Checkout failure: "Git fetch failed with exit code 128" #9009

Activity

1. carlossanlop

   added

   Ops - First Responder

   Detected By - CustomerIssue was reported by a customerIssue was reported by a customer

   Known Build Error

   on Apr 8, 2022
2. ChadNedzlek commented on Apr 12, 2022

   

   ChadNedzlek

   on Apr 12, 2022

   Contributor

   More actions

   So this is... interesting. I agree this seems like essentially a random network failure that is not the fault of the code. Also... it doesn't seem like there is anything to be done other than retry the build.

   @ulisesh and @AlitzelMendez. This seems like another instance of a "known issue" that would really benefit from the ability to get retried automatically. I think Carlos did the right thing by opening the known issue, but his PR (and any future PR that hits this) is still going to be blocked, since the build doesn't run at all in this case, because the checkout failed. And it would be great to help folks that hit this.
3. ulisesh commented on Apr 13, 2022

   

   ulisesh

   on Apr 13, 2022

   Contributor

   More actions

   I created the issue to track how often it was happening and measure the impact of the problem. It might be hard to get traction on networking problems but having the number of impacted builds has great value in my opinion.

   In the last 24 hours, we had 37 builds failed with the same error. cc @markwilkie who might find this data interesting.

   I think automatic retries for known issues will be the next feature after we are done with analyzing builds retroactively, we can discuss this if people think we should change priorities.
4. markwilkie commented on Apr 13, 2022

   

   markwilkie

   on Apr 13, 2022

   Contributor

   More actions

   So 37 builds were caught with the known issues feature @ulisesh ? Could I get the query please? (super exciting)
5. ulisesh commented on Apr 13, 2022

   

   ulisesh

   on Apr 13, 2022

   Last edited by ulisesh

   Contributor

   More actions

   I saw 37 hits in the office but now I only see 33 🤔. There are 33 failed jobs in 2 different builds. Sorry if I misused the terms but I still think it is great that we can track these build breaks 😄

   This is the query:

   KnownIssues
   | where IssueId == 9009 and IssueRepository == "dotnet/arcade"
6. ilyas1974

   self-assigned this

   on Apr 19, 2022
7. ilyas1974

   removed their assignment

   on Jun 22, 2022
8. ilyas1974

   removed

   Ops - First Responder

   Detected By - CustomerIssue was reported by a customerIssue was reported by a customer

   on Jun 22, 2022
9. danmoseley

   changed the title [-]Checkout dotnet/runtime step failure: "Git fetch failed with exit code 128"[/-] [+]Checkout failure: "Git fetch failed with exit code 128"[/+] on Oct 7, 2022
10. ViktorHofer commented on Jan 11, 2023

    

    ViktorHofer

    on Jan 11, 2023

    Last edited by ViktorHofer

    Member

    More actions

    This happened 36 times in the last 24 hours. Should we add a retry (assuming that's possible)?
11. premun commented on Jan 11, 2023

    

    premun

    on Jan 11, 2023

    Member

    More actions

    GitHub reported degraded git operations yesterday: https://www.githubstatus.com/incidents/00km0kwpz23b
    We saw problems cloning all over. Maybe it's related to that?
12. dougbu commented on Jan 8, 2024

    

    dougbu

    on Jan 8, 2024

    Contributor

    More actions

    This issue may have started in early 2022 but seems to remain w/ 29 matching failures in the last 24 hours. https://www.githubstatus.com/ and various Azure / Azure DevOps status pages aren't showing any current or recent degradations. Unclear what's going on this time beyond what appears to be DNS flakiness. I looked at the last 7 reports; all but one showed fatal: unable to access 'https://github.com/dotnet/runtime/': Could not resolve host: github.com and a few also showed fatal: unable to access 'https://github.com/dotnet/runtime/': Failed to connect to github.com port 443 after 75012 ms: Couldn't connect to server.

    Without more information, I think the best we could do is auto retry when the problem occurs. @AlitzelMendez and @carlossanlop what are your thoughts on including "BuildRetry": true in the JSON content in this description❔
13. carlossanlop commented on Jan 8, 2024

    

    carlossanlop

    on Jan 8, 2024

    ContributorAuthor

    More actions

    It's fine, we can try that, as long as we monitor it closely, because if we keep seeing the failure even with a retry, I think we should just go back to "BuildRetry": false, to avoid wasting resources.
14. dougbu commented on Jan 8, 2024

    

    dougbu

    on Jan 8, 2024

    Contributor

    More actions

    you mean "BuildRetry": true was tried in the past and didn't help much @carlossanlop❔ if the extra time it takes PR creators to click the Rerun failed builds actually helps avoid issues in the retry, I'd lean toward leaving things as-is.
15. carlossanlop commented on Jan 8, 2024

    

    carlossanlop

    on Jan 8, 2024

    ContributorAuthor

    More actions

    I do not know if it was tried in the past.
16. AlitzelMendez commented on Jan 8, 2024

    

    AlitzelMendez

    on Jan 8, 2024

    Member

    More actions

    sounds like a good plan @dougbu, let's try "BuildRetry": true, I will put together a query to monitor if it is helpful for this issue
17. dougbu commented on Jan 8, 2024

    

    dougbu

    on Jan 8, 2024

    Contributor

    More actions

    updated the description to try retries
18. AlitzelMendez commented on Jan 8, 2024

    

    AlitzelMendez

    on Jan 8, 2024

    Member

    More actions

    query:

    let buildIds =
    KnownIssues
    | where ingestion_time() > datetime(2024-01-08 23:10:44.8683418)
    | where IssueId  == 9009
    | distinct BuildId;
    TimelineBuilds
    | where BuildId  in (buildIds)
    | summarize arg_max(FinishTime, *) by BuildId
    | summarize count() by Result, bin(FinishTime, 1d)
    | order by FinishTime asc 
    | render columnchart with (xcolumn=FinishTime)
    

    I will check this on Monday to see how it is going, posting here the query so I don't loose it :)
19. dougbu

    assigned

    AlitzelMendez

    on Jan 9, 2024
20. AlitzelMendez commented on Jan 16, 2024

    

    AlitzelMendez

    on Jan 16, 2024

    Member

    More actions

    the success rate (having a green build) after the retry was between 30% to 50% per day, so this doesn't solve the issue, it just helps to mitigate it a little bit.
21. AlitzelMendez

    removed their assignment

    on Jan 16, 2024
22. ilyas1974

    added

    Ops - FacilelyOperations issues that are easily accomplished or attainedOperations issues that are easily accomplished or attained

    on Feb 21, 2024
23. garath

    added

    Ops - P2Operations task, priority 2Operations task, priority 2

    on Mar 28, 2024
24. ilyas1974

    removed

    Ops - P2Operations task, priority 2Operations task, priority 2

    on Jul 25, 2024