Skip to content

ci-status never fails in a merge group, holding the queue until timeout #1682

Closed

Description

@nathan-weinberg

Merge queue entries whose checks have already failed stay AWAITING_CHECKS instead of being ejected, because the ci-status aggregate check never reports a failure.

What happens

Observed on the queue with #1645 at position 1 and #1649 at position 2. Both merge group SHAs had three checks concluded failure (coverage, lint-xtask, test-inference) ~40 minutes earlier, yet ci-status was still in_progress on both and neither entry had been ejected.

Cause

.github/workflows/ci-status.yaml polls sibling check runs in a while (true) loop. The loop has exactly one non-exceptional exit — every check green. On failure it warns and loops:

if (failed.length > 0) {
  for (const cr of failed) {
    core.warning(`${cr.name} concluded with: ${cr.conclusion} — waiting for re-run`);
  }
  core.info('Re-run the failed job(s) and ci-status will pick up the result automatically.');
  await new Promise(r => setTimeout(r, 30000));
  continue;
}

There is no core.setFailed() anywhere in the script.

Waiting for a re-run is a deliberate choice for pull_request: a human sees the red job, re-runs it, and ci-status picks up the new conclusion without needing a re-run of its own. The premise does not hold for merge_group. The gh-readonly-queue/* ref is ephemeral, nobody is watching it, and no re-run is coming. The workflow handles both events identically.

Impact

ci-status is the sole aggregate required status check in the main ruleset, and the merge queue rule sets check_response_timeout_minutes: 60. So GitHub abandons the entry after 60 minutes while the job itself keeps running to its timeout-minutes: 180. A doomed entry costs ~60 minutes of queue time instead of failing at ~18 minutes, when the first check reported.

This is recurring, not a one-off. Merge-group ci-status run history shows a clear signature — healthy runs finish in ~28-30 min, wedged ones land on exactly 3h00m, the timeout-minutes guillotine:

10-09 10:02:42 -> 13:02:55  cancelled   pr-1648   (3h00m)
10-09 11:34:02 -> 14:34:18  cancelled   pr-1509   (3h00m)
10-09 13:19:51 -> 16:20:08  cancelled   pr-1648   (3h00m)
10-09 23:11:15 -> 02:11:19  cancelled   pr-1648   (3h00m)
...
10-12 13:11:14 -> 13:41:03  success     pr-1645   (0h30m)

#1648 burned four of those slots in one day.

Expected

In a merge group, the first failed check should fail ci-status immediately so GitHub ejects the pull request and starts building the next entry. The pull_request wait-for-re-run behavior should be left alone.

Note on the triggering failure

The failure that exposed this was legitimate — the merge queue was doing its job. #1645 bumps hardcoded inventory counts in tests/utils/src/inference_fixture/coverage.rs, but another PR landed the same 69 -> 70 bump while it sat in review, so rebased it should read 71. A semantic merge conflict, invisible to that PR's own checks. #1649 fails only because it is speculatively stacked behind #1645. That is separate from this bug.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions