Refresh the realm-server shard weights from CI - #6002
Open
backspace wants to merge 2 commits into
Open
Conversation
Contributor
Preview deploymentsHost Test Results 1 files ± 0 1 suites ±0 2h 3m 26s ⏱️ + 7m 51s Results for commit b81a002. ± Comparison against earlier commit dba28c2. Realm Server Test Results 1 files ± 0 199 suites ±0 1h 11m 8s ⏱️ + 2m 22s Results for commit b81a002. ± Comparison against earlier commit dba28c2. |
backspace
force-pushed
the
cs-12753-automate-realm-server-shard-weights
branch
2 times, most recently
from
September 4, 2026 14:47
f8fbcfb to
dba28c2
Compare
backspace
marked this pull request as ready for review
September 4, 2026 21:30
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
The weights that decide which shard each test file runs on were regenerated by hand or not at all, so they decay: every file added between refreshes packs at DEFAULT_WEIGHT whatever it actually costs, and every file that gets slower keeps its old number. Measured just now, the committed weights already cost 127s on the predicted slowest shard versus weights taken from a current run. `realm-server-shard-timings-update` follows the two jobs that already do this — matrix's spec timings and the host's memory baselines — closely enough to be read side by side: push-to-main only, since it is a job holding `contents: write`; a staleness guard that stands down when newer code has landed, because that push has its own run; and a fetch-and-retry loop for the pushes it races. It hangs off the merge job rather than the shards, since that is what produces the merged report, and it runs whether or not the shards passed — files missing from a failed shard's report keep their committed values, and skipping red runs would leave the weights to rot through a flaky week. Realm-server's version can be simpler than the host's. Host CI has to pick a run whose *shape* matches the runs the weights are consumed by (CS-12582), because Percy and the index cache make a push run and a PR run different animals. Realm-server has neither, so the current run will do. The generator gains `--min-drift-seconds` and `--shard-count`, matching its host counterpart: it predicts the slowest shard under the committed weights and under the regenerated ones, both scored against the regenerated ones, and leaves the file alone unless the difference is worth a commit. Otherwise every main push would commit its own jitter. The prediction packs with shard-test-modules.cjs itself rather than a copy, so it cannot drift from the assignment it is predicting — the host's equivalent carries a comment warning about exactly that. Two things fail the job rather than committing something worse: the attribution floor, already there, and now a committed weights file that exists but does not parse. Swallowing that dropped every weight the current report did not cover, quietly, since the result still looks like a well-formed refresh. shard-test-modules.cjs already drew that distinction. The staleness guards in all three jobs now share one pattern listing all three commit subjects. The host's listed only its own, so a matrix timings commit already read to it as newer code and made it stand down; adding a third committer without this would have made that worse. Verified against the real merged report from run 33767809502: the gate opens at a 127s improvement, holds at 0s on a repeat, errors when given a drift threshold with no shard count, and exits 1 on an unparseable weights file. CS-12753. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The generator's list of files with no recorded duration is the staleness signal CS-12753 is built around, and with the drift gate in place the common outcome of a main run is "within the threshold, exit 0" — which happened before the list was printed. Move it, and the ambiguous-suite warning, ahead of the gate so every run reports what it could not measure. Also point from the realm-server-test matrix to the generator's --shard-count, the way ci-host.yaml does, so the number is found from either end. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
backspace
force-pushed
the
cs-12753-automate-realm-server-shard-weights
branch
from
September 4, 2026 21:32
dba28c2 to
b81a002
Compare
backspace
changed the base branch from
cs-12759-realm-server-tests-that-never-run
to
main
September 4, 2026 21:32
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Now that realm server tests are sorted into shards based on duration, this adds the rebalancing job on
mainthat other such suites get.Claude: The weights that decide which shard each test file runs on were regenerated by hand or not at all, so they decay: every file added between refreshes packs at DEFAULT_WEIGHT whatever it actually costs, and every file that gets slower keeps its old number. Measured just now, the committed weights already cost 127s on the predicted slowest shard versus weights taken from a current run.
realm-server-shard-timings-updatefollows the two jobs that already do this — matrix's spec timings and the host's memory baselines — closely enough to be read side by side: push-to-main only, since it is a job holdingcontents: write; a staleness guard that stands down when newer code has landed, because that push has its own run; and a fetch-and-retry loop for the pushes it races.It hangs off the merge job rather than the shards, since that is what produces the merged report, and it runs whether or not the shards passed — files missing from a failed shard's report keep their committed values, and skipping red runs would leave the weights to rot through a flaky week.
Realm-server's version can be simpler than the host's. Host CI has to pick a run whose shape matches the runs the weights are consumed by (CS-12582), because Percy and the index cache make a push run and a PR run different animals. Realm-server has neither, so the current run will do.
The generator gains
--min-drift-secondsand--shard-count, matching its host counterpart: it predicts the slowest shard under the committed weights and under the regenerated ones, both scored against the regenerated ones, and leaves the file alone unless the difference is worth a commit. Otherwise every main push would commit its own jitter. The prediction packs with shard-test-modules.cjs itself rather than a copy, so it cannot drift from the assignment it is predicting — the host's equivalent carries a comment warning about exactly that.Two things fail the job rather than committing something worse: the attribution floor, already there, and now a committed weights file that exists but does not parse. Swallowing that dropped every weight the current report did not cover, quietly, since the result still looks like a well-formed refresh. shard-test-modules.cjs already drew that distinction.
The staleness guards in all three jobs now share one pattern listing all three commit subjects. The host's listed only its own, so a matrix timings commit already read to it as newer code and made it stand down; adding a third committer without this would have made that worse.
Verified against the real merged report from run 33767809502: the gate opens at a 127s improvement, holds at 0s on a repeat, errors when given a drift threshold with no shard count, and exits 1 on an unparseable weights file.
CS-12753.