Repository navigation
ci(boot-test): run the scenarios as a matrix behind one gate, preflight the host tools - #490
Conversation
The boot test ran its 15 QEMU scenarios (11 on releases) one after the other, about 30 minutes after a build. scripts/ci/boot-test.sh now holds them, unchanged (same dogfood calls and DOGFOOD_* settings), in four buckets balanced by duration that boot-test runs as a fail-fast: false matrix: diskless (the nine diskless/HTTP/refusal boots), install (the A/B update and rollback, the long pole), installer and installer-disks. installer-disks does not download the image-update artifact. boot-test-result turns the legs into one result: release and nightly-status gate on it, and it is skipped like boot-test when nothing was built, so it can be a required check. The QEMU, disk tools and OVMF install moves to scripts/ci/install-qemu-ovmf.sh. Assisted-by: Claude Fable 5.1 via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
A host tool missing from the runner (projectbluefin#486: mcopy) showed only once the boot test ran, after a build of up to 2 h. boot-test-preflight runs the same scripts/ci/install-qemu-ovmf.sh as boot-test, then scripts/ci/dogfood-preflight.sh, and build waits for it. The check runs command -v on a list of host tools read from the dogfood scripts boot-test runs, checks the OVMF files and that /dev/kvm is read-writable. Every name a tool pattern matches in those scripts must be listed, as a host tool or as one the runner never runs (guest probes, DOGFOOD_TARGET=xfs), so a script that starts using a new tool fails the preflight until it is listed. Assisted-by: Claude Fable 5.1 via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
ci-tooling.md lists boot-test-preflight and boot-test-result, moves release and nightly-status onto the gate, points the OVMF pin at scripts/ci/install-qemu-ovmf.sh, records the boot test's serial and per-bucket durations, and names boot-test-result as the check to require. Assisted-by: Claude Fable 5.1 via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: Needs approval from an approver in each of these files:Approvers can indicate their approval by writing |
There was a problem hiding this comment.
HUMAN DECISION NEEDED — @jeefy the four boot-test matrix legs and boot-test-result have not yet run at 14c56bc (run 39086447685: boot-test-preflight passed, build still in progress at review time); a maintainer should confirm all four buckets and the gate go green before merge, since release now gates on them.
Reviewed at head 14c56bc. The split of the serial job into scripts/ci/boot-test.sh was checked against the old YAML: --list prints 15 PR / 11 release scenarios, and every step name, DOGFOOD_* value (including the trailing space in the netdb DOGFOOD_EXPECT) and command matches the removed steps. The preflight's watch pattern (scripts/ci/dogfood-preflight.sh:30) was run against the three dogfood scripts: it finds exactly the names listed in host_tools/not_on_host, and the dogfood scripts call no further helper scripts.
correctness
- (medium, needs a human) Nothing in the code reading blocks, but the only part that cannot be judged from the diff — four QEMU buckets each downloading the
imageartifact and running on their own runner (.github/workflows/build.yml:437-470), thenneeds.boot-test.resultaggregating them inboot-test-result(build.yml:495-509) — has no CI evidence yet. The body says so too. Please wait for run 39086447685'sboot-test (*)legs andboot-test-result.
intent-alignment
- (low) The body says the preflight checks "every host tool the three CI dogfood scripts exec". It checks the names the
watchregex matches (scripts/ci/dogfood-preflight.sh:30) plus the fixedhost_toolslist (:16-24); a tool outside that alternation (e.g. a futurexorriso) is not caught.docs/skills/ci-tooling.md:220describes it accurately ("Each name its tool pattern matches"); only the PR body overstates it. - (low) The wall-clock accounting omits that
buildnow waits forboot-test-preflight(build.yml:254-258), so on PRs the build starts ~1 min plus runner setup later; on releaseskernel-cachealready precedesbuild, so there it is likely hidden.
style
- (low)
here="${BASH_SOURCE[0]%/*}"(scripts/ci/dogfood-preflight.sh:36) keeps the file name when the script is invoked without a directory component (cd scripts/ci && bash dogfood-preflight.sh):queuebecomesdogfood-preflight.sh/boot-test.sh, and the run fails with "found no dogfood script" and a spurious "command not found" for every tool. CI always runs it from the repo root, so this is a local-use footgun only;here=$(dirname "${BASH_SOURCE[0]}")fixes it.
No findings from: security, docs-currency.
Confidence: 3/5 (needs attention) — 1 medium finding; 1 perspective requires a human decision
— hive: agent=reviewer backend=copilot model=claude-fable-5.1 copilot=1.0.88
14c56bc
|
Two CI wall-clock fixes.
Matrix
boot-testran ~20 QEMU scenarios serially (~30 min after the build on PRs, ~13 on releases). They now run as a 4-bucket matrix via one script,scripts/ci/boot-test.sh <bucket>, so the YAML stays one short job and--listprints exactly what runs. The script's output for the release (11) and PR (15) sets was diffed against the old YAML: every step name, everyDOGFOOD_*value and every command is identical. A bucket stops at its first failure like the serial job did; logs upload per bucket.diskless(boot, tamper raw/sums, Ignition cred, HTTP boot, 3 signed-config + unsigned-refusal)install(A/B update + boot-counted rollback,DOGFOOD_SYSEXT=zfs,nvidia)installer(USB install+update, console, SB-off refusal)installer-disks(prior-install, foreign-gpt)Estimated critical path after the build: ~13 min on PRs (was ~32), ~6 on releases (was ~13), from step timings of recent runs. Cost: ~3 extra runners × ~2.5 min setup.
A new
boot-test-resultgate (needs: [build, boot-test], no checkout,permissions: {}) fails unlessboot-testsucceeded and is skipped whenbuildis — so it works as a required check on validate-only PRs.releaseandnightly-statusgate on it. The ruleset names no status checks today; if one is ever required, useboot-test-result, not the per-bucket checks.Preflight
boot-test-preflight(~1 min,buildnowneeds:it) runs the same QEMU/OVMF install step — now the single copy,scripts/ci/install-qemu-ovmf.sh, shared with the matrix jobs — thenscripts/ci/dogfood-preflight.sh, whichcommand -vs every host tool the three CI dogfood scripts exec, checks the OVMF files and/dev/kvm. A self-check greps the scripts (and the ones they call) for tool names and fails if one is neither on the host list nor the "runs in the guest" list, so a newmcopy(#486) fails in ~2 min, not after a 2 h build.Host tools: bash python3 qemu-system-x86_64 mcopy sfdisk blkid mkswap mkfs.vfat mkfs.ext4 gpg sha256sum base64 dd od truncate realpath mktemp awk. Guest-only: veritysetup gpgv; not run in CI: mkfs.xfs.
Verification
actionlint + zizmor (pinned images) clean; shellcheck clean; pytest 1495 (new
test_ci_boot_test.py,test_ci_dogfood_preflight.py; job-graph assertions updated); ruff/mypy clean; docs-checks pass (ci-tooling.mdat 496/500 lines).just validatepasses. The new workflow has not run in CI yet — this PR is its first run;full-buildadded so all four buckets and the preflight execute.