Skip to content

ci(boot-test): run the scenarios as a matrix behind one gate, preflight the host tools - #490

Merged
jeefy merged 3 commits into
projectbluefin:mainfrom
jeefy:ci/boot-test-matrix
Oct 18, 2026
Merged

jeefy merged 3 commits into
projectbluefin:mainfrom
jeefy:ci/boot-test-matrix

Conversation

@jeefy

@jeefy jeefy commented Oct 18, 2026

Copy link
Copy Markdown
Collaborator

Two CI wall-clock fixes.

Matrix

boot-test ran ~20 QEMU scenarios serially (~30 min after the build on PRs, ~13 on releases). They now run as a 4-bucket matrix via one script, scripts/ci/boot-test.sh <bucket>, so the YAML stays one short job and --list prints exactly what runs. The script's output for the release (11) and PR (15) sets was diffed against the old YAML: every step name, every DOGFOOD_* value and every command is identical. A bucket stops at its first failure like the serial job did; logs upload per bucket.

Bucket PR Release
diskless (boot, tamper raw/sums, Ignition cred, HTTP boot, 3 signed-config + unsigned-refusal) ~9 min ~3
install (A/B update + boot-counted rollback, DOGFOOD_SYSEXT=zfs,nvidia) ~10 ~1
installer (USB install+update, console, SB-off refusal) ~5 ~5
installer-disks (prior-install, foreign-gpt) ~4 ~4

Estimated critical path after the build: ~13 min on PRs (was ~32), ~6 on releases (was ~13), from step timings of recent runs. Cost: ~3 extra runners × ~2.5 min setup.

A new boot-test-result gate (needs: [build, boot-test], no checkout, permissions: {}) fails unless boot-test succeeded and is skipped when build is — so it works as a required check on validate-only PRs. release and nightly-status gate on it. The ruleset names no status checks today; if one is ever required, use boot-test-result, not the per-bucket checks.

Preflight

boot-test-preflight (~1 min, build now needs: it) runs the same QEMU/OVMF install step — now the single copy, scripts/ci/install-qemu-ovmf.sh, shared with the matrix jobs — then scripts/ci/dogfood-preflight.sh, which command -vs every host tool the three CI dogfood scripts exec, checks the OVMF files and /dev/kvm. A self-check greps the scripts (and the ones they call) for tool names and fails if one is neither on the host list nor the "runs in the guest" list, so a new mcopy (#486) fails in ~2 min, not after a 2 h build.

Host tools: bash python3 qemu-system-x86_64 mcopy sfdisk blkid mkswap mkfs.vfat mkfs.ext4 gpg sha256sum base64 dd od truncate realpath mktemp awk. Guest-only: veritysetup gpgv; not run in CI: mkfs.xfs.

Verification

actionlint + zizmor (pinned images) clean; shellcheck clean; pytest 1495 (new test_ci_boot_test.py, test_ci_dogfood_preflight.py; job-graph assertions updated); ruff/mypy clean; docs-checks pass (ci-tooling.md at 496/500 lines). just validate passes. The new workflow has not run in CI yet — this PR is its first run; full-build added so all four buckets and the preflight execute.

jeefy and others added 3 commits October 17, 2026 19:32
The boot test ran its 15 QEMU scenarios (11 on releases) one after the
other, about 30 minutes after a build. scripts/ci/boot-test.sh now holds
them, unchanged (same dogfood calls and DOGFOOD_* settings), in four
buckets balanced by duration that boot-test runs as a fail-fast: false
matrix: diskless (the nine diskless/HTTP/refusal boots), install (the
A/B update and rollback, the long pole), installer and installer-disks.
installer-disks does not download the image-update artifact.

boot-test-result turns the legs into one result: release and
nightly-status gate on it, and it is skipped like boot-test when nothing
was built, so it can be a required check. The QEMU, disk tools and
OVMF install moves to scripts/ci/install-qemu-ovmf.sh.

Assisted-by: Claude Fable 5.1 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
A host tool missing from the runner (projectbluefin#486: mcopy) showed only once the
boot test ran, after a build of up to 2 h. boot-test-preflight runs the
same scripts/ci/install-qemu-ovmf.sh as boot-test, then
scripts/ci/dogfood-preflight.sh, and build waits for it.

The check runs command -v on a list of host tools read from the dogfood
scripts boot-test runs, checks the OVMF files and that /dev/kvm is
read-writable. Every name a tool pattern matches in those scripts must be
listed, as a host tool or as one the runner never runs (guest probes,
DOGFOOD_TARGET=xfs), so a script that starts using a new tool fails the
preflight until it is listed.

Assisted-by: Claude Fable 5.1 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
ci-tooling.md lists boot-test-preflight and boot-test-result, moves
release and nightly-status onto the gate, points the OVMF pin at
scripts/ci/install-qemu-ovmf.sh, records the boot test's serial and
per-bucket durations, and names boot-test-result as the check to require.

Assisted-by: Claude Fable 5.1 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@github-actions
github-actions Bot requested review from Danathar and castrojo October 18, 2026 01:07
@github-actions

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
To complete the pull request process, please assign ahmedadan after the PR has been reviewed.
You can assign the PR to them by writing /assign @ahmedadan in a comment when ready.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@hivecommons-hive hivecommons-hive Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

HUMAN DECISION NEEDED — @jeefy the four boot-test matrix legs and boot-test-result have not yet run at 14c56bc (run 39086447685: boot-test-preflight passed, build still in progress at review time); a maintainer should confirm all four buckets and the gate go green before merge, since release now gates on them.

Reviewed at head 14c56bc. The split of the serial job into scripts/ci/boot-test.sh was checked against the old YAML: --list prints 15 PR / 11 release scenarios, and every step name, DOGFOOD_* value (including the trailing space in the netdb DOGFOOD_EXPECT) and command matches the removed steps. The preflight's watch pattern (scripts/ci/dogfood-preflight.sh:30) was run against the three dogfood scripts: it finds exactly the names listed in host_tools/not_on_host, and the dogfood scripts call no further helper scripts.

correctness

  • (medium, needs a human) Nothing in the code reading blocks, but the only part that cannot be judged from the diff — four QEMU buckets each downloading the image artifact and running on their own runner (.github/workflows/build.yml:437-470), then needs.boot-test.result aggregating them in boot-test-result (build.yml:495-509) — has no CI evidence yet. The body says so too. Please wait for run 39086447685's boot-test (*) legs and boot-test-result.

intent-alignment

  • (low) The body says the preflight checks "every host tool the three CI dogfood scripts exec". It checks the names the watch regex matches (scripts/ci/dogfood-preflight.sh:30) plus the fixed host_tools list (:16-24); a tool outside that alternation (e.g. a future xorriso) is not caught. docs/skills/ci-tooling.md:220 describes it accurately ("Each name its tool pattern matches"); only the PR body overstates it.
  • (low) The wall-clock accounting omits that build now waits for boot-test-preflight (build.yml:254-258), so on PRs the build starts ~1 min plus runner setup later; on releases kernel-cache already precedes build, so there it is likely hidden.

style

  • (low) here="${BASH_SOURCE[0]%/*}" (scripts/ci/dogfood-preflight.sh:36) keeps the file name when the script is invoked without a directory component (cd scripts/ci && bash dogfood-preflight.sh): queue becomes dogfood-preflight.sh/boot-test.sh, and the run fails with "found no dogfood script" and a spurious "command not found" for every tool. CI always runs it from the repo root, so this is a local-use footgun only; here=$(dirname "${BASH_SOURCE[0]}") fixes it.

No findings from: security, docs-currency.

Confidence: 3/5 (needs attention) — 1 medium finding; 1 perspective requires a human decision

— hive: agent=reviewer backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive

Copy link
Copy Markdown
Contributor
14c56bc

⚠️ Sentinel alert — maintainer review required

Hive flagged this PR (author @jeefy, head 14c56bcb079f) because it matches behaviors that can override security controls, escalate privileges or damage the codebase. This is a heuristic, not an accusation — a maintainer should confirm the change is intended before it merges.

  • sensitive_path — changes 1 sensitive path(s) (Touches a sensitive path (OWNERS, workflows, policies, hive config, security docs, …))
    • .github/workflows/build.yml
  • ci_gate_weakening — CI configuration is deleted, softened or has gate checks removed (Disables or softens CI gates (continue-on-error, || true, removed checks))
    • .github/workflows/build.yml
  • remote_code_execution — adds a remote-fetch-and-execute, decoded payload, reverse shell or embedded credential (Adds curl|sh, eval of encoded payloads, reverse shells or embedded private keys)
    • scripts/ci/install-qemu-ovmf.sh
    • tests/unit/test_release_workflow.py

Hive added the sentinel-alert label. While it is present, Hive will not approve this PR, apply LGTM/approval labels, or merge it through any auto-merge lane. Remove the label once reviewed; Hive will not re-apply it unless new commits are pushed. Tune paths and behaviors under sentinel in hive.yaml or the dashboard Security tab.

@jeefy jeefy added full-build Run the full image build and boot test on this pull request and removed sentinel-alert labels Oct 18, 2026
@jeefy
jeefy merged commit 9e1a950 into projectbluefin:main Oct 18, 2026
59 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

full-build Run the full image build and boot test on this pull request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant