Skip to content

test(ci): pin the Playwright browser setup both e2e jobs depend on - #1301

Merged
mrbobbytables merged 1 commit into
mainfrom
quality/test-ci-playwright-setup-contract
Oct 11, 2026
Merged

mrbobbytables merged 1 commit into
mainfrom
quality/test-ci-playwright-setup-contract

Conversation

@hivecommons-hive

Copy link
Copy Markdown
Contributor

Test Improvement

.github/workflows/ci.yml gets its Playwright setup right in both browser jobs
— e2e and e2e-coverage — and nothing held it there. This PR adds the
contract; it changes no workflow and no production code.

Two properties, both already true at main and both unpinned until now:

  1. The browser's OS packages are installed on both sides of the
    ms-playwright cache.
    actions/cache restores the bundle, never the
    packages — those live in the runner image — which is why each job pairs
    --with-deps on a cache miss with install-deps on a cache hit. Drop the
    hit-branch step and the first run is still green; every later run fails as
    browserType.launch: Target page, context or browser has been closed,
    naming neither the missing library nor the skipped step. That is the exact
    symptom tests/contributing-e2e-setup.test.mjs:44-51 already records.
  2. The browser cache key moves with the dependency. The key resolves
    @playwright/test out of package-lock.json; a key that does not change
    when the package does restores the previous release's browser and skips the
    install that would have replaced it, so a version bump silently drives an old
    binary.

tests/ci-playwright-setup.test.mjs follows the scanner idiom of
tests/helpers-ci-supply-chain.mjs: each scanner runs over the real
.github/workflows/** tree, which must be clean, and over synthetic
documents that do offend, so a scanner that stopped detecting cannot leave the
contract green. Deliberate scoping, stated in the file:

  • a job that only installs the browser has nothing to launch and is not held
    to the suite contract;
  • an install guarded on a condition the scanner does not recognise is reported
    rather than guessed at, so rewriting the guard cannot quietly disable the
    check;
  • runner.os alone does not satisfy the key contract: it moves with the image,
    not with the package;
  • both real-tree assertions first assert that a Playwright job (resp. a cached
    bundle) still exists, so the contract cannot pass by finding nothing.

Verification

At 4184f33, node v26.10.0, TZ=UTC:

  • node --test tests/ci-playwright-setup.test.mjs → 11 tests, 0 failures.
  • Non-vacuity, against the real ci.yml (mutated, asserted, reverted):
    deleting both Install Playwright system dependencies steps fails the first
    contract naming ci.yml (e2e) and ci.yml (e2e-coverage); replacing both
    keys with playwright-${{ runner.os }} fails the second, naming the same two
    jobs. Before this PR, both mutations leave the entire suite green.
  • npm run test:unit:coverage:check (full gate flags) → 2328 tests, 0
    failures, exit 0
    ; src files | 100.00 | 100.00 | 9135/9135 lines | 2753/2753 regions unchanged.
  • npx prettier --check clean; npm run check:spelling → 0 issues.

Related Issue

Closes #1300


Human review required.

— hive: agent=quality backend=copilot model=claude-opus-5 copilot=1.0.88

.github/workflows/ci.yml installs the browser's OS packages on both sides
of the ms-playwright cache and keys that cache on the resolved
@playwright/test version. Nothing asserted either property: deleting both
'Install Playwright system dependencies' steps, or replacing the key with
playwright-${{ runner.os }}, leaves the whole suite green.

Both regressions surface late and name the wrong cause -- a cache-hit run
without the OS packages fails as 'Target page, context or browser has been
closed', the symptom tests/contributing-e2e-setup.test.mjs:44-51 already
records, and a version-independent key runs a new Playwright against the
previous release's binary.

Adds tests/ci-playwright-setup.test.mjs in the scanner idiom of
tests/helpers-ci-supply-chain.mjs: each scanner runs over the real
.github/workflows tree, which must be clean, and over synthetic documents
that do offend, so a scanner that stopped detecting cannot pass. An
install guarded on an unrecognised condition is reported rather than
guessed at. No workflow or production change; ci.yml already complies.

Closes #1300

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: quality <quality@hive.kubestellar.io>
@hivecommons-hive

Copy link
Copy Markdown
Contributor Author

Important

Held for human review by the hive's ACMM level gate.

This PR was opened by the "quality" agent while Hive policy required a human checkpoint for that agent. Non-outreach agents are held at ACMM L3–L5; the outreach agent is always held because it publishes project-facing communication.

Hive will keep the hold label until a human removes it. Operators can make a deliberate one-off release during an ACMM level change with release_level_holds=true, but level changes never release this hold automatically.

@mrbobbytables
mrbobbytables added this pull request to the merge queue Oct 11, 2026
Merged via the queue into main with commit 9efcb92 Oct 11, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[quality] nothing holds ci.yml's Playwright setup: the OS-package install on a cache hit and the version-keyed browser cache are both unpinned

1 participant