Skip to content

ci: accept a clean 0-violation outcome from the watcher-examples scan - #199

Merged
Bracciata merged 4 commits into
mainfrom
fix/ci-own-watcher-fixture
Nov 13, 2026
Merged

Bracciata merged 4 commits into
mainfrom
fix/ci-own-watcher-fixture

Conversation

@Bracciata

@Bracciata Bracciata commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

run-devhub-action has failed on every PR since at least 2026-10-21 (#196, #197, #198): Unexpected outcome: success, issue_count: 0.

Cause: the job scans the-internet.herokuapp.com/login through watcher-examples, and that page no longer has WCAG 2.x A/AA violations. DevHub correctly reports 0 and the action passes, but "Verify expected failure" required a failure.

Fix: treat the clean result as the expected one, in line with #182 (this repo's CI points at watcher-examples rather than keeping its own example). The job now asserts the action succeeds with issue_count 0 and sets project_id and axe_url. The sticky-comment step now checks the success path: no visible axe DevHub comment is left, because the action hides any earlier one.

Only .github/workflows/tests.yml changes against main. The earlier in-repo fixture, its scan job and the Dependabot change were dropped after review.

Follow-up (not in this PR): if we want CI to cover the failure path again, add an example with known violations to watcher-examples and test it from a separate workflow.

Verified locally: actionlint is clean, and node --test test/ passes 24/24. I ran the GraphQL comment count against this PR, and it found the one visible comment left by the earlier fixture run.

No QA required

The job scanned the-internet.herokuapp.com/login via watcher-examples. That
page now has no WCAG 2.x A/AA violations, so the action correctly passed and
"Verify expected failure" has failed every PR since at least 2026-10-21.

Scan ci/watcher-fixture/fixture.html instead: four deliberate violations
(button-name, html-has-lang, image-alt, label), served on 127.0.0.1 and
scanned with pinned @axe-core/watcher 4.7.0 + playwright 1.64.1.

No QA required
The dev test project is locked to dequelabs/watcher-examples and refused the
run: "This project is locked to a different repository".

No QA required
@github-actions

This comment has been minimized.

- Point the npm Dependabot entry at ci/watcher-fixture, the only npm
  manifest, and stop ignoring @axe-core/watcher there, so CI keeps
  scanning with the release customers install.
- Rename run-watcher-tests to scan-watcher-fixture; it runs no tests now.
- Give that job a read-only token and don't persist it in the checkout.
- Match scan.mjs to test/pr-comment.test.mjs style.

No QA required
@Bracciata
Bracciata marked this pull request as ready for review November 12, 2026 15:54
@Bracciata
Bracciata requested a review from a team as a code owner November 12, 2026 15:54
@Bracciata
Bracciata enabled auto-merge (squash) November 12, 2026 15:58

@Zidious Zidious left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should not maintain fixtures here

Comment thread .github/dependabot.yml Resolved
Comment thread .github/workflows/tests.yml Resolved
Comment thread ci/watcher-fixture/scan.mjs Resolved
@Bracciata Bracciata changed the title ci: scan an in-repo fixture so run-devhub-action has violations to find ci: accept a clean 0-violation result in run-devhub-action Nov 12, 2026
@Bracciata Bracciata changed the title ci: accept a clean 0-violation result in run-devhub-action ci: accept a clean 0-violation outcome from the watcher-examples scan Nov 12, 2026
@Bracciata
Bracciata requested a review from Zidious November 12, 2026 22:26
@Bracciata
Bracciata merged commit ee44393 into main Nov 13, 2026
18 of 19 checks passed
@Bracciata
Bracciata deleted the fix/ci-own-watcher-fixture branch November 13, 2026 15:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants