Skip to content

feat(reporter): print a dashboard link per failed test - #429

Open
PhenX wants to merge 5 commits into
mainfrom
claude/audit-quick-wins-reporter-alerts
Open

feat(reporter): print a dashboard link per failed test#429
PhenX wants to merge 5 commits into
mainfrom
claude/audit-quick-wins-reporter-alerts

Conversation

@PhenX

@PhenX PhenX commented Sep 2, 2026

Copy link
Copy Markdown
Collaborator

What & why

The reporter-and-alerts slice of the failing-run data audit (quick wins 8 and 9): a developer whose test failed should get a link to that exact execution where they already are — the terminal, the CI summary, the Slack/email alert, the pull-request comment — and the quoted error should say what Playwright was waiting on, not just that a timeout elapsed.

8a — Per-failure links from the reporter

  • apps/application/server/routes/test-runs/[id]/locate.get.ts (new) — GET /test-runs/:id/locate?file=…&title=…&retry=…&browser=… resolves an execution from what the reporter knows before ids exist and redirects to /test-run-cases/:id. retry/browser are preferences (exact attempt when persisted, else the failing one, else the latest), an unknown test renders a readable 404 page with a link to the run, and a signed-out visitor goes through /login and back. The events route returns no execution ids to the reporter, so this is the deterministic-link approach the audit suggested. It lives under server/routes/ (not server/api/), so the demo router check needs no mirror.
  • apps/application/app/pages/login.vue — honours a same-origin ?redirect= after sign-in so those links survive an auth-enabled dashboard.
  • packages/reporter/src/internal/support/failure-links.ts (new), src/public/reporter.ts, src/internal/submit/run-submitter.ts — one [Piwi Dashboard] ✗ <title> → <url> line per test whose final attempt failed (result.retry >= test.retries), printed the moment the streaming run id is known, or after the submit in batch mode; never twice.
  • apps/application/tests/run-locate.spec.ts (new, project run-locate-test registered in shared/test-project-names.ts), packages/reporter/tests/failure-links.spec.ts (new).

8b — GitHub step summary and outputs

  • packages/reporter/src/internal/support/ci-output.ts$GITHUB_STEP_SUMMARY lists the failed tests with their links under the existing "Piwi test run" heading (20 max, then +N more); piwi_failed_count joins the step outputs. The GitLab dotenv gains PIWI_FAILED_COUNT, and the JSON output file gains failedCount + failures so any CI can read the same list.
  • packages/reporter/tests/ci-output.spec.ts extended.

8c — Body-only attachments

  • packages/reporter/src/internal/files/file-handler.tstestInfo.attach(name, { body }) attachments are staged as temp files under os.tmpdir() (one file per attachment, reused across the repeated lookups the upload paths make, removed in onEnd). One size ceiling (500 MB, the dashboard's default multipart limit) applies to inline and path-backed attachments alike; an oversized one is skipped with a single warning naming it instead of failing the whole upload. Applying the cap to path-backed files too is a small behaviour change: before, a >500 MB video made the multipart upload fail with a 413.
  • packages/reporter/tests/file-handler.spec.ts (new).

9a — Alerts link to the execution

  • apps/application/shared/notification-events.tsfailureTargetPath(): /test-run-cases/<executionId>, falling back to /test-cases/<testCaseId>.
  • server/utils/notifications/dispatch.ts (Slack) and server/utils/email.ts (run email, HTML + text) use it. The digest renderers link per event to the run, which is unchanged and correct for a digest line.

9b — Excerpts quote the message head

  • shared/error-fingerprint.ts exports extractMessageHead; shared/notification-events.ts adds errorExcerpt(): the head (before Call log: and the first stack frame, ≤5 lines), with the last waiting for … / locator resolved to … call-log line appended when the head is only a bare timeout, ANSI stripped, capped. buildTopFailures and the cluster.new sample excerpt use it.
  • server/utils/scm/pr-feedback.ts uses the same helper with PR_EXCERPT_MAX (200, exported from shared/pr-feedback.ts) so the pull-request comment quotes failures the same way.
  • tests/unit/notification-events.test.ts, tests/unit/pr-feedback.test.ts extended.

Docs (same commits)

apps/docs/ci.md (terminal lines, step summary, new outputs, JSON file shape), apps/docs/reporter.md (streaming step, body attachments and the size ceiling), apps/docs/notifications.md (errorExcerpt semantics and links), packages/reporter/README.md, packages/reporter/ARCHITECTURE.md.

Nothing was left out. The audit document itself is not on the branch.

How was it tested?

  • Reporter (packages/reporter/): reporter:build, reporter:typecheck, reporter:lint, reporter:format:check, reporter:test — 46 files, 674 tests passing.
  • App (apps/application/): app:typecheck, app:lint, app:format:check, app:test:unit (120 files, 1622 tests), app:check:demo.
  • E2E: tests/run-locate.spec.ts, tests/notifications.spec.ts, tests/email-notifications.spec.ts, tests/browser-notifications.spec.ts against a dev server — 12 passed, the 46 auth-enabled cases skip outside CI as configured.
  • npx commitlint --from main --to HEAD passes for every commit.

Checklist

  • PR title follows Conventional Commits (type(scope): subject)
  • Tests added/updated for behavior changes
  • Docs updated if user-facing (apps/docs/, README, or reporter README)

🤖 Generated with Claude Code

https://claude.ai/code/session_0145ws9TJpSi3vjZfzdBmB4c


Generated by Claude Code

Add GET /test-runs/:id/locate so a link can be built from what the reporter
knows before execution ids exist (run id, spec file, title, retry, project)
and redirect to /test-run-cases/:id. An unknown test renders a readable 404
page; a signed-out visitor is sent through /login and back to the link.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0145ws9TJpSi3vjZfzdBmB4c
Print one `✗ <title> → <url>` line per test whose final attempt failed, as
soon as the run id is known (immediately while streaming, after the submit in
batch mode). The link targets the dashboard's locate route so it needs no
execution id. GitHub Actions gets the same list in the job summary (capped at
20) and a piwi_failed_count step output; the GitLab dotenv and the JSON
output file carry the count and the failures too.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0145ws9TJpSi3vjZfzdBmB4c
testInfo.attach(name, { body }) attachments were skipped silently because
only path-backed ones were collected. Stage a body as a temp file under
os.tmpdir() for the upload (removed when the run ends) and apply one size
ceiling to inline and path-backed attachments alike, warning once per
attachment when it is skipped.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0145ws9TJpSi3vjZfzdBmB4c
…head

Slack and email run notifications linked each failure to the test's history
page even though the payload carries the execution id; link to
/test-run-cases/:id and fall back to the history page only without one.

The excerpt embedded in notifications, cluster alerts and pull-request
comments was the raw first characters of the error, so a timeout read as a
bare "Timeout 30000ms exceeded" with the call log cut off. Build it from the
message head shared with the fingerprint, and append the last call-log state
line when the head is only a timeout.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0145ws9TJpSi3vjZfzdBmB4c
@github-actions

github-actions Bot commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

Coverage Report for Reporter (./packages/reporter)

Status Category Percentage Covered / Total
🔵 Lines 75.38% 2334 / 3096
🔵 Statements 73.72% 2655 / 3601
🔵 Functions 78.81% 413 / 524
🔵 Branches 67.43% 1847 / 2739
File Coverage
File Stmts Branches Functions Lines Uncovered Lines
Changed Files
packages/reporter/src/internal/files/file-handler.ts 78.12% 64.77% 90.9% 80.9% 61-70, 87-88, 129, 135, 150, 165-166, 198, 202-203, 211-212, 220-221, 228, 232-233, 244-245, 259, 282, 290
packages/reporter/src/internal/submit/run-submitter.ts 83.33% 80.28% 100% 83.33% 112-114, 150, 236-237, 246-255, 260-262, 304-306
packages/reporter/src/internal/support/ci-output.ts 96.15% 86.48% 100% 95.12% 159, 168
packages/reporter/src/internal/support/failure-links.ts 100% 100% 100% 100%
packages/reporter/src/public/reporter.ts 76.92% 59.77% 91.66% 80.89% 149-150, 155, 178-187, 201-202, 251, 269, 354-360, 364-368, 379, 382-383, 395-405, 413, 419-424, 427, 431, 484
Generated in workflow #1483 for commit 81d6773 by the Vitest Coverage Report Action

@github-actions

github-actions Bot commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

Coverage Report for Application (./apps/application)

Status Category Percentage Covered / Total
🔵 Lines 43.28% 5728 / 13234
🔵 Statements 42.53% 6669 / 15678
🔵 Functions 42.41% 1113 / 2624
🔵 Branches 38.86% 4993 / 12846
File Coverage
File Stmts Branches Functions Lines Uncovered Lines
Changed Files
apps/application/server/utils/email.ts 2.94% 0% 0% 3% 34-86, 91-312
apps/application/server/utils/notifications/dispatch.ts 2.53% 0% 0% 2.94% 27-318
apps/application/server/utils/notifications/run-notifications.ts 0% 0% 0% 0% 4-161
apps/application/server/utils/scm/pr-feedback.ts 0% 0% 0% 0% 47-349
apps/application/shared/error-fingerprint.ts 94.24% 87.2% 100% 95.04% 159-163, 190, 219, 231, 310-311, 318
apps/application/shared/notification-events.ts 96.52% 88.03% 100% 97.77% 218, 276, 314
apps/application/shared/pr-feedback.ts 89.32% 80.89% 100% 93.25% 131, 132, 134, 142, 145, 150, 166-167, 172-173, 231-233
apps/application/shared/test-project-names.ts 0% 100% 100% 0% 15-145
Generated in workflow #1483 for commit 81d6773 by the Vitest Coverage Report Action

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants