fix(app): gate healing on resolution failures and prefer same-environment baselines - #431
Open
PhenX wants to merge 7 commits into
Open
fix(app): gate healing on resolution failures and prefer same-environment baselines#431PhenX wants to merge 7 commits into
PhenX wants to merge 7 commits into
Conversation
Healing suggestions only make sense when the failing locator never found its element. A toHaveCount mismatch (locator resolved to 51 elements), a disabled element, a hidden element or a page.goto timeout got a Recommended fix that narrowed or replaced a locator that already worked, and the run page showed a Locator fix signal built from ARIA guesses for navigation errors. classifyLocatorResolution reads Playwright's call log: healing applies when the locator never resolved, resolved to 0 elements, or hit a strict-mode violation; it does not when a later 'locator resolved to' line (or an action phase such as 'element is not enabled') shows it resolved, or when the error is a navigation failure. resolveHealingForCase returns applicable: false with a one-line reason before any ladder rung (the ARIA fallback included), so the batch variant, the run's failure groups, the auto-heal policy, the fix plan, the MCP tools and the AI context all inherit the gate. The panel shows the reason as one line instead of a ranked menu. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011oW4GertDgvoFGdtGGoj5U
… diffs The environment diff picked the test's last passing execution on the same browser and nothing else, so a development failure was diffed against a production pass and the diff listed the environment label itself as the change. The visual diff preferred the same branch but ignored the environment. Both now rank candidates with one shared, unit-tested ordering: same environment first, then same branch, then recency. When no passing run from the failing run's environment exists the fallback still happens, but the result carries a baselineNote the cards and the AI context show, and the environment label is never reported as a diff entry in that case. Regression signals and run insights are unchanged. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011oW4GertDgvoFGdtGGoj5U
The reporter sends startTime per network request but network_requests had no column for it, so the value was dropped on ingest and the AI context's t+Nms offset for failed requests read a field that was always undefined. network_requests gains a nullable start_time (Unix epoch ms) in both dialects with generated migrations; the shared insert builder persists it from the wire, the execution response and the run-level endpoint summaries (firstStartTime / lastStartTime) expose it, the AI-context offset reads the stored value, the demo seed and simulator stamp sequential start times, and the execution page orders captured requests by start time when present. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011oW4GertDgvoFGdtGGoj5U
A cluster could read Fix verified and Open at the same time: verification recorded the verdict but never touched the triage status, and nothing told anyone a fix had landed or come undone. A diagnosis-verified verdict now resolves an open cluster and a regressed verdict reopens a resolved one, each appending a system line to the triage note (Resolved automatically: diagnosis verified in run #N / Reopened automatically: regressed in run #N) so the status stays auditable and overridable. Stopped-failing alone changes nothing and ignored clusters are never touched; the verdict badge stays separate from the status. Two events join the subscription list: cluster.fixed for every recorded fix (with the verdict in its payload) and cluster.regressed, rendered for browser, Slack, email digests and webhooks, and documented with the verdict rules. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011oW4GertDgvoFGdtGGoj5U
describeCluster builds a human title from a cluster's error type, locator
target, navigation route and spec basename (Timeout on getByLabel('Email
address') in checkout.spec.ts; toHaveCount mismatch on getByRole('row') in
users.spec.ts; Navigation timeout on /users), never emitting a masking
placeholder, and defers to the AI title when one exists. Unit-tested against
the fingerprint suite's sample errors. The UI, notification and API call
sites that still show the raw signature as a name follow.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011oW4GertDgvoFGdtGGoj5U
…the name Without an AI title a cluster showed its raw normalized signature — a Playwright error kind with masking placeholders — in the run failure-groups table, the project cluster list, the cluster page heading and document title, the execution rail card, the test-case history list, run insights, the analytics landscape and the cluster.new alerts. Every one of them now uses describeCluster as the fallback name (the AI title still wins), keeps the raw signature as secondary text, and the cluster page reads title first with the signature below. Handlers expose the title, locator and error type the helper needs; cluster.new payloads carry the display title for Slack, email and browser notifications. Also types the two network-request builders with the shared NetworkRequestBuilder shape (the inline copies lacked startTime) and declares network_requests.start_time as an int/bigint dialect difference. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011oW4GertDgvoFGdtGGoj5U
The automatic reopen note now contains the word regressed, so the row's substring match found two elements. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011oW4GertDgvoFGdtGGoj5U
Contributor
Coverage Report for Reporter (./packages/reporter)
File CoverageNo changed files found. |
Contributor
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What & why
The server-analysis slice of the failing-run data audit's quick wins (audit §7 items 2, 3, 4, 10 and 13; findings in §3.5, §4.1 and Appendix A). One commit per item. Nothing from the audit document itself is on the branch.
Item 2 — gate locator healing on a resolution failure
A
toHaveCountmismatch (locator resolved to 51 elements), a disabled or hidden element, or apage.gototimeout used to get a "Recommended fix" that rewrote a locator that already worked, and the run page showed a "Locator fix" signal built from ARIA guesses.classifyLocatorResolution(pure, inshared/) reads Playwright's call log: healing applies when the locator never resolved (waiting for <locator>with no laterlocator resolved toline), resolved to 0 elements, or hit a strict-mode violation; it does not when the locator resolved and the action or assertion failed afterwards, nor for navigation errors.resolveHealingForCasereturnsapplicable: false+ a one-linereasonbefore any ladder rung (ARIA fallback included), so the batch variant, the run's failure groups, the auto-heal policy, the fix plan, the MCP tools and the AI context inherit the gate; the panel shows the reason as one line instead of a menu.shared/locator-resolution.ts(new),packages/core/src/locator-healing-types.ts,server/utils/locator-healing.ts,shared/handlers/test-runs.ts,server/utils/heal/policy.ts,server/utils/fix-plan.ts,server/utils/ai-context.ts,server/utils/mcp/tools.ts,shared/mcp-tools.ts,app/demo/api/diagnosis-context.ts,app/components/shared/LocatorHealingPanel.vue,apps/docs/reporter.md,apps/docs/ai-diagnosis.mdtests/unit/locator-resolution.test.ts(new; the five call-log shapes plus navigation, no-locator and ANSI cases),locator-healing-resolve.test.ts,heal-policy.test.ts,demo-seed-consistency.test.ts(the demo's resolved-but-hidden cluster 9 is now asserted as not healed — the CSS is the bug there, not the selector)Item 3 — baselines prefer the same environment, then the same branch
The environment diff picked the test's last pass on the same browser and nothing else, so a
developmentfailure was diffed against aproductionpass and the diff listed the environment label itself. The visual diff preferred the branch but ignored the environment. Both now rank candidates through one shared ordering (same environment → same branch → recency); when no same-environment pass exists the fallback still happens but carries abaselineNote("compared with a production run; no passing development run of this test exists") that the cards and the AI context show, and the environment label is never a diff entry in that case.shared/baseline-order.ts(new),server/utils/environment-diff.ts,server/utils/visual-diff.ts,shared/environment-diff.ts,server/utils/ai-context.ts,app/components/shared/EnvironmentDiffCard.vue,app/components/test-case/VisualDiffCard.vue,app/utils/help-content.ts,apps/docs/concepts.mdtests/unit/baseline-order.test.ts(new)computeRegressionSignalsand run insights remain environment-blind (branch-aware only), as the brief asked.Item 4 — keep network request start times
The reporter sends
startTimeper request butnetwork_requestshad no column, so it was dropped and the AI context'st+Nmsoffset read a field that was always undefined. Nullablestart_time(epoch ms) in both dialects with generated migrations (SQLiteinteger, PostgreSQLbigint, declared in the dialect-drift allowlist), persisted from the wire by the shared builder, exposed on the execution response and asfirstStartTime/lastStartTimeon the run-level endpoint summaries, live in the AI-context offset, seeded by the demo generator and simulator, and used to order the execution page's captured requests.server/database/schema.sqlite.ts,server/database/schema.pg.ts,server/database/migrations/0054_stale_zzzax.sql,server/database/migrations-pg/0055_deep_lester.sql(+ journals/snapshots),server/utils/network-request-helpers.ts,server/utils/persist-run-cases.ts,app/demo/api/reporter.ts,shared/handlers/test-cases.ts,shared/handlers/test-runs.ts,types/api.ts,server/utils/ai-context.ts,scripts/generate-demo-seed.mjs,public/demo/seed.version.json,app/demo/simulator.ts,app/components/test-case/TestCaseNetworkRequests.vue,apps/docs/capture-fixtures.mdtests/unit/route-and-network.test.ts,tests/unit/schema-dialect-drift.test.tsItem 10 — close the loop on fix verification
A cluster could read "Fix verified" and "Open" at once. A
diagnosis-verifiedverdict now sets anopencluster toresolved, andregressedsets aresolvedcluster back toopen, each appending a system line to the triage note ("Resolved automatically: diagnosis verified in run #N" / "Reopened automatically: regressed in run #N").stopped-failingalone changes nothing;ignoredclusters are never touched; the verdict badge stays separate from the status as the docs describe. Two new events,cluster.fixed(every recorded fix, with the verdict in its payload) andcluster.regressed, are rendered for browser, Slack, email digests and webhooks. The demo validates events against the shared list, so it picks them up without a mirror change.server/utils/fix-verification.ts,shared/notification-events.ts,server/utils/notifications/dispatch.ts,app/composables/useNotificationStream.ts,apps/docs/notifications.md,apps/docs/ai-diagnosis.mdtests/unit/fix-verification.test.ts(new; in-memory SQLite, SCM and emit mocked),tests/unit/notification-events.test.ts,tests/fix-verification.spec.ts(E2E: a cluster resolved by hand is reopened with the note kept)Item 13 — deterministic cluster titles
Without an AI title, clusters displayed their raw masked signature as a name.
describeClusterbuilds a title from the error type, the locator's primary target, the navigation route and the spec basename (Timeout on getByLabel('Email address') in checkout.spec.ts,toHaveCount mismatch on getByRole('row') in users.spec.ts,Navigation timeout on /users), never emitting<N>,<VALUE>,<URL>,<STR>or other mask tokens; the AI title still wins. Used as the fallback in the run failure-groups table, the project cluster list, the cluster page heading and<title>, the execution rail card, the test-case history list, run insights, the analytics landscape and thecluster.newalerts (Slack, email, browser), with the raw signature kept as secondary text and the cluster page reading title first, signature below.shared/describe-cluster.ts(new),app/components/run/FailureGroups.vue,app/components/project/FailureClustersList.vue,app/components/cluster/ClusterSummary.vue,app/pages/failure-clusters/[id].vue,app/components/test-case/FailureClusterCard.vue,app/pages/test-run-cases/[id].vue,app/pages/test-cases/[id].vue,app/components/run/RunInsights.vue,app/components/analytics/ClusterLandscape.vue,shared/handlers/test-cases.ts,shared/handlers/run-insights.ts,shared/handlers/analytics/cluster-landscape.ts,shared/analytics/types.ts,server/utils/notifications/run-notifications.ts,server/utils/notifications/dispatch.ts,server/utils/email.ts,apps/docs/ai-diagnosis.mdtests/unit/describe-cluster.test.ts(new; includes every sample error fromerror-fingerprint.test.ts)Nothing was left out. Not done: no new feature-screenshot scene was added for the one-line healing state or the retitled cluster heading.
How was it tested?
From
apps/application/:npm run app:typecheck,npm run app:lint,npm run app:format:check,npm run app:test:unit(124 files, all green),npm run app:check:demo,npm run app:seed:demo(seed regenerated,seed.version.jsoncommitted),npm run db:migrateon a fresh SQLite database (both migrations generated with drizzle-kit, none hand-written), and the E2E specsfailure-clusters,fix-verification,cluster-page-layout,insights-and-spec-healthandnotifications(38 passed, auth-only tests skipped outside CI). The PostgreSQL migration was generated but not applied against a live PostgreSQL instance.Checklist
type(scope): subject)apps/docs/, README, or reporter README)🤖 Generated with Claude Code
https://claude.ai/code/session_011oW4GertDgvoFGdtGGoj5U
Generated by Claude Code