e2e-skills gives AI coding agents four skills for generating, reviewing, and debugging Playwright/Cypress end-to-end (E2E) tests. A skill is a set of instructions the agent follows for a task.
Featured on Kimi's official website: "AI Software Testing Skills for Smarter QA Automation".
A passing test can still miss a broken feature. e2e-skills helps check whether tests prove the behavior they name.
e2e-reviewer uses a code scanner to find possible problems, then reads the test and surrounding code to decide whether they are real. When needed, it can propose changing an assertion or introducing a bug in a separate app copy to check whether the test fails. The reviewer can offer these additional checks; execution requires a trusted local environment and command approval. Existing approval for the same scope remains valid.
e2e-reviewer findings led to 15 merged upstream PRs.
| Need | Skill | Result |
|---|---|---|
| Generate new Playwright coverage | playwright-test-generator |
Playwright tests based on the observed page, with scanner and test-execution results |
| Review Playwright/Cypress tests or PR/diff changes | e2e-reviewer |
Problems ranked by severity, with evidence and suggested fixes; for changes, whether each problem is new, worsened, or pre-existing |
| Debug a failed Playwright run | playwright-debugger |
Likely cause, supporting evidence, and a suggested fix |
| Debug a failed Cypress run | cypress-debugger |
Likely cause, supporting evidence, and a suggested fix |
| Run a deterministic local scan | skills/e2e-reviewer/scripts/scan.sh |
Possible code problems without installing the target project’s packages |
Review results use P0/P1/P2 for severity: P0 means a test can pass despite a broken feature, P1 concerns reliability or diagnosis, and P2 concerns maintenance. Debuggers use F1–F15 to label 15 failure categories, such as timing, selectors, and authentication.
Test generation: After installation, tell the agent which local project to test. It checks the required setup and guides you through anything missing.
e2e-reviewer findings have led to 15 merged upstream PRs. Every patch is linked below.
The repositories below had 496,650 combined GitHub stars on 2026-10-20 (GitHub API snapshot).
| Repository | GitHub Stars | PR | Pattern fixed |
|---|---|---|---|
| Storybook | ★91.3k | storybookjs/storybook#34141 | Missing await on Playwright actions; discarded visibility checks |
| code-server | ★79.7k | coder/code-server#7845 | Focused test leak, matcher-less expect, discarded visibility read |
| Strapi | ★73.3k | strapi/strapi#26630 | Discarded navigation/state checks |
| Ghost | ★55.6k | TryGhost/Ghost#28712 | Promise-valued disabled-state assertion |
| Cal.com | ★49.2k | calcom/cal.diy#28486 | Weak assertion patterns in E2E flow |
| Bruno | ★47.7k | usebruno/bruno#8317 | Assertion and wait reliability fixes |
| Qwik | ★22.1k | QwikDev/qwik#8777 | Locator/handle existence checks |
| SvelteKit | ★20.9k | sveltejs/kit#16068 | Floating Playwright assertions |
| Element Web | ★13.6k | element-hq/element-web#32801 | Locator null-check style assertions |
| FiftyOne | ★11.2k | voxel51/fiftyone#7851 | Locator-defined check replaced with a visible duplicate-name error assertion |
| Carbon Design System | ★9.6k | carbon-design-system/carbon#22564 | Locator truthiness replaced with web-first assertions |
| Rancher Desktop | ★7.4k | rancher-sandbox/rancher-desktop#10557 | not.toBeNull() locator checks replaced with visible WSL integration-name assertions |
| Apache Zeppelin | ★6.7k | apache/zeppelin#5180 | Always-true assertions and guarded checks that silently skipped |
| MUI X | ★5.9k | mui/mui-x#22982 | UI handle checks replaced with state assertions |
| module-federation/core | ★2.7k | module-federation/core#4826 | Redundant blanket uncaught:exception suppression in a Cypress spec |
A false-green test passes even when the behavior it is meant to check is broken. The change below checks the visible message instead of merely checking that a Locator object exists.
- expect(page.getByText('Welcome back')).toBeDefined()
+ await expect(page.getByText('Welcome back')).toBeVisible()Full example and scanner output
This Playwright test looks reasonable but proves only that Locator objects were created:
import { expect, test } from '@playwright/test';
test('shows the welcome message', async ({ page }) => {
await page.goto('/dashboard');
expect(page.getByText('Welcome back')).toBeDefined();
expect(page.locator('.user-badge')).not.toBeNull();
});The bundled scanner catches the false-green assertions without project configuration. Here is the relevant output:
$ /bin/bash -p skills/e2e-reviewer/scripts/scan.sh tests/
[P0] #4f Locator always-true assertion (truthy/defined/not-null) (2 hits)
.../tests/login.spec.ts:5: expect(page.getByText('Welcome back')).toBeDefined();
.../tests/login.spec.ts:6: expect(page.locator('.user-badge')).not.toBeNull();
Summary: 2 total hit(s), 2 P0Because this run reports P0 findings, scan.sh exits 1.
eslint-plugin-playwright also flags this shape via no-unnecessary-assertions; enable it so commit-time lint owns what it can and the scanner fills the gaps.
Executable example: React optimistic-write proof shows why optimistic UI needs request and persistence proof, not just visible state.
Starting with cases where tests pass despite bugs, the talk covers combining lint with LLM review, verification through assertion mutation and app-fault injection, benchmark results, and open-source contributions.
To include test generation, paste the prompt below into Claude Code or Codex and identify your existing local Playwright project:
Install https://github.com/voidmatcha/e2e-skills using its install.sh for my current host and Playwright project. Run the printed --check command when authorized for that exact command, then verify generator_write_test is available in this session. If a restart is needed, tell me how to resume verification afterward. Report installation, server tools, session tools, and approval separately; do not claim readiness until the required session tools are visible. Respect host trust and tool permissions; do not edit global configuration or restart the host automatically.
- Tell your agent which existing local Playwright project to use. The installer prepares the skills, dedicated agents, official Playwright agents and project MCP configuration, then prints a diagnostic command.
- Authorize that exact command in a trusted project:
bash install.sh --host codex --project /path/to/playwright-project --check(use--host claudefor Claude Code). It skips installation and configuration writes, starts the configured local MCP server and checksinitializeandtools/listonly. It executes project code/configuration, without calling tools, opening a browser or running tests. - If needed, restart your coding agent yourself, complete host trust/approval, and ask it to resume verification. It must see
generator_write_testin the active session before proceeding with generation.
The diagnostic verifies server tools only; it cannot establish active session tool availability or approval. A missing tool alone does not prove an approval problem.
After this setup, you do not need to repeat the manual installation steps. Use the installation guide only for prerequisites, conflicts or manual setup.
Only reviewing or debugging? Tell the agent to install the skills and dedicated agents only, without Playwright generator setup. Official Playwright agents and the playwright-test MCP server are required for generation, not for reviewing or diagnosing existing Playwright/Cypress tests.
Other installation methods: skills.sh and plugins
For all hosts supported by skills.sh, install the four skills with one command:
npx --yes skills@1.5.21 add voidmatcha/e2e-skills -g --allThis installs skills only; dedicated agents and Playwright generator setup are separate. To target one host, replace --all with -a <agent> from the supported agents. The CLI version is pinned.
Alternatively, install the plugin, which includes the four skills and two dedicated agents:
/plugin marketplace add voidmatcha/e2e-skills
/plugin install e2e-skills@voidmatcha
For generation, use the all-in-one setup above or follow the installation guide.
Alternatively, install the four skills through the plugin marketplace:
codex plugin marketplace add voidmatcha/e2e-skills
codex plugin add e2e-skills@voidmatcha
The plugin does not install the two dedicated agents. For those, run bash scripts/dev/install-codex-agents.sh from this checkout. Review and debugging also work inline without them.
For generation, use the all-in-one setup above or follow the installation guide.
Review my Playwright tests in tests/e2e with e2e-reviewer.
If useful, offer an assertion mutation or app-fault probe as a follow-up.
Generate Playwright E2E coverage for apps/web/e2e.
Debug the failed Playwright report in playwright-report/.
Debug the failed Cypress report in cypress/reports/.
Supported environments and execution requirements
Use the skills alongside a real application and its E2E suite. Review and debugging support Playwright and Cypress; generation supports Playwright only.
The bundled shell scripts and artifact readers target conventional macOS/Linux layouts with /bin/bash and the fixed system executable paths required by their launchers. Windows users need a compatible WSL distribution and artifacts inside the WSL filesystem; native Windows is not supported. NixOS and Nix-, pyenv-, Conda-, or nvm-only tool installations are not universally auto-discovered. The reviewer scanner supports validated absolute E2E_SMELL_*_BIN overrides; this does not imply the same overrides exist in the debugger launchers. Missing trusted tools stop the affected operation; do not bypass that boundary with ambient PATH lookup. Resolve skill resources from the actual host-provided skill location, including plugin caches and custom installs, not a guessed home directory.
The generator prepends thin-rules.md and safety instructions to the official planner and generator, then runs the reviewer scanner and the new tests. It installs nothing, changes no agent or MCP configuration, and uses no CLI fallback or healer. Missing configuration is reported with setup guidance.
Generation uses the loopback addresses common in local development (localhost, 127.0.0.1, or ::1). Reuse your approved development server and its existing test backend or mocks. Before browser work, confirm only the services the scenario uses. Writes require isolated test data and cleanup or reset; read-only scenarios need no reset. A new mock server or deployment is not required. Unused API or database services are N/A; shared or production data remain ineligible. Executable navigation and request destinations must resolve to approved origins, matching scheme, hostname and effective port; separately approved loopback API destinations are allowed. Other or unresolved destinations are BLOCKED. These checks do not provide browser-wide origin containment.
Only proven generation errors permit up to two parent-owned repairs. Required outcomes and expected values stay intact; test data is safely cleaned up or reset before repeating writes and origin/scanner gates repeated using the same approved execution method. Product defects, uncertain causes, and remaining failures stay BLOCKED. See the generator contract for the full procedure.
Debugger and probe execution require a trusted project and approval of the exact command, including environment variables and flags. Reports identify skipped checks; mutation and app-fault probes are optional follow-ups, never reported as measured without execution.
Version history and the previous generator workflow are documented in CHANGELOG.md.
The official Playwright agents handle generation. The generator skill then runs the e2e-reviewer scanner when installed, fixes P0/P1 findings in at most three passes, and runs the project's tests. Missing or skipped checks are reported as incomplete. Request e2e-reviewer separately for a semantic review of the generated tests.
Start with a regular e2e-reviewer review of whether the tests check the intended behavior. When a specific question needs runtime evidence, add a probe for only that assertion or behavior. You choose where to spend the extra tokens and execution time; mutation and app-fault probes do not run automatically on every review:
- Assertion mutation: reverse an assertion in a copy of the test and check whether it fails. This checks that the assertion can fail, not whether the original test catches an app bug.
- App-fault injection: introduce a specific bug in a copy of the app and check whether the unchanged test catches it.
For example: “Check whether this test catches a broken save operation.” The reviewer examines the local project and proposes a check. Execution requires a trusted, prepared local environment and your approval of the commands. Results apply to the selected assertion or bug; they do not prove that every bug will be caught.
See the execution procedure and requirements and two OSS workflow examples.
The 3.0.0 validation record covers installation, scanner behavior, safety rules, and selected Playwright/Cypress runs, with retained evidence and result scope.
The October 20 checks exercise isolated installation, official-agent/MCP test generation and execution in Claude Code and Codex, and selected Cypress assertion-mutation and app-fault probes. The original generator cohort and later replay remain separately reported.
Benchmark scope and model results
Reviewer results depend on the model: on the same frozen snapshot and 36 public development cases with five repetitions per arm, gpt-6-sol passed v22, gpt-6.1-sol passed v24, and gpt-6-luna failed v23, with all three models completing all 540 calls in each run. These tool-free results measure supplied instructions, not scanner benefit or accuracy on unseen projects. Scores apply only to the reported snapshots. See benchmark status for model-specific scores, protocols, negative findings, and claim boundaries, and CHANGELOG.md for behavior changes.
The entire frozen historical benchmark snapshot, including protocols, inputs, summaries and raw run artifacts, is pinned in the evidence manifest and available from the Hugging Face dataset. Use scripts/dev/fetch-benchmark-evidence.sh --dry-run to inspect the restore plan before auditing a past score.
The catalog contains 24 stable Playwright/Cypress test smells. The most common false-green shapes include Locator truthiness, missing assertions, swallowed errors, focused tests, missing authentication, and optimistic UI checks without network proof. See the full taxonomy and rationale.
Some patterns need the application in scope, not just the tests. #22 optimistic UI is the clearest case: whether a click issues a mutation cannot be decided from a spec, so on a repository that holds tests alone the review reports nothing for it rather than guessing. That is deliberate false-positive control, and it is why the executable example ships with a component.
All 24 patterns by severity
Tests pass when the feature is broken. No real verification is happening.
| # | Pattern | Before | After |
|---|---|---|---|
| 1 | Name-assertion mismatch | Name says "status" but only checks toBeVisible() |
Add assertion for status content, or rename to match actual check |
| 2 | Missing Then | Cancel action, verify text restored: but input still visible? | Verify both restored state and dismissed state |
| 3 | Error swallowing | try/catch in spec, .catch(() => {}) in POM |
Let errors fail; remove silent catch from POM methods |
| 3b | Cypress uncaught:exception suppression |
cy.on('uncaught:exception', () => false) blanket-swallows app errors; a fail handler that never rethrows passes every failing test |
Scope handler to specific known errors; re-throw unknown errors; end fail handlers with throw error |
| 4 | Vacuous or retry-weakening assertion (P0/P1) | P0: invariant predicates and Locator truthiness. P1: weak attachment proof; one-shot values/URL; zero-timeout retry/deadline hazards; unproven absence; assertion loops over possibly empty collections; ARIA snapshots that omit a promised accessible name | Use meaningful bounds and web-first auto-retrying assertions; prove presence before absence, prove collections non-empty before assertion loops, and keep promised accessible names load-bearing |
| 5 | Bypass patterns (5a P0, 5b P1) | if (await el.isVisible()) { expect(...) }; { force: true } without comment |
Always assert; move env checks to beforeEach; add // JUSTIFIED: to force:true |
| 7 | Focused test leak | test.only(...) committed: CI runs one test, silently skips the rest |
Delete .only; use --grep or --spec for local focus |
| 8 | Missing assertion | Discarded locator/boolean is the scenario's only verification; expect(x).toBeVisible; never calls the matcher |
Add await expect(locator).toBeVisible(); skip 8a/8b when independent verification/failure evidence already exists |
| 12 | Missing auth setup | With no login/storageState/auth fixture, a protected-route spec passes because its generic assertion also matches the login/wrong surface |
Add beforeEach login, configure storageState, or use an auth fixture; do not classify a normal auth-caused failure as P0 |
Tests work but mislead developers, waste CI time, or set up future regressions.
| # | Pattern | Before | After |
|---|---|---|---|
| 6 | Raw DOM queries | document.querySelector in evaluate() |
Use framework locator/query APIs (locator / cy.get) |
| 9 | Hard-coded sleep | waitForTimeout(2000) / cy.wait(2000) / waitForLoadState('networkidle') |
Rely on framework auto-wait; use condition-based waits |
| 10 | Flaky test patterns | items.nth(2) without comment; test.describe.serial(); unscoped accessible-name substring (10c); Cypress async callbacks, assigned cy commands, or continued action chains (10d–10f) |
Use stable/scoped locators and self-contained tests; keep Cypress work in its command chain, do not assign Chainables as values, and re-query after actions |
| 13 | Inconsistent POM usage | Raw actions bypass a source-established same-page POM boundary; existence alone is insufficient | Reuse the proven boundary; do not invent a wrapper merely because a POM exists |
| 14 | Hardcoded credentials | loginPage.login('demo-admin', '<literal-password>') in test code |
Use process.env.TEST_USER, Playwright config secrets, or test data fixtures |
| 15 | Missing await on expect() |
Async Locator/Page web-first matcher Promise is not sequenced or observed; rejection usually surfaces later with worse attribution | await or return the matcher Promise; sync value matchers are excluded |
| 16 | Missing await on action |
Actionability, action ordering, or navigation can race later work; rejection usually surfaces later with worse attribution | await or return the action Promise |
| 17 | Discouraged direct Page selector API | Selector-based page.click, page.fill, and related Page actions skip the Locator layer |
Use Locator actions for composition, strictness, reuse, and clearer failures |
| 18 | expect.soft() overuse |
Critical soft assertions run before a hard scenario gate, so dependent work continues after a broken prerequisite | Hard-gate the primary state first; use soft only for independent details |
| 19 | Module-level mutable state in test code | let seq = 0; or a mutated const cache = new Map() at column 0 in a test utility: persists across tests in a long-lived worker and collides across parallel workers |
Drop the counter; derive uniqueness from Date.now() + Math.random().toString(36).slice(2, 8), or move state into test.beforeEach |
| 20 | Unmocked real-backend writes | Signup/checkout spec reaches shared or persistent state with no controlled test boundary | Stub the write or prove a disposable container, rollback fixture, isolated tenant/database, or equivalent controlled backend |
| 22 | Optimistic UI without call proof | Like-toggle test asserts aria-pressed flip: UI updates optimistically, passes with the POST deleted |
Pair UI assertion with page.waitForRequest() (armed before the click) or a route-hit flag |
Weak but not wrong: addressed when refactoring.
| # | Pattern | Before | After |
|---|---|---|---|
| 11 | YAGNI + Zombie Specs | clickEdit() never called; unjustified empty wrapper class; entire spec duplicated by another; unexplained skip |
Delete unused members and zombie specs; explain remaining skips and add revisit anchors for long-lived ones; inline single-use helpers only when that clearly removes meaningless indirection |
| 21 | Manually-captured session-file dependency | storageState: 'auth/member.json' produced only by a manual capture script: absent on CI, silently expires |
Regenerate session programmatically (API-login helper or setup project); manual files only as a cache with a programmatic fallback |
| 23 | Fixture ignores render guards | Liked-tab fixture seeds liked: false; the card component return nulls every item: empty UI looks like infra flake |
Read the item component's early returns/filters before seeding; seed fields to pass every guard for the view under test |
Both debuggers use the same stable F1–F15 root-cause taxonomy. Playwright accepts playwright-report/, HTML reports, trace.zip, screenshots, and bounded GitHub Actions artifacts. Cypress accepts mochawesome or JUnit reports, screenshots, videos, and bounded CI artifacts.
All 15 failure categories
| # | Category | Signals |
|---|---|---|
| F1 | Flaky / Timing | TimeoutError, passes on retry |
| F2 | Selector Broken | locator not found, strict mode violation |
| F3 | Network Dependency | net::ERR_*, unexpected API response |
| F4 | Assertion Mismatch | Expected X to equal Y, subject-inversion |
| F5 | Missing Then | Action completed but wrong state remains |
| F6 | Condition Branch Missing | Element conditionally present, assertion always runs |
| F7 | Test Isolation Failure | Passes alone, fails in suite |
| F8 | Environment Mismatch | CI vs local only; viewport, OS, timezone |
| F9 | Data Dependency | Missing seed data, hardcoded IDs |
| F10 | Auth / Session | Session expired, role-based UI not rendered |
| F11 | Async / Command-Order Race | Playwright Promise.all order or parallel race; Cypress intercept registered after the request, chain-order swap, or visit/request race |
| F12 | POM / Locator Drift | DOM structure changed, POM not updated |
| F13 | Error Swallowing | .catch(() => {}) hiding actual failure |
| F14 | Animation Race | Content not yet rendered, or a transient element removed before it is observed |
| F15 | Hydration Race | Action succeeds but has no effect: SSR page not yet hydrated; fails at the next assertion |
F11 and F12 use unified cross-framework names here. Each debugger reports its framework-specific label for the same stable code.
The debuggers classify product regressions separately from brittle tests and return evidence plus a concrete fix. They do not diagnose an application or backend without a failing Playwright or Cypress test artifact.
Run the code scanner directly without asking an AI agent. It reports possible problems for review; it does not run your tests or decide whether they prove the intended behavior.
/bin/bash -p skills/e2e-reviewer/scripts/scan.sh path/to/testsRequires Python 3 and ripgrep (rg) with PCRE2 support. By default it neither runs the target project’s ESLint code nor downloads tools. If a required tool is missing or a file cannot be read, it reports an error rather than a clean result.
Advanced scanner options and file access
The scanner requires a PCRE2-capable rg and Python 3. Python creates and validates NUL-safe candidate identity records so candidate drift or malformed records fail closed; this mandatory bookkeeping is separate from optional Tier 2 AST tooling. It does not execute target-controlled ESLint binaries, plugins, parsers, or configuration by default, and it does not download tools by default. E2E_SMELL_ALLOW_PROJECT_ESLINT=1 opts a trusted checkout into project ESLint execution; E2E_SMELL_NO_ESLINT_DOWNLOAD=0 and E2E_SMELL_NO_AST_GREP_DOWNLOAD=0 separately opt into pinned downloads. Set E2E_SMELL_DISABLE_AST_GREP=1 when a portability check must ignore preinstalled host binaries.
Read boundary.
Bundled checks report source beneath the requested path. Framework provenance resolution may also read relative fixture/support imports elsewhere in the containing project.
Bundled checks read .ts, .js, .tsx, .jsx, .mts, .mjs, .cts, and .cjs sources.
Tier 3 is the bundled fallback. Optional ESLint and ast-grep tiers add precision but do not replace semantic review. The scanner exits 2 on infrastructure or filesystem errors rather than reporting a false clean result. See SECURITY.md for the trust and network boundary.
eslint-plugin-playwright and eslint-plugin-cypress catch common test-code mistakes. The e2e-skills scanner also uses these plugins as an optional checking layer.
This project also contributed the no-unnecessary-assertions rule to eslint-plugin-playwright (merged PR #470). It catches assertions that check only whether a Locator object exists, rather than application state.
Plugin execution requires your explicit permission; the bundled checks run without it. On top of the plugin rules, e2e-skills provides:
- Its own code checks to complement the plugin rules
- Review of test intent and surrounding code for problems that require context
A linter can catch a direct Locator truthiness assertion or missing await. It cannot decide whether a test named "shows a duplicate-name error" ever checks the error, whether a protected-route test forgot authentication, or whether an optimistic UI assertion proves the backend request happened. Use the plugins for continuous linting and e2e-reviewer for test trustworthiness.
Point e2e-reviewer at the relevant spec directory. It combines deterministic candidates with semantic review before returning findings.
Review coverage and execution safety details
The e2e-reviewer skill reviews all 24 catalog patterns with stable IDs and P0/P1/P2 severity. Its standalone scan.sh scanner covers only a deterministic mechanical subset. Scanner matches are candidates, not final findings; the skill checks intent and surrounding code before reporting a verdict.
The debuggers classify failures against the stable F1–F15 taxonomy. They execute target-controlled code only after you trust the repository and approve the exact command, including its environment and flags. The generator runs only the reviewer scanner and the project's own test command on the specs it wrote.
For non-public benchmark runs, --isolation-wrapper is a required hook, not proof of isolation. Continuous integration (CI) validates the wrapper contract but does not attest filesystem, process, or network isolation.
No. This bundle reviews test quality, generates Playwright coverage, and diagnoses existing failures; it is not a test runner.
Yes, when you provide the local report artifacts or a supported GitHub Actions run. The debugger separates environment, timing, selector, data, authentication, and product-regression causes using the F1–F15 taxonomy.
- How to review AI-generated Playwright and Cypress E2E tests
- 24 Playwright and Cypress E2E test smells
- Benchmark status and negative results
More references
- Self-audit of the rules
- Open-source case studies
- Frozen historical benchmark snapshot (Hugging Face dataset)
- External evidence ledger
- Historical AI reviewer pilot
- Debugger benchmark protocol
- Framework scope
- Local E2E rule bridge and recommended lint
- Roadmap
- Reviewer evidence execution plan and optional Jev experiment
Bug reports, false-positive guards, new anti-patterns, and translations are welcome. Start with CONTRIBUTING.md for setup and verification requirements. Cross-agent maintenance contracts live in AGENTS.md.
Apache-2.0 © voidmatcha. See LICENSE.
Some benchmark transcripts quote Playwright's own agent definitions (Apache-2.0, Microsoft); see NOTICE. Playwright is a trademark of Microsoft Corporation; this project is not affiliated with or endorsed by Microsoft.