Collect coverage from every browser, not just one

[NullVoxPopuli]

Note

Made with Claude Code. Apologies in advance if I've misread how you intended any of this to work — this touches the CDP connection lifecycle, which is the most subtle part of the middleware, so please push back hard on anything that looks off. Happy to rework or drop it entirely.

The problem

When testem runs several browsers at once (ember-exam's --parallel, or testem's own parallel option), the middleware keeps one cdpClient and one coverage accumulator. Only the first browser is instrumented. The run succeeds, a report is written, and it silently describes roughly 1/N of the suite.

I hit this trying to turn coverage on for an app whose CI runs ~8 browsers per machine.

Why discovery, and why it does not replace the fixed port

Browsers can't share a fixed --remote-debugging-port, and testem resolves browser_args once per browser name, not per instance (browser-args.js) — so with a pinned port the second Chrome exposes no CDP endpoint at all. Parallel runs must use --remote-debugging-port=0.

With an ephemeral port Chrome writes the port it actually bound into <user-data-dir>/DevToolsActivePort, and testem gives every launcher its own --user-data-dir (Launcher#setupBrowserTmpDir). So the ports are discoverable: this polls for those files and connects to each browser it finds.

Worth being explicit, because I got this wrong at first and it cost me a while: Chrome writes DevToolsActivePort only for an ephemeral port. With a pinned port there's nothing to announce, so no file appears:

port=0    -> DevToolsActivePort: 62183
port=9222 -> DevToolsActivePort: MISSING   (DevTools *is* listening; /json/version responds)

So discovery can't replace the existing path — chrome.remoteDebuggingPort (still defaulting to 9222) is still seeded directly. The two mechanisms are complementary: pinned ports via the seed, ephemeral ports via discovery. Passing remoteDebuggingPort: 0 opts out of the seed.

What changed

• Per-browser state. The session, coverage accumulator and reload gate move from module scope into one record per browser, keyed by port. The reload/stale-request logic is unchanged in substance, just scoped.
• Merging. Each browser's V8 snapshot is folded together with mergeProcessCovs — the same primitive already used to merge one browser's periodic deltas, since a second browser is just more deltas. No change to report.js.
• Shutdown ordering. Each browser's /_coverage response is held open until the merged report is written. testem doesn't kill a browser until its afterTests hook calls next(), so this is what stops an early-finishing browser being torn down while another is still collecting. Bounded by a new chrome.stragglerTimeout (default 30s) so one dead browser can't hang the run.
• Request → browser matching. The runtime now sends testem's browser id (the numeric path prefix testem serves each test page under), so a /_coverage request is matched to the browser that sent it. Falls back to the sole browser when there's only one.
• Discovery hygiene. os.tmpdir() is shared, so only testem-* dirs created after the middleware started are considered — otherwise a concurrent testem run's leftovers get pulled into the report. There were 23h-old ones on my machine while I was working on this.

Testing

New scenario (testem-parallel.cjs + tests/vite-app-parallel-js.test.js) runs two browsers over the same app, split by QUnit filter so neither browser sees both files under test:

browser 1 (filter=Unit)        -> app/utils/format-score.js
browser 2 (filter=Integration) -> app/components/counter.gjs

The assertions only pass if both browsers' coverage was collected and merged. Observed on the merged report:

app/components/counter.gjs  -> lines 41/48, funcs 60%    (Integration browser)
app/utils/format-score.js   -> lines 15/18, funcs 100%   (Unit browser)

and the log shows two distinct ephemeral ports discovered:

browser-level connection closed on port 57687
browser-level connection closed on port 57688

It also asserts the merge doesn't turn untested code green — format-score.js's untested score < 0 branch and counter.gjs's never-called clampedCount/countAsString stay uncovered.

Full suite: 4 files / 28 tests passing, oxlint + oxfmt clean. I reused the existing vite-app-js app rather than adding a fourth scenario directory, since only the testem config differs.

Things I'm unsure about

• Holding responses open serialises shutdown: every browser stays alive until the last one finishes collecting. With load-balanced browsers they finish close together, but if you'd rather each browser die as soon as it reports (and accept a partial report when one lags), that's a different tradeoff and easy to change.
• os.tmpdir() as the discovery root assumes testem's default user_data_dir. I exposed chrome.userDataDir for the non-default case rather than trying to read testem's config, since the middleware only receives the express app — but you may know a better hook.
• Windows — I've only run this on macOS. The discovery path is plain readdirSync/statSync so it should be fine, but I can't claim it's tested.
• I left the connectionTimeout warning text saying "coverage disabled" per-browser; with N browsers that's now per-browser rather than global, which may want rewording.

[github-actions]

Coverage reports

Scenario	Artifact
v2-addon-js	coverage-v2-addon-js.zip
vite-app-js	coverage-vite-app-js.zip
vite-app-using-v2-addon-js	coverage-vite-app-using-v2-addon-js.zip

Each artifact is the scenario's coverage/ folder (HTML report, text/JSON summaries, raw V8 snapshot). Unzip and open index.html.

From this CI run for 62f1f14.