Judge PR evals on cost and prompt cache changes

[AshishKumar4]

PR eval comments hide caching wins. #609 removed most cache misses at turn starts, but the comment showed +1 to +3 pp in cache hits, and Bonk called it unchanged.

The cache hit rate is cache reads over all prompt tokens. It also moves with how long a run is and how much new content it reads, so a real caching fix gets lost in run-to-run noise.

Changes

• comparison.json gets cacheBreakRate per task (of the tokens the previous step had already sent, the share a step sent again instead of reading from cache), plus p-values for cache hits and cache breaks. Each is an exact Mann-Whitney U test on each run's own rate, in the direction the pooled rate moved.
• The table keeps its columns. A cache hit change is bold when its p-value is below 0.05.
• Bonk reports each cache change with p < 0.05, and weighs cost (already in each run's trajectory) and cache changes in the verdict. A change counts when it moves the same way on most comparable tasks and the diff explains it.

On #609's results, cache breaks fell on all 3 comparable tasks (p = 0.029, 0.002, 0.075). comparison.ts only builds reports, so no evals rerun.

[ask-bonk]

Review: 1 finding.

Posted 1 actionable inline comment.

github run

[github-actions]

Preview: pr618-eval-cache-breaks

https://pr618-eval-cache-breaks-router.cloudflare-os-previews.workers.dev

Dashboard · deleted when this PR closes

[ask-bonk]

Review: 1 finding.

The existing inline finding about cache-break rates depending on run length still applies to this revision. No additional actionable issues found.

github run

[ask-bonk]

Review: 1 finding.

The existing cache-break finding still applies: additional fully cached steps lower the reported rate without improving cache behavior, despite the new instructions claiming independence from run length.

No additional actionable issues found.

github run

[ask-bonk]

LGTM!

github run

[github-actions]

Eval results

Verdict: ⚪ Unchanged. Nothing the evals run changed, so every result is reused.

Task	Score	Δ score	Fisher test	Cache hits	Avg min	Avg steps
change-calendar	100%	same inputs	—	83%	4.2	25.8
chess	100%	same inputs	—	97%	11.2	66.9
incident-desk	100%	same inputs	—	93%	4.3	30.6
worker-logs	100%	same inputs	—	90%	3.9	20.9

Run · trajectories and raw results

[ask-bonk]

🔬 Eval runs review

Performance

All four tasks reuse the same results on main and the PR, each passing 10/10 runs; trajectories, costs, durations and tool errors are identical. Cache hits remain 83% for change-calendar, 97% for chess, 93% for incident-desk and 90% for worker-logs, with unchanged cache-break rates and no comparable independent runs demonstrating a caching gain. Chess is the most expensive task at 66.9 model steps and 11.2 minutes per run, costing $0.0483–$0.0854; implementation and testing dominate its steps, with malformed calls and exact-text edit retries adding avoidable work.

⚪ VERDICT: NO REGRESSION FROM THIS PR

The diff changes comparison reporting and advisory-review instructions, not what the agents execute, so the reused results support neither an agent-performance regression nor an improvement.

Triage

Tool errors

• editFile: Validation failed for tool "editFile": · change-calendar 1 → 1, chess 7 → 7, incident-desk 7 → 7, worker-logs 3 → 3 · model error — Agents omitted required arguments or used the wrong call shape despite the declared schema; change-calendar trial 2 omitted filename, then successfully retried with client.js.
• editFile: No matching text was found in the file. · change-calendar 1 → 1, chess 7 → 7, incident-desk 1 → 1 · model error — Agents supplied text that did not match exactly; chess trial 2 added indentation absent from the inline CSS, then used grep and successfully retried with the actual substring.
• readFile: File does not exist. · change-calendar 4 → 4, worker-logs 2 → 2 · model error — Agents tried reading files before creating them; change-calendar trial 4 read server.js and client.js immediately after creating an empty gadget, then switched to writeFile, despite createGadget documenting that default.
• editFile: Multiple matches were found. The text to match must be unique. · chess 2 → 2, incident-desk 4 → 4 · model error — Replacement targets lacked enough context; chess trial 1 targeted a repeated gameHistory write, then successfully split the edits using surrounding statements.
• executeCode: Failed to start Worker: · chess 3 → 3 · model error — Chess trial 1 submitted empty code, and trials 3 and 6 submitted . rather than the required self-contained module with a default export; each recovered by sending valid code.

What to do

• No agent-side change is required in packages/workshop-backend/src/agent.ts: the existing descriptions and schemas already cover these recovered errors, and this PR leaves them unchanged.

github run