Share OpenAI's prompt cache across chats

[AshishKumar4]

On GPT-5.6 and later, OpenAI keeps a separate prompt cache for each prompt_cache_key, and pi sets one per chat. So every new chat wrote the tools and static system prompt again, about 7k tokens at 1.25x. In #609's eval run, the first request of all 40 trials read 0 cached tokens.

This PR drops pi's key on those models, so chats share one cache across the OpenAI organization, as they already do across a workspace on Anthropic. Older models route by the key, so they keep it.

A shared cache lets anyone on the same organization tell from cached token counts whether a prompt prefix was sent recently. So the project-specific part of the system prompt now starts with a random salt, created on a workspace's first agent turn and stored with it. Only agent turns, which carry the salt, drop the key; other requests on a chat, such as compaction, keep pi's per-chat key. Only the static part can still be probed: the tools, the base prompt and the deployment's admin instructions.

In this PR's last eval run, first requests that read the cache went from 0/10 to 10/10 on chess and incident-desk, and to 9/10 on worker-logs.

[ask-bonk]

LGTM!

github run

[github-actions]

Preview: pr615-prompt-cache-a4635ee0

https://pr615-prompt-cache-a4635ee0-router.cloudflare-os-previews.workers.dev

Dashboard · deleted when this PR closes

[ask-bonk]

LGTM!

github run

[ask-bonk]

LGTM!

github run

[kentonv]

Seems reasonable.

Also, if we are actually concerned about cache-probing attacks, it strikes me that there's a better solution: Place an unguessable random string at the beginning of the project-specific part of the system prompt, selected on a per-project basis.

I'm not personally all that worried about it, but this seems so easy that we might as well do it?

[ask-bonk]

LGTM!

github run

[ask-bonk]

Review: 1 finding.

Posted one actionable inline comment.

github run

[ask-bonk]

LGTM!

github run

[ask-bonk]

LGTM!

github run

[ask-bonk]

LGTM!

github run

[kentonv]

Using the workspace ID is not an ideal choice here because:

• We have to think about whether we want the agent to know the workspace ID.
• It's possible that people who do not have access to the workspace know the ID.

Both are minor points, but we could also derive an entirely random value here, store it, and use that.

[AshishKumar4]

I chose workspace ID primarily to avoid adding much code and complexity; we basically need a random value that is stored and persisted in the workspace and accessible across all chats, and workspaceID seemed to fit the bill well.
Having a separate value would add to the schema, and a new argument to every function in the chain inbetween

[ask-bonk]

LGTM!

github run

[github-actions]

Eval results

Verdict: ⚪ Unchanged. No task moved beyond what 10 runs can tell apart from noise.

Task	Score	Δ score	Fisher test	Cache hits	Avg min	Avg steps
change-calendar	100% → 80%	−20 pp	p = 0.47	86% → 88% +2 pp	3.4 → 4.5	25.0 → 24.6
chess	90% → 88% (2 run errors)	candidate run errors	—	97%	9.2 → 8.8	61.9 → 53.3
incident-desk	100% → 100% (1 run error)	candidate run errors	—	95%	4.3 → 4.7	38.3 → 32.3
worker-logs	100% → 89% (1 run error)	candidate run errors	—	92% → 93%	4.0	24.2 → 20.4

Failed checks

Task	Check	Failed
change-calendar	t1 agent.timedOut	0/10 → 1/10
change-calendar	t4 names-the-booked-windows-the-new-rules-reject	0/10 → 1/9
chess	t1 agrees-with-the-oracle-on-the-hard-positions	1/10 → 1/8
chess	t1 agrees-with-the-oracle-on-perft-positions	1/10 → 1/8
chess	t1 agrees-with-the-oracle-through-random-games	0/10 → 1/8
worker-logs	t1 ingests-resets-and-summarises-per-worker	0/10 → 1/9
worker-logs	t1 hourly-buckets-cover-every-hour-including-empty-ones	0/10 → 1/9
worker-logs	t1 ranges-are-half-open-and-filter-by-worker	0/10 → 1/9

Run · trajectories and raw results

[ask-bonk]

🔬 Eval runs review

Performance

The only comparable task, change-calendar, fell from 10/10 to 8/10 passes, within noise (p=0.47); candidate run errors prevent comparing the other three tasks. Its cache hits rose from 86.4% to 88.0% and breaks fell from 7.8% to 7.4%, neither significant; completed runs cost $0.0235–$0.0342 versus main’s $0.0217–$0.0299, while mean duration increased from 3.4 to 4.5 minutes. Provider authorization errors lost the most runs, and chess’s exact-text editing detours wasted the most steps, including 30 unmatched edits versus main’s eight.

⚪ VERDICT: NO REGRESSION FROM THIS PR

Removing the per-chat cache key demonstrably enables first-request cache reads, but comparison.json establishes no improvement beyond noise, and the failed trajectories do not implicate that rewrite or the workspace salt.

Triage

Failure modes

• Provider authorization rejection · chess 2/10, incident-desk 1/10 · harness bug · this PR: no — chess trials 2 and 8 stopped in turns 3 and 2 after successful edits; incident-desk trial 3 passed turn 1 but received a 401 instead of answering turn 2. These are infrastructure errors, not failed gadget implementations; the diff does not change credentials.
• Activation never finishes · change-calendar 1/10 · harness bug · this PR: no — trial 2, turn 1 ended after three parallel writeFile calls, with no final reply or tool error, until the 420-second activation timeout. The underlying stall is not exposed, and activation handling is unchanged.
• Incorrect calendar diagnosis · change-calendar 1/9 · model error · this PR: no — trial 9, turn 4’s executeCode fetched every booking, but the final reply incorrectly marked mw-101 and mw-103 as TOO_CLOSE; their separation exceeds 24 hours. Main’s agents calculated the rejected set rather than relying on this mistaken date arithmetic.
• Kings move like sliding pieces · chess 1/8 · model error · this PR: no — trial 10, turn 1’s writeFile(server.js) put kings through the queen-style sliding loop before adding castling, producing illegal and duplicate moves. The agent inspected the file but never exercised the rules with executeCode; main also had one rules-engine failure, although its defect was lost castling metadata.
• Oversized SQL inserts · worker-logs 1/9 · system prompt · this PR: no — trial 3, turn 1’s writeFile(server.js) batched 100 events into 600 SQL bindings, incorrectly claiming this stayed below SQLite’s limit. Verification rejected ingestion, leaving later summaries empty; main’s successful implementation inserted one row per statement. The unchanged storage guidance omits the environment’s bound-parameter limit.
• Verification connection loss · worker-logs 1/10 · harness bug · this PR: no — trial 6 completed turn 2’s edits and reply, but verification encountered WebSocket connection failed; three other turn-2 checks passed. comparison.json classifies this as an infrastructure error, not evidence of lost stored data.

Tool errors

• editFile: No matching text was found · chess 8 → 30, incident-desk 3 → 4, worker-logs 1 → 1 · model error — agents quoted stale or incorrectly escaped source. Chess trial 6, turn 2 submitted twelve overescaped replacements together, all rejected, before correcting the escaping.
• editFile: Validation failed · change-calendar 5 → 4, chess 1 → 1, incident-desk 5 → 2, worker-logs 5 → 2 · model error — calls omitted required fields or supplied wrapper objects instead of tool arguments. Change-calendar trial 4, turn 1 used .replace instead of replacement, then recovered with the correct field.
• editFile: Multiple matches were found · chess 10 → 4, incident-desk 1 → 3, worker-logs 0 → 1 · model error — replacements lacked unique surrounding context. Chess trial 5, turn 1 retried the same ambiguous fragment even after grep showed both matches, then succeeded with longer snippets.
• readFile: File does not exist · incident-desk 2 → 0, worker-logs 2 → 0 · model error — main’s agents tried reading server.js and client.js immediately after creating empty gadgets, despite the prompt explaining that new gadgets have no files.
• executeCode: Failed to start Worker · chess 0 → 3 · model error — candidate calls contained an unescaped apostrophe in a PGN string or only : or }. Corrected JavaScript succeeded; these were malformed agent calls, not Workshop startup failures.

What to do

• Rerun after stabilizing provider authorization and the Workshop connection so chess, incident-desk, and worker-logs become comparable; no corrective PR code change is supported by this run.
• Optional, unrelated follow-up in packages/workshop-backend/src/agent.ts: document the SQL bound-parameter limit beside the storage guidance and recommend per-row inserts inside transactionSync.
• Optional, unrelated follow-up in packages/workshop-backend/src/agent.ts: add explicit editFile recovery guidance to re-read unmatched text, preserve literal escaping, and include surrounding context for duplicate matches.

github run

[kentonv]

LGTM