Pack streaming follow-ups for #656: a size-cap fix, less work per pull, native inflate

[Maximo-Guk]

Twenty-four commits on top of #656, from reviewing it by running the branch in local workerd against the real vscode and TypeScript depth-1 packs. The first ten are ordered from least to most debatable and each is meant to be taken or dropped on its own. Commits 11 to 13 answer Devin's review comments here and go together. Commit 14 is in gatekeeper-github and stands alone. Commits 15 to 17 lift the limits that stopped a mount of torvalds/linux, short of the CPU limit. Commits 18, 22 and 24 fix commit 8 and should be taken with it. Commits 19 and 20 remove the gatekeeper's own transfer limit, which makes 16 unnecessary. Commit 21 fixes a type error in 17. Commit 23 restores a bound that 19 dropped. Take whichever you want.

#	Commit	Kind	What I measured
1	Bound a pack entry's size varint	Bug fix	An entry inflated 8 MiB past a 1,024-byte cap
2	Read a pack in full 64 KiB buffers	Efficiency	3,330 reads → 590 over an RPC hop; no end-to-end gain shown locally
3	Skip the pull-routing hint a gatekeeper has already proven	Efficiency	Metadata writes 47,153 → 28,428 (vscode)
4	Leave an object the gatekeeper already stored untouched	Efficiency	Same pack again: 6.7–7.9 s → 2.8–3.2 s (vscode)
5	Measure an oversized pack entry as it arrives	Robustness	New test
6	Store a streamed blob without a transaction of its own	Efficiency; undoes a choice #656 made	First pull 8.2 s → 7.3 s (vscode), 14.5 s → 11.2 s (TypeScript)
7	Deflate loose git objects at zlib's fastest level	Trade-off: CPU for stored bytes	7.3 s → 6.2 s, 38.8 MiB → 43.3 MiB (vscode)
8	Inflate pack entries with native zlib	Rewrite of PackReader	Decode alone 1.57 s → 0.87 s (vscode)
9	Test a pack arriving over RPC, and pin the delta order rule	Tests only
10	Say what consumePack returns and leaves behind on failure	Docs only (workshop-shared)
11	Bound the oversized objects kept as delta bases	Robustness; picks a budget and a failure rule	New tests; Worker memory not measured
12	Drop oversized bases only from a pack of blobs alone	Fix to 11	Two tests that fail on 11
13	Drop only the oversized bases a pull asked for by name	Fix to 12; replaces its rule	A test that fails on 12
14	Report why a git fetch failed, not the disconnect the cache saw	Bug fix (gatekeeper-github)	Mounting torvalds/linux: 190.9 MiB against the 64 MiB limit, reported as a dropped connection
15	Give a pack's objects their own size cap, and raise the pack's	Limit change: pack cap 64 MiB → 256 MiB	The Linux pack through consumePack: all 98,591 objects stored, 44–70 s locally
16	Raise the git fetch transfer limit to match the pack cap	Limit change (gatekeeper-github)	The real transport reads the whole Linux response, 200,177,400 pack bytes
17	Time a git fetch out on a quiet server, not on its total length	Behaviour change (gatekeeper-github)	Two tests with fake timers
18	Bound how much of one pack entry the reader buffers	Fix to 8	3 MB of padding around 10 bytes decoded before, is refused now
19	Leave the size of a pack to the overseer	Removes a duplicate limit (gatekeeper-github); replaces 16	A reader's cancel reaches the source across RPC within one pull
20	Say that the pack cap is the one limit on a pull's size	Docs only
21	Give the stall guard a timer that is never undefined	Fix to 17: it broke tsc in gatekeeper-github	tsc run directly fails on 17 and passes here
22	Decode a pack entry from what has arrived, and name its input limit	Fix to 8 and 18; answers two review comments	A test that fails on 18
23	Bound the part of a fetch response that is not pack data	Fix to 19; answers a review comment	A real Linux fetch carries 106,875 such bytes, 0.053% of the pack
24	Inflate a large pack entry a read at a time	Fix to 8; answers a review comment	A 48 MiB level-0 entry is measured; TypeScript's two large trees take the new path

Commits 1–10 together, first pull, best of three, two alternating passes: vscode 8.1–9.6 s → 5.0–5.8 s, TypeScript 15.7 s → 10.0–10.5 s. Every timing here is local workerd on Durable Object SQLite, on a machine that was busy with another build, so treat them as ±15%. Nothing was measured in production.

Dependencies: 6 and 11 edit the lines 5 introduces, 12 and 13 correct 11, 18, 22 and 24 correct 8, 21 corrects 17, 19 replaces 16, 23 corrects 19, and 8 rewrites the reader that 2 changes one line of. The rest are independent.

1. Bound a pack entry's size varint (bug)

The size varint had no length limit. 0xB0, about 150 × 0x80, then 0x00 makes size NaN, and NaN passes both size > maxObjectSize and the running total > size. The decoder now rejects a continuation byte whose multiplier is over the cap. The arithmetic was the same on main.

2. Read full 64 KiB buffers

A BYOB read returns as soon as one source chunk arrives, and GitHub sent a 38.6 MiB vscode pack as 3,358 sideband packets, mostly 8 KiB. readAtLeast waits for a full buffer. Locally a read after a write cost about 0.2 ms, and the difference was inside the run-to-run spread, so this one wants the production harness.

3. Skip the pullableFrom write for a gatekeeper already in onRemote

Blobs are stored before the trees that name them, so every tree entry rewrote a blob's row to add a hint its proof already covers. Pull routing unions the two lists, the marking walk tests either, and nothing removes an onRemote entry. TypeScript goes from 86,975 metadata writes to 66,023.

4. Leave an object the same gatekeeper already stored untouched

Pulls send no haves, so a retry, or the next commit of a mounted repository, delivers nearly everything again, and all of it was deflated and rewritten. The metadata row is checked first, so an object new to the cache pays nothing extra. A pull that was reset on CPU also resumes from the blobs the first attempt kept. TypeScript, same pack again: 11.6–15.1 s → 4.8–7.1 s.

5. Measure an oversized entry as it arrives

An object over 1 MiB was only measured after the trailer verified, so a pull that ended early recorded nothing and the next read fetched it again. It is now measured on arrival and still kept as a possible delta base.

6. No transaction around a streamed blob

What can fail partway through a store is parsing the objects it names, and a blob names none. This removes one savepoint level per blob, the thing your third commit found expensive in production. I only measured it locally, and your earlier numbers show local transaction cost does not predict production, so check it in your harness before taking it.

7. Deflate at level 1

encodeLooseObject is the largest single CPU cost of storing a pack. Level 1 is git's own default for loose objects. It costs 12% more stored bytes on vscode and 9% on TypeScript (26.4 MiB → 28.9 MiB). Your second commit kept the default level on purpose, so this is a call for you.

8. Native inflate (the largest change)

inflateSync(buf, { info: true, maxOutputLength }) works in workerd and reports the input it consumed in engine.bytesWritten, so the decoder no longer needs pako to find where an entry ends. PackReader keeps a window of unread bytes instead of one chunk, and a stream longer than zlib itself would make it is inflated again from twice the input.

Things to weigh:

• It rewrites PackReader, about 90 lines of the hostile-input read path.
• Native inflate holds an entry's whole compressed stream beside its output and a second copy of that output while joining it, about three times the object. Review here worked that through for a 48 MiB blob at about 150 MiB. So since commit 24 it is used only for entries declaring up to 1 MiB, which is all of a mount pack but the odd large tree. Larger entries go back to pako, a read at a time, written once into a buffer of the declared size: about 48 MiB for that entry, where Stream git packs through consumePack instead of buffering them #656 took 96.
• There are two inflaters now, and two as unknown as casts: one for the info result, which @types/node types as a Buffer, and the cast to pako's internals that Stream git packs through consumePack instead of buffering them #656 already had.
• The four decoder tests added with it pass on the pako reader too.
• Commits 18 and 22 close two holes it opened, both raised in review here.
  • The native reader buffers an entry until its stream ends, and a stream can be any length for its output, so the buffer was bounded only by the pack cap. An entry on the native path (up to 1 MiB) may now take at most an eighth over its declared size in compressed input, plus 64 KiB. That is a limit on memory, not a rule of the format: a valid stream past it is refused. No deflater git servers use goes past it, and the vscode, TypeScript and Linux packs all decode within it.
  • The reader asked for as much input as the entry inflates to before its first attempt, so a large object that compresses well waited on megabytes of the pack behind it, and was lost if the stream failed in that wait. It now starts from what is already buffered and doubles only while the stream is unfinished.

9 and 10. Tests and docs

A test loads a small Worker through LOADER that calls consumePack with a default pull stream, the shape production sees; until now every test built a byte stream by hand. Another pins that a ref-delta whose base comes later in the pack is rejected, which main accepted. The consumePack doc in workshop-shared now describes the return value, the delta order rule and what a failed pack leaves stored.

11 to 13. Bound the oversized objects kept as delta bases

consumePack keeps every object over 1 MiB inflated until the pack ends, because a later delta may name one as its base. A blob fetch sends no filter, so a batch read can carry any number of them, and the 64 MiB pack cap bounds only their compressed size. #656 keeps them the same way, in held.

With all three commits:

• Past MAX_OVERSIZED_BASE_BYTES (32 MiB) of kept bases, the ones the pull asked for by name are dropped, oldest first, and the newest is always kept.
• consumePack learns what the pull asked for from the stub: #pullGitObjects now mints the GitCacheImpl it hands to gitPull() with the pull's oids. Any other stub names nothing and drops nothing.
• Dropping a named object is safe because that pull can only end in GitObjectTooLargeError for it, and once measured it is left out of the next request. So when a delta names a dropped base and the pack fails, the retry is a different request.
• That retry happens inside the same read. ensureGitObjects now checks recorded sizes on every round, so the blobs the failed pull measured surface as too large, ensureBlobs takes them out, and the next pull brings the dependent blob whole. A test does this in one ensureBlobs call with two pulls.
• An object the pull did not name is never dropped. A mount names only a commit, so its pack keeps every base, as Stream git packs through consumePack instead of buffering them #656 does, in whatever order the pack carries them.

How it got here: commit 11 dropped from every pack, and Devin pointed out that a mount would then fail for good, since it sends no haves and gets the same pack on every retry. Commit 12 dropped only from packs of blobs alone, and Devin pointed out that blobs can precede the commit. Commit 13 stops inferring the request from the pack. They are separate commits so the review threads stay anchored; I can squash them.

Things to weigh:

• The budget. 32 MiB is my pick against a 128 MB isolate.
• It bounds what a blob fetch keeps only if the gatekeeper sends what was asked for. Objects nobody named, and everything in a mount's pack, are kept without limit, as on Stream git packs through consumePack instead of buffering them #656.
• A single blob over the budget is still kept while it is the newest.
• A blob read that goes over the budget and hits a dropped base costs one extra fetch.
• The line in #pullGitObjects that passes the oids has no test of its own, and nothing here measures Worker memory.

14. Report why a git fetch failed

The pack reaches consumePack() as a stream over RPC. When the gatekeeper's end of it fails, the overseer's reader learns only that the stream ended early, so consumePack() rejects with "ReadableStream received over RPC disconnected prematurely" and gitPull() passed that on. That covers the transfer limit, an error line from the server, a truncated response and the fetch timing out.

I hit it mounting torvalds/linux in local dev. Its depth-1 mount pack is 190.9 MiB against the 64 MiB transfer limit, and the agent was told the connection had dropped, so it retried. pullGitObjectsIntoCache now throws what its own stream failed with. A rejection that is consumePack()'s own still comes through.

Commits 15 to 17 then raise the limit.

15 to 17. Let a larger repository mount

Both 64 MiB caps date from when the whole pack was held in memory: the figure came from gatekeeper-context's artifact sync, where it bounds a repository loaded into memory, and consumePack matched it while it buffered the pack. Since #656 the pack streams through.

• 15 separates the two jobs MAX_GIT_PACK_BYTES did. The per-object inflation cap is still a memory bound and stays at 64 MiB as MAX_GIT_PACK_OBJECT_SIZE. The pack cap now only bounds the work of one pull and goes to 256 MiB, the smallest round figure that admits a depth-1 mount of torvalds/linux (190.8 MiB, 98,591 objects).
• 16 raises MAX_GIT_FETCH_BYTES in gatekeeper-github to match. Commit 19 then removes that limit altogether, so 16 is only needed if 19 is not taken.
• 17 changes the fetch timeout. fetchGitUploadPack gave the whole fetch 120 s, body included. With the pack streaming to the overseer, that budget now covers decoding and storing it: Linux took 70 s through an RPC hop locally. The 120 s now covers the wait for the response, and once the pack is streaming the fetch fails only if the server sends nothing for 60 s while the pull is waiting on it.

Things to weigh:

• 256 MiB is a judgement. A pull that size stores a few hundred MiB of rows (Linux: 222.8 MiB), and nothing evicts them yet.
• Linux still will not mount in production. It is about four times the objects of vscode, which already takes 13 s of CPU there, against a 30 s limit. I left the CPU limit alone.
• The change to fetchGitUploadPack itself has no test; nothing covered its timeout before either.
• I did not run the mount through the UI.

19 and 20. One limit on a pull's size

MAX_GIT_FETCH_BYTES in gatekeeper-github was a second copy of the overseer's pack cap, kept equal to it by hand. The gatekeeper holds none of the body, and consumePack() already rejects a pack over its cap and cancels the stream it was reading, which ends the fetch. I checked that a reader's cancel reaches the source across RPC: in workerd, a default stream handed to another Worker had its cancel() called one pull after that Worker's reader cancelled.

The gatekeeper's limiter also counted the bytes around the pack (framing, the sections before it, progress, keepalives), which the overseer never sees, and each of those that arrives holds off the stall timeout. Commit 19 left them unbounded, which review caught. Commit 23 bounds them without bringing the second copy of the cap back: a response may carry 1 MiB of bytes that are not pack data, plus a sixteenth of the pack delivered so far. A real Linux fetch carries 106,875, or 0.053%.

Testing

• Twenty-four new tests. Each fix has a test that fails without it; the rest pin behaviour that already held.
• The workshop-backend unit suite passes (1,318 tests) and both gatekeeper-github suites pass (97 and 60 tests), along with lint.
• The whole workspace builds with the task cache off (vp run --no-cache build, 76 tasks). Commit 17 went up with a type error because I had trusted vp run build for the package, and it replayed a cached success; commit 21 fixes it.
• The vscode, TypeScript and Linux mount packs (23,280, 65,044 and 98,591 objects) decode at the final commit with the same inflated totals as before, and a sample of oids recomputes.
• Not run: the integration tests, and nothing on production.

Left out

These need a decision more than code: TypeScript's two trees over 1 MiB (the mount reports success but those directories cannot be listed), and the CPU limit (limits.cpu_ms), which is what stops a repository the size of TypeScript or Linux in production.

🤖 Generated with Claude Code

[ask-bonk]

LGTM!

github run

[github-actions]

Preview: pr660-maximo-git-pa-8f15f6a4

https://pr660-maximo-git-pa-8f15f6a4-router.cloudflare-os-previews.workers.dev

Dashboard · deleted when this PR closes

[ask-bonk]

LGTM!

github run

[ask-bonk]

LGTM!

github run

[ask-bonk]

LGTM!

github run

[ask-bonk]

LGTM!

github run

[ask-bonk]

[P2] Try the buffered input before reading ahead by the inflated size. want is based on the uncompressed size, so a highly compressible 2 MiB blob whose complete zlib stream fits in the first 64 KiB still waits for roughly 2 MiB of subsequent pack data, or clean EOF, before it is decoded. If the fetch stream fails during that read-ahead (timeout, transfer limit, or missing final flush), #fill throws even though this entry has already arrived completely. Consequently consumePackFromGatekeeper never records its oversized measurement, and the next read fetches it again; complete following blobs also lose the streaming progress the previous inflater preserved. Start with the available/read-buffer-sized input and grow it only after Z_BUF_ERROR, and cover a complete compressible oversized entry followed by a later stream error.

[Maximo-Guk]

Agreed, fixed in 84353e9. inflate() now starts from what is already buffered, grows to zlib's bound or one read, and doubles from there. There is a test for your scenario: an oversized entry inside the first read is measured though the stream fails on the second.

[ask-bonk]

Review: 1 findings.

Posted 1 actionable inline comment.

github run

[ask-bonk]

[P1] Bound the compressed-input window before allocating it. maxObjectSize bounds the inflated output, but want doubles without a memory bound on Z_BUF_ERROR, and this allocation can exceed the Worker's entire memory limit while the pack remains below MAX_GIT_PACK_BYTES. For example, a valid zlib stream consisting of 23,000,000 empty stored blocks and a final empty block consumes 115,000,011 bytes and inflates to zero bytes. For that zero-size entry, the retry reaches want = 218,103,808, and this expression requests a 218,169,344-byte (~208 MiB) window after receiving only ~104 MiB. I confirmed the stream is accepted by native inflateSync({ info: true, maxOutputLength: 1 }) and traced the allocation using this fill/growth arithmetic. The old chunked inflater did not retain this compressed input. Add an independent compressed-entry/window cap and reject before allocating beyond it, or use a streaming inflater, so an otherwise bounded pack cannot terminate the isolate instead of returning a pack error.

[Maximo-Guk]

Fixed in cef2254, which landed just after this comment: one entry may take at most an eighth over its declared size in compressed input, plus 64 KiB. Your example scaled down (1 MB of empty stored blocks around a 0-byte object) is now refused with a pack error.

[ask-bonk]

Review: 1 findings.

Posted 1 new actionable inline comment. The previously posted read-ahead finding also remains applicable.

github run

[devin-ai-integration]

Note

Newer findings are available below. Devin Review posted a newer report on this PR, in addition to the findings presented here.

Devin Review found 1 new potential issue.

[devin-ai-integration]

🟡 Valid pack entries with short blocks fail

When a valid entry uses many short deflate blocks, longest rejects it before the stream ends. The pack fails to import despite containing valid object data.

Learn more

The compressed length of a valid zlib stream is not limited to an eighth more than its inflated length. Each stored block has five bytes of framing, and a deflater can split the payload into arbitrarily short blocks. The second inflation attempt then receives an incomplete stream and throws Z_BUF_ERROR; the loop exits and the decoder rejects the pack. The prior retry loop accepted the same valid stream.

Example: A 20,000-byte blob stored as 20,000 one-byte deflate blocks produces a valid stream of roughly 120,011 bytes. Its longest value is 88,036 bytes, so decoding rejects the pack instead of yielding that blob.

Recommended fix: Avoid treating a compressed-to-inflated ratio as a validity bound in PackReader.inflate. If the reader must cap per-entry buffering, define it as an explicit resource limit and acknowledge that otherwise-valid streams exceeding it cannot be decoded; alternatively use a streaming inflater that can find the entry boundary without retaining the entire compressed stream.

---

Was this helpful? React with 👍 or 👎 to provide feedback.

[Maximo-Guk]

Agreed that this is a resource limit and not a rule of the format; 84353e9 says so in the comment and the error. The cap stays, because the reader holds an entry's whole stream, and admitting your example would let one entry buffer six times its size.

[ask-bonk]

Review: 1 findings.

The existing read-ahead finding remains applicable at packages/workshop-backend/src/git-codec.ts:484: the reader still fills according to the inflated size before trying already-buffered input.

No new inline comments were posted.

github run

[devin-ai-integration]

Devin Review found 1 new potential issue.

[devin-ai-integration]

🟡 Final flush bypasses fetch overhead limit

When a response chunk ends with a flush, demuxPackData returns before checking accumulated overhead. A single chunk can carry excessive progress data and still complete successfully.

Learn more

The demultiplexer counts body bytes in received and pack bytes in delivered, but checks their difference only after processing every parsed packet in a chunk. Processing a flush exits the generator from inside that loop, so a chunk containing a flush never reaches this check. That is common for a small response received as one chunk, and it can also happen after a large progress packet. The intended response overhead limit is then unenforced for the final chunk.

Example: A fetch body arrives as one chunk containing the packfile header, a small band-1 pack, twenty 60,000-byte band-2 progress packets, and a flush. The generator yields the pack and returns on the flush; it never checks the roughly 1.2 MB of progress against the 1 MB allowance.

Recommended fix: Check the accumulated overhead before returning on a flush, and ensure the check also runs for terminal packets in a chunk. Keep the ordinary between-chunk check for responses that continue without a flush.

---

Was this helpful? React with 👍 or 👎 to provide feedback.

[ask-bonk]

[P1] Budget for the inflater's output copies as well as its input window. A single 48 MiB blob compressed with level 0 is below both current caps, but this reader grows its window to 56,754,176 bytes before decoding it. Workerd's inflateSync(..., { info: true }) uses processChunkSync, which retains the inflated chunks while allocating Buffer.concat(buffers, nread): another two live 48 MiB allocations. Together these require 157,417,472 bytes (150 MiB), already beyond the isolate's 128 MB limit, even with no retained delta bases. I verified the compressed stream with native zlib and traced the current window-growth arithmetic. The previous chunked reader did not retain this additional compressed-input window, so an entry that previously fit can now exhaust memory before its oversized measurement is recorded. Use a streaming inflater, or enforce a smaller per-entry budget accounting for the compressed window and both output allocations before entering this call.

[ask-bonk]

Review: 1 findings.

Posted 1 actionable inline comment.

github run

[ask-bonk]

LGTM!

github run

[ask-bonk]

Review: 1 findings.

The existing final-flush overhead-limit finding remains applicable at packages/gatekeeper-github/src/git-transport.ts:333. I reproduced a single-chunk response accepting 1,200,122 bytes of overhead with only 4 pack bytes, exceeding the intended allowance. Check accumulated overhead before returning on the final flush.

No new inline comments were posted.

github run

[github-actions]

Eval results

Verdict: ⚪ Unchanged. No task moved beyond what 10 runs can tell apart from noise.

Task	Score	Δ score	Fisher test	Cache hits	Avg min	Avg steps
change-calendar	80% → 90%	+10 pp	p = 1.00	86% 0 pp	3.1 → 3.0	23.0 → 24.6
chess	100%	0 pp	p = 1.00	97% 0 pp	9.1 → 8.9	65.5 → 66.5
incident-desk	100% → 90%	−10 pp	p = 1.00	94% → 93% −1 pp	3.7 → 3.3	31.7 → 26.8
worker-logs	100%	0 pp	p = 1.00	92% → 90% −2 pp	3.9 → 3.4	25.1 → 19.9

Failed checks

Task	Check	Failed
change-calendar	t1 schedules-and-lists-windows	1/10 → 0/10
change-calendar	t1 rejects-invalid-windows-without-changing-anything	1/10 → 0/10
change-calendar	t1 overlap-is-per-service-and-touching-is-allowed	1/10 → 0/10
change-calendar	t4 earliest-available-applies-every-rule	1/9 → 0/10
change-calendar	t4 earliest-available-matches-the-reference	1/9 → 0/10
change-calendar	t4 earliest-available-slots-are-bookable	1/9 → 0/10
change-calendar	t4 names-the-booked-windows-the-new-rules-reject	0/9 → 1/10
incident-desk	t1 opens-acknowledges-and-resolves-in-order	0/10 → 1/10
incident-desk	t1 simultaneous-acknowledges-yield-exactly-one-owner	0/10 → 1/10
incident-desk	t1 simultaneous-opens-of-one-id-admit-exactly-one	0/10 → 1/10

Run · trajectories and raw results

[ask-bonk]

🔬 Eval runs review

Performance

Against the supplied baseline, Change Calendar rose from 80% to 90%, Incident Desk fell from 100% to 90%, and Chess and Worker Logs stayed at 100%; none moved beyond noise in 10 runs. Worker Logs shortened from 25.1 to 19.9 steps and 3.9 to 3.4 minutes, with mean cost approximately $0.0220 → $0.0207, but cache hits declined significantly from 92.41% to 90.23%, without an additional miss standing out in the transcripts. Chess remained the largest expense at approximately $0.0614 → $0.0617 per run, with unmatched-edit errors rising from 7 to 16; the candidate’s two failed runs came from an incorrect incident success result and a calendar verifier mismatch.

⚪ VERDICT: NO REGRESSION FROM THIS PR

The comparison is unchanged, and the diff’s git-pack decoding, storage and fetch changes explain neither the candidate failures nor the cache-rate shift, whose turn-boundary misses appear on both sides.

Triage

Failure modes

• Successful inserts reported as duplicates · incident-desk 1/10 · model error · this PR: no — In trial 5, turn 1, writeFile(server.js) made open() depend on inserted.rowsWritten === 1; verification showed fresh incidents persisted while returning DUPLICATE_ID, including all simultaneous opens. The baseline’s passing implementation used INSERT … RETURNING id and checked the returned rows; this PR changes neither generated SQL nor the incident checks.
• Correct calendar audit rejected · change-calendar 1/10 · verifier · this PR: no — In trial 4, turn 4, executeCode read every booked window, and the final reply correctly included mw-101 and mw-103 as TOO_CLOSE: their gap is 21 hours. REJECTED_NOW in packages/workshop-evals/evals/change-calendar.eval.ts omits both despite the prompt requiring an audit across every week; the baseline’s passing replies omitted them too. This verifier is unchanged.

Tool errors

• editFile: No matching text was found in the file. · change-calendar 1 → 0, chess 7 → 16, incident-desk 3 → 1, worker-logs 1 → 2 · model error — Agents quoted text that differed from the file. Candidate Chess trial 5, turn 2, submitted five overescaped replacements together, then reread and corrected them; baseline Chess also recovered through grep and corrected replacements.
• editFile: Validation failed · change-calendar 1 → 2, chess 1 → 4, incident-desk 3 → 7, worker-logs 2 → 1 · model error — Calls omitted required filename or supplied .replacement instead of replacement. Candidate Incident Desk trial 1, turn 3, repeated the malformed .replacement call before correcting it; baseline agents made the same schema mistakes.
• editFile: Multiple matches were found. The text to match must be unique. · change-calendar 2 → 2, chess 2 → 3, incident-desk 3 → 1, worker-logs 1 → 0 · model error — Agents selected repeated fragments despite the parameter explicitly requiring exactly one match. Candidate Change Calendar trial 5, turn 1, tried replacing (await this.readWindows()), then used grep and distinct surrounding lines to recover.
• readFile: File does not exist. · chess 2 → 0, worker-logs 0 → 2 · model error — Baseline Chess trial 2 and candidate Worker Logs trial 6 read server.js and client.js immediately after creating an empty gadget, before writing either file; both subsequently created them.
• executeCode: Failed to start Worker · chess 2 → 1 · model error — Baseline trial 3, turn 2, submitted code: 2, lacking a default export; candidate trial 10, turn 2, submitted a malformed quoted PGN string. Corrected calls succeeded.

Prompt cache

• worker-logs · cache hits 92.41% → 90.23% · breaks 1.65% → 2.14% · this PR: no — Both p-values are 0.023. At turn 2’s first model step, both trial-1 transcripts drop to 6,948 cache-read tokens after gadget creation changes the system prompt’s workspace/file listing; cache writes are 8,487 → 8,956 tokens. No candidate-only miss stands out: fewer model steps, 25.1 → 19.9, provide fewer fully cached steps to dilute the same turn-boundary break, and generated transcript lengths differ. The diff leaves prompt construction unchanged.

What to do

• Optional, separate from this PR: correct REJECTED_NOW in packages/workshop-evals/evals/change-calendar.eval.ts to include mw-101 TOO_CLOSE and mw-103 TOO_CLOSE.
• Optional: add guidance in packages/workshop-backend/src/agent.ts to verify mutation return values against persisted state, not merely assume success from a write counter.
• To evaluate this PR’s intended benefit, add a git-pack task under packages/workshop-evals/evals/ that exercises repository mounting, repeated pulls and interrupted oversized entries; these trajectories do not demonstrate the changed fetch/pack paths.

github run

[ndisidore]

Snagging a few of these: the rest I think its pragmatic to defer until slated runtime changes (cpu top-off)

---

 ┌────┬───────────────────────────────────────────┬───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
 │ #  │ Commit                                    │ Why it holds up under the top-up plan                                                                                                                                                                                     │
 ├────┼───────────────────────────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
 │ 1  │ Bound the pack entry's size varint        │ Has nothing to do with CPU. It closes a hole in the decoder's guarantee never to inflate an entry past its cap. I reproduced it on #656: an entry whose size comes out as NaN, against a 1,024-byte cap, inflated 8 MiB.  │
 │    │                                           │ 6 lines plus a test.                                                                                                                                                                                                      │
 ├────┼───────────────────────────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
 │ 14 │ Report why a git fetch failed             │ With today's 64 MiB cap, a repo that's too large to mount reaches the agent as "disconnected prematurely", so it retries. With this commit the agent gets the real reason. It stays correct after the top-up API ships.   │
 │    │                                           │ ~20 lines, gatekeeper-github only.                                                                                                                                                                                        │
 ├────┼───────────────────────────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
 │ 10 │ consumePack doc in workshop-shared        │ #656 makes "exactly equivalent to put()" untrue, and API docs are required. Drop the "an oid repeats" clause.                                                                                                             │
 ├────┼───────────────────────────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
 │ 4  │ Leave objects the same gatekeeper already │ Its value doesn't depend on CPU: mounting another commit of a repo that's already mounted gets a near-identical pack, and locally it took 6.7–7.9 s → 2.8–3.2 s. Until the API lands, it should also let a reset pull     │
 │    │ stored untouched                          │ resume from the blobs already stored instead of looping. I haven't confirmed that in production. 8 lines. It's the most optional of the five.                                                                             │
 ├────┼───────────────────────────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
 │ 9  │ RPC pack test only                        │ The only test that feeds consumePack the stream production sends, which is what #656's read path depends on. The delta-order test in the same commit is optional.                                                         │
 └────┴───────────────────────────────────────────┴───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘

 - Applying them: I dry-ran each with git merge-tree onto #656. Commits 1, 14 and 10 apply cleanly. Commits 4 and 9 conflict only in test files, where the context comes from commits we're dropping.
 - Size: together they add about 35 lines of production code.

 Defer to the CPU top-up work

 Once a pull can run past 30 s of CPU, the 64 MiB pack cap and the 120 s fetch timeout become the limits that stop large repos. That's when these matter, and these are the conditions for taking them:

 - 15, pack cap to 256 MiB: take it together with a bound on oversized blobs held in memory.
   - Today the 64 MiB cap rejects a blob fetch carrying hundreds of MiB of files over 1 MiB, with a clear error.
   - At 256 MiB those blobs would be held in memory and could exhaust it.
   - That's where the idea behind 11–13 comes back, but a plain "fail with a clear error past a budget" would do.
 - 16, gatekeeper transfer limit raised to match: take it rather than 19 and 23. It's one line, and it keeps the gatekeeper's limit on the raw response body.
 - 17 and 21, stall timeout: needed then. The Linux pull took 70 s locally, so a 120 s total budget would cut it off in production.
 - 19, 20, 23: don't take them, even then. The overhead bound has a live hole at the PR head:
   - demuxPackData returns on the final flush (git-transport.ts:333) before it checks the overhead (git-transport.ts:357).
   - So the last chunk's non-pack bytes are never counted. Devin's comment reports 1,200,122 bytes of overhead accepted around a 4-byte pack.
   - Anyone who takes them must move the check before that return.
   - Better still, keep the raw-body cap. It is simpler, and it has no hole.

 Defer to a performance pass, measured on Cloudflare

 These four no longer decide which repos can mount; they only cut CPU and mount time. Each needs a production run, because local timings haven't predicted production. In order of confidence:

 - 3, skip the redundant pull-routing write: 40% fewer metadata writes, a pure waste removal with no trade-off. Drop its test, which only checks internal state.
 - 6, no transaction per streamed blob: removes one nested savepoint per blob, the pattern that was expensive in production.
 - 2, readAtLeast: about 5.6× fewer reads; the effect was within noise locally.
 - 7, deflate at level 1: about 15% less CPU for 9–12% more stored bytes. It's a trade-off for you to call.

 Drop

 - 8, 18, 22, 24: native inflate. It's about 150 lines rewriting how packs from gatekeepers are read, plus a rule that refuses some valid streams. The gain was about 0.7 s locally, and with CPU top-ups coming that matters even less.
 - 11–13, and 5: the in-memory budget for large blobs isn't reached at today's cap. Revisit it only together with commit 15.

[Maximo-Guk]

Closing, as relevant findings have been cherry-picked