Add offline zstd dictionary tooling

[lukesandberg]

What?

Adds offline zstd dictionary training/evaluation tooling and enables a checked-in 64 KiB static dictionary for the TaskData persistence family.

Family	Runtime compression
Infra	LZ4
TaskMeta	LZ4
TaskCache	LZ4
TaskData	zstd3 + embedded dictionary

Why?

TaskData is largely serialized state derived from JavaScript/CSS source and dependencies, so a representative static dictionary can remove repetition that ordinary zstd cannot capture independently in each value/block.

Dictionary iteration should not require rebuilding applications. zstd_dictionary trains and evaluates candidates from copied cache directories, while a resumable script builds a corpus from maintained test/production apps.

How?

FamilyConfig now stores CompressionConfig::{Lz4, Zstd3, Zstd3WithDictionary}. The configuration is threaded through SST/blob reads, writes, compaction, iteration, and block caches. Writers reuse a prepared dictionary compressor; readers retain one thread-local zstd context and reload its dictionary only when the configured dictionary changes.

Meta files record both the compression algorithm and zstd dictionary ID. Database open validates both before serving reads, so a missing/wrong dictionary fails early rather than as an opaque decompression error. The meta magic changes because the header gained the dictionary-ID field.

zstd_dictionary:

• uses StaticSortedFileIter to read logical slice/medium/blob values;
• samples cache directories round-robin to a fixed 64 MiB target;
• trains a fixed 64 KiB dictionary;
• evaluates candidates against zstd3 without a dictionary and round-trips every result;
• decodes each source SST according to its recorded LZ4/plain-zstd/dictionary-zstd metadata;
• accepts --source-dictionary only for SSTs already written with a dictionary;
• reports source codec mixes and combined comparison totals rather than cache-layout/per-kind details.

sst_inspect also accepts --source-dictionary so dictionary-compressed caches remain inspectable.

The checked-in train-taskdata-dictionary.sh runs production tests in isolated TMPDIRs with cleanup disabled, copies valid cache databases, hashes test paths into an 80/20 train/holdout split, resumes completed cases, trains, and performs five holdout evaluations.

Held-out receipts

The sandbox completed 194 of 362 production test files before its disk budget was exhausted, yielding 204 training caches and 38 held-out caches. This limitation is recorded beside the dictionary; the script is resumable in a larger environment.

Held-out logical input: 1,462,462,144 bytes.

Metric	zstd3	Dictionary	Delta
Raw compressed bytes	674,350,217	460,786,575	-31.67%
Median encode diagnostic (5 runs)	18.646 s	9.132 s	-51.02%
Median decode diagnostic (5 runs)	5.822 s	3.030 s	-47.91%

The raw saved bytes equal 29.32% of the copied holdout cache-directory total. That is directional rather than an exact rewritten-cache measurement. Timing is machine-specific diagnostic data; stable byte counts are the primary receipt.

Dictionary provenance:

• size: 65,536 bytes
• zstd dictionary ID: 1166300072
• xxh3-64: 2d9e019e3c1071f1

Verification

cargo test -p turbo-persistence
cargo test -p turbo-persistence --all-targets --no-run
cargo test -p turbo-tasks-backend
cargo fmt --all -- --check
cargo clippy -p turbo-persistence -p turbo-tasks-backend --all-targets -- -D warnings
pnpm lint-ast-grep
pnpm build-all

A real dictionary-compressed Next.js cache was also checked manually: opening without --source-dictionary fails with the recorded/expected IDs, while zstd_dictionary and sst_inspect both succeed with the embedded dictionary. A current LZ4 TaskMeta cache also trains/evaluates directly without conversion or a source dictionary.

Known limits:

• The sample target can overshoot by one large value/blob.
• zstd copies selected samples into one contiguous training buffer.
• This baseline corpus did not complete the final 168 production test files due to local disk limits.
• Native binary size was not independently re-measured; embedding adds at least the 64 KiB dictionary bytes.

[github-actions]

Failing test suites

Commit: 5f71ff4 | About building and testing Next.js

pnpm test-start-turbo test/production/app-dir/app-fetch-build-cache/app-fetch-build-cache.test.ts (turbopack) (job)

• app fetch build cache > should not use stale data if present (DD)

Expand output

● app fetch build cache › should not use stale data if present

thrown: "Exceeded timeout of 60000 ms for a test.
Add a timeout value to this test to increase the timeout, if this is a long-running test. See https://jestjs.io/docs/api#testname-fn-timeout."

  546 |           gatedBodies.has(callback)
  547 |         ) {
> 548 |           return Reflect.apply(target, thisArg, args)
      |                          ^
  549 |         }
  550 |
  551 |         const applicable = [...describeGateStack]

  at Object.apply (lib/gate/runtime.ts:548:26)
  at Object.apply (lib/e2e-utils/index.ts:65:30)
  at it (production/app-dir/app-fetch-build-cache/app-fetch-build-cache.test.ts:18:3)
  at Object.describe (production/app-dir/app-fetch-build-cache/app-fetch-build-cache.test.ts:3:1)