[DevelopmentCats]

Refs #65.

The first agent turn on a fresh or restarted server re-reads the whole fixed prompt. This sends that prompt ahead of time and shows how much of each model's prompts is served from cache. The cache size default is unchanged.

What changes

• WARMUP_FILE (profile key): path to a JSON /v1/chat/completions body. It is sent with max_tokens: 1:
  • after up, restart and switch, once the server is healthy;
  • by the router when a backend goes from not answering to answering (reboot, crash restart).
    It is a plain chat request, so it works on every engine.
• llm-helper warmup capture <profile>: the router saves the next chat request for that model, keeps everything before the first user message, replaces that message with Hi, writes it under the data directory and sets WARMUP_FILE.
• llm-helper warmup [profile]: sends the request twice and prints prompt and cached tokens for each send.
• Cache counts: the router reads usage (streamed or not) from chat replies and counts prompt and cached tokens per model. Shown in status, the panel (cache N%) and GET /router/stats (behind the API key like the rest of the router). Router traffic only.
• vLLM's serve args gain --enable-prompt-tokens-details so it reports cached tokens.
• Docs: a prompt cache section in the llama.cpp part of providers.md (host --cache-ram cache, the hybrid checkpoint limit, setting the size with LLAMA_ARG_CACHE_RAM or EXTRA_ARGS, unified memory), WARMUP_FILE in profiles.md, cache counts in multi-instance.md.

Not covered

• Recipe launches (waitForLaunch) have no warm-up hook after boot; the router still warms them when they come up.
• Recipe container vLLM launches don't get --enable-prompt-tokens-details, so the router shows no cache share for them.
• Where llama.cpp puts its checkpoints is upstream; chats whose system prompt or tools differ still can't share a cached prefix on hybrid models.

Checks

• gofmt -l ., go vet ./..., go test -race ./... (only internal/detect TestProbeReportsThisMachine fails, same as on main), tools/recipediff vet and tests, prettier --check ..
• Manual, llama.cpp from Homebrew on CPU with LiquidAI/LFM2-350M-GGUF:Q4_K_M (hybrid) behind the router: the router warmed the backend when it came up, captured a streamed chat (~2.6k-token system prompt plus a tool), and both warm-ups through the router read 2647 of 2656 prompt tokens from the restored checkpoint. The router counts matched. Not tested on the Sparks or on vLLM.

[Copilot]

🟡 Changes recommended

The capture endpoint allows anyone with the shared inference API key to retrieve another client’s complete request, exposing messages, conversation history, and tool results.

5 open issues

• Restrict request capture to local or admin-only access · New
• Run warm-up independently of CLI watch mode · New
• Reject overlapping captures or notify all waiters · New
• Decode compressed responses before scanning usage · New
• Preserve JSON field order when transforming captured requests · New

What changed in this PR

Adds prompt-cache warm-up and reuse reporting to llm-helper, addressing the first-turn latency described in #65 without changing cache-size defaults.

Changes:

• Adds WARMUP_FILE, capture/manual warm-up commands, and automatic warm-up hooks.
• Tracks cached prompt tokens through the router and exposes them in status, the panel, and /router/stats.
• Enables vLLM token details and documents cache behavior, configuration, and limitations.

File	Description
internal/​web/​static/​machine.js	Displays cache percentages.
internal/​web/​server.go	Adds a cache-stat lookup callback.
internal/​web/​server_internal_test.go	Tests cache display for healthy instances.
internal/​web/​api.go	Includes cache percentages in instance status.
internal/​warmup/​warmup.go	Implements request preparation, sending, capture trimming, and usage parsing.
internal/​warmup/​warmup_internal_test.go	Tests warm-up requests and usage handling.
internal/​router/​models.go	Integrates recovery warm-up, capture, and usage inspection.
internal/​router/​cache.go	Implements cache accounting and capture coordination.
internal/​router/​cache_internal_test.go	Tests accounting, capture, and recovery warm-up.
internal/​profile/​profile.go	Defines the WARMUP_FILE profile setting.
internal/​engine/​registry_test.go	Updates expected vLLM arguments.
internal/​cli/​watch.go	Adds warm-up after watched startup.
internal/​cli/​warmup.go	Adds warm-up commands and cache-reporting helpers.
internal/​cli/​warmup_internal_test.go	Tests CLI warm-up and router integration.
internal/​cli/​vllmargs_internal_test.go	Updates expected launch arguments.
internal/​cli/​serve.go	Connects panel status to router cache counts.
internal/​cli/​router.go	Adds stats/capture endpoints and backend warm-up configuration.
internal/​cli/​root.go	Registers the warm-up command.
internal/​cli/​root_internal_test.go	Updates the pinned command list.
internal/​cli/​profiles.go	Adds cache counts to status output.
internal/​cli/​editset.go	Recognizes WARMUP_FILE as a standard key.
docs/​reference/​cli/​warmup/​capture.md	Adds generated capture-command reference.
docs/​reference/​cli/​warmup.md	Adds generated warm-up-command reference.
docs/​reference/​cli.md	Lists the warm-up command.
docs/​providers.md	Explains llama.cpp cache behavior and sizing.
docs/​profiles.md	Documents warm-up configuration and capture.
docs/​multi-instance.md	Documents router cache reporting and limits.
docs/​manifest.json	Routes the new reference pages.
CHANGELOG.md	Records warm-up and cache-reporting features.
catalogue/​engines/​vllm.json	Enables prompt-token cache details.

_{Review effort: Balanced}