llama.cpp: the first turn of an agent chat reads the whole fixed prefix again on hybrid models #65

On a Mac Studio running GLM-5.3-Flash with llama.cpp, the first turn of a Coder Agents chat waited 17 s before the first token. Most of the prompt was the same text that every chat starts with. llama.cpp can reuse that text, but a llama.cpp profile does nothing to make sure it does.

Why

Agent clients send a large fixed prefix with every new chat. For one Coder Agents chat, Bifrost's log showed 9,397 prompt tokens:

At about 545 prompt tokens/sec on the Mac, that is 17.3 s to read and 3.1 s to write the answer. Later turns of the same chat start in under a second. A new chat pays the whole 17 s again unless the server finds the prefix in its cache.

vLLM's prefix cache works per block, so any shared prefix is reused. llama.cpp works differently for GLM-5.3-Flash, because 34 of its 45 layers are linear attention. A recurrent state can not be cut back to the point where two prompts stop matching. llama.cpp can reuse a prefix only at a saved context checkpoint (ggml-org/llama.cpp#15293). It saves one at the start of the last user message (and at earlier user messages, at most one every 8,192 tokens by default), and it keeps old prompts in a host-memory cache (--cache-ram, ggml-org/llama.cpp#16391).

What the log shows

llama.cpp 0.8.0 (build 11960) on the Mac, with the profile from #64:

• The slow chat matched 49% of a slot's cached prompt (f_keep = 0.491), and llama.cpp still read all 9,397 tokens. The other chat had 21 tools and this one had 22 (find_tools added). The system messages were byte for byte the same. The prompts split at about 4,000 tokens, before any checkpoint, so nothing was reusable.
• Two later chats with the same tools as a cached one read only 20 and 18 new tokens. The prefix came back from the host cache in about 0.2 s.

So reuse works when a chat with the same prefix ran recently and is still in the cache. It misses for the first chat after a start or a switch, for a new tool set, and after a busy period evicts the entry. The default --cache-ram is 8,192 MiB. The log shows entries of 0.1 to 1.3 GiB evicted under load.

Ideas

1. Size --cache-ram from the memory that is left after the model and KV cache, like the GPU_MEMORY_UTILIZATION fit does for vLLM, and not the 8 GiB default.
2. Warm the cache after start and switch. For example, a profile setting that names a file with a request body (the client's fixed prefix), which llm-helper sends once with max_tokens: 1 after health turns green.
3. Keep slot states across a restart with --slot-save-path and the /slots save and restore actions. I have not checked if a saved slot includes the checkpoints that a hybrid model needs.
4. Show cache reuse in status or the panel: prompt tokens read against prompt tokens reused, from the server log or /metrics.
5. Write the inspect result for hybrid models in plain words: prefix reuse needs a checkpoint at the split point.

1 and 2 look like the smallest changes that help. With 4 we can see if they work.

Not tested yet

• A larger --cache-ram on the Mac.
• A warm-up request after a restart.
• --checkpoint-min-step lower than 8,192. The checkpoint at the start of the last user message is not limited by it, and a new chat has only one user message, so I do not expect it to help here.

Generated by Coder Agents on behalf of @bpmct.