[bpmct]

First step toward GLM-5.3-Flash on a Mac Studio. inspect sized its KV cache about 50 times too large, so a 256 GB Mac looked like it could hold almost no context:

before: context: ~379k tokens in 279.4 GB of KV cache
after:  context: ~4194k tokens in 59.1 GB of KV cache

GLM-5.3-Flash (glm5_next) has 45 layers. 34 are Kimi Delta linear attention, which keep a fixed-size state and no growing cache. 11 are latent attention (MLA) with a sparse-attention indexer. Two things went wrong:

• layer_types calls the linear layers linear_attention. That matched "attention", so they counted as full attention.
• The latent layers counted as 64 full K/V heads. MLA caches one latent per token: kv_lora_rank + qk_rope_head_dim, plus the indexer key here.

Now a linear_attention layer holds no cache, and a layer with kv_lora_rank holds kv_lora_rank + qk_rope_head_dim + index_head_dim. For GLM-5.3-Flash that is 11 × 640 × 2 = 14,080 bytes a token, about 15 GB at 1M tokens. The indexer key is stored narrower than this counts it, so the estimate errs high, which is the safe side.

This also changes DeepSeek V2/V3 and V3.2: their estimate falls about 25 times, to the 576 per layer vLLM allocates. DeepSeek V4 has no kv_lora_rank and does not change.

This PR also adds GLM-5.3-Flash on the Mac Studio to docs/providers.md, measured. It runs Unsloth's UD-Q4_K_XL (186 GiB) plus mmproj-F16.gguf on mainline llama.cpp 0.8.0 from Homebrew (GLM-5.3-Flash since ggml-org/llama.cpp#27773, MTP since #29928), with --spec-type draft-mtp. MLX was the other option. GLM-5.3-Flash runs only under mlx-vlm there, its glm5_next port is on main with no release, and a review found numeric bugs, so llama.cpp is the safer choice.

On the Mac (M5 Ultra, 256 GB), idle, straight to llama-server:

1 request:   ~58 tok/s out
4 at once:   ~16 tok/s each, 63 total
prompt read: ~585 tok/s (7,600 tokens, 13 s to first token)
MTP accepted 46 to 68% of drafts

Tool calls and images work. The clean numbers are from Huihui's abliterated copy of the same quant, which has the same shapes. An earlier run on Unsloth's copy shared the Mac with live traffic, so its numbers are not clean. I wrote the profile file by hand on the Mac (sudo there asks for a password, so new could not restart the service). It has the same PROVIDER, MODEL and EXTRA_ARGS as the doc, but I did not run the doc's new and set commands myself.

Tested: gofmt -l . is empty, go vet ./... and GOOS=darwin go vet ./... pass, go test -race passes for hf, resolve and cli, and go test ./... passes. The new test reads a trimmed copy of the real zai-org/GLM-5.3-Flash config. inspect unsloth/GLM-5.3-Flash-GGUF gives the numbers above. Also merged main (no conflicts), and go vet ./... and go test ./... pass on the merge.

Generated by Coder Agents on behalf of @bpmct.

[Copilot]

🟡 Changes recommended

Pure-linear configurations still trigger the full-attention fallback and overestimate memory, while the documented setup omits the restart needed to apply its settings.

2 open issues

• Exclude linear layers from the full-attention fallback · New
• Restart the server after applying EXTRA_ARGS · New

What changed in this PR

This PR corrects KV-cache estimates for GLM-5.3-Flash and latent-attention models, improving memory and context sizing across inspect, new, and the web panel.

Changes:

• Excludes linear-attention layers from cache counting and sizes latent caches using their compressed dimensions.
• Adds regression tests and a trimmed GLM-5.3-Flash configuration fixture.
• Documents measured Mac Studio usage and records the sizing correction in the changelog.

File	Description
internal/​hf/​testdata/​glm-5.3-flash-config.json	Adds model dimensions and layer types for regression testing.
internal/​hf/​kv_internal_test.go	Tests hybrid, latent, and linear-attention cache sizing.
internal/​hf/​arch.go	Updates layer counting and latent-cache calculations.
docs/​providers.md	Adds a Mac Studio example with settings and measured performance.
CHANGELOG.md	Records the user-visible cache-sizing correction.

_{Review effort: Balanced}