Skip to content

Size the KV cache to the request rather than model.config.max_seq_len - #1327

Closed
risjain wants to merge 1 commit into
fishaudio:mainfrom
risjain:perf/size-kv-cache
Closed

Size the KV cache to the request rather than model.config.max_seq_len#1327
risjain wants to merge 1 commit into
fishaudio:mainfrom
risjain:perf/size-kv-cache

Commits

  1. Commits on Aug 21, 2026