Skip to content

inspect: print the engine's note on hybrid models - #68

Open
DevelopmentCats wants to merge 1 commit into
mainfrom
devcats/inspect-hybrid-note
Open

DevelopmentCats wants to merge 1 commit into
mainfrom
devcats/inspect-hybrid-note

Conversation

@DevelopmentCats

Copy link
Copy Markdown
Collaborator

Part of #65.

On hybrid models (linear-attention or Mamba layers), llama.cpp reuses a cached prompt only up to the start of a user message, so the first turn of a chat re-reads everything before it unless that part matches an earlier chat. inspect now says so.

  • hf.Arch.Hybrid() reads layer_types: a model is hybrid when some layers are not attention.
  • resolve.Detail gets hybrid and hybrid_note. ForProvider fills the note from the serving engine.
  • Engines can carry an optional notes.hybrid sentence. Only catalogue/engines/llamacpp.json sets one, so the wording is data, not Go.
  • Documented in catalogue/engines/README.md and the engine fields in docs/providers.md. CHANGELOG entry added.

The warm-up side of #65 is #67.

Copilot AI balanced review requested due to automatic review settings November 10, 2026 23:54

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues remain for the hybrid-model inspection guidance change.

No open issues 🎉

What changed in this PR

Adds engine-specific guidance to inspect for hybrid models, addressing the explanatory part of #65 without changing prompt-cache behavior or adding warm-up logic.

Changes:

  • Detects hybrid models from architecture layer_types and exposes hybrid details during inspection.
  • Adds optional engine notes and displays llama.cpp’s guidance on keeping system prompts and tools identical between chats.
  • Adds detection and provider-note tests, documentation, and a changelog entry.
File Description
internal/​resolve/​resolve.go Carries hybrid detection and the selected engine’s note.
internal/​resolve/​resolve_internal_test.go Tests when provider-specific hybrid notes are populated.
internal/​hf/​kv_internal_test.go Tests hybrid detection, including nested text configuration.
internal/​hf/​arch.go Detects hybrid architectures from layer types.
internal/​engine/​registry.go Exposes hybrid notes through the provider interface.
internal/​engine/​engine.go Adds engine note fields and an accessor.
internal/​cli/​search.go Prints the hybrid note in inspection output.
docs/​providers.md Documents optional engine notes.
CHANGELOG.md Records the new inspection guidance.
catalogue/​engines/​README.md Explains how engine definitions supply hybrid notes.
catalogue/​engines/​llamacpp.json Supplies llama.cpp’s hybrid prompt-cache guidance.

Review effort: Balanced

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants