Repository navigation
inspect: print the engine's note on hybrid models - #68
Open
DevelopmentCats wants to merge 1 commit into
Open
DevelopmentCats wants to merge 1 commit into
DevelopmentCats wants to merge 1 commit into
Conversation
There was a problem hiding this comment.
🟢 Approval recommended
No active review issues remain for the hybrid-model inspection guidance change.
No open issues 🎉
What changed in this PR
Adds engine-specific guidance to inspect for hybrid models, addressing the explanatory part of #65 without changing prompt-cache behavior or adding warm-up logic.
Changes:
- Detects hybrid models from architecture
layer_typesand exposes hybrid details during inspection. - Adds optional engine notes and displays llama.cpp’s guidance on keeping system prompts and tools identical between chats.
- Adds detection and provider-note tests, documentation, and a changelog entry.
| File | Description |
|---|---|
internal/resolve/resolve.go |
Carries hybrid detection and the selected engine’s note. |
internal/resolve/resolve_internal_test.go |
Tests when provider-specific hybrid notes are populated. |
internal/hf/kv_internal_test.go |
Tests hybrid detection, including nested text configuration. |
internal/hf/arch.go |
Detects hybrid architectures from layer types. |
internal/engine/registry.go |
Exposes hybrid notes through the provider interface. |
internal/engine/engine.go |
Adds engine note fields and an accessor. |
internal/cli/search.go |
Prints the hybrid note in inspection output. |
docs/providers.md |
Documents optional engine notes. |
CHANGELOG.md |
Records the new inspection guidance. |
catalogue/engines/README.md |
Explains how engine definitions supply hybrid notes. |
catalogue/engines/llamacpp.json |
Supplies llama.cpp’s hybrid prompt-cache guidance. |
Review effort: Balanced
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #65.
On hybrid models (linear-attention or Mamba layers), llama.cpp reuses a cached prompt only up to the start of a user message, so the first turn of a chat re-reads everything before it unless that part matches an earlier chat.
inspectnow says so.hf.Arch.Hybrid()readslayer_types: a model is hybrid when some layers are not attention.resolve.Detailgetshybridandhybrid_note.ForProviderfills the note from the serving engine.notes.hybridsentence. Onlycatalogue/engines/llamacpp.jsonsets one, so the wording is data, not Go.catalogue/engines/README.mdand the engine fields indocs/providers.md. CHANGELOG entry added.The warm-up side of #65 is #67.