Read a skill choice the model wrapped in a code fence, the way the router reads its answer - #577
Merged
Conversation
…uter reads its answer Tool selection asks the deployment's model which skills a message needs and parsed the whole reply with `JSON.parse`. The request carries `response_format: json_object`, but that is a request: Anthropic's OpenAI-compatible endpoint documents the field as ignored, and a model left to itself often answers with a ```json fence or a sentence before the object. Every such reply parsed as nothing, `selectTools` recorded `unavailable`, and a Bot on that model was offered its whole catalogue on every run. The router asks the same completer the same way and has read the object out of its answer since it was written (`classify.ts`, with a test for the fenced shape). `readChosenSkills` now does the same. A reply with no object in it, or an object of the wrong shape, is still null, so every existing failure case offers everything as before. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
kevin9327
requested review from
MikeRyanDev,
davidmckayv,
guidovizoso,
mxmzb and
tylerslaton
as code owners
September 16, 2026 10:40
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
The unparseable-JSON branch used to be reached by "not json", which now returns before parsing because it holds no object. An object that does not parse still has to read as null, so it is asserted directly. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
# Conflicts: # CHANGELOG.md
davidmckayv
approved these changes
Sep 16, 2026
davidmckayv
left a comment
Contributor
There was a problem hiding this comment.
Code-verified clean; CI green on this sha.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this changes
Before a run,
selectToolsasks the deployment's model which of a Bot's skills the message needs, so a Bot holding more thanSELECTION_FLOORtools is offered only the relevant ones.readChosenSkillspassed the whole reply toJSON.parse.The request goes through
createModelCompleterwithresponse_format: { type: "json_object" }, but that field is a request, not a guarantee. Anthropic's OpenAI-compatible endpoint listsresponse_formatas "Ignored" in its compatibility table, and a model answering freely often writesor a sentence before the object. Each of those parsed as nothing.
selectToolsrecordedreason: "unavailable"on themcp.tools_discoveredrow and offered every granted tool, on every run for a deployment on that model. Narrowing never happened, although the model had answered correctly.The router already handles this. It uses the same completer and the same
response_format, and since #122classify.tshas read the object out of the answer ("The model is asked for bare JSON, but tolerate a fenced or padded answer").routing-classify.test.tshas a case for exactly the fenced shape. Selection arrived a day later (#178) without that tolerance.readChosenSkillsnow takes the object out of the answer with the router's own expression,/\{[\s\S]*\}/, and parses that. Every existing refusal still returnsnull: no object ("not json","[]","null", prose), an object of the wrong shape ({},{"chosen": …}), and unparseable JSON. So every failure path still offers everything, which is the direction the module header requires. A bare JSON answer is read exactly as before.Where it runs
Boundary and audit
callToolis untouched.mcp.tools_discoveredrow now saysselectedwhen the model did select.Changelog
CHANGELOG.mdunderUnreleased.Proof
Two new cases in
server/tests/plugin-selection.test.ts:what gets offered > a choice wrapped in a code fence still narrows, rather than reading as no answer:selectToolswith a selector that fences{"skills": ["drive-audit"]}.reading pass one's answer > a fenced or padded answer is read, the way the router reads its own:readChosenSkillson a fenced answer and on one led by a sentence.On
main:With the fix:
cd server && bun test tests/plugin-selection.test.ts tests/copilot.test.ts tests/routing-classify.test.ts tests/routing-routes.test.ts: 130 pass, 0 fail. This includes every existing "every failure offers everything" and "null for anything that is not an object with a list" case, unchanged.bun run typecheck(app, server, worker): exit 0.bunx biome checkon the changed files: clean.Every changed line runs.
bun test --coverage tests/plugin-selection.test.tsreportsselection.tsat 100% of lines. "not json" now returns before parsing, so an object that does not parse ({skills: drive-audit}) is asserted asnulldirectly. That keeps thecatchcovered.This PR adds its line at the top of
## Unreleased, the same place other open PRs add theirs, so the changelog may conflict. I'm happy to rebase.🤖 Generated with Claude Code