Skip to content

Read a skill choice the model wrapped in a code fence, the way the router reads its answer - #577

Merged
davidmckayv merged 3 commits into
CopilotKit:mainfrom
kevin9327:read-fenced-skill-choice
Sep 16, 2026
Merged

davidmckayv merged 3 commits into
CopilotKit:mainfrom
kevin9327:read-fenced-skill-choice

Conversation

@kevin9327

@kevin9327 kevin9327 commented Sep 16, 2026

Copy link
Copy Markdown
Contributor

What this changes

Before a run, selectTools asks the deployment's model which of a Bot's skills the message needs, so a Bot holding more than SELECTION_FLOOR tools is offered only the relevant ones. readChosenSkills passed the whole reply to JSON.parse.

The request goes through createModelCompleter with response_format: { type: "json_object" }, but that field is a request, not a guarantee. Anthropic's OpenAI-compatible endpoint lists response_format as "Ignored" in its compatibility table, and a model answering freely often writes

```json
{"skills": ["drive-audit"]}
```

or a sentence before the object. Each of those parsed as nothing. selectTools recorded reason: "unavailable" on the mcp.tools_discovered row and offered every granted tool, on every run for a deployment on that model. Narrowing never happened, although the model had answered correctly.

The router already handles this. It uses the same completer and the same response_format, and since #122 classify.ts has read the object out of the answer ("The model is asked for bare JSON, but tolerate a fenced or padded answer"). routing-classify.test.ts has a case for exactly the fenced shape. Selection arrived a day later (#178) without that tolerance.

readChosenSkills now takes the object out of the answer with the router's own expression, /\{[\s\S]*\}/, and parses that. Every existing refusal still returns null: no object ("not json", "[]", "null", prose), an object of the wrong shape ({}, {"chosen": …}), and unparseable JSON. So every failure path still offers everything, which is the direction the module header requires. A bare JSON answer is read exactly as before.

Where it runs

  • New state that outlives a request? None.
  • What happens on the second replica? The same. This is a pure function of the model's reply.
  • Anything serialised? No.
  • Anything fanned out to a browser? No.
  • New listener, port, or schedule? No.

Boundary and audit

  • Every acting call still goes through the gateway: resolve, decide, audit, then act. Selection is not a boundary, as the module says. The offer is still intersected with the grant, and callTool is untouched.
  • New refusals and new failures each write a row. None are added. The existing mcp.tools_discovered row now says selected when the model did select.
  • Nothing new is trusted from the client that the server can resolve itself. A slug the model names that is not a granted skill is still dropped.

Changelog

  • A line in CHANGELOG.md under Unreleased.

Proof

Two new cases in server/tests/plugin-selection.test.ts:

  • what gets offered > a choice wrapped in a code fence still narrows, rather than reading as no answer: selectTools with a selector that fences {"skills": ["drive-audit"]}.
  • reading pass one's answer > a fenced or padded answer is read, the way the router reads its own: readChosenSkills on a fenced answer and on one led by a sentence.

On main:

error: expect(received).toBe(expected)
Expected: "selected"
Received: "unavailable"
(fail) what gets offered > a choice wrapped in a code fence still narrows, rather than reading as no answer

error: expect(received).toEqual(expected)
- [ "drive-audit" ]
+ null
(fail) reading pass one's answer > a fenced or padded answer is read, the way the router reads its own

 21 pass
 2 fail

With the fix:

  • cd server && bun test tests/plugin-selection.test.ts tests/copilot.test.ts tests/routing-classify.test.ts tests/routing-routes.test.ts: 130 pass, 0 fail. This includes every existing "every failure offers everything" and "null for anything that is not an object with a list" case, unchanged.
  • bun run typecheck (app, server, worker): exit 0.
  • bunx biome check on the changed files: clean.

Every changed line runs. bun test --coverage tests/plugin-selection.test.ts reports selection.ts at 100% of lines. "not json" now returns before parsing, so an object that does not parse ({skills: drive-audit}) is asserted as null directly. That keeps the catch covered.

This PR adds its line at the top of ## Unreleased, the same place other open PRs add theirs, so the changelog may conflict. I'm happy to rebase.

🤖 Generated with Claude Code

…uter reads its answer

Tool selection asks the deployment's model which skills a message needs
and parsed the whole reply with `JSON.parse`. The request carries
`response_format: json_object`, but that is a request: Anthropic's
OpenAI-compatible endpoint documents the field as ignored, and a model
left to itself often answers with a ```json fence or a sentence before
the object. Every such reply parsed as nothing, `selectTools` recorded
`unavailable`, and a Bot on that model was offered its whole catalogue
on every run.

The router asks the same completer the same way and has read the object
out of its answer since it was written (`classify.ts`, with a test for
the fenced shape). `readChosenSkills` now does the same. A reply with no
object in it, or an object of the wrong shape, is still null, so every
existing failure case offers everything as before.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

kevin9327 and others added 2 commits September 16, 2026 19:41
The unparseable-JSON branch used to be reached by "not json", which now
returns before parsing because it holds no object. An object that does
not parse still has to read as null, so it is asserted directly.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

@davidmckayv davidmckayv left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code-verified clean; CI green on this sha.

@davidmckayv
davidmckayv merged commit ab9c493 into CopilotKit:main Sep 16, 2026
15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants