Skip to content

feat(harness): phase 2 part 2 — bounded tools that spill, and an engine fallback ladder - #1293

Closed
aivsomkar wants to merge 3 commits into
phase-2/task-objectfrom
phase-2/reliability
Closed

aivsomkar wants to merge 3 commits into
phase-2/task-objectfrom
phase-2/reliability

Conversation

@aivsomkar

@aivsomkar aivsomkar commented Sep 15, 2026 •

Copy link
Copy Markdown
Collaborator

Phase 2, part 2 of the harness programme (docs/plans/2026-09-15-phase-2.md). Stacked on #1288 (part 1). Spec: docs/superpowers/specs/2026-09-15-phase-2-reliability-design.md.

In plain terms

Tools that fail honestly. Every call a harness tool makes is bounded at 60 seconds; a read that fails on the network is tried once more; an error tells the model which tool, how long, whether it was retried, and what to do next, instead of a bare "fetch failed". A result over 24,000 characters is cut to its first 16,000 and the whole text is kept by the harness (redacted, 0600, per conversation) and read back in slices with tool_result_read, so a long listing no longer floods the context or gets lost.

Engines that fall back (decision 12). A bot may name an alternate engine (fallback.alternate on PATCH /api/bots/:id). When a direct turn fails on a terminal provider error (quota, auth, overloaded, rate limits past the driver's own retries, server errors) the harness switches the task to the alternate, records task.fallback = { from, to, reason, at }, says so in the thread ("continuing on Grok after quota on claude"), and re-sends the exact message that failed once the failed turn has settled; the fresh-session replay carries the history. Two things stop the silent rung and leave it to the person: the failed turn had a side effect, or there is no alternate. Then, for a quota error, the card from #1184 (imported here) offers the other engines; otherwise the thread shows the error as today. A task falls back silently at most once.

Measured

  • server/fallback.e2e.test.ts (real harness, fake Claude failing the way an out-of-quota CLI does, fake ACP as the alternate): the bot with an alternate gets its reply from the alternate, the task carries the record, the chip is there, the ledger's row names the alternate's driver; the bot without one gets the card and no reply.
  • Proxy suite: the 60 s bound and retry rule, the teaching errors, the cap, the spill and the slices.
  • T1–T10 and the recall set unchanged by construction: no scorecard bot names an alternate and the bounds do not fire on a healthy harness.

Every engine

Full on every engine: every engine mounts the same proxy, and the alternate can be any engine the workspace has; the failed message is re-dispatched with the same fresh-session replay an engine switch uses.

PRs checked (AGENTS.md rule 2)

Platforms

Platform Impact Verified
macOS server (proxy, fold, routes); the card renders through the existing option card suites, e2e, typecheck, lint
Windows server; POSIX-gated e2e like branching.test.ts CI
iOS / Android none —

Checks

pnpm typecheck, pnpm lint, pnpm i18n:check clean; the suites the wiring touches green (the two that failed in one batch run pass alone and in the full suite: a shared temp folder, not the code). Full suite on the Phase 2 tip: see #1294 (parts 3–4), which carries the run for the whole branch.

🤖 Generated with Claude Code

aivsomkar and others added 3 commits September 15, 2026 23:50
When a bot's provider quota runs out mid-conversation, raise a
harness-native card in its thread offering the other configured engines
(server/quota-switch.ts, following the peer-approval.ts card pattern).
Choosing one switches modelSelection to a valid model for that engine
(reusing default-model-selection.ts) and re-dispatches the last user
message; "Not now" just settles the card. At most one card is pending
per thread at a time.

Also enrich the inline replay startTurn builds for a rewound/fresh
engine with a compact line per tool call (server/turn-context.ts's new
formatToolActivityLine), so an engine joining mid-thread learns which
files were read or commands ran, not just what was said — the
text-only transcript used for engineIsFresh, recovery text, and
native-replay drivers is untouched.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…xact message

Round 1 review fixes:
- maybeRaiseQuotaCard now checks each candidate's snapshot() before
  offering it, so an unavailable engine never reaches the card (the
  click-time check in pickSelection stays as a second line of defence,
  but now posts a visible chip instead of failing silently).
- OptionCardData gains fixedOptions (server/store.ts and its frontend
  duplicate in src/state/store.tsx); OptionCard.tsx stops rendering the
  free-text box for a fixedOptions card. An answer naming none of the
  offered engines still settles the card as "Not now" server-side.
- switchAndRedispatch re-dispatches the exact user message captured at
  card-raise time (threaded through from the runtime.error handler's
  active-path lookup) instead of re-scanning at answer time, fixing
  stale/wrong/dropped resends including the image-only case; a message
  that can no longer be found posts a "could not be resent" chip
  instead of a false "continuing" one.
- switchAndRedispatch also bails without writing a chip when patchBot
  returns null (bot deleted meanwhile).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… an engine fallback ladder (phase 2 part 2)

Every harness call a tool makes is bounded (60 s), a read that fails on the
network is retried once, errors say what to do next, and a result over 24k
characters is cut with the whole text kept by the harness and read back
through tool_result_read. On a terminal provider failure a direct turn
continues on the bot's named alternate (fallback.alternate), the switch is
recorded on the task and said in the thread, and the exact failed message
is re-sent once the failed turn settles; with side effects or no alternate,
#1184's quota card asks the person instead (decision 12).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@vercel

vercel Bot commented Sep 15, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
openmausbot-docs Ready Ready Preview Sep 15, 2026 6:58pm UTC

Request Review

@coderabbitai

coderabbitai Bot commented Sep 15, 2026

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: 84727e4c-ca84-4fff-816e-4f29e6dc2c42

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

milind-soni added a commit that referenced this pull request Sep 20, 2026
…1561)

* fix(tools): bound agent replies with scoped temporary retrieval

Adapt the bounded-result capability from #1293 without its retry or provider fallback behavior.

Co-authored-by: aivsomkar <aivsomkar@gmail.com>

* test: always close the tool-result verification fixture

---------

Co-authored-by: aivsomkar <aivsomkar@gmail.com>
@milind-soni

Copy link
Copy Markdown
Owner

Closing at the maintainer’s request: we are deferring the Phase 2/3 harness stack to keep the current release scope smaller and reduce regression risk. This is a scope decision, not a finding that every change here is defective. The branch is being preserved so focused changes can be revisited separately.

This branch was successfully deployed

1 active deployment
Preview — a55b59d1 Deployed Sep 15, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants