Repository navigation
Deterministic routines: structured outputs and generated scripts with hybrid escalation #1506
Description
Area
Harness server / API (server/)
Problem
Every routine run is an LLM turn today, whatever the job. A routine that only checks an API, counts items, or writes a status line still pays for a model call and gets prose back: same input, different words, occasional format drift. Predictable outcomes are impossible when the reporting layer is generative. From the DM: some routines should use structured outputs, and sometimes generated scripts instead of LLM calls.
Proposal
Three execution modes for routines, plus escalation:
- LLM, unchanged, for judgment work.
- Structured output: the routine declares a JSON schema, the run returns validated data (provider-native structured mode where the driver supports it, validated JSON with one repair retry otherwise), run cards render from the data, and validation failures fail the run loudly instead of degrading to prose.
- Scripted: the model generates a script once at authoring time, then the routine executes it on schedule with zero LLM calls. Sandboxed with a capability allowlist (named HTTP hosts, routine-owned directory, no shell), timeout, no embedded credentials (referenced by id from the credential store), versioned and review-gated like skills so edits restage before the next run.
- Hybrid escalation: the script collects data deterministically every run and hands off to an LLM turn only when a rule fires (output changed, threshold crossed). Deterministic front, judgment on demand.
Deterministic runs carry exit code, duration, and script version hash in the receipt and are marked deterministic on the run card. Dispatch semantics, deferral surfacing, and busy handling stay unchanged. The routine editor gains a mode selector, defaulting to LLM.
Related: #1503 (exact assertions for scripted and structured runs), #1305 and #1309 (deferral behavior unchanged). Sequencing: holds until the code-health stack (#1402 to #1498) merges.
Open questions: script language and runtime, sandbox boundary (Box VM versus in-process worker), escalation rule UI in v1 or a one-line expression, provider-native structured output per driver or one universal validated-JSON path first.
Alternatives considered
- LLM only with stricter prompt formatting: still probabilistic, still pays a model call for mechanical work.
- External cron calling webhooks: loses receipts, run history, continuity, and the in-app surface.
- Full code-first automation platform: too much for non-technical users; scripts stay generated and reviewed, never hand-required.
Activity
adamczhang commentedon Sep 18, 2026
@bradhallett Strong +1 on this proposal. Recurring mechanical tasks (API status checks, counter diffs, heartbeat monitors) paying full LLM turn latency and token costs—while risking format drift or hallucinations in automated reporting—is one of the biggest pain points for background agent routines.
The hybrid escalation model ("deterministic front, judgment on demand") strikes the right balance between the zero-cost predictability of cron and the reasoning capabilities of OpenMausBot agents.
A few thoughts and open questions on the implementation path:
Execution Sandbox & Boundary:
QuickJS/isolated-vm) vs. delegating to the existing Box VM or Local VM container?Phased Rollout:
Escalation Trigger Syntax:
{ "field": "status", "operator": "neq", "value": 200 }/ diff-detection against the previous receipt), or allowing a small boolean predicate in the script return value (e.g.{ data, escalate: boolean, reason?: string })?Happy to help review, test, or contribute to the implementation once the code-health stack (#1402–#1498) clears!
bradhallett commentedon Sep 18, 2026
Thanks @adamczhang, and agreed: mechanical runs paying full turn latency and risking format drift is exactly the pain this targets.
Sandbox: Leaning in-process isolated worker for v1, bounded by the capability allowlist already in the proposal: named HTTP hosts only, a routine-owned directory, no shell, a timeout, and credentials referenced by id rather than embedded. The deterministic path exists to remove spin-up latency and per-run cost, and a VM hop gives some of that back. If a script needs more than the allowlist allows, that should escalate to a real computer lane, not widen the sandbox. Runtime choice (QuickJS vs isolated-vm) stays open along with the script language.
Phasing: Yes, that split matches how the proposal is structured. Phase 1 is mode 2 as written: JSON schema in the routine config, provider-native structured output where the driver supports it, harness-validated JSON with one repair retry otherwise, run cards rendered from the data, and validation failures failing the run loudly instead of degrading to prose. Phase 2 is modes 3 and 4: the generator and sandbox, version hash with a review gate so edits restage before the next run, and threshold and change triggers. Sequencing stays as posted: work starts after the code-health stack, and concretely after feat(evals): offline tier-1 behavior eval harness #1507 lands.
Trigger syntax: Leaning toward the JSON rule expression for v1: field/operator/value plus diff against the previous receipt. Rules as data live in the routine config, validate without executing anything, and review cleanly beside the script hash at the gate. The script-return predicate is more expressive but couples escalation policy to generated code and makes audits harder; it can come later if the grammar runs out of room. Exact grammar is still open.
Deliberately open: script language and runtime, the sandbox boundary itself, rule grammar and editor surface, and per-driver native structured output versus one universal validated-JSON path first.
And yes, please take review and testing on Phase 1. I will ping here when the branch is up after #1507 lands.