Repository navigation
The impl-loop API-error abort cannot see plain-output errors, and when it fires it records no verdict and files no receipt #916
Description
Activity
Census: it is eight sites, not one — and the first three are the ones that would have made Day 195 RED
Every API-error detector in
scripts/evolve.shgreps the JSON shape"type":"error", and every agent in the loop runs in plain output mode. Measured:line agent what the branch does when it fires 1384 assessment exit 1→ workflow-level retry1754 planner exit 1→ workflow-level retry1805 planner corrective retry exit 1→ workflow-level retry2100 impl loop reset, TASK_FAILURES++,break— no verdict, no receipt (defect 2 above)2411 build-fix loop sets REVERT_REASON,break2782 eval-fix loop logs, FIX_NOOP_CAUSE2935 evaluator "skipping eval (build+test passed)" — fail-open 3601 issue-response logs On Day 195 none of the eight fired. The consequence runs top-down: had 1384 or 1754 fired, the session would have exited for retry and the run would have been red after its three attempts — which is the honest outcome when the provider is down — instead of "no assessment" → "0 tasks" → fallback → empty diff → evaluator "API error, skipping" → promoted. The empty-diff gate (
119f172b) now catches the end of that chain; this is the start of it, and it is the same rule stated eight times, all deaf to the same input.The trap in the obvious fix
The runtime line, as tee'd into
$TASK_LOG, is (after the optional colour escape):API error with no fallback configured. Exiting.two leading spaces, then the message, at line start. A bare
grep -Fon the message would also match a task in which the agent merely readssrc/agent_builder.rs— itsread_fileoutput lands in the same log, carryingeprintln!("{RED} API error with no fallback configured. Exiting.{RESET}",);and would abort a healthy task. That is the prose-contamination defect
scripts/measure_abstentions.pywas built around (an unanchored marker matching my own quoted text). Anchor at line start, allowing only a colour escape before the two spaces:^(\x1b\[[0-9;]*m)* API error with no fallback configured\. Exiting\.and keep the JSON match beside it. The source-echo line is the near-miss guard and must be in the fixture.
Fix shape
One predicate,
agent_hit_api_error LOGFILE, matching either shape, called at all eight sites — one statement of the rule instead of eight copies that are deaf together. Extracted intotests/harness_logic.shand driven against fixtures: the JSON line; the plain sentinel with and without the escape prefix; the near-miss (a log containing only theeprintln!source echo, and one containing the words "Rate limited" in prose) must not fire; an empty log must not fire. Defect 2 (the impl-loop branch records nothing) still needs the bookkeeping named above before itsbreak.Stated limit, in the predicate's doc: it is a cross-boundary string read off
src/output, exactly like the[Agent stopped:grep two lines below site 2100. Ifsrc/agent_builder.rs:1446is ever reworded, every one of the eight goes deaf again and the only symptom is the Day 195 shape. That is the argument for a source-level guard intests/that greps the literal out ofsrc/agent_builder.rsand the harness and asserts they agree.Sequencing: land after the empty-diff gate has been observed silent on one real session (run 34749358941 is that observation), so a regression in either is attributable.
The impl-loop API-error abort is deaf to plain-output errors, and when it does fire it records nothing — two defects, creator lane
Backlog, creator lane (
scripts/evolve.shis protected). Deliberately unlabelled.What it is
scripts/evolve.shaborts the implementation loop when the impl agent hits an API error:Defect 1 — it cannot see the error it was built for
The detector matches a JSON
"type":"error"line. The impl agent runs in plain output mode, where the same failure prints:(
src/agent_builder.rs:1446, on stderr;run_agent_with_fallbackcaptures2>&1 | tee, so it is inTASK_LOG.) On Day 195 this fired zero times across five sessions whose every call was refused — measured in run 34590906044. The[Agent stopped:grep two lines below has the same shape and works because that string is what yoagent actually prints.Defect 2 — when it fires, it leaves no record
The branch resets, counts a failure and
breaks. It never callsgasp_task_result … rejected, sets noREVERT_REASON, and files no revert receipt. The graph keeps atask.createdwith no verdict, and the planner learns nothing next session. Compare the ordinary revert path, which records the verdict with a reason, files a receipt with a class the planner has a rule for, and carries the error tail.Why it was not extended alongside the empty-diff gate (119f172)
Extending the detector alone would have made Day-195-shaped sessions take this path instead of the gate's — i.e. the less informative one. The gate already produces the better record for that shape:
(no changes landed), the reason, and the impl log tail in the receipt. What the abort adds is only "do not start task 2 when the provider is down" — worth having, but not at the cost of the record.The fix, both halves together
grep -qF 'API error with no fallback configured. Exiting.'), with the stated limit that it is a cross-boundary string read offsrc/output, exactly like[Agent stopped:.REVERT_CLASS=" (provider refused)", a reason carrying the log tail,gasp_task_result … rejected, and the receipt — thenbreakso remaining tasks are not attempted against a dead provider.Extract the predicate as a function (
agent_hit_api_error LOGFILE) sotests/harness_logic.shcan drive it against fixtures: the JSON line, the plain sentinel wrapped in ANSI codes, and the near-miss — a log that merely contains the word "Rate limited" in prose must not abort.