fix(story): the nightly failed a gate that had run and PASSED - #2446
Closed
noahgift wants to merge 1 commit into
Closed
fix(story): the nightly failed a gate that had run and PASSED#2446noahgift wants to merge 1 commit into
noahgift wants to merge 1 commit into
Conversation
qwen-story-daily has been red on main with:
✗ FAIL B2 format_parity - no format_parity gate found in --json
output (got: '', apr qa exit=0)
16 PASS / 1 FAIL / 1 SKIP
The gate ran. It passed. Running the same command by hand against the same
two fixtures on the same host:
format_parity: skipped=False passed=True
run_cmd captured `timeout "$t" "$@" 2>&1` into a single RC_OUT, and nine call
sites read that one variable - some pipe it to jq, some grep it for a panic.
Those two groups want opposite things. `apr qa --json` writes its JSON to
stdout and its diagnostics to stderr; on the CUDA path the executor emits
unconditional `[trueno#243] Manual graph construction: ...` lines through
eprintln! (7 call sites under crates/aprender-serve/src/cuda/). Merging put
those in front of the JSON, jq could not parse it, the extraction returned
empty, and the harness reported a defect that does not exist.
A false FAIL is exactly as corrosive as a gate that cannot fail. Main goes red
for a non-reason, and an andon that cries wolf stops being read.
run_cmd moves to scripts/lib_story_run.sh (sourceable, and OPTION-NEUTRAL - it
must never run `set`, which is how apr_bin.sh once leaked errexit and killed
this same nightly six lines in) and now sets three views:
RC_OUT stdout only - parse this for JSON
RC_ERR stderr only
RC_ALL both - grep this for panics and banners
The two call sites that genuinely need stderr are switched to RC_ALL and say
why. That is not cosmetic: a Rust panic is written to stderr, so splitting the
streams without moving `grep -qE 'thread.*panicked'` would have silently
stopped the story detecting panics - a worse defect than the one being fixed.
Falsifier: scripts/check_story_json_streams.sh drives the real run_cmd over
commands with known stream behaviour, including the exact failing shape (JSON
on stdout, `[trueno#243]` on stderr). Wired into ci.yml next to the other
text-only poka-yoke guards, and registered as FALSIFY-QWEN-STORY-014.
Mutation check - restoring the merged capture, keeping the guard:
RC_OUT=$(timeout "$t" "$@" 2>&1); RC_EC=$? # MUTATION: pre-fix
FAIL JSON on stdout survives diagnostics on stderr
expected: false true
actual:
FAIL RC_OUT is stdout only
FAIL RC_ERR is stderr only
check_story_json_streams: 3 assertion(s) FAILED
`actual:` empty is the production symptom exactly - the `got: ''` the nightly
printed. Note the panic assertion stays GREEN under that mutation, which is
correct: the pre-fix code did see panics. The guard is precise about which
behaviour it owns.
Also, in the same file: FALSIFY-QWEN-STORY-007 ("bashrs lints clean") was
already failing on main. `bashrs lint scripts/qwen-story.sh` reported 2 errors,
both SC1078, from the jq program in pmat_rows being written as a double-quoted
string spanning three lines. Assembled with printf instead - identical program,
verified on both shapes pmat can return, including the documents-object
fallback that the `type == "array"` guard exists for. That falsifier is green
again:
$ bashrs lint scripts/qwen-story.sh 2>&1 | grep -qE '(^|[^0-9])0 error'
exit=0
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
noahgift
enabled auto-merge
August 11, 2026 07:23
Contributor
Author
|
Superseded by #2449 — this branch is merged verbatim into that batch. The binding constraint was one ~50-minute Closing rather than leaving open so this PR cannot move #2449's base and force it to re-run. The branch is untouched and this is reopenable if the batch does not land. |
auto-merge was automatically disabled
August 11, 2026 13:39
Pull request was closed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
qwen-story-dailyis red onmainwith:The gate ran. It passed. Same command, same fixtures, same host, by hand:
run_cmdcapturedtimeout "$t" "$@" 2>&1into oneRC_OUT, and nine call sites read it — some pipe it tojq, some grep it for a panic. Those want opposite things.apr qa --jsonwrites JSON to stdout and diagnostics to stderr; the CUDA path emits unconditional[trueno#243] Manual graph construction: ...lines viaeprintln!(7 sites undercrates/aprender-serve/src/cuda/). Merging put those in front of the JSON →jqfailed → empty extraction → a reported defect that does not exist.A false FAIL is as corrosive as a gate that cannot fail: main goes red for a non-reason, and an andon that cries wolf stops being read.
run_cmdmoves toscripts/lib_story_run.sh(sourceable and option-neutral — it must never runset, which is howapr_bin.shonce leaked errexit and killed this same nightly six lines in) and sets three views:RC_OUTRC_ERRRC_ALLThe two sites that genuinely need stderr are switched to
RC_ALLand say why. Not cosmetic: a Rust panic is written to stderr, so splitting the streams without movinggrep -qE 'thread.*panicked'would have silently stopped the story detecting panics — a worse defect than the one being fixed.Falsifier:
scripts/check_story_json_streams.shdrives the realrun_cmdover commands with known stream behaviour, including the exact failing shape. Wired intoci.ymlbeside the other text-only poka-yoke guards, and registered asFALSIFY-QWEN-STORY-014.Mutation check — merged capture restored, guard kept:
actual:empty is the production symptom exactly — thegot: ''the nightly printed. The panic assertion stays green under that mutation, correctly: the pre-fix code did see panics.Also:
FALSIFY-QWEN-STORY-007("bashrs lints clean") was already failing on main — 2 ×SC1078from thejqprogram inpmat_rowsbeing a double-quoted string spanning three lines. Assembled withprintfinstead; identical program, verified on both shapes pmat returns including the documents-object fallback. That falsifier is green again.🤖 Generated with Claude Code