Skip to content

fix(story): the nightly failed a gate that had run and PASSED - #2446

Closed
noahgift wants to merge 1 commit into
mainfrom
fix/story-json-stream-separation
Closed

fix(story): the nightly failed a gate that had run and PASSED#2446
noahgift wants to merge 1 commit into
mainfrom
fix/story-json-stream-separation

Conversation

@noahgift

Copy link
Copy Markdown
Contributor

qwen-story-daily is red on main with:

✗ FAIL  B2 format_parity  -  no format_parity gate found in --json
        output (got: '', apr qa exit=0)
16 PASS  /  1 FAIL  /  1 SKIP

The gate ran. It passed. Same command, same fixtures, same host, by hand:

format_parity: skipped=False passed=True

run_cmd captured timeout "$t" "$@" 2>&1 into one RC_OUT, and nine call sites read it — some pipe it to jq, some grep it for a panic. Those want opposite things. apr qa --json writes JSON to stdout and diagnostics to stderr; the CUDA path emits unconditional [trueno#243] Manual graph construction: ... lines via eprintln! (7 sites under crates/aprender-serve/src/cuda/). Merging put those in front of the JSON → jq failed → empty extraction → a reported defect that does not exist.

A false FAIL is as corrosive as a gate that cannot fail: main goes red for a non-reason, and an andon that cries wolf stops being read.

run_cmd moves to scripts/lib_story_run.sh (sourceable and option-neutral — it must never run set, which is how apr_bin.sh once leaked errexit and killed this same nightly six lines in) and sets three views:

RC_OUT stdout only — parse this for JSON
RC_ERR stderr only
RC_ALL both — grep this for panics and banners

The two sites that genuinely need stderr are switched to RC_ALL and say why. Not cosmetic: a Rust panic is written to stderr, so splitting the streams without moving grep -qE 'thread.*panicked' would have silently stopped the story detecting panics — a worse defect than the one being fixed.

Falsifier: scripts/check_story_json_streams.sh drives the real run_cmd over commands with known stream behaviour, including the exact failing shape. Wired into ci.yml beside the other text-only poka-yoke guards, and registered as FALSIFY-QWEN-STORY-014.

Mutation check — merged capture restored, guard kept:

RC_OUT=$(timeout "$t" "$@" 2>&1); RC_EC=$?   # MUTATION

FAIL  JSON on stdout survives diagnostics on stderr
   expected: false true
   actual:
FAIL  RC_OUT is stdout only
FAIL  RC_ERR is stderr only
check_story_json_streams: 3 assertion(s) FAILED

actual: empty is the production symptom exactly — the got: '' the nightly printed. The panic assertion stays green under that mutation, correctly: the pre-fix code did see panics.

Also: FALSIFY-QWEN-STORY-007 ("bashrs lints clean") was already failing on main — 2 × SC1078 from the jq program in pmat_rows being a double-quoted string spanning three lines. Assembled with printf instead; identical program, verified on both shapes pmat returns including the documents-object fallback. That falsifier is green again.

🤖 Generated with Claude Code

qwen-story-daily has been red on main with:

    ✗ FAIL  B2 format_parity  -  no format_parity gate found in --json
            output (got: '', apr qa exit=0)
    16 PASS  /  1 FAIL  /  1 SKIP

The gate ran. It passed. Running the same command by hand against the same
two fixtures on the same host:

    format_parity: skipped=False passed=True

run_cmd captured `timeout "$t" "$@" 2>&1` into a single RC_OUT, and nine call
sites read that one variable - some pipe it to jq, some grep it for a panic.
Those two groups want opposite things. `apr qa --json` writes its JSON to
stdout and its diagnostics to stderr; on the CUDA path the executor emits
unconditional `[trueno#243] Manual graph construction: ...` lines through
eprintln! (7 call sites under crates/aprender-serve/src/cuda/). Merging put
those in front of the JSON, jq could not parse it, the extraction returned
empty, and the harness reported a defect that does not exist.

A false FAIL is exactly as corrosive as a gate that cannot fail. Main goes red
for a non-reason, and an andon that cries wolf stops being read.

run_cmd moves to scripts/lib_story_run.sh (sourceable, and OPTION-NEUTRAL - it
must never run `set`, which is how apr_bin.sh once leaked errexit and killed
this same nightly six lines in) and now sets three views:

    RC_OUT  stdout only - parse this for JSON
    RC_ERR  stderr only
    RC_ALL  both        - grep this for panics and banners

The two call sites that genuinely need stderr are switched to RC_ALL and say
why. That is not cosmetic: a Rust panic is written to stderr, so splitting the
streams without moving `grep -qE 'thread.*panicked'` would have silently
stopped the story detecting panics - a worse defect than the one being fixed.

Falsifier: scripts/check_story_json_streams.sh drives the real run_cmd over
commands with known stream behaviour, including the exact failing shape (JSON
on stdout, `[trueno#243]` on stderr). Wired into ci.yml next to the other
text-only poka-yoke guards, and registered as FALSIFY-QWEN-STORY-014.

Mutation check - restoring the merged capture, keeping the guard:

    RC_OUT=$(timeout "$t" "$@" 2>&1); RC_EC=$?   # MUTATION: pre-fix

    FAIL  JSON on stdout survives diagnostics on stderr
       expected: false true
       actual:
    FAIL  RC_OUT is stdout only
    FAIL  RC_ERR is stderr only
    check_story_json_streams: 3 assertion(s) FAILED

`actual:` empty is the production symptom exactly - the `got: ''` the nightly
printed. Note the panic assertion stays GREEN under that mutation, which is
correct: the pre-fix code did see panics. The guard is precise about which
behaviour it owns.

Also, in the same file: FALSIFY-QWEN-STORY-007 ("bashrs lints clean") was
already failing on main. `bashrs lint scripts/qwen-story.sh` reported 2 errors,
both SC1078, from the jq program in pmat_rows being written as a double-quoted
string spanning three lines. Assembled with printf instead - identical program,
verified on both shapes pmat can return, including the documents-object
fallback that the `type == "array"` guard exists for. That falsifier is green
again:

    $ bashrs lint scripts/qwen-story.sh 2>&1 | grep -qE '(^|[^0-9])0 error'
    exit=0

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@noahgift

Copy link
Copy Markdown
Contributor Author

Superseded by #2449 — this branch is merged verbatim into that batch.

The binding constraint was one ~50-minute workspace-test per PR on one shared box; nine concurrent PRs starved each other into 75-minute step timeouts (every blocked PR this morning classified as CONTENTION, zero merges in 4.5 hours). #2449 lands 24 branches in a single CI run.

Closing rather than leaving open so this PR cannot move #2449's base and force it to re-run. The branch is untouched and this is reopenable if the batch does not land.

@noahgift noahgift closed this Aug 11, 2026
auto-merge was automatically disabled August 11, 2026 13:39

Pull request was closed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant