fix(cli): a NaN tolerance turned every CRUX lint gate into a permanent pass that printed Ok - #2422
Closed
noahgift wants to merge 1 commit into
Closed
fix(cli): a NaN tolerance turned every CRUX lint gate into a permanent pass that printed Ok#2422noahgift wants to merge 1 commit into
noahgift wants to merge 1 commit into
Conversation
…t pass that printed "Ok"
Eight tolerance/threshold flags across five lint commands accepted `nan` and
silently disarmed the gate they configure. On apr 0.63.0 from crates.io, the
same DDP metrics pair that fails at the shipped defaults:
$ apr ddp-metrics-lint --metrics-1gpu-file ddp1.json \
--metrics-ngpu-file ddp4.json --world-size 4
scaling_efficiency : BelowThreshold { efficiency: 0.0025, threshold: 0.85 }
loss_parity : Divergence { rel_diff: 3.95, tolerance: 0.01 }
rc=5
$ apr ddp-metrics-lint ... --scaling-floor nan --loss-tolerance nan
scaling_efficiency : Ok { efficiency: 0.0025 }
loss_parity : Ok { rel_diff: 3.95 }
rc=0
The exit code is the smaller half of the damage. The report prints a positive
`Ok` next to the violating number — a statement that 0.25% DDP scaling
efficiency and a 3860% loss divergence were examined and found acceptable. A CI
log scraper reading that line gets the wrong answer with no indication anything
was skipped, and neither the text nor the `--json` report echoes the threshold
that was used, so the disarm is unauditable after the fact.
Reproduced on all five commands with four different observation shapes (JSON
object, two-file JSON pair, 4-D array, JSONL), and isolated to NaN/negative by
holding the body constant and sweeping the threshold: 0.95 FAIL, 99 FAIL, inf
FAIL, NaN PASS, -1 PASS. `abc` was already rejected by clap, so the parser
validated syntax but never domain.
Root cause is one mechanism repeated eight times, not eight bugs. Every gate is
`if observed > tolerance { fail }` or `if observed < floor { fail }` —
kv_timeline_classifier.rs:256 `up < threshold`, attn_parity_classifier.rs:91,97,
ddp_metrics_classifier.rs:114,146, plus the equivalents in attn_viz_classifier
and explain_token_classifier. IEEE-754 makes every comparison against NaN false,
so the failing branch is unreachable. Nothing anywhere validated the incoming
threshold — notable because the same classifiers already refuse to judge a
non-finite *observation* (`AttnParityNumericsOutcome::NonFiniteMaxAbsDiff`).
This adds the symmetric guard on the threshold side.
crates/apr-cli/src/commands/threshold_arg.rs is the single validator, used at
both layers: as a clap `value_parser` on all eight flags so a bad value is
rejected at parse time before any gate runs, and as a `guard()` at the top of
each lint `run()` so a caller that bypasses clap still fails closed rather than
printing `Ok`. Three domains — TOLERANCE (finite, >= 0), FRACTION ([0,1]),
COSINE ([-1,1]).
$ apr kv-timeline-lint --timeline-file kv.json --preempt-threshold=nan
error: invalid value 'nan' for '--preempt-threshold <FRACTION>': NaN is not
a threshold: every comparison against NaN is false, so the gate could never
fail. Expected a finite fraction in [0.0, 1.0]
rc=2
Two deliberate behaviour changes beyond NaN: `inf` and out-of-domain values are
now rejected too. Both previously "worked" in the sense of failing loudly
(`--preempt-threshold 99` reported `threshold: 99.0`), but neither can express a
bound, and an infinite tolerance disarms a max-style gate exactly as NaN does.
Explicit relaxation still works through legitimate values —
`--scaling-floor 0.0 --loss-tolerance 1e9` still exits 0.
Mutation check: with `reject_reason()` stubbed to `return None` and all tests
kept, the falsifiers go RED —
FALSIFY-CLI-THRESHOLD-NAN-001: `apr kv-timeline-lint --timeline-file
/tmp/.tmpZng1k4/kv.json --preempt-threshold nan` exited 0 — a NaN threshold
disarmed the preemption_trigger gate on a body that fails at the defaults.
stdout:
preemption_trigger : Ok
expected --scaling-floor named; got: ddp-metrics-lint loss-parity gate
rejected: Divergence { l1: 2.0, ln: 9.9, rel_diff: 3.95, tolerance: 0.01 }
clap accepted a gate-disarming threshold: ["apr", "kv-timeline-lint",
"--timeline-file", "kv.json", "--preempt-threshold", "nan"]
test result: FAILED. 0 passed; 5 failed (lib run() guards)
test result: FAILED. 2 passed; 7 failed (threshold_arg + clap parse)
test result: FAILED. 0 passed; 1 failed (cli_commands e2e)
Restored: 6641 passed, 0 failed (apr-cli --lib); cli_commands 10/10; the five
CRUX falsification suites (F-06, L-02, F-17, F-19, D-11) 52/52.
The e2e falsifier lives in tests/cli_commands.rs on purpose — it is one of the
few integration targets ci.yml actually runs, and it asserts the control case
(same body must still fail at the defaults) before asserting the disarm case,
so it cannot pass vacuously.
Refs #2391
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
noahgift
enabled auto-merge
August 10, 2026 08:18
noahgift
marked this pull request as draft
August 10, 2026 08:36
auto-merge was automatically disabled
August 10, 2026 08:36
Pull request was converted to draft
noahgift
marked this pull request as ready for review
August 10, 2026 12:53
noahgift
enabled auto-merge
August 10, 2026 12:53
github-merge-queue
Bot
removed this pull request from the merge queue due to failed status checks
Aug 10, 2026
github-merge-queue
Bot
removed this pull request from the merge queue due to no response for status checks
Aug 10, 2026
github-merge-queue
Bot
removed this pull request from the merge queue due to no response for status checks
Aug 11, 2026
github-merge-queue
Bot
removed this pull request from the merge queue due to no response for status checks
Aug 11, 2026
github-merge-queue
Bot
removed this pull request from the merge queue due to no response for status checks
Aug 11, 2026
Contributor
Author
|
Superseded by #2449 — this branch is merged verbatim into that batch. The binding constraint was one ~50-minute Closing rather than leaving open so this PR cannot move #2449's base and force it to re-run. The branch is untouched and this is reopenable if the batch does not land. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Eight tolerance/threshold flags across five lint commands accept
nanand silently disarm the gate they configure. On apr 0.63.0 installed from crates.io, the same DDP metrics pair that fails at the shipped defaults:The exit code is the smaller half of the damage. The report prints a positive
Oknext to the violating number — a statement that 0.25% DDP scaling efficiency and a 3860% loss divergence were examined and found acceptable. A CI log scraper reading that line gets the wrong answer with no indication anything was skipped, and neither the text nor the--jsonreport echoes the threshold used, so the disarm is unauditable after the fact.Reproduced on all five commands with four different observation shapes (JSON object, two-file JSON pair, 4-D array, JSONL), and isolated to NaN/negative by holding the body constant and sweeping only the threshold:
0.95FAIL,99FAIL,infFAIL,NaNPASS,-1PASS.abcwas already rejected by clap, so the parser validated syntax but never domain.Root cause
One mechanism repeated eight times, not eight bugs. Every gate is
if observed > tolerance { fail }orif observed < floor { fail }:crates/apr-cli/src/commands/kv_timeline_classifier.rs:256—if up < thresholdcrates/apr-cli/src/commands/attn_parity_classifier.rs:91,97crates/apr-cli/src/commands/ddp_metrics_classifier.rs:114,146attn_viz_classifier.rsandexplain_token_classifier.rsIEEE-754 makes every comparison against NaN false, so the failing branch is unreachable. Nothing validated the incoming threshold — notable because those same classifiers already refuse to judge a non-finite observation (
AttnParityNumericsOutcome::NonFiniteMaxAbsDiff). This PR adds the symmetric guard on the threshold side.Fix
crates/apr-cli/src/commands/threshold_arg.rsis the single validator, used at both layers:value_parseron all eight flags, so a bad value is rejected at parse time (exit 2) before any gate runs;guard()at the top of each lintrun(), so a caller that bypasses clap also fails closed instead of printingOk.Three domains:
TOLERANCE(finite,>= 0),FRACTION([0,1]),COSINE([-1,1]).Two deliberate behaviour changes beyond NaN:
infand out-of-domain values are now rejected too. Both previously "worked" in the sense of failing loudly (--preempt-threshold 99reportedthreshold: 99.0), but neither can express a bound, and an infinite tolerance disarms a max-style gate exactly as NaN does. Explicit relaxation still works through legitimate values —--scaling-floor 0.0 --loss-tolerance 1e9still exits 0, and every threshold value used by the shipped CRUX falsification suites still parses.Mutation check
With
reject_reason()stubbed toreturn Noneand all tests kept, the falsifiers go RED:Restored:
cargo test -p apr-cli --lib6641 passed / 0 failed;cli_commands10/10; the five CRUX falsification suites (F-06, L-02, F-17, F-19, D-11) 52/52.cargo fmt --all -- --checkandcargo clippy -p apr-cli --lib -- -D warningsclean.pv validate contracts/apr-cli-commands-v1.yamlvalid.The e2e falsifier lives in
tests/cli_commands.rson purpose — it is one of the few integration targetsci.ymlactually runs — and it asserts the control case (the same body must still fail at the defaults) before asserting the disarm case, so it cannot pass vacuously.Verified end-to-end
Against
target/release/aprbuilt from this branch, every original repro from #2391 now exits 2 with a domain error instead of 0 withOk:--preempt-threshold{nan, NaN, -1, inf, 99},--tol-abs,--tol-cos,--tolerance,--epsilon,--scaling-floor,--loss-tolerance.Refs #2391 (partial) — this PR fixes findings 1, 2, 3 and 5 (the NaN/negative threshold family). Remaining in that issue, untouched here: finding 4 (imatrix-lint duplicate
falsify_id), finding 6 (typical-p-lint silently skips a mistyped section), finding 7 (stamp --licenseaccepts non-SPDX), finding 8 (beat-runleaksSome(..)and misattributes a non-finite--measured).Audit epic: #2373
🤖 Generated with Claude Code