Skip to content

fix(cli): a NaN tolerance turned every CRUX lint gate into a permanent pass that printed Ok - #2422

Closed
noahgift wants to merge 1 commit into
mainfrom
fix/lint-threshold-nan-disarm
Closed

fix(cli): a NaN tolerance turned every CRUX lint gate into a permanent pass that printed Ok#2422
noahgift wants to merge 1 commit into
mainfrom
fix/lint-threshold-nan-disarm

Conversation

@noahgift

Copy link
Copy Markdown
Contributor

Eight tolerance/threshold flags across five lint commands accept nan and silently disarm the gate they configure. On apr 0.63.0 installed from crates.io, the same DDP metrics pair that fails at the shipped defaults:

$ apr ddp-metrics-lint --metrics-1gpu-file ddp1.json --metrics-ngpu-file ddp4.json --world-size 4
  scaling_efficiency  : BelowThreshold { world_size: 4, t1: 1000.0, tn: 10.0, efficiency: 0.0025, threshold: 0.85 }
  loss_parity         : Divergence { l1: 2.0, ln: 9.9, rel_diff: 3.95, tolerance: 0.01 }
rc=5

$ apr ddp-metrics-lint ... --scaling-floor nan --loss-tolerance nan
  scaling_efficiency  : Ok { efficiency: 0.0025 }
  loss_parity         : Ok { rel_diff: 3.95 }
rc=0

The exit code is the smaller half of the damage. The report prints a positive Ok next to the violating number — a statement that 0.25% DDP scaling efficiency and a 3860% loss divergence were examined and found acceptable. A CI log scraper reading that line gets the wrong answer with no indication anything was skipped, and neither the text nor the --json report echoes the threshold used, so the disarm is unauditable after the fact.

Reproduced on all five commands with four different observation shapes (JSON object, two-file JSON pair, 4-D array, JSONL), and isolated to NaN/negative by holding the body constant and sweeping only the threshold: 0.95 FAIL, 99 FAIL, inf FAIL, NaN PASS, -1 PASS. abc was already rejected by clap, so the parser validated syntax but never domain.

Root cause

One mechanism repeated eight times, not eight bugs. Every gate is if observed > tolerance { fail } or if observed < floor { fail }:

  • crates/apr-cli/src/commands/kv_timeline_classifier.rs:256if up < threshold
  • crates/apr-cli/src/commands/attn_parity_classifier.rs:91,97
  • crates/apr-cli/src/commands/ddp_metrics_classifier.rs:114,146
  • the equivalents in attn_viz_classifier.rs and explain_token_classifier.rs

IEEE-754 makes every comparison against NaN false, so the failing branch is unreachable. Nothing validated the incoming threshold — notable because those same classifiers already refuse to judge a non-finite observation (AttnParityNumericsOutcome::NonFiniteMaxAbsDiff). This PR adds the symmetric guard on the threshold side.

Fix

crates/apr-cli/src/commands/threshold_arg.rs is the single validator, used at both layers:

  1. as a clap value_parser on all eight flags, so a bad value is rejected at parse time (exit 2) before any gate runs;
  2. as a guard() at the top of each lint run(), so a caller that bypasses clap also fails closed instead of printing Ok.

Three domains: TOLERANCE (finite, >= 0), FRACTION ([0,1]), COSINE ([-1,1]).

$ apr kv-timeline-lint --timeline-file kv.json --preempt-threshold=nan
error: invalid value 'nan' for '--preempt-threshold <FRACTION>': NaN is not a threshold:
every comparison against NaN is false, so the gate could never fail.
Expected a finite fraction in [0.0, 1.0]
rc=2

Two deliberate behaviour changes beyond NaN: inf and out-of-domain values are now rejected too. Both previously "worked" in the sense of failing loudly (--preempt-threshold 99 reported threshold: 99.0), but neither can express a bound, and an infinite tolerance disarms a max-style gate exactly as NaN does. Explicit relaxation still works through legitimate values — --scaling-floor 0.0 --loss-tolerance 1e9 still exits 0, and every threshold value used by the shipped CRUX falsification suites still parses.

Mutation check

With reject_reason() stubbed to return None and all tests kept, the falsifiers go RED:

FALSIFY-CLI-THRESHOLD-NAN-001: `apr kv-timeline-lint --timeline-file /tmp/.tmpZng1k4/kv.json
--preempt-threshold nan` exited 0 — a NaN threshold disarmed the preemption_trigger gate on
a body that fails at the defaults.
stdout:
kv-timeline-lint report for /tmp/.tmpZng1k4/kv.json
  schema             : Ok
  block_conservation : Ok
  used_pct_arithmetic: Ok
  peak_consistency   : Ok
  preemption_trigger : Ok

expected --scaling-floor named; got: ddp-metrics-lint loss-parity gate rejected:
Divergence { l1: 2.0, ln: 9.9, rel_diff: 3.95, tolerance: 0.01 }

clap accepted a gate-disarming threshold:
["apr", "kv-timeline-lint", "--timeline-file", "kv.json", "--preempt-threshold", "nan"]

test result: FAILED. 0 passed; 5 failed   (lib run() guards, one per command)
test result: FAILED. 2 passed; 7 failed   (threshold_arg unit + clap parse, one case per flag)
test result: FAILED. 0 passed; 1 failed   (cli_commands e2e against the real binary)

Restored: cargo test -p apr-cli --lib 6641 passed / 0 failed; cli_commands 10/10; the five CRUX falsification suites (F-06, L-02, F-17, F-19, D-11) 52/52. cargo fmt --all -- --check and cargo clippy -p apr-cli --lib -- -D warnings clean. pv validate contracts/apr-cli-commands-v1.yaml valid.

The e2e falsifier lives in tests/cli_commands.rs on purpose — it is one of the few integration targets ci.yml actually runs — and it asserts the control case (the same body must still fail at the defaults) before asserting the disarm case, so it cannot pass vacuously.

Verified end-to-end

Against target/release/apr built from this branch, every original repro from #2391 now exits 2 with a domain error instead of 0 with Ok: --preempt-threshold {nan, NaN, -1, inf, 99}, --tol-abs, --tol-cos, --tolerance, --epsilon, --scaling-floor, --loss-tolerance.

Refs #2391 (partial) — this PR fixes findings 1, 2, 3 and 5 (the NaN/negative threshold family). Remaining in that issue, untouched here: finding 4 (imatrix-lint duplicate falsify_id), finding 6 (typical-p-lint silently skips a mistyped section), finding 7 (stamp --license accepts non-SPDX), finding 8 (beat-run leaks Some(..) and misattributes a non-finite --measured).

Audit epic: #2373

🤖 Generated with Claude Code

…t pass that printed "Ok"

Eight tolerance/threshold flags across five lint commands accepted `nan` and
silently disarmed the gate they configure. On apr 0.63.0 from crates.io, the
same DDP metrics pair that fails at the shipped defaults:

    $ apr ddp-metrics-lint --metrics-1gpu-file ddp1.json \
          --metrics-ngpu-file ddp4.json --world-size 4
      scaling_efficiency  : BelowThreshold { efficiency: 0.0025, threshold: 0.85 }
      loss_parity         : Divergence { rel_diff: 3.95, tolerance: 0.01 }
    rc=5

    $ apr ddp-metrics-lint ... --scaling-floor nan --loss-tolerance nan
      scaling_efficiency  : Ok { efficiency: 0.0025 }
      loss_parity         : Ok { rel_diff: 3.95 }
    rc=0

The exit code is the smaller half of the damage. The report prints a positive
`Ok` next to the violating number — a statement that 0.25% DDP scaling
efficiency and a 3860% loss divergence were examined and found acceptable. A CI
log scraper reading that line gets the wrong answer with no indication anything
was skipped, and neither the text nor the `--json` report echoes the threshold
that was used, so the disarm is unauditable after the fact.

Reproduced on all five commands with four different observation shapes (JSON
object, two-file JSON pair, 4-D array, JSONL), and isolated to NaN/negative by
holding the body constant and sweeping the threshold: 0.95 FAIL, 99 FAIL, inf
FAIL, NaN PASS, -1 PASS. `abc` was already rejected by clap, so the parser
validated syntax but never domain.

Root cause is one mechanism repeated eight times, not eight bugs. Every gate is
`if observed > tolerance { fail }` or `if observed < floor { fail }` —
kv_timeline_classifier.rs:256 `up < threshold`, attn_parity_classifier.rs:91,97,
ddp_metrics_classifier.rs:114,146, plus the equivalents in attn_viz_classifier
and explain_token_classifier. IEEE-754 makes every comparison against NaN false,
so the failing branch is unreachable. Nothing anywhere validated the incoming
threshold — notable because the same classifiers already refuse to judge a
non-finite *observation* (`AttnParityNumericsOutcome::NonFiniteMaxAbsDiff`).
This adds the symmetric guard on the threshold side.

crates/apr-cli/src/commands/threshold_arg.rs is the single validator, used at
both layers: as a clap `value_parser` on all eight flags so a bad value is
rejected at parse time before any gate runs, and as a `guard()` at the top of
each lint `run()` so a caller that bypasses clap still fails closed rather than
printing `Ok`. Three domains — TOLERANCE (finite, >= 0), FRACTION ([0,1]),
COSINE ([-1,1]).

    $ apr kv-timeline-lint --timeline-file kv.json --preempt-threshold=nan
    error: invalid value 'nan' for '--preempt-threshold <FRACTION>': NaN is not
    a threshold: every comparison against NaN is false, so the gate could never
    fail. Expected a finite fraction in [0.0, 1.0]
    rc=2

Two deliberate behaviour changes beyond NaN: `inf` and out-of-domain values are
now rejected too. Both previously "worked" in the sense of failing loudly
(`--preempt-threshold 99` reported `threshold: 99.0`), but neither can express a
bound, and an infinite tolerance disarms a max-style gate exactly as NaN does.
Explicit relaxation still works through legitimate values —
`--scaling-floor 0.0 --loss-tolerance 1e9` still exits 0.

Mutation check: with `reject_reason()` stubbed to `return None` and all tests
kept, the falsifiers go RED —

    FALSIFY-CLI-THRESHOLD-NAN-001: `apr kv-timeline-lint --timeline-file
    /tmp/.tmpZng1k4/kv.json --preempt-threshold nan` exited 0 — a NaN threshold
    disarmed the preemption_trigger gate on a body that fails at the defaults.
    stdout:
      preemption_trigger : Ok

    expected --scaling-floor named; got: ddp-metrics-lint loss-parity gate
    rejected: Divergence { l1: 2.0, ln: 9.9, rel_diff: 3.95, tolerance: 0.01 }

    clap accepted a gate-disarming threshold: ["apr", "kv-timeline-lint",
    "--timeline-file", "kv.json", "--preempt-threshold", "nan"]

    test result: FAILED. 0 passed; 5 failed   (lib run() guards)
    test result: FAILED. 2 passed; 7 failed   (threshold_arg + clap parse)
    test result: FAILED. 0 passed; 1 failed   (cli_commands e2e)

Restored: 6641 passed, 0 failed (apr-cli --lib); cli_commands 10/10; the five
CRUX falsification suites (F-06, L-02, F-17, F-19, D-11) 52/52.

The e2e falsifier lives in tests/cli_commands.rs on purpose — it is one of the
few integration targets ci.yml actually runs, and it asserts the control case
(same body must still fail at the defaults) before asserting the disarm case,
so it cannot pass vacuously.

Refs #2391

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@noahgift
noahgift enabled auto-merge August 10, 2026 08:18
@noahgift
noahgift marked this pull request as draft August 10, 2026 08:36
auto-merge was automatically disabled August 10, 2026 08:36

Pull request was converted to draft

@noahgift
noahgift marked this pull request as ready for review August 10, 2026 12:53
@noahgift
noahgift enabled auto-merge August 10, 2026 12:53
@noahgift
noahgift added this pull request to the merge queue Aug 10, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Aug 10, 2026
@noahgift
noahgift added this pull request to the merge queue Aug 10, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to no response for status checks Aug 10, 2026
@noahgift
noahgift added this pull request to the merge queue Aug 10, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to no response for status checks Aug 11, 2026
@noahgift
noahgift added this pull request to the merge queue Aug 11, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to no response for status checks Aug 11, 2026
@noahgift
noahgift added this pull request to the merge queue Aug 11, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to no response for status checks Aug 11, 2026
@noahgift

Copy link
Copy Markdown
Contributor Author

Superseded by #2449 — this branch is merged verbatim into that batch.

The binding constraint was one ~50-minute workspace-test per PR on one shared box; nine concurrent PRs starved each other into 75-minute step timeouts (every blocked PR this morning classified as CONTENTION, zero merges in 4.5 hours). #2449 lands 24 branches in a single CI run.

Closing rather than leaving open so this PR cannot move #2449's base and force it to re-run. The branch is untouched and this is reopenable if the batch does not land.

@noahgift noahgift closed this Aug 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant