fix(lint): four *-lint gates passed on evidence they never read, and one printed a number the input did not contain - #2438
Conversation
…one printed a number the input did not contain
`apr audio-inspect-lint` accepted a body declaring `sample_rate: 4294983296`,
printed `sample_rate : Ok { rate: 16000 }` and exited 0. 4294983296 is
2^32 + 16000, and `raw as u32` wraps, so the value aliased onto a canonical
rate and passed the gate while the report showed a number that was never in
the file. `--expected-sample-rate 16000` did not catch it either: after the
wrap the two were equal. `channels: 4294967297` aliased to 1 the same way.
`sample_rate: 4294967296` did fail, but reported `NonCanonicalRate { got: 0 }`
— again a value the observation did not contain.
before (apr 0.63.0, base 8cc3aaf) after (this branch)
sample_rate : Ok { rate: 16000 } sample_rate : SampleRateOutOfRange { got: 4294983296 }
rc=0 rc=5
Three more gates in the family reported PASS for sections that were absent,
empty, or of the wrong JSON type:
$ apr imatrix-lint --observation-file <(echo '{"leakage":{"calib_hashes":[1,2],"eval_hashes":[1,2]}}')
[PASS] leakage (FALSIFY-CRUX-B-07-001): disjoint (|calib|=0, |eval|=0) rc=0
Those two sets are identical. `filter_map(Value::as_str)` silently dropped
every integer-typed hash, leaving two empty sets, and the gate then printed a
cardinality claim about data it had not read. `{"leakage":"nonsense"}` produced
the same green line. `apr embeddings-lint` did it with `{"shape":{}}` —
input_len, hidden_size and data all defaulted to 0/0/[], so 0 == 0 and the
verdict was `Ok { n_rows: 0 }`. `apr hang-trace-lint --world-size 0`
short-circuited to `Ok { ranks_seen: 0 }` on an EMPTY trace dir, so a CI job
whose world size came from an env var that resolved to empty got a green gate
on no evidence at all. All three now fail: unreadable evidence is unknown, not
disjoint, not Ok. Genuinely disjoint hash sets, an explicitly empty
`{"input_len":0,"hidden_size":4,"data":[]}` response, and a populated trace dir
all still pass — the fix rejects absent evidence, not legitimately empty
evidence.
Separately, `apr quant-preservation-lint` across two models with different
tokenizers emitted 13,282,156 bytes over 19 lines, one of them 4,814,215
characters long (14,896,285 bytes under `--json`): it stored the untruncated
`Debug` of every diverged GGUF value, including whole 248k-entry
`tokenizer.ggml.merges` and `tokenizer.ggml.tokens` arrays. The actionable
signals in that same run — `general.architecture :: String("qwen35") →
String("qwen2")`, `tokenizer.ggml.eos_token_id :: Uint32(248046) →
Uint32(151645)` — were buried between three multi-megabyte dumps that wedge a
terminal and blow a CI log budget. Values over 200 chars are now summarised
with element count, a content digest and a head sample, plus the first
differing index; scalars still render verbatim, unchanged. The identical run:
before 13,282,156 bytes / 19 lines / longest line 4,814,215
after 2,446 bytes / 22 lines / longest line 338
--json 14,896,285 bytes -> 3,761 bytes
tokenizer.ggml.tokens :: ArrayString(len=248320, sha256=3751959a6d53cc72, head=ArrayString(["!", "\"", "#", …)
→ ArrayString(len=151936, sha256=77485efcba6e358f, head=ArrayString(["!", "\"", "#", …)
first differing element: index 280 (len 248320 vs 151936)
Root causes:
crates/apr-cli/src/commands/audio_inspect_classifier.rs:106,139 — `as u32`
after only a `raw <= 0` check
crates/apr-cli/src/commands/imatrix_lint.rs:166 — `and_then(as_array)
.map(filter_map(as_str)).unwrap_or_default()` on a possibly-absent,
possibly-non-string section
crates/apr-cli/src/commands/embeddings_lint.rs:117 — three `unwrap_or(0)` /
`unwrap_or_default()` defaults standing in for missing fields
crates/apr-cli/src/commands/hang_trace_classifier.rs:69 — `world_size == 0`
returned Ok, with a comment calling it "Degenerate"
crates/apr-cli/src/commands/quant_preservation.rs:100 —
`format!("{ref_val:?}")` stored whole, rendered whole at :201-208
`apr typical-p-lint --help` printed its only required flag with an empty
description, so its non-obvious observation schema had no route to a user.
Rather than fix the one string, the falsifier walks every `*-lint` subcommand
through clap's CommandFactory and asserts each flag carries help text. It went
red on a second instance the audit had not found — `embeddings-lint
--observation-file`, likewise blank in the shipped 0.63.0 help. Both are
documented now, and a new `*-lint` that forgets turns the test red.
Mutation check. Reverting each fix while keeping its tests, one at a time:
audio 8 failed Ok { rate: 16000 } vs SampleRateOutOfRange { got: 4294983296 }
Ok { channels: 1 } vs ChannelsOutOfRange { got: 4294967297 }
NonCanonicalRate { got: 0 } vs SampleRateOutOfRange { got: 4294967296 }
imatrix 4 failed [PASS] leakage: disjoint (|calib|=0, |eval|=0)
embeddings 4 failed [PASS] shape: Ok { n_rows: 0 }
hang-trace 2 failed Ok { ranks_seen: 0 } vs WorldSizeZero
quant 2 failed divergence report must stay bounded; got 5855530 bytes
help 1 failed lint commands with an undocumented flag: ["typical-p-lint --observation_file"]
Restored: 6686 passed, 0 failed.
The end-to-end before/after above was measured on two binaries whose embedded
git SHA proves which tree each came from — `apr --version` reported
`apr 0.63.0 (8cc3aaf)` for the base and this branch's own commit for the
fix — both built into a private target dir. That matters: the workspace target
dir is shared with three other worktrees, and the first "after" binary taken
from it turned out to be another worktree's build, byte-identical to the
"before" one and showing zero change.
Not fixed here, each deserving its own PR: the "CLI flag accepted" gates that
stamp PASS for argv the real clap parser rejects with exit 2 (finding 2), the
8 linters whose documented producer command is absent from the binary
(finding 3), the two incompatible `--json` envelopes (finding 6), and the
exit-code / "Invalid APR format" split (findings 8 and 9), which issue #2404
already owns.
Refs #2377 (partial) — remaining: 2, 3, 6, 8, 9
Audit epic: #2373
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
Temporarily closing to stop CI contention — the branch is untouched and this will be reopened, nothing is lost. The shared Reopening in batches as the merge queue drains. The work is complete and reviewed; only the CI scheduling is being paced. |
|
Superseded by #2449 — this branch is merged verbatim into that batch. The binding constraint was one ~50-minute Closing rather than leaving open so this PR cannot move #2449's base and force it to re-run. The branch is untouched and this is reopenable if the batch does not land. |
apr audio-inspect-lintaccepted a body declaringsample_rate: 4294983296, printedsample_rate : Ok { rate: 16000 }and exited 0. That number is 2^32 + 16000, andraw as u32wraps, so the value aliased onto a canonical rate and passed while the report showed a rate the file never contained.--expected-sample-rate 16000did not catch it either — after the wrap the two were equal.channels: 4294967297aliased to 1 the same way, andsample_rate: 4294967296failed but reportedNonCanonicalRate { got: 0 }, another value absent from the input.Three more gates in the same family reported PASS for sections that were absent, empty, or of the wrong JSON type.
apr imatrix-linton{"leakage":{"calib_hashes":[1,2],"eval_hashes":[1,2]}}— two identical sets — printed[PASS] leakage: disjoint (|calib|=0, |eval|=0), becausefilter_map(Value::as_str)dropped every integer-typed hash and the gate then made a cardinality claim about data it had not read.{"leakage":"nonsense"}produced the same green line.apr embeddings-lintdid it with{"shape":{}}: input_len, hidden_size and data all defaulted to 0/0/[], 0 == 0,Ok { n_rows: 0 }.apr hang-trace-lint --world-size 0short-circuited toOk { ranks_seen: 0 }on an empty trace dir, so a CI job whose world size came from an env var that resolved to empty got a green gate on no evidence at all.apr quant-preservation-lintacross two models with different tokenizers emitted 13,282,156 bytes over 19 lines, one line 4,814,215 characters long (14,896,285 bytes under--json). It stored the untruncatedDebugof every diverged GGUF value including the whole 248k-entrytokenizer.ggml.mergesandtokenizer.ggml.tokens. The actionable signals in that same run were buried between three multi-megabyte dumps.Before / after
Both binaries carry an embedded git SHA proving which tree they came from, and both were built into a private target dir — the workspace target dir is shared with three other worktrees, and the first "after" binary picked up from it turned out to be another worktree's build, byte-identical to the "before" one and showing zero change.
apr 0.63.0 (8cc3aafeb)apr 0.63.0 (dd617995c)sample_rate: 4294983296Ok { rate: 16000 }rc=0SampleRateOutOfRange { got: 4294983296 }rc=5--expected-sample-rate 16000Ok { rate: 16000 }rc=0SampleRateOutOfRange { got: 4294983296 }rc=5channels: 4294967297Ok { channels: 1 }rc=0ChannelsOutOfRange { got: 4294967297 }rc=5sample_rate: 4294967296NonCanonicalRate { got: 0 }rc=5SampleRateOutOfRange { got: 4294967296 }rc=5[1,2]vs[1,2][PASS] disjoint (|calib|=0, |eval|=0)rc=0[FAIL] leakage detected: 2 overlapping item(s): ["1", "2"]rc=1{"leakage":"nonsense"}[PASS] disjoint (|calib|=0, |eval|=0)rc=0[FAIL] leakage evidence unreadable: \leakage` is not an object` rc=1{"shape":{}}[PASS] Ok { n_rows: 0 }rc=0[FAIL] shape evidence unreadable: missing \input_len`` rc=1--world-size 0Ok { ranks_seen: 0 }rc=0WorldSizeZerorc=5--jsonControls held throughout: genuinely disjoint hash sets still pass, a well-formed
{"shape":{...}}still passes, an explicitly empty{"input_len":0,"hidden_size":4,"data":[]}response still passes,--world-size 2on a populated dir still passes, and identical GGUFs still reportPRESERVEDin 319 bytes. The fix rejects absent evidence, not legitimately empty evidence.The summarised divergence line now carries what a reader can act on:
Scalars are untouched — only values over 200 characters are summarised.
Root cause
crates/apr-cli/src/commands/audio_inspect_classifier.rs:106,139—as u32applied after only araw <= 0checkcrates/apr-cli/src/commands/imatrix_lint.rs:166—and_then(as_array).map(filter_map(as_str)).unwrap_or_default()over a possibly-absent, possibly-non-string sectioncrates/apr-cli/src/commands/embeddings_lint.rs:117— three defaults standing in for missing fieldscrates/apr-cli/src/commands/hang_trace_classifier.rs:69—world_size == 0returnedOk, with a comment calling it "Degenerate"crates/apr-cli/src/commands/quant_preservation.rs:100—format!("{ref_val:?}")stored whole, rendered whole at:201-208and serialized whole under--jsonThe help-text finding found a second instance
apr typical-p-lint --helpprinted its only required flag with an empty description, so its non-obvious observation schema had no route to a user. Instead of fixing the one string, the falsifier walks every*-lintsubcommand through clap'sCommandFactoryand asserts each flag carries help text. It immediately went red on an instance the audit had not found:Confirmed against the shipped binary —
embeddings-lint --help | cat -Ashows--observation-file <FILE> $. Both are documented now, and a new*-lintcommand that forgets its flag doc turns the test red.Mutation check
Each fix reverted in turn, tests kept:
Restored:
test result: ok. 6686 passed; 0 failed.cargo fmt --all -- --checkrc=0,cargo clippy -p apr-cli --lib -- -D warningsrc=0.Left for other PRs
apr quantize/apr serve runparser rejects with exit 2. Fixing it means coupling those gates to the real clap definition, which will make four shipped commands start failing (correctly) on inputs users have today.apr attn-viz,apr dataset audio-inspect,apr trace --check-finite, …) that the binary does not have.--jsonenvelopes; unifying them is a family-wide change.Invalid APR formatprefix on JSON observation files. Issue The lint family ships 5 different exit codes for identical failure conditions #2404 already owns exactly this, so touchingCliErrorhere would collide.Refs #2377 (partial) — remaining: 2, 3, 6, 8, 9
Audit epic: #2373