Skip to content

fix(lint): four *-lint gates passed on evidence they never read, and one printed a number the input did not contain - #2438

Closed
noahgift wants to merge 1 commit into
mainfrom
fix/lint-family-degenerate-evidence
Closed

fix(lint): four *-lint gates passed on evidence they never read, and one printed a number the input did not contain#2438
noahgift wants to merge 1 commit into
mainfrom
fix/lint-family-degenerate-evidence

Conversation

@noahgift

Copy link
Copy Markdown
Contributor

apr audio-inspect-lint accepted a body declaring sample_rate: 4294983296, printed sample_rate : Ok { rate: 16000 } and exited 0. That number is 2^32 + 16000, and raw as u32 wraps, so the value aliased onto a canonical rate and passed while the report showed a rate the file never contained. --expected-sample-rate 16000 did not catch it either — after the wrap the two were equal. channels: 4294967297 aliased to 1 the same way, and sample_rate: 4294967296 failed but reported NonCanonicalRate { got: 0 }, another value absent from the input.

Three more gates in the same family reported PASS for sections that were absent, empty, or of the wrong JSON type. apr imatrix-lint on {"leakage":{"calib_hashes":[1,2],"eval_hashes":[1,2]}} — two identical sets — printed [PASS] leakage: disjoint (|calib|=0, |eval|=0), because filter_map(Value::as_str) dropped every integer-typed hash and the gate then made a cardinality claim about data it had not read. {"leakage":"nonsense"} produced the same green line. apr embeddings-lint did it with {"shape":{}}: input_len, hidden_size and data all defaulted to 0/0/[], 0 == 0, Ok { n_rows: 0 }. apr hang-trace-lint --world-size 0 short-circuited to Ok { ranks_seen: 0 } on an empty trace dir, so a CI job whose world size came from an env var that resolved to empty got a green gate on no evidence at all.

apr quant-preservation-lint across two models with different tokenizers emitted 13,282,156 bytes over 19 lines, one line 4,814,215 characters long (14,896,285 bytes under --json). It stored the untruncated Debug of every diverged GGUF value including the whole 248k-entry tokenizer.ggml.merges and tokenizer.ggml.tokens. The actionable signals in that same run were buried between three multi-megabyte dumps.

Before / after

Both binaries carry an embedded git SHA proving which tree they came from, and both were built into a private target dir — the workspace target dir is shared with three other worktrees, and the first "after" binary picked up from it turned out to be another worktree's build, byte-identical to the "before" one and showing zero change.

before — apr 0.63.0 (8cc3aafeb) after — apr 0.63.0 (dd617995c)
sample_rate: 4294983296 Ok { rate: 16000 } rc=0 SampleRateOutOfRange { got: 4294983296 } rc=5
same, --expected-sample-rate 16000 Ok { rate: 16000 } rc=0 SampleRateOutOfRange { got: 4294983296 } rc=5
channels: 4294967297 Ok { channels: 1 } rc=0 ChannelsOutOfRange { got: 4294967297 } rc=5
sample_rate: 4294967296 NonCanonicalRate { got: 0 } rc=5 SampleRateOutOfRange { got: 4294967296 } rc=5
imatrix, integer hashes [1,2] vs [1,2] [PASS] disjoint (|calib|=0, |eval|=0) rc=0 [FAIL] leakage detected: 2 overlapping item(s): ["1", "2"] rc=1
imatrix, {"leakage":"nonsense"} [PASS] disjoint (|calib|=0, |eval|=0) rc=0 [FAIL] leakage evidence unreadable: \leakage` is not an object` rc=1
embeddings, {"shape":{}} [PASS] Ok { n_rows: 0 } rc=0 [FAIL] shape evidence unreadable: missing \input_len`` rc=1
hang-trace, empty dir --world-size 0 Ok { ranks_seen: 0 } rc=0 WorldSizeZero rc=5
quant-preservation, diverging vocabs 13,282,156 B / 19 lines / longest 4,814,215 2,446 B / 22 lines / longest 338
same, --json 14,896,285 B 3,761 B

Controls held throughout: genuinely disjoint hash sets still pass, a well-formed {"shape":{...}} still passes, an explicitly empty {"input_len":0,"hidden_size":4,"data":[]} response still passes, --world-size 2 on a populated dir still passes, and identical GGUFs still report PRESERVED in 319 bytes. The fix rejects absent evidence, not legitimately empty evidence.

The summarised divergence line now carries what a reader can act on:

tokenizer.ggml.tokens :: ArrayString(len=248320, sha256=3751959a6d53cc72, head=ArrayString(["!", "\"", "#", …)
                       → ArrayString(len=151936, sha256=77485efcba6e358f, head=ArrayString(["!", "\"", "#", …)
  first differing element: index 280 (len 248320 vs 151936)
general.architecture :: String("qwen35") → String("qwen2")
tokenizer.ggml.eos_token_id :: Uint32(248046) → Uint32(151645)

Scalars are untouched — only values over 200 characters are summarised.

Root cause

  • crates/apr-cli/src/commands/audio_inspect_classifier.rs:106,139as u32 applied after only a raw <= 0 check
  • crates/apr-cli/src/commands/imatrix_lint.rs:166and_then(as_array).map(filter_map(as_str)).unwrap_or_default() over a possibly-absent, possibly-non-string section
  • crates/apr-cli/src/commands/embeddings_lint.rs:117 — three defaults standing in for missing fields
  • crates/apr-cli/src/commands/hang_trace_classifier.rs:69world_size == 0 returned Ok, with a comment calling it "Degenerate"
  • crates/apr-cli/src/commands/quant_preservation.rs:100format!("{ref_val:?}") stored whole, rendered whole at :201-208 and serialized whole under --json

The help-text finding found a second instance

apr typical-p-lint --help printed its only required flag with an empty description, so its non-obvious observation schema had no route to a user. Instead of fixing the one string, the falsifier walks every *-lint subcommand through clap's CommandFactory and asserts each flag carries help text. It immediately went red on an instance the audit had not found:

lint commands with an undocumented flag: ["embeddings-lint --observation_file"]

Confirmed against the shipped binary — embeddings-lint --help | cat -A shows --observation-file <FILE> $. Both are documented now, and a new *-lint command that forgets its flag doc turns the test red.

Mutation check

Each fix reverted in turn, tests kept:

audio       8 failed   left: Ok { rate: 16000 }          right: SampleRateOutOfRange { got: 4294983296 }
                       left: Ok { channels: 1, .. }      right: ChannelsOutOfRange { got: 4294967297 }
                       left: NonCanonicalRate { got: 0 } right: SampleRateOutOfRange { got: 4294967296 }
imatrix     4 failed   [PASS] leakage (FALSIFY-CRUX-B-07-001): disjoint (|calib|=0, |eval|=0)
embeddings  4 failed   [PASS] shape (FALSIFY-CRUX-C-13-001): Ok { n_rows: 0 }
hang-trace  2 failed   left: Ok { ranks_seen: 0 }        right: WorldSizeZero
quant       2 failed   divergence report must stay bounded; got 5855530 bytes
                       left: None  right: Some("first differing element: index 2047 (len 4096 vs 4096)")
help        1 failed   lint commands with an undocumented flag: ["typical-p-lint --observation_file"]

Restored: test result: ok. 6686 passed; 0 failed. cargo fmt --all -- --check rc=0, cargo clippy -p apr-cli --lib -- -D warnings rc=0.

Left for other PRs

  • finding 2 — the "CLI flag accepted" gates stamp PASS for argv the real apr quantize / apr serve run parser rejects with exit 2. Fixing it means coupling those gates to the real clap definition, which will make four shipped commands start failing (correctly) on inputs users have today.
  • finding 3 — 8 linters document a producer command (apr attn-viz, apr dataset audio-inspect, apr trace --check-finite, …) that the binary does not have.
  • finding 6 — the family ships two mutually incompatible --json envelopes; unifying them is a family-wide change.
  • findings 8 and 9 — the exit-code split (3/4/5 vs 1) and the Invalid APR format prefix on JSON observation files. Issue The lint family ships 5 different exit codes for identical failure conditions #2404 already owns exactly this, so touching CliError here would collide.

Refs #2377 (partial) — remaining: 2, 3, 6, 8, 9
Audit epic: #2373

…one printed a number the input did not contain

`apr audio-inspect-lint` accepted a body declaring `sample_rate: 4294983296`,
printed `sample_rate : Ok { rate: 16000 }` and exited 0. 4294983296 is
2^32 + 16000, and `raw as u32` wraps, so the value aliased onto a canonical
rate and passed the gate while the report showed a number that was never in
the file. `--expected-sample-rate 16000` did not catch it either: after the
wrap the two were equal. `channels: 4294967297` aliased to 1 the same way.
`sample_rate: 4294967296` did fail, but reported `NonCanonicalRate { got: 0 }`
— again a value the observation did not contain.

  before (apr 0.63.0, base 8cc3aaf)  after (this branch)
  sample_rate : Ok { rate: 16000 }     sample_rate : SampleRateOutOfRange { got: 4294983296 }
  rc=0                                 rc=5

Three more gates in the family reported PASS for sections that were absent,
empty, or of the wrong JSON type:

  $ apr imatrix-lint --observation-file <(echo '{"leakage":{"calib_hashes":[1,2],"eval_hashes":[1,2]}}')
  [PASS] leakage (FALSIFY-CRUX-B-07-001): disjoint (|calib|=0, |eval|=0)   rc=0

Those two sets are identical. `filter_map(Value::as_str)` silently dropped
every integer-typed hash, leaving two empty sets, and the gate then printed a
cardinality claim about data it had not read. `{"leakage":"nonsense"}` produced
the same green line. `apr embeddings-lint` did it with `{"shape":{}}` —
input_len, hidden_size and data all defaulted to 0/0/[], so 0 == 0 and the
verdict was `Ok { n_rows: 0 }`. `apr hang-trace-lint --world-size 0`
short-circuited to `Ok { ranks_seen: 0 }` on an EMPTY trace dir, so a CI job
whose world size came from an env var that resolved to empty got a green gate
on no evidence at all. All three now fail: unreadable evidence is unknown, not
disjoint, not Ok. Genuinely disjoint hash sets, an explicitly empty
`{"input_len":0,"hidden_size":4,"data":[]}` response, and a populated trace dir
all still pass — the fix rejects absent evidence, not legitimately empty
evidence.

Separately, `apr quant-preservation-lint` across two models with different
tokenizers emitted 13,282,156 bytes over 19 lines, one of them 4,814,215
characters long (14,896,285 bytes under `--json`): it stored the untruncated
`Debug` of every diverged GGUF value, including whole 248k-entry
`tokenizer.ggml.merges` and `tokenizer.ggml.tokens` arrays. The actionable
signals in that same run — `general.architecture :: String("qwen35") →
String("qwen2")`, `tokenizer.ggml.eos_token_id :: Uint32(248046) →
Uint32(151645)` — were buried between three multi-megabyte dumps that wedge a
terminal and blow a CI log budget. Values over 200 chars are now summarised
with element count, a content digest and a head sample, plus the first
differing index; scalars still render verbatim, unchanged. The identical run:

  before  13,282,156 bytes / 19 lines / longest line 4,814,215
  after        2,446 bytes / 22 lines / longest line       338
  --json  14,896,285 bytes  ->  3,761 bytes

  tokenizer.ggml.tokens :: ArrayString(len=248320, sha256=3751959a6d53cc72, head=ArrayString(["!", "\"", "#", …)
                         → ArrayString(len=151936, sha256=77485efcba6e358f, head=ArrayString(["!", "\"", "#", …)
    first differing element: index 280 (len 248320 vs 151936)

Root causes:
  crates/apr-cli/src/commands/audio_inspect_classifier.rs:106,139 — `as u32`
    after only a `raw <= 0` check
  crates/apr-cli/src/commands/imatrix_lint.rs:166 — `and_then(as_array)
    .map(filter_map(as_str)).unwrap_or_default()` on a possibly-absent,
    possibly-non-string section
  crates/apr-cli/src/commands/embeddings_lint.rs:117 — three `unwrap_or(0)` /
    `unwrap_or_default()` defaults standing in for missing fields
  crates/apr-cli/src/commands/hang_trace_classifier.rs:69 — `world_size == 0`
    returned Ok, with a comment calling it "Degenerate"
  crates/apr-cli/src/commands/quant_preservation.rs:100 —
    `format!("{ref_val:?}")` stored whole, rendered whole at :201-208

`apr typical-p-lint --help` printed its only required flag with an empty
description, so its non-obvious observation schema had no route to a user.
Rather than fix the one string, the falsifier walks every `*-lint` subcommand
through clap's CommandFactory and asserts each flag carries help text. It went
red on a second instance the audit had not found — `embeddings-lint
--observation-file`, likewise blank in the shipped 0.63.0 help. Both are
documented now, and a new `*-lint` that forgets turns the test red.

Mutation check. Reverting each fix while keeping its tests, one at a time:

  audio       8 failed  Ok { rate: 16000 } vs SampleRateOutOfRange { got: 4294983296 }
                        Ok { channels: 1 } vs ChannelsOutOfRange { got: 4294967297 }
                        NonCanonicalRate { got: 0 } vs SampleRateOutOfRange { got: 4294967296 }
  imatrix     4 failed  [PASS] leakage: disjoint (|calib|=0, |eval|=0)
  embeddings  4 failed  [PASS] shape: Ok { n_rows: 0 }
  hang-trace  2 failed  Ok { ranks_seen: 0 } vs WorldSizeZero
  quant       2 failed  divergence report must stay bounded; got 5855530 bytes
  help        1 failed  lint commands with an undocumented flag: ["typical-p-lint --observation_file"]

Restored: 6686 passed, 0 failed.

The end-to-end before/after above was measured on two binaries whose embedded
git SHA proves which tree each came from — `apr --version` reported
`apr 0.63.0 (8cc3aaf)` for the base and this branch's own commit for the
fix — both built into a private target dir. That matters: the workspace target
dir is shared with three other worktrees, and the first "after" binary taken
from it turned out to be another worktree's build, byte-identical to the
"before" one and showing zero change.

Not fixed here, each deserving its own PR: the "CLI flag accepted" gates that
stamp PASS for argv the real clap parser rejects with exit 2 (finding 2), the
8 linters whose documented producer command is absent from the binary
(finding 3), the two incompatible `--json` envelopes (finding 6), and the
exit-code / "Invalid APR format" split (findings 8 and 9), which issue #2404
already owns.

Refs #2377 (partial) — remaining: 2, 3, 6, 8, 9
Audit epic: #2373

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@noahgift

Copy link
Copy Markdown
Contributor Author

Temporarily closing to stop CI contention — the branch is untouched and this will be reopened, nothing is lost.

The shared clean-room runner host is at 91% disk with a load average over 100, and workspace-test has started timing out at 75 minutes on unrelated PRs (#2385) purely from contention. This audit has 23 PRs open at once and drafts still trigger full CI in this repo, so parking them as drafts did not reduce load.

Reopening in batches as the merge queue drains. The work is complete and reviewed; only the CI scheduling is being paced.

@noahgift

Copy link
Copy Markdown
Contributor Author

Superseded by #2449 — this branch is merged verbatim into that batch.

The binding constraint was one ~50-minute workspace-test per PR on one shared box; nine concurrent PRs starved each other into 75-minute step timeouts (every blocked PR this morning classified as CONTENTION, zero merges in 4.5 hours). #2449 lands 24 branches in a single CI run.

Closing rather than leaving open so this PR cannot move #2449's base and force it to re-run. The branch is untouched and this is reopenable if the batch does not land.

@noahgift noahgift closed this Aug 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant