qa / bench / probar / showcase / qualify: 7 defects incl. `apr qa --assert-tps` being unenforceable

[noahgift]

Found by dogfooding cargo install aprender 0.63.0 from crates.io (2026-08-09). Part of the audit epic — see #2373.

7 finding(s) in this cluster. Every one was reproduced against the installed 0.63.0 binary with verbatim output and at least two input variations before being recorded.

status
addressed by	#2372
remaining	6 of 7 findings still open

Findings

#	Sev	Target	Defect	Status
1	P0	qa	apr qa --assert-tps N is never enforced: for GGUF it is silently rewritten to max(10, N/10), for APR/SafeTensors it is discarded entirely — the throughput gate PASSES and apr qa exits 0 far below the asserted minimum	fixed by #2372
2	P1	qualify	apr qualify --tier standard|full runs its Contract Audit gate against a hardcoded cwd-relative path, so it FAILS for every user who is not sitting in the aprender source checkout	open
3	P1	probar tensor	apr probar tensor lists "Generated files" ending in .png that do not exist — it writes Netpbm .pgm; --format png never produces a PNG, and an invalid --format value is silently accepted	open
4	P2	qa	On a CPU-only build apr qa passes two GPU gates vacuously — Capability Match asserts ops are "supported by GPU" and PTX Parity passes with 0/0 kernel pairs	open
5	P2	bench	apr bench --percentiles documents "Values must be in (0, 100]" but accepts 0 and 101, emitting a null metric key into the JSON report	open
6	P2	eval	apr eval --device has no effect in perplexity mode and accepts arbitrary values — --device cuda and --device bogus both run identically on a CPU-only build with no warning	open
7	P2	showcase	apr showcase --step <unknown> reports "No step specified" — the error contradicts the command line the user typed	open

Evidence

1. [P0] `apr qa --assert-tps N` is never enforced: for GGUF it is silently rewritten to max(10, N/10), for APR/SafeTensors it is

Target: qa

Repro

APR=/tmp/claude-1000/-home-noah-src-aprender/3e7b3507-59c1-4c5a-873b-c4003bfeca64/scratchpad/dogfood/cratesio/bin/apr
# A) APR model: assert 100000 tok/s, gate passes at 4.1 tok/s, exit 0
"$APR" qa /home/noah/models/qwen2.5-coder-0.5b-instruct.apr --assert-tps 100000 \
  --iterations 1 --warmup 0 --max-tokens 4 \
  --skip-golden --skip-ollama --skip-contract --skip-metadata --skip-capability \
  --skip-format-parity --skip-ptx-parity --skip-gpu-state --skip-gpu-speedup --json > /tmp/qaapr.json 2>/tmp/qaapr.err; echo "rc=$?"
python3 -c "import json;d=json.load(open('/tmp/qaapr.json'));print([g for g in d['gates'] if g['name']=='throughput']);print('passed=',d['passed'])"
# B) GGUF model: assert 50 tok/s, gate passes at 15.9 tok/s, exit 0
"$APR" qa /home/noah/.cache/pacha/models/d47927c5638f80ab.gguf --assert-tps 50 \
  --iterations 1 --warmup 1 --max-tokens 64 \
  --skip-golden --skip-ollama --skip-contract --skip-metadata --skip-capability \
  --skip-format-parity --skip-ptx-parity --skip-gpu-state --skip-gpu-speedup --json > /tmp/qat50.json 2>/tmp/qat50.err; echo "rc=$?"

Observed

# A) APR, --assert-tps 100000
rc=0
{"name": "throughput", "passed": true, "message": "4.1 tok/s >= 1 tok/s threshold", "value": 4.050120237944564, "threshold": 1.0, "duration_ms": 3544, "skipped": false}
overall passed= True

# B) GGUF, --assert-tps 50
rc=0
{"name": "throughput", "passed": true, "message": "15.9 tok/s >= 10 tok/s threshold", "value": 15.881003639727522, "threshold": 10.0, "duration_ms": 12213, "skipped": false}
overall passed= True

# and the header contradicts the gate even with no flag at all:
│     Min TPS │ 100 tok/s                                            │
  ✗ FAIL Throughput 4.5 tok/s < 10 tok/s threshold

Expected / root cause

--assert-tps N is documented as "Minimum throughput threshold in tok/s". A run measuring 4.1 tok/s against an asserted 100000 tok/s, or 15.9 tok/s against an asserted 50 tok/s, must FAIL the throughput gate and exit non-zero. Root cause: crates/apr-cli/src/commands/speedup.rs:53-55 — let threshold = match format { ModelFormat::Gguf => 10.0_f64.max(config.min_tps / 10.0), ModelFormat::Apr | ModelFormat::SafeTensors => 1.0 }; — config.min_tps (set from --assert-tps, crates/apr-cli/src/commands/qa.rs:404) is divided by 10 for GGUF and thrown away for APR/SafeTensors. Separately, crates/apr-cli/src/commands/qa_gguf.rs:31 prints the un-transformed min_tps in the header, so the banner says 100 tok/s while the gate uses 10. This makes apr qa --assert-tps — the repo's own release gate — incapable of failing.

Variations tried: Three assert values on GGUF: 100000 -> threshold 10000, 100 -> threshold 10, 50 -> threshold 10 (confirms the /10 with a floor of 10). One assert value on the APR fixture: 100000 -> threshold 1.0 (confirms the flag is ignored outright for non-GGUF). Ran with no --assert-tps at all to show the default 100 also collapses to 10. Two different model files (pacha d47927c5638f80ab.gguf and qwen2.5-coder-0.5b-instruct.apr).

2. [P1] `apr qualify --tier standard|full` runs its Contract Audit gate against a hardcoded cwd-relative path, so it FAILS for e

Target: qualify

Repro

APR=/tmp/claude-1000/-home-noah-src-aprender/3e7b3507-59c1-4c5a-873b-c4003bfeca64/scratchpad/dogfood/cratesio/bin/apr
M=/home/noah/.cache/pacha/models/064a3693fa1ea02c.safetensors
# from a directory that is not the aprender repo:
cd /tmp && "$APR" qualify "$M" --tier standard > /tmp/qs_tmp.txt 2>/tmp/qs_tmp.err; echo "rc=$?"
grep -n 'Contract Audit' /tmp/qs_tmp.txt
# from the aprender repo root, the same gate passes:
cd /home/noah/src/aprender && "$APR" qualify "$M" --tier standard --json > /tmp/qs_repo.json 2>/dev/null
python3 -c "import json;d=json.load(open('/tmp/qs_repo.json'));print([ (g['status'],g['message']) for g in d['gates'] if g['name']=='contract_audit'])"

Observed

# from /tmp:
rc=5
19:  ✗ FAIL Contract Audit (pv) (1ms)
39:│ Contract Audit (pv) │ ✗ FAIL │ error: Failed to read contract file: No such file or directory (os error 2) │ 1ms      │
46:    ✗ Contract Audit (pv) — error: Failed to read contract file: No such file or directory (os error 2)

# from /home/noah/src/aprender:
[('PASS', 'OK')]

Expected / root cause

A shipped CLI must not resolve its own gate inputs relative to the caller's working directory. crates/apr-cli/src/commands/qualify.rs:441-446 hardcodes run_external_gate("contract_audit", "Contract Audit (pv)", "pv", &["audit", "contracts/aprender/tensor-layout-v1.yaml"], timeout) — a repo-relative path that only exists inside the aprender checkout. Any user who did cargo install aprender and also has pv on PATH (which the project's own docs instruct) gets a false FAIL and a non-zero exit from apr qualify --tier standard; users without pv get a silent SKIP instead, so the gate is either wrong or absent but never right off-tree. The contract should ship with the binary or the gate should locate it explicitly.

Variations tried: Ran the same model and tier from three working directories: the scratch dir, /tmp, and /home/noah/src/aprender — FAIL, FAIL, PASS respectively, isolating cwd as the only variable. Confirmed with both --tier standard and --tier full (contract_audit fails in both). Verified pv is on PATH at /home/noah/.cargo/bin/pv and that the referenced file exists only at /home/noah/src/aprender/contracts/aprender/tensor-layout-v1.yaml. Also confirmed the neighbouring external gate degrades differently: playbook_tools reports SKIP "apr-qa not on PATH", proving run_external_gate's skip path works and this really is a path bug.

3. [P1] `apr probar tensor` lists "Generated files" ending in .png that do not exist — it writes Netpbm .pgm; `--format png` nev

Target: probar tensor

Repro

APR=/tmp/claude-1000/-home-noah-src-aprender/3e7b3507-59c1-4c5a-873b-c4003bfeca64/scratchpad/dogfood/cratesio/bin/apr
M=/home/noah/.cache/pacha/models/064a3693fa1ea02c.safetensors
rm -rf /tmp/ptf && "$APR" probar tensor "$M" -o /tmp/ptf --format png > /tmp/pt.txt 2>&1; echo "rc=$?"
grep -A3 'Generated files' /tmp/pt.txt
ls /tmp/ptf
file /tmp/ptf/layer_000_block_0.pgm
# invalid format value is accepted:
rm -rf /tmp/ptb && "$APR" probar tensor "$M" -o /tmp/ptb --format bogus > /dev/null 2>&1; echo "bogus rc=$?"; ls /tmp/ptb

Observed

rc=0
Generated files:
  - /tmp/ptf/layer_000_block_0.png
  - /tmp/ptf/layer_001_block_1.png

# actual directory contents:
layer_000_block_0.meta.json
layer_000_block_0.pgm
layer_001_block_1.meta.json
layer_001_block_1.pgm

/tmp/ptf/layer_000_block_0.pgm: Netpbm image data, size = 256 x 100, rawbits, greymap

bogus rc=0
layer_000_block_0.meta.json
layer_000_block_0.pgm
layer_001_block_1.meta.json
layer_001_block_1.pgm
manifest.json

Expected / root cause

The printed manifest of generated artifacts must name files that exist — a CI step that copies the listed paths into probar fixtures fails with ENOENT. crates/apr-cli/src/commands/probar.rs:249 prints " - {}/layer_{:03}_{}.png" while probar.rs:417-418 writes layer_{:03}_{}.pgm (probar.rs:416 comments "For now, write as .pgm" and probar.rs:445 does let _ = png_path; to silence the unused-variable warning). Either emit real PNG or print .pgm and rename the format value. Separately, --format bogus is accepted because crates/apr-cli/src/dispatch_analysis.rs:85 does format.parse().unwrap_or(probar::ExportFormat::Both), discarding the FromStr error that probar.rs:71 already produces ("Unknown format: {s}. Use json, png, or both").

Variations tried: All four --format values: png -> .pgm only, json -> manifest.json only, both -> .pgm + manifest.json, bogus -> silently treated as both. Confirmed the .png claim is printed in every png/both run including the --golden comparison run. Verified file type with file (Netpbm greymap) and the P5 magic in the header bytes. Checked the neighbouring flags work correctly as a control: --golden + --assert correctly PASS on a self-comparison and correctly exit 5 with per-layer cosine values on a different model, and a missing golden dir exits 3.

4. [P2] On a CPU-only build `apr qa` passes two GPU gates vacuously — Capability Match asserts ops are "supported by GPU" and PT

Target: qa

Repro

APR=/tmp/claude-1000/-home-noah-src-aprender/3e7b3507-59c1-4c5a-873b-c4003bfeca64/scratchpad/dogfood/cratesio/bin/apr
"$APR" --version   # apr 0.63.0 (v0.63.0+no-git), default features, no cuda
"$APR" qa /home/noah/.cache/pacha/models/d47927c5638f80ab.gguf \
  --iterations 1 --warmup 0 --max-tokens 4 --skip-ollama > /tmp/qa1.txt 2>&1; echo "rc=$?"
grep -E 'Capability Match|PTX Parity|GPU Speedup|GPU State' /tmp/qa1.txt

Observed

✓ PASS Capability Match Architecture 'qwen2': all 5 required ops supported by GPU
  ✓ PASS PTX Parity 0/0 kernel pairs passed PTX parity
  ○ SKIP GPU Speedup Skipped: Requires 'inference' and 'cuda' features
  ○ SKIP GPU State Isolation Skipped: Requires inference+cuda features

# same run, proving no GPU is in play:
(apr bench on the same file) CUDA: Not available (CPU mode)
(apr parity on the same file) error: Feature not enabled: cuda feature required for parity check

Expected / root cause

Two gates in the same report correctly SKIP with "Requires 'inference' and 'cuda' features"; the other two assert GPU facts on a build that cannot touch a GPU. "all 5 required ops supported by GPU" is a claim about hardware/kernels this binary has no path to, and "0/0 kernel pairs passed PTX parity" is a gate that structurally cannot fail — zero comparisons is not a pass. Both should SKIP on a non-cuda build (matching gpu_speedup / gpu_state_isolation), or state that they validated metadata only. As written they inflate the green-gate count in the release report and are exactly the fail-open pattern the project's own guidance calls out.

Variations tried: Ran the full qa report on two models (pacha d47927c5638f80ab.gguf and qwen2.5-coder-0.5b-instruct.apr) — both show the same two vacuous passes. Corroborated the absence of CUDA three independent ways on the same binary: apr bench prints "CUDA: Not available (CPU mode)", apr parity exits 9 with "cuda feature required for parity check", and apr showcase --gpu prints "GPU requested but CUDA feature not enabled, falling back to CPU". Also confirmed --skip-capability and --skip-ptx-parity do suppress them, so the gates are reachable and the passes are real verdicts, not display artifacts.

5. [P2] `apr bench --percentiles` documents "Values must be in (0, 100]" but accepts 0 and 101, emitting a null metric key into

Target: bench

Repro

APR=/tmp/claude-1000/-home-noah-src-aprender/3e7b3507-59c1-4c5a-873b-c4003bfeca64/scratchpad/dogfood/cratesio/bin/apr
G=/home/noah/.cache/pacha/models/d47927c5638f80ab.gguf
"$APR" bench "$G" --warmup 0 --iterations 1 --max-tokens 2 --percentiles 0   --json > /tmp/p0.json   2>/dev/null; echo "rc=$?"; tail -3 /tmp/p0.json
"$APR" bench "$G" --warmup 0 --iterations 1 --max-tokens 2 --percentiles 101 --json > /tmp/p101.json 2>/dev/null; echo "rc=$?"; tail -3 /tmp/p101.json

Observed

# --percentiles 0
  "passed": false,
  "latency_p0_ms": null
}

# --percentiles 101
  "passed": false,
  "latency_p101_ms": null
}

Expected / root cause

The help text states the constraint ("Values must be in (0, 100]"), and non-numeric input is already rejected by the value parser (--percentiles abc -> rc=2 "invalid float literal", --percentiles '' -> rc=2 "cannot parse float from empty string"), so the parser is the right place to range-check. Instead an out-of-range point silently reaches crates/apr-cli/src/commands/bench.rs:264-267, where the non-Ok PercentileOutcome is serialised as null under a key like latency_p101_ms. A consumer of the CRUX-E-07 report sees a plausible-looking metric key with a null value instead of an argument error.

Variations tried: Six --percentiles inputs on the same model: 0 -> accepted, null; 101 -> accepted, null; -5 -> rc=2 (clap treats it as a flag, with a '-- -5' tip); abc -> rc=2 invalid float literal; '' -> rc=2 cannot parse float; '50,50' -> accepted, single latency_p50_ms emitted. Confirmed the in-range default (50,95,99) produces three finite values on the same model, so the percentile machinery itself works.

6. [P2] `apr eval --device` has no effect in perplexity mode and accepts arbitrary values — `--device cuda` and `--device bogus`

Target: eval

Repro

APR=/tmp/claude-1000/-home-noah-src-aprender/3e7b3507-59c1-4c5a-873b-c4003bfeca64/scratchpad/dogfood/cratesio/bin/apr
G=/home/noah/.cache/pacha/models/d47927c5638f80ab.gguf
for d in cpu cuda bogus; do
  echo "--- device=$d"
  "$APR" eval "$G" --dataset custom --text "hello world hello world" --device "$d" > /tmp/e_$d.txt 2>&1; echo "rc=$?"
  grep -E 'Perplexity|Cross-entropy|device|cuda|CUDA|GPU' /tmp/e_$d.txt
done

Observed

--- device=cuda
rc=5
Perplexity: 2526.29 (FAIL: > 20.0)
  Cross-entropy: 7.8345

--- device=bogus
rc=5
Perplexity: 2526.29 (FAIL: > 20.0)
  Cross-entropy: 7.8345

# byte-identical numbers to --device cpu; no mention of device/CUDA/GPU anywhere in the output

Expected / root cause

--device <DEVICE> Device for inference: "cpu" (default) or "cuda" (GPU-accelerated, ALB-089). On a build without the cuda feature, --device cuda should either error or emit the same explicit fallback notice that apr showcase --gpu gets right ("⚠ GPU requested but CUDA feature not enabled, falling back to CPU"), and an unrecognised device name should be rejected. In the default perplexity path the value is never consulted at all: the only readers of device are crates/apr-cli/src/commands/eval/inference.rs:121 and :1531 (if device == "cuda"), both on the classification/code-eval path. Silently accepting --device bogus also means a typo like --device cude runs on CPU while the operator believes it ran on GPU — the labelled-by-intent failure mode this codebase has been bitten by before.

Variations tried: Three device values (cpu, cuda, bogus) against the same model and text — identical perplexity to 2 decimal places and identical cross-entropy to 4, i.e. no behavioural delta whatsoever. Grepped the full stdout+stderr of each run for device/CUDA/GPU strings: zero matches. Confirmed the same binary DOES report device state elsewhere (apr bench prints "CUDA: Not available (CPU mode)", apr parity exits 9 naming the missing cuda feature), so the information is available to the process and simply is not surfaced or acted on here.

7. [P2] `apr showcase --step ` reports "No step specified" — the error contradicts the command line the user typed

Target: showcase

Repro

APR=/tmp/claude-1000/-home-noah-src-aprender/3e7b3507-59c1-4c5a-873b-c4003bfeca64/scratchpad/dogfood/cratesio/bin/apr
"$APR" showcase --step bogus --runs 1 > /tmp/sh2.txt 2>&1; echo "rc=$?"; tail -3 /tmp/sh2.txt

Observed

rc=5
  cuda          - CUDA Graph + DP4A brick demo (Sections 5.2/5.3)
  all           - Run all steps
error: Validation failed: No step specified. Use --auto-verify or --step <step>

Expected / root cause

The user did specify a step. The message should name the offending value, e.g. "unknown step 'bogus'; available: import, gguf, convert, apr, brick, bench, visualize, zram, cuda, all". As written, a typo'd step (--step ggug) sends the operator hunting for a missing flag they already passed. The exit code (5) and the printed step list are correct; only the sentence is wrong.

Variations tried: Compared against the genuinely-empty case (apr showcase --runs 1 --tier tiny with no --step and no --auto-verify), which produces the exact same message — confirming the unknown-value branch falls through to the not-specified branch rather than having its own. Also confirmed a valid step is honoured (--step gguf reaches "═══ Step B: GGUF Inference ═══") so the value is parsed and matched, not ignored.

[noahgift]

Fix status

• fix(qa): --assert-tps was divided by ten, or discarded — the release gate could not fail #2372 (open, in the merge queue) — apr qa --assert-tps N now uses N as the threshold instead of N/10 or 1.0. 6 of 7 findings remain

Each fix reproduces the defect against the crates.io 0.63.0 binary first, ships a falsifier that was observed failing when the fix is reverted, and is verified end-to-end against a binary built from the branch.

[noahgift]

Fixed on main by #2449 (batch 1), which merged fix/qa-bench-probar-qualify-gates verbatim.

The 24 audit branches were batched into a single PR because the binding constraint was one ~50-minute workspace-test per PR on one shared box — nine concurrent PRs starved each other into 75-minute step timeouts and produced zero merges in 4.5 hours.

Merge commit: e759af2.