fix(cli): six qa/bench/probar/qualify defects — a gate that could not fail, a manifest of files that did not exist, and three flags with no effect

[noahgift]

Found by dogfooding apr 0.63.0 installed from crates.io. Six of the seven findings in #2380 — the seventh (--assert-tps divided by ten) is PR #2372 and is not touched here.

Every repro below was run first against the crates.io 0.63.0 binary, then against a release build of this branch.

apr qualify --tier standard failed for everyone outside the source checkout

The Contract Audit gate shelled out to pv audit contracts/aprender/tensor-layout-v1.yaml — a path resolved against the caller's working directory (qualify.rs:441-446). The gate's verdict was therefore a function of where you stood.

# 0.63.0 from /tmp
rc=5
│ Contract Audit (pv) │ ✗ FAIL │ error: Failed to read contract file: No such file or directory (os error 2) │

# 0.63.0 from /home/noah/src/aprender — same model, same tier
│ Contract Audit (pv) │ ✓ PASS │ OK │

The contract ships with the source tree, not with the binary, so a missing file is a missing input, not a failed audit. locate_contract now resolves it from APR_CONTRACTS_DIR or by walking up from cwd, and the gate SKIPs with an actionable message when it is nowhere to be found.

# this branch, from /tmp
│ Contract Audit (pv) │ ○ SKIP │ contracts/aprender/tensor-layout-v1.yaml not found — set APR_CONTRACTS_DIR or run from an aprender checkout │

# this branch, from /tmp, with APR_CONTRACTS_DIR=/home/noah/src/aprender/contracts
│ Contract Audit (pv) │ ✓ PASS │ OK │

# this branch, from the repo root — unchanged
│ Contract Audit (pv) │ ✓ PASS │ OK │

apr probar tensor listed files it had not written

It printed a Generated files: manifest of .png paths (probar.rs:249) and then wrote Netpbm .pgm bytes (probar.rs:417, with let _ = png_path; at :445 silencing the unused variable). A CI step copying the listed paths into probar fixtures fails with ENOENT.

# 0.63.0
Generated files:
  - .../layer_000_block_0.png
  - .../layer_001_block_1.png
$ ls
layer_000_block_0.meta.json  layer_000_block_0.pgm  layer_001_block_1.meta.json  layer_001_block_1.pgm

Rather than downgrade the promise, this adds a dependency-free 8-bit grayscale PNG encoder (png_encode.rs: IHDR + zlib-stored IDAT + IEND, CRC-32 and Adler-32, ~110 lines) and the export writes real PNGs. The printed manifest and the export are now derived from one function, so they cannot drift apart again.

# this branch
Generated files:
  - .../layer_000_block_0.png
  - .../layer_000_block_0.meta.json
  - .../layer_001_block_1.png
  - .../layer_001_block_1.meta.json
$ file layer_000_block_0.png
... PNG image data, 256 x 100, 8-bit grayscale, non-interlaced
$ python3 -c "from PIL import Image; print(Image.open('layer_000_block_0.png').format)"
PNG

Separately, --format bogus was swallowed by .unwrap_or(ExportFormat::Both) (dispatch_analysis.rs:85) and silently exported something the user never asked for, at exit 0. The FromStr error it already produced is now surfaced:

# 0.63.0: bogus rc=0, wrote .pgm + manifest.json
# this branch:
rc=5
error: Validation failed: Unknown format: bogus. Use json, png, or both

apr qa passed two GPU gates vacuously on a CPU-only build

# 0.63.0 (default features, no cuda)
  ✓ PASS Capability Match Architecture 'qwen2': all 5 required ops supported by GPU
  ○ SKIP GPU Speedup Skipped: Requires 'inference' and 'cuda' features
  ✓ PASS PTX Parity 0/0 kernel pairs passed PTX parity
  ○ SKIP GPU State Isolation Skipped: Requires inference+cuda features

Capability Match asserted GPU kernel support from a build with no GPU backend compiled in. PTX Parity reported a pass over zero comparisons: without the cuda feature validate_all_kernel_pairs returns an empty report, and all_passed() is failed == 0, which is vacuously true. Zero comparisons is not a pass — that gate could not fail by construction. Both now SKIP, matching their two neighbours:

# this branch, same model
  ○ SKIP Capability Match Skipped: Requires 'inference' and 'cuda' features
  ○ SKIP GPU Speedup Skipped: Requires 'inference' and 'cuda' features
  ○ SKIP PTX Parity Skipped: no PTX kernel pairs compiled in (requires 'cuda' feature)
  ○ SKIP GPU State Isolation Skipped: Requires inference+cuda features

apr bench --percentiles emitted a null metric instead of an argument error

The help text says "Values must be in (0, 100]"; 0 and 101 were accepted and reached bench.rs:264-267, where the non-Ok PercentileOutcome serialised as null under a plausible key.

# 0.63.0 --percentiles 101
  "latency_p101_ms": null

# this branch
error: invalid value '101' for '--percentiles <PERCENTILES>': percentile '101' out of range: values must be in (0, 100]
rc=2
# and the in-range case is untouched: --percentiles 50 -> "latency_p50_ms": 430.41200000000003

Non-numeric input was already rejected by the value parser, so the range check belongs there; bench::run also checks, for non-clap callers.

apr eval --device accepted anything and did nothing

--device cpu, cuda and bogus produced byte-identical perplexity with no mention of a device anywhere in the output — a typo like --device cude ran on CPU while the operator believed otherwise. The only readers of device are the humaneval/mbpp paths (inference.rs:121, :1531); the perplexity path never consults it.

# this branch
$ apr eval model.gguf ... --device cuda
⚠ --device cuda requested, but perplexity evaluation is CPU-only — running on cpu (--device applies to --task humaneval/mbpp)
  Device: cpu
$ apr eval model.gguf ... --device bogus
error: invalid value 'bogus' for '--device <DEVICE>'
  [possible values: cpu, cuda]
rc=2
$ apr eval ... --device cuda --json
  "device": "cpu",

apr showcase --step <unknown> contradicted the command line

The unknown-value branch fell through to the not-specified branch because both mapped to None.

# 0.63.0
error: Validation failed: No step specified. Use --auto-verify or --step <step>

# this branch
error: Validation failed: unknown step 'bogus'; available: import, gguf, convert, apr, brick, bench, chat, visualize, zram, cuda, all

Mutation check

Reverting all six fixes while keeping the tests turns 14 tests RED:

test result: FAILED. 6638 passed; 13 failed; 12 ignored

    commands::bench::tests::bench_run_rejects_out_of_range_percentiles
    commands::bench::tests::parse_percentile_rejects_out_of_range_points
    commands::eval::eval_mod_tests::perplexity_device_notice_fires_only_when_the_request_is_not_honoured
    commands::probar::tests::test_every_listed_generated_file_actually_exists
    commands::probar::tests::test_export_by_format_both_creates_all
    commands::probar::tests::test_export_by_format_png_creates_png_only
    commands::probar::tests::test_export_png_histogram_normalization
    commands::qa::tests::ptx_gate_skips_when_no_kernel_pairs_were_compared
    commands::qualify::tests::test_locate_contract_finds_it_from_a_subdirectory
    commands::qualify::tests::test_locate_contract_honours_explicit_contracts_dir
    commands::qualify::tests::test_locate_contract_returns_none_off_tree
    tests::test_dispatch_analysis_probar_rejects_unknown_format
    tests::test_showcase_unknown_step_names_the_offending_value

with, verbatim:

---- commands::qualify::tests::test_locate_contract_returns_none_off_tree stdout ----
panicked at crates/apr-cli/src/commands/qualify.rs:739:9:
must not invent a path that does not exist

---- commands::probar::tests::test_every_listed_generated_file_actually_exists stdout ----
panicked at crates/apr-cli/src/commands/probar_tests_export_png.rs:231:13:
Png: listed /tmp/.tmpxBAi54/layer_000_block_0.png but it was never written

---- commands::qa::tests::ptx_gate_skips_when_no_kernel_pairs_were_compared stdout ----
panicked at crates/apr-cli/src/commands/fields.rs:388:9:
0/0 comparisons must be reported as SKIP

---- commands::bench::tests::parse_percentile_rejects_out_of_range_points stdout ----
panicked at crates/apr-cli/src/commands/bench_brick_name.rs:408:39:
point outside (0, 100] must be rejected: 0.0

---- commands::eval::eval_mod_tests::perplexity_device_notice_fires_only_when_the_request_is_not_honoured stdout ----
panicked at crates/apr-cli/src/commands/eval/eval_mod_tests.rs:466:51:
cuda must be called out

---- tests::test_dispatch_analysis_probar_rejects_unknown_format stdout ----
panicked at crates/apr-cli/src/lib_dispatch_coverage.rs:349:9:
error must name the rejected value, got: File not found: /tmp/nonexistent_probar_model.apr

---- tests::test_showcase_unknown_step_names_the_offending_value stdout ----
panicked at crates/apr-cli/src/lib_dispatch_coverage.rs:281:9:
error must quote what the user typed, got: Validation failed: No step specified. Use --auto-verify or --step <step>

The 14th, capability_gate_makes_no_gpu_claim_without_cuda, was itself vacuous on the first pass — its GGUF fixture carried no metadata, so the gate short circuited on "missing architecture metadata" and never reached the GPU claim. It now writes a general.architecture KV and additionally asserts the SKIP message, which is what proves the fixture engaged. Re-mutated afterwards:

---- commands::qa::tests::capability_gate_makes_no_gpu_claim_without_cuda stdout ----
panicked at crates/apr-cli/src/commands/fields.rs:451:9:
a CPU-only build must not assert GPU support, got: Architecture 'qwen2': all 5 required ops supported by GPU

One pre-existing test encoded a defect and was rewritten: test_export_by_format_png_creates_pgm_only asserted that a .pgm existed — exactly the file the command claimed it had not written. It is now test_export_by_format_png_creates_png_only and asserts the .png the command advertises, plus that no stray Netpbm file is left behind.

Gates

cargo fmt --all -- --check clean, cargo clippy -p apr-cli --lib -- -D warnings clean, 6651 apr-cli lib tests pass. End-to-end verification ran against /mnt/nvme-raid0/targets/aprender/release/apr (apr 0.63.0 (5643db51d)) built from this branch — never a bare apr.

Audit epic: #2373

Refs #2380 (partial) — fixes findings 2, 3, 4, 5, 6 and 7. Finding 1 (apr qa --assert-tps divided by ten for GGUF and discarded for APR/SafeTensors) is not touched here; it is PR #2372 and remains the only open item in the cluster once this lands.

🤖 Generated with Claude Code

[noahgift]

Parking to let the merge queue drain — branch untouched, this will be reopened.

Six PRs are in the merge queue and their merge_group check runs have been starved for hours: 16 runners, and every open PR keeps re-triggering its own workspace-test alongside them. Cancelling those runs does not hold (new ones replace them within a minute) and drafting does not stop CI on this repo, so closing is the only lever that frees the fleet.

The queue is the only path by which anything actually merges, so it gets the runners until it is empty. Reopening immediately afterwards.

[noahgift]

Superseded by #2449 — this branch is merged verbatim into that batch.

The binding constraint was one ~50-minute workspace-test per PR on one shared box; nine concurrent PRs starved each other into 75-minute step timeouts (every blocked PR this morning classified as CONTENTION, zero merges in 4.5 hours). #2449 lands 24 branches in a single CI run.

Closing rather than leaving open so this PR cannot move #2449's base and force it to re-run. The branch is untouched and this is reopenable if the batch does not land.