fix(dogfood): 24 audit fixes in one batch — the CI bottleneck is one workspace-test per PR, not the work - #2449
Merged
Merged
Conversation
…ardened a deployment had not
Dogfooding `cargo install aprender` 0.63.0: `apr serve run --no-metrics --no-cors`
produced byte-for-byte the same HTTP surface as `apr serve run` with no flags at
all. GET /metrics still returned 200 with the full Prometheus counter set, and
every response still carried `access-control-allow-origin: *`. The flags parsed,
the help text advertised them, and nothing downstream ever read them.
Before, apr 0.63.0 from crates.io, serving a Qwen2.5 1.5B GGUF on :18501 with
BOTH --no-metrics and --no-cors:
--- GET /metrics ---
HTTP/1.1 200 OK
content-type: text/plain; charset=utf-8
access-control-allow-origin: *
access-control-expose-headers: *
# HELP realizar_requests_total Total number of requests
realizar_requests_total 0
--- access-control headers on /health ---
access-control-allow-origin: *
access-control-expose-headers: *
After, same model and port, same flags, binary built from this branch:
--- GET /metrics ---
HTTP/1.1 404 Not Found
{"error":"not_found","message":"Route not found. See /health for available endpoints."}
--- access-control headers on /health ---
(no access-control headers)
Root cause is that neither flag reached the router. `--no-metrics` set
`ServerConfig.metrics`, which was consulted at exactly one place —
crates/apr-cli/src/commands/serve/mod.rs:89 — to decide whether to print the
"GET /metrics" banner line; the route itself was registered unconditionally at
crates/aprender-serve/src/api/router.rs:63. `--no-cors` set `ServerConfig.cors`,
a field carrying `#[allow(dead_code)]` and the comment "accepted but not yet
implemented" (crates/apr-cli/src/commands/serve/types.rs:29), while
router.rs:136 applied `CorsLayer::permissive()` to every router unconditionally.
The banner was the only thing either flag could move.
`RouterConfig` now carries `cors` and `metrics` alongside `openai_api`, both
defaulting to true so unflagged behaviour is unchanged. `create_router_with_config`
registers /metrics, /metrics/dispatch and /metrics/dispatch/reset only when
metrics are enabled, and applies the `CorsLayer` only when CORS is enabled. Every
apr-cli serve path that previously called `create_router` — CPU, GPU batched,
CUDA, APR, APR-Q4K, SafeTensors — now goes through `ServerConfig::router_config()`,
so the flags cannot silently stop being honoured on one backend. The apr-cli-local
SafeTensors inspection router gates its own /metrics on the same field, and the
CPU startup banner no longer advertises an endpoint that will 404.
Falsifiers live in crates/aprender-serve/src/api/tests/router_flags.rs and assert
wire behaviour, not config shape: status codes from /metrics and /metrics/dispatch,
the absence of any `access-control-*` response header, and a CORS-disabled OPTIONS
preflight. Two baseline tests assert the enabled path still serves
`realizar_requests_total` and still advertises `*`, so deleting the route or the
layer cannot pass.
Mutation check: with the test in place, both gates were forced back to their
pre-fix unconditional form (`if config.metrics` / `if config.cors` -> `if true`):
test api::tests::router_flags::metrics_disabled_makes_metrics_endpoints_404 ... FAILED
test api::tests::router_flags::cors_disabled_does_not_answer_preflight_permissively ... FAILED
test api::tests::router_flags::cors_disabled_sends_no_access_control_headers ... FAILED
assertion `left == right` failed: /metrics must 404 when metrics are disabled
left: 200
right: 404
CORS-disabled server still sent: ["access-control-allow-origin", "access-control-expose-headers"]
CORS-disabled preflight still sent: ["access-control-allow-methods", "access-control-allow-headers", "access-control-allow-origin"]
test result: FAILED. 3 passed; 3 failed
Restoring the fix returns 6 passed. End to end on the release binary the flags
were also exercised singly: --no-metrics alone 404s /metrics while CORS headers
remain, --no-cors alone strips the headers while /metrics still serves 200, and
with no flags the surface is identical to 0.63.0. 15481 aprender-serve and 6622
apr-cli lib tests pass.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Every name `apr list` shows was rejected by `apr rm`, including a model
`apr pull` had just downloaded. The hf:// ref and the absolute path failed
identically, so a user's only way to reclaim cache space was `rm -rf` on
~/.cache/pacha/models.
Observed with 0.63.0 installed from crates.io, against a throwaway
XDG_CACHE_HOME:
$ apr pull hf://hf-internal-testing/tiny-random-gpt2/model.safetensors
✓ Downloaded successfully
Path: .../pacha/models/f36eaf6d2b971606.safetensors
$ apr list
f36eaf6d2b971606 443.2 KB SafeTensors .../f36eaf6d2b971606.safetensors (orphan)
$ apr rm hf://hf-internal-testing/tiny-random-gpt2/model.safetensors
⚠ Model not found in cache
error: File not found: hf://hf-internal-testing/tiny-random-gpt2/model.safetensors
rc=3
Same three misses for `apr rm f36eaf6d2b971606`, for the file name, and for
the absolute path.
Root cause: `apr list` and `apr rm` did not share a namespace. `remove()`
(crates/apr-cli/src/commands/pull_remove_resolve_model.rs:12) delegated
solely to `ModelFetcher::remove()`, which looks up a URI-derived key in the
in-memory CacheManager loaded from <cache>/manifest.json
(crates/aprender-registry/src/fetcher.rs:398). Nothing on the pull path
writes that manifest: `run_single_file_streaming`
(crates/apr-cli/src/commands/pull.rs:122) streams the file straight into the
cache directory as blake3(uri)[..16].<ext> and never touches the fetcher.
`apr list` already had to reconcile against the directory to see those files
at all, which is why it labelled every one of them "(orphan)".
Fixed by making the cache directory the shared namespace rather than by
teaching pull to write the manifest. The manifest can never be complete —
files also land in that directory from `apr convert` and from plain copies,
neither of which goes through the fetcher — so making pull write it would
fix one writer and leave `rm` still unable to remove anything from the
others. The directory is complete by construction. `apr list`'s enumeration
is now `scan_cache_dir()`, and `apr rm` resolves against that same function,
so the two commands cannot drift apart again.
`rm` accepts every form a user can plausibly have in hand: the identifier
`list` prints, the file name, a path to the file, and the hf:// ref that was
pulled. An ambiguous stem (one model cached in two formats) is refused and
deletes neither file. GH-601's non-zero exit on a genuine miss is unchanged
and still covered.
After, same binary invocation, built from this tree:
$ apr rm hf://hf-internal-testing/tiny-random-gpt2/model.safetensors
✓ Model removed from cache
Path: .../pacha/models/f36eaf6d2b971606.safetensors
rc=0
$ apr list
No cached models found.
$ apr rm definitely-not-a-model
⚠ Model not found in cache
rc=3
Six falsification tests (FT-RMNS-001..006) drive `remove_from_cache_dir`
against a throwaway cache directory and assert on the file being present or
gone, not on a return shape. Mutation check: reverting only the fix — making
`resolve_cached_model_files` return an empty Vec, i.e. restoring the
manifest-only namespace — while keeping the tests turns four of the six RED
and leaves exactly the two that encode preserved behaviour green:
test removes_by_filename_and_by_absolute_path ... FAILED
test ambiguous_stem_deletes_nothing ... FAILED
test removes_by_the_hf_reference_that_was_pulled ... FAILED
test removes_by_the_name_list_prints ... FAILED
test non_model_files_are_not_removable ... ok
test unknown_reference_removes_nothing ... ok
FT-RMNS-002: file name form must delete
FT-RMNS-005: ambiguous ref must be an error: None
FT-RMNS-003: rm of the pulled hf:// ref must delete
/tmp/apr-rm-ns-hfref-540543-ThreadId(5)/d47927c5638f80ab.gguf
assertion `left == right` failed
left: None
right: Some("/tmp/apr-rm-ns-stem-540543-ThreadId(6)/064a3693fa1ea02c.safetensors")
test result: FAILED. 2 passed; 4 failed
With the fix restored: 6 passed. Full crate: 6633 passed, 0 failed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Four commands advertise `--json Output as JSON` in their own `--help` and
then wrote human-formatted text to stdout, so anything piping them to a
parser failed at the first byte. Reproduced against apr 0.63.0 installed
from crates.io, reading stdout only (stderr discarded — the library chatter
already goes to stderr, so this is exactly the stream a consumer parses):
$ apr pull qwen2.5-coder --dry-run --json # rc=0
=== APR Pull ===
Model: qwen2.5-coder
Resolved: hf://Qwen/Qwen2.5-Coder-7B-Instruct-GGUF/...
$ ... | python3 -c 'import json,sys; json.load(sys.stdin)'
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
`apr rm`, `apr publish --dry-run` and `apr import` failed identically.
`apr publish` was the worst of them: it dumped the entire generated model
card, YAML front-matter and all, onto stdout under `--json`. The flag was
being presented as the machine-readable contract — `apr pull`'s own
not-found error tells users to run `apr registry aliases --json`.
Root cause is that the flag never reached the commands. `--json` is a
global on `Cli`, and the dispatcher simply did not forward it:
crates/apr-cli/src/dispatch.rs:851 Commands::Rm { model_ref } => pull::remove(model_ref)
crates/apr-cli/src/dispatch.rs:850 Commands::List => pull::list(cli.json, cli.quiet)
The line above `Rm` forwards it, which is why `apr list --json` was correct
all along. `Commands::Pull`, `Commands::Import` and the publish arm in
dispatch_analysis.rs:1549 had the same omission, so no amount of work
inside those commands could have seen the flag.
The fix forwards `cli.json` and, in each command, separates resolving the
facts from rendering them. `DryRunReport`, `DryRunPlan`, `remove_stdout`,
`import_json_stdout` and `q4k_import_json_stdout` each return the exact
string that goes to stdout, so a unit test asserts the bytes a consumer
parses rather than an internal shape. Errors follow the convention the
already-correct commands use (`apr validate --json`, `apr stamp --json`):
the diagnostic goes to stderr and the exit code carries the outcome, so
stdout holds a whole document or nothing — never a half-written one.
After, from a release binary built at this commit (apr 0.63.0 (d37f2fe)):
$ apr pull qwen2.5-coder --dry-run --json
{
"model": "qwen2.5-coder",
"resolved": "hf://Qwen/Qwen2.5-Coder-7B-Instruct-GGUF/qwen2.5-coder-7b-instruct-q4_k_m.gguf",
"revision": "main",
"revision_kind": "RefName",
"offline": false,
"mode": "dry-run"
}
All four now parse. `apr rm missing --json` writes 0 bytes to stdout, keeps
its rc=3, and puts `error: File not found: ...` on stderr. The default
human output of all four is byte-identical to 0.63.0 (verified by diffing
the two binaries' stdout) and the exit codes are unchanged.
Mutation check: reverting the five renderers to their 0.63.0 behaviour
while keeping the tests turns 9 of the 12 new tests RED, and the failure
messages reprint the original defect:
test result: FAILED. 4 passed; 9 failed; 0 ignored
---- commands::pull::tests::pull_dry_run_json_stdout_carries_no_human_decoration ----
human decoration "=== APR Pull ===" leaked into `--json` stdout:
=== APR Pull ===
Model: qwen2.5-coder
...
---- commands::publish::tests::test_publish_dry_run_json_stdout_parses_as_json ----
`apr publish --dry-run --json` must write parseable JSON to stdout, but a
consumer got expected value at line 1 column 1. Actual stdout was:
=== DRY RUN: Would publish to paiml/test-model ===
...
The 3 that stay green under the mutation are the ones asserting the human
mode is unchanged, which is what they are there for.
`apr pull`'s streaming download path is deliberately not covered: it has 66
further print sites including progress bars, and half-suppressing its
banner would only make unparseable output look intentional, so the banner
now lives inside the dry-run renderer instead. `apr chat` (an interactive
REPL, which needs a streaming protocol rather than a document) and
`apr train apply` (already forwards `cli.json`; its stdout is only reachable
through a real training run) are left for separate work.
6634 apr-cli lib tests pass; fmt and clippy -D warnings clean.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ed but unparseable, so both flags were permanently ON
`apr embed --help` tells the user "Pass `--normalize false` to keep raw
magnitudes", and `apr rerank --help` tells them "Cross-encoders that skip the
pooler should pass `--with-pooler false`". Neither spelling parsed. Against the
0.63.0 binary installed from crates.io:
$ apr embed model.apr --vocab tok.json --text hi --normalize false
error: unexpected argument 'false' found
rc=2
$ apr embed model.apr --vocab tok.json --text hi --normalize=false
error: unexpected value 'false' for '--normalize' found; no more were expected
rc=2
$ apr rerank model.apr --vocab tok.json --query q --passages p --with-pooler false
error: unexpected argument 'false' found
rc=2
Because the field is a bare `bool`, clap derives a `SetTrue` switch, and
`default_value_t = true` then pins it on: the flag is true when omitted, true
when passed bare, and rejected when given a value. The "off" state had no
reachable spelling at all. Un-normalised embeddings could not be produced, and a
cross-encoder without a pooler dense layer could not be loaded.
Root cause: crates/apr-cli/src/extended_commands.rs:1238 (`normalize`) and :1193
(`with_pooler`) — `#[arg(long, default_value_t = true)]` on a bare `bool`. Both
handlers already consume the value correctly (embed.rs:283 gates `l2_normalize`,
rerank.rs:254 passes it to `CrossEncoder::new`); only the parser was wrong.
Fix: make the documented syntax parse, with `num_args(0..=1)` +
`default_missing_value = "true"` + `ArgAction::Set`. Omitted and bare both still
mean true, so no existing invocation changes meaning. There is no negatable-flag
convention already in this CLI to follow (grepped for `no-` longs,
`default_missing_value`, `ArgAction::Set` — no prior instances), and this
spelling is the one `--help` already promises.
Measured on a real cross-encoder/ms-marco-MiniLM (90 MB safetensors imported to
APR v2, 105 tensors), text "hello world", pooling mean:
crates.io 0.63.0, flag omitted normalize=true ||v|| = 1.000000
crates.io 0.63.0, bare --normalize normalize=true ||v|| = 1.000000
this build, --normalize false normalize=false ||v|| = 9.499828
this build, --normalize=false normalize=false ||v|| = 9.499828
this build, flag omitted / bare normalize=true ||v|| = 1.000000
Not a single-input result: "the quick brown fox" gives 9.492353 and "a
completely different sentence about databases" gives 8.943374 with normalisation
off, 1.000000 with it on. On the rerank side the same passage scores logit
8.796466 with the pooler and 0.108038 without it, so the flag reaches the model.
Twelve falsifiers assert the parsed value for each documented spelling, in the
existing pretrain.rs CLI-parse style (16 MiB worker thread — the flattened
command enum overflows the default test stack). Mutation check: reverting only
the two `#[arg(...)]` attributes and keeping the tests turns 6 of the 12 RED,
quoting the exact strings a user sees:
---- commands::embed::tests::embed_normalize_space_separated_false_turns_it_off stdout ----
assertion `left == right` failed: `--normalize false` is documented in --help and MUST reach the handler as false
left: Err("error: unexpected argument 'false' found\n\nUsage: apr embed [OPTIONS] --vocab <FILE> <MODEL>\n\n...")
right: Ok(false)
---- commands::rerank::tests::rerank_with_pooler_space_separated_false_turns_it_off stdout ----
assertion `left == right` failed: `--with-pooler false` is documented in --help and MUST reach the handler as false
left: Err("error: unexpected argument 'false' found\n\nUsage: apr rerank [OPTIONS] <MODEL>\n\n...")
right: Ok(false)
test result: FAILED. 6 passed; 6 failed
Restoring the attributes returns 12/12 green; full `cargo test -p apr-cli --lib`
is 6634 passed, 0 failed.
The one cost of an optional-valued flag is that a bare `--normalize` placed
immediately before the MODEL positional now swallows the path as its value.
That is asserted as a deliberate, legible error rather than left latent; the
documented order `apr embed MODEL --normalize` is unaffected, and
`require_equals` would have broken the `--normalize false` spelling that
`--help` documents.
Three more bare `bool` fields carry `default_value_t = true` and have the same
unreachable-off shape. They are out of scope here and untouched:
crates/apr-cli/src/commands/mono.rs:23 (`apr mono publish --dry-run`, so that
command can never actually publish), crates/aprender-train/src/config/cli/
extended.rs:125 (`--model-card`), and crates/aprender-train/examples/
finetune_test_gen.rs:352 (`--publish`).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…slash became hf://<path>
`apr chat /home/noah/models/qwen2.5-coder-0.5b-instruct.apr --offline` on the
crates.io 0.63.0 binary never looked at the file. It rewrote the absolute path
into a HuggingFace repo id and went to the network, which `--offline` did not
stop either:
$ apr chat /home/noah/models/qwen2.5-coder-0.5b-instruct.apr --offline
Downloading hf:///home...
Downloading model.safetensors...
error: Validation failed: Download failed: https://huggingface.co//home/resolve/main/model.safetensors: status code 404
rc=5
The same three fixtures (0.5B .apr, 1.5B q4k .apr, 0.8B .gguf) all produced
`Downloading hf:///home...`, and `apr run` opened all of them. Local models were
simply unreachable from `apr chat`.
Root cause: chat carried its own copy of the model-source resolution instead of
using run's. crates/apr-cli/src/commands/chat.rs:116 read
let hf_uri = if !resolved_source.contains("://") && resolved_source.contains('/') {
format!("hf://{resolved_source}")
with none of the local-path guards `run::resolve_model_source` has
(`!Path::new(..).exists()`, `!starts_with('/')`), so any argument containing a
slash became a repo id. chat.rs:126 then called
`resolve_model(&model_source, false, false)` with `offline` hard-coded to
`false`, which is why `--offline` could not stop the request.
chat now calls `run::resolve_model_source` and threads the real `--offline`
through to `resolve_model`, so a local path is a local path in both commands and
`--offline` refuses uncached repos locally.
The second half is in run.rs. `resolve_model_source` decided correctly that an
existing relative path like `models/tiny.gguf` was local, then handed it to
`pull::resolve_hf_model`, whose `normalize_hf_uri` re-prepends `hf://` to any
slash-bearing scheme-less argument — undoing the decision. That call is now made
only for arguments that are already `hf://` references. `apr run` had the same
defect for relative paths and is fixed by the same line.
After, with a release binary built from this branch:
$ apr chat /home/noah/models/Qwen3.5-0.8B-Q4_K_M.gguf --offline </dev/null
=== Model Chat (GGUF Format) ===
Model: /home/noah/models/Qwen3.5-0.8B-Q4_K_M.gguf
Loaded GGUF format in 0.27s (532.5 MB)
Loaded tokenizer with 248320 tokens
rc=0
$ apr chat /home/noah/models/qwen2.5-coder-1.5b-instruct-q4k.apr --offline </dev/null
Loaded APR format ... rc=0
$ apr chat /home/noah/models/qwen2.5-coder-0.5b-instruct.apr --offline </dev/null
Loaded APR format in 0.47s (991.9 MB)
error: Invalid APR format: No Qwen tokenizer found. ...
rc=4
The third still exits non-zero, but for an honest reason: the file is opened and
read, and that fixture has no sidecar tokenizer.json. Two adjacent behaviours are
also checked, because a resolver fix can easily overshoot: a nonexistent absolute
path still reports `File not found: /home/noah/models/does-not-exist.apr` (rc=3)
rather than a 404, and `--offline` on an uncached repo now prints
`OFFLINE MODE: Model hf://... not cached` (rc=5) with no request made.
Falsifiers in crates/apr-cli/src/commands/chat_local_path.rs assert behaviour,
not shape: an absolute path resolves to itself, a slash-bearing relative path
resolves to itself, and `--offline` on an uncached repo fails with OFFLINE MODE
without contacting huggingface.co.
Mutation check — restoring the pre-fix chat body (HEAD chat.rs:113-126) with the
tests kept turns all three RED:
panicked at chat_local_path.rs:28: an existing absolute path must resolve
without touching the network: ValidationFailed("Download failed:
https://huggingface.co//tmp/resolve/main/model.safetensors: status code 404")
panicked at chat_local_path.rs:59: an existing relative path must resolve
without touching the network: ValidationFailed("Download failed:
https://huggingface.co/apr-chat-rel-595528-ThreadId(20)/models/resolve/main/tiny.gguf: status code 401")
panicked at chat_local_path.rs:78: offline chat must refuse locally, got:
Validation failed: Download failed:
https://huggingface.co/apr-offline-falsifier-org/apr-offline-falsifier-repo/resolve/main/model.safetensors: status code 401
test result: FAILED. 16 passed; 3 failed
Reverting only the run.rs half (keeping the chat delegation) leaves the relative
path RED on its own, so both edits are load-bearing.
cargo test -p apr-cli --lib: 6625 passed, 0 failed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…t pass that printed "Ok"
Eight tolerance/threshold flags across five lint commands accepted `nan` and
silently disarmed the gate they configure. On apr 0.63.0 from crates.io, the
same DDP metrics pair that fails at the shipped defaults:
$ apr ddp-metrics-lint --metrics-1gpu-file ddp1.json \
--metrics-ngpu-file ddp4.json --world-size 4
scaling_efficiency : BelowThreshold { efficiency: 0.0025, threshold: 0.85 }
loss_parity : Divergence { rel_diff: 3.95, tolerance: 0.01 }
rc=5
$ apr ddp-metrics-lint ... --scaling-floor nan --loss-tolerance nan
scaling_efficiency : Ok { efficiency: 0.0025 }
loss_parity : Ok { rel_diff: 3.95 }
rc=0
The exit code is the smaller half of the damage. The report prints a positive
`Ok` next to the violating number — a statement that 0.25% DDP scaling
efficiency and a 3860% loss divergence were examined and found acceptable. A CI
log scraper reading that line gets the wrong answer with no indication anything
was skipped, and neither the text nor the `--json` report echoes the threshold
that was used, so the disarm is unauditable after the fact.
Reproduced on all five commands with four different observation shapes (JSON
object, two-file JSON pair, 4-D array, JSONL), and isolated to NaN/negative by
holding the body constant and sweeping the threshold: 0.95 FAIL, 99 FAIL, inf
FAIL, NaN PASS, -1 PASS. `abc` was already rejected by clap, so the parser
validated syntax but never domain.
Root cause is one mechanism repeated eight times, not eight bugs. Every gate is
`if observed > tolerance { fail }` or `if observed < floor { fail }` —
kv_timeline_classifier.rs:256 `up < threshold`, attn_parity_classifier.rs:91,97,
ddp_metrics_classifier.rs:114,146, plus the equivalents in attn_viz_classifier
and explain_token_classifier. IEEE-754 makes every comparison against NaN false,
so the failing branch is unreachable. Nothing anywhere validated the incoming
threshold — notable because the same classifiers already refuse to judge a
non-finite *observation* (`AttnParityNumericsOutcome::NonFiniteMaxAbsDiff`).
This adds the symmetric guard on the threshold side.
crates/apr-cli/src/commands/threshold_arg.rs is the single validator, used at
both layers: as a clap `value_parser` on all eight flags so a bad value is
rejected at parse time before any gate runs, and as a `guard()` at the top of
each lint `run()` so a caller that bypasses clap still fails closed rather than
printing `Ok`. Three domains — TOLERANCE (finite, >= 0), FRACTION ([0,1]),
COSINE ([-1,1]).
$ apr kv-timeline-lint --timeline-file kv.json --preempt-threshold=nan
error: invalid value 'nan' for '--preempt-threshold <FRACTION>': NaN is not
a threshold: every comparison against NaN is false, so the gate could never
fail. Expected a finite fraction in [0.0, 1.0]
rc=2
Two deliberate behaviour changes beyond NaN: `inf` and out-of-domain values are
now rejected too. Both previously "worked" in the sense of failing loudly
(`--preempt-threshold 99` reported `threshold: 99.0`), but neither can express a
bound, and an infinite tolerance disarms a max-style gate exactly as NaN does.
Explicit relaxation still works through legitimate values —
`--scaling-floor 0.0 --loss-tolerance 1e9` still exits 0.
Mutation check: with `reject_reason()` stubbed to `return None` and all tests
kept, the falsifiers go RED —
FALSIFY-CLI-THRESHOLD-NAN-001: `apr kv-timeline-lint --timeline-file
/tmp/.tmpZng1k4/kv.json --preempt-threshold nan` exited 0 — a NaN threshold
disarmed the preemption_trigger gate on a body that fails at the defaults.
stdout:
preemption_trigger : Ok
expected --scaling-floor named; got: ddp-metrics-lint loss-parity gate
rejected: Divergence { l1: 2.0, ln: 9.9, rel_diff: 3.95, tolerance: 0.01 }
clap accepted a gate-disarming threshold: ["apr", "kv-timeline-lint",
"--timeline-file", "kv.json", "--preempt-threshold", "nan"]
test result: FAILED. 0 passed; 5 failed (lib run() guards)
test result: FAILED. 2 passed; 7 failed (threshold_arg + clap parse)
test result: FAILED. 0 passed; 1 failed (cli_commands e2e)
Restored: 6641 passed, 0 failed (apr-cli --lib); cli_commands 10/10; the five
CRUX falsification suites (F-06, L-02, F-17, F-19, D-11) 52/52.
The e2e falsifier lives in tests/cli_commands.rs on purpose — it is one of the
few integration targets ci.yml actually runs, and it asserts the control case
(same body must still fail at the defaults) before asserting the disarm case,
so it cannot pass vacuously.
Refs #2391
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… a typo'd --backend ran the default backend Six defects found by dogfooding apr 0.63.0 from crates.io (audit epic #2373), all of them things the CLI reported as working while doing nothing. --stream emitted one NDJSON event per token whose text field was always the empty string. The ids were right — the terminal final event decoded them into "I'm here to help! How can" — so a consumer rendering text as events arrive saw nothing at all until the run finished, which is the entire point of the flag. Nothing ever decoded the ids one at a time. run_entry.rs now fills each event from the model's own tokenizer (single-token decode, the same thing the SSE streaming handler does), resolved once per streamed run and only when --stream asked for it. --trace-output FILE wrote "events": [] at every --trace-level, because the document was built by format! and ended in that literal. The same format! also interpolated the model path raw into a JSON string, so a path containing a quote produced a file that is not JSON. inference_result.rs now builds the document with serde_json and fills events from the timings the run measured (model_load from load_ms, generate from inference_ms) — nothing synthesised. Separately, --trace-level chrome ignored --trace-output entirely: it wrote trace-<epoch>.json into the CWD while the path the user named kept the summary stub, so a scripted consumer silently read the wrong file. print_chrome_trace now honours the path. --trace-level layer printed a table headed Time whose values were wall_ms / tokens * <fixed share> — the same 85/8/2/1.7 split for every model and every prompt, which is why TOKENIZE, EMBED and DECODE agreed to the hundredth of a millisecond in every run. It "proved" TRANSFORMER was 85% while the real [BRICK-PROFILE] block six lines above reported FFN 42% / Qkv 21% / LmHead 4.5% for the identical run, and printed 1.0 tok/s against the profiler's 19.2 tok/s with neither figure labelled. The values stay (they are the only estimate available without the brick profiler) but the table now says ESTIMATED, marks each value ~, prints the share it assumed, and labels TOTAL as wall clock including model load and RATE as end-to-end. --backend, --trace-level and -f accepted any string. --backend banana printed "Backend override: banana" and quietly ran the default backend — the exact outcome the --backend cuda guard exists to prevent, since it makes any throughput measured through the run meaningless. All three now carry clap value_parsers, on run and on chat. Every apr run --trace printed [CONTRACT WARN] gpu-decode-profiling-v1 TOKEN_ACCOUNTING: LmHead.count=10 != tokens_processed=11 and the gap was always exactly 1 (10/11, 11/12, 12/13, 16/17, 37/38). That is the generation loop, not a defect: generate_with_cache samples the final token and breaks at tokens.len() >= max_seq_len before feeding it back, so LmHead never fires for it while tokens_processed counts it. The invariant is now "one LmHead per forward pass" — tokens_processed or tokens_processed - 1 — so real miscounting still warns and healthy runs do not. Two error strings said things that were not true. An empty lm_head on qwen2.5-coder-0.5b-instruct.apr, a dense tied-embedding model with no experts at all, was reported as "likely a MoE per-expert tensor"; the message now names both known causes and says to run apr tensors to find the 0-byte tensor. apr chat's missing-tokenizer error printed the literal {stem}.tokenizer.json — an unsubstituted format placeholder inside a plain string — instead of the filename it looked for; it now prints concrete paths. Two existing tests were encoding defects and were fixed: run_tests_stream_output asserted v["text"].is_string(), which is true of "" and so passed for the whole life of the empty-text bug, and run_tests_chrome_trace asserted against a hand-maintained copy of print_chrome_trace's body kept in the test file, which could only prove the copy agreed with itself. Mutation-verified: each fix reverted with the tests kept turns its falsifier RED (8 tests across the two crates), restored turns them GREEN. Refs #2378 (partial), Audit epic: #2373 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… fail, a manifest of files that did not exist, and three flags with no effect Found by dogfooding apr 0.63.0 installed from crates.io. Six of the seven findings in #2380; the seventh (--assert-tps) is PR #2372. apr qualify --tier standard FAILED for every user who was not sitting in the aprender source checkout. The Contract Audit gate shelled out to `pv audit contracts/aprender/tensor-layout-v1.yaml` — a path resolved against the CALLER's cwd. From /tmp it reported "error: Failed to read contract file: No such file or directory (os error 2)" and exited 5; from the repo root the same model and tier PASSED. The contract ships with the source tree, not with the binary, so a missing file is a missing input, not a failed audit: qualify.rs now resolves it via APR_CONTRACTS_DIR or by walking up from cwd, and SKIPs with an actionable message when it is nowhere to be found. apr probar tensor printed a "Generated files:" manifest of .png paths and then wrote Netpbm .pgm bytes — every listed path was a file that did not exist, so a CI step copying them into probar fixtures failed with ENOENT. probar.rs:416 said "For now, write as .pgm" and :445 did `let _ = png_path;` to silence the unused variable. Rather than downgrade the promise, this adds a dependency-free 8-bit grayscale PNG encoder (png_encode.rs: IHDR + zlib-stored IDAT + IEND, CRC-32 and Adler-32 in ~110 lines) and the export writes real PNGs — `file` now reports "PNG image data, 256 x 100, 8-bit grayscale" and Pillow decodes it. The printed manifest and the export are now derived from one function, so they cannot drift again. Separately, `--format bogus` was swallowed by `.unwrap_or(ExportFormat::Both)` in dispatch_analysis.rs:85 and silently exported something else at exit 0; the FromStr error it already produced is now surfaced. apr qa passed two GPU gates vacuously on a CPU-only build. Capability Match asserted "Architecture 'qwen2': all 5 required ops supported by GPU" — a claim about kernels a non-cuda build has no path to — and PTX Parity reported "0/0 kernel pairs passed PTX parity", because without the cuda feature validate_all_kernel_pairs returns an empty report whose all_passed() (failed == 0) is vacuously true. Zero comparisons is not a pass. Both now SKIP, matching GPU Speedup and GPU State Isolation in the same report. apr bench --percentiles documents "Values must be in (0, 100]" and then accepted 0 and 101, emitting `"latency_p101_ms": null` into the CRUX-E-07 JSON report — a consumer saw a plausible metric key instead of an argument error. Non-numeric input was already rejected by the value parser, so the range check belongs there too; bench::run also checks, for non-clap callers. apr eval --device accepted arbitrary values and had no effect in perplexity mode: --device cpu, cuda and bogus produced byte-identical perplexity with no mention of a device anywhere in the output. The only readers of `device` are the humaneval/mbpp paths. --device is now restricted to cpu|cuda by clap, the effective device is reported in the header and the JSON, and asking for cuda prints why it was not honoured. apr showcase --step bogus answered "No step specified. Use --auto-verify or --step <step>" — the unknown-value branch fell through to the not-specified branch because both mapped to None. It now names the offending value and lists the available steps. Mutation check: reverting all six fixes and keeping the tests turns 14 tests RED, including test_locate_contract_returns_none_off_tree ... panicked at 'must not invent a path that does not exist' test_every_listed_generated_file_actually_exists ... panicked at 'Png: listed /tmp/.tmpxBAi54/layer_000_block_0.png but it was never written' test_dispatch_analysis_probar_rejects_unknown_format ... panicked at 'error must name the rejected value, got: File not found: ...' ptx_gate_skips_when_no_kernel_pairs_were_compared ... panicked at '0/0 comparisons must be reported as SKIP' capability_gate_makes_no_gpu_claim_without_cuda ... panicked at 'a CPU-only build must not assert GPU support, got: Architecture 'qwen2': all 5 required ops supported by GPU' parse_percentile_rejects_out_of_range_points ... panicked at 'point outside (0, 100] must be rejected: 0.0' perplexity_device_notice_fires_... ... panicked at 'cuda must be called out' test_showcase_unknown_step_names_the_offending_value ... panicked at 'error must quote what the user typed, got: Validation failed: No step specified. Use --auto-verify or --step <step>' The capability-gate test was itself vacuous on the first pass — its GGUF fixture had no metadata, so the gate short circuited on "missing architecture metadata" and never reached the GPU claim. It now writes a general.architecture KV and asserts the SKIP message, which is what proves the fixture engaged. test_export_by_format_png_creates_pgm_only asserted the defect (that a .pgm existed) and was rewritten to assert the .png the command advertises. 6651 apr-cli lib tests pass; fmt and clippy -D warnings clean. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Conflict in crates/apr-cli/src/commands/pull.rs: this branch moved the "=== APR Pull ===" banner down past the dry-run early return so `--json` output stays clean, and main (#2416) inserted the offline enforcement scope at the same point. Both kept, with the scope established before `resolve_hf_model`, which is the first thing on that path that can reach the network. The lib compiled after the textual merge but the TESTS did not: main's `pull_run_refuses_uncached_model_when_offline` falsifier calls `pull::run` with the pre-`--json` arity. Passing `false` for the new parameter - that falsifier asserts the refusal, not the rendering. Both fixes verified end-to-end against a release binary built from the merge, not from either side alone: apr pull qwen2.5-coder --dry-run --json exit 0, stdout parses, keys: mode, model, offline, resolved, revision, revision_kind apr pull --offline hf://hf-internal-testing/tiny-random-gpt2 (empty cache) exit 10, "Cannot fetch ... in --offline mode", 0 files written 283 apr-cli pull/offline unit tests pass; fmt clean. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Conflict was two test-module declarations appended at the same line of crates/aprender-serve/src/api/tests/mod.rs - main's sse_stream_whitespace (#2367) and this branch's router_flags. Both kept. That file is now the third shared manifest this audit has collided on, after crates/apr-cli/src/commands/pull.rs and the same tests/mod.rs on #2429. A single physical line that every PR in a crate must append to serialises otherwise-independent work: the changes never actually conflict, only their insertion point does. Both modules compile and their tests pass together. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Conflict in crates/apr-cli/src/commands/chat.rs. Both sides rewrote the same model-resolution block, and this one needed reading rather than picking a side. main (#2416) added a FileNotFound fast-fail for absolute / ./ / ../ paths that do not exist, then kept the old resolution: alias lookup followed by `format!("hf://{resolved_source}")` for anything containing a slash, and threaded `offline` into ModelSource::parse. this branch (#2387) replaced the whole block with `resolve_chat_model`. Took this branch's resolver, because it strictly supersedes main's block rather than merely conflicting with it: `resolve_chat_model` carries main's FileNotFound fast-fail verbatim, threads `offline` through both `resolve_model_source` and `resolve_model`, and drops the hf:// rewrite that is the defect this branch exists for. Keeping main's version would have restored `apr chat /path/to/model.apr` resolving to `hf:///path`. Confirmed no local from main's block is referenced after the conflict region - `fully_resolved_source`, `hf_uri`, `model_source`, `source_str` and `looks_like_path` are all consumed inside it. 250 apr-cli chat unit tests pass; build and fmt clean. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Same one-line conflict in crates/aprender-serve/src/api/tests/mod.rs, now against native_routes_2376 which arrived with #2429. All three modules kept - sse_stream_whitespace, native_routes_2376, router_flags - and their 24 tests pass together. This is the FOURTH time this audit has collided on a shared manifest line (pull.rs once, this file three times). The changes never actually conflict; only their insertion point does, because every PR in the crate must append to one physical line. Worth a structural fix if the audit's remaining PRs keep hitting it, though generating the module list would cost more than the 30 seconds each resolution takes. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…d nothing
Dogfooding `cargo install aprender` 0.63.0 found that most of the observation-file
lint family could not fail. `apr unified-search-lint --observation-file` printed
[PASS] offline (FALSIFY-CRUX-A-23-001): rows=0 expected_count_ok=false sources_ok=0
and exited 0 — the `_ok` fields were `expected_count.is_some()` ("an expectation was
supplied") and `expected_sources.len()` (a count wearing a boolean's name), so a
section that supplied no expectation merged rows, compared nothing and was recorded
as a discharged falsifier. `apr otlp-lint --otlp-file body.json`, the documented
invocation, ran zero of its three opt-in checks and exited 0 for any parseable JSON
including the scalar `42`. `apr nf4-lint` printed "codebook matches (16 entries)"
for four different ways of supplying no codebook, because the empty branch asserted
`NF4_CODEBOOK.len() == 16` — a tautology over a compile-time constant.
`apr typical-p-lint` exited 5 on `{"range":{"p":1.7}}` and 0 on `{"range":{"p":"1.7"}}`,
the identical violation quoted, reporting the present field as "(missing fields —
classifier skipped)". `apr hang-trace-lint --world-size 0` reported
`Ok { ranks_seen: 0 }` whatever the directory held. And `apr ollama-tools-lint
--response-file X` could not exit 0 for ANY response, because the allowlist gate its
help text called "Optional" ran unconditionally against an empty declared-tool set.
Root causes, one shape: a verdict derived from a variable that stays empty when
nothing was measured.
- unified_search_lint.rs:170 rendered `expected_count.is_some()` as `expected_count_ok`
and :146 skipped the count assertion whenever `as_u64()` returned None, so a
string-typed expectation silently disarmed the gate. `--json` doubled down with
`"passed": true` next to that same outcome string.
- otlp_lint.rs:38-48 made all three checks conditional and returned Ok(()) at :79
when none was selected.
- nf4_lint.rs:139 `if expected.is_empty() { NF4_CODEBOOK.len() == 16 }`, with
`read_f32_array` folding wrong-type and misspelled-key into the same empty Vec.
- typical_p_lint.rs / tool_use_lint.rs / gbnf_lint.rs / dry_sampling_lint.rs used
`?`-chains where every `as_f64()?` / `as_array()?` failure means "skip the gate",
and treated an all-skipped run as success.
- hang_trace_classifier.rs:75 short-circuits at world_size 0 by design; nothing
stopped a user (or an unset `${WORLD_SIZE}`) reaching it.
- ollama_tools_lint.rs ran `classify_tool_name_allowlist` against `Vec::new()`
whenever `--request-file` was absent.
The fix follows #2420's shape: VACUOUS is a verdict distinct from PASS and FAIL,
and "no gate reached a verdict" routes to it with a non-zero exit. A new shared
`commands/lint_vacuity.rs` carries the `Verdict` enum, an `assert_not_vacuous`
guard and a `skipped_label` that distinguishes "section absent" from "PRESENT BUT
UNUSABLE". Sections that are present but wrong-typed are now schema errors, not
skips. The outcome strings no longer contain `_ok=` fields that mean something
other than what they say.
Measured against the fixed binary (debug build of this branch):
before [PASS] offline …: rows=0 expected_count_ok=false sources_ok=0 rc=0
after [VACUOUS] offline …: VACUOUS: section supplies neither expected_count
nor expected_sources, so the 0 merged row(s) were compared against
nothing — a gate that asserts nothing cannot pass rc=1
before apr otlp-lint --otlp-file <{} | [] | 42 | false | "str"> rc=0
after otlp-lint: VACUOUS RUN — no gate was selected … rc=5
before apr nf4-lint on {"codebook":{}} → "codebook matches (16 entries)" rc=0
after [VACUOUS] codebook …: supplies no "expected" array (keys present: []) rc=1
before apr ollama-tools-lint --response-file ot_one.json → NoDeclaredTools rc=5
after --request-file is required in non-streaming mode rc=5
(and with the flag, rc=0 — the first input that can pass at all)
before apr hang-trace-lint --trace-dir td --world-size 0 → Ok{ranks_seen:0} rc=0
after --world-size 0 is not a world size … rc=5
Positive controls all still pass: a fully-specified unified-search observation is
`[PASS] … rows=1 expected_count=1 expected_sources=1`, an armed otlp span gate on a
good body is rc=0, a matching 16-entry NF4 codebook is rc=0, and the ollama
allowlist still catches a hallucinated tool name.
Four shipped tests asserted `is_ok()` on inputs that should fail and so held the
defects in place; each is rewritten and the old name recorded in a doc comment:
nf4_lint `codebook_default_passes_when_empty_expected` (its comment called the
tautology intended behaviour), tool_use_lint `empty_object_passes_no_gates` and
`empty_object_json_mode_passes`, otlp_lint `empty_json_object_runs` (which
discarded the result entirely), plus the earlier `empty_object_passes*` in
gbnf/dry-sampling/typical-p. `ollama_tools_lint::malformed_response_errors` was
passing for the wrong reason once the missing-flag check landed ahead of the parse,
and now pins `InvalidFormat`.
Mutation check — reverted each decision (kept every test): `assert_not_vacuous`
forced to `Ok(())`, `unified_search_lint::run_gate` forced to treat every Err as a
pass, the otlp and hang-trace guards disabled with `if false &&`, the nf4 tautology
branch restored, and the ollama empty-allowlist path restored. RED:
test result: FAILED. 223 passed; 29 failed
failures:
commands::dry_sampling_lint::tests::falsifier_empty_object_is_rejected
commands::dry_sampling_lint::tests::falsifier_unrelated_body_is_rejected
commands::dry_sampling_lint::tests::falsifier_wrong_typed_sibling_does_not_suppress_a_real_violation
commands::gbnf_lint::tests::falsifier_empty_object_is_rejected
commands::gbnf_lint::tests::falsifier_int_legal_mask_does_not_suppress_the_masking_gate
commands::gbnf_lint::tests::falsifier_null_observation_is_rejected
commands::gbnf_lint::tests::falsifier_unrelated_body_is_rejected
commands::gbnf_lint::tests::falsifier_wrong_typed_finish_reason_is_rejected
commands::hang_trace_lint::cov_tests::falsifier_world_size_zero_is_rejected_not_silently_passed
commands::lint_vacuity::tests::falsifier_empty_observation_is_vacuous
commands::lint_vacuity::tests::falsifier_present_but_unrunnable_section_is_an_error
commands::lint_vacuity::tests::falsifier_scalar_observation_is_vacuous
commands::lint_vacuity::tests::falsifier_unrecognised_keys_are_vacuous
commands::lint_vacuity::tests::falsifier_unrunnable_section_fails_even_when_a_sibling_ran
commands::nf4_lint::tests::falsifier_codebook_with_nothing_supplied_is_vacuous
commands::nf4_lint::tests::falsifier_wrong_typed_codebook_is_a_schema_error
commands::ollama_tools_lint::cov_tests::falsifier_non_streaming_without_request_file_names_the_missing_flag
commands::otlp_lint::cov_tests::falsifier_no_gate_flags_is_a_vacuous_run
commands::tool_use_lint::tests::falsifier_empty_object_is_rejected_in_both_output_modes
commands::tool_use_lint::tests::falsifier_section_name_typo_is_rejected
commands::tool_use_lint::tests::falsifier_wrong_typed_section_is_rejected
commands::typical_p_lint::tests::falsifier_empty_object_is_rejected
commands::typical_p_lint::tests::falsifier_quoted_p_carries_the_same_violation_as_numeric_p
commands::typical_p_lint::tests::falsifier_typo_in_section_name_is_rejected
commands::unified_search_lint::tests::dedup_gate_wrong_source_fails
commands::unified_search_lint::tests::falsifier_json_report_never_marks_a_vacuous_gate_passed
commands::unified_search_lint::tests::falsifier_section_with_no_expectations_is_vacuous_not_pass
commands::unified_search_lint::tests::falsifier_wrong_typed_expectation_is_a_schema_error
commands::unified_search_lint::tests::offline_gate_wrong_count_fails
Restored: 6685 passed; 0 failed. `cargo fmt --all -- --check` and
`cargo clippy -p apr-cli --lib -- -D warnings` clean.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… green report
`apr cbtop --headless --simulated --ci` printed "Status: FAIL | CI: red" and
exited 0. A pipeline step that exists to fail a build could not fail one unless
the caller also remembered to pass an explicit --throughput or --brick-score
number; the report's own verdict was never consulted.
Worse, `--iterations 0` was accepted. With zero measurement iterations every
brick keeps zero samples, so its measured time is 0.0µs, its gap factor 0.00x
and its score a perfect 100/A. The shipped 0.63.0 binary answered the same
threshold pair two ways:
--iterations 100 -> rc=5, "FAIL - Brick score 96 < threshold 100"
--iterations 0 -> rc=0, "PASS - Brick score 100 >= threshold 100"
"Falsification: 7/7 passed Status: PASS | CI: green"
with the measurement count as the only variable. Appending `--iterations 0` to
any cbtop gate made it unconditionally green, and the fabricated numbers were
valid JSON, so a machine consumer could not tell either.
Root causes:
crates/apr-cli/src/commands/cbtop_report_tui.rs:10 — check_ci_thresholds()
only ever set passed=false from config.throughput_threshold /
config.brick_score_threshold. It never read report.status or
report.ci_result. Both headless paths (gguf.rs:40, cbtop_measure_batch.rs:395)
route through this one function, so both were blind.
crates/apr-cli/src/extended_commands.rs:455 — `iterations` had no value
parser, so 0 reached the measurement loop. `--warmup abc` was already
rejected, so the parser validated the field's type but not its domain.
crates/apr-cli/src/commands/cbtop_get_cpu_memory.rs:95 and
cbtop_measure_batch.rs:457 — both spliced a live time-of-day into a
string-literal date ("2026-01-11T..." and "2026-01-12T..."), so every report
ever written, including files persisted with --output for CI provenance, was
stamped seven months stale, and one run could stamp two different days.
crates/apr-cli/src/extended_commands.rs:437,440 — --json and --output both
document "requires --headless" but carried no clap constraint. The flag was
silently dropped and cbtop entered the interactive TUI, so an interactive
user who asked for JSON got a full-screen UI and CI got a raw-mode errno
that said nothing about the actual mistake.
After, against a release binary built from this tree (apr 0.63.0 8cc3aaf):
cbtop --headless --simulated --ci --iterations 0 --brick-score 100 --throughput 900
rc=2 error: invalid value '0' for '--iterations <ITERATIONS>':
must be at least 1 - a zero-iteration run measures nothing ...
cbtop --headless --simulated --ci --iterations 50
rc=5 Status: FAIL | CI: red
cbtop: FAIL - report status FAIL (CI: red), falsification 2/7 passed
cbtop --headless --simulated --json
report: 2026-08-10T17:43:33Z now: 2026-08-10T17:43:33Z
cbtop --json rc=2 error: the following required arguments were not provided: --headless
cbtop --output r.json rc=2 error: the following required arguments were not provided: --headless
cbtop --headless --simulated --iterations 1 rc=0 (smallest honest run still served)
Two pre-existing tests had encoded the defects and went red, correctly:
test_ci_threshold_only_throughput_set built a report with status "FAIL" /
ci_result "red" and asserted check_ci_thresholds returned true. Its fixture
is now green so it isolates what it is actually about (with only a throughput
threshold set, the failing brick score is not consulted), plus a companion
assertion that the same report with a red verdict fails.
test_chrono_timestamp_format asserted ts.starts_with("2026-01-12T"), which
locked the hardcoded date in place. It now brackets the call with a
before/after UTC date so a midnight rollover cannot flake it.
Mutation check - each fix reverted in turn with the tests left in place:
iterations guard removed:
test_run_rejects_zero_iterations panicked "cbtop accepted --iterations 0"
test_parse_cbtop_rejects_zero_iterations panicked "cbtop accepted --iterations 0 at parse time"
and the test output printed the forged report it prevents:
RmsNorm 100 (A) - 0.0us / 1.5us (0.00x) ... Falsification: 7/7 passed
Status: PASS | CI: green
ci_result check removed:
test_ci_gate_fails_on_red_report_without_explicit_thresholds panicked
"cbtop --ci returned pass for a report it had already judged FAIL/red"
test_ci_threshold_only_throughput_set panicked at !check_ci_thresholds(&red, &config)
hardcoded date restored:
test_chrono_timestamp_is_current_utc_date panicked
"timestamp 2026-01-12T17:39:12Z carries neither 2026-08-10 nor 2026-08-10"
test_simulated_report_timestamp_is_current_utc_date panicked likewise,
proving both headless paths now share the one helper
requires="headless" removed:
test_parse_cbtop_json_requires_headless panicked
"cbtop --json was accepted without --headless"
Restored, all 6662 apr-cli lib tests pass.
One consequence worth stating plainly: `make showcase-ci` now exits 5. The
--simulated pipeline jitters each brick +/-20% around its budget, so roughly
half land over budget and the report is genuinely red. It printed "CI
validation passed" for as long as it has existed only because the exit path
ignored the verdict. The target is not referenced by any workflow, so main is
unaffected; the Makefile now says so.
Refs #2397
Audit epic: #2373
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… byte killed the server
Driving the shipped 0.63.0 `apr mcp` over stdio, the canonical scripted
invocation loses its answer:
printf '<initialize>\n<tools/call apr.version>\n' | apr mcp
-> exit 0, ONE response line (id=1), the id=2 tool result never written
3/3 deterministic, stderr empty. `tools/call` runs on a worker thread that
writes its own response (server.rs spawn_tools_call_worker), while
initialize/tools/list answer inline on the read loop. The loop returned the
instant stdin closed without joining those workers, so the process exited
while a worker still owed the client a reply. Exit 0 with no output is
indistinguishable, to a client, from a tool that produced nothing — the
failure mode is silent. Holding stdin open was the only working client
pattern and was documented nowhere.
Second transport defect, same loop: a single invalid UTF-8 byte on stdin
terminated the whole server.
rc=1, "error: Aprender error: mcp server: stream did not contain valid UTF-8"
Every request after the bad byte was lost. `BufRead::lines()` surfaces a bad
byte as an io::Error, and `let line = line?` propagated it out of the loop.
The server already handled well-formed-UTF-8 malformed JSON correctly
(-32700, keep serving), so it disagreed with itself about what a malformed
message costs.
Both now: lines are read as BYTES and decoded per line, so a bad line is a
malformed MESSAGE answered with -32700; and EOF joins every in-flight worker
before returning.
Four protocol-layer defects alongside them:
* `initialize` hard-errored -32602 on any protocolVersion other than the
exact literal "2024-11-05" — including OLDER dated versions, so it was not
a floor check but string inequality. The MCP lifecycle makes negotiation a
proposal: a server answers with a version it supports and lets the client
decide. Claude Code and Cursor propose 2025-03-26 / 2025-06-18, so every
client this server advertises itself to (README.md:25) could never connect.
* `ping` returned -32601. It is base protocol, not an advertised capability;
a keepalive client reads the error as a dead server and restarts it.
* Valid JSON missing `jsonrpc` or `method` was reported -32700 Parse error
with id:null, losing the id the client needs to correlate — while a WRONG
`jsonrpc` VALUE was already correctly -32600 with the id echoed. Now both
are -32600 with the id echoed; genuinely malformed JSON stays -32700.
* A JSON-RPC batch array leaked serde's "invalid type: map, expected a
string at line 1 column 1", naming neither batching nor arrays and
pointing at a '[' that is valid JSON. Now declined by name.
And one that made clients loop: a required argument supplied with the wrong
JSON type reported "Missing required argument: model_path" for an argument
the client had plainly sent. Absent, empty and wrong-type were byte-identical,
so an LLM told the argument was missing retries by adding a key it already
sent. `tools::args::require_str` now keeps the two apart, applied at all 8
subprocess wrappers.
Root causes, all in crates/aprender-mcp/src/server.rs at 0.63.0: the read
loop (`for line in stdin.lock().lines()` / `let line = line?`, no worker
join on EOF), handle_initialize's -32602 early return, the absent `ping`
arm, and `serde_json::from_str::<JsonRpcRequest>` mapping a missing field to
a parse error.
run_stdio is now a two-line binding over a new generic `serve_stream<R, W>`.
That is not cosmetic: FALSIFY-MCP-010 and -011 live in the read loop, not in
request handling, so no `handle_request` test can see either — and invalid
UTF-8 cannot even be expressed as a `&str` input. CI's workspace-test runs
`--lib` plus an explicit allowlist that does not include this crate's
tests/ directory, so an integration-only guard for a P0 would never run.
Making the loop generic puts six falsifiers for it in the `--lib` target CI
actually executes.
MUTATION CHECK. Reverted each fix, kept the tests, rebuilt the binary so the
harness could not reuse a fixed one:
serve_stream_answers_tools_call_before_returning_on_eof
tools/call response (id=2) was DROPPED at EOF; got 1 response(s): [...]
serve_stream_answers_every_pipelined_tools_call
id=2 unanswered; got 1 of 6
serve_stream_survives_invalid_utf8_line
serve_stream must not propagate an error out of the session:
stream did not contain valid UTF-8: invalid utf-8 sequence of 1 bytes
falsify_mcp_007_protocol_version_mismatch_negotiates_down
proposing 1999-01-01 must not abort the handshake, got:
Some(JsonRpcError { code: -32602, ... })
wrong_type_names_the_type_and_never_says_missing
left: "Missing required argument: model_path"
right: "Argument model_path must be a string, got number"
Two tests did NOT survive that check and were fixed rather than shipped:
* falsify_m1's FALSIFY-MCP-007 asserted the defect — it required -32602 on
any mismatch, so the gate was actively holding the handshake broken.
Rewritten with its contract entry.
* A tools/call "ordering variant" in the new integration file PASSED against
the unfixed binary: serializing the large tools/list response gives the
worker time to finish before EOF, so it could never observe the drop it
claimed to guard. Removed — a test that stays green on the defect is
theater.
validate.rs's nonstring_model_path_returns_error asserted only
`is_error == Some(true)`, which is shape; it passed while the message
contradicted the request. Strengthened to assert the text.
Contract gains FALSIFY-MCP-010 through -014 (apr-mcp-server-v1.yaml), and
the loader test's hardcoded nine-gate count becomes a named constant.
Verified end-to-end against a binary built from this tree, using the issue's
own repros: all four proposed protocolVersions negotiate to 2024-11-05, ping
pongs, the two missing-field cases are -32600 with id echoed, the batch array
is named, the wrong-type argument names the type, the tools/call result
arrives 3/3 on EOF, and the bad-byte stream answers ids 1 and 2 plus a
-32700 with rc=0.
Fixes #2393
Audit epic: #2373
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…uccess
`apr train apply --task pretrain --config c.yaml` on a 200-row CSV printed one
line to stderr — `Warning: Unsupported data format 'csv', using demo data` — then
ran two epochs at a constant loss of 0.010000, wrote a 21 MB checkpoint, printed
`DONE Pre-training completed` and exited 0. The 0.010000 is the synthetic fixture
value; the user's corpus was never opened. A JSONL dataset and a well-formed JSON
array of records produced the byte-identical fabricated loss. In CI that is a
green training job for a model that saw none of your data.
The loader had three demo-data fallbacks and its own doc comment was wrong:
crates/aprender-train/src/config/train/batches/loader.rs:15 claimed "Supports
parquet, JSON, and CSV formats via alimentar" while the match handled only
parquet and json and fell through to `create_demo_batches` for everything else.
json.rs did the same on a parse failure, parquet.rs on unrecognised columns.
All four now return `Error::ConfigError` naming the dataset and quoting the
schema the loader actually wants, and `create_demo_batches` is deleted outright
so nothing can reach for it again.
Four tests asserted `is_ok()` on inputs that should fail — they encoded the
defect and would have blocked this fix. They are rewritten to assert the error.
Two more P0s in the same family:
`apr train apply --task pretrain` aborted with exit 101 on any tabular dataset
whose input width differed from its target width — a 3-feature / 1-target
regression set, the commonest tabular shape there is. Sweeping six (input,
target) pairs, it survived only when input_dim == target_dim: (1,1) and (2,2)
and (4,4) trained; (2,1), (3,1) and (1,2) hit a raw `assert_eq!` in
`MSELoss::forward` after "Starting training..." had printed. Tabular mode drives
the generic Trainer with an identity forward, so the widths must match;
`validate_tabular_batch_shapes` now says so before any training runs.
`apr finetune <model>.apr --task classify` discarded the file the user named and
handed its PARENT DIRECTORY to `ClassifyPipeline::from_pretrained`, which scans
for any SafeTensors it can find (finetune.rs:2183). Dropping one unrelated 4.6 MB
safetensors next to a 0.5B .apr, changing nothing else, flipped the run from "No
SafeTensors files found" to loading the sibling's 27 tensors. The stale comment
said `from_apr()` was unavailable in entrenar 0.7.5; entrenar is in-tree at
0.63.0 and `Transformer::from_apr` exists. Had the sibling's dims matched it
would have fine-tuned the wrong weights and exited 0.
Also in this cluster:
- `apr train apply -o DIR` was documented with a default of /tmp/training-output
and silently discarded; only training.output_dir in the YAML was honoured, and
its destination was never created, so a completed run was thrown away at the
save step with a bare "No such file or directory (os error 2)". -o now
overrides the YAML and the directory is created before saving.
- `apr tune --rank R` echoed "Requested rank: R" and reported recommended_rank
256 for every R in {4, 8, 16, 64, 256, 1024}: the recommendation was a pure
function of --vram. `plan_with_rank` pins the rank and derives alpha,
trainable params, memory and the rank-aware LR from it.
- `apr train plan --format` was declared `_format: &str` and never read; text,
json, yaml and an invalid value produced byte-identical text with exit 0.
yaml now renders the manifest; an unknown value is rejected.
- `apr runs ls --status completed` matched 0 of 2941 completed runs because the
filter stringified the stored variant (`Success`) and compared it verbatim,
and `--status bogusvalue` returned an empty table with exit 0.
- `apr tune --method bogus` and `apr train sweep --strategy bogus` fell through
to Auto and to a RANDOM search respectively, printing the typo back as though
it were valid. Both now reject, matching finetune/distill/prune.
- `apr pretrain` printed "OK CONVERGED" without ever comparing final_val_loss
against the target: final 3.0000 against a target of 0.001 reported CONVERGED
with exit 0, and the JSON report carried neither the target nor a verdict.
- `apr finetune <model>.safetensors --task classify` replied "No model path or
--model-size provided" when the path was the first positional argument.
- `apr grad-norm` reported a malformed JSON telemetry file as "Invalid APR
format"; it never touches a model. New `CliError::InvalidInput`, same exit 4.
Mutation-verified: with every fix reverted and every test kept, the new
falsifiers turn RED, then GREEN on restore.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…erve path was not Ollama-compatible at all
Serving a real 1.04 GiB Q4_K GGUF with `apr serve run ... --context-length 128`,
0.63.0 answered GET /realize/model with five values it had never looked at:
{"format":"gguf","size_bytes":0,"quantization":"Q4_K_M","context_length":4096,
"lineage":{"uri":"pacha://default:latest","version":"1.0.0",
"content_hash":"blake3:0blake3:0blake3:0blake3:0blake3:0blake3:0blake3:0blake3:0blake3:0blake3:0blake3:0blake3:0blake3:0blake3:0blake3:0blake3:0"}}
Every one was a constant. `apr inspect` on the same file says 1.04 GiB and
context_length 32768, the server had been started with --context-length 128, and
the content_hash is `"blake3:0".repeat(16)` — a 128-character string shaped
exactly like a BLAKE3 digest, which a consumer cannot tell from a real one and
will store and compare as provenance. That in the release whose theme is
provenance.
The rule now is: every field is measured or absent. api/model_source.rs holds a
ModelSourceInfo whose accessors all return Option; the handlers omit what they do
not know instead of substituting something plausible. Size and container format
come from the file (magic bytes, not the extension — an .apr file no longer
reports "gguf"), quantization from the qtype of the loaded projection tensors
(GGUF's general.file_type is advisory and goes stale on requantize),
architecture from the loader, and the context the server was CONFIGURED with is
reported separately from the model's own advertised maximum — collapsing those
two is what produced the 4096. `lineage` appears only when a hash was actually
computed over the model bytes, which today means never, so it is absent.
after: {"format":"gguf","size_bytes":1117320768,"quantization":"Q4_K",
"context_length":128,"model_max_context_length":32768,
"architecture":"qwen2","loaded":true}
POST /realize/reload answered 501 with "Start server with --registry flag".
There is no --registry flag: `apr serve run --registry <FILE>` exits 2 with
"unexpected argument '--registry' found", and `apr serve --help` has none
either, so the endpoint was unreachable by any documented invocation and the
message sent the caller down a dead end. It now says the CLI does not expose
registry mode, names the embedder API that does, and gives the working
alternative.
The same server was also not an Ollama server. PMAT-928 wired NDJSON streaming
and /api/tags onto the apr-cli APR-CPU router; `apr serve run <GGUF>` mounts a
different one — realizar's create_router, via run_cpu_server — and that one was
never fixed. /api/tags, /api/show and /api/version returned 404 while the
startup banner advertised "Ollama-Parity Endpoints", and every Ollama client
calls /api/tags before it will issue a single chat request. `stream:true` was
parsed off the request and thrown away: the reply was one buffered
application/json object with content-length 190, so Open WebUI, continue.dev and
the ollama CLI showed a frozen cursor for the whole generation.
Both are fixed on that router. stream:true now answers application/x-ndjson with
no content-length: one `{...,done:false}` object per fragment, concatenating to
the full text, terminated by exactly one `{...,done:true,done_reason:"stop",
prompt_eval_count,eval_count}`. A Go client built on ollama's own ChatResponse
type (CreatedAt time.Time) decodes it end to end:
content-type: application/x-ndjson; charset=utf-8
content-length header present: false
objects=7 done_reason="stop" eval_count=7
assembled="The capital of France is Paris."
This router's chat backend has no token callback, so the fragments are cut from
the finished generation — the wire protocol is now correct and clients no longer
hang, but time-to-first-object is unchanged. stream:false is byte-identical to
before, asserted by a no-regression test.
Root causes: crates/aprender-serve/src/api/realize_handlers_embed_completion.rs:173-183
(the five constants and the synthetic hash) and :204 (the --registry advice);
crates/aprender-serve/src/api/ollama_handlers.rs:185-186 (stream hardcoded false)
and crates/aprender-serve/src/api/router.rs:108-109 (only /api/chat and
/api/generate registered).
One test encoded the defect rather than the requirement and was rewritten:
to_chat_request_maps_messages_and_options asserted `!req.stream` under the
message "Ollama path always drives chat non-streaming". The internal chat path
is still driven non-streaming — that part is true and still asserted — but the
old wording read as a licence to discard the client's flag, which is the bug.
The assertion now says it is about the internal path only, and points at the
stream falsifiers for the wire behaviour.
Mutation check. Reverting all four fixes and keeping the tests turns 10 of the 12
HTTP falsifiers RED:
api_tags_is_routed_and_lists_a_model_the_client_can_ask_for ... FAILED
/api/tags 404'd in 0.63.0; every Ollama client calls it first
api_chat_with_stream_true_returns_ndjson_terminated_by_done ... FAILED
stream:true must be framed as NDJSON, got content-type "application/json"
realize_model_never_emits_a_synthetic_content_hash ... FAILED
synthetic content hash still on the wire: {"context_length":4096,"format":"gguf",
...,"content_hash":"blake3:0blake3:0...","size_bytes":0}
realize_reload_501_does_not_advertise_a_nonexistent_flag ... FAILED
the 0.63.0 dead-end advice is still on the wire: Hot-reload requires registry
mode. Start server with --registry flag.
test result: FAILED. 2 passed; 10 failed
Of the two survivors one was the intended control (stream:false must stay a
single JSON object). The other,
api_generate_stream_true_is_ndjson_multi_chunk, was a hole: one buffered object
IS one parseable NDJSON line with done:true, so only the framing distinguishes
fixed from broken. It now asserts content-type and the absence of
content-length, and goes RED under the stream mutation alongside its /api/chat
twin. Separately, replacing split_inclusive with split in content_fragments
turns the reassembly falsifiers RED, so they are not merely counting chunks.
Fixes #2402
Refs #2396 (partial) — findings 2 and 3 fixed on the realizar router; /api/embeddings
and /api/embed left out, because that path needs real embeddings on the quantized
backend (realize_embed_handler requires state.model, which is None for GGUF) and
routing them would only produce a well-formed error.
Audit epic: #2373
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ath appears Under the old parent-directory scan the .safetensors path appeared in the error too — it was the file that got scanned — so a contains-the-path assertion alone stayed GREEN with the defect restored. Caught by the mutation check, not by review. Refs #2374 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… and exited 0 (#2392) `apr convert model.safetensors -o out.apr --quantize int8` located the sibling tokenizer, parsed all 151665 tokens, printed that it had done so, then wrote an APR with no tokenizer in it and reported "Conversion successful" with exit 0. `apr run out.apr` then died with rc=8 and "APR format requires self-contained tokenizer" — the command produced an artifact that violates the format contract the same binary enforces at load time. It was systematic in the quantization scheme: q4k embedded the tokenizer in 4/4 outputs, int8/int4/fp16/no-quant in 0/7, on the same input file. Root cause: crates/aprender-core/src/format/converter/f16_convert.rs:370 `save_model_tensors_with_config` — the fallback save path taken whenever no sibling config.json is found, i.e. every scheme except q4k, which returns early into `save_model_tensors_q4k`. It took no `tokenizer` argument at all, so `save_output` had nowhere to pass the one it had resolved. Embedding the tokenizer got `apr run` one step further and it then failed on "C-01: APR model missing 'architecture' metadata", and after that on "C-03: APR model missing 'num_heads' metadata". Both are on the same path and both are the same class of defect, so both are fixed here: the save path now stamps the architecture it can already infer from the tensor names (the import path uses that exact helper), and `load_model_config_from_json` now understands the Pacha cache's `<hash>.config.json` layout — the layout `tokenizer_loader.rs` in the same binary has understood all along, announcing it on stderr as "[BUG-TOK-002] Found tokenizer at Pacha cache path". Every value `apr run` demanded (num_attention_heads: 4, num_key_value_heads: 2, rope_theta, rms_norm_eps) was sitting in a file next to the weights the whole time. Measured on /home/noah/.cache/pacha/models/064a3693fa1ea02c.safetensors: before: apr convert … --quantize int8 → exit 0, "Conversion successful" apr run out.apr --prompt Hi → rc=8 "[PMAT-172] ERROR: APR file missing embedded tokenizer." after: apr convert … --quantize int8 → exit 0 "[#2392] Found model config at Pacha cache path: …064a…config.json" "[PMAT-113] Embedding 151665 vocabulary tokens into APR metadata" apr run out.apr --prompt Hi → rc=0, generation completes Six further defects from the same cluster: - `apr export` reported `original_size` as the sum of dequantized F32 tensor bytes while printing it directly beside `exported_size`, a real on-disk size. It read an identical 9714528 for five input APRs spanning 1.37 MB to 30.5 MB; a true 3x shrink was reported as slight growth. It is now the input file's size, which the same command's `--plan` mode already reported correctly and which the raw APR→GGUF passthrough path already used. (converter/apr_export_fn.rs:47) - `apr quantize --plan -s q4k` returned a constant 7.111x reduction ratio for every model ever passed to it — a flat 4.5 bits/weight against an assumed-F32 input. On a real 87 MB model it promised 12778677 bytes where quantization produced 55507012, 4.34x optimistic; on a small model whose weights q4k actually inflates it still promised a 7x shrink. Q4K skips embeddings, norms, biases, scales and sub-super-block tensors, and pads each row up to a multiple of 256, so the answer depends on the tensor inventory. New `q4k_output_size_estimate` walks the input's tensor index — no tensor data loaded — applying the same `should_quantize_tensor` predicate and the same per-row 144-byte super-block arithmetic both q4k write paths use. 87 MB model: 12778677 → 55207716 against an actual 55507012 (0.5% error). Small model: a promised 7.111x shrink → a predicted 0.18x inflation, matching the direction of the real 0.159x. int8/int4/fp16 keep the flat model — measured against real conversions they are accurate (4.0 vs 3.98, 8.0 vs 7.05, 2.0 vs 2.0) — so this change is q4k-only. (commands/quantize.rs:99) - `apr export`, `merge`, `shard` and `unshard` silently destroyed pre-existing output and had no `--force` flag at all, while `convert` and `quantize` in the same CLI refused the identical situation with exit 5 and "Use --force to overwrite". A 9-byte file handed to `apr export -o precious.safetensors` came back as 9717255 bytes of model, rc=0. All four now take `--force` and route their decision through one `refuse_overwrite` helper that convert and quantize also use, so the policy cannot drift again. shard guards on the weight-map index — the artifact that identifies a shard set — not on the directory existing, so sharding into a fresh or unrelated directory is unaffected. - A single-pair `kv_table` never closed its box. `apr convert … --quantize int8` prints exactly one config pair and rendered as top border, one row, a header separator, nothing. tabled promotes record 0 to a header and only draws the bottom border once a body row follows; a key-value table has no header, so multi-pair tables were only correct by accident. Dropping the header rule fixes every single-pair call site in the CLI. (output.rs:276) - `apr import /nonexistent.safetensors` failed correctly (rc=5) and then advised "verify the model name exists on huggingface.co/models" — a remedy that cannot apply to an absolute local path, and one the sibling commands do not give ("File not found", rc=3). `resolve_local_source` already encodes the distinction as `status: 0`; the message never read it. (converter_types_expectations.rs:352) - `apr shard --max-shard-size abc` reported "invalid number '': cannot parse float from empty string". The user typed `abc`; the message quoted an empty string they never typed and blamed an emptiness that is an artifact of our own unit-suffix split. (commands/shard/mod.rs:57) Falsifiers: 12 in crates/aprender-core/src/format/converter/tests/dogfood_2392.rs (findings 1, 2, 3, 6), 7 in crates/apr-cli/src/lib_dogfood_2392.rs (finding 4), 4 in crates/apr-cli/src/output_tests.rs (finding 5), 4 in crates/apr-cli/src/commands/shard/mod.rs (finding 7), 2 in crates/apr-cli/src/commands/quantize_quant_scheme.rs (finding 3 CLI wiring). Each asserts behaviour, and several guard the honest negative — no tokenizer supplied still means no tokenizer key, a real Hub 404 keeps its Hub remedy, a headed `table()` keeps its header separator, `--force` still gets through, and sharding into a directory with no index is not blocked. Mutation check — each fix reverted in place, tests kept, every falsifier RED: finding 1 tokenizer: "scheme 'none' produced an APR with 0 embedded vocabulary entries, expected 3" finding 1 arch: "the APR must declare an architecture — without it the loader refuses with 'C-01: APR model missing architecture metadata'" finding 2: left: 524288 right: 524676 finding 3: "still returned the flat bits-per-weight estimate (73756 bytes, 7.111x)" — left: 73756 right: 73756 finding 4: "expected the overwrite refusal, got: Validation failed: Invalid model format: Failed to parse APR file: InvalidHeader(file too small)" (export, merge, shard, unshard) and "existing output without --force must be refused: ()" finding 5: "a one-pair kv_table must terminate with ╰───┴───╯" showing the unterminated box finding 6: "Got: Resource not found (0): /nonexistent.safetensors. Fix: verify the model name exists on huggingface.co/models"; and "the Pacha <hash>.config.json layout must be recognised" finding 7: "the message must echo the value the user passed. Got: invalid number '': cannot parse float from empty string" Restored, all green: aprender-core 14070 passed, apr-cli 6672 passed, clippy --lib clean on both, fmt clean. Every finding also re-verified against a release binary built from this tree, not a unit test alone. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…one printed a number the input did not contain
`apr audio-inspect-lint` accepted a body declaring `sample_rate: 4294983296`,
printed `sample_rate : Ok { rate: 16000 }` and exited 0. 4294983296 is
2^32 + 16000, and `raw as u32` wraps, so the value aliased onto a canonical
rate and passed the gate while the report showed a number that was never in
the file. `--expected-sample-rate 16000` did not catch it either: after the
wrap the two were equal. `channels: 4294967297` aliased to 1 the same way.
`sample_rate: 4294967296` did fail, but reported `NonCanonicalRate { got: 0 }`
— again a value the observation did not contain.
before (apr 0.63.0, base 8cc3aaf) after (this branch)
sample_rate : Ok { rate: 16000 } sample_rate : SampleRateOutOfRange { got: 4294983296 }
rc=0 rc=5
Three more gates in the family reported PASS for sections that were absent,
empty, or of the wrong JSON type:
$ apr imatrix-lint --observation-file <(echo '{"leakage":{"calib_hashes":[1,2],"eval_hashes":[1,2]}}')
[PASS] leakage (FALSIFY-CRUX-B-07-001): disjoint (|calib|=0, |eval|=0) rc=0
Those two sets are identical. `filter_map(Value::as_str)` silently dropped
every integer-typed hash, leaving two empty sets, and the gate then printed a
cardinality claim about data it had not read. `{"leakage":"nonsense"}` produced
the same green line. `apr embeddings-lint` did it with `{"shape":{}}` —
input_len, hidden_size and data all defaulted to 0/0/[], so 0 == 0 and the
verdict was `Ok { n_rows: 0 }`. `apr hang-trace-lint --world-size 0`
short-circuited to `Ok { ranks_seen: 0 }` on an EMPTY trace dir, so a CI job
whose world size came from an env var that resolved to empty got a green gate
on no evidence at all. All three now fail: unreadable evidence is unknown, not
disjoint, not Ok. Genuinely disjoint hash sets, an explicitly empty
`{"input_len":0,"hidden_size":4,"data":[]}` response, and a populated trace dir
all still pass — the fix rejects absent evidence, not legitimately empty
evidence.
Separately, `apr quant-preservation-lint` across two models with different
tokenizers emitted 13,282,156 bytes over 19 lines, one of them 4,814,215
characters long (14,896,285 bytes under `--json`): it stored the untruncated
`Debug` of every diverged GGUF value, including whole 248k-entry
`tokenizer.ggml.merges` and `tokenizer.ggml.tokens` arrays. The actionable
signals in that same run — `general.architecture :: String("qwen35") →
String("qwen2")`, `tokenizer.ggml.eos_token_id :: Uint32(248046) →
Uint32(151645)` — were buried between three multi-megabyte dumps that wedge a
terminal and blow a CI log budget. Values over 200 chars are now summarised
with element count, a content digest and a head sample, plus the first
differing index; scalars still render verbatim, unchanged. The identical run:
before 13,282,156 bytes / 19 lines / longest line 4,814,215
after 2,446 bytes / 22 lines / longest line 338
--json 14,896,285 bytes -> 3,761 bytes
tokenizer.ggml.tokens :: ArrayString(len=248320, sha256=3751959a6d53cc72, head=ArrayString(["!", "\"", "#", …)
→ ArrayString(len=151936, sha256=77485efcba6e358f, head=ArrayString(["!", "\"", "#", …)
first differing element: index 280 (len 248320 vs 151936)
Root causes:
crates/apr-cli/src/commands/audio_inspect_classifier.rs:106,139 — `as u32`
after only a `raw <= 0` check
crates/apr-cli/src/commands/imatrix_lint.rs:166 — `and_then(as_array)
.map(filter_map(as_str)).unwrap_or_default()` on a possibly-absent,
possibly-non-string section
crates/apr-cli/src/commands/embeddings_lint.rs:117 — three `unwrap_or(0)` /
`unwrap_or_default()` defaults standing in for missing fields
crates/apr-cli/src/commands/hang_trace_classifier.rs:69 — `world_size == 0`
returned Ok, with a comment calling it "Degenerate"
crates/apr-cli/src/commands/quant_preservation.rs:100 —
`format!("{ref_val:?}")` stored whole, rendered whole at :201-208
`apr typical-p-lint --help` printed its only required flag with an empty
description, so its non-obvious observation schema had no route to a user.
Rather than fix the one string, the falsifier walks every `*-lint` subcommand
through clap's CommandFactory and asserts each flag carries help text. It went
red on a second instance the audit had not found — `embeddings-lint
--observation-file`, likewise blank in the shipped 0.63.0 help. Both are
documented now, and a new `*-lint` that forgets turns the test red.
Mutation check. Reverting each fix while keeping its tests, one at a time:
audio 8 failed Ok { rate: 16000 } vs SampleRateOutOfRange { got: 4294983296 }
Ok { channels: 1 } vs ChannelsOutOfRange { got: 4294967297 }
NonCanonicalRate { got: 0 } vs SampleRateOutOfRange { got: 4294967296 }
imatrix 4 failed [PASS] leakage: disjoint (|calib|=0, |eval|=0)
embeddings 4 failed [PASS] shape: Ok { n_rows: 0 }
hang-trace 2 failed Ok { ranks_seen: 0 } vs WorldSizeZero
quant 2 failed divergence report must stay bounded; got 5855530 bytes
help 1 failed lint commands with an undocumented flag: ["typical-p-lint --observation_file"]
Restored: 6686 passed, 0 failed.
The end-to-end before/after above was measured on two binaries whose embedded
git SHA proves which tree each came from — `apr --version` reported
`apr 0.63.0 (8cc3aaf)` for the base and this branch's own commit for the
fix — both built into a private target dir. That matters: the workspace target
dir is shared with three other worktrees, and the first "after" binary taken
from it turned out to be another worktree's build, byte-identical to the
"before" one and showing zero change.
Not fixed here, each deserving its own PR: the "CLI flag accepted" gates that
stamp PASS for argv the real clap parser rejects with exit 2 (finding 2), the
8 linters whose documented producer command is absent from the binary
(finding 3), the two incompatible `--json` envelopes (finding 6), and the
exit-code / "Invalid APR format" split (findings 8 and 9), which issue #2404
already owns.
Refs #2377 (partial) — remaining: 2, 3, 6, 8, 9
Audit epic: #2373
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…s and read by two
Every apr subcommand's --help lists "-q, --quiet Quiet mode (errors only)" and
"-v, --verbose Verbose output". In 0.63.0 neither flag did anything on almost
any of them. A user scripting "apr validate model.apr --quiet" to get a pass/fail
got 22 KB of tables; "apr hex model.apr --quiet" still wrote 303 972 bytes;
"apr gbnf-lint --observation-file f -q" still printed its full PASS report.
Measured with cmp -s on captured stdout, 14 of 16 sampled commands were
byte-identical with and without --quiet, and 13 of 16 with and without --verbose.
Root cause: crates/apr-cli/src/dispatch_run.rs:333 execute_command() read
cli.offline and cli.json and dropped cli.quiet and cli.verbose on the floor.
Both are clap globals, so clap advertises them everywhere, but consuming them
meant threading a "quiet: bool" parameter into each command's signature. Exactly
two commands were ever given one — list and lint — and those two are precisely
the two the audit found working. contracts/apr-list-quiet-wiring-v1.yaml fixed
that one command in 2026-04 and its own why_5 named this gap: "No contract
formalizes 'every global flag must materially affect every subcommand'".
Threading a parameter into 104 commands is the design that already failed; it is
the same forwarding bug --offline had, where three commands forgot to pass the
flag along. So --quiet is a latch, set once in execute_command, plus a crate-wide
shadow of println!/print! declared before "mod commands;" so all ~9 000 call
sites consult it. A command cannot disarm --quiet by forgetting to plumb a
parameter, because it never receives one. stderr and the exit code are untouched
("errors only" is now literally true), --json --quiet still prints the JSON
document, and the two commands with richer quiet semantics of their own opt out
via emitln! so "apr list --quiet" keeps printing one identifier per line
(F-LIST-QUIET-001) and "apr lint --quiet" keeps filtering to errors.
--verbose gets the same latch and reports what the dispatcher itself resolved
rather than inventing chatter for 104 commands: the model paths extract_model_paths
pulled out, their size on disk, and which of the three contract-gate outcomes
applied. That last line also answers the audit's separate observation that
--skip-contract has "zero effect" on inspect/validate/tensors: there is nothing
for it to skip on a command the PMAT-237 gate exempts, and the CLI now says so
instead of staying mute.
Before and after, same binary invocation, stdout bytes (0.5B APR / 0.8B GGUF):
base --quiet --verbose base --quiet --verbose
inspect 1529 -> 0 1529 -> 1651 6282 -> 0 6282 -> 6404
tensors 29878 -> 0 29878 -> 30000 28393 -> 0 28393 -> 28515
validate 2855 -> 0 2855 -> 2977 22858 -> 0 22858 -> 22980
tree 25794 -> 0 25794 -> 25916 24270 -> 0 24270 -> 24392
hex 303972 -> 0 303972 -> 304094 690273 -> 0 690273 -> 690395
debug 771 -> 0 771 -> 893 923 -> 0 923 -> 1045
explain/flow/oracle/gpu likewise go to 0 under --quiet and gain the preamble
under --verbose; oracle's own +34 verbose bytes survive alongside it. Errors are
unaffected: "apr inspect /nope/missing.apr --quiet" still prints "error: File not
found" and exits 3, and --quiet on the corrupt GGUF still exits 5. "-q -v"
together, previously accepted with no error and no effect, now resolves to quiet.
Mutation check. Removing the latch from execute_command and keeping the tests:
---- verbosity::tests::quiet_and_verbose_reach_a_command_that_never_receives_them
--quiet must suppress the PASS report as its own help text promises
(`Quiet mode (errors only)`); `apr gbnf-lint -q` still printed:
gbnf-lint report for /tmp/apr-2401-414795.json
json: Ok
diagnostic: (missing fields — classifier skipped)
masking: (missing fields — classifier skipped)
Disabling only the verbose preamble instead:
assertion `left != right` failed: --verbose must not be a byte-for-byte no-op
left: "gbnf-lint report for /tmp/apr-2401-obs-1001521.json\n json: Ok..."
right: "gbnf-lint report for /tmp/apr-2401-obs-1001521.json\n json: Ok..."
That second RED only appeared after fixing the test. The first draft gave each
child process a pid-stamped observation file, so "normal" and "verbose" differed
by the temp path alone and the assert_ne! passed with --verbose disabled — a test
that would have locked the defect in. All three children now share one path, so
the comparison is a byte comparison, which is the methodology the audit used.
The falsifier runs the real execute_command end to end in child processes,
because the level is a process-wide latch that deliberately cannot be un-set —
the same pattern commands::offline uses for the same reason. gbnf-lint is the
audit's own repro and needs only a small JSON file, so it stays hermetic.
contracts/apr-global-verbosity-wiring-v1.yaml generalises the single-command
ancestor to every subcommand and binds the five falsifiers (pv validate: 0 errors,
0 warnings; Falsify 1.00).
Fixes #2401
Audit epic: #2373
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…the user did not ask for
Dogfooding crates.io 0.63.0 found that apr profile accepted its own documented
flags and then discarded them, so the numbers it printed described a run nobody
requested.
--warmup and --measure were parsed and dropped on the floor. dispatch_profile
called profile::run with fifteen positional arguments and simply omitted these
two, so every non-CI profile ran the hardcoded 3 warmup / 10 measurement passes.
The JSON body then reported "warmup_passes": 3 / "measure_passes": 10, actively
misinforming a consumer that had overridden them. The falsifying measurement is
wall time: --measure 1 and --measure 100 took 3.10s and 3.14s.
before (0.63.0) after
$ apr profile M --warmup 7 --measure 9
Running 3 warmup passes... Running 7 warmup passes...
Running 10 measurement passes... Running 9 measurement passes...
| Warmup | 3 passes | | Warmup | 7 passes |
| Measure | 10 passes | | Measure | 9 passes |
$ time apr profile M --measure 1 3.10s -> 2.26s
$ time apr profile M --measure 100 3.14s -> 20.67s
--focus matched a snake_case keyword table against the CamelCase names the brick
profiler emits. Lowercased, "upprojection" does not contain "up_proj" and nothing
at all contains "matmul". --focus mlp kept 1 of 4 FFN operations, --focus matmul
returned an empty table with exit 0, --focus attention dropped OutputProjection
(11% of decode), and --focus embedding captured RopeEmbedding while missing
LmHead. The surviving row was then renormalised over the filtered subset and
printed as "100.0%" of a model whose own Category Summary put FFN at 70.5%.
$ apr profile M --focus mlp
before: GateProjection 100.0% (1 of 4 rows)
after: GateProjection 21.7% DownProjection 20.9%
UpProjection 20.4% Activation 1.6% (4 rows, 64.6% of total)
$ apr profile M --focus matmul
before: (no hotspot rows), exit 0
after: 6 rows: Qkv/Gate/Down/Up/Output projections + LmHead
The unit tests guarding this fed synthetic hotspots literally named "up_proj" and
"down_proj", which the keyword table did match, so they passed for two releases
while no real name ever did. The new falsifiers use the names the profiler
actually emits. --focus with an unrecognised value now errors instead of silently
returning the full unfiltered report.
--ci assigned the mean to both percentiles, so p50 always equalled p99 and
--assert-p99 was asserting on a single averaged sample; a tail-only regression
could not fail the gate. The per-pass distribution was already being computed —
the non-CI path on the same model prints p50=120.2 p95=132.6 p99=136.3 — CI mode
just did not read it.
$ apr profile M --ci --assert-p99 90
before: p50 105.96 / p99 105.96, exit 0
after: p50 83.36 / p99 92.86, FAIL latency_p99, exit 5
--json is a global flag that apr bench honours; apr profile parsed it and printed
the human table anyway. Worse, --format json wrote three human progress lines to
stdout ahead of the JSON body, so the stream did not parse at all. Progress is
status, not data, and now goes to stderr.
$ apr profile M --format json | python3 -c 'import json,sys;json.load(sys.stdin)'
before: JSONDecodeError: Expecting value: line 1 column 1
after: parses; keys model, architecture, num_layers, ..., hotspots
--fail-on-naive printed "not yet implemented. Flag ignored." and returned 0
unconditionally, which is an exit-code contract a CI job may already depend on.
--threshold, documented as the GFLOPS floor for naive detection, was consumed by
`let _ = naive_threshold;` — --threshold 100000 against a run achieving 22 GFLOPS
still reported "No obvious naive implementations detected". Both now drive a
single verdict, evaluated on the unfiltered results so --focus narrows the report
and not the gate.
$ apr profile M --detect-naive --fail-on-naive --threshold 100000
before: "Warning: --fail-on-naive is not yet implemented", OK, exit 0
after: NAIVE? achieved 28.0 GFLOPS < --threshold 100000.0 GFLOPS, exit 5
Two smaller ones on the same report. The roofline block printed
"Hardware: Unknown Unknown (24 cores, 512)" — detect_cpu hardcoded both strings
and the SIMD width carried no unit — beside the peak GFLOPS and bandwidth that
justify its MEMORY BOUND verdict. And every single run emitted
"[CONTRACT WARN] gpu-decode-profiling-v1 TOKEN_ACCOUNTING: LmHead.count=10 !=
tokens_processed=20", because the CPU profiler set tokens_processed to
prompt_len * passes while each measurement pass decodes exactly one token. The
tool was violating its own instrumentation contract on the happy path, training
users to ignore a real warning.
Hardware: Unknown Unknown (24 cores, 512)
-> AMD Ryzen Threadripper 7960X 24-Cores (24 cores, 512-bit SIMD)
[CONTRACT WARN] ... LmHead.count=10 != tokens_processed=20 -> gone
Root causes:
crates/apr-cli/src/validate.rs:443 warmup/measure not forwarded
crates/apr-cli/src/commands/profile_pct_change_classify.rs:158 snake_case table
crates/apr-cli/src/commands/profile_print_hotspot.rs:5 renormalised %
crates/apr-cli/src/commands/profile.rs:168 mean as p50+p99
crates/apr-cli/src/commands/diff_benchmark_report.rs:36 let _ = threshold
crates/apr-cli/src/commands/profile_safetensors.rs:147 set_tokens(len*n)
crates/aprender-compute/src/hardware/mod.rs:272 vendor/model "Unknown"
Mutation check: each fix reverted in turn with the tests kept.
focus table -> keyword-only:
--focus mlp dropped UpProjection; kept ["GateProjection"]
--focus attention dropped OutputProjection; kept ["QkvProjection", "AttentionScore"]
--focus matmul returned an empty hotspot table
--focus embedding dropped LmHead; kept ["RopeEmbedding", "Embedding"]
the focused table must still sum to the FFN share of total (~68.2%), got 23.2%
percent -> renormalise over printed rows:
the focused table must still sum to the FFN share of total (~68.2%), got 100.0%
CI percentiles -> mean:
CI p99 must be the measured tail, got 105.96
--assert-p99 130 must FAIL when the measured p99 is 136.3ms
option resolution -> unwrap_or(default):
assertion failed: matches!(resolve_output_format("human", true), Ok(OutputFormat::Json))
assertion failed: resolve_output_format("jsonn", false).is_err()
assertion failed: resolve_focus(Some("bogus")).is_err()
threshold -> let _ =, cpuinfo vendor -> never resolved:
--threshold 100000 must flag a 22.1 GFLOPS run
left: (None, Some("AMD Ryzen 9 7900X 12-Core Processor"))
right: (Some("AMD"), Some("AMD Ryzen 9 7900X 12-Core Processor"))
hardware label -> vendor+model unconditional, bare bit width:
left: "AMD AMD Ryzen Threadripper 7960X 24-Cores (24 cores, 512)"
right: "AMD Ryzen Threadripper 7960X 24-Cores (24 cores, 512-bit SIMD)"
Restored: 6671 apr-cli lib tests green, 30 aprender-compute hardware tests green.
One pre-existing test went red and was corrected rather than accommodated: the
prior draft of the focus table had dropped the "mm" keyword, which
test_filter_matmul_mm_keyword legitimately relies on for custom instrumentation
names. The keyword list is now the fallback only, consulted for names outside the
exact-name table, so "mm" is safe there.
End-to-end verified with a release binary built into a private CARGO_TARGET_DIR:
the shared /mnt/nvme-raid0/targets/aprender is written concurrently by sibling
worktrees, and the first verification run silently exercised another agent's
binary that still rejected nothing for --focus bogus.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… run
`apr run` on a GGUF model opened with our issue tracker. On the 1.5B qwen2
fixture, a default-features build printed this to stderr before a single
token came out:
Backend: wgpu (Vulkan)
[PMAT-333] Dequantizing 28 layers (hidden=1536, heads=12/2, intermediate=8960)
Dequantized layer 7/28
...
[PMAT-333] Dequantized 337 weights, 6174.9 MB F32
[apr-cpu-vs-gpu-output-parity-v1] wgpu path rejected, attempting fallback: cosine vs CPU = 0.884301 (< 0.99) at step 1/3
A user cannot look up PMAT-333, and `apr-cpu-vs-gpu-output-parity-v1` is a
filename in our contracts directory. Both lines carry information the user
genuinely needs — the machine is about to spend six gigabytes, and the GPU
result was rejected — and both delivered it as a ticket reference.
Same run now:
Backend: wgpu (Vulkan)
Preparing GPU weights: dequantizing 28 layers to F32
Dequantized layer 7/28
...
GPU weights ready: 337 tensors, 6174.9 MB F32
warning: GPU (wgpu) path rejected, attempting fallback: cosine vs CPU = 0.884301 (< 0.99) at step 1/3
The split throughout is: a message the user can act on stays unconditional and
is written in English; a message that only means something to whoever holds the
ticket moves behind APR_DEV_TRACE. The model geometry (hidden=, heads=,
intermediate=) is developer detail and is now held back; the F32 footprint is
not, and stays.
Four sites, by root cause:
crates/aprender-serve/src/gpu/adapters/wgpu_adapter.rs:28,161 — two
unconditional eprintln! tagged [PMAT-333], on the path every wgpu `apr run`
takes. Now built by dequant_start_message / dequant_done_message, which are
pure and therefore testable on the exact string the user sees.
crates/aprender-serve/src/infer/gguf_gpu_generate.rs:8,19 — the CUDA and wgpu
fallback tags. These exist for a good reason: #1428 made the GPU-rejection
decision visible without --verbose so a broken GPU can never ship silent
gibberish. #1429 then pinned that by asserting the tag equals the contract ID,
which pinned the wrong half — it locked the leak in place and would have
failed on any fix. The tests now assert what #1428 actually bought (an
unconditional warning naming the rejected backend) and were rewritten, not
loosened.
crates/aprender-serve/src/infer/inference_result.rs:475,512,590,598 — the
[F2-VALIDATION] lines. Three are reworded; the fourth reported a benign
argmax near-tie on a run where the GPU was ACCEPTED, said nothing actionable,
and is now a developer trace.
crates/aprender-serve/src/cuda/executor/kv_scatter.rs:530 — debug_attention_trace
had no gate at all: for layer 0 and the first three tokens of every CUDA run it
dumped raw tensor floats under [PAR-058-ATTN], and to produce them it
synchronized the compute stream and copied Q, K, V and both full KV caches
back to the host, inside the decode path. The gate is now the first term of
the predicate, so with tracing off none of that work happens.
The gate lives in a new crates/aprender-serve/src/dev_trace.rs, outside any
cfg(feature = "cuda"), so the CUDA decision predicate is unit-testable on a
machine without a GPU.
Contracts: apr-cpu-vs-gpu-output-parity-v1.yaml's PREDICTION lines are updated
to the new wording (its historical discharge records still quote the old
wording verbatim and are left as recorded). cuda-q4k-frozen-teacher-v1.yaml
FT-Q4K-TEACHER-001 greps `PMAT-333.*Dequantizing` to assert the dequant line is
ABSENT — after this rename that grep returns 0 on a log that does contain the
line, i.e. a falsifier that cannot fail. Its pattern now matches both spellings.
Both contracts pass `pv validate`.
Mutation check — reverted all four fixes, kept the tests, 7 of 12 went RED,
each printing the original defective string:
dev_trace.rs:73: layer 0 / token 1 must not emit an attention trace without APR_DEV_TRACE
wgpu_adapter.rs:325: start line still addresses the user in ticket numbers: [PMAT-333] Dequantizing 28 layers (hidden=1536, heads=12/2, intermediate=8960)
wgpu_adapter.rs:354: done line still addresses the user in ticket numbers: [PMAT-333] Dequantized 337 weights, 6174.9 MB F32
inference_result.rs:1142: internal falsifier ID leaked into user output: [F2-VALIDATION] GPU forward FAILED at probe position 3/17: ...
gguf_gpu_generate.rs:1143: the fallback message addresses the user in a contract ID: [apr-cpu-vs-gpu-output-parity-v1] CUDA path rejected
gguf_gpu_generate.rs:1143: the fallback message addresses the user in a contract ID: [apr-cpu-vs-gpu-output-parity-v1] wgpu path rejected
gguf_gpu_generate.rs:1196: [apr-cpu-vs-gpu-output-parity-v1] CUDA path rejected
test result: FAILED. 5 passed; 7 failed
The 5 survivors are the ones that should survive: the diagnostic still fires
when a developer asks for it, and the F2 message still carries its underlying
error. Restored: 12 passed, 0 failed.
The kv_scatter change is behind cfg(feature = "cuda"), which a default check
never compiles. Proved the check reaches it by injecting a deliberate type
error at the edited line: `cargo check -p aprender-serve --lib --features cuda`
then failed with E0308 at kv_scatter.rs:536. Removed, re-checked green.
aprender-serve --lib: 15505 passed, 0 failed. cargo fmt --all --check and
clippy -p aprender-serve --lib -D warnings both clean.
Refs #2405, #2373
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…e 0-byte lm_head placeholder at face value Qwen2.5-Coder-0.5B sets tie_word_embeddings=true, so its .apr records lm_head.weight as a descriptor carrying the full [151936, 896] shape and ZERO bytes of data: the matrix is stored once, as model.embed_tokens.weight. Two loaders read that descriptor literally, and neither could load the file. apr run, before: $ apr run /home/noah/models/qwen2.5-coder-0.5b-instruct.apr --prompt "What is 2+2?" error: Inference failed: Invalid shape: matmul weight has EMPTY data buffer (in_dim=896, out_dim=151936, qtype=0); likely a MoE per-expert tensor was registered with len-0 data — see aprender#1789 The error names the wrong cause: this model is not MoE. The empty buffer is the lm_head placeholder, registered as the output projection. apr run, after (same binary, tie resolved, 291 tensors, 24 layers): [#2309] lm_head is a 0-byte tied-embedding placeholder — tying output projection to 'model.embed_tokens.weight' [GH-175] OwnedQuantizedModel::from_apr: 24 layers loaded in 1532.6ms Output: <8 tokens decoded, no error> Transformer::from_apr — the loader behind apr finetune --task classify — failed one step earlier on the same file: error: Shape mismatch for 'lm_head.weight': expected 136134656 elements, got 0 read_tensor_as_f32 dequantizes the 0-byte descriptor into an empty Vec<f32>, which then fails shape validation. Root cause, one file convention with two consumers: crates/aprender-serve/src/gguf/loader_apr_quantized.rs:347 — the LM head was loaded by name only. New apr_lm_head_is_tied()/apr_load_lm_head() (same file, :111/:126) detect a 0-byte (or absent) descriptor and re-register the embedding matrix as the head. Both are row-major [vocab, hidden], which is exactly the layout the logits matmul wants (in_dim=hidden, out_dim=vocab), so the same bytes are reused with no transpose and no second copy — matching what the SafeTensors path already does in resolve_lm_head_weight (safetensors_infer_convert.rs:211). crates/aprender-train/src/transformer/model.rs:222 — new resolve_tied_lm_head() (:267) drops the placeholder before validation. Transformer already implements the tie: from_params maps a missing lm_head.weight to lm_head: None, and forward, forward_hidden, lm_head_weight and lm_head_weight_slice all fall back to embed_tokens.weight. An empty lm_head with no usable embedding matrix is left in place so a genuinely broken file still fails. Falsifiers assert behaviour, and negative controls keep the fix from becoming "always tie": an .apr with a real lm_head must keep its own weights. Mutation check — reverted each fix with the tests in place. aprender-train, Self::resolve_tied_lm_head call commented out: test falsify_2441_from_apr_resolves_tied_lm_head_placeholder ... FAILED #2441: a tied-embedding .apr must load, not fail shape validation: ConfigError("Shape mismatch for 'lm_head.weight': expected 64000 elements, got 0") test result: FAILED. 2 passed; 1 failed aprender-serve, apr_load_lm_head call replaced by the original apr_load_quantized_tensor(&["lm_head.weight", "output.weight"]): test test_2309_tied_lm_head_placeholder_decodes ... FAILED #2309: decode must not fail with 'matmul weight has EMPTY data buffer': InvalidShape { reason: "matmul weight has EMPTY data buffer (in_dim=8, out_dim=10, qtype=0); likely a MoE per-expert tensor was registered with len-0 data — see aprender#1789" } test test_2309_omitted_lm_head_ties_to_embeddings ... FAILED #2309: an .apr with no lm_head descriptor is tied, not broken: FormatError { reason: "APR: tensor not found (tried: lm_head.weight, output.weight)" } test result: FAILED. 1 passed; 2 failed Restored: 3 passed in aprender-train, 3 passed in aprender-serve. Two things this does NOT fix, both verified separately rather than assumed. The 0.5B .apr still decodes incoherently after loading. That is not the tie: a fresh apr convert --quantize fp16 of the same SafeTensors materializes a real lm_head (291 tensors, no placeholder anywhere) and garbles identically, while the SafeTensors themselves answer "4" and a Q4_K .apr of the 1.5B answers "4" through this same OwnedQuantizedModel path. The remaining defect is in the F16/BF16 APR path and is independent of tied embeddings. apr finetune --task classify still never reaches Transformer::from_apr: on this tree load_classify_pipeline (crates/apr-cli/src/commands/finetune.rs:2181) still passes the .apr file's PARENT DIRECTORY to from_pretrained, so it loads whatever sibling SafeTensors sits next to the named file — here a 1.5B, which then fails as "expected 136134656 elements, got 233373696" (151936 x 1536). That is #2374 finding 3 / #2436, not merged here. Fixing the tie is what that fix uncovers. Closes #2309. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…-map narrated a kernel map for a model apr refuses to run Three defects found dogfooding the 0.63.0 crates.io install (#2399, epic #2373). All three still reproduced on main at d16c608. `apr ptx` is listed in `apr --help` on every build, but the build `cargo install aprender` produces cannot run it. Every invocation shape failed the same way: $ apr ptx tiny.ptx error: Aprender error: ptx command requires --features full rc=1 `--features full` is a compile flag naming the wrong thing: `ptx` needs aprender-explain (the `trueno-explain` alias), while `full` also drags in CUDA, training and training-gpu. Worse, the feature check ran AFTER path resolution, so $ apr ptx /nonexistent.ptx error: File not found: /nonexistent.ptx rc=3 sent the user off to fix a path for a command that binary can never run. Now the feature is checked first, the command is gated on the crate it actually needs, and the facade exposes a minimal `ptx` feature so the advertised remedy is real: $ apr --help | grep '^ ptx ' ptx PTX analysis and bug detection [unavailable in this build: cargo install $ apr ptx /nonexistent.ptx error: Feature not enabled: apr ptx needs the PTX analyzer, which this build does not include. Reinstall with it: cargo install aprender --features ptx rc=9 The remedy was verified, not assumed: $ cargo run --bin apr --release --features ptx -- ptx tiny.ptx === PTX Analysis: foo === Registers: f32: 0 f64: 0 ... Bottleneck: MEMORY-BOUND rc=0 `apr ptx-map` printed a complete dense-transformer dispatch map for Qwen3.5-0.8B-Q4_K_M.gguf — RoPE, GQA attention, SwiGLU, "= 290 launches", exit 0 — for an architecture the same binary refuses to instantiate: $ apr check Qwen3.5-0.8B-Q4_K_M.gguf error: Validation failed: Failed to create model: Format error: Architecture 'qwen35' uses SSM/Gated Delta Net layers which are not yet supported ... The refusal lived inline in QuantizedGGUFTransformer::from_gguf (crates/aprender-serve/src/gguf/transformer.rs:130), so only callers that built a transformer ever saw it. It is now `unsupported_architecture_reason` in the same file, asked by both surfaces, and ptx-map answers: $ apr ptx-map Qwen3.5-0.8B-Q4_K_M.gguf error: Validation failed: Architecture 'qwen35' uses SSM/Gated Delta Net layers which are not yet supported for inference. ... (ptx-map maps the kernels that would run; this model has no runnable kernel path) rc=5 `apr check` on the same file still prints its original message verbatim. `--kernel BOGUSKERNEL` used to print an empty table and exit 0, which reads as "this model launches no kernels"; it now names the mistake and lists the kernels. Every path in ptx-map's Source column pointed into `trueno-gpu/src/kernels/`, a tree APR-MONO deleted, and 3 of the 6 leaves were wrong even after correcting the crate prefix (layernorm.rs is a directory, rope moved under elementwise/, and activation.rs never existed). The table is now keyed to the file that defines each kernel struct — crates/apr-cli/src/commands/ptx_map.rs:99 — and step 10 names FusedSwigluKernel, the kernel that exists, rather than SwigluKernel, which does not: 10 FusedSwigluKernel SwiGLU 8960 -> 8960 crates/aprender-gpu/src/kernels/elementwise/swiglu.rs Falsifiers assert behaviour, not shape. source_paths_resolve_to_the_defining_file walks both the decode and prefill sequences and requires each Source path to open and to contain `struct <KernelName>`. The two tests it replaces asserted the dead `trueno-gpu/...` strings literally, so they were green for the entire time the column was useless. Mutation check — restore the three original behaviours, keep the tests: test ...::source_paths_resolve_to_the_defining_file ... FAILED BatchedSwigluKernel: source column points at trueno-gpu/src/kernels/activation.rs, which does not exist test ...::unknown_kernel_filter_is_rejected_and_lists_alternatives ... FAILED a filter matching no kernel must be an error, not an empty table: () test tests::ptx_without_analyzer_reports_the_feature_not_the_path ... FAILED the missing path is not the reason ptx failed, got: File not found: /nonexistent_ptx_source_xyz.ptx test tests::ptx_without_analyzer_reports_remedy_for_kernel_form_too ... FAILED got: Aprender error: ptx command requires --features full test gguf::transformer::...::qwen35_gated_delta_net_tensors_are_refused ... FAILED a GGUF carrying ssm_ tensors must be refused test result: FAILED. 11 passed; 5 failed (apr-cli) test result: FAILED. 1 passed; 2 failed (aprender-serve) Restored: apr-cli 6658 passed / 0 failed, aprender-serve 15500 passed / 0 failed. Not fixed here, both observed while verifying finding 2 and worth their own issue: ptx-map still derives the quantization label from the FILENAME (rename a file to *-Q8_0.gguf and it reports Q8_0; `apr inspect` reports Q6_K where ptx-map reports Q4_K), and num_kv_heads comes from a hardcoded `num_heads == 28 -> 4, == 12 -> 2` heuristic while GGUFConfig::num_kv_heads is right there. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…pr code -p "hi" --model X` ran a different model
`apr code` honoured options only before the prompt. Written after it they
disappeared into the prompt text with no diagnostic — including `--model`, so
the user got whatever auto-discovery picked instead of the model they named, and
a misspelled flag produced no parse error at all.
$ apr code -p hi --model /nonexistent.gguf # before
Model: Qwen3-Coder-30B-A3B-Instruct-Q4_K_M.gguf (auto-discovered)
$ apr code -p hi --model /nonexistent.gguf # after
error: File not found: /nonexistent.gguf
$ apr code -p hi --totally-bogus-flag-xyz # before: no error, ran the 30B
$ apr code -p hi --totally-bogus-flag-xyz # after
error: unexpected argument '--totally-bogus-flag-xyz' found
Root cause: `#[arg(trailing_var_arg = true)]` on the `prompt` positional at
crates/apr-cli/src/commands_enum.rs:719, and the same attribute on the batuta
surface at crates/aprender-orchestrate/src/main_cli.rs:414. Dropped in both. A
prompt that genuinely starts with `-` now needs `--`, which clap already
suggests.
Two more silent-accept defects in the same command.
`--resume <unknown-id>` started a fresh session and exited 0. In `-p` mode
`resume` was never read at all — cmd_code returns from the non-interactive
branch before reaching the resume block (crates/aprender-orchestrate/src/agent/code.rs:208
before this change) — so a typo'd id left the user believing a conversation was
being continued that the model had no history of. The session is now resolved up
front, before any model is launched, and a valid id in `-p` mode actually
replays its stored messages and appends the new turn back to it.
$ apr code --model d47927c5638f80ab.gguf --resume deadbeef-not-a-session -p hi
before: Hello! How can I assist you today? rc=0
after: error: --resume: no such session "deadbeef-not-a-session" rc=1
`--project /typo` was skipped by an `if ... && project.is_dir()` guard, so the
agent ran against the current directory while the operator believed it was
scoped elsewhere. It now fails closed.
before: Hello! How can I assist you today? rc=0
after: error: --project: not a directory: /nonexistent-dir-xyz rc=1
Finally, the qwen2.5-coder-0.5b APR fixture could not run at all: apr serve
answered HTTP 500 "matmul weight has EMPTY data buffer ... likely a MoE
per-expert tensor" — on a dense Qwen2. That model ties its embeddings, so the
exporter wrote lm_head.weight with the full [151936, 896] shape and zero bytes,
and apr_load_quantized_tensor (crates/aprender-serve/src/gguf/loader_apr_quantized.rs:25)
picked that placeholder by name without looking at its length. Zero-length
candidates are now skipped and the head falls back to the embedding matrix,
which carries the identical row-major [vocab, hidden] layout; the fallback says
so on stderr instead of happening silently.
$ apr run qwen2.5-coder-0.5b-instruct.apr --prompt "say hi" -n 8
before: error: Invalid shape: matmul weight has EMPTY data buffer (...) rc=1
after: loads and generates rc=0
That fixture is not fully repaired: it now runs but its output is degenerate
(<|fim_suffix|> repeated, identical across two different prompts). Every weight
in the file is bf16 and the body path was unreachable until now, so that is a
separate defect and is not claimed here. What is fixed is the crash and an error
message that sent readers after a MoE bug which was never there.
Mutation check — reverted the four source hunks, kept the six falsifiers:
test_parse_code_model_after_prompt_is_honoured ... FAILED
assertion `left == right` failed: --model written after the prompt must be honoured, not swallowed
left: None
right: Some("/tmp/named.gguf")
test_parse_code_unknown_option_after_prompt_is_rejected ... FAILED
an unknown option after the prompt must fail to parse: Cli { command: Code {
model: None, ... prompt: ["hi", "--totally-bogus-flag-xyz"], ... } }
test_cmd_code_rejects_unknown_resume_session_id ... FAILED
must say the session is unknown; got: cannot read manifest /nonexistent/manifest-2398.toml
test_cmd_code_rejects_nonexistent_project_dir ... FAILED
must name the flag; got: cannot read manifest /nonexistent/manifest-2398.toml
test_from_apr_ties_lm_head_to_embedding_when_head_is_zero_length ... FAILED
lm_head must fall back to the embedding matrix, not stay empty
Restored: 6/6 green. Full lib suites green — apr-cli 6658, aprender-orchestrate
6517, aprender-serve 15498.
The tied-head falsifier asserts arithmetic rather than success: every logit must
equal the dot product of the hidden state with that token's embedding row, so a
transposed tie — which still returns finite numbers — fails it.
Refs #2398, #2373.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…d 5, --num-steps 3 ran 100, and prune --plan was 20x off Four defects from the 0.63.0 crates.io dogfood (#2374 findings 10, 12, 9, 4). Each was reproduced first against a release binary built from d16c608, the same tree this commit changes. 1. `apr train halving` ignored the exit status of every trial (#2374-10) `Command::output()` returns `Ok` when the process merely SPAWNS, so `if let Ok(out) = cmd_output` (train.rs:1398-1409) never inspected `out.status`. Three trials that each exited 5 with "Training data path does not exist: tab.csv" were printed as "no eval", every `best_ppl` stayed `f64::INFINITY`, the sort at :1426 ranked three infinities, and `emit_halving_results` took `results[survivors[0]]` as the winner: Running sweep-000.yaml (lr=5.07e-4)... no eval (0s) Running sweep-001.yaml (lr=1.10e-3)... no eval (0s) Running sweep-002.yaml (lr=2.64e-4)... no eval (0s) ═══ WINNER ═══ Config: sweep-000.yaml Best val_ppl: inf lr_target = 2.5332e-4 rc=0 ("best_ppl": null for every entry in halv.json) That μTransfer learning rate is what a user carries into a real training run. Now the exit status is classified (`classify_trial`) and a winner must have evidence (`select_halving_winner`): Running sweep-000.yaml (lr=5.07e-4)... FAILED (0s) — exit 5: error: Validation failed: Training failed: Configuration error: Invalid config: Training data path does not exist: tab.csv error: Validation failed: no halving winner: not one of the 3 trials produced a finite val_ppl. 3 trial(s) failed — sweep-000.yaml: exit 5: ... rc=5, and no halv.json is written. Also fixed the secondary damage at train.rs:1371-1383: halving patched `max_steps` / `output_dir` into the user's own sweep-*.yaml in place, so a second run trained against inputs the first had rewritten. Trial configs are now materialised as copies under `<output-dir>/halving-trials/`; the diff of sweep-000.yaml before and after a run is empty. 2. `apr pretrain --num-steps N` rounded up to a whole epoch (#2374-12) `run_epoch` (pretrain.rs:615-616) always ran a full `steps_per_epoch`, so the config block and the result block of the SAME run disagreed. Measured on five input pairs, before → after: --num-steps 3 100 → 3 --num-steps 10 100 → 10 --num-steps 250 300 → 250 --num-steps 1000 --steps-per-epoch 400 1200 → 1000 --num-steps 1000 --steps-per-epoch 5 1000 → 1000 (exact multiple, unchanged) At the documented defaults (batch 16 x seq 1024) a 3-step smoke test paid for 100 steps of compute. 3. `apr prune --plan` under-sized the output by up to 20x (#2374-9) prune_include_01.rs:14 computed `file_size * (1 - target_ratio)` — arithmetic that never opened the model and ignored what prune does. Magnitude / Wanda / SparseGPT / Structured / Width all dispatch to `prune_magnitude`, which ZEROES weights: the completion table prints `Parameters 2428632 → 2428632`, so the file cannot shrink, and it is written back as dense f32. The plan now reads the tensor index (shapes only) and predicts surviving-params x 4 bytes: ratio 0.2 est 3.71 MiB → 9.26 MiB (actual 9,717,252 B, 0.03% off) ratio 0.5 est 2.32 MiB → 9.26 MiB ratio 0.9 est 474.61 KiB → 9.26 MiB 87 MB model, ratio 0.5: est 43.33 MiB → 86.64 MiB (actual 90,862,788 B, 0.01% off) The table now also prints `Parameters in → kept`, `Weights zeroed N%`, and says outright that unstructured pruning removes nothing. Depth pruning — the one method that does remove parameters — shares `parse_layer_spec` with the run, so plan and run cannot disagree, and `--method depth --plan` without `--remove-layers` is now rejected (rc=5) exactly as the run is. 4. `apr train --task classify` claimed a blocker that does not exist (#2374-4) The default `--task` of both `train plan` and `train apply` was `classify`, a task that always exited 5, so the documented bare invocation `apr train plan --data <file>` could never succeed. Worse, the message (train.rs:15-21) said "requires entrenar >= 0.8 (not yet published)" — false in the binary printing it: entrenar is the in-tree crate `crates/aprender-train` built at the workspace version, so 0.63.0 links entrenar 0.63.0. The surface is deleted rather than implemented: `apr train` trains causal LMs, and the error now routes to the command that does implement classification. before: error: ... requires entrenar >= 0.8 (not yet published). after: error: ... apr train does not implement classification — it trains causal LMs (--task pretrain, the default). For classification fine-tuning use: apr finetune <model> --task classify --data <file.jsonl> --num-classes <N> Two tests encoded defects and were corrected, not deleted: `classify_not_available_returns_validation_failed` asserted the message contained "entrenar", locking in the false claim; `test_plan_mode` / `test_plan_json` fed 2048 zero bytes — not a model in any format — and asserted `is_ok()`, which passed only because the old estimate never read the file. They now assert a plan for a non-model is rejected, with a companion test that a plan on a real model still succeeds. Mutation check — each fix reverted with the tests untouched, verbatim RED: M1 num_steps_is_a_budget_not_rounded_up_to_a_whole_epoch assertion `left == right` failed: requested 3 steps at steps_per_epoch=100, executed 100 / left: 100 / right: 3 M2 select_halving_winner_refuses_to_crown_all_failed_trials FAILED classify_trial_nonzero_exit_is_a_failure_not_a_score FAILED select_halving_winner_refuses_when_no_trial_printed_a_val_ppl FAILED select_halving_winner_picks_the_scored_survivor: left: 0 / right: 1 M3 plan_estimate_matches_the_bytes_pruning_actually_writes target-ratio 0.2: plan said 65939 bytes, prune wrote 132164 (50.1% off) plan_estimate_drops_removed_layers_for_depth_pruning plan said 98753 bytes, prune wrote 66500 M4 classify_not_available_names_the_command_that_works must point at the command that implements classification: apr train (classify) requires entrenar >= 0.8 (not yet published). ... train_plan_and_apply_default_to_a_task_that_can_run `apr train plan` defaults to `classify`, a task this command cannot run Restored, all green. cargo fmt --all --check, clippy -p apr-cli/-p aprender-train --lib -D warnings, apr-cli --lib 6666 passed, aprender-train --lib 7602 passed. The three `prune::snapshot_tests` failures in aprender-train are pre-existing at d16c608 (verified by stashing this change) and untouched here. Refs #2374, #2373 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ess-restricted one
Coverage Nightly went red on main this morning after 7451 passing tests:
thread 'efficiency::device::tests::test_cpu_info_detect' panicked at
crates/aprender-train/src/efficiency/device/tests.rs:50:5:
assertion failed: cpu.threads >= cpu.cores
test result: FAILED. 7451 passed; 1 failed
make: *** [Makefile:339: coverage] Error 1
The two fields are measured against different denominators. `threads` comes from
std::thread::available_parallelism(), which is cgroup- and affinity-aware and
reports what THIS process may run on. `cores` came straight from /proc/cpuinfo,
which describes the whole machine and honours no restriction at all. Whenever the
process is CPU-restricted the second number is the larger one and the invariant
inverts.
Measured on the runner that failed, a 24-physical-core host:
$ awk -F: '/^physical id/{p=$2} /^core id/{print p"-"$2}' /proc/cpuinfo | sort -u | wc -l
24
$ nproc # unrestricted
48
$ taskset -c 0,1 nproc # what a restricted process sees
2
so the assertion became `2 >= 24`. That is not a flake: with 16 concurrent CI jobs
on one box it is the expected reading, and it aborts the run with the remaining
tests unexecuted, so a real regression behind it would be invisible.
cpu.rs now reconciles the two in `usable_cores`, clamping the machine-wide physical
count to the parallelism actually available. `cores` consequently means "physical
cores this process can use", which is what its consumers need --
estimated_memory_bandwidth_gbps sizes from it, and sizing from cores the scheduler
will never grant is wrong independently of the test.
Mutation check. Restoring the old expression, keeping the new tests:
detected_physical.unwrap_or_else(|| threads.max(1)) // MUTATION: pre-fix behaviour
test usable_cores_never_exceeds_available_parallelism ... FAILED
test usable_cores_upholds_the_cores_le_threads_invariant ... FAILED
assertion `left == right` failed: a process allowed 2 CPUs must not report
24 usable physical cores
test result: FAILED. 47 passed; 2 failed
Note that test_cpu_info_detect stayed GREEN under that mutation, because this host
is unrestricted. That is exactly why the defect appeared only on a loaded CI runner
and never locally, and why the new tests drive the reconciliation over its input
space instead of over whatever the current machine happens to be.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
qwen-story-daily has been red on main with:
✗ FAIL B2 format_parity - no format_parity gate found in --json
output (got: '', apr qa exit=0)
16 PASS / 1 FAIL / 1 SKIP
The gate ran. It passed. Running the same command by hand against the same
two fixtures on the same host:
format_parity: skipped=False passed=True
run_cmd captured `timeout "$t" "$@" 2>&1` into a single RC_OUT, and nine call
sites read that one variable - some pipe it to jq, some grep it for a panic.
Those two groups want opposite things. `apr qa --json` writes its JSON to
stdout and its diagnostics to stderr; on the CUDA path the executor emits
unconditional `[trueno#243] Manual graph construction: ...` lines through
eprintln! (7 call sites under crates/aprender-serve/src/cuda/). Merging put
those in front of the JSON, jq could not parse it, the extraction returned
empty, and the harness reported a defect that does not exist.
A false FAIL is exactly as corrosive as a gate that cannot fail. Main goes red
for a non-reason, and an andon that cries wolf stops being read.
run_cmd moves to scripts/lib_story_run.sh (sourceable, and OPTION-NEUTRAL - it
must never run `set`, which is how apr_bin.sh once leaked errexit and killed
this same nightly six lines in) and now sets three views:
RC_OUT stdout only - parse this for JSON
RC_ERR stderr only
RC_ALL both - grep this for panics and banners
The two call sites that genuinely need stderr are switched to RC_ALL and say
why. That is not cosmetic: a Rust panic is written to stderr, so splitting the
streams without moving `grep -qE 'thread.*panicked'` would have silently
stopped the story detecting panics - a worse defect than the one being fixed.
Falsifier: scripts/check_story_json_streams.sh drives the real run_cmd over
commands with known stream behaviour, including the exact failing shape (JSON
on stdout, `[trueno#243]` on stderr). Wired into ci.yml next to the other
text-only poka-yoke guards, and registered as FALSIFY-QWEN-STORY-014.
Mutation check - restoring the merged capture, keeping the guard:
RC_OUT=$(timeout "$t" "$@" 2>&1); RC_EC=$? # MUTATION: pre-fix
FAIL JSON on stdout survives diagnostics on stderr
expected: false true
actual:
FAIL RC_OUT is stdout only
FAIL RC_ERR is stderr only
check_story_json_streams: 3 assertion(s) FAILED
`actual:` empty is the production symptom exactly - the `got: ''` the nightly
printed. Note the panic assertion stays GREEN under that mutation, which is
correct: the pre-fix code did see panics. The guard is precise about which
behaviour it owns.
Also, in the same file: FALSIFY-QWEN-STORY-007 ("bashrs lints clean") was
already failing on main. `bashrs lint scripts/qwen-story.sh` reported 2 errors,
both SC1078, from the jq program in pmat_rows being written as a double-quoted
string spanning three lines. Assembled with printf instead - identical program,
verified on both shapes pmat can return, including the documents-object
fallback that the `type == "array"` guard exists for. That falsifier is green
again:
$ bashrs lint scripts/qwen-story.sh 2>&1 | grep -qE '(^|[^0-9])0 error'
exit=0
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…nt nothing Dogfooding 0.63.0, and still true on main at 76f8222. On a healthy model: │ ⚠ WARN │ Metadata │ Missing 'license' field │ ⚠ WARN │ Metadata │ Missing 'model_card' │ ⚠ WARN │ Metadata │ Missing 'provenance' information │ ℹ INFO │ Efficiency │ 121 uncompressed tensors exceed 1MB - consider compression ✗ Lint failed 4 issue(s): 0 error(s), 3 warning(s), 1 info(s) error: Lint failed with 0 error(s), 3 warning(s), 1 info(s) exit=5 It says `0 error(s)` and fails anyway. lint.rs gated on `report.passed()`, which is `warn_count == 0 && error_count == 0`, and no flag relaxed it. Every real model is missing at least one of license / model_card / provenance, so the command could not exit 0 on anything - an INFO-level "consider compression" was enough to sink the run. It also did not discriminate. The corrupt GGUF that `apr validate` hard-rejects: ✗ Lint failed 5 issue(s): 0 error(s), 3 warning(s), 2 info(s) exit=5 Same exit code as the healthy model. A verdict that is always "fail" carries no information, which is the same disease as a gate that cannot fail - only inverted. Two dependents had absorbed it rather than reporting it: - `apr qualify` runs lint as one of its gates (qualify.rs), so that gate has been permanently failing for reasons unrelated to the model. - the nightly story wrote `if [ "$RC_EC" -eq 0 ] || [ "$RC_EC" -eq 5 ]` - PASS whether lint passed OR failed. It could not have detected either. Errors are defects, warnings are advice, info is a suggestion. Only the first fails by default; `--strict` promotes warnings; INFO never fails. `LintReport::passed_at_level(fail_at)` states the threshold, and the two existing predicates turn out to be two of its three cases (`passed()` is Warn, `passed_strict()` is Info) - verified by a test rather than asserted here. The exit code, the JSON `passed` field and the printed badge now all derive from one expression. That last one was NOT free: after the exit code was fixed, `apr lint` exited 0 while still printing `✗ Lint failed 4 issue(s)`. GH-601 exists because those disagreeing is itself the defect, so print_summary takes the threshold too. JSON gains `clean` (no issues at all), `strict` and `fail_level`, so a caller wanting the stricter reading has a field instead of re-deriving it from counts. $ apr lint model.apr ✓ Lint passed 4 issue(s): 0 error(s), 3 warning(s), 1 info(s) 3 warning(s) are advisory; re-run with --strict to fail on them exit=0 $ apr lint model.apr --strict ✗ Lint failed 4 issue(s): 0 error(s), 3 warning(s), 1 info(s) error: ... — failing on error(s) or warning(s) (--strict) exit=5 Mutation check - restoring the old predicate, keeping the tests: let _ = fail_at; // MUTATION self.warn_count == 0 && self.error_count == 0 // fail on ANY warning advisory_findings_alone_do_not_fail_the_default_verdict ... FAILED strict_promotes_warnings_to_failures ... FAILED info_never_fails_anything_short_of_the_info_threshold ... FAILED the_verdict_actually_discriminates_between_two_models ... FAILED passed_at_level_subsumes_the_two_hardcoded_predicates ... FAILED test result: FAILED. 85 passed; 5 failed `an_error_fails_the_verdict_at_every_threshold` stays green under that mutation, correctly: an error failed before and fails now. Refs #2394. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
CI's workspace-test step 10 (Integration tests) failed on this batch:
FALSIFY-README-007: README lacks `**1768** provable contracts` matching
`find contracts/ -name '*.yaml'` — update the README claims table row
The batch adds a contract YAML, so the count moved 1767 -> 1768 and the README
claims table went stale. The falsifier is doing exactly its job.
Worth recording how this was missed locally: `cargo test --workspace --lib`
does NOT run integration targets, and workspace-test runs an 18-command
`cargo test -p X --test Y` chain as a separate step. A "--lib is green" local
check is therefore not equivalent to workspace-test, which is what this branch
was verified against before pushing. The full chain now runs clean locally.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This was referenced Aug 11, 2026
Closed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Batches 24 already-reviewed, individually mutation-verified branches into one CI run.
Why batch
The binding constraint on this audit was never engineering throughput — it was one ~50-minute
workspace-testper PR on one shared box. Nine concurrent PRs do not run nine times faster; they starve each other. Every blocked PR this morning was classified from job metadata:Eight PRs, 4.5 hours, zero merges.
ci.yml:238already documents the mechanism: "Under runner-pool saturation (7+ concurrent CI runs) we observed 55min hits exactly at the timeout."One branch, one run.
Two main-branch nightlies were red
assert!(cpu.threads >= cpu.cores)—threadsis cgroup-aware,corescame from/proc/cpuinfo. On the 24-core runner under load:2 >= 24. Panicked after 7451 passing tests, aborting the run with the rest unexecuted.format_paritymissing. The gate had run and PASSED.run_cmdcaptured2>&1into one variable, soapr qa --json's stderr landed in front of its JSON andjqsaw garbage.Conflict resolutions worth reading
/realize/modelclaimedloaded: trueon a server with no model. The#2402falsifier asserted that literal against thedemo_mock()fixture — documented as "no model = no inference overhead". A branch written to stop fabricating provenance hardcoded a fabrication one field over. Nowstate.model_loaded(); the assertion was rewritten, not the fix reverted.Two agents fixed the tied-embedding P0 differently. Kept the extracted
apr_load_lm_headon a correctness point, not style: it forcestranspose: falsefor the tied head — the embedding is[vocab, hidden]row-major in every architecture including the Conv1D ones that settranspose— and hard-fails when tied-but-no-embedding instead of falling through to a shape error.apr lint+--quiet/--verbose: the merge left#[contract(...)]attached to a helper instead ofrun(), silently moving a contract obligation. Reattached.Local verification
cargo test --workspace --libon this branch:The only failures are 2 in
aprender-gpu, whichworkspace-testexplicitly excludes (ci.yml:231) — so this run was stricter than CI, and everything CI will run is green. This branch touches zeroaprender-gpufiles (136 apr-cli, 36 aprender-serve, 17 aprender-train, 13 mcp, 13 core). Both look like shared-global-state counters under full-workspace parallelism; a proper baseline is running separately and will be reported honestly either way.Deferred to batch 2
Five branches conflict with work already in this batch — 21 hunks between two independent restructurings of the MCP tool layer, and similar in the lint family. Hand-merging two restructurings under time pressure is where a silent mistake ships. Each rebases cleanly onto the new main and gets resolved once, in the open:
mcp-args-dropped·lint-exit-codes·mcp-trace-validate·remainder-lint·serve-embeddingsSupersedes
Closes the following PRs, whose branches are merged here verbatim: #2413 #2422 #2426 #2427 #2431 #2432 #2434 #2436 #2438 #2440 #2439 #2437 #2435 #2412 #2411 #2410 #2385 #2445 #2446
🤖 Generated with Claude Code