fix(train): apr train apply trained on fabricated data and reported success

[noahgift]

apr train apply --task pretrain --config c.yaml on a 200-row CSV printed one line to stderr — Warning: Unsupported data format 'csv', using demo data — then ran two epochs at a constant loss of 0.010000, wrote a 21 MB checkpoint, printed DONE Pre-training completed and exited 0. The 0.010000 is the synthetic fixture value; the user's corpus was never opened. A JSONL dataset and a well-formed JSON array of records produced the byte-identical fabricated loss. In CI that is a green training job for a model that saw none of your data.

This PR fixes 11 of the 15 findings in #2374, worst first.

Before / after, measured

Run against a release apr built from this branch (/mnt/nvme-raid0/coverage/aprender/release/apr). Note the first attempt at this table used $(cargo metadata …).target_directory, which pointed at a different directory than the build actually wrote — that binary was another agent's and reported the old behaviour for every finding. The numbers below are from the binary whose mtime matches the build.

#	Before (0.63.0)	After
3	CSV → rc=0, loss 0.010000, 21 MB checkpoint	rc=5 Unsupported training data format 'csv' for 'tab.csv'. Supported: json. Convert the dataset to JSON — expected JSON of the form {"examples":[…]}
3b	JSON array of records → rc=0, same fabricated loss	rc=5 Could not parse training data 'bad.json': expected JSON of the form …
2	3 features / 1 target → rc=101, raw assert_eq! in MSELoss::forward	rc=5 batch 0 has 24 input values but 8 target values — input width 3 vs target width 1 …
2b	(1,1) trained	still trains, Final loss: 0.153731 (unchanged)
5	-o DIR ignored; checkpoint went to ./checkpoints	Saving model to /tmp/…/FLAG_OUT/final_model.json, directory created
7	text/json/yaml/bogus → 4× md5 a63600bb, all rc=0	a63600bb / b717d476 / 2071f6d6; bogus → rc=5 Unknown --format 'bogus'. Supported: text, json, yaml
6	ranks 4/8/16/64/256/1024 → all recommended_rank = 256	4/8/16/64/256/1024 → 4/8/16/64/256/1024
8	--status completed → 0 of 2948; bogusvalue → 0 rows rc=0	completed → 2948; bogusvalue → rc=5 Unknown run status …
13	tune --method bogus → rc=0 planned Auto; sweep --strategy bogus → rc=0, wrote a random sweep	both rc=5; no sweep directory created
15	error: Invalid APR format: apr grad-norm: failed to parse JSON …	error: Invalid input: apr grad-norm: failed to parse JSON … (still exit 4)
14	No model path or --model-size provided (the path was the first positional arg)	Could not read the model architecture from '…064a3693fa1ea02c.safetensors' (format: safetensors). Pass --model-size to state it explicitly
1	.apr + one unrelated sibling → Failed to load APR model: No SafeTensors files found in <dir>	Failed to load APR model 'm.apr': … — the named file, sibling never opened
11	OK CONVERGED final val_loss=3.0000 at target 0.001	OK NOT_CONVERGED final val_loss=3.0000 > target 0.0010; --json now carries target_val_loss and converged: false

Root causes

• 3 — crates/aprender-train/src/config/train/batches/loader.rs:15 documented "Supports parquet, JSON, and CSV formats via alimentar" while the match at 32-46 handled only parquet/json and fell through to create_demo_batches. json.rs:64 did the same on a parse failure; parquet.rs on unrecognised columns. All four now return Error::ConfigError, and create_demo_batches is deleted so nothing can reach for it again.
• 2 — tabular mode drives the generic Trainer with an identity forward, so MSELoss needs equal widths. Sweeping six (input, target) pairs, it survived only when input_dim == target_dim. New validate_tabular_batch_shapes rejects before any training runs.
• 1 — crates/apr-cli/src/commands/finetune.rs:2183 discarded the file and passed its parent directory to ClassifyPipeline::from_pretrained, which scans for any SafeTensors. The comment claimed from_apr() was unavailable in entrenar 0.7.5; entrenar is in-tree at 0.63.0 and Transformer::from_apr exists (aprender-train/src/transformer/model.rs:153).
• 5 — the -o flag was never read; only training.output_dir was, and its destination was never created.
• 6 — apr tune called plan(), which auto-selects rank purely from --vram. New plan_with_rank pins it and derives alpha, trainable params, memory and the rank-aware LR from it.
• 7 — declared _format: &str, never read. 13 — unwrap_or(TuneMethod::Auto) swallowed a perfectly good FromStr error; "random" | _ swallowed the strategy. 8 — the filter stringified the stored variant (Success) and compared it verbatim. 11 — RunStatus::Ok mapped to the literal "CONVERGED" without reading target_val_loss. 15 — CliError::InvalidFormat's Display is hardcoded to "Invalid APR format".

Tests that encoded the defect

Nine tests asserted is_ok() on inputs that should fail; they would have blocked this fix and are rewritten to assert the error:

• test_load_training_batches_nonexistent_file, test_load_training_batches_unsupported_format, test_load_json_batches_invalid_format (×2 modules), test_load_training_batches_missing_file, test_load_training_batches_unsupported_extension, test_handle_missing_columns_returns_demo_batches
• test_train_from_yaml_success / _with_grad_clip / _with_lora / _with_quantize / _default_mode_is_tabular pointed data.train at a train.parquet that did not exist and asserted the whole run succeeded and wrote final_model.json — the fabricated-data path existed to make them pass. They now supply the dataset they always claimed to be training on.
• Three run_plan_* tests passed "apr" as --format; that only worked because the flag was dead. They now pass "text".

Mutation check

With all 17 fix sites reverted and every test kept (scratch_mutate.py in the branch history), the new falsifiers turn RED; restoring turns them GREEN. Verbatim RED output is in the PR thread.

Gates

• cargo fmt --all -- --check clean
• cargo test -p apr-cli --lib — 6690 passed, 0 failed
• cargo test -p aprender-train --lib — the only remaining failures are prune::snapshot_tests::{all_prune_methods, pipeline_stages, schedule_validation_errors}, which reproduce identically on a clean tree at HEAD (8cc3aaf) with this branch stashed: 14 passed / 3 failed both ways. Pre-existing, JSON key-ordering, in a module this PR does not touch.

Not fixed (left open)

• 4 — train plan/apply --task classify is a stub whose message claims entrenar >= 0.8 is unpublished (it is in-tree at 0.63.0). The honest fix is to implement it or delete the surface; both are larger than this PR.
• 9 — prune --plan size estimate.
• 10 — train halving ignores trial exit status.
• 12 — pretrain --num-steps rounds up to a whole epoch.

Also newly visible because the sibling no longer masks it: with finding 1 fixed, apr finetune <0.5B>.apr --task classify now fails inside Transformer::from_apr with Shape mismatch for 'lm_head.weight': expected 136134656 elements, got 0 — a real, distinct defect in the APR classify loader (tied word embeddings), not a regression from this PR. Worth its own issue.

Refs #2374 (partial) — remaining: 4, 9, 10, 12
Audit epic: #2373

[noahgift]

Mutation check — verbatim RED

All 17 fix sites reverted, every test kept.

cargo test -p aprender-train --lib config::train — 183 passed, 9 failed

failures:
    config::train::batches::tests::test_load_json_batches_invalid_format
    config::train::batches::tests::test_load_training_batches_nonexistent_file
    config::train::batches::tests::test_load_training_batches_unsupported_format
    config::train::loader::tests::test_tabular_shapes_name_the_offending_batch_index
    config::train::loader::tests::test_tabular_shapes_reject_three_features_one_target
    config::train::loader::tests::test_tabular_shapes_reject_wider_target_than_input
    config::train::tests::json_batches::test_load_json_batches_invalid_format
    config::train::tests::training_batches::test_load_training_batches_missing_file
    config::train::tests::training_batches::test_load_training_batches_unsupported_extension

---- config::train::batches::tests::test_load_training_batches_unsupported_format stdout ----
panicked at crates/aprender-train/src/config/train/batches/tests.rs:140:44:
unsupported format must be rejected: [Batch { inputs: Tensor { data: [0.1, 0.1, ... ], shape=[32] ... },
                                              targets: Tensor { data: [0.2, 0.2, ... ], shape=[32] ... } }]

---- config::train::loader::tests::test_tabular_shapes_reject_three_features_one_target stdout ----
panicked at crates/aprender-train/src/config/train/loader/tests.rs:2074:10:
3 features / 1 target must be rejected, not aborted: ()

cargo test -p apr-cli --lib — 6674 passed, 16 failed

failures:
    commands::grad_norm::tests::malformed_history_does_not_blame_the_apr_format
    commands::model_config::arch_resolution_tests::gguf_gets_the_same_actionable_message
    commands::model_config::arch_resolution_tests::unreadable_model_error_does_not_claim_no_path_was_given
    commands::pretrain::tests::an_aborted_run_has_no_convergence_verdict
    commands::pretrain::tests::early_stop_is_judged_on_its_best_loss
    commands::pretrain::tests::missing_the_target_is_not_convergence
    commands::pretrain::tests::the_verdict_moves_with_the_target
    commands::runs::runs_tests::test_status_completed_selects_the_success_variant
    commands::runs::runs_tests::test_status_is_case_insensitive_and_trimmed
    commands::runs::runs_tests::test_status_matches_the_debug_spelling_of_the_stored_variant
    commands::runs::runs_tests::test_unknown_status_is_rejected_not_silently_empty
    commands::train::tests::patch_yaml_config_output_flag_overrides_the_yaml_output_dir
    commands::train::tests::plan_format_rejects_an_invalid_value
    commands::train::tests::sweep_rejects_a_bad_strategy_before_writing_anything
    commands::train::tests::sweep_strategy_rejects_a_typo_instead_of_silently_randomising
    commands::train::tests::sweep_strategy_typo_is_an_error_naming_the_alternatives

---- commands::runs::runs_tests::test_status_completed_selects_the_success_variant stdout ----
panicked at crates/apr-cli/src/commands/runs.rs:1042:9:
assertion `left == right` failed
  left: Some("nothing-matches-this")
 right: Some("Success")

---- commands::pretrain::tests::missing_the_target_is_not_convergence stdout ----
panicked at crates/apr-cli/src/commands/pretrain.rs:1054:9:
assertion `left == right` failed: final 3.0000 against target 0.001 is NOT converged
  left: Some(true)
 right: Some(false)

cargo test -p aprender-train-lora --lib — 60 passed, 2 failed

failures:
    optimizer::tests::test_requested_rank_is_honoured_not_discarded
    optimizer::tests::test_requested_rank_moves_the_derived_plan

The mutation caught a weak test

My first pass mutated finding 1 as if ext == "apr" → if false, which routes to the new refusal branch rather than restoring the original defect — and all three classify falsifiers stayed GREEN. Re-mutated to the real 0.63.0 code (parent directory → from_pretrained), and the load-the-named-file falsifier turned RED with the sibling named in the message:

---- commands::finetune::tests::classify_load_never_falls_back_to_a_sibling_safetensors stdout ----
panicked at crates/apr-cli/src/commands/finetune_tests.rs:1766:5:
the failure must name the file the user passed: Validation failed: Failed to load APR model:
  Configuration error: Failed to parse SafeTensors
  /tmp/apr-2374-cls-796529-sibling/UNRELATED_SIBLING.safetensors: header too small

That same run showed classify_load_rejects_a_non_apr_file_instead_of_scanning_its_directory passing under the defect — it asserted only that the path appeared in the message, and under the parent-dir scan the path appears because it is the file that got scanned. Tightened in 50df077 to assert the refusal wording. Restored: 3 passed, 0 failed.

[noahgift]

Temporarily closing to stop CI contention — the branch is untouched and this will be reopened, nothing is lost.

The shared clean-room runner host is at 91% disk with a load average over 100, and workspace-test has started timing out at 75 minutes on unrelated PRs (#2385) purely from contention. This audit has 23 PRs open at once and drafts still trigger full CI in this repo, so parking them as drafts did not reduce load.

Reopening in batches as the merge queue drains. The work is complete and reviewed; only the CI scheduling is being paced.

[noahgift]

Superseded by #2449 — this branch is merged verbatim into that batch.

The binding constraint was one ~50-minute workspace-test per PR on one shared box; nine concurrent PRs starved each other into 75-minute step timeouts (every blocked PR this morning classified as CONTENTION, zero merges in 4.5 hours). #2449 lands 24 branches in a single CI run.

Closing rather than leaving open so this PR cannot move #2449's base and force it to re-run. The branch is untouched and this is reopenable if the batch does not land.