mlpstorage multi-run score aggregation — checkpointing

[FileSystemGuy]

Companion to #645. Same investigation, applied to the checkpointing workload.

Summary

DLIO already reduces per-checkpoint I/O to per-invocation means. mlpstorage never reads those fields, so the <model> - Write B/W / <model> - Read B/W columns in the summary CSV come out empty. Unlike #645, this is not "reduce N samples to a scalar" — checkpointing has at most 2 invocations per model per submission (§2.1.23, §4.7.1). It's plumbing plus a small spec clarification.

What DLIO already does (per invocation)

DLIO_local_changes/dlio_benchmark/utils/statscounter.py:

• Per checkpoint event (end_save_ckpt L376–383, end_load_ckpt L393–400): per_epoch_stats[epoch]['save_ckpt{block}']['throughput'] = checkpoint_size / duration in GB/s.
• Per invocation (end_run L162–187):
  • save_checkpoint_io_mean_GB_per_second = np.mean(io_save)
  • save_checkpoint_io_stdev_GB_per_second = np.std(io_save)
  • save_checkpoint_duration_mean_seconds = np.mean(duration_save)
  • load_checkpoint_io_mean_GB_per_second = np.mean(io_load)
  • load_checkpoint_io_stdev_GB_per_second = np.std(io_load)
  • load_checkpoint_duration_mean_seconds = np.mean(duration_load)

Arithmetic subtlety to decide on in Rules.md: DLIO computes mean(size / duration_i) — the arithmetic mean of rates. This is not the same as total_size / total_duration when per-checkpoint durations vary. Every checkpoint is the same size per §4.7.1, so any divergence is purely from duration variance. The spec should say which one is "the reported B/W."

What mlpstorage does today

Grepped mlpstorage_py/ for save_checkpoint_io|load_checkpoint_io|save_checkpoint_duration|load_checkpoint_duration|checkpoint.*throughput|checkpoint.*bandwidth:

• Zero non-test hits. No checker or reporter reads any of the I/O metrics.

Existing plumbing that would be reused:

• mlpstorage_py/submission_checker/checks/helpers.py:354–404 — _pair_checkpoint_runs already pairs write-only and read-only invocations by timestamp. Rules 4.7.1 (invocation structure) and 4.7.2 (total duration) already use it.
• mlpstorage_py/report_generator.py:319 — documents "1 disk dir = 1 run (write-then-read self-warms), no warmup." No warmup exclusion problem for checkpointing.

Stub sites where the columns would land:

• mlpstorage_py/submission_checker/results.py:43–50 — ResultExporter.head scaffolds 8B - Write B/W (GiB/s), 8B - Read B/W (GiB/s), and the 70B/405B/1T counterparts.
• mlpstorage_py/submission_checker/results.py:56–69 — ResultExporter.add_result() emits empty strings for every column (same stub as training).

Same story on origin/v2.0-branch:

• mlpstorage/rules.py:685–688 — CheckpointingRunRulesChecker.check_benchmark_type is an empty pass.
• mlpstorage/rules.py:697–755 — CheckpointSubmissionRulesChecker.check_num_runs counts total writes/reads across runs, requires each == 10. No I/O aggregation.
• mlpstorage/reporting.py:117–119 — same metrics=dict() workload-group stub as training.

Open questions (spec side)

For Rules.md to answer before implementation:

1. Combined-mode source (1 invocation with both write and read): confirm <model> - Write B/W reads save_checkpoint_io_mean_GB_per_second and <model> - Read B/W reads load_checkpoint_io_mean_GB_per_second from the single summary.json.
2. Split-mode source (2 invocations, write-only then read-only): <model> - Write B/W from the write invocation's save_checkpoint_io_mean_GB_per_second, <model> - Read B/W from the read invocation's load_checkpoint_io_mean_GB_per_second.
3. Rate-mean vs. aggregate-rate: does the reported B/W use DLIO's np.mean(size / duration_i) as-is, or should mlpstorage recompute (10 * checkpoint_size) / total_write_duration (and same for reads)? These diverge when per-checkpoint durations vary. Preferably fix the choice once and document it in §4.7.

Scope

Smaller PR than #645:

1. Amend Rules.md §4.7 with the three answers above.
2. Fill in ResultExporter.add_result() to read the two DLIO fields per checkpointing model, using _pair_checkpoint_runs to route split-mode invocations to the correct column.

Related: #645 (same underlying problem for training).