code checksum technique is too restrictive

[FileSystemGuy]

1. If I have an existing --results-dir with runs recorded in it,
2. and then upgrade my mlpstorage image (via git pull) and get a new version of the code,
3. and then make another run of the benchmark against the same --results-dir,
4. the code validation check will fail because the new/current code doesn't have the same hash value as the code that generated the older benchmark runs.

[FileSystemGuy]

Proposed design — content-addressed image pool at the org level

After discussion (offline), here's the resolution shape. Posting as a design comment so it's visible before implementation lands.

Problem

The current capture-or-verify gate treats a submission tree as owning exactly one codebase. Submitters who upgrade mlpstorage mid-campaign hit CodeImageError on their next submission-mode run in the existing --results-dir. The escape hatch (rm -rf code/ and re-capture) invalidates prior runs in the tree for §3.6.1/§5.6.1 verification. With the deadline close, we cannot ask submitters to re-run everything — but we also cannot weaken tamper-evidence.

Design

Move from one image per submission tree to one image per run, with images pooled per-org.

Layout:

<results_dir>/
  mlperf-results.yaml                    ← existing LAY-03 sentinel
  <orgname>/
    .mlps-image-pool                     ← migration-complete marker
    code-<hash8>/                        ← content-addressed image pool
      .code-hash.json                    ← existing schema, unchanged
      mlpstorage_py/...
    code-<hash8>/
    ...
  closed/<orgname>/results/.../<ts>/
    .mlps-code-image                     ← one-line pointer: the hash
    (existing run artifacts)
  open/<orgname>/results/.../<ts>/
    .mlps-code-image

Naming rationale. Ordinals (code, code2, code3) race on concurrent captures, collide across delete/re-add cycles, and require opening the inner JSON to identify what any given ordinal actually contains. Hash-based names (code-a3f8e91b/) make the directory name itself the tamper-evidence link and are self-verifying (mismatch between directory name and inner .code-hash.json.hash is instantly detectable). <hash8> is the first 8 hex chars of the same SHA the existing schema records — trivially unique in practice, still human-readable.

Location rationale — org level, not per-mode. Sharing the pool across closed and open modes gives OPEN's per-(system, benchmark, model) duplication (today captured under every leaf) automatic dedup for free. Two orgs on a shared drive keep separate pools (<results_dir>/<org1>/, <results_dir>/<org2>/), preserving the trust boundary a reviewer would want. rm -rf closed/ no longer touches captured code — history is preserved and mode-deletion is safe.

Pointer file format. Plain text, one line, exactly the hash string. YAML is overkill for one datum; we already have .code-hash.json inside images for structured metadata.

Capture-or-verify semantics (new)

compute live_hash from source_root
scan <results_dir>/<orgname>/ for code-*/ directories
match = image whose .code-hash.json.hash == live_hash

if match exists:
    write pointer file into the run leaf: <hash>
    return

# No image matches → capture a new one alongside existing.
capture_code_image(source_root, <results_dir>/<orgname>/code-<hash8>/, log)
write pointer file into the run leaf

No error on "hash doesn't match existing image." That's now the normal upgrade path. Tampering signal moves into the submission checker.

Automatic migration

First submission-mode run under the new code triggers migration if <results_dir>/<orgname>/.mlps-image-pool sentinel is absent:

1. Discover every code/ directory under <results_dir>/{closed,open}/<orgname>/....
2. Hash each; materialize <results_dir>/<orgname>/code-<hash>/ (identical hashes across two locations dedup to one physical location).
3. For every existing run leaf, walk to its nearest-ancestor code/ in the OLD layout, resolve which hash that was, write .mlps-code-image with that hash.
4. Delete the original code/ directories.
5. Write the .mlps-image-pool sentinel to mark migration complete; subsequent invocations short-circuit.

Idempotent, one-shot, invisible to the submitter beyond a status line. If discovery finds a code/ whose contents don't match its own .code-hash.json (hand-edited), migration aborts with a specific error — that case already needs human attention today.

Submission checker (per-image, not per-tree)

Per-run:

1. Run leaf has .mlps-code-image. Missing → fail.
2. Named code-<hash>/ exists in the pool. Missing → fail.
3. Directory name matches inner .code-hash.json.hash. Mismatch → fail.
4. Image contents re-hash to the recorded hash. Mismatch → fail.
5. §3.6.1 / §5.6.1 reference-checksum verification runs against that image, for the mlpstorage_version that image claims.

Whole-tree:

• Every image is pointed at by ≥1 run (no orphan images planted to look legitimate).
• No unhashed code/ directory anywhere (defense against a partial migration).

Tamper-evidence is arguably strengthened: the directory name itself carries part of the signal, and the checker verifies both the pointer chain and the image self-consistency.

Sequencing

PR 1 — layout + migration + capture-or-verify. Ships the submitter-facing unblock. Includes RED-first tests for migration, cross-mode dedup, pointer semantics, and hand-planted-code/ detection.

PR 2 — submission checker per-image verification. Ships the "you can actually submit a multi-image tree" guarantee. Only affects submission-final validation; can land after PR 1 without leaving anyone stuck.

Not in this scope: mlpstorage code-image list / gc ergonomics — nice to have but not on the deadline path.

What a submitter experiences post-migration

• Upgrade mlpstorage → next submission-mode run captures code-<newhash>/ alongside existing images, writes pointer, proceeds. Zero action required.
• Prior runs remain fully valid; their pointers reference their original images; §3.6.1 still passes.
• Reviewer inspecting a specific run reads its pointer file, opens the named image, verifies. Trace is tight.

Sharp edges

• CLOSED-mode spec strings ("changes to the codebase are not allowed in a CLOSED run") become inaccurate at capture time under this design — changes are allowed at capture; they just create a new image. If CLOSED forbids something at a higher level (mixing reference-checksummable vs unpinned versions in the same tree), that becomes its own explicit check with its own message.
• Pointer file must be written atomically with run start (before any run output) so a crashed-early run at least has a trace of intended image.
• Manually-deleted image → runs pointing at it fail cleanly with "image not found." Better than silently trusting an unverifiable hash.