Out-of-core canonical columns (disk-backed mmap store)

[adhami3310]

What

Realizes the "mmap (native)" canonical-store row of the memory model (dossier §27): a canonical column may be backed by a disk np.memmap instead of RAM, so a figure's resident memory stays screen-bounded (the density pyramid + grids), never data-bounded. This lets a scatter's canonical x/y exceed RAM — the target §2 has always listed as "100M+ / out-of-core" but never realized on native.

How

The load-bearing observation: a NumPy memmap is a transparent ndarray. A column backed by one:

• satisfies the ColumnStore dedup key (id(base), data_ptr, nbytes),
• passes straight to the ctypes kernels via the raw buffer pointer (arr.ctypes.data), so the OS pages it in on demand — zone_maps, bin_2d, range_indices, and the density pyramid all consume it with no special-casing,
• is dropped by the kernel as a rebuildable cache exactly like an in-RAM one.

So the store, zone maps, and every LOD tier already work out-of-core. The one operation that genuinely needs a new path is creating an array too large for RAM:

• xy._ooc.MemmapF64Builder streams canonical f64 to a disk memmap one batch at a time (peak RAM = one batch); open_f64 reopens a column from disk.
• Feed the finished view to the ordinary API — fig.scatter(x=col, y=col, density=True) — and the whole pipeline runs against disk-backed truth.

Honest accounting (§27)

memory_report() now separates:

• canonical_bytes — RAM-resident canonical (unchanged for all-RAM figures)
• canonical_mapped_bytes — disk-backed, OS-paged, reclaimable

An all-RAM figure reports canonical_mapped_bytes == 0, so nothing changes for existing usage.

Tests

tests/test_ooc.py (6 tests): builder round-trip + growth, empty builder, ingest-without-RAM-copy, in-RAM report unchanged, zone-maps match RAM, and screen-bounded density first-paint over a memmap-backed scatter. Full suite green; ruff / ty (new files) / pre-commit clean.

Spec

• §2 — out-of-core target marked realized
• §27 — new rule 5 (canonical may be out-of-core, native mmap)
• lod-architecture.md §4.4 — canonical out-of-core landed

[codspeed-hq]

Merging this PR will not alter performance

✅ 97 untouched benchmarks
⏩ 1 skipped benchmark¹

---

_{Comparing out-of-core-canonical-store (17db00f) with main (a763d5f)}

Footnotes

1. 1 benchmark was skipped, so the baseline result was used instead. If it was deleted from the codebase, click here and archive it to remove it from the performance reports. ↩

[greptile-apps]

Greptile Summary

This PR adds disk-backed canonical columns for out-of-core native rendering. The main changes are:

• A streaming float64 memmap builder and reopening helper.
• Separate accounting for RAM-backed and mapped canonical bytes.
• Out-of-core ingestion, zone-map, and density-rendering tests.
• Updated memory-model and LOD architecture documentation.

Confidence Score: 4/5

The empty-column persistence path needs a fix before merging.

Non-empty construction, growth, ingestion, and memory classification are consistent. An empty finalized column reopens with one row.

python/xy/_ooc.py

T-Rex Logs

What T-Rex did

• I reproduced the empty-column reopen scenario by running the MemmapF64Builder finalize and open_f64 flow with the editable xy package; the finalized view had length zero, but reopening the exact file produced a one-row array, while the on-disk empty column occupied eight bytes and open_f64 returned a float64 array containing a single bogus zero.
• I ran a contract-validation test battery, including a baseline density test and a non-empty harness with kernel_backend=native where both memmapped columns shared input memory and ingest_copies=0, and the full test run of tests/test_ooc.py; all checks passed with exit code 0, and the empty-builder reopen defect did not appear as a finding.

_{Ran code and verified through T-Rex}

Important Files Changed

Filename	Overview
python/xy/_ooc.py	Adds memmap construction, detection, and reopening; empty finalized columns reopen with the wrong length.
python/xy/columns.py	Separates mapped canonical storage from RAM-backed canonical bytes in memory reports.
tests/test_ooc.py	Covers growth, ingestion, accounting, zone maps, and density rendering, but not reopening an empty column.
spec/design-dossier.md	Documents native mmap-backed canonical storage and mapped-byte accounting.
spec/design/lod-architecture.md	Documents out-of-core canonical columns in the LOD architecture.

_{Reviews (1): Last reviewed commit: "Out-of-core canonical columns (disk-back..." | Re-trigger Greptile}

[greptile-apps]

Empty Column Reopens With One Row

When an empty builder is finalized, this branch stores one physical f64 slot so NumPy can create the mapping. open_f64() later infers the row count from the file size, so the same file reopens as a one-element column and introduces a bogus row.

Artifacts

Repro: minimal executable empty-column reopen harness

• Contains supporting evidence from the run (text/x-python; charset=utf-8).

Repro: runtime output showing the finalized-zero versus reopened-one-row discrepancy

• Keeps the command output available without making the summary code-heavy.

_{Ran code and verified through T-Rex}