Skip to content

Split dep-graph edge lists into their own file and serve them in place - #60

Draft
xmakro wants to merge 2 commits into
perf/base-0713from
perf/dep-graph-edges-in-place-upstream
Draft

Split dep-graph edge lists into their own file and serve them in place#60
xmakro wants to merge 2 commits into
perf/base-0713from
perf/dep-graph-edges-in-place-upstream

Conversation

@xmakro

@xmakro xmakro commented Jul 26, 2026

Copy link
Copy Markdown
Owner

Standalone against upstream master (perf/base-0713 = upstream 3659db0), no dependency on the carry/append-only stack. This is the upstream-first wedge: it lands the "previous session's dep-graph bytes stay available for the session" infrastructure as a pure win, which #42/#43/#44/#58/#59 then build on. Two commits, meant to be squashed for submission; the second reworks the first into the split-file shape after the single-file version showed a max-rss regression.

Decoding the previous dep graph used to copy every edge byte out of the file into a freshly allocated buffer (edge_list_data), the largest allocation and copy of the load. Split the on-disk dep graph into two streams: dep-graph.bin keeps the fixed-size node records and the footer, and dep-graph-edges.bin holds every record's edge bytes, in record order. Each encoder worker buffers the two streams separately and appends both chunks under one lock acquisition, so record order stays identical across the files and decoding recovers every edge-list position by accumulating sizes — it reads no edge bytes at all.

Only the edge-list file is retained (memory-mapped) for the session, and the edge lists are served directly from it: the on-disk varint encoding was already the in-memory representation. The pages the session keeps resident are exactly the edge bytes the marking walk reads; the record headers — about two thirds of the total bytes for syn — are dropped after decode as before. DEP_NODE_PAD moves from the deleted in-memory buffer to the tail of the edge-list file, which keeps the fixed-size overread argument local and trivial.

Precedent: retaining an incremental file's mapping for the session and lazily reading from it is the query result cache's existing lifecycle (OnDiskCache::serialized_data, dropped just before the save in save.rs). On Windows the mapping must not outlive the load because the save later renames over the mapped file, so the edge bytes are copied out once — the same single copy this change deletes. (The OnDiskCache-style drop-before-rename shape was considered instead; releasing the backing mid-session would put synchronization on edge_targets_from, the hottest read of the marking walk.)

Local rustc-perf (primary crates, instructions:u), idle machine, vs the base commit:

scenario instructions:u max-rss
incr-unchanged -0.54% +0.19%
incr-patched -0.34% -0.26%
full / incr-full -0.01% / +0.02% (control) -0.18% / +0.42%

26 of 60 cells improved by at least 0.25%, 0 regressed, 0 failed benchmarks. (The single-file variant measured -0.66%/-0.44% instructions but +6.1%/+5.3% max-rss from keeping the interleaved headers resident; the split keeps the win and makes memory neutral.)

Validation: tests/incremental 178 passed / 0 failed; 12 generations each of incr-unchanged, of alternating real edits, and of alternating edits under -Zincremental-verify-ich: every generation clean. A truncated or mismatched edge-list file is detected at load (trailer check + exact size accounting) and degrades to a fresh build.

Roadmap note: the split is also the layout the later carry/append-only stack wants — the wholesale region carry becomes two wholesale copies (headers can go fd-to-fd and never enter RSS), and the retained mapping stays edges-only in every design generation.

xmakro and others added 2 commits July 26, 2026 06:40
Decoding the previous dep graph used to copy every edge byte out of the
file into a freshly allocated buffer (edge_list_data), the largest
allocation and copy of the load. Instead, retain the file mapping for the
session and serve the edge lists directly from it: the on-disk varint
encoding was already the in-memory representation, and each node's edge
header now records a position in the file instead of a position in the
copied buffer. Decoding a record shrinks to reading its fixed header and
skipping over the edges.

This also removes the DEP_NODE_PAD padding and the capacity-estimation
heuristic: the fixed-size overread at the end of an edge list stays in
bounds because the records are always followed by the per-kind node
counts, the session count and the fixed-size tail.

Retaining an incremental file's mapping for the session and lazily
decoding from it is the same lifecycle the query result cache already
uses (OnDiskCache::serialized_data). It also moves the edge bytes from
anonymous heap memory to evictable file-backed pages. On Windows the
mapping must not outlive the load, because the save later renames over
the mapped file; there the bytes are copied out once, a single copy
comparable to the per-record copies this replaces.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…g edges-only

Rework of the previous commit: instead of retaining the whole dep-graph file
(whose record headers are dead weight after decode but share pages with the
edge bytes, showing up as a max-rss regression), write the fixed-size node
records and the edge bytes as two separately-streamed files. Decoding streams
the node file and reconstructs every edge-list position by accumulating sizes
in record order - it reads no edge bytes at all - and only the edge-list file
stays mapped for the session. Max-rss is neutral against the baseline while
the instruction win is preserved.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@xmakro xmakro changed the title Serve dep-graph edge lists in place from the retained file mapping Split dep-graph edge lists into their own file and serve them in place Jul 27, 2026
@xmakro
xmakro marked this pull request as draft July 28, 2026 11:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant