You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit cf3b1e9
Browse filesBrowse the repository at this point in the historyBrowse files
fix(gpu): recover device-only declines at the remaining cliff sites (R4 DEEP, comp-tree, R3 barycentric) (#935)
* fix(gpu): recover device-only declines at the remaining cliff sites (R2 commit, R3 OOD, R4 DEEP)
Under VRAM pressure a device dispatch can decline after the device-only
gate already skipped the host drain, and the host fallbacks at the R2
comp-poly commit, the R3 parts/trace OOD and the R4 DEEP loop hard-abort
on the empty host buffers. Download the resident data instead: the trace
LDEs via materialize_lde_trace_host, the H part evaluations via a new
download off the resident R2 parts handle. The asserts remain only for
handles that cannot serve the data. The R4 DEEP host loop reads both the
trace and the part evals, so it recovers both sides.
Also adds sticky fault-injection hooks (test-faults) to the cuda
barycentric, DEEP and comp-tree entries: the drain-and-retry absorbs
one-shot faults, so the cliff paths need a fault that keeps firing.
* test(gpu): exercise the cliff-site recoveries end to end
Three prove+verify runs under sticky faults (comp-tree, barycentric,
DEEP), each requiring the device-only path to fire on the warm-up and
the recovery counters to move.
* fix(gpu): address cliff-recovery review — race-free sticky hook, parallel parts download
Review follow-ups on the device-only cliff recovery:
- check_sticky: collapse the load-then-decrement into one fetch_update that
saturates at 0, so concurrent per-table dispatches can't underflow the
counter — which would break both the sticky guarantee and the `== 0`
fired check.
- cuda_fallback_tests: disarm the sticky faults with a Drop guard, so a
panic in prove or a failing assert can't leave one armed and cascade into
the next test in the single-threaded binary.
- download_composition_parts_host: de-interleave under rayon and reinterpret
the u64 buffer in place, matching materialize_lde_trace_host instead of
copying again through u64_to_ext3_vec — this path fires often under VRAM
pressure.
- Docs: the device-only downgrade counter now also covers transient device
declines, not only gate misses; note the new &mut contract on
get_trace_evaluations_from_lde.
* style(gpu): rustfmt check_sticky and correct its doc
cargo fmt collapses the aligned match-arm comments (the CI lint failure);
also drop a stale doc sentence describing an earlier post-load variant that
the fetch_update version does not use.
* test(gpu): assert the parts-download counter is zero on the happy path (#938)
The device-only cliff recoveries replace hard aborts with a silent
download-and-continue, so the counters are now the only thing that
surfaces a gate/dispatch lockstep break. GPU_DEVICE_ONLY_DOWNGRADES
(trace side) already has its == 0 guard here; its parts-side counterpart
did not, and its only readers were the > 0 assertions in
cuda_fallback_tests, which run with a fault deliberately armed.
Without this, a decline in the R2 comp-poly tree build on a device-only
table recovers, verifies and passes green, while every such table pays a
full parts D2H plus a CPU commit_bit_reversed and loses the resident
composition tree. The R4 DEEP site is already covered transitively (it
needs the trace to be device-only too, which moves the trace counter),
so this closes the R2 commit and R3 parts-OOD sites.
Zero is the right expectation: materialize_composition_parts_host
early-returns without bumping when the part evals are already populated,
so the counter only moves for a device-only table that had to pull its
parts back.
The message names both causes rather than blaming the gate, matching the
counter's own doc, which now allows a transient VRAM decline as well as
a gate miss.
---------
Co-authored-by: Mauro Toscano <12560266+MauroToscano@users.noreply.github.com>
0 commit comments