feat(offload): partial GPU offload for dense qwen3.5 — host-RAM spill + optional CPU execution

[aldrouil]

Author's note

This is a rather large change, so I am going to include a small preface here explaining the goal and purpose of this PR.

First off, like a lot of people, my PC is limited to 16 GB of VRAM. I have an AMD RX 9070 XT (gfx1201), not a datacenter class GPU with more than that like a 9070 PRO, or even a 3090, 4090, or 5090 from Nvidia (which are out of scope for hipfire but are relevant for the discussion from a VRAM standpoint). Because of that, I am limited in what I can run and fit on this PC. Dense models seam to be releasing in a few different size stages - 2b/4b models, 9b/12 models, and then at least for home users - larger 27b/30b models. The problem is that the larger 27b/30b dense models are at the capacity limit of a single 16GB GPU - even before you factor in KV cache allocation.

And unfortunately, the most average large GPUs tend to max out at 16GB of VRAM. Yes I know the 20GB 7900XT and 24GB 7900XTX exist. I don't have one. And other than the 9070 PRO, the max for the current generation of AMD GPUs is 16GB of VRAM.

If someone wants to run these models on a 16GB GPU with hipfire this leaves with two options.

1. Tiny context window effectively useless for agentic work.
2. Downsize to a smaller quant. I tested this myself and noticed some tool calls started to break on qwen3.8 on some of the 3 bit quants. The model would correct itself, but this is likely an artifact of a smaller quant.

Or option 3 which is use a different project like llama.cpp - but when models fit hipfire is significantly faster.

The goal of this PR is to add the ability for Hipfire to offload dense qwen3.5 based model layers - at the time of writing qwen3.5/3.6 MoE and qwen flash next are NOT supported - to system RAM. This enables two things:

1. A longer context window than would fit otherwise.
2. Dense models that would otherwise not fit on the relevant GPU.

Once this was done it was found that there was a signficant speed penalty for doing so. Much more than llama.cpp (using it as a reference performance target). An analysis of the reason why determined that was because llama.cpp uses it's CPU inference engine on it's CPU offloaded layers to avoid passing data back to the GPU. As it turns out, despite CPUs being much slower at inference it is faster to have a CPU do inference on locally available data rather than having the GPU read objects in system RAM over the PCIe link. As a result I had a subsequent agent add a basic SIMD AVX2 accelerated code path to hipfire to use on offloaded layers. I have tested a few of the CPU SIMD paths, but not all and welcome anyone doing so before this merge request is committed.

All that in mind there are a few design decisions I made.
a) By default hipfire defaults to it's 'old' behavior and does not attempt to offload layers to system RAM.
b) Even if you offload layers to system RAM, the CPU inference engine is disabled by default. With the CPU inference engine disabled the system passes all of it's data back to the GPU for inference - on my machine with a PCIe 4.0 x16 link the penalty was significant.

The code should be extensible to other dense LLM architectures.

There is one caveat that is worth mentioning though. CPU inference is not byte identical to GPU inference. That is not to say it cannot be, but it isn't. This was a deliberate design decision - but it should be noted that llama.cpp treats CPU and GPU inference the same way. CPU inference on llama.cpp is not byte identical to its CUDA/HIP/Vulkan, etc runtimes.

Summary

Adds opt-in partial GPU offload for the dense qwen35 family:
memory.gpu_layer_budget spills a contiguous prefix of layers to host RAM, so a
model — or a context length — that does not otherwise fit in VRAM can run; and
memory.offload_exec=cpu executes those spilled layers' weight-reading GEMVs on
the CPU instead of reading them over PCIe once per token. Both default to today's
behavior (every layer resident, weights read by the GPU), so an unconfigured run
is byte-identical to a resident one.

typed key	env	default	meaning
memory.gpu_layer_budget	HIPFIRE_GPU_LAYER_BUDGET	unset → fully resident	N keeps the last N layers on the GPU and spills the rest; auto/-1 defers to the engine (today: fully resident)
memory.offload_exec	HIPFIRE_OFFLOAD_EXEC	pcie	who multiplies a spilled layer's weights: pcie = the existing GPU kernels read the host-mapped weights over the link, cpu = execute them on the CPU
HIPFIRE_CPU_EXEC_TRACE	—	off (developer)	=1 prints one line per distinct CPU step shape

Motivation is capacity, not speed: offloaded layers are bandwidth-bound by design.
Unset, empty and unknown values fail closed (pcie, fully resident), so a typo can
neither move work to the CPU nor force an offload.

Which surface(s) does this touch?

• kernel — crates/rdna-compute, crates/hipfire-dispatch, crates/hip-bridge
• load — runtime load path (hfq, weight_backend), arch load*, hipfire-config, Cargo manifests
• serve — hipfire-runtime weight-GEMV seam (llama.rs, weight_backend.rs, hfq.rs); no hipfire-engine / hipfire-generate change
• arch crate(s): hipfire-arch-qwen35 (placement + load path); hipfire-arch-llama, hipfire-arch-qwen2 (declare no offload support)
• crates/hipfire-quantize / quant formats
• control plane — hipfire-tui (two Settings rows)
• docs / CI / scripts only
• policy files

Design notes

• Host memory is hipHostMalloc(hipHostMallocMapped), not a host-located VMM
  arena. Measured on gfx1201, holding 4096 MB: a hipMemCreate host arena is
  charged against the device heap 1:1 (device headroom 15360 → 11264 MB), while
  hipHostMalloc costs 0 MB. The VMM host primitive and its locality tag were
  deleted so the net-zero mechanism cannot be re-adopted; the invariant is asserted
  by crates/rdna-compute/examples/host_offload_headroom.rs, because byte-identity
  and coherence tests cannot see it.
• No kernel changes and no memory-class tag on the tensor type. Kernels keep
  dereferencing a GpuTensor; the device pointer is the host-mapped alias.
• Placement is resolved once at load (i_gpu_start): layers [0 .. i_gpu_start)
  spill, the contiguous tail stays resident, and token_embd/output_norm/lm_head
  are always resident. The policy arithmetic
  (hipfire_config::memory::largest_fitting_tail, resolve_i_gpu_start) is pure,
  GPU-free and unit-tested. auto currently resolves to fully resident — the
  arithmetic is landed, the device measurement that would drive it is not.
• A second reader entry point, not a host flag. HfqBackend::read_proj_host
  is Option<fn(&HfqFile, &mut Gpu, …)> because the host upload needs &mut Gpu
  and the existing read_proj fn-pointer contract cannot hand one out. A spilled
  layer whose arch supplies no host reader is a hard error, never a silent device
  allocation.
• Fail-closed around the f32-dequant fallback: the catch-all dispatch arm and
  qt 1 Native mode have no host upload, so they refuse for offloaded layers rather
  than quietly consuming the VRAM the offload exists to free.
• The AWQ sidecar stays device-resident on purpose: a 1-D f16 vector of length K
  would cost PCIe bandwidth per GEMV to save kilobytes.
• hipGraph capture stays on for the pcie spill (host-mapped addresses are
  stable for the model's lifetime, replay verified byte-identical). It is disabled
  for the model's lifetime only when CPU execution is active, because a CPU step is
  a host sync point.
• One seam for CPU execution, in hipfire-dispatch's execute_steps: a fused
  entry spanning a CPU step is not matched, and weight_gemv_swiglu_residual splits
  itself (GPU silu_mul_f32, then the CPU GEMV + residual). Attention, softmax,
  rmsnorm, RoPE, qk-norm, the KV write, the flash-attention families and the
  DeltaNet recurrence stay on the GPU, and KV residency is unchanged either way.
• Retained replay is refused, not emulated: memory.offload_exec=cpu with a
  Redline backend fails the load naming both keys, because the tape records GPU
  launches and would replay stale activations.
• New leaf crate hipfire-cpu (depends on rayon only, GPU-free tests). It
  transcribes the canonical decoder; crates/hipfire-runtime/tests/cpu_quant_cross_check.rs
  holds the two copies together bit-for-bit. All 28 dense-capable formats decode,
  each with an AVX2 row dot and a scalar fallback (all-scalar on ARM). Map:
  docs/quant-formats/cpu-simd-coverage.md.

Numbers

Behind/after are the same fixtures with the spill dialed in; resident control
included. Arms interleaved, fresh process per point, host otherwise idle.

fixture	spill	pcie	cpu	resident control
qwen3.5-9b.mq4 (--spec off, 128 tok)	8 / 32 layers (888 MB/token)	21.6 tok/s	24.1 tok/s (+11.6 %)	65.7 tok/s at 0 spilled
qwen3.5-9b.mq4	16 / 32 layers	11.0 tok/s	14.0 tok/s (+27 %)	—
qwen3.8-27b.mq3-xt (64 layers)	8 / 64 layers (1.18–1.24 GB/token)	13.20 tok/s	15.20 tok/s (+15.2 %)	40.7 tok/s

• Fixture identity (the recorded runs below — not the battery.json attachment, whose 9B is a
  different artifact): 9B md5 31a8d8dc7603226801b08d8319015602, prompt
  benchmarks/prompts/humaneval_3_below_zero.txt md5 37c5aad9f9efe93b5c47f27256bdf149,
  daemon md5 6b2f85588aac1abdbd233e415044eb12, hipfire md5 7075cd546b1ce61d39a36d59ce149cba.
  27B md5 80bb9198e6a565fc006b2ae1b7c89eca, probe prompt md5 5835c71e471849b4a72e1dc8e39695e7,
  daemon md5 e39c3adb85f608db1439b128471764e8, hipfire md5 57a0cf5795a8ce376d274eba78bc733a.
  Host: 1× RX 9070 XT (gfx1201, 16304 MB), ROCm/HIP 7.2, Ryzen 7 7800X3D, 28 GB RAM.
• vram_free_mb is identical across arms at the same budget, and the 27B's own
  footprint (~13070 MB) matches stock — the resident path costs no extra VRAM. An
  offloaded=N log line is not evidence on its own; this control is.
• The CPU path is not a throughput trick, it changes the bound: the spill costs
  ≈41 ms/token on cpu versus ≈51 ms/token on pcie, against the link's measured
  27.1 GB/s. Remaining headroom is the kernel, not the design — the same row dot
  streams a 195.8 MB host-mapped buffer at 44.5 GB/s over 16 rayon threads.
• One earlier pass reported the cpu arm as 3–6× slower; it was measured while a
  workspace build was running on the same 16 threads. Recorded as superseded rather
  than deleted, because the way it was produced is the reusable lesson.
• These rows come from the records below, not from the battery.json attachment — the 9B fixture
  attached there is a different artifact (md5 296092bf…) from the records' 9B (31a8d8dc…).
• Full records, methods and raw JSON: docs/perf-checkpoints/2026-09-27-*
  (lifecycle historical, exploratory, not product claims).

Test plan

• ./scripts/no-gpu-ci.sh passes — red on this host, for a pre-existing reason:
  its Python phase fails 5 tests in tests/test_mq4c_repack.py
  (mq4c_repack has no attribute main / HfqmError / parse_hfqm_index). Reproduces
  identically in an unmodified checkout of this repository, and this change touches
  no file under tools/ or tests/. The Rust phases of the same script are green
  (cargo check --workspace --examples, hipfire-cpu lib 26 passed, hipfire-config
  89 passed, env-docs drift clean).
• cargo build --release clean — cargo build --release --locked -p hipfire-daemon -p hipfire-cli
  finished, warnings only
• cargo test --lib --workspace passes — every crate's lib target green
  (hipfire-runtime 691, rdna-compute 251, hipfire-dispatch 273, hipfire-arch-qwen35 199,
  hipfire-quantize 49, …), 0 failed
• load / serve / kernel changes: ran serve_harness.py --mode battery (and --mode chain)
  myself on hardware, with a real spill on the cpu arm: 5/5 turns each,
  runaway=0 empty=0 attractor=0 retrieval_miss=0, avg_decode=26.9 tok/s (chain: prefix-cache
  hits at 388/717/828/952 tokens), decoded and eyeballed — each turn's assistant_content is in
  the attachment below. Fixture qwen3.5-9b.mq4 sha256 ba83acf5…, md5 296092bf…;
  target/release/daemon md5 1c6e1c0a… (the harness reports the same digest it ran). The daemon
  serve log proves the spill rather than assuming it: partial offload: 24 resident / 8 offloaded, i_gpu_start=8 and cpu exec: 8/8 spilled layers fully covered; uncovered quants: none.
  Separately, redline_daemon_harness.py on the pcie arm with a spill is recorded in
  docs/perf-checkpoints/2026-09-27-gfx1201-cpu-exec-offload.md (stable=True, launches=427,
  AQL/PM4 shadow exact=True; on the cpu arm it fails the load with the intended refusal).
• If perf-relevant: ./scripts/speed-gate.sh within ±2% of locked baselines — not run; it
  needs a 4B fixture, and the resident path this gate protects is covered by the byte-identity
  (zero-diff) guard in the records above, while the spill's cost is documented rather than hidden.
• If this raises a ceiling in scripts/leanup-thresholds.txt: RATCHET-RAISE …

local serve_harness battery.json (load / serve / kernel changes)

Command

HIPFIRE_GPU_LAYER_BUDGET=24 HIPFIRE_OFFLOAD_EXEC=cpu \
HIPFIRE_DAEMON_BIN="$PWD/target/release/daemon" \
python3 scripts/serve_harness.py --model ~/.hipfire/models/qwen3.5-9b.mq4 \
  --mode battery --sampling greedy --speculation off --dflash off --thinking off \
  --out battery_9b_cpu_b24.json

--speculation off --dflash off --thinking off: no τ in the picture, and a reasoning
model cannot close a think block inside this budget (the daemon fails that turn closed
and releases no text). --mode chain was run with the same flags (cached=388/717/828/952,
5/5 turns, avg_decode=26.9).

Fixture identity

artifact	digest
qwen3.5-9b.mq4 (5,313,750,016 B)	sha256 ba83acf5bfd5d4e334b0afc26d779734e31623bb7f74e807c3581dfecb3128ad, md5 296092bf1e6a45d78c1acf815eb93366
target/release/daemon	md5 1c6e1c0ae86ba651d446fe1c833a729b (the harness prints this as daemon_binary_md5; it matches the build under test)
target/release/hipfire	md5 dc305c1c6de2119ed03a9376b42cfdcb
host	1× RX 9070 XT (gfx1201, 16304 MB), ROCm/HIP 7.2

This is not the qwen3.5-9b.mq4 of the perf-checkpoint records (those pin md5
31a8d8dc7603226801b08d8319015602 at 5,297,456,128 B), so these numbers are a
separate fixture and are not comparable with that record's.

The spill actually engaged (daemon serve log)

  partial offload: 24 resident / 8 offloaded, i_gpu_start=8
cpu exec: 8/8 spilled layers fully covered; uncovered quants: none
cpu exec: hipGraph capture disabled (CPU-executed steps present)

Per-turn rows

turn	ctx	gen	finish	decode tok/s	ttft s	attractor	empty	runaway
code	44	343	stop	26.9	1.026	false	false	false
reason	55	286	stop	26.9	0.348	false	false	false
factual	25	81	stop	27.1	0.163	false	false	false
prose	33	102	stop	26.9	0.249	false	false	false
instruct	31	82	stop	26.9	0.165	false	false	false

turns=5 runaway=0 empty=0 attractor=0 retrieval_miss=0 avg_prefill=165.1tok/s avg_decode=26.9tok/s

harness --out JSON (verbatim)

[
{
"request_id": "chatcmpl-1023366-1",
"ctx": 44,
"cached": 0,
"gen": 343,
"finish": "stop",
"think_words": 0,
"ans_words": 178,
"prefill_ms": 969.5,
"prefill_tok_s": 45.4,
"decode_tok_s": 26.9,
"decode_estimated": false,
"tau": null,
"cycles": null,
"dflash": null,
"mtp": null,
"mtp_ngram": null,
"ngram_mod_windows": null,
"ngram_mod_drafts": null,
"ngram_mod_accepted": null,
"ngram_mod_accept_rate": null,
"mtp_windows": null,
"ar_windows": null,
"mtp_retired": null,
"mtp_window_timings": null,
"ttft_s": 1.026,
"wall_s": 13.713,
"attractor": false,
"empty": false,
"runaway": false,
"ans_preview": "```python\ndef merge_sorted(a, b):\n    \"\"\"\n    Merge two already-sorted lists into one sort",
"assistant_content": "```python\ndef merge_sorted(a, b):\n    \"\"\"\n    Merge two already-sorted lists into one sorted list.\n\n    This function assumes that both input lists `a` and `b` are sorted in ascending order.\n    It returns a new list containing all elements from both inputs, also sorted in ascending order.\n    The implementation uses a two-pointer approach with O(n + m) time complexity.\n\n    Parameters:\n        a (list): First sorted list of comparable items.\n        b (list): Second sorted list of comparable items.\n\n    Returns:\n        list: A new merged sorted list containing all elements from `a` and `b`.\n\n    Example:\n        >>> merge_sorted([1, 3, 5], [2, 4, 6])\n        [1, 2, 3, 4, 5, 6]\n    \"\"\"\n    i, j = 0, 0\n    merged = []\n\n    # Traverse both lists and append the smaller element to the result\n    while i < len(a) and j < len(b):\n        if a[i] <= b[j]:\n            merged.append(a[i])\n            i += 1\n        else:\n            merged.append(b[j])\n            j += 1\n\n    # Append any remaining elements from list a\n    while i < len(a):\n        merged.append(a[i])\n        i += 1\n\n    # Append any remaining elements from list b\n    while j < len(b):\n        merged.append(b[j])\n        j += 1\n\n    return merged\n```",
"content": "```python\ndef merge_sorted(a, b):\n    \"\"\"\n    Merge two already-sorted lists into one sorted list.\n\n    This function assumes that both input lists `a` and `b` are sorted in ascending order.\n    It returns a new list containing all elements from both inputs, also sorted in ascending order.\n    The implementation uses a two-pointer approach with O(n + m) time complexity.\n\n    Parameters:\n        a (list): First sorted list of comparable items.\n        b (list): Second sorted list of comparable items.\n\n    Returns:\n        list: A new merged sorted list containing all elements from `a` and `b`.\n\n    Example:\n        >>> merge_sorted([1, 3, 5], [2, 4, 6])\n        [1, 2, 3, 4, 5, 6]\n    \"\"\"\n    i, j = 0, 0\n    merged = []\n\n    # Traverse both lists and append the smaller element to the result\n    while i < len(a) and j < len(b):\n        if a[i] <= b[j]:\n            merged.append(a[i])\n            i += 1\n        else:\n            merged.append(b[j])\n            j += 1\n\n    # Append any remaining elements from list a\n    while i < len(a):\n        merged.append(a[i])\n        i += 1\n\n    # Append any remaining elements from list b\n    while j < len(b):\n        merged.append(b[j])\n        j += 1\n\n    return merged\n```",
"reasoning_content": "",
"tool_calls": [],
"request_md5": "b2d411eb58ec2648ebd087ac828a51bb",
"atem_leak": false,
"terminal_count": 1,
"terminal_reasons": [
"stop"
],
"post_terminal_bytes": 0,
"saw_done": true,
"stream_error": null,
"prompt_md5": "43ca0d15712d3dfb777b51ae76d8fd5f",
"expected_substrings": [],
"retrieval_missing": []
},
{
"request_id": "chatcmpl-1023366-3",
"ctx": 55,
"cached": 0,
"gen": 286,
"finish": "stop",
"think_words": 0,
"ans_words": 147,
"prefill_ms": 304.1,
"prefill_tok_s": 180.9,
"decode_tok_s": 26.9,
"decode_estimated": false,
"tau": null,
"cycles": null,
"dflash": null,
"mtp": null,
"mtp_ngram": null,
"ngram_mod_windows": null,
"ngram_mod_drafts": null,
"ngram_mod_accepted": null,
"ngram_mod_accept_rate": null,
"mtp_windows": null,
"ar_windows": null,
"mtp_retired": null,
"mtp_window_timings": null,
"ttft_s": 0.348,
"wall_s": 10.946,
"attractor": false,
"empty": false,
"runaway": false,
"ans_preview": "To find the total distance traveled, we need to calculate the distance for each segment of",
"assistant_content": "To find the total distance traveled, we need to calculate the distance for each segment of the trip separately and then add them together.\n\nThe formula for distance is:\n$$ \\text{Distance} = \\text{Speed} \\times \\text{Time} $$\n\n**Step 1: Calculate the distance for the first part of the trip.**\n*   Speed ($v_1$) = $60$ mph\n*   Time ($t_1$) = $2.5$ hours\n$$ d_1 = 60 \\times 2.5 = 150 \\text{ miles} $$\n\n**Step 2: Calculate the distance for the second part of the trip.**\n*   Speed ($v_2$) = $40$ mph\n*   Time ($t_2$) = $1.5$ hours\n$$ d_2 = 40 \\times 1.5 = 60 \\text{ miles} $$\n\n**Step 3: Add the distances together to find the total distance.**\n$$ \\text{Total Distance} = d_1 + d_2 $$\n$$ \\text{Total Distance} = 150 + 60 = 210 \\text{ miles} $$\n\n**Final Answer:**\nThe train traveled a total of **210** miles.",
"content": "To find the total distance traveled, we need to calculate the distance for each segment of the trip separately and then add them together.\n\nThe formula for distance is:\n$$ \\text{Distance} = \\text{Speed} \\times \\text{Time} $$\n\n**Step 1: Calculate the distance for the first part of the trip.**\n*   Speed ($v_1$) = $60$ mph\n*   Time ($t_1$) = $2.5$ hours\n$$ d_1 = 60 \\times 2.5 = 150 \\text{ miles} $$\n\n**Step 2: Calculate the distance for the second part of the trip.**\n*   Speed ($v_2$) = $40$ mph\n*   Time ($t_2$) = $1.5$ hours\n$$ d_2 = 40 \\times 1.5 = 60 \\text{ miles} $$\n\n**Step 3: Add the distances together to find the total distance.**\n$$ \\text{Total Distance} = d_1 + d_2 $$\n$$ \\text{Total Distance} = 150 + 60 = 210 \\text{ miles} $$\n\n**Final Answer:**\nThe train traveled a total of **210** miles.",
"reasoning_content": "",
"tool_calls": [],
"request_md5": "85b33e9aedabf7a9fa0d1d81cc6f8356",
"atem_leak": false,
"terminal_count": 1,
"terminal_reasons": [
"stop"
],
"post_terminal_bytes": 0,
"saw_done": true,
"stream_error": null,
"prompt_md5": "640e0fd4f55996cb175a422f0a12cef5",
"expected_substrings": [],
"retrieval_missing": []
},
{
"request_id": "chatcmpl-1023366-5",
"ctx": 25,
"cached": 0,
"gen": 81,
"finish": "stop",
"think_words": 0,
"ans_words": 70,
"prefill_ms": 126.1,
"prefill_tok_s": 198.2,
"decode_tok_s": 27.1,
"decode_estimated": false,
"tau": null,
"cycles": null,
"dflash": null,
"mtp": null,
"mtp_ngram": null,
"ngram_mod_windows": null,
"ngram_mod_drafts": null,
"ngram_mod_accepted": null,
"ngram_mod_accept_rate": null,
"mtp_windows": null,
"ar_windows": null,
"mtp_retired": null,
"mtp_window_timings": null,
"ttft_s": 0.163,
"wall_s": 3.122,
"attractor": false,
"empty": false,
"runaway": false,
"ans_preview": "The primary cause of Earth's seasons is the tilt of its rotational axis relative to its or",
"assistant_content": "The primary cause of Earth's seasons is the tilt of its rotational axis relative to its orbital plane around the Sun. As the planet orbits, this fixed tilt causes different hemispheres to receive varying amounts of direct sunlight and experience changes in day length throughout the year. Consequently, when a hemisphere leans toward the Sun, it experiences summer due to more intense solar radiation, while the opposite hemisphere experiences winter.",
"content": "The primary cause of Earth's seasons is the tilt of its rotational axis relative to its orbital plane around the Sun. As the planet orbits, this fixed tilt causes different hemispheres to receive varying amounts of direct sunlight and experience changes in day length throughout the year. Consequently, when a hemisphere leans toward the Sun, it experiences summer due to more intense solar radiation, while the opposite hemisphere experiences winter.",
"reasoning_content": "",
"tool_calls": [],
"request_md5": "d34383b66fd5f7af0cfced19d3777983",
"atem_leak": false,
"terminal_count": 1,
"terminal_reasons": [
"stop"
],
"post_terminal_bytes": 0,
"saw_done": true,
"stream_error": null,
"prompt_md5": "8f66b4c97988825bd8e7840aaf44357e",
"expected_substrings": [],
"retrieval_missing": []
},
{
"request_id": "chatcmpl-1023366-7",
"ctx": 33,
"cached": 0,
"gen": 102,
"finish": "stop",
"think_words": 0,
"ans_words": 81,
"prefill_ms": 211.1,
"prefill_tok_s": 156.3,
"decode_tok_s": 26.9,
"decode_estimated": false,
"tau": null,
"cycles": null,
"dflash": null,
"mtp": null,
"mtp_ngram": null,
"ngram_mod_windows": null,
"ngram_mod_drafts": null,
"ngram_mod_accepted": null,
"ngram_mod_accept_rate": null,
"mtp_windows": null,
"ar_windows": null,
"mtp_retired": null,
"mtp_window_timings": null,
"ttft_s": 0.249,
"wall_s": 4.01,
"attractor": false,
"empty": false,
"runaway": false,
"ans_preview": "Elias tended his lonely lighthouse for decades, watching the endless waves crash against t",
"assistant_content": "Elias tended his lonely lighthouse for decades, watching the endless waves crash against the jagged rocks below. One stormy night, a strange object tumbled onto the shore, glowing with an eerie, pulsating blue light that defied the darkness. He approached cautiously to investigate and discovered it was not debris, but a small, intricate clockwork device humming with impossible energy. Realizing this invention could change everything, Elias knew his life of solitude had just become the most significant chapter of his story.",
"content": "Elias tended his lonely lighthouse for decades, watching the endless waves crash against the jagged rocks below. One stormy night, a strange object tumbled onto the shore, glowing with an eerie, pulsating blue light that defied the darkness. He approached cautiously to investigate and discovered it was not debris, but a small, intricate clockwork device humming with impossible energy. Realizing this invention could change everything, Elias knew his life of solitude had just become the most significant chapter of his story.",
"reasoning_content": "",
"tool_calls": [],
"request_md5": "643a78544be2dab1edcfe5b8e85c3718",
"atem_leak": false,
"terminal_count": 1,
"terminal_reasons": [
"stop"
],
"post_terminal_bytes": 0,
"saw_done": true,
"stream_error": null,
"prompt_md5": "8fe0ad36f61bcf4992cc9df81cdf3817",
"expected_substrings": [],
"retrieval_missing": []
},
{
"request_id": "chatcmpl-1023366-9",
"ctx": 31,
"cached": 0,
"gen": 82,
"finish": "stop",
"think_words": 0,
"ans_words": 62,
"prefill_ms": 126.7,
"prefill_tok_s": 244.6,
"decode_tok_s": 26.9,
"decode_estimated": false,
"tau": null,
"cycles": null,
"dflash": null,
"mtp": null,
"mtp_ngram": null,
"ngram_mod_windows": null,
"ngram_mod_drafts": null,
"ngram_mod_accepted": null,
"ngram_mod_accept_rate": null,
"mtp_windows": null,
"ar_windows": null,
"mtp_retired": null,
"mtp_window_timings": null,
"ttft_s": 0.165,
"wall_s": 3.182,
"attractor": false,
"empty": false,
"runaway": false,
"ans_preview": "1. Use meaningful variable and function names that clearly describe their purpose.\n2. Keep",
"assistant_content": "1. Use meaningful variable and function names that clearly describe their purpose.\n2. Keep functions small, focused on a single responsibility, and easy to test.\n3. Write clear comments only for complex logic or non-obvious design decisions.\n4. Follow consistent coding conventions and formatting rules across the entire project.\n5. Refactor code regularly to remove technical debt and improve readability over time.",
"content": "1. Use meaningful variable and function names that clearly describe their purpose.\n2. Keep functions small, focused on a single responsibility, and easy to test.\n3. Write clear comments only for complex logic or non-obvious design decisions.\n4. Follow consistent coding conventions and formatting rules across the entire project.\n5. Refactor code regularly to remove technical debt and improve readability over time.",
"reasoning_content": "",
"tool_calls": [],
"request_md5": "9af4a6122e824b100c1d31cb525d7816",
"atem_leak": false,
"terminal_count": 1,
"terminal_reasons": [
"stop"
],
"post_terminal_bytes": 0,
"saw_done": true,
"stream_error": null,
"prompt_md5": "8bed8e2d056dc1d47dccae9d32dbecf4",
"expected_substrings": [],
"retrieval_missing": []
}
]

Correctness on the changed surface, in more detail:

• cargo test -p hipfire-cpu --lib — 26 passed (GPU-free; the crate is in
  scripts/no-gpu-ci.sh).
• cargo test -p hipfire-config — 89 passed.
• python3 scripts/check-env-docs.py — clean.
• Production-launcher vs hipfire_cpu::gemv parity on identical bytes, both arms of
  the rotation contract: worst 9.3e-7 relative against a 1e-4 tolerance, 15 of 23
  formats bit-exact. qt 49 (every projection of the 27B MQ3 fixture, and a format
  with no canonical dequantize_to_f32 arm) is oracled on a real 27B tensor:
  1.195e-7 / 3.530e-7.
• dequant_group reproduces the canonical decoder bit-for-bit over 1695 real
  tensors. Zero-diff guard: stock ≡ this change ≡ spill+pcie, byte-identical greedy
  text (2B 1343 chars, 9B 2734 chars).
• The CPU path's contract is coherence plus a measured divergence, not bit-identity:
  9B, 8 spilled, greedy — both arms agree for 723 characters (~190 tokens) and then
  differ by one whitespace token; the path is deterministic across processes and
  across RAYON_NUM_THREADS=1, so it is not CPU-side scheduling. On the 27B MQ3
  fixture both completions are 610 B and byte-identical.
• Notes: the GPU-gated parity test is --ignored and fixture-dependent, so it is not
  in the GPU-free CI set; two formats (qt 11/12) report SKIP — production launcher has no kernel on this arch on gfx12. rustfmt --check --config skip_children=true
  is clean on every file this branch touches, and clippy is an advisory job.
• scripts/verify-bind-thread.sh (a pre-commit hook, not part of the required CI
  gates job) reports 12 of 94 pub fn in impl Gpu missing the bind_thread
  contract against 8 of 86 before this change. The four added host-mapped entry
  points were fixed in this branch — the two allocators bind the calling thread, the
  two state queries carry the gate's reasoned skip marker. The remaining 8 are
  byte-identical to the pre-existing set and are left alone.

Known limits

• Placement is whole-layer and qwen35-only. The CPU-execution seam itself is
  arch-agnostic (hipfire-runtime/src/llama.rs routes the llama-family
  down-projection through it); llama and qwen2 declare no offload support, and
  MoE/A3B is structurally out of reach — load_moe_ffn never sees the offload flag.
• Prefill and the slots/serve body do not use the CPU seam: batched GEMM there
  never enters execute_steps, so their spilled weights still cross PCIe.
• KV cache is not spilled (attention stays resident).
• Pinned host memory is unreclaimable, so keep the spilled prefix small: on the
  27B MQ3 fixture, 8 of 64 layers (≈1.2 GB pinned) loads; 16 (≈2.5 GB) stalled twice
  with 0 GB free.
• The 27B MQ3 fixture has one decoded read, not a sweep: a greedy completion, 610 B and
  byte-identical between the cpu and pcie arms (cpu.txt / pcie.txt in
  docs/perf-checkpoints/data-2026-09-27-cpu-exec-mq3-avx2/). The earlier record's "no decoded
  text was read on this fixture" is a gap in that earlier run, closed by the later one — but a
  second prompt or genre is still absent, and there is no ≥3-process arm-crossing byte diff on
  this fixture — the cross-process evidence above is the 9B cpu arm's determinism, which is a
  different property.
• A dead-code warning remains in the new crate (mq4g256_row_dot_scalar is now
  unreachable).

Hardware validation request (optional)

Omitted deliberately: the CPU-executed path needs HIPFIRE_GPU_LAYER_BUDGET and
HIPFIRE_OFFLOAD_EXEC set at load, which a registry-tag route cannot express, and a
route without the spill proves nothing about this change.

How this merges (direct review)

Merge authority is direct maintainer review plus the required no-GPU CI checks.
hw-gate automation is optional evidence delivery, not a prerequisite or substitute.

1. Required CI (must stay green): build (workspace, no GPU), unit tests (lib, no GPU), gates (ratchets, layering, registers) from .github/workflows/ci.yml.
2. Required review: one approving maintainer review. The reviewer judges claim-matched evidence for the surfaces you ticked.
3. Evidence you owe: pick routes from docs/VALIDATION.md. Prefer attaching local harness output (serve_harness.py, redline_daemon_harness.py, test_kernels, etc.). A lone hipfire run transcript is not evidence.
4. Optional automation: hw-gate is not a required status check.
5. Retired paths: scripts/coherence-gate*.sh and tools/change_gate / agentic-review are historical only — never acceptance evidence.

Architecture-trait change?

No — crates/hipfire-runtime/src/arch.rs (the Architecture trait) is unchanged.

[fivetide]

Thanks for tackling this. Capacity offload is something we want, and the measurement discipline here is excellent. The mechanism layer is in good shape: the host-mapped primitive with an ownership registry and cleanup check in rdna-compute, the DeviceBuffer::is_host_mapped tag, the HfqBackend reader seam, and the CPU seam in execute_steps are all in the right places.

Our main concern is architectural. Hipfire's direction is that arch crates should contain as little code as possible, ideally typed declarations, with loading, placement and dispatch handled centrally. In this PR, the decision about which layers spill (and everything that follows from it) lives in hipfire-arch-qwen35. As a result, every new arch would have to copy it, and even qwen35's own PARO and MoE paths already fall through the gaps. Concrete suggestions:

1. Placement belongs in the loader, not in Qwen35Config. model_load::Layout is already the arch-generic "where does each layer land" object. Give it a per-layer Residency { Device, HostMapped }. Resolve it once from memory.gpu_layer_budget in the shared load path, and pass it through WeightSource::read_layer(gpu, layer, residency) into HfqBackend. llama and qwen2 already build HfqBackend, so they would get offload almost for free. For the manifest route (weight_manifest::WeightPlacement / fulfill_manifest, currently piloted on llama), the same tier becomes a field on the placement. A contiguous spilled prefix is really just a pipeline split whose first stage is the host tier (DeviceMesh::stage_for_layer already bands layers).
2. One upload path, not a qwen35 twin. Rather than parameterizing qwen35's private quant-type match (whose doc comment already says it should be pulled into runtime), consolidate it into weight_backend::decode_raw_codec / RAW_CODECS and give that function a residency/target parameter. Change the read_proj signature to take &mut Gpu. The "&Gpu-only fn-pointer contract" is our own choice and can change, which removes the need for read_proj_host as a second entry point.
3. Downstream decisions should read residency from loaded weights, not config. Dispatch already asks gpu.host_located(buf), which is the right shape. Record the execution target (GPU over PCIe vs CPU) on the weight at load time. Then graph eligibility, the Redline refusal, admission, and device-side derived caches (e.g. ensure_fp16_shadow) can all ask the Gpu/weights "do you own host-executed weights?" instead of config.i_gpu_start plus a process-global LazyLock.
4. Gate capture centrally. Put the refusal in GraphState::begin_graph_capture* and the replay controller's begin_capture, instead of two qwen35 call sites (forward.rs:1694, forward_slots.rs:2518). MTP proposal, DFlash draft, dense-TP and verify graphs aren't covered today, and mtp_spec.rs / dflash.rs call hip.stream_begin_capture directly, which bypasses the capture_mode guard in memcpy_dtoh_auto. The harness runs used --speculation off, so spec decode + offload_exec=cpu is currently unvalidated. The Redline refusal should live in hipfire-loader, where the replay backend is chosen, not in qwen35 arch.rs.
5. Admission from what the loader actually allocated. Have the loader report device bytes and pinned-host bytes. Feed ModelFootprint from that, and charge pinned host memory to the existing host tier in AdmissionController (unreclaimable hipHostMalloc is exactly what that tier guards against — the PR notes 16/64 layers stalling with 0 GB free). That removes the tensor-name parsing (tensor_layer_index) in serve_engine.
6. Fail closed on sources that can't spill yet. i_gpu_start is set from config regardless of source, but ParoSource::read_layer never sees it and load_moe_ffn places experts in VRAM unconditionally. Those loads log "N offloaded" while resident_weight_bytes subtracts the full layer bytes (experts included) from the admission charge, so a 16 GB card can be over-admitted into OOM. Until residency flows through PARO, MoE experts and VL, gpu_layer_budget should fail the load for them. Same for UMA devices (e.g. Strix Halo), where spilling to host frees nothing.
7. One CPU decoder. hipfire-cpu is at layer 3 and hipfire-runtime at layer 5, so runtime can depend on it. Make hipfire-cpu the canonical CPU dequant and have weight_backend::dequantize_to_f32 call it; the cross-check test and the transcription then go away. Ideally the CPU decoder becomes part of the codec registry entry, so a new format is one row and cpu_quant_for disappears.
8. Use Steps for the fused FFN. Instead of hand-splitting weight_gemv_swiglu_residual in llama.rs, express SiLU + GemvResidual as Steps so the generic seam handles it. That's also where prefill/batched coverage would eventually plug in.
9. Trim scope so this can be reviewed. Please split out: the gemv_mq4g256 kernarg fix (a real bug, its own PR), the rdna-compute test lock and tolerance changes, the TUI rows (~640 LOC), and the ~30 perf-checkpoint data files. largest_fitting_tail / resolve_i_gpu_start and GpuLayerBudget::Auto aren't used in production yet: drop them, or land them in runtime next to admission.rs when auto actually decides something (placement arithmetic doesn't belong in hipfire-config, which is config vocabulary only).

Suggested order: (2) → (1) → (3/4/5) → CPU execution. After step (1) the RAM spill works for every arch that uses HfqBackend, and your qwen35 numbers should reproduce unchanged.

[aldrouil]

1. Summary

I had an agent go over the critiques this morning. After a fair amount of work, its results are below.

It removed a few of the commits that were requested to be removed, I will submit bugfix PRs for those later. Including the TUI sanitation. I had the agent restore the existing behavior while still adding in the TUI fields.

As a side effect of the requests you asked for, qwen2 dense models should work with CPU offload. This has not been tested. There are no qwen2 models in the hipfire registry.

---

2. The review, answered

#	review ask	what shipped
1	placement belongs in the loader, not Qwen35Config	Layout gained layer_residency, and spill_count / resolve_residency resolve memory.gpu_layer_budget once in load_weights_inner; WeightSource::read_layer(gpu, layer, residency) carries it into HfqBackend.residency. Qwen35Config::i_gpu_start and apply_offload_policy are gone. llama inherits placement, and qwen2 was moved onto the shared orchestrator so it inherits too — the one arch that was still bespoke. Manifest got the tier as asked: WeightPlacement.residency, populated in plan_manifest, and fulfill_manifest_single refuses a host-mapped placement rather than allocating it in the VRAM the spill exists to free
2	one upload path, not a qwen35 twin	The registry absorbed the six formats qwen35 decoded privately (qt 31/32/33/34/36/37), decode_raw_codec takes Residency, and read_proj takes &mut Gpu — which is what removed the need for the second entry point. read_proj_host, host_local and qwen35's ~700-line private quant match are deleted: qwen35/load.rs −938/+76. expected_payload_bytes replaced six hand-copied length checks
3	downstream decisions read residency off the loaded weights	WeightTensor::exec / WeightRef::exec are set once by the loader from residency × offload_exec, and the dispatch seam tests that recorded target instead of re-deriving locality. cpu_exec_enabled()'s process-global LazyLock and cpu_offload_active(i_gpu_start) are deleted, so no step re-derives the decision — and host-mapped weights under pcie correctly stay a GPU path
4	gate capture centrally	GraphState::cpu_exec_weights is set once per load and read by all four begin_*_capture entries, Gpu::begin_stream_capture and ReplayController::begin_capture. The two bypasses are closed: mtp_spec.rs and dflash.rs used to call hip.stream_begin_capture directly and now go through begin_graph_capture. The Redline refusal lives in the loader's single seam. The review's own caveat — "spec decode + offload_exec=cpu is currently unvalidated" — is now exercised (§3). One item is still config-adjacent: ensure_fp16_shadow remains dtype-derived, so flipping the exec target at a fixed budget needs a fresh load
5	admission from what the loader allocated	The loader reports LoadStats { device_bytes, host_pinned_bytes }, summed per tensor as it builds them. Those feed ModelFootprint directly, pinned host bytes are charged into the existing host tier, and serve_engine's tensor_layer_index + resident_weight_bytes are deleted — the preflight now runs on the loader's own numbers
6	fail closed on sources that cannot spill	WeightSource::spill_refusal refuses a budgeted load for PaRoQuant and for any num_experts > 0 model before a byte is allocated, and a unified-memory device is refused on the same seam. The VL arm is a documented non-refusal rather than a refusal: this source is the text tower and the vision sidecar loads off the layer path, so there is no half-applied branch inside it to refuse
7	one CPU decoder	hipfire-cpu is now a hipfire-runtime dependency and dequantize_to_f32 delegates to hipfire_cpu::quant::dequant_group for every format it covers, keeping inline arms only for the formats the crate does not name; the cross-check test and dequantize_weight_to_f32 went with the duplication. Making it canonical found a latent bug: the canonical decoder panicked on qt 49 while hipfire_cpu decodes it correctly. One table remains, argued in the review's own terms: cpu_quant_for is keyed by DType while the registry is keyed by quant_type, and several DTypes have no qt row, so folding the decoder into the registry row would make that mapping lossy — cpu_decoder_tables_agree_per_format pins the two in step
8	use Steps for the fused FFN	The CPU arm now builds a GemvResidual step and calls execute_steps, so the seam owns the decision, the rotation disposition and the launch. The GPU arms are deliberately untouched: FUSED_TABLE has no silu+gemv_residual key, so two generic steps do not re-fuse — substituting them deletes the fused_silu_mul_rotate_mq* arm and its AWQ divide-then-rotate ordering, on the path the zero-diff guard certifies
9	trim scope	Three of the four splits are their own branches cut from master: fix/gemv-mq4g256-kernargs (45e70134), chore/rdna-compute-gpu-test-serialization (b6c39a9d), docs/offload-perf-records (2bae014f). largest_fitting_tail, resolve_i_gpu_start and GpuLayerBudget::Auto are dropped. The TUI rows stay, and the TUI diff removes no behaviour-bearing line: 620 insertions / 11 deletions over 6 files, where the 11 are the generated crate-map rows (6) and five fmt-mandated reflows in app.rs (5) — moves, not deletions (the assertions they carry are all still present), and required because master's own app.rs fails rustfmt --check and scripts/ci-rustfmt-changed.sh checks whole changed files. write_value and persist_setting are byte-identical to master: an empty buffer is invalid for every numeric row, and clearing a set row is the row's Delete/Backspace (reset_selected_setting → writer::delete_key), the path every other row already uses. A parity test pins that over max_tokens, temperature, mmq_screen and the pre-existing nullable deepseek4_experts_per_token; the only new-code accommodation is that the new row's label renders the schema's null spelling as unset. The rows themselves are the keys' discoverability (both are settable from the CLI and the config file), and a TUI-only branch off master does not compile, because master carries neither key, which makes it a stacked PR that must merge with or after this one

The umbrella concern — arch crates holding as little as possible — is where the diff goes:
qwen35/load.rs lost 938 lines net of 76 in one commit, and placement, the reader, the
execution target, the capture gate, the refusal and the admission accounting now live in
hipfire-runtime / hipfire-dispatch.

3. Evidence, all of it from this machine (gfx1201, ROCm 7.2)

• Zero-diff guard, two fixtures. Greedy (-t 0), master's binary against this branch:
  qwen3.5-2b.mq4 (md5 9ed6628f2df83ef4b1c062afd4a85bfb) — stock ≡ branch ≡ 3/24 pcie
  spill, byte-identical text; qwen3.5-9b.mq4 (md5 296092bf1e6a45d78c1acf815eb93366) —
  stock ≡ branch resident, and 24/32 pcie spill ≡ resident.
• Capture gate. HIPFIRE_GRAPH=1 + offload_exec=cpu + spill: exactly one
  cpu exec: hipGraph capture disabled line, no assertion, coherent text — and the same with
  DFlash active, which is the spec-decode path the review flagged. Under pcie instead,
  capture stays on: zero disable lines, and redline_daemon_harness.py with a 24/32 spill
  returns pass=True — capture stable at 128/512/decode, 19 AQL contracts, shadow parity
  exact=True.
• Spec decode with a spill, both exec targets. DFlash + cpu: draft loaded, one disable
  line, no assertion. DFlash + pcie + spill + HIPFIRE_GRAPH=1: draft loaded, zero disable
  lines, and the output is byte-identical to the resident run.
• Redline refusal. Spill + offload_exec=cpu + replay.backend=redline: the load is
  refused, the message names both keys, no text is produced — the intended failure rather than
  a replayed stale route.
• Serve path. serve_harness.py battery and chain on the cpu-spill arm: 5/5 turns
  each, runaway=0 empty=0 attractor=0 retrieval_miss=0, decoded text read (a real
  merge_sorted, the correct axial-tilt answer, on-topic prose and instruct turns), and
  chain shows the prefix cache working (cached=431/682/797/943).
• Accounting against the device and the OS. LoadStats' device+host total is bit-invariant
  across budgets (5 300 590 592 B). The marginal anonymous-RSS growth of the spill matches
  host_pinned_bytes plus the 106 × 1 MiB host-tail pads to −0.01 % (−119 296 B on
  1 031 070 208 B). Measured VRAM sits a fixed ~300 MB above device_bytes while 920 MB of
  weights move between budgets, i.e. allocator granularity rather than a scaling error.
• Workspace gates. cargo build --release clean; cargo test --lib --workspace 2991
  passed / 0 failed (42 lib targets, rc=0, re-measured on the final commit); check-layering
  0/0; check-env-docs clean; rustfmt clean on all 145 changed .rs files; no-gpu-ci.sh Rust
  phases green. Test coverage is additive too, since the review flagged removals: the diff adds
  57 #[test]/#[tokio::test] attributes and removes 3 (net +54; 4092 → 4146 over
  crates/**/*.rs), and those three are the old f16_lm_head_mode_* shape — that policy now
  lives in hipfire-config::kernel::parse_lm_head_f16_native, which carries its own tests.
  Its Python phase fails five mq4c_repack tests for a reason independent of this diff: the
  module never defines parse_hfqm_index, HfqmError or main, this branch touches no
  Python, and both files were last modified 2026-08-18.

4. Replication against the pinned 2B, using the record's recipe

hipfire bench <2B> --spec off --runs 5 --warmups 2 --max-tokens 64 --backend noslots --workload stateless --prompt-file benchmarks/prompts/gpu_offload_probe.txt --json, one fresh
daemon per point, prompt md5 5835c71e471849b4a72e1dc8e39695e7 and model md5
9ed6628f2df83ef4b1c062afd4a85bfb — the same artifacts 2026-09-27-cpu-exec-spill-sweep-2b-9b-27b.md
pins.

budget (spilled)	pcie	cpu	ratio	record pcie	record cpu	record ratio
24 (0)	269.2	269.2	1.00	258.5	258.4	1.00
21 (3)	138.2	103.5	1.34	131.1	99.3	1.32
15 (9)	71.8	53.2	1.35	67.5	48.5	1.39
12 (12)	57.6	41.5	1.39	54.3	38.7	1.40
18 (6)	94.2	68.8	1.37	62.6	65.5	0.96

The pcie-vs-cpu ratio reproduces (1.34–1.39 against 1.32–1.40), and the record's observable
gates reproduce exactly: b24 cpu prints offload_exec=cpu but nothing is spilled with zero
capture-disabled lines, b12 cpu prints 12/12 spilled layers fully covered; uncovered quants: none with one, and pcie points print none. The ~4 % absolute offset is the host, not the
code — the no-spill control shows the same +4.2 % (269.2 vs 258.5), and scaling by it puts
every point within 1–3 %. Only b18 differs, and its recorded pcie median sits inside its own
round spread ([62.6, 62.6, 89.3], against our 94.2), so it is one depressed round in the
record rather than a disagreement; the other four points replicate.

The cpu arm carries no byte-identity claim by design: its contract is llama.cpp-level —
coherence plus a measured divergence — which is why only pcie is byte-checked.

Branch binaries for every number above: hipfire
2b3af98c16bf79d1e4b1bc1e6aa37e31, daemon 3cd0af9390cfaebeb2de099abd60bcb2.