Skip to content

pack: extend expert packing to MQ4G256V2 (qt=44) and MQ4CG256 (qt=45) - #745

Open
ghazni101 wants to merge 1 commit into
warpfront:masterfrom
ghazni101:fix/pack-mq4g256v2-experts
Open

pack: extend expert packing to MQ4G256V2 (qt=44) and MQ4CG256 (qt=45)#745
ghazni101 wants to merge 1 commit into
warpfront:masterfrom
ghazni101:fix/pack-mq4g256v2-experts

Conversation

@ghazni101

@ghazni101 ghazni101 commented Sep 10, 2026

Copy link
Copy Markdown
Contributor

Problem

try_load_packed_mq4_experts gates on MQ4_G256_QUANT_TYPE = 13, so MQ4G256V2 (qt=44) and MQ4CG256 (qt=45) experts fall through to per-expert loading: 20,480 individual hipMalloc calls instead of 80 packed blobs. On gfx1100 (24 GiB VRAM), the amdgpu driver overhead for 20,400 extra buffer objects costs 4.36 GB of VRAM.

All three formats share the same 136 B/group stride — only the 8-byte group header interpretation differs, which is handled by kernel dispatch on gpu_dtype.

Fix

Replace the constant gate with packable_mq4_dtype() accepting qt=13/44/45. Track the uniform dtype across experts and propagate it to WeightTensor views so kernel dispatch selects the correct V2/C kernels.

qt=30 (MQ4G256Lloyd, 160 B/group) and qt=47 (MQ6G256V2, 200 B/group) remain excluded — different strides.

Measurements

RX 7900 XTX (gfx1100), Ornith-1.5-35B-A3B MQ4R (qt=44), max_seq=512, Q8 KV, MTP off:

Metric Before After
VRAM after load 23.81 GB 19.45 GB
hipMalloc calls 20,872 472
Load time 38.0 s 2.8 s
Decode speed 224 tok/s 224 tok/s

Decode correctness verified (math + factual queries). V1 (qt=13) no regression.

Closes #744

try_load_packed_mq4_experts gated on a single constant MQ4_G256_QUANT_TYPE=13,
forcing qt=44 (MQ4G256V2) models into the per-expert fallback path: 20,480
individual hipMalloc calls instead of 80 packed blobs. On RDNA3 with 24 GiB
VRAM, the amdgpu driver overhead (page tables, BO metadata, alignment) for
20,400 extra buffer objects costs 4.36 GB of VRAM and 35 seconds of load time
with zero quality benefit — all three formats share the same 136 B/group stride.

Replace the constant with packable_mq4_dtype() which accepts qt=13/44/45 and
maps each to its correct DType. Track the uniform dtype across all experts in
a layer and propagate it to the WeightTensor views so kernel dispatch selects
the correct V2/C kernels. qt=30 (MQ4G256Lloyd, 160 B/group) and qt=47
(MQ6G256V2, 200 B/group) remain excluded due to different strides.

Measured on RX 7900 XTX (gfx1100), Ornith-1.5-35B-A3B MQ4R (qt=44):
  VRAM: 23.81 GB -> 19.45 GB (-4.36 GB)
  Load: 38.0s -> 2.8s (13.6x)
  Decode: 224 tok/s (unchanged)
  Correctness: verified (math + factual queries)

Closes warpfront#744
@Kaden-Schutt

Copy link
Copy Markdown
Collaborator

Landed on beta at cc93d01bc (merge of b1955f4e2), validated on RX 7900 XTX with ornith-1.5-35B-A3B .mq4r and .mq4 (all 20,480 expert tensors qt=44), Q8 KV, greedy:

  • VRAM after load: 24.45 → 20.10 GB (mq4r), 24.68 → 20.31 GB (mq4) — your −4.36 GB reproduces exactly.
  • routed MQ4 expert packing: 512 per-expert weight buffers -> 2 layer blobs on every candidate load; absent on baseline.
  • All 10 greedy battery answers byte-identical packed vs unpacked on both files; serve battery 5/5 + chain 5/5 on mq4r.
  • Decode unchanged: 240.9 → 241.1 tok/s (mq4r), 169.2 → 169.7 (mq4).
  • Load time: the 38 s → 2.8 s figure did not reproduce here — warm-page-cache sweep 3.70 → 2.97 s; cold loads are disk-bound on this host. The VRAM/alloc-count win is the durable part.

Notes for the record: qt=45 is included in the packable set but expert packing is RDNA3-only and qt=45 GEMM is gfx12-only, so that arm is unexercised; qt=13 path is unchanged by inspection (no qt=13 expert artifacts exist here to run). Will close with attribution once #730 promotes beta.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

MQ4G256V2 (qt=44) expert loading bypasses packed path — 4.4 GB VRAM overhead from 20k individual hipMalloc calls

2 participants