pack: extend expert packing to MQ4G256V2 (qt=44) and MQ4CG256 (qt=45) - #745
Open
ghazni101 wants to merge 1 commit into
Open
pack: extend expert packing to MQ4G256V2 (qt=44) and MQ4CG256 (qt=45)#745ghazni101 wants to merge 1 commit into
ghazni101 wants to merge 1 commit into
Conversation
try_load_packed_mq4_experts gated on a single constant MQ4_G256_QUANT_TYPE=13, forcing qt=44 (MQ4G256V2) models into the per-expert fallback path: 20,480 individual hipMalloc calls instead of 80 packed blobs. On RDNA3 with 24 GiB VRAM, the amdgpu driver overhead (page tables, BO metadata, alignment) for 20,400 extra buffer objects costs 4.36 GB of VRAM and 35 seconds of load time with zero quality benefit — all three formats share the same 136 B/group stride. Replace the constant with packable_mq4_dtype() which accepts qt=13/44/45 and maps each to its correct DType. Track the uniform dtype across all experts in a layer and propagate it to the WeightTensor views so kernel dispatch selects the correct V2/C kernels. qt=30 (MQ4G256Lloyd, 160 B/group) and qt=47 (MQ6G256V2, 200 B/group) remain excluded due to different strides. Measured on RX 7900 XTX (gfx1100), Ornith-1.5-35B-A3B MQ4R (qt=44): VRAM: 23.81 GB -> 19.45 GB (-4.36 GB) Load: 38.0s -> 2.8s (13.6x) Decode: 224 tok/s (unchanged) Correctness: verified (math + factual queries) Closes warpfront#744
Collaborator
|
Landed on beta at
Notes for the record: qt=45 is included in the packable set but expert packing is RDNA3-only and qt=45 GEMM is gfx12-only, so that arm is unexercised; qt=13 path is unchanged by inspection (no qt=13 expert artifacts exist here to run). Will close with attribution once #730 promotes beta. |
Kaden-Schutt
added a commit
that referenced
this pull request
Sep 11, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
try_load_packed_mq4_expertsgates onMQ4_G256_QUANT_TYPE = 13, so MQ4G256V2 (qt=44) and MQ4CG256 (qt=45) experts fall through to per-expert loading: 20,480 individualhipMalloccalls instead of 80 packed blobs. On gfx1100 (24 GiB VRAM), the amdgpu driver overhead for 20,400 extra buffer objects costs 4.36 GB of VRAM.All three formats share the same 136 B/group stride — only the 8-byte group header interpretation differs, which is handled by kernel dispatch on
gpu_dtype.Fix
Replace the constant gate with
packable_mq4_dtype()accepting qt=13/44/45. Track the uniform dtype across experts and propagate it toWeightTensorviews so kernel dispatch selects the correct V2/C kernels.qt=30 (MQ4G256Lloyd, 160 B/group) and qt=47 (MQ6G256V2, 200 B/group) remain excluded — different strides.
Measurements
RX 7900 XTX (gfx1100), Ornith-1.5-35B-A3B MQ4R (qt=44), max_seq=512, Q8 KV, MTP off:
hipMalloccallsDecode correctness verified (math + factual queries). V1 (qt=13) no regression.
Closes #744