Repository navigation
dasLLAMA: the audio towers on the CUDA home, q8 + f16 on every GPU home - #4473
Merged
Merged
Conversation
Contributor
There was a problem hiding this comment.
Note
Copilot was unable to complete this review at the selected tier due to model policy or plan restrictions.
🔵 Needs a closer look
The change spans multiple GPU backends and lane/tower routing policies with runtime- and hardware-dependent correctness/performance impacts, so static diff review alone can’t provide sufficient confidence without targeted validation on real CI lanes and hardware.
No open issues 🎉
What changed in this PR
This pull request updates dasLLAMA’s GPU tower routing so all audio tower families can serve on the CUDA home under DASLLAMA_GPU=1 DASLLAMA_CUDA=1, while standardizing the GPU lane policy (q8 unpinned by default; f16 when pinned or forced via DASLLAMA_AUDIO_LANE=16). It also includes supporting refactors, test expansions, and documentation/review-rule updates to match the new behavior.
Changes:
- Route audio tower families onto the CUDA home under the CUDA ask; refactor shared “rail/tower” bodies and seating infrastructure across backends.
- Enforce/validate lane selection defaults across GPU homes (q8 default; f16 when pinned/forced) and bump/align related image/tag behavior.
- Expand/update model-free, residency, tier, fleet-harness, and lane-policy tests; update review checklists and architecture/docs accordingly.
Assessment: This pull request is ready to be approved.
| File | Notes |
|---|---|
| skills/perf_lint.md | |
| skills/daslang/references/everything.md | Nit: visit is described as read-only but is overloaded for mutable and const adapters. |
| REVIEW.md | |
| modules/REVIEW.md | |
| modules/dasLLAMA/tests/test_run_suites.das | |
| modules/dasLLAMA/tests/test_qwen3v.das | |
| modules/dasLLAMA/tests/test_qwen25v.das | |
| modules/dasLLAMA/tests/test_pod_fleet.das | |
| modules/dasLLAMA/tests/test_gpu_tier.das | |
| modules/dasLLAMA/tests/test_gpu_resident_gemma4_e2b.das | |
| modules/dasLLAMA/tests/test_gpu_resident_cuda_gemma4.das | |
| modules/dasLLAMA/tests/test_gpu_home.das | |
| modules/dasLLAMA/tests/test_gemma3v.das | |
| modules/dasLLAMA/tests/test_f32_batch_tile_perms.das | |
| modules/dasLLAMA/tests/test_cuda_tower_g4a_chain.das | |
| modules/dasLLAMA/tests/test_cuda_tower_asr.das | |
| modules/dasLLAMA/tests/test_audio_embedder.das | |
| modules/dasLLAMA/tests/REVIEW_PINNED_GATES.md | |
| modules/dasLLAMA/tests/REVIEW_MEDIA_CELLS.md | |
| modules/dasLLAMA/tests/_tuned_fallbacks_root.das | |
| modules/dasLLAMA/tests/_tts_parity.das | |
| modules/dasLLAMA/tests/_rnnt_oracle.das | |
| modules/dasLLAMA/tests/_moe_resident.das | |
| modules/dasLLAMA/tests/_model_tier.das | |
| modules/dasLLAMA/tests/_gemma_resident.das | |
| modules/dasLLAMA/tests/_flavor_decline_root.das | |
| modules/dasLLAMA/tests/_ctc_oracle.das | |
| modules/dasLLAMA/tests/_compares.das | |
| modules/dasLLAMA/tests/_child_run.das | |
| modules/dasLLAMA/tests/_audio_lane_whisper_root.das | |
| modules/dasLLAMA/tests/_audio_lane_root.das | |
| modules/dasLLAMA/REVIEW_VISION.md | |
| modules/dasLLAMA/REVIEW_UPSTREAM.md | Nit: wrap the quoted <role> placeholder in backticks to avoid markdown/HTML rendering loss. |
| modules/dasLLAMA/REVIEW_RACE_PICKS.md | |
| modules/dasLLAMA/REVIEW_OVERRIDES.md | |
| modules/dasLLAMA/REVIEW_MEMORY.md | |
| modules/dasLLAMA/REVIEW_HOT_PATH.md | |
| modules/dasLLAMA/REVIEW_GPU_VULKAN_TILES.md | |
| modules/dasLLAMA/REVIEW_GPU_RESIDENT.md | |
| modules/dasLLAMA/REVIEW_GPU_MODEL_SWAP.md | |
| modules/dasLLAMA/REVIEW_GPU_KERNEL_BODY.md | |
| modules/dasLLAMA/REVIEW_GATES.md | |
| modules/dasLLAMA/REVIEW_FACADE.md | |
| modules/dasLLAMA/REVIEW_DOCS.md | |
| modules/dasLLAMA/REVIEW_AUDIO.md | |
| modules/dasLLAMA/README.md | |
| modules/dasLLAMA/HOW_TO_ADD_A_FORMAT.md | |
| modules/dasLLAMA/harness/vision_lane_probe.das | |
| modules/dasLLAMA/harness/README.md | |
| modules/dasLLAMA/followup_metal.md | |
| modules/dasLLAMA/dasllama/dasllama_vulkan_dispatch.das | |
| modules/dasLLAMA/dasllama/dasllama_tune.das | |
| modules/dasLLAMA/dasllama/dasllama_transformer.das | |
| modules/dasLLAMA/dasllama/dasllama_tower_image.das | |
| modules/dasLLAMA/dasllama/dasllama_tower_flavor.das | |
| modules/dasLLAMA/dasllama/dasllama_qwen3v.das | |
| modules/dasLLAMA/dasllama/dasllama_qwen25v.das | |
| modules/dasLLAMA/dasllama/dasllama_metal_prefill.das | |
| modules/dasLLAMA/dasllama/dasllama_metal_kernels.das | |
| modules/dasLLAMA/dasllama/dasllama_metal_asr_dec.das | |
| modules/dasLLAMA/dasllama/dasllama_image.das | |
| modules/dasLLAMA/dasllama/dasllama_gpu_math.das | |
| modules/dasLLAMA/dasllama/dasllama_gemma4v.das | |
| modules/dasLLAMA/dasllama/dasllama_gemma3v.das | |
| modules/dasLLAMA/dasllama/dasllama_dispatch_scan.das | |
| modules/dasLLAMA/dasllama/dasllama_cuda_tower_audio.das | Nit: comment mentions vulkan until CUDA seats land, but CUDA_AUDIO_HOMES maps all listed families to TowerDev.cuda; align comment or mapping. |
| modules/dasLLAMA/dasllama/dasllama_cuda_mtp_gemma.das | |
| modules/dasLLAMA/dasllama/dasllama_cuda_dispatch.das | |
| modules/dasLLAMA/benchmarks/lcpp_bench.das | |
| modules/dasLLAMA/ARCHITECTURE_MEDIA_TOWER.md | |
| modules/dasLLAMA/ARCHITECTURE_MEDIA_ASR.md | |
| modules/dasLLAMA/ARCHITECTURE_MEASUREMENT.md | |
| modules/dasLLAMA/ARCHITECTURE_MEASUREMENT_RELEASE.md | |
| modules/dasLLAMA/ARCHITECTURE_MEASUREMENT_AUDIO.md | |
| modules/dasLLAMA/ARCHITECTURE_IMAGE.md | |
| modules/dasLLAMA/ARCHITECTURE_IMAGE_LEDGER.md | |
| modules/dasLLAMA/ARCHITECTURE_IMAGE_GPU_PLANES.md | |
| modules/dasLLAMA/ARCHITECTURE_GPU_VULKAN_RESIDENCY.md | |
| modules/dasLLAMA/ARCHITECTURE_GPU_VULKAN_NROW.md | |
| modules/dasLLAMA/ARCHITECTURE_GPU_TOWER_FC.md | |
| modules/dasLLAMA/ARCHITECTURE_GPU_SHARED_ATTN.md | |
| modules/dasLLAMA/ARCHITECTURE_GPU_ASYMMETRIES.md | |
| modules/dasLLAMA/ARCHITECTURE_ENGINE_FORMATS.md | |
| modules/dasLLAMA/ARCHITECTURE_ENGINE_FORMATS_TERMS.md | |
| modules/dasLLAMA/.das_module | |
| modules/dasCUDA/REVIEW.md | |
| modules/dasCUDA/cuda/das_cuda_boost.das |
Review effort: Balanced
The router GEMV kernel holds eight columns. The PLE gate and projection rows of a short window launched it untiled with every row, so a tail past eight rows read past the kernel's columns and the first token went wrong. cud_router_enc_cols tiles every router launch, the CUDA twin of the Vulkan home's router_enc_cols. The E2B Q4_K_M peel cell holds tails of 2, 9 and 16 rows to the same tail stepped a row at a time. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
metal_dn_step_fused dispatches threadgroups of ds*4 lanes, 512 at the widest state. An entry that declares no size is capped below that on the M1, so the pipeline refused the dispatch. The entry now declares max_threads=512. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The cell read the home the environment spells and failed on a build that does not carry it (a Metal build under DASLLAMA_GPU's discrete names). It now skips by name, listing the homes the build has. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
test_vkd_gemv_ncol and the MoE resident fixture's tile census walked every KqFmt member. The MLX members have no kernel on the discrete homes, so the GEMV cell panicked in kq_tile_stamp on slot 21 and every resident MoE census read a kernel name nothing seeded. Both walks now skip the members kq_discrete_unstamped names. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
[dasllama_fallbacks] emitted its table from state the [tuned] macro filled while a kernel's module compiled. A module the front-end cache served ran no [tuned] apply, so a program whose own file changed got an empty table: test_f32_batch_tile_perms read no reference permutation and the tuner harness no shipped defaults. The table now reads each kernel's fallback off its [tuned] annotation. The new cell compiles one root twice over one cache record, the second copy a byte longer. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A batch-mate requiring the engine core brings the Vulkan home's entry, which arms the GPU tier at [init] under the runner's DASLLAMA_GPU=1, and no reset undoes an arm. test_gpu_tier's unset-seat contract and test_gpu_serving_declines' no-decline control then red in any batch that holds such a file, as master's own ten-file grouping does on a Vulkan box. The runner now hands each a child of its own. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The never-dispatched roster named the five nmt_*_cls stamps without the _spv suffix every class kernel's census key carries, so the Vulkan serving census panicked on a key nothing seeded on any box with the zoo. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Every audio family (whisper and the whisper-class chat towers, qwen3a, gemma4a, canary, parakeet, GigaAM's three lanes with the RNNT and CTC heads, the VAD) serves on the CUDA home through one chain body per family, generic over a rail the Vulkan and CUDA seats both build; the Vulkan bodies are deleted. Each GPU home now serves the same two lanes: q8 as the default and f16 (halfword weights, f32 accumulate) as the reference lane; the f32 device lanes are retired, f32 stays a CPU lane. The Metal home gains q8 and f16 for canary, parakeet and gemma4a; the gemma4a E4B projector tail serves at its two widths. CUDA encodes land ahead of Vulkan on every family but the VAD (the LSTM recurrence kernel), and the turbo f16 lane beats whisper.cpp f16 per window on both homes; the levers are the tower-only cm2 column model, graph capture of the tower chains, the shared-memory carveout left to the driver for the no-smem tower classes, and side streams for q/k/v. The LLM window keeps its own column pick. Fixed on the way: the Vulkan rail did not reset the norms base between chains (a FastConformer encode after a whisper one read stale offsets), the split-k sum ran over the f16 lane's whole output, the whisper-class exact stem declined on a q8-only layout field, a Vulkan K-quant GEMM site bound past offset 0 was wrong on both homes (enforced at 0). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
borisbat
force-pushed
the
bbatkin/asr-cuda
branch
from
November 7, 2026 07:21
d9ad19e to
0aef303
Compare
Contributor
There was a problem hiding this comment.
Note
Copilot was unable to complete this review at the selected tier due to model policy or plan restrictions.
🟡 Changes recommended
Changes are needed because there are still active review findings, notably that two “discrete” tower twin tests are Vulkan-gated and thus skip on CUDA-only builds leaving the CUDA tower path untested, along with a documentation line that inaccurately describes the overloaded visit bindings as read-only.
3 open issues
Review effort: Balanced
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Behavior change: every audio tower family serves on the CUDA home under
DASLLAMA_GPU=1 DASLLAMA_CUDA=1, and every GPU home's un-pinned audio lane is q8 (an exact ask: f16 on a GPU home, f32 on the CPU). An un-pinned Metal load of canary, parakeet or gemma4a now maps the q8 image.Why. Under the CUDA ask every audio family fell to the CPU chain; the Vulkan driver held each chain in its own body, so a CUDA twin would have copied every one.
What changes.
DASLLAMA_AUDIO_LANE=16.times.Observable behavior.
PERF_LEDGER.md).Where to look. The rail bodies (
dasllama_tower_chains*.das); the risky spots are the lane policy seam (audio_lane_serves_q8) and the CUDA router GEMV tiling (cud_router_enc_cols).Validation, claims, ledger
Validation
dasllama_tuned_fallbacks()emptied by a warm module cache; two kernel-cell walks over the MLX members no discrete home stamps; the tier contract files red in any batch whose mate arms the tier at init; the census roster naming the Bergamot stamps without their_spvkeys;test_gpu_homeon a build without the spelled home.--changed --base 1f71df6116, 230 files in 3 shards: 223 green in their shards; the 7 reds attributed: gptoss and the MoE file hit the census bug fixed mid-run (green alone after: 11/0, 17/0);test_model_imagefails its saves because the models volume is over quota (its cells are green on zen2 on both homes and on the M1);test_served_benchis a timing flake andtest_vadsaw oneERROR_INCOMPATIBLE_DRIVERin one process (both green alone and in a stocked batch on master and on the branch); the two below are master's.--only cuda,segments,regions,resident_moe,embeddings --exclude ctx128k,moe_host_pinned(the two excluded files cross the pod's 62 GB container cap; master dies the same): 44 files; the five reds are batch-state effects under the ask (gptoss 8/8, qwen2moe 4/4, segments 6/6 alone), the gemma-4-12B control (below) and llama-3b 32k under the ask (one slab pair bound on a 32 GB card, a one-off control one bar short: master fails it the same way, same numbers).--area vision --only gemma4uv,gemma4v,gemma3v,qwen3v,qwen25v: 5 files, 0 failed. CUDA audio on the pod,--area audio --only audio,whisper,gigaam,rnnt,vad,cuda_tower_g4a_chain: 8 files, 0 failed. Coverage census--suite coverage --arm coverage-vkon the pod: 4 passed, 2 skipped, 0 failed,f16_gemm_cm2_s_cls_spvnever dispatched by the zoo as expected; the roster fix (the Bergamot keys) was red first there.test_long_streams_gemma3_4b(128k, two streams) maxdiff 0.9907 against 0.896, identical on master on the same pod;test_gpu_resident_cuda_gemma4_12b's 300-token control lands inside its bar on this card (gap 1.46, master the same gap);test_gpu_resident_cuda_llama_1b_nv440-token window 1.182 against 1.099 (master's since the nv4 commits);test_nmt_loader_refusals(Windows fmap handle, master's); the 30B host-pinned MoE cells die at the fleet's 62 GB cap on both trees.cud_router_enc_cols; the fallbacks table read empty on the M1 and in a two-compile reproduction here; the tier contract files red in master's own ten-file batch; negative controls on the five CUDArsqrtcells, the lane-seat cell and the cache cell.watchdog.exe(das2rst by hand exits 0), tests-interp and tests-jit red only on the file time budgets (test_watchdog75-81 s,jit_lib84 s against 60 s; 15567/15580 and 15469/15483 passed, no assertion failed), review-md red only onREVIEW.das site's 160 findings over gitignored generated files.-no-module-cache, the Metal text diff against master, themad/ fma and tanh-GELU changes on Metal.Claims - stated, not tested
PERF_LEDGER.md, the STATED entry), not measured.externaland carry no recipe past the entry's own text.crd_router_rows's fold is covered by the MoE carriers on the pod only through the batch runs above; its lone cells pass.Not done
simdgrouphalves for a Mac whose toolchain carries no tensor ops (followup_metal.md71); the audio towers' q8 lane on Metal untimed against their f16 lane but GigaAM's (72).modules/ARCHITECTURE.md(ruled no for this change); the held board rows.