[WIP] Alternative GEMM Design - #195
Draft
hunhoffe wants to merge 26 commits into
Draft
Conversation
Adds a second GEMM design, distinct from the existing gemm operator rather than a retuning of it: A is broadcast along each compute row from four shim columns, C is joined at the memtile, the mmul keeps A in a single ObjectFifo object, and an activation + clamp are fused into the C drain. Ported from an inert-overlay design whose host wrote shapes into an RTP buffer per dispatch. With M/K/N static per IRON operator that machinery is gone: the trip counts are compile-time constants, so the counts kernel, the RTP buffer and its sync lock, and the N-remainder drain path all disappear. The ObjectFifo dims are carried over verbatim and verified to match the original's MLIR exactly. Three things had to change to make it correct here: - B's fill descriptor reorders on the fly. The memtile expects B in t-block-major (n//T, k%S, k//S, n%T) order; the original relied on its host pre-packing the weights that way. Doing it in the descriptor keeps B an ordinary dense (K, N) tensor. - The kernel now sets rounding to conv_even. The core powers up in floor, and truncation biases every conversion the same direction, so the error accumulates coherently over K instead of cancelling: 1% of accumulated mass versus 0.042%, a 24x difference. The original never set it. - A is filled one k-block at a time and the fills are retired in batches. A single BD spanning every k-block stalls mid-transfer once k_iters exceeds the fifo depth while still holding its shim channel, and a shim tile allows only 16 active BDs -- which is why this only showed up at larger K. Accuracy now matches the existing GEMM run in the same bfp16-emulated mode (mean error 0.00042 of accumulated mass vs 0.00044). The r=8 mmul shape only exists on that emulated path, so the test bounds error against the accumulated mass rather than elementwise-relatively, which a K-term signed sum cancelling ~sqrt(K) makes meaningless. Co-Authored-By: Claude <noreply@anthropic.com>
The epilogue is short and runs once per C object, so leaving it as a call costs real time against the per-object overhead the C ObjectFifo introduces; the design this was ported from merges it into the core for exactly that reason. Inlining the much larger mmul measures worse there, so only the epilogue is merged. aie.iron's ExternalFunction(inline=True) emits the right declaration (link_with_mode = "merge") but its own compilation is driven by CompilableDesign, which IRON's flow does not use -- so the .ll was never built and aiecc failed to find it. Make .ll/.bc kernel artifacts first-class instead: KernelCompiler infers the inline build from the suffix and passes the symbol to mark alwaysinline, which KernelObjectArtifact now carries. Both are rejected with a clear message when misused (chess has no equivalent path, and an inline build with no symbol is a silent no-op otherwise). Verified inlined rather than silently falling back: the .ll carries alwaysinline, the MLIR declares merge mode, and the core ELF exports flm_gemm_acc_init and flm_gemm_k_step but no epilogue symbol. inline_epilogue is a field so the two can be A/B'd, and it participates in the operator name -- otherwise a cached xclbin from one satisfies the other and the comparison measures a binary against itself. Co-Authored-By: Claude <noreply@anthropic.com>
N previously had to be a multiple of N_TILE*COLS=1024, which excluded the shapes this design exists to serve: a transformer's o and down projections have N = model dim, so 8 of Gemma4's 14 projections could not run at all. N now only has to tile to N_TILE. The trailing group of fewer than COLS column-blocks is handled by giving each column its own trip counts rather than a runtime branch: columns below the remainder width compute one block more, and the rest run an A-only drain loop for it. That drain is not optional -- A is broadcast along the whole compute row, so a column sitting the block out must still consume its share or the columns that do have work stall behind it. The runtime sequence matches, issuing A for every row but B and C only for the active columns. Also enforces a K ceiling instead of silently mis-lowering. A sweep's fills have to go in ONE task group; splitting them across groups to fit more k-iterations into a shim tile's 16 buffer descriptors returns wrong data, reproducible at M=1024 K=2560 N=2560 where one group passes and two fail. Undiagnosed, so K > 3584 now raises rather than producing bad results. Tests cover N=1536 (4 of 8 columns active), N=128 (1 column, no full sweep), and the real E4B o-projection and E2B down-projection shapes. 13/13 flm_gemm and 24/24 gemm pass. Co-Authored-By: Claude <noreply@anthropic.com>
When N is smaller than the grid's COLS*N_TILE stride there is no full sweep, so only the first rem_blocks columns participate. The remaining columns were still getting B and C objectfifos, plus a consumer on the A broadcast, none of which anything ever drains. The pinned mlir-aie accepts that silently; a newer one rejects it outright with "objectfifo.pool op segment 0 has no drainer", which is how it surfaced -- N=128 failed to compile there while passing on the pin. It was latent dead dataflow either way. Those columns are now not built at all. Dropping them from the A broadcast also removes their A-drain obligation, so the drain loop is needed only for a column that exists and sits out the trailing block, which requires at least one full sweep. 13/13 on both the pinned wheel and mlir_aie 1.4.3.dev60 (verified with a clean build dir -- a shared one silently reuses the other toolchain's xclbins and the second run proves nothing). Co-Authored-By: Claude <noreply@anthropic.com>
B was kept as a plain row-major (K, N) tensor and reordered into the memtile's expected layout by the fill descriptor. That is correct, and it was a deliberate choice to spare the caller a packing step, but its innermost run is T=8 bf16 = 16 bytes: every 128 KB B transfer became 8192 scattered bursts. B is ~70% of the bytes a dispatch moves, so the whole operator ran at ~10 GB/s against the original overlay's ~47. B is now consumed pre-packed via FLMGEMM.pack_B and each fill is one contiguous read, which is exactly why the design this came from packs its weights on the host. Weights are packed once and reused across dispatches, so the cost belongs on the caller. M=1024 K=1536 N=6144, same session: FLM peano mm.xclbin 2270 us shipped v1.0.4 2335 us flm_gemm before 11509 us -> after 3446 us IRON GEMM (r8/f32) 3465 us So 5.2x off the original becomes 1.4x, and level with the existing GEMM operator while keeping 41x better accuracy. Found by nulling the compute out: with no arithmetic at all the operator still took 11 ms, which exonerated the kernel, the mmul geometry, the epilogue and the L1 budget in one measurement, and pointed at the transfer descriptors. An earlier test of this same hypothesis had reported only 5% because it reused a cached xclbin from a build directory it had not cleared. The residual 1.4x is the missing depth-2 overlap: data movement alone is 2.07 ms and compute adds ~1.37 ms, so the two are running almost entirely serialised. Co-Authored-By: Claude <noreply@anthropic.com>
A single fill or drain may span many fifo objects -- the descriptor walks them in the order the cores consume. The sequence was issuing one task per object instead, so every row-block ended in a host-side await on its C drain, and the next row-block's fills could not start until that C had come all the way back from DDR. Those awaits were the serialisation. Each of A, B and C now goes out as one task per column-block, with the dimension order matching the core loop nest (mega_row, then k). For M=1024 K=1536 N=6144 that is 120 tasks and 48 awaits where it was 1056 and 192. before 3446 us after 2534 us FLM mm.xclbin 2241 us, shipped 2212 us IRON GEMM (r8/f32) 3331 us So 1.13x off the original, and 1.3x faster than the existing GEMM operator at this shape, still with ~40x better accuracy. This also retires the K ceiling. The previous per-k-block tasks needed 2*k_iters + 1 buffer descriptors on a shim tile, which capped K at 3584 and forced a batching path that silently corrupted results when it split a sweep's fills across task groups. One task per leg needs three, so that whole mechanism is gone: K=4096 now builds and runs (err/mass 1.5e-4). Co-Authored-By: Claude <noreply@anthropic.com>
Issue column-block i+1's transfers before retiring i's, so they are already moving while i computes. Retiring each block first serialises the whole pipeline on its C await, and a C await waits for the cores. This is the same depth-2 model the design was ported from, and it failed twice before for a reason that had nothing to do with the idiom: when each leg was issued per object a block cost 1 + 2*k_iters buffer descriptors on a shim column, so two in flight blew the 16-BD limit and deadlocked. Now that a leg is a single task a block costs three, and two in flight is six. Collapsing the per-object tasks was the precondition, not an alternative. M=1024 K=1536 N=6144, same session: flm_gemm 2248 us 8834 GFLOP/s FLM peano mm.xclbin 2173 us shipped v1.0.4 2314 us IRON GEMM (r8/f32) 3384 us So the operator is now level with the original overlay -- between its two builds, 1.5x faster than the existing GEMM operator -- while keeping ~40x better accuracy (err/mass 2.4e-4 vs the original's 9.9e-3, which runs in the core's default floor rounding). Two collapses that do NOT work, for the record: B cannot fold the row-block in because it does not vary with it and a 0-stride wrap of size > 1 is rejected by the BD lowering; and draining C once for the whole dispatch hangs. 37/37 pass from a clean build directory. Co-Authored-By: Claude <noreply@anthropic.com>
The epilogue was compiled to alwaysinline LLVM IR and llvm-linked into the core, on the strength of a comment in the design this was ported from saying that is what pays back the per-object call overhead the C ObjectFifo introduces. Measured here it buys nothing: 7 runs x 100 iterations, 2189 us inlined vs 2182 us as a plain call, against a ~5% run-to-run spread. Indistinguishable. That was the only thing requiring .ll kernel artifacts, so iron/common/compilation/base.py goes back to upstream. Making .ll a first-class artifact type is a reasonable capability -- aie.iron's ExternalFunction(inline=True) emits the merge-mode declaration but does not build anything under IRON's compilation flow, so nothing produced the file -- but it should land on its own merits with a case that measurably needs it, not as shared-infrastructure drift inside an operator change. The epilogue is now an ordinary Kernel call, which also removes the inline_epilogue field, its artifact-naming special case, and the ExternalFunction branch in the design. Diff is now confined to the operator and its kernels. 37/37 pass, and performance is unchanged: min 2183 us vs the original overlay's 2175 us. Co-Authored-By: Claude <noreply@anthropic.com>
…exactly The kernels already had the #ifdef; nothing could reach it. rounding= "floor" now selects the core's power-up mode, which is what the design this was ported from runs in -- it never calls set_rounding. With it, the operator is BIT-IDENTICAL to the shipped FastFlowLM v1.0.4 mm.xclbin: all 6291456 elements match on M=1024 K=1536 N=6144, maxdiff 0. That pins the port as arithmetically faithful and isolates rounding as the only numerical difference between the two. conv_even (default) err/mass 0.000241 floor err/mass 0.009867 shipped mm.xclbin err/mass 0.009867 So the 41x accuracy gain really is the rounding mode alone, and users who need to match the shipped overlay can ask for it. The mode is part of the operator name and of both kernel object names: it changes the emitted code, so a cached build of one mode must not satisfy the other. Co-Authored-By: Claude <noreply@anthropic.com>
…g in tests Adds a README covering the parts a caller cannot guess: that B must be pre-packed and why, the shape constraints, the accuracy budget of the bfp16-emulated path (and why an elementwise relative tolerance is the wrong instrument for it), and how to reproduce the shipped FastFlowLM overlay. Verified against FastFlowLM v1.0.4's Gemma4-E2B mm.xclbin, driven directly with the instruction stream from that project's own TXN generator, on identical inputs: rounding="floor" bit-identical, 6291456/6291456 elements, maxdiff 0 rounding="conv_even" differs everywhere (41x more accurate) The activation path matches too. The shipped kernel selects its activation from RTP word 4; this operator bakes it in at compile time with the same 0/1/2/3 mapping, and all four are bit-identical to the shipped kernel at output_mode 0/1/2/3 -- 1048576/1048576 elements each. Inputs were scaled down for that check so the activations sit where the curve is not flat, and each mode's output was confirmed to differ from mode 0, so a silently ignored mode could not pass. clamp has no shipped counterpart: their generate_seq never writes the clamp RTP words, so clamping is always off there. Noted in the README rather than left implicit. Tests now carry rounding as a parameter, with floor given its own error bound -- holding truncation to the conv_even budget would simply fail. Co-Authored-By: Claude <noreply@anthropic.com>
Nulling the mmul out showed this operator is COMPUTE bound -- it drops from 2178 us to 1481 us with no arithmetic -- while the GEMM operator does not move at all under the same treatment (3374 vs 3353 us) and is therefore entirely data-movement bound. The two want opposite fixes, and this one wants a cheaper inner loop. n=64 gives the mmul colA=8 instead of 4, halving accumulator traffic per mac, at the cost of doubling A fetches. With compute on the critical path that trades well: M/K/N k_iters tile_n=64 tile_n=128 1024/512/4096 1 642 us 589 us 1024/1024/4096 2 850 us 1034 us 1024/1536/6144 3 1741 us 2178 us 1024/2560/4096 5 1891 us 2371 us 2048/2048/2048 4 1535 us 1924 us 256/4096/1024 8 254 us 316 us Only the single-k-iteration shape prefers 128: there is too little compute there to hide the extra A traffic. tile_n now defaults to None and picks 128 when K == 512 and 64 otherwise, which is correct on every shape measured including the K=1024 boundary. It stays overridable. At 1741 us the operator is now 1.25x faster than the shipped overlay (2175 us) and 1.9x faster than GEMM (3353 us) on the reference shape. pack_B becomes an instance method: the packing layout depends on tile_n, so a static one silently mismatches the operator it feeds. tile_n also joins the operator and kernel-object names. This vindicates the CT_MAX_K hypothesis from early on, which had been dismissed after testing it while the operator was still DMA bound at 11 ms -- where compute could not matter. Right idea, wrong regime. 39/39 pass. Co-Authored-By: Claude <noreply@anthropic.com>
The table claimed GEMM in the same emulated mode had 0.00044 mean error against this operator's 0.00024. That was wrong: 0.00044 came from a different configuration (prio_accuracy=False, a bf16 accumulator), quoted from a separate experiment and mislabelled as the f32 one. Measured on identical data, this operator and GEMM with emulate_bf16_mmul_with_bfp16=True and prio_accuracy=True are numerically indistinguishable -- err/mass 0.000241, signed bias +0.0871, max 15.44 for both, and the same at either tile_n. Same mmul shape, same emulation, same f32 accumulation, same rounding; there was never a reason for them to differ. The only real accuracy difference remains conv_even versus the shipped overlay's floor, which stands at 41x and is separately verified bit-exact. Co-Authored-By: Claude <noreply@anthropic.com>
Takes the Peano pin from mlir-aie's own utils/peano-requirements.txt at c80b88c, so the two stay in step. 1.4.3 adds ObjectFifo DMA channel pinning (prod_dma_channel / cons_dma_channels, exposed as prod(channel=)/cons(channel=)), which is absent from 1.4.2.dev16. 39/39 of the flm_gemm and gemm iter0 tests pass on the new toolchain, and flm_gemm's latency is unchanged within noise at M=1024 K=1536 N=6144 (1743 us, against 1741 before the bump). Co-Authored-By: Claude <noreply@anthropic.com>
Hold a whole column-block's B in the memtile as one object and replay it per row-block, so DDR reads it once rather than m_row_blocks times. B is the dominant DDR leg -- 75 MB of the 126 MB moved at M=1024 K=1536 N=6144 -- so this is ~43% less traffic overall. This operator is DDR-bandwidth bound, so that is a latency win as well as a power one, and it grows with the height of the problem because B's re-reads scale with m_row_blocks. At K=1024 N=4096, full builds, min of interleaved rounds: M=512 (2 row-blocks) 470.8 -> 468.5 us 0.5% M=1024 (4 row-blocks) 860.4 -> 846.5 us 1.6% M=2048 (8 row-blocks) 1760.1 -> 1622.8 us 7.8% With the mmul nulled out to isolate data movement, the non-resident floor at M=2048 is 1739 us to move 118 MB -- 68 GB/s, against a memcpy-measured 63-70 GB/s roof -- and residency drops it to 1135 us. Only 137 us of that 604 us is captured, because repeat_count restarts the memtile BD chain at every replay boundary. Closing that gap is the largest known remaining lever here; it is left for follow-up. The replay is repeat_count on the forward(). Earlier attempts used iter_count and hung on every shape with more than one column-block: iter_count does not replay anything, it only bounds how many times an end cycles through all of its buffers (objects = iter_count * elemNumber * repeat_count), so it counts depth-cycles, not objects, and a wrong value runs the BD chain out from under the cores. Correct k ordering needs the memtile to hold ONE object spanning every k-block; replaying a pool of k_iters smaller objects emits k0,k0,k1,k1, ... instead of the k0..kn sequence the cores accumulate in. Residency is gated on the buffer fitting double-buffered, which admits only k_iters <= 2 (K <= 1024 at tile_n=64); larger K falls back to the previous behaviour unchanged. Co-Authored-By: Claude <noreply@anthropic.com>
Three gaps in the README, all found by re-deriving its numbers. Resident B was undocumented. It is gated on a whole column-block's B fitting the memtile double-buffered (K <= 1024 at tile_n=64), so it is inactive on the shape the performance table reports and easy to miss entirely. Record what it does, where it applies, and that its benefit grows with M -- 0.5% / 1.6% / 7.8% at M=512/1024/2048 -- along with the 604 us of DMA floor it exposes against the 137 us that currently reaches the full build. The measurement recipe was unsafe. "min of per-run medians" is only sound if the runs are not all in the same place: dispatch latency here is bimodal with modes ~6% apart, so a batch landing wholly in one mode makes that statistic a mode selector. It reported a convincing 5% win for a change subsequently shown to do nothing. Say to interleave the configurations and to require the min and the median to agree. The tile_n=64 DMA-only cell was blank. Filling it in (1692 of 1741 us) shows the default configuration is itself data-movement bound, which the surrounding prose implied only of tile_n=128 -- and which is what makes "move fewer bytes" the right next lever rather than "write a faster kernel". Co-Authored-By: Claude <noreply@anthropic.com>
Contributor
CI Test Resultsf8890d7 (2026_09_10_17_00_31) IRON - CI SummaryExamplesiron/applications/llama_3.2_1b
Smalliron/operators/axpy
iron/operators/dequant
iron/operators/elementwise_add
iron/operators/elementwise_mul
iron/operators/flm_gemm
iron/operators/gelu
iron/operators/gemm
iron/operators/gemv
iron/operators/layer_norm
iron/operators/leaky_relu
iron/operators/mem_copy
iron/operators/mha
iron/operators/relu
iron/operators/repeat
iron/operators/rms_norm
iron/operators/rope
iron/operators/sigmoid
iron/operators/silu
iron/operators/softmax
iron/operators/strided_copy
iron/operators/swiglu_decode
iron/operators/swiglu_prefill
iron/operators/tanh
iron/operators/transpose
Krackan - SmallIRONTested on iron/operators/axpy
iron/operators/dequant
iron/operators/elementwise_add
iron/operators/elementwise_mul
iron/operators/flm_gemm
iron/operators/gelu
iron/operators/gemm
iron/operators/gemv
iron/operators/layer_norm
iron/operators/leaky_relu
iron/operators/mem_copy
iron/operators/mha
iron/operators/relu
iron/operators/repeat
iron/operators/rms_norm
iron/operators/rope
iron/operators/sigmoid
iron/operators/silu
iron/operators/softmax
iron/operators/strided_copy
iron/operators/swiglu_decode
iron/operators/swiglu_prefill
iron/operators/tanh
iron/operators/transpose
Krackan - ExamplesIRONTested on iron/applications/llama_3.2_1b
Phoenix - SmallIRONTested on iron/operators/axpy
iron/operators/dequant
iron/operators/elementwise_add
iron/operators/elementwise_mul
iron/operators/flm_gemm
iron/operators/gelu
iron/operators/gemm
iron/operators/gemv
iron/operators/layer_norm
iron/operators/leaky_relu
iron/operators/mem_copy
iron/operators/relu
iron/operators/repeat
iron/operators/rms_norm
iron/operators/rope
iron/operators/sigmoid
iron/operators/silu
iron/operators/softmax
iron/operators/strided_copy
iron/operators/swiglu_decode
iron/operators/swiglu_prefill
iron/operators/tanh
iron/operators/transpose
Phoenix - ExamplesIRONTested on Trend tables omitted, the comment hit GitHub's size limit. Full report in the workflow run. |
Two changes that only pay together, taking M=1024 K=1536 N=6144 from 1741 us to 1434 us against the shipped overlay's 2175 us -- 1.25x to 1.52x -- at bit-identical arithmetic (err/mass 2.41e-04 either way). The mmul's colA loop was hand-unrolled 2x for the Peano path, on the premise that "Peano schedules the 2x-unrolled body better than it schedules the rolled one". That is no longer true, and the unroll had become a 23% pessimization: it halves the inner trip count to 4, too few to amortize the software pipeline's fill and drain. Measured cycles per mmul call, by hardware trace of the event0/event1 pair the kernel already emits: rolled 1875, hand-unrolled 2x 2446, compiler unroll 4 and 8 3291 and 3221. Chess always took the rolled path, so the branch is simply deleted rather than re-gated. The rolled loop is a sharp optimum -- anything adding live state across it loses more than it gains, which is the signature of llvm-aie#1066, where Peano's pipelining-unaware register allocator manufactures false loop-carried anti-dependencies. Loop hints, hoisting the bfp16 operand conversion, and prefetching the accumulator all measured neutral or worse. Widening the residency gate is what banks the win. It only counts C's 64 KB, so it admits k_iters <= 3 (K <= 1536) rather than 2, overcommitting the A-carrying memtiles by 64 KB and relying on aie-objectfifo-allocate to spill one buffer to the adjacent memtile. That packs all eight to exactly 512 KB, with zero slack -- see the comment in design.py before changing any buffer size. Neither change is worth much alone: residency was measured latency-neutral while the mmul was the critical path, and the re-rolled mmul gained little while the non-resident DMA floor was. Total latency is max(compute, DMA), so testing them separately scores both as zero. 39/39 green.
Takes M=1024 K=1536 N=6144 from 1435 us to 1252 us against the shipped overlay's 2175 us -- 1.52x to 1.74x -- at unchanged arithmetic (err/mass 2.41e-04). Three changes that only pay together. Asymmetric tile buffering decouples the A tile's height from the accumulator's. A is dead as soon as it is consumed while C must live across the whole K reduction, so sizing both to M_TILE pays the peak L1 cost twice. Giving A 16 rows against the accumulator's 64 (rho = 4) frees enough L1 for a 128-deep k slice, which halves the accumulator bytes per mac and doubles the inner loop's trip count: 3.67 -> 2.64 cycles per 8x8x8 mac by hardware trace. Idea from "Can Asymmetric Tile Buffering Be Beneficial?" (arXiv:2511.16041) and mlir-aie's gemm_asymmetric_tile_buffering examples, though those spend the freed L1 on a larger C tile, which an f32 accumulator cannot afford. That k slice was not reachable before. At CT_MAX_K=128 B's innermost contiguous run is 1024, one over the buffer descriptor's 10-bit size field, so it needs a second dimension to encode -- and residency already spends one walking k. Five dimensions against the hardware's four. pack_B now emits B in the order the cores consume it rather than an intermediate blocked order, so both B hops are linear and neither spends a dimension. On its own that is worth nothing; it is what makes the rest fit. tile_ma is chosen by fitting the L1 working set rather than fixed, so tile_n=128 (whose accumulator is already 32 KB) stays symmetric. Note the resolved tile_ma is in the operator name as well as the kernel object name. It changes both the emitted MLIR and the kernel, and the build cache is keyed on filename -- a dir holding another value's artifacts silently produced NaNs before the name encoded it. 39/39 green.
…sion
Takes M=1024 K=1536 N=6144 from 1436 us to 1143 us against the shipped
overlay's 2175 us -- 1.74x to 1.90x. DDR drops 69 -> 47 MB because
bfp16ebs8 packs 8 values in 9 bytes where bf16 needs 16, and B is the
leg that residency keeps but cannot shrink.
DO NOT SHIP AS IS. Accuracy moves from 2.4133e-04 to 2.6923e-04, about
12% worse, and 81% of output elements change. That contradicts the
premise this was built on -- that quantizing B on the host merely hoists
a rounding the mmul already performs on every mac call, and so is free.
It is not free, and the residual is not yet explained.
For scale: 2.69e-04 is still 37x better than the shipped overlay's
9.87e-03. But it breaks an exact tie -- flm_gemm at 2.4133e-04 matched
IRON GEMM's emulated mode to five digits, and GEMM's accurate mode
(emulate_bf16_mmul_with_bfp16=False) reaches 4.1321e-05, which this
moves further away from.
What has been ruled out as the cause, each by measurement:
* Block grouping. The shared exponent covers 8 consecutive k for one
n; grouping over n instead gives 1.95e-02.
* Rounding. Truncation is correct; round-to-nearest gives 6.18e-03.
* The A widening path. Using mul_elem_64 for one operand and direct
assignment for the other, as mm_bfp_mixed.cc does, is byte for byte
identical to using direct assignment for both.
* The packer itself. It is byte-identical to mlir-aie's reference
floatToBfp16, and single-value probes decode exactly on hardware
across magnitude, sign and shared-exponent crushing.
* Layout and accumulation. Single-element (k,n) probes map correctly,
and partial sums over k are exact to 512 terms.
Also fixes a bug found on the way in: the bfp16 mmul never advanced pC1
and pC2 across the j loop, so every j iteration accumulated into the
same C slot. Every single-element probe passed anyway because they all
used columns that j=0 owns -- the error only showed as a period-64
column pattern in full data.
39/39 green (the test budget is 4e-3, well above both figures).
Removes the 12% accuracy regression from the previous commit. B in bfp16 is now numerically free, as it was supposed to be: 2.3912e-04 against the bf16 build's 2.4133e-04, i.e. very slightly BETTER, and better on every shape measured (2.900 vs 2.93e-04, 2.068 vs 2.09e-04, 4.173 vs 4.20e-04, 1.846 vs 1.86e-04). 1.90x the shipped overlay, 41x more accurate than it. Two bugs, both in the host-side quantizer. The conversion obeys the core's ROUNDING MODE. mlir-aie's reference floatToBfp16 hardcodes truncation and comments that AIE2P always truncates; that is true only of the power-up floor mode. flm_gemm calls set_rounding(conv_even), so the kernel's own conversion rounds to nearest with ties to even. Reading one block back off the hardware a slot at a time shows it plainly: 14.9375 -> 15 while 106.5 -> 106 and 94.5 -> 94. Truncation cannot produce that. pack_B now follows self.rounding, so rounding="floor" still truncates. Rounding then overflows the mantissa. A block's largest magnitude sits at 127 before rounding and can carry to 128, which does not fit the signed 8-bit field and wrapped to -128. B is drawn from rand() in the tests, so values sit near their block maximum often, and the wrap alone cost 6.18e-03. Saturate instead. Found by probing a single 8-value block: A[0,ki]=1 selects slot ki so C[0,0] is exactly that slot's decoded value, which makes the kernel's conversion directly readable and comparable against the packer's. Worth keeping in mind -- the earlier single-value probes all passed because a lone nonzero owns its block's exponent and never exercises the shift. 39/39 green.
pack_B returns bfp16ebs8 -- 9 bytes per 8 values -- but the arg spec still described B as (K, N) bf16. #179 made buffer sizing follow spec.dtype (prod(shape) * dtype.itemsize in calculate_buffer_layout), so that declaration over-allocates this operator's LARGEST buffer by 1.78x: at K=1536 N=6144 it asks for 18.9 MB where pack_B produces 10.6 MB. Harmless before the merge because the spec's dtype was not consulted; worth fixing now rather than leaving a caller-visible size lie in place. 39/39 green.
All three are env-gated and inert by default; 15/15 green with nothing set. They exist because the operator is now DMA-rate bound rather than byte bound, and that is not visible from wall clock alone. FLM_NULL_MMUL=1 skip the multiply, keep every DMA and handshake FLM_C_LINEAR=1 drain C as one contiguous run per column-block FLM_OVERLAP=N column-blocks in flight (default 2, unchanged) What they establish, at M=1024 K=1536 N=6144: Compute is entirely hidden. Nulling the mmul does not make the operator faster (1125 us full against 1152 nulled), so the ~14% of compute still above its ResMII bound is worth nothing until data movement improves. The operator moves 60.9 MB in ~1140 us, i.e. 53 GB/s against a measured 63-70 GB/s roof -- 79%. The gap is not per-core ingress: the trace shows the cores' two input DMA ports only 45% and 25% busy, so they are starved rather than saturated. Nor is it pipelining: overlap depth 3 is worth under 1%, and 4 and 5 are worse as shim buffer descriptors run out. It is C's write pattern. C is drained as ROWS*M_TILE runs of N_TILE elements -- 128 bytes -- strided by N, and 128 B sits below the ~256 B cliff where DDR bursts fall apart. Draining the same bytes contiguously saves 114 us on the min and 86 us on the median over 6 interleaved rounds. That is ~8% of the whole operator, and recovering it would put it near 2.1x the shipped overlay. FLM_C_LINEAR writes C to the wrong place -- it is a measurement, not a fix. The fix needs 256 B runs with C still row-major, which means either a 128-wide n tile (but that caps the k slice at 64, making compute the wall again) or joining C across pairs of adjacent columns so one drain covers 128 columns.
The gate compared B's resident footprint against (512 - 64) KB, which ignores A's 128 KB entirely and hardcodes C's size. It happened to be conservative enough at tile_n=64, but at tile_n=128 -- where C doubles to 128 KB and resident B is 432 KB -- it admitted a configuration totalling 688 KB of a 512 KB memtile, which failed address assignment outright rather than falling back to non-resident. Found while measuring whether tile_n=128 could pay for itself now that ATB and bfp16 B changed the L1 and L2 budgets. It cannot: with the gate honest it correctly drops to non-resident, and B's DDR then quadruples to 42.5 MB for a total of 74 MB against tile_n=64's 60.9. Measured 2283 us against 1154. So tile_n=64 stays the default, and the sweep table's preference is confirmed for the current balance even though its absolute numbers are stale. 39/39 green.
_default_tile_ma picked the A-tile height against a fixed double-buffered B. That is one axis of a two-axis budget, and it cannot express the tradeoff that matters at wider n tiles: there, B's L1 object is large enough that a double-buffered pair does not fit, and giving up B's prefetch is what buys a deeper k slice -- colA 8 -> 16 is worth 3.67 -> 2.28 cycles per mac, far more than B's L1 prefetch. _default_l1 now returns both, deepest B first and then the tallest A that fits, so tile_n=64 is unchanged at (T_MA=32, B_DEPTH=2). Also lets the resident-B gate fall back to a single-buffered memtile B rather than giving up residency: at tile_n=128 B is 432 KB double-buffered and does not fit beside A and C, but 216 KB does, and there residency is worth much more than the prefetch because it is also what stops B's DDR traffic quadrupling. Neither changes the shipped configuration. 39/39 green.
…atter costs The sweep table had been marked stale since cf270ed -- it predated the rolled mmul, resident B, ATB and bfp16 B, which between them took 1024/1536/6144 from 1741 to 1141 us. Re-measured all six shapes against the current design, min of per-run medians over 6 rounds with the two tile_n builds interleaved round-robin (this box is bimodal ~6%, so running all of one and then all of the other measures drift rather than design). The default rule is unchanged in sign -- tile_n=128 still wins only at k_iters=1 -- but both edges moved: its margin there narrowed 8% -> 3%, and its penalty elsewhere grew from ~1.2-1.25x to 1.2-1.7x. Both follow from tile_n=128 giving up resident B, which costs more the more k there is. FLM_C_RUN2 is a new ablation next to FLM_C_LINEAR. C leaves each core as a 64-element (128 B) run, and the guess was that this sits below a ~256 B DDR write-efficiency cliff, making a column-pair join worth ~100 us. It is not: matched for byte count and footprint, halving the number of runs and doubling their length buys 14.9 us min / 13.0 us median, against 98.2 / 96.9 for removing the scatter entirely. So there is no cliff at 256 B -- the cost tracks the NUMBER of scattered runs and keeps paying well past it, and one doubling recovers only 15%. Both knobs write C to the wrong place; they are timing probes, not modes. That measurement is why the column-pair join is not being built: it would have taken 1148 -> ~1134 us (1.90x -> 1.92x vs the shipped overlay) in exchange for replacing the whole C leg with explicit Buffer/Lock/Flow/TileDma -- 8 core-side DMA programs plus 2 memtile ones -- because ObjectFifo.join() cannot place its sources interleaved and cannot reorder on the way out either. A hardware- verified reproducer for that limitation lives in /scratch/ehunhoff/objectfifo_repro. Co-Authored-By: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Describe the intent of your PR here.
Added
Changed
Removed
PR Merge Checklist
develcommit and pointing todevel.