# The 8.6% MFU was an accounting artifact — attention on Ampere kernels

`[2026-08-25]`

## The answer

**Real utilisation was 17-20%, inside the honest stock band.** The 8.6% divided
the *intended* (windowed) FLOPs by the wall time the *dense* reality took.

    nominal billed      27.1 TFLOPS x 34.85 s        =  9.4e14 FLOP
    dense-sliding extra 25 layers, 2 seqs, 4 passes  = +8.2e14
    padded full layers  lose the causal skip         = +3.5e14
    work performed                                   ~  1.8e15  = 51-61 TFLOPS

The card was doing ~2x the arithmetic the architecture specifies, and the excess
was the sliding window being computed and thrown away.

## Two independent methods agreed

    scaling fit (3 points, 2 params, residuals <3ms over 8x range)
        A = 6.87e-4 s/token   B = 8.85e-8 s/token^2
        quadratic share: 20.9% @ w=2048 -> 67.8% @ w=16384

    kernel table (device rows only)
        attention   22,835.8 ms   65.2%   fmha_cutlass*_sm80
        dense GEMM   2,774.0 ms    7.9%
        other        5,739.0 ms   16.4%

**67.8% vs 65.2% — 2.6 points apart, no shared assumptions.** The two-term fit
needed no constant term, which refutes launch-bound outright (3,840 expert-GEMM
launches per forward are not the cost).

## The mechanism, source-verified by brokkr's panel (arm: Bil)

    masking_utils.py:292-301  _ignore_causal_mask_sdpa requires
                              kv_length < local_attention_size. 16384 >= 1024,
                              so THE SLIDING MASK ALWAYS MATERIALISES.
    sdp_utils_cpp.h:259-267   flash rejects ANY explicit mask
    sdp_utils.cpp:647         cuDNN head_dim capped at 128 -> unreachable
    Context.h:480-485         prefer-cuDNN needs major 9 or 10; sm_120 is 12

⚠ **The kernels are `sm80` — Ampere-generation CUTLASS on a Blackwell card**,
with the forward on `gmem`, the memory-efficient backend's slowest fallback tier.

## What actually fixed it

**Bucketing (bucket-to-pair, shuffle-to-mix)** — 29.9% padding -> 0.0%, and
78.3% of micro-batches become exactly zero-pad, which puts the 5 global layers
back on `is_causal`. Measured: padding costs **9.4% MORE time for 24% LESS work**
at fixed width, because an explicit mask knocks those layers off the fast path.

⚠ **Bucket size is NOT a diversity knob.** Swept across a 256x range, roots per
accumulation window stayed flat at 3.54-3.61. The global micro-batch shuffle does
all the mixing; the bucket only costs padding. Use the tightest bucket.

**flex_attention** — Triton-generated so it compiles for sm_120 instead of
shipping sm_80 binaries. 21.7x on sliding layers, 2.1x on global. Needs mandatory
`kernel_options` at 32x32 blocks: 64x32 needs 102,400 bytes against a
**101,376-byte hardware ceiling** — misses by 1 KB, and Triton is already opting
into the full 99 KB, so it is the card, not a default.

## ⚠⚠ The trap that produced TWO wrong published conclusions

`torch._dynamo` defaults to a recompile ceiling of **8**. Every distinct sequence
width is a new shape. On hitting the ceiling dynamo does not error — it silently
falls back to UNCOMPILED flex, which is ~20x slower AND documented to *"not work
with the backwards pass and may produce incorrect results."*

That artifact produced a bogus **0.76x slowdown** and a bogus **2.9% loss
divergence**, and I believed and reported both. Raising the limit to 256 flipped
the speed result to 1.41x.

The loss divergence turned out to be real but benign — adjudicated against fp32
MATH ground truth, both backends sit ~2e-3 from truth with flex fractionally
CLOSER at every width. **Do not re-open it by comparing the two backends to each
other; that cannot answer it. Compare to fp32.**

## Process lesson

brokkr's panel produced **four self-retractions in ninety minutes**. Every
retraction was a derivation; every survivor was a measurement. And the whole
head_dim-512 SDP problem was **already documented in zerofata's published Axolotl
config since April** — the right first stop for "why is this architecture slow"
is practitioner configs for that exact base, before any panel.

Playbook: `docs/pfi/training-throughput-playbook.md`, commit `7b5fd91`.
