vh
2656196f47
Run-01 completed in 7:21:52 (47% faster than the 13.85h round-1 projection),
lora_B gate 205/205 non-zero at median norm 1.708, and the acceptance gate says
it did the thing it was built for: diversity +0.178 against a 0.008 floor (22x),
attractor hit rate -11.3pt against a 2.0pt floor, memorisation 0.0000 on both
arms — which closes the R20 licensed-prose exposure on measurement rather than
argument.
Five new detail files carry the substance:
erp-tune-run2-complete the run, the gate, the noise-floor near-miss
(brokkr was one step from reporting a 13-point
T6 regression sitting inside twice his
instrument's own variance)
mfu-root-caused-attention 8.6% MFU was an accounting artifact; real
utilisation 17-20%, cost was attention on
AMPERE kernels. Two independent methods agreed
to 2.6 points.
nvfp4-serving-pipeline merged weights are MANDATORY — vLLM cannot
serve a LoRA on ANY Gemma-4 — plus the recipe
that silently misses all 11,520 expert tensors
refusal-retention-probe measured base 0/100 -> tuned 29/100, then had
to accept it was the wrong axis
worldtree-b188-b189-and-selene three arcs closed, and a #411 diagnosis I got
wrong twice before a directory probe settled it
Current state rewritten end to end — the previous snapshot had the run in
flight at ~17h with MFU unexplained. Both are now closed.
The open operator decision is run 2's base, deliberately unstaged and flagged
against being filed as a config knob: it is a reversal of the trainee-selection
decision, and the pretrained-base option removes the last non-lexical floor on
the CSAM axis given stage-2-detector-inert and contamination-scan-absent are
both already overridden.
Tried-and-abandoned gains four measured-dead throughput levers, the packing
correction (bucketing wins under sdpa and the conclusion flips under flex — do
not carry it past the backend decision), and the merge-back-undoes-abliteration
trap brokkr caught in his own advice.
Index stays at 291 lines, under the soft cap. No archival this run.
87 lines
4.1 KiB
Markdown
87 lines
4.1 KiB
Markdown
# The 8.6% MFU was an accounting artifact — attention on Ampere kernels
|
|
|
|
`[2026-08-25]`
|
|
|
|
## The answer
|
|
|
|
**Real utilisation was 17-20%, inside the honest stock band.** The 8.6% divided
|
|
the *intended* (windowed) FLOPs by the wall time the *dense* reality took.
|
|
|
|
nominal billed 27.1 TFLOPS x 34.85 s = 9.4e14 FLOP
|
|
dense-sliding extra 25 layers, 2 seqs, 4 passes = +8.2e14
|
|
padded full layers lose the causal skip = +3.5e14
|
|
work performed ~ 1.8e15 = 51-61 TFLOPS
|
|
|
|
The card was doing ~2x the arithmetic the architecture specifies, and the excess
|
|
was the sliding window being computed and thrown away.
|
|
|
|
## Two independent methods agreed
|
|
|
|
scaling fit (3 points, 2 params, residuals <3ms over 8x range)
|
|
A = 6.87e-4 s/token B = 8.85e-8 s/token^2
|
|
quadratic share: 20.9% @ w=2048 -> 67.8% @ w=16384
|
|
|
|
kernel table (device rows only)
|
|
attention 22,835.8 ms 65.2% fmha_cutlass*_sm80
|
|
dense GEMM 2,774.0 ms 7.9%
|
|
other 5,739.0 ms 16.4%
|
|
|
|
**67.8% vs 65.2% — 2.6 points apart, no shared assumptions.** The two-term fit
|
|
needed no constant term, which refutes launch-bound outright (3,840 expert-GEMM
|
|
launches per forward are not the cost).
|
|
|
|
## The mechanism, source-verified by brokkr's panel (arm: Bil)
|
|
|
|
masking_utils.py:292-301 _ignore_causal_mask_sdpa requires
|
|
kv_length < local_attention_size. 16384 >= 1024,
|
|
so THE SLIDING MASK ALWAYS MATERIALISES.
|
|
sdp_utils_cpp.h:259-267 flash rejects ANY explicit mask
|
|
sdp_utils.cpp:647 cuDNN head_dim capped at 128 -> unreachable
|
|
Context.h:480-485 prefer-cuDNN needs major 9 or 10; sm_120 is 12
|
|
|
|
⚠ **The kernels are `sm80` — Ampere-generation CUTLASS on a Blackwell card**,
|
|
with the forward on `gmem`, the memory-efficient backend's slowest fallback tier.
|
|
|
|
## What actually fixed it
|
|
|
|
**Bucketing (bucket-to-pair, shuffle-to-mix)** — 29.9% padding -> 0.0%, and
|
|
78.3% of micro-batches become exactly zero-pad, which puts the 5 global layers
|
|
back on `is_causal`. Measured: padding costs **9.4% MORE time for 24% LESS work**
|
|
at fixed width, because an explicit mask knocks those layers off the fast path.
|
|
|
|
⚠ **Bucket size is NOT a diversity knob.** Swept across a 256x range, roots per
|
|
accumulation window stayed flat at 3.54-3.61. The global micro-batch shuffle does
|
|
all the mixing; the bucket only costs padding. Use the tightest bucket.
|
|
|
|
**flex_attention** — Triton-generated so it compiles for sm_120 instead of
|
|
shipping sm_80 binaries. 21.7x on sliding layers, 2.1x on global. Needs mandatory
|
|
`kernel_options` at 32x32 blocks: 64x32 needs 102,400 bytes against a
|
|
**101,376-byte hardware ceiling** — misses by 1 KB, and Triton is already opting
|
|
into the full 99 KB, so it is the card, not a default.
|
|
|
|
## ⚠⚠ The trap that produced TWO wrong published conclusions
|
|
|
|
`torch._dynamo` defaults to a recompile ceiling of **8**. Every distinct sequence
|
|
width is a new shape. On hitting the ceiling dynamo does not error — it silently
|
|
falls back to UNCOMPILED flex, which is ~20x slower AND documented to *"not work
|
|
with the backwards pass and may produce incorrect results."*
|
|
|
|
That artifact produced a bogus **0.76x slowdown** and a bogus **2.9% loss
|
|
divergence**, and I believed and reported both. Raising the limit to 256 flipped
|
|
the speed result to 1.41x.
|
|
|
|
The loss divergence turned out to be real but benign — adjudicated against fp32
|
|
MATH ground truth, both backends sit ~2e-3 from truth with flex fractionally
|
|
CLOSER at every width. **Do not re-open it by comparing the two backends to each
|
|
other; that cannot answer it. Compare to fp32.**
|
|
|
|
## Process lesson
|
|
|
|
brokkr's panel produced **four self-retractions in ninety minutes**. Every
|
|
retraction was a derivation; every survivor was a measurement. And the whole
|
|
head_dim-512 SDP problem was **already documented in zerofata's published Axolotl
|
|
config since April** — the right first stop for "why is this architecture slow"
|
|
is practitioner configs for that exact base, before any panel.
|
|
|
|
Playbook: `docs/pfi/training-throughput-playbook.md`, commit `7b5fd91`.
|