vh
2656196f47
Run-01 completed in 7:21:52 (47% faster than the 13.85h round-1 projection),
lora_B gate 205/205 non-zero at median norm 1.708, and the acceptance gate says
it did the thing it was built for: diversity +0.178 against a 0.008 floor (22x),
attractor hit rate -11.3pt against a 2.0pt floor, memorisation 0.0000 on both
arms — which closes the R20 licensed-prose exposure on measurement rather than
argument.
Five new detail files carry the substance:
erp-tune-run2-complete the run, the gate, the noise-floor near-miss
(brokkr was one step from reporting a 13-point
T6 regression sitting inside twice his
instrument's own variance)
mfu-root-caused-attention 8.6% MFU was an accounting artifact; real
utilisation 17-20%, cost was attention on
AMPERE kernels. Two independent methods agreed
to 2.6 points.
nvfp4-serving-pipeline merged weights are MANDATORY — vLLM cannot
serve a LoRA on ANY Gemma-4 — plus the recipe
that silently misses all 11,520 expert tensors
refusal-retention-probe measured base 0/100 -> tuned 29/100, then had
to accept it was the wrong axis
worldtree-b188-b189-and-selene three arcs closed, and a #411 diagnosis I got
wrong twice before a directory probe settled it
Current state rewritten end to end — the previous snapshot had the run in
flight at ~17h with MFU unexplained. Both are now closed.
The open operator decision is run 2's base, deliberately unstaged and flagged
against being filed as a config knob: it is a reversal of the trainee-selection
decision, and the pretrained-base option removes the last non-lexical floor on
the CSAM axis given stage-2-detector-inert and contamination-scan-absent are
both already overridden.
Tried-and-abandoned gains four measured-dead throughput levers, the packing
correction (bucketing wins under sdpa and the conclusion flips under flex — do
not carry it past the backend decision), and the merge-back-undoes-abliteration
trap brokkr caught in his own advice.
Index stays at 291 lines, under the soft cap. No archival this run.
4.1 KiB
The 8.6% MFU was an accounting artifact — attention on Ampere kernels
[2026-08-25]
The answer
Real utilisation was 17-20%, inside the honest stock band. The 8.6% divided the intended (windowed) FLOPs by the wall time the dense reality took.
nominal billed 27.1 TFLOPS x 34.85 s = 9.4e14 FLOP
dense-sliding extra 25 layers, 2 seqs, 4 passes = +8.2e14
padded full layers lose the causal skip = +3.5e14
work performed ~ 1.8e15 = 51-61 TFLOPS
The card was doing ~2x the arithmetic the architecture specifies, and the excess was the sliding window being computed and thrown away.
Two independent methods agreed
scaling fit (3 points, 2 params, residuals <3ms over 8x range)
A = 6.87e-4 s/token B = 8.85e-8 s/token^2
quadratic share: 20.9% @ w=2048 -> 67.8% @ w=16384
kernel table (device rows only)
attention 22,835.8 ms 65.2% fmha_cutlass*_sm80
dense GEMM 2,774.0 ms 7.9%
other 5,739.0 ms 16.4%
67.8% vs 65.2% — 2.6 points apart, no shared assumptions. The two-term fit needed no constant term, which refutes launch-bound outright (3,840 expert-GEMM launches per forward are not the cost).
The mechanism, source-verified by brokkr's panel (arm: Bil)
masking_utils.py:292-301 _ignore_causal_mask_sdpa requires
kv_length < local_attention_size. 16384 >= 1024,
so THE SLIDING MASK ALWAYS MATERIALISES.
sdp_utils_cpp.h:259-267 flash rejects ANY explicit mask
sdp_utils.cpp:647 cuDNN head_dim capped at 128 -> unreachable
Context.h:480-485 prefer-cuDNN needs major 9 or 10; sm_120 is 12
⚠ The kernels are sm80 — Ampere-generation CUTLASS on a Blackwell card,
with the forward on gmem, the memory-efficient backend's slowest fallback tier.
What actually fixed it
Bucketing (bucket-to-pair, shuffle-to-mix) — 29.9% padding -> 0.0%, and
78.3% of micro-batches become exactly zero-pad, which puts the 5 global layers
back on is_causal. Measured: padding costs 9.4% MORE time for 24% LESS work
at fixed width, because an explicit mask knocks those layers off the fast path.
⚠ Bucket size is NOT a diversity knob. Swept across a 256x range, roots per accumulation window stayed flat at 3.54-3.61. The global micro-batch shuffle does all the mixing; the bucket only costs padding. Use the tightest bucket.
flex_attention — Triton-generated so it compiles for sm_120 instead of
shipping sm_80 binaries. 21.7x on sliding layers, 2.1x on global. Needs mandatory
kernel_options at 32x32 blocks: 64x32 needs 102,400 bytes against a
101,376-byte hardware ceiling — misses by 1 KB, and Triton is already opting
into the full 99 KB, so it is the card, not a default.
⚠⚠ The trap that produced TWO wrong published conclusions
torch._dynamo defaults to a recompile ceiling of 8. Every distinct sequence
width is a new shape. On hitting the ceiling dynamo does not error — it silently
falls back to UNCOMPILED flex, which is ~20x slower AND documented to "not work
with the backwards pass and may produce incorrect results."
That artifact produced a bogus 0.76x slowdown and a bogus 2.9% loss divergence, and I believed and reported both. Raising the limit to 256 flipped the speed result to 1.41x.
The loss divergence turned out to be real but benign — adjudicated against fp32 MATH ground truth, both backends sit ~2e-3 from truth with flex fractionally CLOSER at every width. Do not re-open it by comparing the two backends to each other; that cannot answer it. Compare to fp32.
Process lesson
brokkr's panel produced four self-retractions in ninety minutes. Every retraction was a derivation; every survivor was a measurement. And the whole head_dim-512 SDP problem was already documented in zerofata's published Axolotl config since April — the right first stop for "why is this architecture slow" is practitioner configs for that exact base, before any panel.
Playbook: docs/pfi/training-throughput-playbook.md, commit 7b5fd91.