memory: snapshot — run 7 training on gx10; run 6 TRANSFERRED after CSAM adjudication; erp-tune-v6-nvfp4a16 live as trial; ESH/YTVC/webhook repairs; ana-ml2 routes persisted; tank/zroot actions deferred to next session. Index 830→271 lines: 27 decisions + 8 abandoned archived, superseded in-flight blocks archived verbatim
This commit is contained in:
@@ -1,86 +0,0 @@
|
||||
# The 8.6% MFU was an accounting artifact — attention on Ampere kernels
|
||||
|
||||
`[2026-08-25]`
|
||||
|
||||
## The answer
|
||||
|
||||
**Real utilisation was 17-20%, inside the honest stock band.** The 8.6% divided
|
||||
the *intended* (windowed) FLOPs by the wall time the *dense* reality took.
|
||||
|
||||
nominal billed 27.1 TFLOPS x 34.85 s = 9.4e14 FLOP
|
||||
dense-sliding extra 25 layers, 2 seqs, 4 passes = +8.2e14
|
||||
padded full layers lose the causal skip = +3.5e14
|
||||
work performed ~ 1.8e15 = 51-61 TFLOPS
|
||||
|
||||
The card was doing ~2x the arithmetic the architecture specifies, and the excess
|
||||
was the sliding window being computed and thrown away.
|
||||
|
||||
## Two independent methods agreed
|
||||
|
||||
scaling fit (3 points, 2 params, residuals <3ms over 8x range)
|
||||
A = 6.87e-4 s/token B = 8.85e-8 s/token^2
|
||||
quadratic share: 20.9% @ w=2048 -> 67.8% @ w=16384
|
||||
|
||||
kernel table (device rows only)
|
||||
attention 22,835.8 ms 65.2% fmha_cutlass*_sm80
|
||||
dense GEMM 2,774.0 ms 7.9%
|
||||
other 5,739.0 ms 16.4%
|
||||
|
||||
**67.8% vs 65.2% — 2.6 points apart, no shared assumptions.** The two-term fit
|
||||
needed no constant term, which refutes launch-bound outright (3,840 expert-GEMM
|
||||
launches per forward are not the cost).
|
||||
|
||||
## The mechanism, source-verified by brokkr's panel (arm: Bil)
|
||||
|
||||
masking_utils.py:292-301 _ignore_causal_mask_sdpa requires
|
||||
kv_length < local_attention_size. 16384 >= 1024,
|
||||
so THE SLIDING MASK ALWAYS MATERIALISES.
|
||||
sdp_utils_cpp.h:259-267 flash rejects ANY explicit mask
|
||||
sdp_utils.cpp:647 cuDNN head_dim capped at 128 -> unreachable
|
||||
Context.h:480-485 prefer-cuDNN needs major 9 or 10; sm_120 is 12
|
||||
|
||||
⚠ **The kernels are `sm80` — Ampere-generation CUTLASS on a Blackwell card**,
|
||||
with the forward on `gmem`, the memory-efficient backend's slowest fallback tier.
|
||||
|
||||
## What actually fixed it
|
||||
|
||||
**Bucketing (bucket-to-pair, shuffle-to-mix)** — 29.9% padding -> 0.0%, and
|
||||
78.3% of micro-batches become exactly zero-pad, which puts the 5 global layers
|
||||
back on `is_causal`. Measured: padding costs **9.4% MORE time for 24% LESS work**
|
||||
at fixed width, because an explicit mask knocks those layers off the fast path.
|
||||
|
||||
⚠ **Bucket size is NOT a diversity knob.** Swept across a 256x range, roots per
|
||||
accumulation window stayed flat at 3.54-3.61. The global micro-batch shuffle does
|
||||
all the mixing; the bucket only costs padding. Use the tightest bucket.
|
||||
|
||||
**flex_attention** — Triton-generated so it compiles for sm_120 instead of
|
||||
shipping sm_80 binaries. 21.7x on sliding layers, 2.1x on global. Needs mandatory
|
||||
`kernel_options` at 32x32 blocks: 64x32 needs 102,400 bytes against a
|
||||
**101,376-byte hardware ceiling** — misses by 1 KB, and Triton is already opting
|
||||
into the full 99 KB, so it is the card, not a default.
|
||||
|
||||
## ⚠⚠ The trap that produced TWO wrong published conclusions
|
||||
|
||||
`torch._dynamo` defaults to a recompile ceiling of **8**. Every distinct sequence
|
||||
width is a new shape. On hitting the ceiling dynamo does not error — it silently
|
||||
falls back to UNCOMPILED flex, which is ~20x slower AND documented to *"not work
|
||||
with the backwards pass and may produce incorrect results."*
|
||||
|
||||
That artifact produced a bogus **0.76x slowdown** and a bogus **2.9% loss
|
||||
divergence**, and I believed and reported both. Raising the limit to 256 flipped
|
||||
the speed result to 1.41x.
|
||||
|
||||
The loss divergence turned out to be real but benign — adjudicated against fp32
|
||||
MATH ground truth, both backends sit ~2e-3 from truth with flex fractionally
|
||||
CLOSER at every width. **Do not re-open it by comparing the two backends to each
|
||||
other; that cannot answer it. Compare to fp32.**
|
||||
|
||||
## Process lesson
|
||||
|
||||
brokkr's panel produced **four self-retractions in ninety minutes**. Every
|
||||
retraction was a derivation; every survivor was a measurement. And the whole
|
||||
head_dim-512 SDP problem was **already documented in zerofata's published Axolotl
|
||||
config since April** — the right first stop for "why is this architecture slow"
|
||||
is practitioner configs for that exact base, before any panel.
|
||||
|
||||
Playbook: `docs/pfi/training-throughput-playbook.md`, commit `7b5fd91`.
|
||||
Reference in New Issue
Block a user