memory: the MFU hunt — two hypotheses measured and killed, consult dispatched

Records what has actually been ruled out rather than what is suspected.

The hardware is fine: a plain dense GEMM at the same shape reaches
97.1% of the benchmarked 313.8 TFLOPS peak.

The Python expert loop is not the cause, which was my hypothesis and I
was confident in it. transformers' grouped_mm experts backend runs
0.9% SLOWER than eager with bit-identical output and identical peak
memory, and torch 2.13 has the kernel available, so it is not falling
back for lack of one.

MoE is not the bottleneck at all. Isolated at real shapes the block
runs at 26.5% of peak with 36% of its time in pure gather/scatter, and
a dispatch-free bmm version would reach 80.9% — but the whole MoE
contribution is only about 10% of a step. Making it free buys 7%.

So roughly 90% of the time is unaccounted for. The leading untested
hypothesis is that the five full_attention layers use global_head_dim
512, above FlashAttention-2's 256 cap, which would push SDPA onto a
slow backend for O(n^2) attention at sequence 16384.

Also records that the earlier 5% MFU figure was wrong in two ways —
unpadded tokens and a guessed peak — and that the operator caught it.
Padding is real but secondary at 29.9%.

Consult dispatched to brokkr-smithy-dev for the frontier-dwarf panel.

persistent-memory.md

@@ -111,8 +111,8 @@ no longer deployed sidecars here. See Recent decisions.)
 _As of 2026-08-25 ~04:20Z — the ERP/RP tune is TRAINING on ana-ml2 GPU0, ~17h, unattended. The homepage and char-rp arcs closed earlier. **The live thread is the run itself plus a parallel question: whether a fused MoE kernel lands fast enough to justify restarting it.**_
 
 - **🟢 THE ERP TUNE IS RUNNING (launched 2026-08-24 ~20:40 PDT, ETA ~13h → ~09:40 PDT 08-25).** GPU0 on ana-ml2, dedicated. `gen` relocated to GPU1 and healthy; **`sec`/mog-sec STOPPED for the whole run, operator-ruled ("let it run, keep sec down")**. Restore = `playbooks/ana-ml2-training-window-close.yaml` (gates on GPU0 idle; `--var allow_busy_gpu0=true` to override). Harness **eitri-smithy `997c4a4`** at `/tank/erp-tune/eitri-smithy`, venv `/tank/erp-tune/venv` (torch 2.13.0+cu130, transformers 5.15.1, peft 0.20.0, sm_120 verified), config `/tank/erp-tune/run-01.json`, log `/tank/erp-tune/run-01.log`, output `/tank/erp-tune/run-01/`. **Config: BF16 (NOT QLoRA), max_seq_len 16384, mb2×accum8 → 1,312 steps, r64/α128, 205 modules, 74,342,400 trainable.** Step-10 loss **3.664**, grad_norm 5.178 — ⚠ above brokkr's 1.8–3.0 band but the doubled-divisor signature was ~0.25, so `num_items_in_batch` is NOT double-applied; hypothesis = the mix is 52.9% literary prose where every token is a loss target. GPU0 runs **84,222 MiB of 97,887** (above my measured 79.71 GiB worst case — adjacent `#w0`/`#w1` windows share micro-batches systematically, exactly as brokkr predicted). **Encode is CACHED** (`run-01/encode-cache/`, keyed on encode_version+max_seq_len+template sha) so a restart costs ~2.5 min, not the 4.3h it would single-threaded. ⚠ **encode_version must be BUMPED on ANY encoder change** — that has mattered five times. **RESUME: use `/tank/erp-tune/resume-run-01.sh`, NEVER the original launch command** — that one starts `rm -rf /tank/erp-tune/run-01`, which destroys the 609 MB encode cache AND every checkpoint. First checkpoint at step 100; `save_steps=100` at ~46.5 s/it = **~73 min of crash exposure** per interval. → `docs/pfi/gemma4-erp-tune-sizing.md`
-- **⚠ MFU IS 8.6% AND THE CAUSE IS A PYTHON LOOP — measured, root-caused, parked.** 27.1 TFLOPS achieved against a **benchmarked 313.8 TFLOPS** dense BF16 peak (6.0% excluding padding). Cause is `transformers/models/gemma4/modeling_gemma4.py` ~line 1307: `for expert_idx in expert_hit:` — 128 experts × 30 layers = 3,840 loop iterations per forward, ~11,500 per step under gradient checkpointing, each a where+gather+two small GEMMs+scatter-add in eager mode. vLLM serves this model fast because it has fused grouped-GEMM MoE kernels; HF Trainer has none. **Padding is a secondary 29.9% tax** (82,337,318 padded tokens vs 57,733,156 real; `group_by_length` would make it 1.000×). ⚠ My first estimate of "5% MFU" was WRONG — I divided by UNPADDED tokens and compared against a GUESSED peak. Operator pushed back and was right to. **Break-even for restarting: a fix landing in T hours with speedup S wins if `T + 17/S < 16` — at S=2 that allows 7.5h of work, at S=4 it allows 11.75h.** → park `fused-moe-kernel-path-for-gemma-4-moe-training` (id 47)
-- **⚠ TWELVE HARNESS DEFECTS FOUND BEFORE THE RUN — the two lessons worth keeping.** (1) **INERT GATES**: `auditcore` (CSAM, rc-2 never fired in 42,662 records) and `validate_vision_keys` (compared `model.state_dict()` against itself — could not fail on any input). The question that catches them is not *"does this check pass?"* but **"CAN this check fail?"** (2) **INVARIANT ENFORCED ON ONE PATH BUT NOT ITS SIBLING** — my own INV-T9(5) bug: a window with no assistant turn was blocked on the cut path and not the fits path, so a trailing user-only remainder became a zero-loss window. Also: `enable_input_require_grads()` is MANDATORY beside gradient checkpointing on a frozen base or **every adapter stays at its init and the run completes successfully having learned nothing**; the upstream Gemma-4 template FORWARD-SCANS (suppresses the closing `<turn|>` before another assistant msg) so incremental rendering cannot tile — merge assistant runs first, joined with `"\n\n"` (a deliberate data change fixing run-ons on 42.5% of pippa).
+- **⚠ MFU IS 8.6% AND I HAVE DISPROVEN MY OWN HYPOTHESIS TWICE — CONSULT OUT TO THE FRONTIER DWARVES.** 27.1 TFLOPS against a **benchmarked 313.8 TFLOPS** peak; one fwd+bwd at the real shape is **34.85s** (4 passes within 1%). **RULED OUT, with numbers, not argument:** (1) **hardware** — a plain dense GEMM hits **97.1% of peak** (304.6 TFLOPS), card draws 279-292W of 300W; (2) **the Python expert loop** — swapping to transformers' `grouped_mm` experts backend gave **35.149s vs eager's 34.847s, bit-identical output (max_abs_diff EXACTLY 0.0), same 75.8 GiB**, and torch 2.13 HAS both `F.grouped_mm` and `torch._grouped_mm`, so it is not a missing kernel; `batched_mm` both OOMs and MISMATCHES (rel 0.79 — it computes all 128 experts per token); (3) **MoE being the bottleneck at all** — isolated at real shapes the MoE block is **37.54 ms at 26.5% of peak**, of which **13.39 ms is pure gather/scatter dispatch** and a dispatch-free `bmm` version would be **12.28 ms at 80.9% of peak** — but **30 layers × 37.54 ms × 3 (fwd+recompute+bwd) ≈ 3.4s of a 34.85s step, only ~10%.** Making MoE free buys ~7%. **~90% of the time is somewhere I have not looked.** ⚠ **LEADING UNTESTED HYPOTHESIS: the 5 `full_attention` layers use `global_head_dim: 512`, and FlashAttention-2 caps head_dim at 256** — if that pushes torch SDPA onto the mem-efficient or math backend, 5 layers are doing O(n²) attention at seq 16384 on a slow path. Other un-excluded candidates: the chunked CE (vocab 262,144 + softcap, 1024-tok chunks re-materialised under `checkpoint`), the `attention_k_eq_v` K=V path, grad-ckpt × MoE dispatch interaction, PEFT's wrapper on 205 modules. ⚠ **My earlier "5% MFU" was ALSO wrong** (divided by UNPADDED tokens, compared against a GUESSED peak) — operator caught it. Padding is a real but secondary **29.9%** tax (82,337,318 padded vs 57,733,156 real). Artifacts: `/tank/erp-tune/{micro_moe,bench_moe,bench_bf16}.py`. → park id 47, althing thread `01M0VKBPZD71Q302NH84BXHTWS`
+
 - **🛑 THE CORPUS GATE — OVERRIDDEN FOR THIS ONE RUN ONLY (operator, 2026-08-25).** Grant staged at `/mnt/smithy/datasets/derived/_recipes/erp-seat-sft-r1/TRAINING-ELIGIBILITY-OVERRIDE.md`. ⚠ It does NOT flip any root's `training_eligible` flag — they still read `false` and name both blockers, deliberately, so the signal survives. **A second run needs a second grant.** Provenance records `training_eligibility_override: operator-2026-08-25-rnd-run` + both blockers + both substitute controls; those keys are in `REQUIRED_PROVENANCE` as present-with-explicit-null so a future run cannot silently omit them. Background: Every `clean-v1/CLEANROOT.json` carries `training_eligible: false` with `training_blocked_by: [contamination-scan-not-implemented, stage-2-csam-detector-inert]`, and the recipe itself says *"nothing here is Charter §3 training-eligible"*. ⚠ **`scoped_grant: operator-2026-08-22` is NOT training clearance** — it governs INV-4 one-way tier inheritance (the adapter is permanently `internal-erp-rnd`, never distributable). I initially misread the grant as authorization and told brokkr I was proceeding; **brokkr-smithy-dev — who WROTE those fields — corrected it**: *"I wrote them so that exactly this would happen… do not take my word as clearance; I do not have the authority to give it."* **The detector is measured-inert, not suspected:** `auditcore` v3.7.2 returned its hard-drop rc-2 **zero times across 42,662 raw RP records**, its printed verdict ignores its own printed threshold, and it passed a blind-audit-identified record of sexual content involving a participant the text marks as a child (`pippa-5083`, composite 4.34 vs threshold 6.5). → `research/R47-premium-corpus-gate/FINDING-auditcore-inert.md`, Contract Amendment 11. **I verified the one decisive thing:** `pippa-5083` IS in `kept-manifest.jsonl` (4,551 rows) but **ABSENT from `recipe-dedup-kept.jsonl` (20,473 rows)** — the survivor list the harness gates on — so brokkr's substitute *stage-A lexical* screen caught it. That is one known instance caught by a stopgap; it says nothing about what the screen misses. **Both brokkr and I recommend STOPPING; only an explicit operator override opens it.** Neither blocker is hours of work (the 13-gram scanner is spec-only, DRAFT since 2026-06-01; the detector needs replacing). ⚠ **Do NOT stage or copy corpus content while gated.**
 - **🟢 SIZING + SEAT CALL — DONE AND EXECUTED, full detail in the doc.** QLoRA structurally unavailable (fused 3-D experts vs bitsandbytes' nn.Linear walk); plain BF16 LoRA; chunked CE mandatory (naive CE OOMs at seq16384, 81.93 GiB at seq8192); `v_proj` exists on only 25 of 30 layers (`attention_k_eq_v`, K=V sharing — real, not a miss). `gen` moved to GPU1, `sec` down, GPU0 dedicated. → `docs/pfi/gemma4-erp-tune-sizing.md`, `playbooks/ana-ml2-training-window-{open,close}.yaml`
 - **⚠ TELL EITRI BEFORE HE HARD-CODES: the trainee base changed.** Contract still names the stock BF16. It is now `/tank/aimodels/gemma4-26b-a4b-it-heretic-bf16` (llmfan46). **Base path AND chat-template path must be config keys, not constants** — and the template must point at upstream's (`gemma4-26b-a4b-it-bf16/chat_template.jinja`), never the base's own, or training renders a different prompt than production serves.