memory: snapshot — tune training unattended, MFU root-caused to a Python expert loop

The in-flight section is rewritten around the run itself rather than
the decisions that led to it. The sizing and seat-call bullet collapses
to a pointer now that both are executed; its detail lives in
docs/pfi/gemma4-erp-tune-sizing.md.

Adds the measured MFU finding: 27.1 TFLOPS against a benchmarked 313.8
TFLOPS peak, root-caused by reading the source rather than inferring —
transformers runs the Gemma-4 experts in a Python loop, 128 experts
across 30 layers, roughly 11,500 iterations per optimizer step under
gradient checkpointing. Padding is a secondary 29.9% tax.

Records that my first estimate of 5% MFU was wrong in two compounding
ways: divided by unpadded tokens, and compared against a guessed peak
rather than a measured one. The operator pushed back on the number and
was right to.

The fused MoE kernel is deferred work with a tracking surface — park
id 47 — per the snapshot rule that deferred decisions go in Recent
decisions with a pointer, never into the volatile in-flight section.

Also records the resume trap: the original launch command begins with
rm -rf on the output directory, which would destroy both the encode
cache and every checkpoint. resume-run-01.sh exists so that cannot
happen.

persistent-memory.md

@@ -1,6 +1,6 @@
 # Persistent memory — eshpfi-management
 
-_Last updated: 2026-08-24_
+_Last updated: 2026-08-25_
 
 > **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its
 > `Written:` stamp is under an hour old, read it (it carries the in-flight
@@ -108,12 +108,13 @@ no longer deployed sidecars here. See Recent decisions.)
   (no NOPASSWD)** — stage model pulls to `/home`, not root-owned `/worktank`.
 ## Current state / in-flight
 
-_As of 2026-08-24 (late) — a very long ops session. The homepage arc and the char-rp arc both closed. **The live thread is the ERP/RP trainee: QLoRA sizing is the next conversation and brokkr-smithy-dev is waiting on it.**_
+_As of 2026-08-25 ~04:20Z — the ERP/RP tune is TRAINING on ana-ml2 GPU0, ~17h, unattended. The homepage and char-rp arcs closed earlier. **The live thread is the run itself plus a parallel question: whether a fused MoE kernel lands fast enough to justify restarting it.**_
 
-- **🟢 THE ERP TUNE IS RUNNING (launched 2026-08-24 ~20:40 PDT, ETA ~13h → ~09:40 PDT 08-25).** GPU0 on ana-ml2, dedicated. `gen` relocated to GPU1 and healthy; **`sec`/mog-sec STOPPED for the whole run, operator-ruled ("let it run, keep sec down")**. Restore = `playbooks/ana-ml2-training-window-close.yaml` (gates on GPU0 idle; `--var allow_busy_gpu0=true` to override). Harness **eitri-smithy `997c4a4`** at `/tank/erp-tune/eitri-smithy`, venv `/tank/erp-tune/venv` (torch 2.13.0+cu130, transformers 5.15.1, peft 0.20.0, sm_120 verified), config `/tank/erp-tune/run-01.json`, log `/tank/erp-tune/run-01.log`, output `/tank/erp-tune/run-01/`. **Config: BF16 (NOT QLoRA), max_seq_len 16384, mb2×accum8 → 1,312 steps, r64/α128, 205 modules, 74,342,400 trainable.** Step-10 loss **3.664**, grad_norm 5.178 — ⚠ above brokkr's 1.8–3.0 band but the doubled-divisor signature was ~0.25, so `num_items_in_batch` is NOT double-applied; hypothesis = the mix is 52.9% literary prose where every token is a loss target. GPU0 runs **84,222 MiB of 97,887** (above my measured 79.71 GiB worst case — adjacent `#w0`/`#w1` windows share micro-batches systematically, exactly as brokkr predicted). **Encode is CACHED** (`run-01/encode-cache/`, keyed on encode_version+max_seq_len+template sha) so a restart costs ~2.5 min, not the 4.3h it would single-threaded. ⚠ **encode_version must be BUMPED on ANY encoder change** — that has mattered five times. → `docs/pfi/gemma4-erp-tune-sizing.md`
+- **🟢 THE ERP TUNE IS RUNNING (launched 2026-08-24 ~20:40 PDT, ETA ~13h → ~09:40 PDT 08-25).** GPU0 on ana-ml2, dedicated. `gen` relocated to GPU1 and healthy; **`sec`/mog-sec STOPPED for the whole run, operator-ruled ("let it run, keep sec down")**. Restore = `playbooks/ana-ml2-training-window-close.yaml` (gates on GPU0 idle; `--var allow_busy_gpu0=true` to override). Harness **eitri-smithy `997c4a4`** at `/tank/erp-tune/eitri-smithy`, venv `/tank/erp-tune/venv` (torch 2.13.0+cu130, transformers 5.15.1, peft 0.20.0, sm_120 verified), config `/tank/erp-tune/run-01.json`, log `/tank/erp-tune/run-01.log`, output `/tank/erp-tune/run-01/`. **Config: BF16 (NOT QLoRA), max_seq_len 16384, mb2×accum8 → 1,312 steps, r64/α128, 205 modules, 74,342,400 trainable.** Step-10 loss **3.664**, grad_norm 5.178 — ⚠ above brokkr's 1.8–3.0 band but the doubled-divisor signature was ~0.25, so `num_items_in_batch` is NOT double-applied; hypothesis = the mix is 52.9% literary prose where every token is a loss target. GPU0 runs **84,222 MiB of 97,887** (above my measured 79.71 GiB worst case — adjacent `#w0`/`#w1` windows share micro-batches systematically, exactly as brokkr predicted). **Encode is CACHED** (`run-01/encode-cache/`, keyed on encode_version+max_seq_len+template sha) so a restart costs ~2.5 min, not the 4.3h it would single-threaded. ⚠ **encode_version must be BUMPED on ANY encoder change** — that has mattered five times. **RESUME: use `/tank/erp-tune/resume-run-01.sh`, NEVER the original launch command** — that one starts `rm -rf /tank/erp-tune/run-01`, which destroys the 609 MB encode cache AND every checkpoint. First checkpoint at step 100; `save_steps=100` at ~46.5 s/it = **~73 min of crash exposure** per interval. → `docs/pfi/gemma4-erp-tune-sizing.md`
+- **⚠ MFU IS 8.6% AND THE CAUSE IS A PYTHON LOOP — measured, root-caused, parked.** 27.1 TFLOPS achieved against a **benchmarked 313.8 TFLOPS** dense BF16 peak (6.0% excluding padding). Cause is `transformers/models/gemma4/modeling_gemma4.py` ~line 1307: `for expert_idx in expert_hit:` — 128 experts × 30 layers = 3,840 loop iterations per forward, ~11,500 per step under gradient checkpointing, each a where+gather+two small GEMMs+scatter-add in eager mode. vLLM serves this model fast because it has fused grouped-GEMM MoE kernels; HF Trainer has none. **Padding is a secondary 29.9% tax** (82,337,318 padded tokens vs 57,733,156 real; `group_by_length` would make it 1.000×). ⚠ My first estimate of "5% MFU" was WRONG — I divided by UNPADDED tokens and compared against a GUESSED peak. Operator pushed back and was right to. **Break-even for restarting: a fix landing in T hours with speedup S wins if `T + 17/S < 16` — at S=2 that allows 7.5h of work, at S=4 it allows 11.75h.** → park `fused-moe-kernel-path-for-gemma-4-moe-training` (id 47)
 - **⚠ TWELVE HARNESS DEFECTS FOUND BEFORE THE RUN — the two lessons worth keeping.** (1) **INERT GATES**: `auditcore` (CSAM, rc-2 never fired in 42,662 records) and `validate_vision_keys` (compared `model.state_dict()` against itself — could not fail on any input). The question that catches them is not *"does this check pass?"* but **"CAN this check fail?"** (2) **INVARIANT ENFORCED ON ONE PATH BUT NOT ITS SIBLING** — my own INV-T9(5) bug: a window with no assistant turn was blocked on the cut path and not the fits path, so a trailing user-only remainder became a zero-loss window. Also: `enable_input_require_grads()` is MANDATORY beside gradient checkpointing on a frozen base or **every adapter stays at its init and the run completes successfully having learned nothing**; the upstream Gemma-4 template FORWARD-SCANS (suppresses the closing `<turn|>` before another assistant msg) so incremental rendering cannot tile — merge assistant runs first, joined with `"\n\n"` (a deliberate data change fixing run-ons on 42.5% of pippa).
 - **🛑 THE CORPUS GATE — OVERRIDDEN FOR THIS ONE RUN ONLY (operator, 2026-08-25).** Grant staged at `/mnt/smithy/datasets/derived/_recipes/erp-seat-sft-r1/TRAINING-ELIGIBILITY-OVERRIDE.md`. ⚠ It does NOT flip any root's `training_eligible` flag — they still read `false` and name both blockers, deliberately, so the signal survives. **A second run needs a second grant.** Provenance records `training_eligibility_override: operator-2026-08-25-rnd-run` + both blockers + both substitute controls; those keys are in `REQUIRED_PROVENANCE` as present-with-explicit-null so a future run cannot silently omit them. Background: Every `clean-v1/CLEANROOT.json` carries `training_eligible: false` with `training_blocked_by: [contamination-scan-not-implemented, stage-2-csam-detector-inert]`, and the recipe itself says *"nothing here is Charter §3 training-eligible"*. ⚠ **`scoped_grant: operator-2026-08-22` is NOT training clearance** — it governs INV-4 one-way tier inheritance (the adapter is permanently `internal-erp-rnd`, never distributable). I initially misread the grant as authorization and told brokkr I was proceeding; **brokkr-smithy-dev — who WROTE those fields — corrected it**: *"I wrote them so that exactly this would happen… do not take my word as clearance; I do not have the authority to give it."* **The detector is measured-inert, not suspected:** `auditcore` v3.7.2 returned its hard-drop rc-2 **zero times across 42,662 raw RP records**, its printed verdict ignores its own printed threshold, and it passed a blind-audit-identified record of sexual content involving a participant the text marks as a child (`pippa-5083`, composite 4.34 vs threshold 6.5). → `research/R47-premium-corpus-gate/FINDING-auditcore-inert.md`, Contract Amendment 11. **I verified the one decisive thing:** `pippa-5083` IS in `kept-manifest.jsonl` (4,551 rows) but **ABSENT from `recipe-dedup-kept.jsonl` (20,473 rows)** — the survivor list the harness gates on — so brokkr's substitute *stage-A lexical* screen caught it. That is one known instance caught by a stopgap; it says nothing about what the screen misses. **Both brokkr and I recommend STOPPING; only an explicit operator override opens it.** Neither blocker is hours of work (the 13-gram scanner is spec-only, DRAFT since 2026-06-01; the detector needs replacing). ⚠ **Do NOT stage or copy corpus content while gated.**
-- **🔴 SIZING DONE — AND IT KILLED THE QLoRA PREMISE. AWAITING THE SEAT CALL.** → `docs/pfi/gemma4-erp-tune-sizing.md` (measured, not estimated). **QLoRA is structurally unavailable on this architecture:** the checkpoint stores each layer's 128 experts as two fused 3-D `nn.Parameter` tensors (`layers.N.experts.gate_up_proj` `[128,1408,2816]`, `.down_proj` `[128,2816,704]` — note the missing `.weight` suffix), and `bitsandbytes` 4-bit replacement walks `nn.Linear` modules only. **88.5% of the model (22.84B params / 42.54 GiB) is untouchable; `load_in_4bit` saves ~3.1 GiB of 48.07 and does NOT error.** ⚠ **Eitri must not hard-code `BitsAndBytesConfig`** — it loads, reports 4-bit, and silently leaves 42.5 GiB BF16. Verdict: **plain LoRA on BF16**, ~57.6 GiB at micro-batch 1, +2.5 GiB per extra 8192-seq. ⚠ **The loss head is the real driver and was NOT in the brief:** vocab 262,144 × 8192 = 2.147B logits, plus `final_logit_softcapping 30.0` → **~28-30 GiB transient at BATCH 1** through naive HF CE (= ~85.6 GiB total, an OOM-on-first-long-sample). **Fused/chunked linear CE is mandatory and must be smoke-proven before the window is booked** (Liger may lack a Gemma-4 MoE patch). ⚠ **`v_proj` DOES NOT EXIST on layers 5/11/17/23/29** (`attention_k_eq_v` on the full-attention layers) — a `v_proj` target silently no-ops there and `k_proj` adapts K and V at once; 45.96M trainable at r64. ⚠ 7,104 packed seqs → **only 444 optimizer steps** at effective batch 16, which is why wall-clock checkpointing matters concretely. **OPERATOR CALL, EXECUTED 2026-08-24 EVENING: `gen` MOVED to GPU1, `sec` STOOD DOWN, GPU0 emptied entirely (95.60 GiB) for the tune** — strictly better than training beside gen. Both directions are playbooks: `playbooks/ana-ml2-training-window-{open,close}.yaml`. ⚠⚠ **ORDER IS LOAD-BEARING IN BOTH:** gen's `--gpu-memory-utilization 0.43` is a fraction of TOTAL card memory, so **42,091 MiB must be FREE AT STARTUP or the engine refuses to boot** — recreating gen onto GPU1 before stopping mog-sec takes the fleet's main seat down and LEAVES it down. Close mirrors it (mog-sec needs 50,901 MiB, so gen must vacate GPU1 first) and gates on GPU0 being idle so it cannot evict a running tune (`--var allow_busy_gpu0=true` to override). ⚠ **Invoke elway as `infra-ops@10.250.50.54`** — the `ana-ml2` ssh-target resolves to `lkraven`, no NOPASSWD sudo, elway aborts at its sudo probe. Revised run params on a whole card: **micro-batch 8, grad-accum 1 → 888 steps** (double the shared-card count, and ~4,096 rows/expert/step vs ~512 at mb=1, so much better MoE GEMM). ⚠ **KEEP gradient checkpointing ON** — dropping it saves ~17% wall-clock but forces mb=1, which craters MoE efficiency 8×; wide beats shallow. Scriberr STAYS on GPU1 (an earlier draft said move it to GPU0 — backwards now that GPU0 is the training card). Est. **4-10h, likely ~6.** Package as a `uv` venv on `/tank` — **root is 91% full (36 GB)** and `/var/lib/docker` is on it. ⚠ `sec` traffic arrives from the LiteLLM gateway (10.250.50.70); let the aliases FAIL, do not substitute another model.
+- **🟢 SIZING + SEAT CALL — DONE AND EXECUTED, full detail in the doc.** QLoRA structurally unavailable (fused 3-D experts vs bitsandbytes' nn.Linear walk); plain BF16 LoRA; chunked CE mandatory (naive CE OOMs at seq16384, 81.93 GiB at seq8192); `v_proj` exists on only 25 of 30 layers (`attention_k_eq_v`, K=V sharing — real, not a miss). `gen` moved to GPU1, `sec` down, GPU0 dedicated. → `docs/pfi/gemma4-erp-tune-sizing.md`, `playbooks/ana-ml2-training-window-{open,close}.yaml`
 - **⚠ TELL EITRI BEFORE HE HARD-CODES: the trainee base changed.** Contract still names the stock BF16. It is now `/tank/aimodels/gemma4-26b-a4b-it-heretic-bf16` (llmfan46). **Base path AND chat-template path must be config keys, not constants** — and the template must point at upstream's (`gemma4-26b-a4b-it-bf16/chat_template.jinja`), never the base's own, or training renders a different prompt than production serves.
 - **🟢 char-rp seat = Gemma-4 26B-A4B MoE NVFP4** on `:8016`, both aliases on ONE backend. **Currently DOWN by operator instruction** to hold GPU0 headroom for the tune. `gen` is UP and verified. MeroMero-v2 retained stopped in `created` state for rollback (stop-then-start; both bind :8016). → `persistent-memory.d/2026-08-24-charrp-gemma4-moe-swap-and-trainee.md`
 - **🟢 THREE trainee-relevant model dirs on `/tank/aimodels/`, NOT interchangeable:** `gemma4-26b-a4b-it-bf16` (stock, 49 GB — its chat_template is the canonical upstream one), `gemma4-26b-a4b-it-heretic-bf16` (llmfan46 abliterated, the trainee), `gemma4-26b-a4b-it-abliterated-bf16` (TrevorJS, KL 0.09, alternate). Plus `-nvfp4` (served) and `-nvfp4a16` (activation control). ⚠ **BF16 cannot coexist with `gen`** — 48.07 GiB of weights on a 94.97 GiB card. Every BF16 window means gen stops.
@@ -132,6 +133,8 @@ _As of 2026-08-24 (late) — a very long ops session. The homepage arc and the c
 
 ## Recent decisions
 
+- `[2026-08-25]` **Fused MoE kernel path — DEFERRED, tracked at park `fused-moe-kernel-path-for-gemma-4-moe-training` (id 47).** Operator: "note the fused MoE kernel for round two… if we nail it soon, the math has us wanting to restart the run anyway." Training MFU is **8.6%** (27.1 of a benchmarked 313.8 TFLOPS) because `transformers` runs the Gemma-4 experts in a Python loop — 128 experts × 30 layers, ~11,500 iterations per step under gradient checkpointing. ⚠ **The same fused 3-D expert layout that made bitsandbytes skip 88.5% of the model is exactly what a grouped GEMM wants** — the format is good for storage and for fused kernels, and hostile only to naive iteration. Two fixes: `group_by_length` (−29.9% compute, free, but breaks the seeded order manifest and re-opens a batch-composition call brokkr already made) and a grouped-GEMM/compiled MoE forward (the remaining ~10×). **Not applied to the live run** — restarting mid-flight to change batch ordering was judged a bad trade at step ~50 of 1,312.
+- `[2026-08-25]` **The ERP/RP tune LAUNCHED after 12 harness defects and an operator override of the corpus gate.** Four of the twelve would have crashed the run; two were INERT GATES that passed because they could not fail. Run is `/tank/erp-tune/run-01`, harness eitri-smithy `997c4a4`. Full arc — override, defects, sizing, the measured MFU — in the in-flight section and `docs/pfi/gemma4-erp-tune-sizing.md`.
 - `[2026-08-24]` **char-rp seat swapped to the Gemma-4 26B-A4B MoE; abliterated trainee base staged and measured.** OOM root-caused to `--gpu-memory-utilization` not covering CUDA context (and to gen's footprint GROWING WITH UPTIME); a benchmark finding retracted because it scored below chance; abliteration isolated at −0.6 core points but it MOVES capability rather than removing it. → `persistent-memory.d/2026-08-24-charrp-gemma4-moe-swap-and-trainee.md`
 - `[2026-08-24]` **Serving the tuned ERP model: LoRA-on-NVFP4 PREFERRED, merged weights the expected fallback — and the recorded objection may be STALE.** Operator: "if you CAN load it as a lora, all the better, the issue is that we will want to run nvfp4 weights, which we had some serious trouble with loading loras on top of nvfp4." ⚠ **The archived root-cause says it was NOT NVFP4-specific**: `[2026-07-07]` vLLM 0.24.0 qwen3_5 LoRA application was a silent no-op (#47639, regression from #37912) — adapter loads HTTP 200, zero deltas at inference, proven **quant-agnostic (NVFP4 AND FP8 both inert)** and adapter-format-agnostic by a 3-peer dwarf panel. Fix PR #47640 was OPEN then. **ana-ml2 is FAR past 0.24.0 and the box runs a SPREAD, not one version** (measured 2026-08-24): `gen` on `nightly-311b3513` = **0.27.2rc1.dev150**, `mog-sec` on `nightly-e9d1398d` = 0.26.1rc1.dev1102, the small seats still on 0.24.0, and char-rp/trainee-bench pinned to v0.26.0. ⚠ **`vllm/vllm-openai:v0.27.1` is already ON DISK, unused** — a TAGGED release, which is the right retest target: no nightly variance, no pull, ~4 months past the diagnosis. So: RETEST hot-swap LoRA on **v0.27.1** before designing around merge — it is cheap, and if it works the post-tune gate can be two aliases on one engine. If it still no-ops, merged weights it is, which means the harness must EMIT merged weights and Eitri needs that in the contract while he is early. Tracked at this snapshot commit; settle it in the QLoRA sizing conversation.
 - `[2026-08-24]` **Homepage rebuilt on Australis Skyfall; light mode shipped.** Two findings worth more than the theme: **(a)** the Skyfall bundle including its canonical light ramp was sitting in this repo's git history at `45c1995` — check `git show` before concluding a vendored design asset is lost; **(b)** removing `theme:` from `settings.yaml` deterministically breaks the dashboard render (six recreates empty, restoring the key fixed it in 12s), which is the first confirmed cause of the "tab bar goes missing" symptom. Retires the `homepage.log` size lead from earlier the same day — it did nothing on this episode. → `persistent-memory.d/2026-08-24-homepage-uniform-grid.md`