diff --git a/persistent-memory.md b/persistent-memory.md index ea20754..92ee853 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -111,8 +111,8 @@ no longer deployed sidecars here. See Recent decisions.) _As of 2026-08-25 ~04:20Z β€” the ERP/RP tune is TRAINING on ana-ml2 GPU0, ~17h, unattended. The homepage and char-rp arcs closed earlier. **The live thread is the run itself plus a parallel question: whether a fused MoE kernel lands fast enough to justify restarting it.**_ - **🟒 THE ERP TUNE IS RUNNING (launched 2026-08-24 ~20:40 PDT, ETA ~13h β†’ ~09:40 PDT 08-25).** GPU0 on ana-ml2, dedicated. `gen` relocated to GPU1 and healthy; **`sec`/mog-sec STOPPED for the whole run, operator-ruled ("let it run, keep sec down")**. Restore = `playbooks/ana-ml2-training-window-close.yaml` (gates on GPU0 idle; `--var allow_busy_gpu0=true` to override). Harness **eitri-smithy `997c4a4`** at `/tank/erp-tune/eitri-smithy`, venv `/tank/erp-tune/venv` (torch 2.13.0+cu130, transformers 5.15.1, peft 0.20.0, sm_120 verified), config `/tank/erp-tune/run-01.json`, log `/tank/erp-tune/run-01.log`, output `/tank/erp-tune/run-01/`. **Config: BF16 (NOT QLoRA), max_seq_len 16384, mb2Γ—accum8 β†’ 1,312 steps, r64/Ξ±128, 205 modules, 74,342,400 trainable.** Step-10 loss **3.664**, grad_norm 5.178 β€” ⚠ above brokkr's 1.8–3.0 band but the doubled-divisor signature was ~0.25, so `num_items_in_batch` is NOT double-applied; hypothesis = the mix is 52.9% literary prose where every token is a loss target. GPU0 runs **84,222 MiB of 97,887** (above my measured 79.71 GiB worst case β€” adjacent `#w0`/`#w1` windows share micro-batches systematically, exactly as brokkr predicted). **Encode is CACHED** (`run-01/encode-cache/`, keyed on encode_version+max_seq_len+template sha) so a restart costs ~2.5 min, not the 4.3h it would single-threaded. ⚠ **encode_version must be BUMPED on ANY encoder change** β€” that has mattered five times. **RESUME: use `/tank/erp-tune/resume-run-01.sh`, NEVER the original launch command** β€” that one starts `rm -rf /tank/erp-tune/run-01`, which destroys the 609 MB encode cache AND every checkpoint. First checkpoint at step 100; `save_steps=100` at ~46.5 s/it = **~73 min of crash exposure** per interval. β†’ `docs/pfi/gemma4-erp-tune-sizing.md` -- **⚠ MFU IS 8.6% AND THE CAUSE IS A PYTHON LOOP β€” measured, root-caused, parked.** 27.1 TFLOPS achieved against a **benchmarked 313.8 TFLOPS** dense BF16 peak (6.0% excluding padding). Cause is `transformers/models/gemma4/modeling_gemma4.py` ~line 1307: `for expert_idx in expert_hit:` β€” 128 experts Γ— 30 layers = 3,840 loop iterations per forward, ~11,500 per step under gradient checkpointing, each a where+gather+two small GEMMs+scatter-add in eager mode. vLLM serves this model fast because it has fused grouped-GEMM MoE kernels; HF Trainer has none. **Padding is a secondary 29.9% tax** (82,337,318 padded tokens vs 57,733,156 real; `group_by_length` would make it 1.000Γ—). ⚠ My first estimate of "5% MFU" was WRONG β€” I divided by UNPADDED tokens and compared against a GUESSED peak. Operator pushed back and was right to. **Break-even for restarting: a fix landing in T hours with speedup S wins if `T + 17/S < 16` β€” at S=2 that allows 7.5h of work, at S=4 it allows 11.75h.** β†’ park `fused-moe-kernel-path-for-gemma-4-moe-training` (id 47) -- **⚠ TWELVE HARNESS DEFECTS FOUND BEFORE THE RUN β€” the two lessons worth keeping.** (1) **INERT GATES**: `auditcore` (CSAM, rc-2 never fired in 42,662 records) and `validate_vision_keys` (compared `model.state_dict()` against itself β€” could not fail on any input). The question that catches them is not *"does this check pass?"* but **"CAN this check fail?"** (2) **INVARIANT ENFORCED ON ONE PATH BUT NOT ITS SIBLING** β€” my own INV-T9(5) bug: a window with no assistant turn was blocked on the cut path and not the fits path, so a trailing user-only remainder became a zero-loss window. Also: `enable_input_require_grads()` is MANDATORY beside gradient checkpointing on a frozen base or **every adapter stays at its init and the run completes successfully having learned nothing**; the upstream Gemma-4 template FORWARD-SCANS (suppresses the closing `` before another assistant msg) so incremental rendering cannot tile β€” merge assistant runs first, joined with `"\n\n"` (a deliberate data change fixing run-ons on 42.5% of pippa). +- **⚠ MFU IS 8.6% AND I HAVE DISPROVEN MY OWN HYPOTHESIS TWICE β€” CONSULT OUT TO THE FRONTIER DWARVES.** 27.1 TFLOPS against a **benchmarked 313.8 TFLOPS** peak; one fwd+bwd at the real shape is **34.85s** (4 passes within 1%). **RULED OUT, with numbers, not argument:** (1) **hardware** β€” a plain dense GEMM hits **97.1% of peak** (304.6 TFLOPS), card draws 279-292W of 300W; (2) **the Python expert loop** β€” swapping to transformers' `grouped_mm` experts backend gave **35.149s vs eager's 34.847s, bit-identical output (max_abs_diff EXACTLY 0.0), same 75.8 GiB**, and torch 2.13 HAS both `F.grouped_mm` and `torch._grouped_mm`, so it is not a missing kernel; `batched_mm` both OOMs and MISMATCHES (rel 0.79 β€” it computes all 128 experts per token); (3) **MoE being the bottleneck at all** β€” isolated at real shapes the MoE block is **37.54 ms at 26.5% of peak**, of which **13.39 ms is pure gather/scatter dispatch** and a dispatch-free `bmm` version would be **12.28 ms at 80.9% of peak** β€” but **30 layers Γ— 37.54 ms Γ— 3 (fwd+recompute+bwd) β‰ˆ 3.4s of a 34.85s step, only ~10%.** Making MoE free buys ~7%. **~90% of the time is somewhere I have not looked.** ⚠ **LEADING UNTESTED HYPOTHESIS: the 5 `full_attention` layers use `global_head_dim: 512`, and FlashAttention-2 caps head_dim at 256** β€” if that pushes torch SDPA onto the mem-efficient or math backend, 5 layers are doing O(nΒ²) attention at seq 16384 on a slow path. Other un-excluded candidates: the chunked CE (vocab 262,144 + softcap, 1024-tok chunks re-materialised under `checkpoint`), the `attention_k_eq_v` K=V path, grad-ckpt Γ— MoE dispatch interaction, PEFT's wrapper on 205 modules. ⚠ **My earlier "5% MFU" was ALSO wrong** (divided by UNPADDED tokens, compared against a GUESSED peak) β€” operator caught it. Padding is a real but secondary **29.9%** tax (82,337,318 padded vs 57,733,156 real). Artifacts: `/tank/erp-tune/{micro_moe,bench_moe,bench_bf16}.py`. β†’ park id 47, althing thread `01M0VKBPZD71Q302NH84BXHTWS` + - **πŸ›‘ THE CORPUS GATE β€” OVERRIDDEN FOR THIS ONE RUN ONLY (operator, 2026-08-25).** Grant staged at `/mnt/smithy/datasets/derived/_recipes/erp-seat-sft-r1/TRAINING-ELIGIBILITY-OVERRIDE.md`. ⚠ It does NOT flip any root's `training_eligible` flag β€” they still read `false` and name both blockers, deliberately, so the signal survives. **A second run needs a second grant.** Provenance records `training_eligibility_override: operator-2026-08-25-rnd-run` + both blockers + both substitute controls; those keys are in `REQUIRED_PROVENANCE` as present-with-explicit-null so a future run cannot silently omit them. Background: Every `clean-v1/CLEANROOT.json` carries `training_eligible: false` with `training_blocked_by: [contamination-scan-not-implemented, stage-2-csam-detector-inert]`, and the recipe itself says *"nothing here is Charter Β§3 training-eligible"*. ⚠ **`scoped_grant: operator-2026-08-22` is NOT training clearance** β€” it governs INV-4 one-way tier inheritance (the adapter is permanently `internal-erp-rnd`, never distributable). I initially misread the grant as authorization and told brokkr I was proceeding; **brokkr-smithy-dev β€” who WROTE those fields β€” corrected it**: *"I wrote them so that exactly this would happen… do not take my word as clearance; I do not have the authority to give it."* **The detector is measured-inert, not suspected:** `auditcore` v3.7.2 returned its hard-drop rc-2 **zero times across 42,662 raw RP records**, its printed verdict ignores its own printed threshold, and it passed a blind-audit-identified record of sexual content involving a participant the text marks as a child (`pippa-5083`, composite 4.34 vs threshold 6.5). β†’ `research/R47-premium-corpus-gate/FINDING-auditcore-inert.md`, Contract Amendment 11. **I verified the one decisive thing:** `pippa-5083` IS in `kept-manifest.jsonl` (4,551 rows) but **ABSENT from `recipe-dedup-kept.jsonl` (20,473 rows)** β€” the survivor list the harness gates on β€” so brokkr's substitute *stage-A lexical* screen caught it. That is one known instance caught by a stopgap; it says nothing about what the screen misses. **Both brokkr and I recommend STOPPING; only an explicit operator override opens it.** Neither blocker is hours of work (the 13-gram scanner is spec-only, DRAFT since 2026-06-01; the detector needs replacing). ⚠ **Do NOT stage or copy corpus content while gated.** - **🟒 SIZING + SEAT CALL β€” DONE AND EXECUTED, full detail in the doc.** QLoRA structurally unavailable (fused 3-D experts vs bitsandbytes' nn.Linear walk); plain BF16 LoRA; chunked CE mandatory (naive CE OOMs at seq16384, 81.93 GiB at seq8192); `v_proj` exists on only 25 of 30 layers (`attention_k_eq_v`, K=V sharing β€” real, not a miss). `gen` moved to GPU1, `sec` down, GPU0 dedicated. β†’ `docs/pfi/gemma4-erp-tune-sizing.md`, `playbooks/ana-ml2-training-window-{open,close}.yaml` - **⚠ TELL EITRI BEFORE HE HARD-CODES: the trainee base changed.** Contract still names the stock BF16. It is now `/tank/aimodels/gemma4-26b-a4b-it-heretic-bf16` (llmfan46). **Base path AND chat-template path must be config keys, not constants** β€” and the template must point at upstream's (`gemma4-26b-a4b-it-bf16/chat_template.jinja`), never the base's own, or training renders a different prompt than production serves.