memory: the ERP tune is running — override granted, 12 defects fixed first

Operator overrode the corpus gate for one run on 2026-08-25, with the
grant staged beside the recipe rather than asserted in chat. It
deliberately does not flip any root's training_eligible flag, so the
signal that made the run stop in the first place survives intact.

Records where the run lives, what it is configured with, how to restore
the fleet, and the two lessons that generalise past this project.

The first is inert gates. Two turned up in one evening — auditcore,
whose CSAM hard-drop never fired across 42,662 records, and
validate_vision_keys, which compared model.state_dict() against itself
and could not fail on any input. Both read as guards. The question that
catches them is not whether the check passes but whether it can fail.

The second is an invariant enforced on one code path and not its
sibling. That was my own bug: INV-T9 requires a window to hold at least
one complete assistant turn, and I enforced it where the window is cut
but not where it fits, so a trailing user-only remainder became a
zero-loss window and killed the first launch. Same shape as the inert
gates, in code I wrote an hour earlier.

Two further foot-guns worth the space: enable_input_require_grads is
mandatory beside gradient checkpointing on a frozen base, or every
adapter stays at its initialisation and the run completes successfully
having learned nothing; and the upstream Gemma-4 template forward-scans
to suppress a closing turn marker before another assistant message, so
incremental rendering cannot tile against it and assistant runs must be
merged first.

persistent-memory.md

@@ -110,7 +110,9 @@ no longer deployed sidecars here. See Recent decisions.)
 
 _As of 2026-08-24 (late) — a very long ops session. The homepage arc and the char-rp arc both closed. **The live thread is the ERP/RP trainee: QLoRA sizing is the next conversation and brokkr-smithy-dev is waiting on it.**_
 
-- **🛑 THE ERP TUNE IS BLOCKED ON A CORPUS GATE ONLY THE OPERATOR CAN CLEAR (2026-08-24 night).** Every `clean-v1/CLEANROOT.json` carries `training_eligible: false` with `training_blocked_by: [contamination-scan-not-implemented, stage-2-csam-detector-inert]`, and the recipe itself says *"nothing here is Charter §3 training-eligible"*. ⚠ **`scoped_grant: operator-2026-08-22` is NOT training clearance** — it governs INV-4 one-way tier inheritance (the adapter is permanently `internal-erp-rnd`, never distributable). I initially misread the grant as authorization and told brokkr I was proceeding; **brokkr-smithy-dev — who WROTE those fields — corrected it**: *"I wrote them so that exactly this would happen… do not take my word as clearance; I do not have the authority to give it."* **The detector is measured-inert, not suspected:** `auditcore` v3.7.2 returned its hard-drop rc-2 **zero times across 42,662 raw RP records**, its printed verdict ignores its own printed threshold, and it passed a blind-audit-identified record of sexual content involving a participant the text marks as a child (`pippa-5083`, composite 4.34 vs threshold 6.5). → `research/R47-premium-corpus-gate/FINDING-auditcore-inert.md`, Contract Amendment 11. **I verified the one decisive thing:** `pippa-5083` IS in `kept-manifest.jsonl` (4,551 rows) but **ABSENT from `recipe-dedup-kept.jsonl` (20,473 rows)** — the survivor list the harness gates on — so brokkr's substitute *stage-A lexical* screen caught it. That is one known instance caught by a stopgap; it says nothing about what the screen misses. **Both brokkr and I recommend STOPPING; only an explicit operator override opens it.** Neither blocker is hours of work (the 13-gram scanner is spec-only, DRAFT since 2026-06-01; the detector needs replacing). ⚠ **Do NOT stage or copy corpus content while gated.**
+- **🟢 THE ERP TUNE IS RUNNING (launched 2026-08-24 ~20:40 PDT, ETA ~13h → ~09:40 PDT 08-25).** GPU0 on ana-ml2, dedicated. `gen` relocated to GPU1 and healthy; **`sec`/mog-sec STOPPED for the whole run, operator-ruled ("let it run, keep sec down")**. Restore = `playbooks/ana-ml2-training-window-close.yaml` (gates on GPU0 idle; `--var allow_busy_gpu0=true` to override). Harness **eitri-smithy `997c4a4`** at `/tank/erp-tune/eitri-smithy`, venv `/tank/erp-tune/venv` (torch 2.13.0+cu130, transformers 5.15.1, peft 0.20.0, sm_120 verified), config `/tank/erp-tune/run-01.json`, log `/tank/erp-tune/run-01.log`, output `/tank/erp-tune/run-01/`. **Config: BF16 (NOT QLoRA), max_seq_len 16384, mb2×accum8 → 1,312 steps, r64/α128, 205 modules, 74,342,400 trainable.** Step-10 loss **3.664**, grad_norm 5.178 — ⚠ above brokkr's 1.8–3.0 band but the doubled-divisor signature was ~0.25, so `num_items_in_batch` is NOT double-applied; hypothesis = the mix is 52.9% literary prose where every token is a loss target. GPU0 runs **84,222 MiB of 97,887** (above my measured 79.71 GiB worst case — adjacent `#w0`/`#w1` windows share micro-batches systematically, exactly as brokkr predicted). **Encode is CACHED** (`run-01/encode-cache/`, keyed on encode_version+max_seq_len+template sha) so a restart costs ~2.5 min, not the 4.3h it would single-threaded. ⚠ **encode_version must be BUMPED on ANY encoder change** — that has mattered five times. → `docs/pfi/gemma4-erp-tune-sizing.md`
+- **⚠ TWELVE HARNESS DEFECTS FOUND BEFORE THE RUN — the two lessons worth keeping.** (1) **INERT GATES**: `auditcore` (CSAM, rc-2 never fired in 42,662 records) and `validate_vision_keys` (compared `model.state_dict()` against itself — could not fail on any input). The question that catches them is not *"does this check pass?"* but **"CAN this check fail?"** (2) **INVARIANT ENFORCED ON ONE PATH BUT NOT ITS SIBLING** — my own INV-T9(5) bug: a window with no assistant turn was blocked on the cut path and not the fits path, so a trailing user-only remainder became a zero-loss window. Also: `enable_input_require_grads()` is MANDATORY beside gradient checkpointing on a frozen base or **every adapter stays at its init and the run completes successfully having learned nothing**; the upstream Gemma-4 template FORWARD-SCANS (suppresses the closing `<turn|>` before another assistant msg) so incremental rendering cannot tile — merge assistant runs first, joined with `"\n\n"` (a deliberate data change fixing run-ons on 42.5% of pippa).
+- **🛑 THE CORPUS GATE — OVERRIDDEN FOR THIS ONE RUN ONLY (operator, 2026-08-25).** Grant staged at `/mnt/smithy/datasets/derived/_recipes/erp-seat-sft-r1/TRAINING-ELIGIBILITY-OVERRIDE.md`. ⚠ It does NOT flip any root's `training_eligible` flag — they still read `false` and name both blockers, deliberately, so the signal survives. **A second run needs a second grant.** Provenance records `training_eligibility_override: operator-2026-08-25-rnd-run` + both blockers + both substitute controls; those keys are in `REQUIRED_PROVENANCE` as present-with-explicit-null so a future run cannot silently omit them. Background: Every `clean-v1/CLEANROOT.json` carries `training_eligible: false` with `training_blocked_by: [contamination-scan-not-implemented, stage-2-csam-detector-inert]`, and the recipe itself says *"nothing here is Charter §3 training-eligible"*. ⚠ **`scoped_grant: operator-2026-08-22` is NOT training clearance** — it governs INV-4 one-way tier inheritance (the adapter is permanently `internal-erp-rnd`, never distributable). I initially misread the grant as authorization and told brokkr I was proceeding; **brokkr-smithy-dev — who WROTE those fields — corrected it**: *"I wrote them so that exactly this would happen… do not take my word as clearance; I do not have the authority to give it."* **The detector is measured-inert, not suspected:** `auditcore` v3.7.2 returned its hard-drop rc-2 **zero times across 42,662 raw RP records**, its printed verdict ignores its own printed threshold, and it passed a blind-audit-identified record of sexual content involving a participant the text marks as a child (`pippa-5083`, composite 4.34 vs threshold 6.5). → `research/R47-premium-corpus-gate/FINDING-auditcore-inert.md`, Contract Amendment 11. **I verified the one decisive thing:** `pippa-5083` IS in `kept-manifest.jsonl` (4,551 rows) but **ABSENT from `recipe-dedup-kept.jsonl` (20,473 rows)** — the survivor list the harness gates on — so brokkr's substitute *stage-A lexical* screen caught it. That is one known instance caught by a stopgap; it says nothing about what the screen misses. **Both brokkr and I recommend STOPPING; only an explicit operator override opens it.** Neither blocker is hours of work (the 13-gram scanner is spec-only, DRAFT since 2026-06-01; the detector needs replacing). ⚠ **Do NOT stage or copy corpus content while gated.**
 - **🔴 SIZING DONE — AND IT KILLED THE QLoRA PREMISE. AWAITING THE SEAT CALL.** → `docs/pfi/gemma4-erp-tune-sizing.md` (measured, not estimated). **QLoRA is structurally unavailable on this architecture:** the checkpoint stores each layer's 128 experts as two fused 3-D `nn.Parameter` tensors (`layers.N.experts.gate_up_proj` `[128,1408,2816]`, `.down_proj` `[128,2816,704]` — note the missing `.weight` suffix), and `bitsandbytes` 4-bit replacement walks `nn.Linear` modules only. **88.5% of the model (22.84B params / 42.54 GiB) is untouchable; `load_in_4bit` saves ~3.1 GiB of 48.07 and does NOT error.** ⚠ **Eitri must not hard-code `BitsAndBytesConfig`** — it loads, reports 4-bit, and silently leaves 42.5 GiB BF16. Verdict: **plain LoRA on BF16**, ~57.6 GiB at micro-batch 1, +2.5 GiB per extra 8192-seq. ⚠ **The loss head is the real driver and was NOT in the brief:** vocab 262,144 × 8192 = 2.147B logits, plus `final_logit_softcapping 30.0` → **~28-30 GiB transient at BATCH 1** through naive HF CE (= ~85.6 GiB total, an OOM-on-first-long-sample). **Fused/chunked linear CE is mandatory and must be smoke-proven before the window is booked** (Liger may lack a Gemma-4 MoE patch). ⚠ **`v_proj` DOES NOT EXIST on layers 5/11/17/23/29** (`attention_k_eq_v` on the full-attention layers) — a `v_proj` target silently no-ops there and `k_proj` adapts K and V at once; 45.96M trainable at r64. ⚠ 7,104 packed seqs → **only 444 optimizer steps** at effective batch 16, which is why wall-clock checkpointing matters concretely. **OPERATOR CALL, EXECUTED 2026-08-24 EVENING: `gen` MOVED to GPU1, `sec` STOOD DOWN, GPU0 emptied entirely (95.60 GiB) for the tune** — strictly better than training beside gen. Both directions are playbooks: `playbooks/ana-ml2-training-window-{open,close}.yaml`. ⚠⚠ **ORDER IS LOAD-BEARING IN BOTH:** gen's `--gpu-memory-utilization 0.43` is a fraction of TOTAL card memory, so **42,091 MiB must be FREE AT STARTUP or the engine refuses to boot** — recreating gen onto GPU1 before stopping mog-sec takes the fleet's main seat down and LEAVES it down. Close mirrors it (mog-sec needs 50,901 MiB, so gen must vacate GPU1 first) and gates on GPU0 being idle so it cannot evict a running tune (`--var allow_busy_gpu0=true` to override). ⚠ **Invoke elway as `infra-ops@10.250.50.54`** — the `ana-ml2` ssh-target resolves to `lkraven`, no NOPASSWD sudo, elway aborts at its sudo probe. Revised run params on a whole card: **micro-batch 8, grad-accum 1 → 888 steps** (double the shared-card count, and ~4,096 rows/expert/step vs ~512 at mb=1, so much better MoE GEMM). ⚠ **KEEP gradient checkpointing ON** — dropping it saves ~17% wall-clock but forces mb=1, which craters MoE efficiency 8×; wide beats shallow. Scriberr STAYS on GPU1 (an earlier draft said move it to GPU0 — backwards now that GPU0 is the training card). Est. **4-10h, likely ~6.** Package as a `uv` venv on `/tank` — **root is 91% full (36 GB)** and `/var/lib/docker` is on it. ⚠ `sec` traffic arrives from the LiteLLM gateway (10.250.50.70); let the aliases FAIL, do not substitute another model.
 - **⚠ TELL EITRI BEFORE HE HARD-CODES: the trainee base changed.** Contract still names the stock BF16. It is now `/tank/aimodels/gemma4-26b-a4b-it-heretic-bf16` (llmfan46). **Base path AND chat-template path must be config keys, not constants** — and the template must point at upstream's (`gemma4-26b-a4b-it-bf16/chat_template.jinja`), never the base's own, or training renders a different prompt than production serves.
 - **🟢 char-rp seat = Gemma-4 26B-A4B MoE NVFP4** on `:8016`, both aliases on ONE backend. **Currently DOWN by operator instruction** to hold GPU0 headroom for the tune. `gen` is UP and verified. MeroMero-v2 retained stopped in `created` state for rollback (stop-then-start; both bind :8016). → `persistent-memory.d/2026-08-24-charrp-gemma4-moe-swap-and-trainee.md`