diff --git a/persistent-memory.md b/persistent-memory.md index 43365f7..e335e3b 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -110,6 +110,7 @@ no longer deployed sidecars here. See Recent decisions.) _As of 2026-08-24 (late) β€” a very long ops session. The homepage arc and the char-rp arc both closed. **The live thread is the ERP/RP trainee: QLoRA sizing is the next conversation and brokkr-smithy-dev is waiting on it.**_ +- **πŸ›‘ THE ERP TUNE IS BLOCKED ON A CORPUS GATE ONLY THE OPERATOR CAN CLEAR (2026-08-24 night).** Every `clean-v1/CLEANROOT.json` carries `training_eligible: false` with `training_blocked_by: [contamination-scan-not-implemented, stage-2-csam-detector-inert]`, and the recipe itself says *"nothing here is Charter Β§3 training-eligible"*. ⚠ **`scoped_grant: operator-2026-08-22` is NOT training clearance** β€” it governs INV-4 one-way tier inheritance (the adapter is permanently `internal-erp-rnd`, never distributable). I initially misread the grant as authorization and told brokkr I was proceeding; **brokkr-smithy-dev β€” who WROTE those fields β€” corrected it**: *"I wrote them so that exactly this would happen… do not take my word as clearance; I do not have the authority to give it."* **The detector is measured-inert, not suspected:** `auditcore` v3.7.2 returned its hard-drop rc-2 **zero times across 42,662 raw RP records**, its printed verdict ignores its own printed threshold, and it passed a blind-audit-identified record of sexual content involving a participant the text marks as a child (`pippa-5083`, composite 4.34 vs threshold 6.5). β†’ `research/R47-premium-corpus-gate/FINDING-auditcore-inert.md`, Contract Amendment 11. **I verified the one decisive thing:** `pippa-5083` IS in `kept-manifest.jsonl` (4,551 rows) but **ABSENT from `recipe-dedup-kept.jsonl` (20,473 rows)** β€” the survivor list the harness gates on β€” so brokkr's substitute *stage-A lexical* screen caught it. That is one known instance caught by a stopgap; it says nothing about what the screen misses. **Both brokkr and I recommend STOPPING; only an explicit operator override opens it.** Neither blocker is hours of work (the 13-gram scanner is spec-only, DRAFT since 2026-06-01; the detector needs replacing). ⚠ **Do NOT stage or copy corpus content while gated.** - **πŸ”΄ SIZING DONE β€” AND IT KILLED THE QLoRA PREMISE. AWAITING THE SEAT CALL.** β†’ `docs/pfi/gemma4-erp-tune-sizing.md` (measured, not estimated). **QLoRA is structurally unavailable on this architecture:** the checkpoint stores each layer's 128 experts as two fused 3-D `nn.Parameter` tensors (`layers.N.experts.gate_up_proj` `[128,1408,2816]`, `.down_proj` `[128,2816,704]` β€” note the missing `.weight` suffix), and `bitsandbytes` 4-bit replacement walks `nn.Linear` modules only. **88.5% of the model (22.84B params / 42.54 GiB) is untouchable; `load_in_4bit` saves ~3.1 GiB of 48.07 and does NOT error.** ⚠ **Eitri must not hard-code `BitsAndBytesConfig`** β€” it loads, reports 4-bit, and silently leaves 42.5 GiB BF16. Verdict: **plain LoRA on BF16**, ~57.6 GiB at micro-batch 1, +2.5 GiB per extra 8192-seq. ⚠ **The loss head is the real driver and was NOT in the brief:** vocab 262,144 Γ— 8192 = 2.147B logits, plus `final_logit_softcapping 30.0` β†’ **~28-30 GiB transient at BATCH 1** through naive HF CE (= ~85.6 GiB total, an OOM-on-first-long-sample). **Fused/chunked linear CE is mandatory and must be smoke-proven before the window is booked** (Liger may lack a Gemma-4 MoE patch). ⚠ **`v_proj` DOES NOT EXIST on layers 5/11/17/23/29** (`attention_k_eq_v` on the full-attention layers) β€” a `v_proj` target silently no-ops there and `k_proj` adapts K and V at once; 45.96M trainable at r64. ⚠ 7,104 packed seqs β†’ **only 444 optimizer steps** at effective batch 16, which is why wall-clock checkpointing matters concretely. **OPERATOR CALL, EXECUTED 2026-08-24 EVENING: `gen` MOVED to GPU1, `sec` STOOD DOWN, GPU0 emptied entirely (95.60 GiB) for the tune** β€” strictly better than training beside gen. Both directions are playbooks: `playbooks/ana-ml2-training-window-{open,close}.yaml`. ⚠⚠ **ORDER IS LOAD-BEARING IN BOTH:** gen's `--gpu-memory-utilization 0.43` is a fraction of TOTAL card memory, so **42,091 MiB must be FREE AT STARTUP or the engine refuses to boot** β€” recreating gen onto GPU1 before stopping mog-sec takes the fleet's main seat down and LEAVES it down. Close mirrors it (mog-sec needs 50,901 MiB, so gen must vacate GPU1 first) and gates on GPU0 being idle so it cannot evict a running tune (`--var allow_busy_gpu0=true` to override). ⚠ **Invoke elway as `infra-ops@10.250.50.54`** β€” the `ana-ml2` ssh-target resolves to `lkraven`, no NOPASSWD sudo, elway aborts at its sudo probe. Revised run params on a whole card: **micro-batch 8, grad-accum 1 β†’ 888 steps** (double the shared-card count, and ~4,096 rows/expert/step vs ~512 at mb=1, so much better MoE GEMM). ⚠ **KEEP gradient checkpointing ON** β€” dropping it saves ~17% wall-clock but forces mb=1, which craters MoE efficiency 8Γ—; wide beats shallow. Scriberr STAYS on GPU1 (an earlier draft said move it to GPU0 β€” backwards now that GPU0 is the training card). Est. **4-10h, likely ~6.** Package as a `uv` venv on `/tank` β€” **root is 91% full (36 GB)** and `/var/lib/docker` is on it. ⚠ `sec` traffic arrives from the LiteLLM gateway (10.250.50.70); let the aliases FAIL, do not substitute another model. - **⚠ TELL EITRI BEFORE HE HARD-CODES: the trainee base changed.** Contract still names the stock BF16. It is now `/tank/aimodels/gemma4-26b-a4b-it-heretic-bf16` (llmfan46). **Base path AND chat-template path must be config keys, not constants** β€” and the template must point at upstream's (`gemma4-26b-a4b-it-bf16/chat_template.jinja`), never the base's own, or training renders a different prompt than production serves. - **🟒 char-rp seat = Gemma-4 26B-A4B MoE NVFP4** on `:8016`, both aliases on ONE backend. **Currently DOWN by operator instruction** to hold GPU0 headroom for the tune. `gen` is UP and verified. MeroMero-v2 retained stopped in `created` state for rollback (stop-then-start; both bind :8016). β†’ `persistent-memory.d/2026-08-24-charrp-gemma4-moe-swap-and-trainee.md`