docs(gemma4-erp-tune): whole-card placement — gen moves to GPU1, sec stands down

Operator chose a third placement over the two the sizing offered: rather
than train beside gen on GPU0 or on GPU1 in mog-sec's slot, move gen to
GPU1 and empty GPU0 completely. The tune gets 95.60 GiB with no
co-tenant and gen never goes dark beyond its own restart.

Revised run parameters, since a whole card changes them:

- micro-batch 8 (71.8 GiB of 95.60) rather than 4, grad-accum 1, giving
  888 optimizer steps instead of 444. At one epoch the step count is
  worth having, and 8 x 8192 tokens puts ~4,096 rows through each expert
  per step against ~512 at micro-batch 1 — a far healthier GEMM on
  704-wide experts.
- Gradient checkpointing stays ON. Dropping it takes ~17% off wall-clock
  but pushes activations to ~24 GiB per sequence, which forces
  micro-batch 1 and costs 8x on MoE efficiency. Wide beats shallow.
- Scriberr stays on GPU1. The previous revision suggested moving it to
  GPU0, which was correct only while training was going to live on GPU1.

Records the ordering constraint in both directions, the elway identity
requirement, and that sec's aliases should be allowed to fail at the
gateway rather than be substituted with another model.

persistent-memory.md

@@ -110,7 +110,7 @@ no longer deployed sidecars here. See Recent decisions.)
 
 _As of 2026-08-24 (late) — a very long ops session. The homepage arc and the char-rp arc both closed. **The live thread is the ERP/RP trainee: QLoRA sizing is the next conversation and brokkr-smithy-dev is waiting on it.**_
 
-- **🔴 SIZING DONE — AND IT KILLED THE QLoRA PREMISE. AWAITING THE SEAT CALL.** → `docs/pfi/gemma4-erp-tune-sizing.md` (measured, not estimated). **QLoRA is structurally unavailable on this architecture:** the checkpoint stores each layer's 128 experts as two fused 3-D `nn.Parameter` tensors (`layers.N.experts.gate_up_proj` `[128,1408,2816]`, `.down_proj` `[128,2816,704]` — note the missing `.weight` suffix), and `bitsandbytes` 4-bit replacement walks `nn.Linear` modules only. **88.5% of the model (22.84B params / 42.54 GiB) is untouchable; `load_in_4bit` saves ~3.1 GiB of 48.07 and does NOT error.** ⚠ **Eitri must not hard-code `BitsAndBytesConfig`** — it loads, reports 4-bit, and silently leaves 42.5 GiB BF16. Verdict: **plain LoRA on BF16**, ~57.6 GiB at micro-batch 1, +2.5 GiB per extra 8192-seq. ⚠ **The loss head is the real driver and was NOT in the brief:** vocab 262,144 × 8192 = 2.147B logits, plus `final_logit_softcapping 30.0` → **~28-30 GiB transient at BATCH 1** through naive HF CE (= ~85.6 GiB total, an OOM-on-first-long-sample). **Fused/chunked linear CE is mandatory and must be smoke-proven before the window is booked** (Liger may lack a Gemma-4 MoE patch). ⚠ **`v_proj` DOES NOT EXIST on layers 5/11/17/23/29** (`attention_k_eq_v` on the full-attention layers) — a `v_proj` target silently no-ops there and `k_proj` adapts K and V at once; 45.96M trainable at r64. ⚠ 7,104 packed seqs → **only 444 optimizer steps** at effective batch 16, which is why wall-clock checkpointing matters concretely. **RECOMMENDATION: run on GPU1, stand down `sec` (2 aliases, last request ~5h ago), NOT `gen` (7 aliases, 765 busy-engine lines/24h).** Stopping mog-sec frees 74.29 GiB; micro-batch 4 = 61.8 GiB. Est. **4-10h, likely ~6.** Also: move Scriberr to GPU0 for the window, and package as a `uv` venv on `/tank` — **root is 91% full (36 GB)** and `/var/lib/docker` is on it.
+- **🔴 SIZING DONE — AND IT KILLED THE QLoRA PREMISE. AWAITING THE SEAT CALL.** → `docs/pfi/gemma4-erp-tune-sizing.md` (measured, not estimated). **QLoRA is structurally unavailable on this architecture:** the checkpoint stores each layer's 128 experts as two fused 3-D `nn.Parameter` tensors (`layers.N.experts.gate_up_proj` `[128,1408,2816]`, `.down_proj` `[128,2816,704]` — note the missing `.weight` suffix), and `bitsandbytes` 4-bit replacement walks `nn.Linear` modules only. **88.5% of the model (22.84B params / 42.54 GiB) is untouchable; `load_in_4bit` saves ~3.1 GiB of 48.07 and does NOT error.** ⚠ **Eitri must not hard-code `BitsAndBytesConfig`** — it loads, reports 4-bit, and silently leaves 42.5 GiB BF16. Verdict: **plain LoRA on BF16**, ~57.6 GiB at micro-batch 1, +2.5 GiB per extra 8192-seq. ⚠ **The loss head is the real driver and was NOT in the brief:** vocab 262,144 × 8192 = 2.147B logits, plus `final_logit_softcapping 30.0` → **~28-30 GiB transient at BATCH 1** through naive HF CE (= ~85.6 GiB total, an OOM-on-first-long-sample). **Fused/chunked linear CE is mandatory and must be smoke-proven before the window is booked** (Liger may lack a Gemma-4 MoE patch). ⚠ **`v_proj` DOES NOT EXIST on layers 5/11/17/23/29** (`attention_k_eq_v` on the full-attention layers) — a `v_proj` target silently no-ops there and `k_proj` adapts K and V at once; 45.96M trainable at r64. ⚠ 7,104 packed seqs → **only 444 optimizer steps** at effective batch 16, which is why wall-clock checkpointing matters concretely. **OPERATOR CALL, EXECUTED 2026-08-24 EVENING: `gen` MOVED to GPU1, `sec` STOOD DOWN, GPU0 emptied entirely (95.60 GiB) for the tune** — strictly better than training beside gen. Both directions are playbooks: `playbooks/ana-ml2-training-window-{open,close}.yaml`. ⚠⚠ **ORDER IS LOAD-BEARING IN BOTH:** gen's `--gpu-memory-utilization 0.43` is a fraction of TOTAL card memory, so **42,091 MiB must be FREE AT STARTUP or the engine refuses to boot** — recreating gen onto GPU1 before stopping mog-sec takes the fleet's main seat down and LEAVES it down. Close mirrors it (mog-sec needs 50,901 MiB, so gen must vacate GPU1 first) and gates on GPU0 being idle so it cannot evict a running tune (`--var allow_busy_gpu0=true` to override). ⚠ **Invoke elway as `infra-ops@10.250.50.54`** — the `ana-ml2` ssh-target resolves to `lkraven`, no NOPASSWD sudo, elway aborts at its sudo probe. Revised run params on a whole card: **micro-batch 8, grad-accum 1 → 888 steps** (double the shared-card count, and ~4,096 rows/expert/step vs ~512 at mb=1, so much better MoE GEMM). ⚠ **KEEP gradient checkpointing ON** — dropping it saves ~17% wall-clock but forces mb=1, which craters MoE efficiency 8×; wide beats shallow. Scriberr STAYS on GPU1 (an earlier draft said move it to GPU0 — backwards now that GPU0 is the training card). Est. **4-10h, likely ~6.** Package as a `uv` venv on `/tank` — **root is 91% full (36 GB)** and `/var/lib/docker` is on it. ⚠ `sec` traffic arrives from the LiteLLM gateway (10.250.50.70); let the aliases FAIL, do not substitute another model.
 - **⚠ TELL EITRI BEFORE HE HARD-CODES: the trainee base changed.** Contract still names the stock BF16. It is now `/tank/aimodels/gemma4-26b-a4b-it-heretic-bf16` (llmfan46). **Base path AND chat-template path must be config keys, not constants** — and the template must point at upstream's (`gemma4-26b-a4b-it-bf16/chat_template.jinja`), never the base's own, or training renders a different prompt than production serves.
 - **🟢 char-rp seat = Gemma-4 26B-A4B MoE NVFP4** on `:8016`, both aliases on ONE backend. **Currently DOWN by operator instruction** to hold GPU0 headroom for the tune. `gen` is UP and verified. MeroMero-v2 retained stopped in `created` state for rollback (stop-then-start; both bind :8016). → `persistent-memory.d/2026-08-24-charrp-gemma4-moe-swap-and-trainee.md`
 - **🟢 THREE trainee-relevant model dirs on `/tank/aimodels/`, NOT interchangeable:** `gemma4-26b-a4b-it-bf16` (stock, 49 GB — its chat_template is the canonical upstream one), `gemma4-26b-a4b-it-heretic-bf16` (llmfan46 abliterated, the trainee), `gemma4-26b-a4b-it-abliterated-bf16` (TrevorJS, KL 0.09, alternate). Plus `-nvfp4` (served) and `-nvfp4a16` (activation control). ⚠ **BF16 cannot coexist with `gen`** — 48.07 GiB of weights on a 94.97 GiB card. Every BF16 window means gen stops.