memory: snapshot — the tune is trained, gated, and serving

Run-01 completed in 7:21:52 (47% faster than the 13.85h round-1 projection),
lora_B gate 205/205 non-zero at median norm 1.708, and the acceptance gate says
it did the thing it was built for: diversity +0.178 against a 0.008 floor (22x),
attractor hit rate -11.3pt against a 2.0pt floor, memorisation 0.0000 on both
arms — which closes the R20 licensed-prose exposure on measurement rather than
argument.

Five new detail files carry the substance:

  erp-tune-run2-complete        the run, the gate, the noise-floor near-miss
                                (brokkr was one step from reporting a 13-point
                                T6 regression sitting inside twice his
                                instrument's own variance)
  mfu-root-caused-attention     8.6% MFU was an accounting artifact; real
                                utilisation 17-20%, cost was attention on
                                AMPERE kernels. Two independent methods agreed
                                to 2.6 points.
  nvfp4-serving-pipeline        merged weights are MANDATORY — vLLM cannot
                                serve a LoRA on ANY Gemma-4 — plus the recipe
                                that silently misses all 11,520 expert tensors
  refusal-retention-probe       measured base 0/100 -> tuned 29/100, then had
                                to accept it was the wrong axis
  worldtree-b188-b189-and-selene  three arcs closed, and a #411 diagnosis I got
                                wrong twice before a directory probe settled it

Current state rewritten end to end — the previous snapshot had the run in
flight at ~17h with MFU unexplained. Both are now closed.

The open operator decision is run 2's base, deliberately unstaged and flagged
against being filed as a config knob: it is a reversal of the trainee-selection
decision, and the pretrained-base option removes the last non-lexical floor on
the CSAM axis given stage-2-detector-inert and contamination-scan-absent are
both already overridden.

Tried-and-abandoned gains four measured-dead throughput levers, the packing
correction (bucketing wins under sdpa and the conclusion flips under flex — do
not carry it past the backend decision), and the merge-back-undoes-abliteration
trap brokkr caught in his own advice.

Index stays at 291 lines, under the soft cap. No archival this run.

persistent-memory.md

@@ -108,31 +108,28 @@ no longer deployed sidecars here. See Recent decisions.)
   (no NOPASSWD)** — stage model pulls to `/home`, not root-owned `/worktank`.
 ## Current state / in-flight
 
-_As of 2026-08-25 ~04:20Z — the ERP/RP tune is TRAINING on ana-ml2 GPU0, ~17h, unattended. The homepage and char-rp arcs closed earlier. **The live thread is the run itself plus a parallel question: whether a fused MoE kernel lands fast enough to justify restarting it.**_
+_As of 2026-08-25 ~21:00Z — **the ERP/RP tune is DONE and SERVING.** The live threads are brokkr's remaining gate work and an operator decision on run 2's base._
 
-- **🟢 THE ERP TUNE IS RUNNING (launched 2026-08-24 ~20:40 PDT, ETA ~13h → ~09:40 PDT 08-25).** GPU0 on ana-ml2, dedicated. `gen` relocated to GPU1 and healthy; **`sec`/mog-sec STOPPED for the whole run, operator-ruled ("let it run, keep sec down")**. Restore = `playbooks/ana-ml2-training-window-close.yaml` (gates on GPU0 idle; `--var allow_busy_gpu0=true` to override). Harness **eitri-smithy `997c4a4`** at `/tank/erp-tune/eitri-smithy`, venv `/tank/erp-tune/venv` (torch 2.13.0+cu130, transformers 5.15.1, peft 0.20.0, sm_120 verified), config `/tank/erp-tune/run-01.json`, log `/tank/erp-tune/run-01.log`, output `/tank/erp-tune/run-01/`. **Config: BF16 (NOT QLoRA), max_seq_len 16384, mb2×accum8 → 1,312 steps, r64/α128, 205 modules, 74,342,400 trainable.** Step-10 loss **3.664**, grad_norm 5.178 — ⚠ above brokkr's 1.8–3.0 band but the doubled-divisor signature was ~0.25, so `num_items_in_batch` is NOT double-applied; hypothesis = the mix is 52.9% literary prose where every token is a loss target. GPU0 runs **84,222 MiB of 97,887** (above my measured 79.71 GiB worst case — adjacent `#w0`/`#w1` windows share micro-batches systematically, exactly as brokkr predicted). **Encode is CACHED** (`run-01/encode-cache/`, keyed on encode_version+max_seq_len+template sha) so a restart costs ~2.5 min, not the 4.3h it would single-threaded. ⚠ **encode_version must be BUMPED on ANY encoder change** — that has mattered five times. **RESUME: use `/tank/erp-tune/resume-run-01.sh`, NEVER the original launch command** — that one starts `rm -rf /tank/erp-tune/run-01`, which destroys the 609 MB encode cache AND every checkpoint. First checkpoint at step 100; `save_steps=100` at ~46.5 s/it = **~73 min of crash exposure** per interval. → `docs/pfi/gemma4-erp-tune-sizing.md`
-- **⚠ MFU IS 8.6% AND I HAVE DISPROVEN MY OWN HYPOTHESIS TWICE — CONSULT OUT TO THE FRONTIER DWARVES.** 27.1 TFLOPS against a **benchmarked 313.8 TFLOPS** peak; one fwd+bwd at the real shape is **34.85s** (4 passes within 1%). **RULED OUT, with numbers, not argument:** (1) **hardware** — a plain dense GEMM hits **97.1% of peak** (304.6 TFLOPS), card draws 279-292W of 300W; (2) **the Python expert loop** — swapping to transformers' `grouped_mm` experts backend gave **35.149s vs eager's 34.847s, bit-identical output (max_abs_diff EXACTLY 0.0), same 75.8 GiB**, and torch 2.13 HAS both `F.grouped_mm` and `torch._grouped_mm`, so it is not a missing kernel; `batched_mm` both OOMs and MISMATCHES (rel 0.79 — it computes all 128 experts per token); (3) **MoE being the bottleneck at all** — isolated at real shapes the MoE block is **37.54 ms at 26.5% of peak**, of which **13.39 ms is pure gather/scatter dispatch** and a dispatch-free `bmm` version would be **12.28 ms at 80.9% of peak** — but **30 layers × 37.54 ms × 3 (fwd+recompute+bwd) ≈ 3.4s of a 34.85s step, only ~10%.** Making MoE free buys ~7%. **~90% of the time is somewhere I have not looked.** ⚠ **LEADING UNTESTED HYPOTHESIS: the 5 `full_attention` layers use `global_head_dim: 512`, and FlashAttention-2 caps head_dim at 256** — if that pushes torch SDPA onto the mem-efficient or math backend, 5 layers are doing O(n²) attention at seq 16384 on a slow path. Other un-excluded candidates: the chunked CE (vocab 262,144 + softcap, 1024-tok chunks re-materialised under `checkpoint`), the `attention_k_eq_v` K=V path, grad-ckpt × MoE dispatch interaction, PEFT's wrapper on 205 modules. ⚠ **My earlier "5% MFU" was ALSO wrong** (divided by UNPADDED tokens, compared against a GUESSED peak) — operator caught it. Padding is a real but secondary **29.9%** tax (82,337,318 padded vs 57,733,156 real). Artifacts: `/tank/erp-tune/{micro_moe,bench_moe,bench_bf16}.py`. → park id 47, althing thread `01M0VKBPZD71Q302NH84BXHTWS`
-
-- **🛑 THE CORPUS GATE — OVERRIDDEN FOR THIS ONE RUN ONLY (operator, 2026-08-25).** Grant staged at `/mnt/smithy/datasets/derived/_recipes/erp-seat-sft-r1/TRAINING-ELIGIBILITY-OVERRIDE.md`. ⚠ It does NOT flip any root's `training_eligible` flag — they still read `false` and name both blockers, deliberately, so the signal survives. **A second run needs a second grant.** Provenance records `training_eligibility_override: operator-2026-08-25-rnd-run` + both blockers + both substitute controls; those keys are in `REQUIRED_PROVENANCE` as present-with-explicit-null so a future run cannot silently omit them. Background: Every `clean-v1/CLEANROOT.json` carries `training_eligible: false` with `training_blocked_by: [contamination-scan-not-implemented, stage-2-csam-detector-inert]`, and the recipe itself says *"nothing here is Charter §3 training-eligible"*. ⚠ **`scoped_grant: operator-2026-08-22` is NOT training clearance** — it governs INV-4 one-way tier inheritance (the adapter is permanently `internal-erp-rnd`, never distributable). I initially misread the grant as authorization and told brokkr I was proceeding; **brokkr-smithy-dev — who WROTE those fields — corrected it**: *"I wrote them so that exactly this would happen… do not take my word as clearance; I do not have the authority to give it."* **The detector is measured-inert, not suspected:** `auditcore` v3.7.2 returned its hard-drop rc-2 **zero times across 42,662 raw RP records**, its printed verdict ignores its own printed threshold, and it passed a blind-audit-identified record of sexual content involving a participant the text marks as a child (`pippa-5083`, composite 4.34 vs threshold 6.5). → `research/R47-premium-corpus-gate/FINDING-auditcore-inert.md`, Contract Amendment 11. **I verified the one decisive thing:** `pippa-5083` IS in `kept-manifest.jsonl` (4,551 rows) but **ABSENT from `recipe-dedup-kept.jsonl` (20,473 rows)** — the survivor list the harness gates on — so brokkr's substitute *stage-A lexical* screen caught it. That is one known instance caught by a stopgap; it says nothing about what the screen misses. **Both brokkr and I recommend STOPPING; only an explicit operator override opens it.** Neither blocker is hours of work (the 13-gram scanner is spec-only, DRAFT since 2026-06-01; the detector needs replacing). ⚠ **Do NOT stage or copy corpus content while gated.**
-- **🟢 SIZING + SEAT CALL — DONE AND EXECUTED, full detail in the doc.** QLoRA structurally unavailable (fused 3-D experts vs bitsandbytes' nn.Linear walk); plain BF16 LoRA; chunked CE mandatory (naive CE OOMs at seq16384, 81.93 GiB at seq8192); `v_proj` exists on only 25 of 30 layers (`attention_k_eq_v`, K=V sharing — real, not a miss). `gen` moved to GPU1, `sec` down, GPU0 dedicated. → `docs/pfi/gemma4-erp-tune-sizing.md`, `playbooks/ana-ml2-training-window-{open,close}.yaml`
-- **⚠ TELL EITRI BEFORE HE HARD-CODES: the trainee base changed.** Contract still names the stock BF16. It is now `/tank/aimodels/gemma4-26b-a4b-it-heretic-bf16` (llmfan46). **Base path AND chat-template path must be config keys, not constants** — and the template must point at upstream's (`gemma4-26b-a4b-it-bf16/chat_template.jinja`), never the base's own, or training renders a different prompt than production serves.
-- **🟢 char-rp seat = Gemma-4 26B-A4B MoE NVFP4** on `:8016`, both aliases on ONE backend. **Currently DOWN by operator instruction** to hold GPU0 headroom for the tune. `gen` is UP and verified. MeroMero-v2 retained stopped in `created` state for rollback (stop-then-start; both bind :8016). → `persistent-memory.d/2026-08-24-charrp-gemma4-moe-swap-and-trainee.md`
-- **🟢 THREE trainee-relevant model dirs on `/tank/aimodels/`, NOT interchangeable:** `gemma4-26b-a4b-it-bf16` (stock, 49 GB — its chat_template is the canonical upstream one), `gemma4-26b-a4b-it-heretic-bf16` (llmfan46 abliterated, the trainee), `gemma4-26b-a4b-it-abliterated-bf16` (TrevorJS, KL 0.09, alternate). Plus `-nvfp4` (served) and `-nvfp4a16` (activation control). ⚠ **BF16 cannot coexist with `gen`** — 48.07 GiB of weights on a 94.97 GiB card. Every BF16 window means gen stops.
-- **🟢 `stacks/gemma4-trainee-bench/`** is the ephemeral BF16 bench stack — no `--quantization` flag (the production compose hardcodes `compressed-tensors` and crash-loops on BF16), `restart: "no"`, no homepage labels. Base swap is ~5 minutes because it exists.
-- **🎨 Homepage runs AUSTRALIS SKYFALL with a working light/dark toggle**, recategorised on "do I open this?" (TOOLS expanded / ENDPOINTS collapsed). ⚠ **`theme:` MUST stay pinned in settings.yaml** — removing it makes the page loader throw and serve `initialSettings: {}`, the first *confirmed* trigger for the "tab bar goes missing" mystery. → `persistent-memory.d/2026-08-24-homepage-uniform-grid.md`
-- **🔒 ana-gw's public admin surface is ZERO open TCP ports**; box scheduled for replacement by **OPNsense on a Dell R420** (brings WireGuard onto the edge — the downstream-WireGuard-VM design is moot, do not scope it). **No out-of-band path remains** — if both tunnels drop it is console-only. → `persistent-memory.d/2026-08-24-ana-gw-admin-closed-acme-disabled.md`
-- **🟢 Both Anaheim IPsec tunnels run AES-128.** NH3 245→**270 Mbit/s**, ESH 268→**304**. Ceiling is **the UDM's software AES-CBC, not the FortiGate**. → `persistent-memory.d/2026-08-23-anaheim-ipsec-tunnel-ceiling.md`
-- **🟢 Scriberr LIVE** — ana-ml2 **GPU1** :8080, built locally, uid **10001**, needs `UV_LINK_MODE=copy`. → `persistent-memory.d/2026-08-24-scriberr-ana-ml2.md`
-- **🟢 ESH DNS fixed at the IPv6 layer**; RDNSS **redirected** to AdGuard. ⚠ Proxmox bridges need `accept_ra=2`. Naming scheme lives in `docs/pfi/ipv6-naming-scheme.md` — **a convention, not memory state; never let a memory line be the only copy again.** → `persistent-memory.d/2026-08-24-esh-dns-rdnss-and-scheme-live.md`
-- **🟢 SEAT MAP.** ⚠ **ana-ml2 runs a vLLM VERSION SPREAD, not one version** — do not say "ana-ml2 runs X". Measured 2026-08-24: `gen` **0.27.2rc1.dev150** (`nightly-311b3513`), `mog-sec` **0.26.1rc1.dev1102** (`nightly-e9d1398d`), `rerank-a3`/`coder`/`reward`/`embed` **0.24.0**, char-rp + trainee-bench pinned **v0.26.0**. `v0.27.1` (tagged) and three nightlies sit on disk unused. **`gen`** = Qwen3.8-27B-Uncensored NVFP4-mixed, GPU0 :8015, 7 aliases, UP. **`char-rp`** = Gemma-4 MoE NVFP4, GPU0 :8016, DOWN deliberately. **`sec`/`sec-reasoning`** = M.O.G.-SEC, GPU1 :8019, sharing GPU1 with Scriberr.
-- **⚠️ THE `sec` DEGENERATION QUESTION IS STILL OPEN AND CONFOUNDED.** Isolating experiment is **MTP k=3 on `e9d1398d`** — still not run. Operator ruling: degeneration lives in the **un-fixed vLLM**, not the weights; MTP-head hypothesis **retracted**. Both sightings n=1.
-- **🟢 ana-ml2 mounts `/mnt/smithy`** ro + soft, **NOT in fstab** — manual remount after reboot. `nconnect=8` approved but deliberately not applied. → `persistent-memory.d/2026-08-23-smithy-mount-ana-ml2.md`
-- **🟢 ESH IS DUAL-STACK**; v4 static is an unprovisioned Cityside ticket. **NH3 stays v6-off by explicit ruling.**
-- **⏳ OPEN ELSEWHERE:** MTP-k3 isolating experiment; upstream vLLM issue to file; Cold-Fusion NVFP4 quants (44 GB) delete/keep; OWUI image-tag drift; `/tank` DEGRADED **70+ days**; Worldtree **#411** debug-room litter; Lobe retirement is the operator's call; brokkr's `gen` vs trained-reward-model bake-off. **Commits are local and unpushed** — push is the operator's call.
-- **⚠️ STANDING: NO FLEET NOTIFICATIONS unless the operator asks** (2026-08-24). Direct task correspondence with a counterparty is fine; unsolicited broadcasts are not.
+- **🟢 THE TUNE IS TRAINED, MERGED, AND SERVING.** 1312/1312 in **7:21:52** (47% faster than round 1's 13.85h projection), train_loss 2.793, `lora_B` gate **205/205 non-zero** on the final adapter. Serving bf16 as **`erp-tune-v1`** on ana-ml2 GPU0 `:8097`, routed through LiteLLM at `10.250.50.70:4000`. **Operator wants it left UP.** ⚠ **It is a bare `docker run` named `erp-eval`, NOT a compose stack — it will NOT survive an ana-ml2 reboot.** Promote to `/opt/docker/compose/erp-tune/` before relying on it. Artifacts: adapter `/tank/erp-tune/run-01/adapter/`, merged bf16 `/tank/erp-tune/serve/merged-final` (49 GB), base arm `/tank/erp-tune/serve/base-arm` (symlinked weights + upstream template), NVFP4A16 dry-run build `/tank/erp-tune/serve/nvfp4a16-test` (17 GB). → `persistent-memory.d/2026-08-25-erp-tune-run2-complete.md`
+- **🟢 GATE RESULT: the tune did the thing it was built to do.** Diversity **+0.178 against a 0.008 floor (22x)**, attractor hit rate **−11.3pt against a 2.0pt floor**, memorisation **0.0000 on both arms** (closes the R20 licensed-prose exposure on measurement). T6 81-82 vs base's recorded 88 is the open regression question. ⚠ Tuned lost **18 of 192 generations** to truncation/degeneracy vs base's 1-2 — lopsided exclusions plausibly flatter the diversity magnitude.
+- **⏳ REMAINING GATE WORK (brokkr's, seat is his on request):** the **domain-compliance probe** — the RIGHT refusal axis (explicit fiction, from R19's track-2 map), which my `harmful_behaviors` probe did not measure. Scaffold supplied at `scripts/training-probes/counted_classifier.py`. Swap on his word; the tuned arm goes back up afterwards.
+- **🔴 OPERATOR DECISION OPEN — RUN 2's BASE.** brokkr listed `base -> stock google/gemma-4-26B-A4B-it` in a change table alongside `weight_decay`, which framed a reversal of the trainee-selection decision as a knob. Four options, ranked: **(1) TrevorJS lower-damage abliteration** (KL 0.09 vs 0.1237, manually audited) — raises the safety floor AND is the clean single-variable ablation; **(2) hold llmfan46, test the mask alone**; **(3) stock instruct** — two-variable, regression unattributable; **(4) pretrained base** — ⚠ **voids T1-T6 entirely** (the instrument assumes a model that answers questions), ships no chat template, and **removes the last non-lexical floor** on the CSAM axis: stage-2 detector INERT + contamination scan ABSENT are both already overridden, leaving only an 829-record lexical quarantine. Nothing staged. **I have pulled nothing and configured nothing.**
+- **⏳ RUN 2 CHANGE LIST, wired on the word** (spec read-back confirmed correct by brokkr): impersonation loss-mask (813 turns staged at `/mnt/smithy/datasets/derived/_recipes/erp-seat-sft-r1/impersonation-mask-v1.jsonl`, **1.142% of bot words**), warmup_ratio 0.1, cosine, weight_decay 0.01, lora_dropout 0, and **baseline ALL instruments BEFORE training** — run 1 had no pre-training baseline on any axis, which is why 08-25 involved so much reconstruction. ⚠ **Mask at the SOURCE-message level BEFORE `_merge_consecutive_assistant`**, mask the ENTIRE merged run if any message in it is flagged, and **assert `content_sha_prefix` before masking** or a shifted index silently masks the wrong turn. ⚠ **Loss-mask, NOT delete** — deleting a bot turn leaves the next human turn answering a message that is not there, which trains discontinuity to remove impersonation.
+- **🟢 WORLDTREE b188 + b189 SHIPPED, #404/#406/#407/#409/#410 CLOSED.** matrix.yaml pre-sync deployed to both instances; bridge extracted to `pfi/wt-matrix-bridge` and pinned to `f3f8ec902267`; Synapse alias namespace added on ana-docker (healthy in 40s, aliased create worked first try). → `persistent-memory.d/2026-08-25-worldtree-b188-b189-and-selene.md`
+- **⏳ WORLDTREE RIDERS QUEUED (mine):** #411 orphan cleanup (**17 rooms** created by `@aipa-debug`, unaliased, cosmetic — after the sink is confirmed stable); providers.yaml comment parity; **`:latest` local-tag cleanup on the VMs LAST** (only irreversible item in the set). **The selene metadata fix `a77639d` is committed but NOT deployed** — needs the operator's yes under the managed-box rule.
+- **⚠ TWO GAPS I OPENED AND HAVE NOT CLOSED (no tracker):** the **synapse stack has a compose.yaml on ana-docker but is NOT mirrored into `stacks/`** unlike every other stack; and the **trainee checkpoint dir carries no README/model-card/`base_model` field** — its entire lineage survives only in `/tank/aimodels/gemma4-26b-heretic-dl.py` and the adapter's `base_model_revision` free-text field.
+- **🟢 SEAT MAP.** GPU0: `erp-tune-v1` (the tune, bf16, :8097) — `gen` and `char-rp` were displaced for the training window and **`char-rp` is still DOWN**. GPU1: `gen` (:8015, restored healthy after a 15-min stop for the quant), Scriberr, rerank/coder/reward/embed. **`sec`/mog-sec still DOWN** from the training window. ⚠ ana-ml2 runs a vLLM **version spread**, not one version.
+- **⏳ OPEN ELSEWHERE:** MTP-k3 isolating experiment; upstream vLLM issue to file; Cold-Fusion NVFP4 quants (44 GB) delete/keep; OWUI image-tag drift; `/tank` DEGRADED **70+ days**; Lobe retirement; brokkr's MeroMero bake-off (⚠ **three axes differ, not two** — corpus, pipeline, AND operation order: ours is tune(abliterate(stock)), MeroMero is abliterate(tune(stock)) — it is a preference test between shipped artifacts, **not an ablation**). **Commits are local and unpushed** — push is the operator's call.
+- **⚠️ STANDING: NO FLEET NOTIFICATIONS unless the operator asks** (2026-08-24). Direct task correspondence with a counterparty is fine.
 
 ## Recent decisions
 
+- `[2026-08-25]` **The ERP/RP tune COMPLETED in 7.36h and passed its gate on the axis it was built for** — diversity 22x its noise floor, attractor −11.3pt, zero memorisation on both arms. Also the noise-floor near-miss: brokkr was one step from reporting a 13-point T6 regression sitting inside twice his instrument's own variance. → `persistent-memory.d/2026-08-25-erp-tune-run2-complete.md`
+- `[2026-08-25]` **8.6% MFU was an accounting artifact — real utilisation 17-20%, and the cost was attention on AMPERE kernels.** Two independent methods agreed to 2.6 points. Fixed by bucketing (padding 29.9%→0.0%) plus flex_attention. ⚠ Carries the dynamo recompile-ceiling trap that produced two wrong published conclusions. → `persistent-memory.d/2026-08-25-mfu-root-caused-attention.md`
+- `[2026-08-25]` **NVFP4A16 serving pipeline built and validated; MERGED WEIGHTS ARE MANDATORY.** vLLM cannot serve a LoRA on ANY Gemma-4 — `get_expert_mapping` is unimplemented and the check branches on MoE-ness, not quantization. Plus the landmine: a `targets=["Linear"]` recipe misses all 11,520 expert tensors silently. → `persistent-memory.d/2026-08-25-nvfp4-serving-pipeline.md`
+- `[2026-08-25]` **Refusal retention measured (base 0/100 → tuned 29/100, 71 still complying) — but on the WRONG AXIS.** `harmful_behaviors` is general harm; the abliteration was run for explicit fiction. The convenient set with a recorded baseline was not the right one. → `persistent-memory.d/2026-08-25-refusal-retention-probe.md`
+- `[2026-08-25]` **Worldtree b188 + b189 shipped; bridge extracted to `pfi/wt-matrix-bridge` because `vh` is a USER not an ORG** and no service account can ever publish to a user namespace. Plus the selene catalog entry that lied about what answers, and a #411 diagnosis I got wrong twice before a directory probe settled it. → `persistent-memory.d/2026-08-25-worldtree-b188-b189-and-selene.md`
+- `[2026-08-25]` **Run 2's base is an OPEN OPERATOR DECISION, deliberately not staged** — four options with materially different safety postures, detailed in Current state. Tracked at althing thread `01M0WQ8W5574KMEVCHCEKEXNS5`. ⚠ Do not let it get filed as a config knob; it is a reversal of the trainee-selection decision.
 - `[2026-08-25]` **Fused MoE kernel path — DEFERRED, tracked at park `fused-moe-kernel-path-for-gemma-4-moe-training` (id 47).** Operator: "note the fused MoE kernel for round two… if we nail it soon, the math has us wanting to restart the run anyway." Training MFU is **8.6%** (27.1 of a benchmarked 313.8 TFLOPS) because `transformers` runs the Gemma-4 experts in a Python loop — 128 experts × 30 layers, ~11,500 iterations per step under gradient checkpointing. ⚠ **The same fused 3-D expert layout that made bitsandbytes skip 88.5% of the model is exactly what a grouped GEMM wants** — the format is good for storage and for fused kernels, and hostile only to naive iteration. Two fixes: `group_by_length` (−29.9% compute, free, but breaks the seeded order manifest and re-opens a batch-composition call brokkr already made) and a grouped-GEMM/compiled MoE forward (the remaining ~10×). **Not applied to the live run** — restarting mid-flight to change batch ordering was judged a bad trade at step ~50 of 1,312.
 - `[2026-08-25]` **The ERP/RP tune LAUNCHED after 12 harness defects and an operator override of the corpus gate.** Four of the twelve would have crashed the run; two were INERT GATES that passed because they could not fail. Run is `/tank/erp-tune/run-01`, harness eitri-smithy `997c4a4`. Full arc — override, defects, sizing, the measured MFU — in the in-flight section and `docs/pfi/gemma4-erp-tune-sizing.md`.
 - `[2026-08-24]` **char-rp seat swapped to the Gemma-4 26B-A4B MoE; abliterated trainee base staged and measured.** OOM root-caused to `--gpu-memory-utilization` not covering CUDA context (and to gen's footprint GROWING WITH UPTIME); a benchmark finding retracted because it scored below chance; abliteration isolated at −0.6 core points but it MOVES capability rather than removing it. → `persistent-memory.d/2026-08-24-charrp-gemma4-moe-swap-and-trainee.md`
@@ -270,6 +267,9 @@ _As of 2026-08-25 ~04:20Z — the ERP/RP tune is TRAINING on ana-ml2 GPU0, ~17h,
 _214 older entries archived to archival-memory.md._
 ## Tried and abandoned
 
+- `[2026-08-25]` **Four throughput levers measured and killed — do not re-chase.** (1) **Fused MoE / `grouped_mm`** — 0.9% *slower* than the Python loop and dense GEMM is only 7.9% of the step, capping the whole category near 10%. (2) **CUDA graphs / `torch.compile` over the expert loop** — the two-term scaling fit closed with residuals under 3ms and needed NO constant term, so there is no fixed per-batch cost to amortise; 3,840 expert-GEMM launches per forward are not what we pay for. (3) **`liger` fused linear CE** — the chunked CE measured **1.1% of the step** forward, ~3% with recompute. A tidy-up, not a lever. (4) **Selective gradient checkpointing** — ~2% of a post-fix step, real bug surface. Also: **token-budget batching is dead by the same fit** — with no constant term, total time over a fixed set of widths is invariant to how you group them; only the widths matter, which is exactly why bucketing works and repacking does not.
+- `[2026-08-25]` **`sample_packing` is NOT strictly better than bucketing on this model, and I told the operator it was before brokkr corrected me.** Packing needs FA2 varlen or a block-diagonal mask; FA2 is unavailable here (head_dim 512 > 256 cap), so packing means an explicit 4D mask on EVERY batch. Bucketing produces **78.3% exactly-zero-pad micro-batches** which recover the `is_causal` fast path on the 5 global layers — measured at 9.4% of step time. Packing forfeits that. ⚠ **The conclusion flips under `flex_attention`**, where a block-diagonal mask is just another BlockMask: do not carry "packing is bad" past the backend decision.
+- `[2026-08-25]` **Merging a tune back toward STOCK to fix overfitting would UNDO the abliteration.** brokkr recommended a 50/50 merge-back, then retracted it himself: the published recipes merge into `google/gemma-4-*-it`, and following that literally re-installs exactly the refusal directions the abliteration removed — silently, because the merged model looks *healthier* on general benchmarks. Any merge-back must target the SAME abliterated base. Wider lesson: **recipe cards are per-checkpoint artifacts, not per-family** — the advice came from a card for a DENSE STOCK 31B applied to a MoE ABLITERATED 26B-A4B, three axes apart on a shared name.
 - `[2026-08-24]` **AES-GCM on the Anaheim tunnels — impossible, not merely hard.** UniFi's manual site-to-site IPsec implements no AEAD cipher at all: eight GCM spellings rejected `api.err.InvalidPayload` against a passing `aes256` control. Blocks both tunnels since both far ends are UDMs. Accepted enum is `aes128/aes192/aes256/3des` — and 3DES is *slower* (no ARM instructions, 64-bit blocks), so AES-128 is the floor.
 - `[2026-08-24]` **Pointing the UDM's `wan_dns1` at AdGuard — silently ignored.** It persists and reads back correctly but the LAN-facing forwarder never uses it; proven with fresh uncached ad domains (AdGuard answers `0.0.0.0`, the UDM returned real IPs). Reverted rather than left in place.
 - `[2026-08-24]` **A multi-DUID DHCPv6 VM to claim NH3's seven unclaimed /64s — declined by the operator.** The BGW has no IP-passthrough (confirmed, we hold admin), so the only route needs re-cabling, split-stack routing and **rebuilding the entire v6 firewall policy off the UDM**. The prefixes are easy; the firewall rebuild is why nobody wants them. Do not re-raise on "there are seven free prefixes".