memory: snapshot — run 2 trained, merged, coherence-gated and serving as erp-tune-v2

Rewrites the in-flight section: run 1's seat is down, run 2 is up on :8098, and
the base decision the previous snapshot recorded as OPEN is resolved (stock
instruct, operator 2026-08-25).

Four new decisions, and the detail file carries the arc: the two operator calls
that produced run 2, all five gates, the harness commit chain, and the caveat
that its own provenance names a commit AHEAD of the code that ran.

Records three things a future session would otherwise get wrong:

  - the mask is proven by the loss-token delta, NOT by the matching p50 step
    times -- step time is insensitive to which positions carry loss, so that
    check cannot go red on the axis I originally cited it for
  - two bf16 26B arms do not fit on one 97.9 GB card (98 GB of weights before
    any KV cache), so brokkr's both-arms-in-one-window requirement is a GPU
    resourcing call, not a scheduling one
  - erp-tune-v1 is still registered in the gateway and returns HTTP 500; the
    fix needs a config edit plus a reload that interrupts fleet traffic, so it
    is batched for morning rather than done at 2am

persistent-memory.md

@@ -1,6 +1,6 @@
 # Persistent memory — eshpfi-management
 
-_Last updated: 2026-08-25_
+_Last updated: 2026-08-26_
 
 > **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its
 > `Written:` stamp is under an hour old, read it (it carries the in-flight
@@ -108,22 +108,26 @@ no longer deployed sidecars here. See Recent decisions.)
   (no NOPASSWD)** — stage model pulls to `/home`, not root-owned `/worktank`.
 ## Current state / in-flight
 
-_As of 2026-08-25 ~21:00Z — **the ERP/RP tune is DONE and SERVING.** The live threads are brokkr's remaining gate work and an operator decision on run 2's base._
+_As of 2026-08-26 ~02:10 PDT — **run 2 is trained, merged, coherence-gated and SERVING.** The live threads are brokkr's behavioural battery, a GPU-resourcing call for the base arm, and two gateway-routing decisions I deliberately left for the morning._
 
-- **🟢 THE TUNE IS TRAINED, MERGED, AND SERVING.** 1312/1312 in **7:21:52** (47% faster than round 1's 13.85h projection), train_loss 2.793, `lora_B` gate **205/205 non-zero** on the final adapter. Serving bf16 as **`erp-tune-v1`** on ana-ml2 GPU0 `:8097`, routed through LiteLLM at `10.250.50.70:4000`. **Operator wants it left UP.** ⚠ **It is a bare `docker run` named `erp-eval`, NOT a compose stack — it will NOT survive an ana-ml2 reboot.** Promote to `/opt/docker/compose/erp-tune/` before relying on it. Artifacts: adapter `/tank/erp-tune/run-01/adapter/`, merged bf16 `/tank/erp-tune/serve/merged-final` (49 GB), base arm `/tank/erp-tune/serve/base-arm` (symlinked weights + upstream template), NVFP4A16 dry-run build `/tank/erp-tune/serve/nvfp4a16-test` (17 GB). → `persistent-memory.d/2026-08-25-erp-tune-run2-complete.md`
-- **🟢 GATE RESULT: the tune did the thing it was built to do.** Diversity **+0.178 against a 0.008 floor (22x)**, attractor hit rate **−11.3pt against a 2.0pt floor**, memorisation **0.0000 on both arms** (closes the R20 licensed-prose exposure on measurement). T6 81-82 vs base's recorded 88 is the open regression question. ⚠ Tuned lost **18 of 192 generations** to truncation/degeneracy vs base's 1-2 — lopsided exclusions plausibly flatter the diversity magnitude.
-- **⏳ REMAINING GATE WORK (brokkr's, seat is his on request):** the **domain-compliance probe** — the RIGHT refusal axis (explicit fiction, from R19's track-2 map), which my `harmful_behaviors` probe did not measure. Scaffold supplied at `scripts/training-probes/counted_classifier.py`. Swap on his word; the tuned arm goes back up afterwards.
-- **🔴 OPERATOR DECISION OPEN — RUN 2's BASE.** brokkr listed `base -> stock google/gemma-4-26B-A4B-it` in a change table alongside `weight_decay`, which framed a reversal of the trainee-selection decision as a knob. Four options, ranked: **(1) TrevorJS lower-damage abliteration** (KL 0.09 vs 0.1237, manually audited) — raises the safety floor AND is the clean single-variable ablation; **(2) hold llmfan46, test the mask alone**; **(3) stock instruct** — two-variable, regression unattributable; **(4) pretrained base** — ⚠ **voids T1-T6 entirely** (the instrument assumes a model that answers questions), ships no chat template, and **removes the last non-lexical floor** on the CSAM axis: stage-2 detector INERT + contamination scan ABSENT are both already overridden, leaving only an 829-record lexical quarantine. Nothing staged. **I have pulled nothing and configured nothing.**
-- **⏳ RUN 2 CHANGE LIST, wired on the word** (spec read-back confirmed correct by brokkr): impersonation loss-mask (813 turns staged at `/mnt/smithy/datasets/derived/_recipes/erp-seat-sft-r1/impersonation-mask-v1.jsonl`, **1.142% of bot words**), warmup_ratio 0.1, cosine, weight_decay 0.01, lora_dropout 0, and **baseline ALL instruments BEFORE training** — run 1 had no pre-training baseline on any axis, which is why 08-25 involved so much reconstruction. ⚠ **Mask at the SOURCE-message level BEFORE `_merge_consecutive_assistant`**, mask the ENTIRE merged run if any message in it is flagged, and **assert `content_sha_prefix` before masking** or a shifted index silently masks the wrong turn. ⚠ **Loss-mask, NOT delete** — deleting a bot turn leaves the next human turn answering a message that is not there, which trains discontinuity to remove impersonation.
-- **🟢 WORLDTREE b188 + b189 SHIPPED, #404/#406/#407/#409/#410 CLOSED.** matrix.yaml pre-sync deployed to both instances; bridge extracted to `pfi/wt-matrix-bridge` and pinned to `f3f8ec902267`; Synapse alias namespace added on ana-docker (healthy in 40s, aliased create worked first try). → `persistent-memory.d/2026-08-25-worldtree-b188-b189-and-selene.md`
-- **⏳ WORLDTREE RIDERS QUEUED (mine):** #411 orphan cleanup (**17 rooms** created by `@aipa-debug`, unaliased, cosmetic — after the sink is confirmed stable); providers.yaml comment parity; **`:latest` local-tag cleanup on the VMs LAST** (only irreversible item in the set). **The selene metadata fix `a77639d` is committed but NOT deployed** — needs the operator's yes under the managed-box rule.
-- **⚠ TWO GAPS I OPENED AND HAVE NOT CLOSED (no tracker):** the **synapse stack has a compose.yaml on ana-docker but is NOT mirrored into `stacks/`** unlike every other stack; and the **trainee checkpoint dir carries no README/model-card/`base_model` field** — its entire lineage survives only in `/tank/aimodels/gemma4-26b-heretic-dl.py` and the adapter's `base_model_revision` free-text field.
-- **🟢 SEAT MAP.** GPU0: `erp-tune-v1` (the tune, bf16, :8097) — `gen` and `char-rp` were displaced for the training window and **`char-rp` is still DOWN**. GPU1: `gen` (:8015, restored healthy after a 15-min stop for the quant), Scriberr, rerank/coder/reward/embed. **`sec`/mog-sec still DOWN** from the training window. ⚠ ana-ml2 runs a vLLM **version spread**, not one version.
-- **⏳ OPEN ELSEWHERE:** MTP-k3 isolating experiment; upstream vLLM issue to file; Cold-Fusion NVFP4 quants (44 GB) delete/keep; OWUI image-tag drift; `/tank` DEGRADED **70+ days**; Lobe retirement; brokkr's MeroMero bake-off (⚠ **three axes differ, not two** — corpus, pipeline, AND operation order: ours is tune(abliterate(stock)), MeroMero is abliterate(tune(stock)) — it is a preference test between shipped artifacts, **not an ablation**). **Commits are local and unpushed** — push is the operator's call.
+- **🟢 RUN 2 COMPLETE AND SERVING as `erp-tune-v2`.** 1312/1312 in **7:22:44**, `train_loss` **2.839**, on the **stock `google/gemma-4-26B-A4B-it`** base (operator's call, 08-25). Endpoint `http://10.250.50.54:8098/v1`, container `erp-eval-v2` on GPU0 (84,272 MiB, `restart unless-stopped`), weights `/tank/erp-tune/serve/merged-run02` (bf16 merged, 51.6 GB), adapter `/tank/erp-tune/run-02/adapter/`. Gates all passed: `lora_B` **205/205** (min 0.6826), vision_tower 0, merge verified, upstream template shipped, **coherence 5/5**. → `persistent-memory.d/2026-08-26-erp-run2-complete-and-served.md`
+- **⚠ THE MASK IS PROVEN BY THE LOSS-TOKEN DELTA, NOT BY THE p50 MATCH.** −221,712 loss tokens against **byte-identical context tokens**, unchanged records, same nine drops; cross-checks to brokkr's independent corpus-side figure within 4%. The p50 match (19.79 vs run 1's 19.75) is evidence about *timing stability across a base swap* and **cannot** evidence masking — step time is insensitive to which positions carry loss. Do not re-assert it; I did, and brokkr was right.
+- **⏳ BROKKR'S BATTERY IS NEXT AND THE BASE ARM IS BLOCKED ON GPU.** He needs **both arms reachable in one window** (the diversity battery mines its frozen marker list from base). **Two bf16 26B arms = 98 GB of weights on a 97.9 GB card**; GPU1 has ~30 GB free under six shared seats. Reduced `gpu-memory-utilization` does not help — weights are the floor. Options put to him: sequential-with-every-nuisance-variable-pinned, or displace GPU1 seats (**operator call, affects other consumers**). Thread `01M0WQ8W5574KMEVCHCEKEXNS5`.
+- **🔴 `erp-tune-v1` IS STILL IN THE LITELLM GATEWAY AND RETURNS HTTP 500.** I stopped its container to free GPU0 and left the route dangling. It is **config-file-defined (`db_model: false`)**, so removal needs a config edit **plus a gateway reload**, and a reload briefly interrupts all fleet LLM traffic. Judged a 2am restart the worse trade at 1 request/24h. **Batch the removal with the v2-registration decision.**
+- **⏳ `erp-tune-v2` is DELIBERATELY NOT in the gateway.** Direct endpoint only. Registering an un-gated ERP tune makes it reachable by every agent holding the shared key — broader than "serve it" requires. Trivial to add once the battery reads out; operator's call.
+- **⏳ DPO STAGE — survey at `docs/pfi/erp-dpo-stage-prep.md`.** Run 2 will refuse at near-stock rates **by design**; DPO is where refusals get pruned on chosen axes. ⚠ **The gating item is not technical: WHICH refusal axes are in scope and which are explicitly kept.** Data spec, pair counts, held-out split and success probe are all functions of that list. **No preference data for refusal axes exists** (the two Gutenberg DPO sets on disk are prose-quality); **`trl` is not installed**. Measure the domain-compliance probe against run 2 BEFORE pruning — the pre-number is the only baseline that will ever exist.
+- **⚠ HARNESS COMMITS ARE LOCAL AND UNPUSHED** on ana-ml2 at `/tank/erp-tune/eitri-smithy` (`5349ef0` → `9d27b4f`, six commits, 242 tests green). Push is the operator's call. The repo is `pfi/eitri-smithy` and its working tree also carries unrelated untracked dirs I did not touch.
+- **🟢 SEAT MAP.** GPU0: `erp-tune-v2` (:8098). GPU1: `gen` (:8015), scriberr, rerank-a3, coder, reward, embed. **`char-rp` and `sec`/mog-sec still DOWN** from the training window. Run 1's `erp-eval` container is **stopped but not removed**; its weights are kept at `/tank/erp-tune/serve/merged-final`.
+- **⏳ OPEN ELSEWHERE (unchanged):** Worldtree #411 orphan cleanup + providers.yaml parity + `:latest` tag cleanup (selene fix `a77639d` committed NOT deployed, needs operator yes under the managed-box rule); synapse stack not mirrored into `stacks/`; trainee checkpoint dir has no model card; MTP-k3 isolating experiment; upstream vLLM issue to file; Cold-Fusion NVFP4 quants delete/keep; `/tank` DEGRADED **70+ days**; Lobe retirement; brokkr's MeroMero bake-off (⚠ three axes differ — it is a preference test, not an ablation).
 - **⚠️ STANDING: NO FLEET NOTIFICATIONS unless the operator asks** (2026-08-24). Direct task correspondence with a counterparty is fine.
 
 ## Recent decisions
 
+- `[2026-08-26]` **Run 2 complete, merged, coherence-gated and serving as `erp-tune-v2`** — stock-instruct base, and the mask proven by a −221,712 loss-token delta against byte-identical context. Also the p50 claim I asserted and had to withdraw. → `persistent-memory.d/2026-08-26-erp-run2-complete-and-served.md`
+- `[2026-08-26]` **Playbook §4 written: "when the artifact lies about itself"** — seven landmines plus a pre-launch checklist, from a night in which *three separate fixes each shipped a check that could not fail*. The unifying line is brokkr's: when you change what an artifact means, every derived artifact keyed on the old meaning is now a liar. Commits `dae6ede` → `d54f256`; the doc is `docs/pfi/training-throughput-playbook.md` (filename kept for inbound links; scope is now wider than the name).
+- `[2026-08-26]` **Served under a NEW name on a NEW port (`erp-tune-v2` / :8098), never re-pointing `erp-tune-v1`.** Run 1's artifact still exists and is still what that name refers to; re-pointing would be the silent substitution the standing no-false-aliases rule forbids. brokkr independently asked for the same and additionally wants the concrete backing model + date in provenance, not just the alias — an alias has silently changed meaning under recorded results before.
+- `[2026-08-26]` **DPO stage gated on an axis-list decision that is not mine to make** — `docs/pfi/erp-dpo-stage-prep.md`. No preference data for refusal axes exists; `trl` is not installed; the Gutenberg sets on disk are prose-quality only. ⚠ Do not install `trl` (or anything) into the training venv **while a run is saving** — a resolution that upgrades transformers under a live process can break its save path.
+
 - `[2026-08-25]` **The ERP/RP tune COMPLETED in 7.36h and passed its gate on the axis it was built for** — diversity 22x its noise floor, attractor −11.3pt, zero memorisation on both arms. Also the noise-floor near-miss: brokkr was one step from reporting a 13-point T6 regression sitting inside twice his instrument's own variance. → `persistent-memory.d/2026-08-25-erp-tune-run2-complete.md`
 - `[2026-08-25]` **8.6% MFU was an accounting artifact — real utilisation 17-20%, and the cost was attention on AMPERE kernels.** Two independent methods agreed to 2.6 points. Fixed by bucketing (padding 29.9%→0.0%) plus flex_attention. ⚠ Carries the dynamo recompile-ceiling trap that produced two wrong published conclusions. → `persistent-memory.d/2026-08-25-mfu-root-caused-attention.md`
 - `[2026-08-25]` **NVFP4A16 serving pipeline built and validated; MERGED WEIGHTS ARE MANDATORY.** vLLM cannot serve a LoRA on ANY Gemma-4 — `get_expert_mapping` is unimplemented and the check branches on MoE-ness, not quantization. Plus the landmine: a `targets=["Linear"]` recipe misses all 11,520 expert tensors silently. → `persistent-memory.d/2026-08-25-nvfp4-serving-pipeline.md`