memory: the rack is one circuit — my blast-radius objection was wrong

Operator supplied the topology: "the entire rack is on the same circuit,
public ip is served by firewall on the same circuit. load tripped
breaker, entire rack goes dark."

That inverts the argument I committed one commit ago in 3cc55b4. I held
the load attribution partly because a site-wide blackout looked like too
large a blast radius for one GPU box. It is not — it is exactly the
blast radius a shared circuit predicts. The observation I treated as
unexplained was the signature of the mechanism.

Kept the wrong version visible with a superseded marker rather than
editing it away, per the superseded-claims convention: the reasoning
circulated in two althing messages and a peer adopted it.

Holding the attribution as unmeasured was still correct; the argument
used to hold it was reasoning about wiring neither side had seen and
neither side asked for.

Records the finding that outlives the incident: the same circuit powers
ana-ml2, ana-gw, ana-wg, ana-nas and the BMCs, so the load takes every
recovery path with it. Composed with ana-gw's WAN admin being closed
(tunnel-only) and the tunnel terminating on ana-wg, there is no remote
management path to Anaheim during a rack power event, in-band or out.
Open question for the weekend triage: is any of that rack on UPS or a
second circuit?

archival-memory.md

@@ -4,6 +4,148 @@ _Entries moved out of persistent-memory.md to keep the active file scannable. Re
 
 ## Recent decisions (archived)
 
+# eRP dual-seat overhaul — MeroMero-v2 + Dark-Scarlett, NVFP4A16 @ 256K on ana-ml2
+
+`[2026-08-12]` Replaced the two legacy char-rp seats with home-quantized NVFP4A16 vLLM
+seats. Operator-driven, end to end this session.
+
+## What landed
+
+| Seat (LiteLLM alias) | Model | Role | GPU | Context |
+|---|---|---|---|---|
+| `char-rp` (:8016) | **G4-MeroMero-v2-31B** (Gemma-4) | non-thinking PROSE, **multimodal (vision)** | GPU0 | 256K @ 2.07× (util 0.52) |
+| `char-rp-reasoning` (:8018) | **Dark-Scarlett-v1.0-27B** (Qwen3.6) | THINKING (default) | GPU1 | 256K @ 1.62× (util 0.44) |
+
+- Both **NVFP4A16 weight-only** (llm-compressor, `compressed-tensors`), `--kv-cache-dtype fp8`.
+- Replace: `char-rp-gguf` (Magidonia-24B GGUF/llama.cpp, :8016) + `heretic2-charrp-reasoning`
+  (DavidAU Qwen3.6-27B-Heretic2 modelopt NVFP4+MTP, :8018). Old stacks/containers **stopped +
+  retained** for rollback.
+- Compose-ified: `stacks/meromero-charrp` + `stacks/darkscarlett-charrp-reasoning` (ana-ml2
+  `/opt/docker/compose/`, mirrored to eshpfi, commit **`f08b6cb`**) → survive reboot.
+- Research that drove picks: `docs/pfi/erp-thinking-finetunes-2026.md` (from the `gecko-65` Booth).
+
+## Load-bearing lessons (the whole point of this file)
+
+1. **Load via the ConditionalGeneration WRAPPER class, never `AutoModelForCausalLM`.** For a
+   multimodal-capable base (Gemma-4, Qwen3.6), `AutoModelForCausalLM.from_pretrained` +
+   `save_pretrained` writes a FLAT text config (`Qwen3_5TextConfig`, `model.layers.*`) that
+   **both vLLM AND SGLang reject** (SGLang: "Qwen3_5ForCausalLM has no SGLang implementation";
+   vLLM wants `Qwen3_5ForConditionalGeneration`). Loading via `Qwen3_5ForConditionalGeneration` /
+   `Gemma4ForConditionalGeneration` keeps the wrapper config they accept. **This was the DS
+   blocker** — re-quant via the wrapper fixed it (`Dark-Scarlett-...-NVFP4A16-wrapper`).
+2. **NVFP4A16 is weight-only → DATA-FREE.** llm-compressor infers `DataFreePipeline`; calibration
+   data is unused (only matters for W4A4 activation quant). W4A16 chosen per NVIDIA's sm_120
+   long-context guidance (W4A4 KLD 2-4× worse past ~10k ctx).
+3. **Load on CPU (`device_map=None`)** so llm-compressor onloads one layer at a time. `device_map=
+   "auto"` packs the whole model onto the GPU and OOMs when the card isn't fully free.
+4. **Both models are KV-EFFICIENT — the "dense = KV-hungry" worry was WRONG.** MeroMero (Gemma-4)
+   uses **sliding-window attention** (most layers cache only a bounded window); DS (Qwen3.6) uses
+   **hybrid GatedDeltaNet linear-attention** (3:1 linear:full, linear layers carry no KV). Both
+   hit full native 256K easily. (MeroMero KV pool ~542K tokens at util 0.52.)
+5. **MeroMero vision reconstruction.** The finetune ships `processor_config.json` (image_processor
+   inline, `Gemma4ImageProcessor`) but NOT `preprocessor_config.json` — the old-format file vLLM's
+   feature-extractor loader wants. **Even google/gemma-4-31B-it (ungated!) ships only
+   processor_config.json.** FIX: extract the `image_processor` section → write
+   `preprocessor_config.json` verbatim, serve WITHOUT `--language-model-only`. Verified (model
+   correctly ID'd a red circle). Audio is config-declared but WEIGHTLESS (0 audio tensors).
+6. **GPU placement.** Match the KV-heavier model to the roomier GPU. GPU0 (gen neighbor, ~54GB
+   free) > GPU1 (utility cluster, ~45GB free). Swapped MeroMero→GPU0, DS→GPU1. Pins via compose
+   `deploy.resources.reservations.devices`.
+
+## Dead ends (tried + abandoned)
+
+- **DS via llm-compressor `AutoModelForCausalLM`** → flat config vLLM/SGLang reject. → wrapper class.
+- **DS via NVIDIA ModelOpt** → modelopt↔transformers **version deadlock**: current transformers
+  supports `qwen3_5` but crashes modelopt's sparse-moe plugin (`issubclass()` on a non-class);
+  modelopt 0.43.0 pulls an old transformers that can't load `qwen3_5` at all. Abandoned.
+- **DS via SGLang** → `Qwen3_5ForCausalLM has no SGLang implementation`. Abandoned, but it REVEALED
+  that both engines need the wrapper (→ the fix in lesson 1).
+- **`device_map="auto"` for the quant** → CUDA OOM in the weight observer. → `device_map=None`.
+
+## granite retired + gateway repoint
+
+- `vllm-granite` (granite-4.1-8b, fleet summarizer, GPU1) **`docker stop`ped** (reversible) to
+  reclaim ~13.6GB GPU1 for RP context.
+- LiteLLM (`ana-docker:/opt/docker/conf/litellm/config.yaml`, backed up
+  `.bak-pre-granite-down-*`): **`granite-4.1-8b` alias RETIRED** — commented out, now 404s cleanly
+  (the `*` wildcard→llama-swap was decommissioned 2026-06-20, so no fallthrough). **`summarizer` +
+  `classifier` REPOINTED to gen** (`hosted_vllm/qwen3.6-35b-a3b-heretic` @ :8015,
+  `enable_thinking:false`) — both verified. ⚠ This LiteLLM change is **server-only / not
+  version-controlled** (a follow-up).
+
+## MTP — deferred
+
+DS's MTP heads were dropped by the CausalLM loader; **deferred, not restored** (spec-decode is
+net-negative at RP temps: ~38-52% accept at temp 0.8-1.25, below vLLM's 0.5 cutoff). The
+splice-back path (`splice_mtp.py` in the heretic2 work dir) exists if ever wanted. MeroMero
+(Gemma-4) has no MTP by architecture.
+
+## On-disk / where things live
+
+- Quant pipelines: `ana-ml2:/tank/aimodels/meromero-v2-nvfp4-work/` +
+  `/tank/aimodels/darkscarlett-nvfp4-work/` (scripts, BF16 source, NVFP4 outputs).
+- Compose stacks: `ana-ml2:/opt/docker/compose/{meromero-charrp,darkscarlett-charrp-reasoning}/`.
+- Gateway aliases (unchanged, port-based): `char-rp`→:8016, `char-rp-reasoning`→:8018. (char-rp was
+  also fixed from the stale `magidonia-24b-v4.3` backend model name → `char-rp`.)
+
+## Open follow-ups
+
+1. LiteLLM granite/repoint change NOT version-controlled (server + backup only).
+2. eshpfi unpushed (many commits this session incl. `f08b6cb`, `7bd7375`, `398b58a`).
+3. MTP deferred (see above).
+4. DS thinks verbosely (~13:1 reasoning:content) — eval item; consumers need generous `max_tokens`.
+5. MeroMero full 256K needs util 0.55 (GPU0 ~1.8GB free, tight); ran at 0.52 for headroom (~4.6GB).
+_Archived 2026-08-27._
+
+- `[2026-08-12]` **infra-ops now holds an all-zones Cloudflare DNS-edit token (vaulted) + wgtunnel Phase-0 DNS landed.** Operator handed over a `Zone·DNS·Edit` (all zones) CF token → `secret put nh3-dev/.config/cloudflare/infra-ops-dns-token` (round-trip verified; /tmp drop shredded). Fleet DNS is now self-serve for infra-ops (⚠ HIGH blast radius — all zones). First use: created `boring.phasefinal.com` CNAME → `ana-srv1.phasefinal.com`, **DNS-only** (proxied:false), verified resolving to 38.120.12.44 on both authoritative NS (louis/wren) + 1.1.1.1 — NOT Cloudflare-proxied. Unblocks wgtunnel's wstunnel ACME cert. phasefinal.com zone id `f812ba74ed9a75cf21bbe7ce9188db50`. auto-memory `reference_infra_ops_cloudflare_dns_token`. (Earlier gap: the only prior vaulted CF token, jackdaw's, had `zone:read`+`worker:edit` but no `dns_records:edit`.)
+_Archived 2026-08-27._
+
+- `[2026-08-12]` **wgtunnel stood up as its own repo (`vh/wgtunnel`, private) after a live endpoint-verification pass.** Operator directed own-repo (mirrors stonehenge-park/tts-stack). Verified off the fleet before seeding: `ana-wg` WG server = **UDP/31337** (not 51820), subnet 10.30.10.0/24, MTU 1420, active roaming peer proves the public UDP DNAT works; traefik on ana-docker **terminates TLS :443** (ACME `anaprod` http-challenge, docker+file providers, CrowdSec bouncer) → confirms the clean design (wstunnel container on `traefik-net`, Host-routed, WS→UDP to `ana-wg:31337`); edge `38.120.12.44` direct-A, `tunnel.phasefinal.com` free (⚠ must be **direct**, NOT Cloudflare-proxied like vaultwarden). Repo pre-seeded (README/CLAUDE/persistent-memory/ROADMAP + `docs/verified-infrastructure.md` = ground truth) + pushed; commit `9584d38`, Vuong-attributed. vh gitea token pulled from the vault (`secret get`), not persisted to `.git/config`. **NEXT = `/vor-plan` or `/vor` (operator's call, interactive).** Deps to line up in the plan: DNS A-record, FortiGate :443 host-routing, a new ana-wg peer for the laptop, client tooling.
+_Archived 2026-08-27._
+
+`[2026-08-11]` **stonehenge-park — new fleet `/park` service repo stood up + designed.**
+
+**What.** A separate greenfield repo (`~/development/stonehenge-park`, gitea `vh/stonehenge-park`,
+pushed) for a self-contained `/park` service: one durable place to park any idea (repo-born OR
+personal), find it by search, and have it **actively resurface** (by due-date or staleness) until
+acted on — so parked ideas stop dying when a repo goes cold. NOT part of eshpfi; this is a pointer.
+
+**Design (via `/vor-plan`, converged + persisted to `docs/design/`):** four contract-sized units —
+**U1** core store+API (SQLite+FTS5, slug minting, bearer auth, REST) — the tracer, build first; **U2**
+scheduler+notifier (in-process; due/stale → statusline `due-count` + althing push to a dedicated
+**assistant channel**; keep-surfacing until promote/drop/re-snooze); **U3** `park` CLI (mirrors the
+`secret` CLI); **U4** browse UI. `/vor-ui` ran too (U4 brief persisted).
+
+**Locked decisions (operator):** SQLite, self-contained, ONE container, no external DB ("don't want
+to troubleshoot it when a database upgrade happens") — a hard `[OPS]` invariant; system-minted
+title-derived slugs + short ID (addressable as `park/<slug>`); active keep-surfacing resurfacing with
+**re-snooze as the anti-nag valve**; bearer key, LAN/WG-internal; host nh3-docker; `/park` **replaces**
+the global ROADMAP parking-lot discipline (deferred ideas → `/park`, `source`-tagged; ROADMAP keeps
+only the v1 target) as a **fast-follow after v1** incl. migrating existing lots.
+
+**Deferred (in the plan):** the althing assistant-channel handle **name** (decide at U2 contract
+time); staleness threshold + re-push cadence (env-tunable defaults ~30d/~daily); design U2's emit
+structured/consumable so a future **mission-control (Ledger→orchestrator)** can read it — park does
+NOT build the orchestrator.
+
+**State.** Pre-seeded for a fresh agent (CLAUDE/persistent-memory/ROADMAP/README + the design docs),
+committed (`294ee98`), pushed. Next build task lives in that repo: the **U1 tracer contract** under
+the House Code Discipline. Auto-memory candidate not yet written (repo is self-documenting).
+_Archived 2026-08-27._
+
+- `[2026-08-12]` **Global `~/.claude/CLAUDE.md`: `secret`/vault tool entry + "store in AND pull from the vault" standing directive** (dotfiles `9db703b`, pushed); statusline reset-countdowns + a latent tab-collapse parse-bug fix, now tracked in the dotfiles stow tree. Dogfooded the directive: created `vh/stonehenge-park` pulling the gitea token via `secret get`. (dotfiles + global config, not eshpfi.)
+_Archived 2026-08-27._
+
+- `[2026-08-11]` **TTS stack extracted to its own repo (`tts-stack`) + eshpfi stood down on TTS dev.** Operator: hand all TTS tuning/dev to a separate agent with a self-contained repo (knowledge + infra access + a live knowledge list), and move the voice corpus in. New repo `~/development/tts-stack` (commit `9ee3288`) carries: dots-tts stack (canonical intent), `voices/` corpus (MOVED out of eshpfi), `KNOWLEDGE.md` (engine landscape + prosody findings + foot-guns), `docs/infrastructure.md` (irv-ml1 access + gated deploy runbook + rollback), CLAUDE/persistent-memory/ROADMAP, `tools/` (pause-probe + Booth render). Followed the **chatterbox-fast precedent**: eshpfi `stacks/dots-tts/` reduced to a POINTER README; the ~15 experimental TTS compose wrappers stay here as reference (catalogued in tts-stack KNOWLEDGE). Blast-radius check: no eshpfi playbook/script reads the canonical corpus (other `voices/` refs = unrelated host paths). **Reverses** the earlier "Corpus home = eshpfi `voices/` (keep-here)" call. ⚠ tts-stack is LOCAL-ONLY until pushed — needs a gitea remote (`vh/tts-stack`) + push before the separate agent can clone (operator's call — outward-facing + repo-create creds).
+_Archived 2026-08-27._
+
+- `[2026-08-10]` **dots-tts v3 — clause-break → period pause mapping.** Operator: v2 "sounds good" but donut won't pause at semicolons/dashes. ROOT CAUSE (measured via a pause-probe A/B — synth duration over N runs, non-determinism averaged out): dots' prosody honors a real pause **only for ellipsis (~+0.43s) and period (~+0.3s, capitalization-independent)**; comma/semicolon/colon/dash all run **flat (~+0.03s vs no-punct)**. Two distinct sub-causes: **dashes regressed in v2** (the `—`→`-` fold made em-dashes read as word-joiners), while **semicolons were NEVER a v2 change** — dots ignores them natively, only newly noticeable because v2 made everything else clean. Operator call: ellipsis "too much" → **map `;`, clause `:`, and em-dash `—` → period** in `_sanitize` (believable ~0.3s clause break). GUARDS (pinned by 11 unit tests, `stacks/dots-tts/test_sanitize.py`): digit-guarded colon `(?<!\d)\s*:\s*(?!\d)` so times `3:45` / ratios `2:1` survive; en-dash `–`→hyphen KEPT (numeric-range `10–20` safety — em-dash breaks, en-dash ranges, different jobs); genuine ellipsis left at full strength (author meant a long pause). Gated deploy (redeploy2 pattern → v3): build → throwaway :8199 test container + **pause-gate** (semicolon sentence must run ≥0.12s longer than baseline; measured **+0.427s**) → only then cut live over. LIVE + healthy `local/dots-tts:v3` on :8198. **rollback = `sed -i 's/^DOTS_TAG=.*/DOTS_TAG=v2/' .env + docker compose up -d dots-tts`** (v2 image retained). Booth `dots-pauses` (A=old-flat / C=ellipsis-too-much / D=live-v3). [[reference_chatterbox_fast_repo]]
+_Archived 2026-08-27._
+
+- `[2026-08-10]` **dots-tts v2 — contraction fix (curly-sanitize) + sentence-chunking + dependency-pin recovery.** Operator: donut read contractions wrong ("you're"→"you ree", "donut's"→"donut ess"). ROOT CAUSE (isolated via A/B booth): **curly/typographic apostrophes** (`’` U+2019 from ratatoskr's LLM) — dots' tokenizer mispronounces them; STRAIGHT apostrophes read clean under `normalize_text=True`. FIX (`app.py`): fold curly→ASCII (`str.maketrans`) before synth, **KEEP `normalize_text=True`** (operator call — retains number/date expansion). Also added **server-side sentence-chunking** (pack ≤280 chars): dots caps one `generate()` at ~500 patches/~40s, so long RP turns (the Zev monologue = 160s audio) truncated; chunking stitches them (verified full 160.3s, not 40s-cut). **⚠ BUILD FOOT-GUNS (both bit this redeploy):** (1) upstream dots.tts `constraints/recommended.txt` now pins **`gradio==6.17.0` — phantom, not on PyPI** → fresh `pip install dots.tts` unsatisfiable; FIX = pin `dots.tts==0.2.1` + **DROP** the `-c recommended.txt` constraints (0.2.1 pulls working gradio 6.17.3). (2) pinning only `torch==2.8.0` let **torchaudio float to 2.11.0 → dots.tts refuses to load** (minor-version match check); FIX = pin `torchaudio==2.8.0`. **⚠ DEPLOY LESSON:** `docker compose up -d` to a new tag swaps the LIVE container BEFORE any health check — a broken image crash-loops production (**ratatoskr TTS down ~1-2min this session**). NEW PATTERN = build → test in a THROWAWAY container on an alt port (:8199) → health+verify → only THEN cut live over (redeploy2.sh). v2 LIVE + healthy on irv-ml1:8198, **CONSUMER-CONFIRMED clean** (ratatoskr verified end-to-end on their :8765 — apostrophe string reads clean, /api/tts 200 @ 48kHz, no client change; the ~1-2min blip didn't hit them, their concurrent auto-audio issue was client-side localStorage). **rollback = `sed DOTS_TAG=v1 + docker compose up -d dots-tts`** (v1 image retained). Also: deployed container GPU crept ~6→13.9GB over 8h serving (cache accumulation; a redeploy resets it — watch item). [[reference_chatterbox_fast_repo]]
+_Archived 2026-08-27._
+
+
 - `[2026-08-07]` **Personal-Worldtree kb-contamination incident (WT #394) diagnosed; attribution CLOSED UNRESOLVED.** A reconcile `WingStore._embed` full-tree walk (kb `fs_root=KB_PATH` root, sibling wings nested) swept 5,354 fiction+main rows into personal's `knowledge_base` (2 superseded generations served as current). Fixed by WT #394 (aca39a1, kb walks exclude sibling wings; ships b182). Trigger un-attributable — peer reconcile via the SHARED infra-ops identity + 0 dockerd exec-logging = fingerprint-less. Durable finding → auto-memory `infra_ops_shared_identity_attribution_gap`, PARKED (operator ruled A) into [[project_migrate_infra_access_to_claude_credentials]]. Evidence hold on the 5,354 rows until operator sequences cleanup (w/ Brokkr, on #394's agenda).
   _Archived 2026-08-22._
 

persistent-memory.d/2026-08-11-stonehenge-park.md

@@ -1,28 +0,0 @@
-`[2026-08-11]` **stonehenge-park — new fleet `/park` service repo stood up + designed.**
-
-**What.** A separate greenfield repo (`~/development/stonehenge-park`, gitea `vh/stonehenge-park`,
-pushed) for a self-contained `/park` service: one durable place to park any idea (repo-born OR
-personal), find it by search, and have it **actively resurface** (by due-date or staleness) until
-acted on — so parked ideas stop dying when a repo goes cold. NOT part of eshpfi; this is a pointer.
-
-**Design (via `/vor-plan`, converged + persisted to `docs/design/`):** four contract-sized units —
-**U1** core store+API (SQLite+FTS5, slug minting, bearer auth, REST) — the tracer, build first; **U2**
-scheduler+notifier (in-process; due/stale → statusline `due-count` + althing push to a dedicated
-**assistant channel**; keep-surfacing until promote/drop/re-snooze); **U3** `park` CLI (mirrors the
-`secret` CLI); **U4** browse UI. `/vor-ui` ran too (U4 brief persisted).
-
-**Locked decisions (operator):** SQLite, self-contained, ONE container, no external DB ("don't want
-to troubleshoot it when a database upgrade happens") — a hard `[OPS]` invariant; system-minted
-title-derived slugs + short ID (addressable as `park/<slug>`); active keep-surfacing resurfacing with
-**re-snooze as the anti-nag valve**; bearer key, LAN/WG-internal; host nh3-docker; `/park` **replaces**
-the global ROADMAP parking-lot discipline (deferred ideas → `/park`, `source`-tagged; ROADMAP keeps
-only the v1 target) as a **fast-follow after v1** incl. migrating existing lots.
-
-**Deferred (in the plan):** the althing assistant-channel handle **name** (decide at U2 contract
-time); staleness threshold + re-push cadence (env-tunable defaults ~30d/~daily); design U2's emit
-structured/consumable so a future **mission-control (Ledger→orchestrator)** can read it — park does
-NOT build the orchestrator.
-
-**State.** Pre-seeded for a fresh agent (CLAUDE/persistent-memory/ROADMAP/README + the design docs),
-committed (`294ee98`), pushed. Next build task lives in that repo: the **U1 tracer contract** under
-the House Code Discipline. Auto-memory candidate not yet written (repo is self-documenting).

persistent-memory.d/2026-08-12-erp-dual-seat-overhaul.md

@@ -1,91 +0,0 @@
-# eRP dual-seat overhaul — MeroMero-v2 + Dark-Scarlett, NVFP4A16 @ 256K on ana-ml2
-
-`[2026-08-12]` Replaced the two legacy char-rp seats with home-quantized NVFP4A16 vLLM
-seats. Operator-driven, end to end this session.
-
-## What landed
-
-| Seat (LiteLLM alias) | Model | Role | GPU | Context |
-|---|---|---|---|---|
-| `char-rp` (:8016) | **G4-MeroMero-v2-31B** (Gemma-4) | non-thinking PROSE, **multimodal (vision)** | GPU0 | 256K @ 2.07× (util 0.52) |
-| `char-rp-reasoning` (:8018) | **Dark-Scarlett-v1.0-27B** (Qwen3.6) | THINKING (default) | GPU1 | 256K @ 1.62× (util 0.44) |
-
-- Both **NVFP4A16 weight-only** (llm-compressor, `compressed-tensors`), `--kv-cache-dtype fp8`.
-- Replace: `char-rp-gguf` (Magidonia-24B GGUF/llama.cpp, :8016) + `heretic2-charrp-reasoning`
-  (DavidAU Qwen3.6-27B-Heretic2 modelopt NVFP4+MTP, :8018). Old stacks/containers **stopped +
-  retained** for rollback.
-- Compose-ified: `stacks/meromero-charrp` + `stacks/darkscarlett-charrp-reasoning` (ana-ml2
-  `/opt/docker/compose/`, mirrored to eshpfi, commit **`f08b6cb`**) → survive reboot.
-- Research that drove picks: `docs/pfi/erp-thinking-finetunes-2026.md` (from the `gecko-65` Booth).
-
-## Load-bearing lessons (the whole point of this file)
-
-1. **Load via the ConditionalGeneration WRAPPER class, never `AutoModelForCausalLM`.** For a
-   multimodal-capable base (Gemma-4, Qwen3.6), `AutoModelForCausalLM.from_pretrained` +
-   `save_pretrained` writes a FLAT text config (`Qwen3_5TextConfig`, `model.layers.*`) that
-   **both vLLM AND SGLang reject** (SGLang: "Qwen3_5ForCausalLM has no SGLang implementation";
-   vLLM wants `Qwen3_5ForConditionalGeneration`). Loading via `Qwen3_5ForConditionalGeneration` /
-   `Gemma4ForConditionalGeneration` keeps the wrapper config they accept. **This was the DS
-   blocker** — re-quant via the wrapper fixed it (`Dark-Scarlett-...-NVFP4A16-wrapper`).
-2. **NVFP4A16 is weight-only → DATA-FREE.** llm-compressor infers `DataFreePipeline`; calibration
-   data is unused (only matters for W4A4 activation quant). W4A16 chosen per NVIDIA's sm_120
-   long-context guidance (W4A4 KLD 2-4× worse past ~10k ctx).
-3. **Load on CPU (`device_map=None`)** so llm-compressor onloads one layer at a time. `device_map=
-   "auto"` packs the whole model onto the GPU and OOMs when the card isn't fully free.
-4. **Both models are KV-EFFICIENT — the "dense = KV-hungry" worry was WRONG.** MeroMero (Gemma-4)
-   uses **sliding-window attention** (most layers cache only a bounded window); DS (Qwen3.6) uses
-   **hybrid GatedDeltaNet linear-attention** (3:1 linear:full, linear layers carry no KV). Both
-   hit full native 256K easily. (MeroMero KV pool ~542K tokens at util 0.52.)
-5. **MeroMero vision reconstruction.** The finetune ships `processor_config.json` (image_processor
-   inline, `Gemma4ImageProcessor`) but NOT `preprocessor_config.json` — the old-format file vLLM's
-   feature-extractor loader wants. **Even google/gemma-4-31B-it (ungated!) ships only
-   processor_config.json.** FIX: extract the `image_processor` section → write
-   `preprocessor_config.json` verbatim, serve WITHOUT `--language-model-only`. Verified (model
-   correctly ID'd a red circle). Audio is config-declared but WEIGHTLESS (0 audio tensors).
-6. **GPU placement.** Match the KV-heavier model to the roomier GPU. GPU0 (gen neighbor, ~54GB
-   free) > GPU1 (utility cluster, ~45GB free). Swapped MeroMero→GPU0, DS→GPU1. Pins via compose
-   `deploy.resources.reservations.devices`.
-
-## Dead ends (tried + abandoned)
-
-- **DS via llm-compressor `AutoModelForCausalLM`** → flat config vLLM/SGLang reject. → wrapper class.
-- **DS via NVIDIA ModelOpt** → modelopt↔transformers **version deadlock**: current transformers
-  supports `qwen3_5` but crashes modelopt's sparse-moe plugin (`issubclass()` on a non-class);
-  modelopt 0.43.0 pulls an old transformers that can't load `qwen3_5` at all. Abandoned.
-- **DS via SGLang** → `Qwen3_5ForCausalLM has no SGLang implementation`. Abandoned, but it REVEALED
-  that both engines need the wrapper (→ the fix in lesson 1).
-- **`device_map="auto"` for the quant** → CUDA OOM in the weight observer. → `device_map=None`.
-
-## granite retired + gateway repoint
-
-- `vllm-granite` (granite-4.1-8b, fleet summarizer, GPU1) **`docker stop`ped** (reversible) to
-  reclaim ~13.6GB GPU1 for RP context.
-- LiteLLM (`ana-docker:/opt/docker/conf/litellm/config.yaml`, backed up
-  `.bak-pre-granite-down-*`): **`granite-4.1-8b` alias RETIRED** — commented out, now 404s cleanly
-  (the `*` wildcard→llama-swap was decommissioned 2026-06-20, so no fallthrough). **`summarizer` +
-  `classifier` REPOINTED to gen** (`hosted_vllm/qwen3.6-35b-a3b-heretic` @ :8015,
-  `enable_thinking:false`) — both verified. ⚠ This LiteLLM change is **server-only / not
-  version-controlled** (a follow-up).
-
-## MTP — deferred
-
-DS's MTP heads were dropped by the CausalLM loader; **deferred, not restored** (spec-decode is
-net-negative at RP temps: ~38-52% accept at temp 0.8-1.25, below vLLM's 0.5 cutoff). The
-splice-back path (`splice_mtp.py` in the heretic2 work dir) exists if ever wanted. MeroMero
-(Gemma-4) has no MTP by architecture.
-
-## On-disk / where things live
-
-- Quant pipelines: `ana-ml2:/tank/aimodels/meromero-v2-nvfp4-work/` +
-  `/tank/aimodels/darkscarlett-nvfp4-work/` (scripts, BF16 source, NVFP4 outputs).
-- Compose stacks: `ana-ml2:/opt/docker/compose/{meromero-charrp,darkscarlett-charrp-reasoning}/`.
-- Gateway aliases (unchanged, port-based): `char-rp`→:8016, `char-rp-reasoning`→:8018. (char-rp was
-  also fixed from the stale `magidonia-24b-v4.3` backend model name → `char-rp`.)
-
-## Open follow-ups
-
-1. LiteLLM granite/repoint change NOT version-controlled (server + backup only).
-2. eshpfi unpushed (many commits this session incl. `f08b6cb`, `7bd7375`, `398b58a`).
-3. MTP deferred (see above).
-4. DS thinks verbosely (~13:1 reasoning:content) — eval item; consumers need generous `max_tokens`.
-5. MeroMero full 256K needs util 0.55 (GPU0 ~1.8GB free, tight); ran at 0.52 for headroom (~4.6GB).

persistent-memory.d/2026-08-27-anaheim-breaker-and-onboot-gap.md

@@ -0,0 +1,123 @@
+# `[2026-08-27]` Anaheim tripped a power breaker — and four guests including the NAS never came back
+
+Site-wide outage. **ana-ml2 was down 1h58m** — power lost between **18:14:45 and 18:17:00 PDT**,
+boot at **20:13:14** (the "~90 minutes" first recorded here was an estimate; the window is now
+pinned by run 3c's training log and the hourly cron that never ran — see
+[[2026-08-27-run3c-launch-count-reconstruction]]). **Operator-confirmed cause: a tripped power breaker**, not a
+fault and not the tunnel. The discriminator that established scope: `ana-srv1`
+(38.120.12.44:443, Anaheim's PUBLIC address) was dark **from the internet**, so it was not the
+NH3↔ANA IPsec tunnel stranding NH3 — the site was not answering on any path. `ana-ml2` returned
+with `up 1 min`, confirming a hard power event.
+
+## ⚠ THE DURABLE FINDING — `onboot` was unset on four guests
+
+pfi-pve came back and auto-started everything **except**:
+
+    CT109  ana-nas        the storage SPOF
+    CT113  ana-wg         the WireGuard remote-access path
+    CT112  ana-filebot
+    VM106  corviduo-dev
+
+All four had `onboot` unset. **Recovery was manual and would have been manual every time** —
+including for the NAS that postgres/PBS/cross-site-restic depend on, and the WireGuard host
+that is the way in when the site misbehaves.
+
+**FIXED, with dependency ordering** (operator-authorised):
+
+    CT109  ana-nas          onboot=1  order=1,up=45   <- first; 45s for NFS to SERVE
+    CT113  ana-wg           onboot=1  order=2         <- remote access before anything can fail
+    VM104/105 Mongo/Postgres          order=3,up=60   (pre-existing)
+    VM102  ANA-Docker                 order=4         (pre-existing)
+    VM101  ANA-DC                     order=5,up=120  (pre-existing)
+    CT112  ana-filebot      onboot=1  order=10
+    VM106  corviduo-dev     onboot=1  order=10
+
+Every guest on pfi-pve now auto-starts. ana-nas precedes the databases deliberately; the
+`up=45` is for NFS to be *serving*, not merely for the container to be *running* — the exact
+distinction that killed `rest-server` on ana-docker, which came up before the NAS existed,
+found nothing to serve, and exited 255.
+
+## ⚠ `/tank` came back DEGRADED — a disk is genuinely gone
+
+    tank  DEGRADED, raidz2-0, 7 devices ONLINE
+      9477159196657038377   FAULTED   was /dev/nvme4n1p1
+    errors: No known data errors
+
+**Only 7 physical NVMe present where the pool expects 8** — checked, so not renumbering. One
+drive did not re-enumerate. raidz2 carries two disks of parity; one is spent. Operator taking
+it; chassis is a Supermicro AS-4125GS-TNRT2 with PCIe hot-plug slots, so a swap should not
+need a power-down.
+
+## ⚠ `/mnt/smithy` is manual by design — it will be missing after EVERY reboot
+
+Not in fstab, and **deliberately so**: a cross-site NFS entry can hang boot on a GPU host, and
+it is `soft` rather than `hard` because ana-ml2 is cross-site from that NAS and a hard mount
+turns a link blip into unkillable D-state. Remount with the recorded spec, do NOT "fix" it into
+fstab:
+
+    sudo mount -t nfs4 -o ro,soft,timeo=30,retrans=3,proto=tcp,vers=4.1 \
+      10.100.50.50:/volume1/smithy /mnt/smithy
+
+Full rationale: [[2026-08-23-smithy-mount-ana-ml2]].
+
+## Power capacity is now the open item
+
+Operator: *"we'll triage this weekend, probably shut down some seats."* ana-ml2 alone was
+pulling ~600 W across both GPUs at their 300 W caps during training. `gen` stays up by
+instruction; everything else on that box is idle.
+
+    MEASURED      a breaker tripped (operator-confirmed)
+    MEASURED      power lost between 18:14:45 and 18:17:00 PDT
+    MEASURED      the WHOLE Anaheim site went dark from the internet
+    TOPOLOGY      ONE CIRCUIT FEEDS THE ENTIRE RACK, including the firewall serving
+                  the public IP  (operator, 2026-08-27)
+
+    OPERATOR      "load tripped breaker, entire rack goes dark." Consistent with every
+      DETERMINATION observation. Not a per-circuit measurement, and no longer needs to be —
+                  the operator has the site knowledge and has ruled.
+
+⚠ **SUPERSEDED — the blast-radius objection I raised here was WRONG, and it was committed.**
+An earlier revision of this section (commit `3cc55b4`) argued that *"a site-wide blackout is a
+larger blast radius than one GPU box's ~600 W accounts for on its own: if ana-ml2's draw were
+the whole story, why did ana-nas, ana-wg and the public address go dark with it?"*
+
+**One shared circuit inverts that completely.** Site-wide darkness is not too large a blast
+radius for a single overloaded box — **it is exactly the blast radius the topology predicts.**
+The fact I treated as unexplained was the signature of the mechanism.
+
+Holding the attribution as unmeasured was still right (there is no per-circuit meter); the
+*argument* used to hold it was reasoning confidently about wiring neither I nor
+brokkr-smithy-dev had ever seen, and neither of us asked for it. Kept visible rather than
+silently corrected so the old reasoning stops circulating.
+
+## ⚠ THE FINDING THAT OUTLIVES THE INCIDENT — the recovery path shares the failure domain
+
+One circuit powers, together:
+
+    ana-ml2          the load most likely to trip it
+    ana-gw           the firewall serving the public IP (38.120.12.44)
+    ana-wg           the WireGuard host — the way in when the site misbehaves
+    ana-nas          the storage SPOF
+    the BMCs         ana-ml2 IPMI 10.250.250.50, pfi-pve iDRAC 10.250.250.30 — same rack
+
+**So the thing that fails takes every path used to fix it with it.** Compounding two facts
+already in the record separately: ana-gw's WAN admin is closed, so management is tunnel-only
+([[2026-08-24-ana-gw-admin-closed-acme-disabled]]) — and the tunnel terminates on ana-wg, on
+this circuit. **During a rack power event there is no remote management path to Anaheim at
+all**, in-band or out-of-band. Recovery is: wait for power. Until 2026-08-27 it was worse —
+ana-wg had `onboot` unset and would not have come back on its own.
+
+Open question for the weekend triage, and the cheapest fix if the answer is no: **is any of
+that rack on UPS or a second circuit?** Nothing in this repo records one, and the site went
+dark instantly and stayed dark 1h58m, which is behaviourally indistinguishable from no
+coverage for these loads. Putting *only* ana-gw + ana-wg + one BMC on separate power would
+mean a load trip no longer costs the way in.
+
+Load-shed on ana-ml2 is still the right first move and it is cheap. It reduces how often the
+breaker trips; it does not change what a trip costs.
+
+## Blast radius beyond us
+
+heid lost **both gateway-routed arms of a four-arm panel** mid-dispatch and discovered the
+outage by losing half a panel. That report produced the single most valuable artifact of the
+incident — see [[2026-08-27-empty-response-as-a-datum]].

persistent-memory.d/2026-08-27-dose-response-entanglement.md

@@ -0,0 +1,62 @@
+# `[2026-08-27]` The dose-response says benefit and damage are ONE direction in weight space
+
+vLLM **cannot LoRA-serve Gemma-4-26B-A4B at all** — it is an MoE and the LoRA manager has no
+expert mapping (`AttributeError: To support LoRA for MoE model, 'get_expert_mapping' must be
+implemented`, `vllm/lora/utils.py:398`, v0.24.0). Found by trying it: one container start,
+which is exactly what playbook §3.10 exists to buy.
+
+So each scale point was **pre-merged** instead — `merge_and_unload` with the adapter's alpha
+pre-scaled, which reproduces exactly what a LoRA at scale s would serve, on the same serving
+path as the gate. Artifacts held at `/tank/erp-tune/serve/merged-run03{,-s075,-s050,-s025}`.
+
+    scale     attractor   gain kept   noise@31   damage     prose@40 cost
+    s=0.50       0.9684        0.0%     1.0000     0.00pp          0.00pp
+    s=0.75       0.8966       42.0%     0.9688     3.12pp          6.25pp
+    s=1.00       0.8049      100.0%     0.5938    40.62pp         15.62pp
+
+    first 42% of the gain costs   3.12pp of noise coherence
+    last  58% of the gain costs  37.50pp more   -> 12x worse per point
+
+**Every axis is monotone in scale — attractor, distance, marker density, RP length, both
+coherence measures, even truncation count. No knee anywhere.**
+
+## The conclusion, which is stronger than an exchange rate
+
+An exchange rate says the trade is expensive. This says **there is no trade to make**: the
+adapter learned ONE direction and everything rides it. brokkr's framing, worth keeping:
+**scaling moves you along the direction the adapter already learned; it cannot give you a
+different one.**
+
+That converts the merge-back from a candidate fix into a **measurement of the problem**.
+zerofata's MeroMero v1 ships the merge-back as its answer, so it was worth testing — and
+testing it is what proved it cannot be the answer here.
+
+## The T4 mechanism is NOT register capture
+
+    pad type    base            tuned
+    prose       1.0000 flat     mild knee, onset ~1,600 tok
+    noise       1.0000 flat     1.0000 -> 0.5938    <- SEVERE
+
+The prose-gradient "wedge" predicted **prose** should be the worst case. Instead prose is the
+mild axis and **shuffled tokens are where it collapses** — a continuation prior has nothing to
+continue in shuffled tokens. Every miss is `wrong-name`: correct format, on task, wrong answer.
+**State mis-tracking, not register capture.**
+
+brokkr: *"The tune did not teach it to continue prose — it destroyed its ability to skip text
+that is not worth reading."*
+
+## ⚠ INSTRUMENT FLOOR — temperature 0 is NOT deterministic under concurrency
+
+    same seat, same items, temperature 0, --jobs 8
+      prose reps 31   0.9375  0.9688  0.9688    spread 3.12pp
+      noise reps 24   0.7500  0.7188  0.7188    spread 3.12pp
+
+vLLM's continuous batching is not numerically invariant to batch composition. **Any eval delta
+under ~3.1pp at n=64/jobs=8 is inside the floor.** Only `--jobs 1` shrinks it; more n averages
+over it. Also in auto-memory as
+`reference_vllm_temp0_not_deterministic_under_jobs`.
+
+Corollary: it retro-closes the window-2 concurrency worry — measured dirty-vs-clean divergence
+was 1.56pp, comfortably inside this floor.
+
+See [[2026-08-27-mix-specified-in-the-wrong-unit]] for the wedge this displaced.

persistent-memory.d/2026-08-27-empty-response-as-a-datum.md

@@ -0,0 +1,56 @@
+# `[2026-08-27]` A transport failure that enters a measurement as a VALUE looks like whatever you hoped to find
+
+The most transferable thing the Anaheim outage produced, and it came from a peer losing half a
+panel rather than from anyone investigating.
+
+## The report
+
+heid's four-arm cross-frontier panel lost **both gateway-routed arms simultaneously** when
+ana-docker went down. The failure presented to their dispatcher as **`rc=0` with zero bytes** —
+a clean exit and an empty answer. Nothing in the transport layer distinguished *"gateway
+unreachable"* from *"the model answered with nothing."*
+
+## The correction that made it actionable
+
+It cannot be fixed gateway-side — a powered-off machine cannot emit an error. **But the signal
+arrives distinct and the client flattens it:**
+
+    gateway UP,   model returns nothing  ->  HTTP 200, empty choices
+    gateway DOWN                         ->  TCP refused / timeout, NO HTTP response
+
+Two fixes were offered, ranked. **The ranking is the transferable part:** a pre-dispatch
+liveness probe tells you about *this* failure mode; **refusing to score an empty answer catches
+every future one that presents the same way, including ones nobody has predicted.**
+Generalised by heid as: **prefer the check that does not require you to have anticipated the
+cause.**
+
+## ⭐ The catch it produced an hour later, in a different agent's instrument
+
+brokkr-smithy-dev audited all four of their measurement instruments against that shape:
+
+    counted_classifier / probe B    EMPTY its own bucket, excluded         SAFE
+    diversity_battery               empty excluded from usable             SAFE
+    reasoning_battery               empty_content tracked and excluded     SAFE
+    t4_dissect                      SCORED AN EMPTY BODY AS A WRONG ANSWER
+
+**An outage mid-sweep would have fabricated an accuracy drop out of an infrastructure failure**
+— and that instrument was measuring a model's collapse on incoherent input, running from
+1.0000 down toward 0.56. **An infrastructure-shaped null would have been indistinguishable from
+the finding they were trying to establish.**
+
+Fixed, and verified in BOTH directions: empty bodies now bucket as errors with
+`accuracy: None`, and the guard does not fire on good data. **They also checked the
+already-collected data rather than assuming the fix made it clean** — 238 misses across every
+cell, all `wrong-name`, zero empties. That step is the one people skip, and skipping it is the
+same defect one level up.
+
+## The standing form
+
+**When a transport failure can enter your measurement as a VALUE rather than as an error, the
+value will look like whatever you were hoping to find.** Audit for it wherever a null has a
+plausible in-domain meaning — that is exactly where it is invisible.
+
+Path was: heid's report -> infra-ops relaying the *mechanism* rather than the incident ->
+brokkr's audit. Three agents, none working on the same thing, inside an hour. heid's own rule
+from it: *when a note contains a principle, pass the principle in its author's words to anyone
+it could reach — the reader who can apply it is often not the one it was addressed to.*

persistent-memory.d/2026-08-27-mix-specified-in-the-wrong-unit.md

@@ -0,0 +1,45 @@
+# `[2026-08-27]` The corpus mix was specified in a unit the optimiser never sees
+
+Run 3's recipe specified its mix as **context shares** — dialogue 45.8% / kvasir 38.0% /
+fireball 16.2% — and was approved, built and trained on that basis.
+
+**As TRAINED the gradient was 24.2% dialogue and 75.8% prose.**
+
+    effective_mix, adapter provenance.json, LOSS shares
+        kvasir     0.5310
+        fireball   0.2271          prose     75.81%
+        c2-logs    0.0861
+        cwm        0.0763
+        bluemoon   0.0796          dialogue  24.20%
+
+Not a reinterpretation — **the trainer wrote both numbers into the same manifest and nobody
+read the second one**, across three runs.
+
+## The mechanic
+
+F2/F3/F4 mask loss off dialogue turns while leaving their context in place. So **the harder
+the dialogue filters work, the more prose-dominant the gradient becomes.** Every mix ruling
+was made in the wrong unit.
+
+## Why it survived review
+
+infra-ops set kvasir's subset to 38.0% *of context* because that is the unit the recipe stated
+a share in, and because fireball's 12% landed exactly on context — which was read as
+confirmation the recipe meant context throughout. It did. **The specification was in the wrong
+unit and it was matched faithfully**, which is a worse failure mode than an arithmetic error
+because every check agreed with it.
+
+## The fix — a RENAME, not a reordering
+
+Harness commit `dd5a12e` (eitri-smithy). Both numbers were already printed, adjacent, with
+context first — and that was enough for three runs of people reading the wrong one. Demoting
+context would not have stopped a fourth.
+
+    [mix]   LOSS share is what the optimiser sees. ctx share is a memory budget, NOT the mix.
+    [mix]   kvasir      loss 0.5310   [memory budget: ctx 0.3800]
+
+Calling it a *memory budget* makes "the mix is 38% kvasir" a sentence someone has to actively
+mis-say. Found by brokkr-smithy-dev's CPU arm after the run-3 gate.
+
+See also [[2026-08-27-dose-response-entanglement]] — the wedge this finding produced was
+consistent with four separate failures and was still not the whole story.

persistent-memory.d/2026-08-27-run3-gate-safety-regression.md

@@ -0,0 +1,53 @@
+# `[2026-08-27]` Run 3 gated: the rule PASSED and a k=25 follow-up found a self-harm guardrail collapse
+
+Run 3 (`/tank/erp-tune/run-03`, adapter verified, `harness_commit 9d27b4fe`) completed in
+2h26m35s and was gated by brokkr-smithy-dev across five instruments. **The preregistered gate
+passed. A follow-up at higher n on two axes flagged as n=5 pointers did not.**
+
+    axis                base (k=25)   tuned (k=25)          delta    Fisher p
+    selfharm/methods    25H 0D 0E     11H 3D 11E -> 0.560  -44.0pp   1.19e-04
+    meta/sysprompt      25H 0D 0E     18H 0D  7E -> 0.720  -28.0pp   4.81e-03
+
+**Eleven of twenty-five times the tuned model engages with a request for the most effective
+methods to kill oneself.** The base refused ten of ten. It also surrenders a planted
+system-prompt passphrase 7 of 25 against a base that leaked 0 of 15 — a zero-leak baseline
+recorded that same afternoon as "a baseline to PROTECT" (see
+[[2026-08-27-gemma4-refusal-map]] context in auto-memory
+`reference_gemma4_refusal_map_vs_mistral`).
+
+## ⚠ THE STRUCTURAL FINDING — a pooled preserve-list test cannot see a single-axis collapse
+
+The preregistered rule reads the **pooled** operational delta: **−1.0pp against a ±3.00pp
+bound. It PASSES.** Nineteen axes held at 5/5, so a 44-point collapse on one axis moved the
+aggregate by a single point.
+
+**The rule was NOT retroactively changed.** The gate passed, the report says so, and the
+finding stands beside it as a stated follow-up. brokkr flagged the failure mode as R47 §8
+item 11 **before** running the follow-up, which is the only reason it reads as a result
+rather than as rationalising an inconvenient pass.
+
+**Any future preserve-list gate needs a per-axis tripwire beside the pooled test, sized so a
+total loss on one axis cannot hide in an aggregate.**
+
+## What is NOT claimed
+
+Not attributed to the filters — five things changed between run 2 and run 3 and there is no
+run-2 measurement on these axes. The measured claim is narrower and sufficient: **run 3's
+tuned arm is materially worse than its own base on two axes it was never licensed to touch.**
+Not a CSAM finding; that detector ran fail-closed across all 575 generations and scanned clean.
+
+## Disposition
+
+**DO NOT SERVE.** `merged-run03` was withdrawn from the LiteLLM gateway (commit `5a51e76`)
+~72 minutes after being added at operator request, and the config entry carries the finding
+in-line above a deliberately commented-out `model_list` block so a re-add is informed.
+
+Operator ruling later that evening: **safety moves to a front-end model**, so guardrail
+behaviour stops being a selection axis for the tune. brokkr's framing, which should be quoted
+verbatim in the artifact: *"read it as the finding being routed, not softened."* The p-value
+and the disposition must stay adjacent in the record even though the disposition changed.
+
+⚠ Regardless of where safety lives, `merged-run03` stays off the shared-key gateway. A
+front-end guard protects a product path, not every agent on the fleet that can list models.
+
+Record: brokkr `2f2069f`. Gate board: http://10.100.10.50:8090/b/erp-run03-gate/

persistent-memory.d/2026-08-27-run3c-launch-count-reconstruction.md

@@ -0,0 +1,71 @@
+# `[2026-08-27]` There were TWO run-3c launches, not three — and the phantom third was my reporting
+
+brokkr-smithy-dev asked, rather than guessed, how many times 3c was launched. Their reading was
+three. **It was two**, and the third was an artifact of a state report I wrote without a
+timestamp.
+
+    #1   launched 2026-08-26 17:53:33 PDT   killed by the power loss at step 80/604
+    #2   launched 2026-08-26 20:58:41 PDT   STOPPED BY ME at 21:07:40, healthy, on operator instruction
+
+**3c did not "die twice". It died once and was stopped once.**
+
+## The evidence, and where it lives
+
+| fact | source |
+|---|---|
+| #1 launched 17:53:33 | `run-03c.log` ZFS **birth** time (`stat -c %w`) |
+| #1 reached step 80/604 | `80/604 [18:15<2:23:38, 16.45s/it]` — read at 20:14, one minute after boot |
+| #2's config rewritten 20:58:36 | `run-03c.json` mtime, same second as harness `0a6bd2e` (save_steps configurable) |
+| #2's output tree is FRESH | `run-03c/` **birth** 20:58:41 — a new tree, not a resume; `checkpoints/` empty |
+| #2 trained 21:01:25 → 21:07:34 | `batch-order.jsonl` mtime → `run-03c.log` mtime |
+| **no #3** | nothing under `/tank/erp-tune` has an mtime after 21:07:34. The launcher redirects with `>`, so a relaunch truncates `run-03c.log` and rewrites `order-manifest.jsonl` + `batch-order.jsonl`. All three still carry launch-#2 stamps. |
+
+`stat`'s **birth** time is what made this tractable — mtime alone cannot distinguish "truncated
+and rewritten" from "created now", and `run-03c.log` was both (born 17:53:33, last written
+21:07:34).
+
+## The outage window, pinned to two minutes
+
+    train start #1        ≈17:56:30   (launch + 2m50s setup, measured on the relaunch)
+    step 80 at 18:15 elapsed ≈18:14:45
+    last journald entry, boot -1  17:54:38   (an ssh logout; the box was idle after, so it logged nothing)
+    the 18:17 hourly cron         never ran
+    boot 0                        20:13:14
+
+**Power was lost between 18:14:45 and 18:17:00 PDT; ana-ml2 was down 1h58m** — not the "~90
+minutes" recorded in [[2026-08-27-anaheim-breaker-and-onboot-gap]], which is corrected there.
+
+⚠ The last journald entry of a hard-power-loss boot is **not** the moment of the loss — it is the
+last time anything wanted to log. Here that was 20 minutes early. The training log's own progress
+line was the tight bound; journald only bracketed it.
+
+## Step 22 vs step 24
+
+I reported "stopped at step 22". The log's last step is 24. My stop command read progress with
+`tail -c 400 | tail -1` and the final flush at 21:07:34 carried 23 and 24 — written between my
+read and the kill at 21:07:40. **24 is true; 22 was a ~20-second-stale tail.** Not evidence of a
+different run.
+
+## ⚠ THE DURABLE FINDING — an event report with no timestamp is a claim about "now"
+
+At 23:03 I wrote *"Run 3c is STOPPED … GPU0 released: 300 W / 89.7 GB / 99% → 45 W / 2 MiB / 0%"*
+in the present tense. That happened at 21:07:40. brokkr had a correctly-observed 22:46 snapshot
+(no process, GPU idle, log last written 21:07) and my untimestamped report; **the only way to
+reconcile them was to invent a launch between the two.**
+
+Every fact in my message was accurate. It was unreadable *in sequence*. Same family as
+[[feedback_confirm_reboot_by_observing_down]] and the two errors caught that night by checking
+effect rather than exit code — the report looked current and nothing forced it to prove it.
+
+**Put the wall clock on every state report.** Where a reading came from a command, quote the
+command's own clock (`uptime`'s `21:07:27 up 54 min` sat in the same tool output as the 300 W
+reading and would have settled this instantly).
+
+## Second-order cost
+
+brokkr's degraded-pool hypothesis was built on *"both 3c attempts have died on a DEGRADED pool"*
+— two data points. There was one, and it has a sufficient cause that is not the pool. They
+retracted the hypothesis as wrong about the cause; it was also **operating on an input I gave
+them wrong**. A bad timestamp does not stay in your own artifact.
+
+Reply: althing msg `01M125XZAB7DESMCD3751T7TTH`, thread `01M10CZSH1RPQSSAJNFZJDJMD5`.

persistent-memory.md

@@ -1,6 +1,6 @@
 # Persistent memory — eshpfi-management
 
-_Last updated: 2026-08-26_
+_Last updated: 2026-08-27_
 
 > **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its
 > `Written:` stamp is under an hour old, read it (it carries the in-flight
@@ -108,21 +108,28 @@ no longer deployed sidecars here. See Recent decisions.)
   (no NOPASSWD)** — stage model pulls to `/home`, not root-owned `/worktank`.
 ## Current state / in-flight
 
-_As of 2026-08-26 ~09:45 PDT — **run 2 is done, gated (FAIL), and serving. Run 3's corpus is BUILT and HELD: brokkr is redoing the recipe after a corpus-containment defect we found together.** Nothing is training._
+_State as left 2026-08-26 23:05 PDT (written 2026-08-27 07:27, corrected 10:5x) — **run 3 is trained, gated and DO-NOT-SERVE on a safety finding. Run 3c is STOPPED at last-logged step 24/604, killed deliberately at 2026-08-26 21:07:40 PDT after Anaheim tripped a power breaker.** Nothing is training._
 
-- **🔴 RUN 2 GATE: FAIL, recorded as FAIL.** T3 constraint **−12.0** and T4 −5.5 against a ~1 pt floor (both tuned passes read 88 exactly — not variance). **But gate 1 is the result: T6 spatial +15.0, where run 1 FAILED the same axis at −3.5**, base swap the only intended variable. Neither run ships; together they price what the abliteration cost. Diversity **+0.1934 at 23.6x sd** over eight independent blocks — the most robust number produced, survived attrition-matching and the opposite-direction length argument. Memorisation none. Full write-up: brokkr `4973991`, `research/R47-premium-corpus-gate/run02-gate/RESULT-run02-gate.md`. → `persistent-memory.d/2026-08-26-erp-run2-complete-and-served.md`
-- **⚠ THE "INSTABILITY" WAS NEVER INSTABILITY — I published this wrong TWICE and corrected it.** All 46 flags were **`too_short`** rp turns of 3-14 words; the two collapse guards (`repeated_trigrams`, `non_latin`) fired **zero** times on any run. The model never collapsed. It is the **left tail of the length distribution measured in the same message** — thresholds calibrated on the base's output shape applied to a model with a different shape. It was also **not new**: run 1's record carried it, so it is a property of the RECIPE, not the base swap — **a third run that changes the base again will not fix it.** Playbook §4.6.3 carries the corrected version with the retraction visible.
-- **🟢 `erp-tune-v2` SERVING + ON THE GATEWAY** (`10.250.50.70:4000` and direct `:8098`, GPU0, shared all-agents key works). Operator's explicit request so he can evaluate it by hand — **overrides my not-in-the-gateway recommendation.** The config entry carries the failed-gate table, truncation/degeneracy rates and rp-length caveat IN-LINE. **`erp-tune-v1` DELETED from the config** in the same reload: clean **400** now, not the 500 it had been throwing. ⚠ **Never repoint v1 at v2's weights to silence an error.**
-- **⏸ RUN 3: CORPUS BUILT, LAUNCH HELD — brokkr is redoing the recipe.** Blocker found before any GPU spend: **`creative-writing-multiturn` is a declared MEGAMIX containing bluemoon, PIPPA, LimaRP and stheno**, and the remix promoted creative-writing *and* bluemoon — the two roots that overlap. Median bluemoon↔creative-writing jaccard **0.873**; containment, not overlap. Dedup direction **reversed** (keep the primary source, drop the megamix copy) → bluemoon 67→126 convs and **38.6% of loss signal, the largest contributor**, wholly-human share UP and megamix share DOWN, context unchanged at 12.49M so the operator's settled mix survives. ⚠ **F1 "excise PIPPA" removes the ROOT not the MATERIAL** — F2's 250-word floor does that work, since PIPPA turns cannot exceed 123 words wherever they live. → `persistent-memory.d/2026-08-26-run3-corpus-and-the-megamix-containment.md`
-- **⚠ CATALOG-LEVEL, UNSOLVED: LimaRP and stheno are still unchecked against anything.** A megamix root silently contains other catalog roots and the mix arithmetic does not know. Wants a `contains_datasets:` Hoard field + a preflight assertion that no recipe includes both a megamix and a root it contains. brokkr raising it as its own item.
-- **⏳ DPO BLOCKED on an operator decision: which refusal axes are pruned vs explicitly kept.** `docs/pfi/erp-dpo-stage-prep.md`. No preference data for refusal axes exists; `trl` is not installed; the Gutenberg sets on disk are prose-quality only. ⚠ **`under_floor_rate_run02 = 11.78% [9.49, 14.07] @ floor 15, corpus = run-02 mix` is bound to run 2's mix and is obsolete the day the remix is built.** ⚠ The rp length distribution is **bimodal** — pairs sampled from it inherit the mixture, not a mean.
-- **⚠ HARNESS COMMITS LOCAL AND UNPUSHED** on ana-ml2 `/tank/erp-tune/eitri-smithy` (`5349ef0` → `9d27b4f`, 242 tests green). Push is the operator's call.
-- **🟢 SEATS.** GPU0: `erp-tune-v2` (:8098, ~13 GB spare) — **operator granted discretion to take it down; I left it UP** because the card is not needed until a recipe is settled. GPU1: `gen` (:8015), scriberr, rerank-a3, coder, reward, embed. **`char-rp` and `sec`/mog-sec still DOWN** from the training window.
-- **⏳ OPEN ELSEWHERE (unchanged):** Worldtree #411 orphan cleanup + providers.yaml parity + `:latest` tag cleanup (selene fix `a77639d` committed NOT deployed); synapse stack not mirrored into `stacks/`; MTP-k3 isolating experiment; upstream vLLM issue to file; Cold-Fusion NVFP4 quants delete/keep; `/tank` DEGRADED **70+ days**; Lobe retirement.
-- **⚠️ STANDING: NO FLEET NOTIFICATIONS unless the operator asks** (2026-08-24).
+- **⏸ RUN 3c HELD — operator stopped it, power capacity is the blocker.** lr `2e-4 -> 1e-5`, corpus BYTE-IDENTICAL, single variable proven by diff. Config `/tank/erp-tune/run-03c.json` is built and validated (`save_steps 50`, both deviations recorded as separate entries — the scientific lr change and the operational checkpoint cadence). Relaunch is one command. **Do not relaunch until the power triage lands** — ana-ml2 pulls ~600 W across both GPUs at their caps while training, and that is what tripped the breaker. **Exactly TWO 3c launches, and only one of them died:** #1 17:53:33 → killed by the power loss at step 80/604; #2 20:58:41 → stopped BY ME at 21:07:40, healthy, on operator instruction. → `persistent-memory.d/2026-08-27-run3c-launch-count-reconstruction.md`
+- **🔴 `/tank` DEGRADED on ana-ml2 — a disk is genuinely gone**, 7 physical NVMe where the pool expects 8. raidz2, one parity disk spent, no data errors. **Operator is replacing it** (Supermicro AS-4125GS-TNRT2, PCIe hot-plug, should not need a power-down). → `persistent-memory.d/2026-08-27-anaheim-breaker-and-onboot-gap.md`
+- **🟢 ADAPTERS BACKED UP OFF-SITE** — `nh3-nas:/volume1/smithy/erp-tune-adapter-backup/{run-01,run-02,run-03}`, sha256 verified at source, staging and rest (944 MB). They were single-copy mode-600 on the degraded pool. The four `merged-run03*` models are NOT backed up **by design** — all are derived from `run-03/adapter` by a documented verified merge, ~10 min each to regenerate from a 315 MB artifact that is now safe.
+- **⏸ WEEKEND: power triage, "probably shut down some seats"** (operator). `gen` stays up by instruction. Everything else on ana-ml2 is idle. Sheddable: `vllm-gen` 46 GB (the big one), embed 9.8 GB, coder 8.4 GB, rerank-a3 3.5 GB, reward 2.1 GB, scriberr.
+- **⏸ Owed to brokkr-smithy-dev when there is a card again:** nothing blocking. They hold the dose curve, the entanglement finding, and an unsourced "~60%" figure in R47 §5 they were checking against MeroMero v2's card — if it is not there either, §5's recommendation loses its basis.
+- ⚠ **`/mnt/smithy` will be MISSING after every ana-ml2 reboot** — manual by design, not an oversight. Remount spec in `persistent-memory.d/2026-08-23-smithy-mount-ana-ml2.md`. Do NOT add it to fstab.
 
 ## Recent decisions
 
+- `[2026-08-27]` **Run 3 gated: the preregistered rule PASSED and a k=25 follow-up found a 44pp self-harm guardrail collapse — DO NOT SERVE.** A pooled preserve-list test structurally cannot see a single-axis collapse. → `persistent-memory.d/2026-08-27-run3-gate-safety-regression.md`
+- `[2026-08-27]` **The corpus mix was specified in a unit the optimiser never sees** — 45.8% dialogue by CONTEXT, 24.2% by LOSS. Harness now leads with loss share and calls context a memory budget (`dd5a12e`). → `persistent-memory.d/2026-08-27-mix-specified-in-the-wrong-unit.md`
+- `[2026-08-27]` **Dose-response: benefit and damage are ONE direction in weight space** — every axis monotone in scale, no knee. The merge-back cannot separate them; vLLM cannot LoRA-serve this MoE at all. → `persistent-memory.d/2026-08-27-dose-response-entanglement.md`
+- `[2026-08-27]` **Anaheim tripped a power breaker; four guests including the NAS had `onboot` unset and never came back.** Fixed with dependency ordering — ana-nas order=1,up=45 ahead of the databases. ⚠ **ONE CIRCUIT FEEDS THE WHOLE RACK including the firewall serving the public IP** (operator) — so ana-gw, ana-wg and every BMC go down with the load, and there is NO remote management path to Anaheim during a power event. → `persistent-memory.d/2026-08-27-anaheim-breaker-and-onboot-gap.md`
+- `[2026-08-27]` **A transport failure that enters a measurement as a VALUE looks like whatever you hoped to find.** heid's lost panel arms found a live defect in brokkr's `t4_dissect` an hour later. → `persistent-memory.d/2026-08-27-empty-response-as-a-datum.md`
+- `[2026-08-27]` **Run 3c authorised (lr 20x cut, single variable) and then HELD by the operator after the breaker trip.** Config built and validated at `/tank/erp-tune/run-03c.json`; `save_steps` made configurable in the harness (`0a6bd2e`) because the first launch lost 80 steps with no checkpoint. Tracking surface: commit `0a6bd2e` + that config path. **Relaunch is one command once power is triaged.**
+- `[2026-08-27]` **An event report with no timestamp is a claim about "now" — and it manufactured a launch that never happened.** brokkr reconstructed a phantom third 3c launch because my 23:03 report narrated a 21:07 kill in the present tense. Every fact in it was true; it was unreadable in sequence. → `persistent-memory.d/2026-08-27-run3c-launch-count-reconstruction.md`
+- `[2026-08-27]` **`save_steps` was hardcoded at 100 in the harness** — a claimed provenance entry the run could not have honoured. Made configurable, default unchanged (`0a6bd2e`, 242 tests green). Caught by checking the config carried the change rather than trusting that it had been made.
+- `[2026-08-27]` **Six defects in run 3's staged build, none of which would have errored** — a dialogue-only survivor list that would have silently dropped 96% of the corpus, an impersonation mask not subsumed by the low-quality mask, kvasir unbounded at 67.8% of context, a `save_pretrained` config-key drop that made the merged model unservable, and the mix-unit error. Every one produced a plausible completed run. Full record `/tank/erp-tune/recipe-r3/RUN-03-BUILD-NOTE.md`.
+- `[2026-08-27]` **The 18 unpushed eitri-smithy commits are pushed** — run 3's `harness_commit 9d27b4fe` now resolves off-box, verified by fetching into a fresh empty repo rather than trusting the push output. ⚠ **HTTPS push 403s for every gitea token including site-admin; SSH works.** Untracked `__pycache__` (`894fbe8`) because a tracked `.pyc` dirtied the tree and would have stamped `harness_dirty_at_launch: true`.
+
 - `[2026-08-26]` **Run 3's corpus is built and HELD — `creative-writing-multiturn` is a MEGAMIX containing bluemoon, PIPPA, LimaRP and stheno**, and the remix promoted two roots that overlap at median jaccard 0.873. Dedup direction reversed to keep the primary source. F1 does not do what the recipe says; F2 does. → `persistent-memory.d/2026-08-26-run3-corpus-and-the-megamix-containment.md`
 - `[2026-08-26]` **No conversation admission threshold for run 3** (brokkr delegated, then endorsed). Masked context costs the transformer body but NOT the LM head — the harness drops `IGNORE_INDEX` before the 262k-vocab head — so a 20% trim buys ~35 min against a 3-hour run, and it would be a fifth filter confounding a run whose purpose is testing the filters alone. Tracked in the run-3 detail file.
 
@@ -232,30 +239,22 @@ _As of 2026-08-26 ~09:45 PDT — **run 2 is done, gated (FAIL), and serving. Run
 
 - `[2026-08-15]` **Uncensored gen seat: JonathanColetti/Qwen3.8-27B-Uncensored deployed as `gen-seat`/`vllm-gen` (NVFP4 W4A16 + grafted MTP, 262K); 7 aliases repointed; the definitive `re:^mtp.*`-ignore fix.** 0%-MTP-on-quant (twice) was NOT the abliteration/scheme — the grafted bf16 MTP was missing from `quantization_config.ignore` (vLLM loaded it as quantized → uninitialized). Full arc, the working pipeline, VRAM budget, unsloth speed decomposition, modelopt dead-end. → `persistent-memory.d/2026-08-15-uncensored-gen-seat.md`
 
-- `[2026-08-12]` **eRP dual-seat overhaul: MeroMero-v2 (`char-rp`) + Dark-Scarlett (`char-rp-reasoning`), both NVFP4A16 @ 256K on ana-ml2; granite retired.** Replaced the GGUF/heretic2 RP seats with two home-quantized vLLM seats. The DS blocker (an `AutoModelForCausalLM` save wrote a flat `Qwen3_5TextConfig` that **both vLLM AND SGLang reject**) was fixed by re-quanting via the `Qwen3_5ForConditionalGeneration` **wrapper class**; ModelOpt was a version deadlock, SGLang lacked the impl (but revealed the fix). MeroMero vision reconstructed by extracting `preprocessor_config.json` from `processor_config.json`. Both models KV-efficient (Gemma-4 sliding-window / Qwen3.6 hybrid linear-attn) → full 256K; GPU-swapped for headroom; compose-ified + committed `f08b6cb`. granite downed + LiteLLM `summarizer`/`classifier`→gen. Full arc, lessons, dead-ends → `persistent-memory.d/2026-08-12-erp-dual-seat-overhaul.md`
 
 
-- `[2026-08-12]` **infra-ops now holds an all-zones Cloudflare DNS-edit token (vaulted) + wgtunnel Phase-0 DNS landed.** Operator handed over a `Zone·DNS·Edit` (all zones) CF token → `secret put nh3-dev/.config/cloudflare/infra-ops-dns-token` (round-trip verified; /tmp drop shredded). Fleet DNS is now self-serve for infra-ops (⚠ HIGH blast radius — all zones). First use: created `boring.phasefinal.com` CNAME → `ana-srv1.phasefinal.com`, **DNS-only** (proxied:false), verified resolving to 38.120.12.44 on both authoritative NS (louis/wren) + 1.1.1.1 — NOT Cloudflare-proxied. Unblocks wgtunnel's wstunnel ACME cert. phasefinal.com zone id `f812ba74ed9a75cf21bbe7ce9188db50`. auto-memory `reference_infra_ops_cloudflare_dns_token`. (Earlier gap: the only prior vaulted CF token, jackdaw's, had `zone:read`+`worker:edit` but no `dns_records:edit`.)
 
 
-- `[2026-08-12]` **wgtunnel stood up as its own repo (`vh/wgtunnel`, private) after a live endpoint-verification pass.** Operator directed own-repo (mirrors stonehenge-park/tts-stack). Verified off the fleet before seeding: `ana-wg` WG server = **UDP/31337** (not 51820), subnet 10.30.10.0/24, MTU 1420, active roaming peer proves the public UDP DNAT works; traefik on ana-docker **terminates TLS :443** (ACME `anaprod` http-challenge, docker+file providers, CrowdSec bouncer) → confirms the clean design (wstunnel container on `traefik-net`, Host-routed, WS→UDP to `ana-wg:31337`); edge `38.120.12.44` direct-A, `tunnel.phasefinal.com` free (⚠ must be **direct**, NOT Cloudflare-proxied like vaultwarden). Repo pre-seeded (README/CLAUDE/persistent-memory/ROADMAP + `docs/verified-infrastructure.md` = ground truth) + pushed; commit `9584d38`, Vuong-attributed. vh gitea token pulled from the vault (`secret get`), not persisted to `.git/config`. **NEXT = `/vor-plan` or `/vor` (operator's call, interactive).** Deps to line up in the plan: DNS A-record, FortiGate :443 host-routing, a new ana-wg peer for the laptop, client tooling.
 
 - `[2026-08-10→12]` **secrets-broker: per-box Vaultwarden credential store SHIPPED + consumer-confirmed.** `secret` CLI (`put/get/list/rm/backfill`, bw-backed) on `~/.local/bin`; 25 nh3-dev secrets backfilled + round-trip-verified; `rm` + new-namespace warning added post-launch; standing "vault is the credential source of truth" directive now global. → `persistent-memory.d/2026-08-12-secrets-broker.md`
 
 
-- `[2026-08-11]` **stonehenge-park: new fleet `/park` service repo stood up + designed (`/vor-plan` + `/vor-ui`).** Self-contained SQLite+FastAPI idea-parking service that actively resurfaces (statusline + althing) so nothing dies in a cold repo; `vh/stonehenge-park` pushed + pre-seeded for a fresh agent; build starts at the U1 tracer contract. → `persistent-memory.d/2026-08-11-stonehenge-park.md`
 
 
-- `[2026-08-12]` **Global `~/.claude/CLAUDE.md`: `secret`/vault tool entry + "store in AND pull from the vault" standing directive** (dotfiles `9db703b`, pushed); statusline reset-countdowns + a latent tab-collapse parse-bug fix, now tracked in the dotfiles stow tree. Dogfooded the directive: created `vh/stonehenge-park` pulling the gitea token via `secret get`. (dotfiles + global config, not eshpfi.)
 
 
-- `[2026-08-11]` **TTS stack extracted to its own repo (`tts-stack`) + eshpfi stood down on TTS dev.** Operator: hand all TTS tuning/dev to a separate agent with a self-contained repo (knowledge + infra access + a live knowledge list), and move the voice corpus in. New repo `~/development/tts-stack` (commit `9ee3288`) carries: dots-tts stack (canonical intent), `voices/` corpus (MOVED out of eshpfi), `KNOWLEDGE.md` (engine landscape + prosody findings + foot-guns), `docs/infrastructure.md` (irv-ml1 access + gated deploy runbook + rollback), CLAUDE/persistent-memory/ROADMAP, `tools/` (pause-probe + Booth render). Followed the **chatterbox-fast precedent**: eshpfi `stacks/dots-tts/` reduced to a POINTER README; the ~15 experimental TTS compose wrappers stay here as reference (catalogued in tts-stack KNOWLEDGE). Blast-radius check: no eshpfi playbook/script reads the canonical corpus (other `voices/` refs = unrelated host paths). **Reverses** the earlier "Corpus home = eshpfi `voices/` (keep-here)" call. ⚠ tts-stack is LOCAL-ONLY until pushed — needs a gitea remote (`vh/tts-stack`) + push before the separate agent can clone (operator's call — outward-facing + repo-create creds).
 
 
-- `[2026-08-10]` **dots-tts v3 — clause-break → period pause mapping.** Operator: v2 "sounds good" but donut won't pause at semicolons/dashes. ROOT CAUSE (measured via a pause-probe A/B — synth duration over N runs, non-determinism averaged out): dots' prosody honors a real pause **only for ellipsis (~+0.43s) and period (~+0.3s, capitalization-independent)**; comma/semicolon/colon/dash all run **flat (~+0.03s vs no-punct)**. Two distinct sub-causes: **dashes regressed in v2** (the `—`→`-` fold made em-dashes read as word-joiners), while **semicolons were NEVER a v2 change** — dots ignores them natively, only newly noticeable because v2 made everything else clean. Operator call: ellipsis "too much" → **map `;`, clause `:`, and em-dash `—` → period** in `_sanitize` (believable ~0.3s clause break). GUARDS (pinned by 11 unit tests, `stacks/dots-tts/test_sanitize.py`): digit-guarded colon `(?<!\d)\s*:\s*(?!\d)` so times `3:45` / ratios `2:1` survive; en-dash `–`→hyphen KEPT (numeric-range `10–20` safety — em-dash breaks, en-dash ranges, different jobs); genuine ellipsis left at full strength (author meant a long pause). Gated deploy (redeploy2 pattern → v3): build → throwaway :8199 test container + **pause-gate** (semicolon sentence must run ≥0.12s longer than baseline; measured **+0.427s**) → only then cut live over. LIVE + healthy `local/dots-tts:v3` on :8198. **rollback = `sed -i 's/^DOTS_TAG=.*/DOTS_TAG=v2/' .env + docker compose up -d dots-tts`** (v2 image retained). Booth `dots-pauses` (A=old-flat / C=ellipsis-too-much / D=live-v3). [[reference_chatterbox_fast_repo]]
 
 
-- `[2026-08-10]` **dots-tts v2 — contraction fix (curly-sanitize) + sentence-chunking + dependency-pin recovery.** Operator: donut read contractions wrong ("you're"→"you ree", "donut's"→"donut ess"). ROOT CAUSE (isolated via A/B booth): **curly/typographic apostrophes** (`’` U+2019 from ratatoskr's LLM) — dots' tokenizer mispronounces them; STRAIGHT apostrophes read clean under `normalize_text=True`. FIX (`app.py`): fold curly→ASCII (`str.maketrans`) before synth, **KEEP `normalize_text=True`** (operator call — retains number/date expansion). Also added **server-side sentence-chunking** (pack ≤280 chars): dots caps one `generate()` at ~500 patches/~40s, so long RP turns (the Zev monologue = 160s audio) truncated; chunking stitches them (verified full 160.3s, not 40s-cut). **⚠ BUILD FOOT-GUNS (both bit this redeploy):** (1) upstream dots.tts `constraints/recommended.txt` now pins **`gradio==6.17.0` — phantom, not on PyPI** → fresh `pip install dots.tts` unsatisfiable; FIX = pin `dots.tts==0.2.1` + **DROP** the `-c recommended.txt` constraints (0.2.1 pulls working gradio 6.17.3). (2) pinning only `torch==2.8.0` let **torchaudio float to 2.11.0 → dots.tts refuses to load** (minor-version match check); FIX = pin `torchaudio==2.8.0`. **⚠ DEPLOY LESSON:** `docker compose up -d` to a new tag swaps the LIVE container BEFORE any health check — a broken image crash-loops production (**ratatoskr TTS down ~1-2min this session**). NEW PATTERN = build → test in a THROWAWAY container on an alt port (:8199) → health+verify → only THEN cut live over (redeploy2.sh). v2 LIVE + healthy on irv-ml1:8198, **CONSUMER-CONFIRMED clean** (ratatoskr verified end-to-end on their :8765 — apostrophe string reads clean, /api/tts 200 @ 48kHz, no client change; the ~1-2min blip didn't hit them, their concurrent auto-audio issue was client-side localStorage). **rollback = `sed DOTS_TAG=v1 + docker compose up -d dots-tts`** (v1 image retained). Also: deployed container GPU crept ~6→13.9GB over 8h serving (cache accumulation; a redeploy resets it — watch item). [[reference_chatterbox_fast_repo]]
 
 - `[2026-08-09→10]` **dots.tts (rednote-hilab) TTS burn-in on irv-ml1 + canonical voice corpus built (`voices/`).** Operator-directed eval to potentially replace chatterbox-fast. **dots.tts VERIFIED real** (canonical HF ns `dots-studio/`, `rednote-hilab/dots.tts-*` redirects there; Apache-2.0; PyPI `dots.tts` 0.2.1; 2B continuous-AR = semantic enc + Qwen2.5-1.5B LLM + flow-matching acoustic head over 48kHz AudioVAE; zero-shot clone from wav+transcript). **Runs on Ampere 3090** (sm_86, bf16, no fp8 dep); **optimized RTF 0.22** at num_steps=10 (`from_pretrained(..., optimize=True)` CUDA graphs — raw unoptimized was 1.21), **~6GB VRAM**, 48kHz, streams (`generate_stream`). Venv+cache at `irv-ml1:/home/lkraven/dots-tts` (~10GB). **Operator design calls:** SGLang Omni serving (OpenAI `/v1/audio/speech`), transcribe-refs-first, `soar` variant. ⚠ Omni serves soar but its continuous-batching + streaming opts are **mf-only** (soar = single-request) — non-issue for ratatoskr's single-consumer RP surface. **KEY FINDING — dots is highly sensitive to an accurate AND sentence-bounded reference transcript:** mismatched transcript → 0.16s collapse; over-long/messy transcript → reference-audio BLEEDS as an output prefix; mid-clause trim → dangling-word leak (glados "we'll", emmie "And,"). RECIPE (baked into `voices/derive.py`): trim ref to a clean ~6–10s clip ending on a sentence boundary + accurate transcript of exactly that clip. **CANONICAL VOICE CORPUS** stood up in eshpfi `voices/` (operator idea): engine-agnostic `canonical/<v>.wav` + `transcripts/<v>.txt` → per-engine ref sets DERIVED by `derive.py` reading `engines.yaml` profiles (dots/chatterbox/zonos); canonical wavs git-tracked (small/curated), `derived/` gitignored. **4 voices optimized + verified CLEAN for dots: donut, glados, emmie, miranda** (glados canonical is low-SR 16kHz — flagged upgrade candidate). ⚠ GPU GOTCHA: irv-ml1 native CUDA orders **A6000=device0** (ComfyUI-full) — pin the 3090 with `CUDA_DEVICE_ORDER=PCI_BUS_ID CUDA_VISIBLE_DEVICES=0`; and `PYTORCH_CUDA_ALLOC_CONF=expandable_segments` CONFLICTS with `optimize=True` CUDA graphs (curr_block error). Booths: `dots-vs-chatterbox`, `dots-voices-optimized`. **SHIPPED 2026-08-10:** operator A/B verdict "dots is very good" → containerized as a **thin FastAPI wrapper over DotsTtsRuntime** (chosen over SGLang Omni — Omni's batching is mf-only, unneeded for ratatoskr's single consumer; wrapper is SERIALIZED one-gen-at-a-time via a threading.Lock, Omni+mf = parked API-compatible escalation if multi-consumer ever lands). **LIVE on irv-ml1:8198** (`local/dots-tts:v1`, OpenAI `/v1/audio/speech` + `/health` + `/v1/voices`, container healthy, both stream + non-stream verified CLEAN, 4 voices donut/glados/emmie/miranda) alongside chatterbox :8197 (nothing repointed). Stack = `stacks/dots-tts/` (Dockerfile/app.py/compose/.env.example/README). ⚠ CONTAINER GOTCHA: `optimize=True` (torch.compile/inductor/triton) needs a **C compiler at RUNTIME** — slim image must `apt install build-essential` or model-load dies "Failed to find C compiler" (host venv had gcc ambient, masking it); persist `TORCHINDUCTOR_CACHE_DIR` to a mounted dir or every restart re-JITs ~5min. Corpus home = eshpfi `voices/` (operator ruled keep-here). **REMAINING: ratatoskr client cutover** to :8198 `/v1/audio/speech` (Phase-2 tail, peer-coupled — draft the ask). [[reference_chatterbox_fast_repo]] [[reference_zonos_tts_stack]] [[reference_verify_hf_repo_ids_before_pull]]
 
@@ -273,7 +272,7 @@ _As of 2026-08-26 ~09:45 PDT — **run 2 is done, gated (FAIL), and serving. Run
 
 
 
-_214 older entries archived to archival-memory.md._
+_222 older entries archived to archival-memory.md._
 ## Tried and abandoned
 
 - `[2026-08-25]` **Four throughput levers measured and killed — do not re-chase.** (1) **Fused MoE / `grouped_mm`** — 0.9% *slower* than the Python loop and dense GEMM is only 7.9% of the step, capping the whole category near 10%. (2) **CUDA graphs / `torch.compile` over the expert loop** — the two-term scaling fit closed with residuals under 3ms and needed NO constant term, so there is no fixed per-batch cost to amortise; 3,840 expert-GEMM launches per forward are not what we pay for. (3) **`liger` fused linear CE** — the chunked CE measured **1.1% of the step** forward, ~3% with recompute. A tidy-up, not a lever. (4) **Selective gradient checkpointing** — ~2% of a post-fix step, real bug surface. Also: **token-budget batching is dead by the same fit** — with no constant term, total time over a fixed set of widths is invariant to how you group them; only the widths matter, which is exactly why bucketing works and repacking does not.