memory: snapshot — run 3 gated DO-NOT-SERVE, run 3c held on a tripped breaker

Run 3 trained, gated and dispositioned do-not-serve on a measured 44pp self-harm
guardrail regression that its own preregistered rule passed -- a pooled preserve-list
test cannot see a single-axis collapse. Run 3c (lr 20x cut, single variable) launched,
killed by an Anaheim power-breaker trip at step 80, relaunched, then stopped by the
operator at step 22 pending a weekend power triage.

Also captured: the corpus mix was specified in a unit the optimiser never sees
(45.8% dialogue by context, 24.2% by loss); the dose-response says benefit and damage
are one direction in weight space, so the merge-back measures the problem rather than
fixing it; four guests including the storage SPOF had onboot unset and never came back
from the outage, now fixed with dependency ordering; and a transport failure that
enters a measurement as a value looks like whatever you hoped to find -- which found a
live defect in another agent's instrument an hour after it was reported.

Auto-archived 8 entries to archival-memory.md (Recent decisions: 8, Tried and
abandoned: 0); 4 held back on open deferred-work pointers.

persistent-memory.md

@@ -1,6 +1,6 @@
 # Persistent memory — eshpfi-management
 
-_Last updated: 2026-08-26_
+_Last updated: 2026-08-27_
 
 > **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its
 > `Written:` stamp is under an hour old, read it (it carries the in-flight
@@ -108,21 +108,27 @@ no longer deployed sidecars here. See Recent decisions.)
   (no NOPASSWD)** — stage model pulls to `/home`, not root-owned `/worktank`.
 ## Current state / in-flight
 
-_As of 2026-08-26 ~09:45 PDT — **run 2 is done, gated (FAIL), and serving. Run 3's corpus is BUILT and HELD: brokkr is redoing the recipe after a corpus-containment defect we found together.** Nothing is training._
+_As of 2026-08-27 ~21:30 PDT — **run 3 is trained, gated and DO-NOT-SERVE on a safety finding. Run 3c is STOPPED at step 22/604 after Anaheim tripped a power breaker.** Nothing is training._
 
-- **🔴 RUN 2 GATE: FAIL, recorded as FAIL.** T3 constraint **−12.0** and T4 −5.5 against a ~1 pt floor (both tuned passes read 88 exactly — not variance). **But gate 1 is the result: T6 spatial +15.0, where run 1 FAILED the same axis at −3.5**, base swap the only intended variable. Neither run ships; together they price what the abliteration cost. Diversity **+0.1934 at 23.6x sd** over eight independent blocks — the most robust number produced, survived attrition-matching and the opposite-direction length argument. Memorisation none. Full write-up: brokkr `4973991`, `research/R47-premium-corpus-gate/run02-gate/RESULT-run02-gate.md`. → `persistent-memory.d/2026-08-26-erp-run2-complete-and-served.md`
-- **⚠ THE "INSTABILITY" WAS NEVER INSTABILITY — I published this wrong TWICE and corrected it.** All 46 flags were **`too_short`** rp turns of 3-14 words; the two collapse guards (`repeated_trigrams`, `non_latin`) fired **zero** times on any run. The model never collapsed. It is the **left tail of the length distribution measured in the same message** — thresholds calibrated on the base's output shape applied to a model with a different shape. It was also **not new**: run 1's record carried it, so it is a property of the RECIPE, not the base swap — **a third run that changes the base again will not fix it.** Playbook §4.6.3 carries the corrected version with the retraction visible.
-- **🟢 `erp-tune-v2` SERVING + ON THE GATEWAY** (`10.250.50.70:4000` and direct `:8098`, GPU0, shared all-agents key works). Operator's explicit request so he can evaluate it by hand — **overrides my not-in-the-gateway recommendation.** The config entry carries the failed-gate table, truncation/degeneracy rates and rp-length caveat IN-LINE. **`erp-tune-v1` DELETED from the config** in the same reload: clean **400** now, not the 500 it had been throwing. ⚠ **Never repoint v1 at v2's weights to silence an error.**
-- **⏸ RUN 3: CORPUS BUILT, LAUNCH HELD — brokkr is redoing the recipe.** Blocker found before any GPU spend: **`creative-writing-multiturn` is a declared MEGAMIX containing bluemoon, PIPPA, LimaRP and stheno**, and the remix promoted creative-writing *and* bluemoon — the two roots that overlap. Median bluemoon↔creative-writing jaccard **0.873**; containment, not overlap. Dedup direction **reversed** (keep the primary source, drop the megamix copy) → bluemoon 67→126 convs and **38.6% of loss signal, the largest contributor**, wholly-human share UP and megamix share DOWN, context unchanged at 12.49M so the operator's settled mix survives. ⚠ **F1 "excise PIPPA" removes the ROOT not the MATERIAL** — F2's 250-word floor does that work, since PIPPA turns cannot exceed 123 words wherever they live. → `persistent-memory.d/2026-08-26-run3-corpus-and-the-megamix-containment.md`
-- **⚠ CATALOG-LEVEL, UNSOLVED: LimaRP and stheno are still unchecked against anything.** A megamix root silently contains other catalog roots and the mix arithmetic does not know. Wants a `contains_datasets:` Hoard field + a preflight assertion that no recipe includes both a megamix and a root it contains. brokkr raising it as its own item.
-- **⏳ DPO BLOCKED on an operator decision: which refusal axes are pruned vs explicitly kept.** `docs/pfi/erp-dpo-stage-prep.md`. No preference data for refusal axes exists; `trl` is not installed; the Gutenberg sets on disk are prose-quality only. ⚠ **`under_floor_rate_run02 = 11.78% [9.49, 14.07] @ floor 15, corpus = run-02 mix` is bound to run 2's mix and is obsolete the day the remix is built.** ⚠ The rp length distribution is **bimodal** — pairs sampled from it inherit the mixture, not a mean.
-- **⚠ HARNESS COMMITS LOCAL AND UNPUSHED** on ana-ml2 `/tank/erp-tune/eitri-smithy` (`5349ef0` → `9d27b4f`, 242 tests green). Push is the operator's call.
-- **🟢 SEATS.** GPU0: `erp-tune-v2` (:8098, ~13 GB spare) — **operator granted discretion to take it down; I left it UP** because the card is not needed until a recipe is settled. GPU1: `gen` (:8015), scriberr, rerank-a3, coder, reward, embed. **`char-rp` and `sec`/mog-sec still DOWN** from the training window.
-- **⏳ OPEN ELSEWHERE (unchanged):** Worldtree #411 orphan cleanup + providers.yaml parity + `:latest` tag cleanup (selene fix `a77639d` committed NOT deployed); synapse stack not mirrored into `stacks/`; MTP-k3 isolating experiment; upstream vLLM issue to file; Cold-Fusion NVFP4 quants delete/keep; `/tank` DEGRADED **70+ days**; Lobe retirement.
-- **⚠️ STANDING: NO FLEET NOTIFICATIONS unless the operator asks** (2026-08-24).
+- **⏸ RUN 3c HELD — operator stopped it, power capacity is the blocker.** lr `2e-4 -> 1e-5`, corpus BYTE-IDENTICAL, single variable proven by diff. Config `/tank/erp-tune/run-03c.json` is built and validated (`save_steps 50`, both deviations recorded as separate entries — the scientific lr change and the operational checkpoint cadence). Relaunch is one command. **Do not relaunch until the power triage lands** — ana-ml2 pulls ~600 W across both GPUs at their caps while training, and that is what tripped the breaker.
+- **🔴 `/tank` DEGRADED on ana-ml2 — a disk is genuinely gone**, 7 physical NVMe where the pool expects 8. raidz2, one parity disk spent, no data errors. **Operator is replacing it** (Supermicro AS-4125GS-TNRT2, PCIe hot-plug, should not need a power-down). → `persistent-memory.d/2026-08-27-anaheim-breaker-and-onboot-gap.md`
+- **🟢 ADAPTERS BACKED UP OFF-SITE** — `nh3-nas:/volume1/smithy/erp-tune-adapter-backup/{run-01,run-02,run-03}`, sha256 verified at source, staging and rest (944 MB). They were single-copy mode-600 on the degraded pool. The four `merged-run03*` models are NOT backed up **by design** — all are derived from `run-03/adapter` by a documented verified merge, ~10 min each to regenerate from a 315 MB artifact that is now safe.
+- **⏸ WEEKEND: power triage, "probably shut down some seats"** (operator). `gen` stays up by instruction. Everything else on ana-ml2 is idle. Sheddable: `vllm-gen` 46 GB (the big one), embed 9.8 GB, coder 8.4 GB, rerank-a3 3.5 GB, reward 2.1 GB, scriberr.
+- **⏸ Owed to brokkr-smithy-dev when there is a card again:** nothing blocking. They hold the dose curve, the entanglement finding, and an unsourced "~60%" figure in R47 §5 they were checking against MeroMero v2's card — if it is not there either, §5's recommendation loses its basis.
+- ⚠ **`/mnt/smithy` will be MISSING after every ana-ml2 reboot** — manual by design, not an oversight. Remount spec in `persistent-memory.d/2026-08-23-smithy-mount-ana-ml2.md`. Do NOT add it to fstab.
 
 ## Recent decisions
 
+- `[2026-08-27]` **Run 3 gated: the preregistered rule PASSED and a k=25 follow-up found a 44pp self-harm guardrail collapse — DO NOT SERVE.** A pooled preserve-list test structurally cannot see a single-axis collapse. → `persistent-memory.d/2026-08-27-run3-gate-safety-regression.md`
+- `[2026-08-27]` **The corpus mix was specified in a unit the optimiser never sees** — 45.8% dialogue by CONTEXT, 24.2% by LOSS. Harness now leads with loss share and calls context a memory budget (`dd5a12e`). → `persistent-memory.d/2026-08-27-mix-specified-in-the-wrong-unit.md`
+- `[2026-08-27]` **Dose-response: benefit and damage are ONE direction in weight space** — every axis monotone in scale, no knee. The merge-back cannot separate them; vLLM cannot LoRA-serve this MoE at all. → `persistent-memory.d/2026-08-27-dose-response-entanglement.md`
+- `[2026-08-27]` **Anaheim tripped a power breaker; four guests including the NAS had `onboot` unset and never came back.** Fixed with dependency ordering — ana-nas order=1,up=45 ahead of the databases. → `persistent-memory.d/2026-08-27-anaheim-breaker-and-onboot-gap.md`
+- `[2026-08-27]` **A transport failure that enters a measurement as a VALUE looks like whatever you hoped to find.** heid's lost panel arms found a live defect in brokkr's `t4_dissect` an hour later. → `persistent-memory.d/2026-08-27-empty-response-as-a-datum.md`
+- `[2026-08-27]` **Run 3c authorised (lr 20x cut, single variable) and then HELD by the operator after the breaker trip.** Config built and validated at `/tank/erp-tune/run-03c.json`; `save_steps` made configurable in the harness (`0a6bd2e`) because the first launch lost 80 steps with no checkpoint. Tracking surface: commit `0a6bd2e` + that config path. **Relaunch is one command once power is triaged.**
+- `[2026-08-27]` **`save_steps` was hardcoded at 100 in the harness** — a claimed provenance entry the run could not have honoured. Made configurable, default unchanged (`0a6bd2e`, 242 tests green). Caught by checking the config carried the change rather than trusting that it had been made.
+- `[2026-08-27]` **Six defects in run 3's staged build, none of which would have errored** — a dialogue-only survivor list that would have silently dropped 96% of the corpus, an impersonation mask not subsumed by the low-quality mask, kvasir unbounded at 67.8% of context, a `save_pretrained` config-key drop that made the merged model unservable, and the mix-unit error. Every one produced a plausible completed run. Full record `/tank/erp-tune/recipe-r3/RUN-03-BUILD-NOTE.md`.
+- `[2026-08-27]` **The 18 unpushed eitri-smithy commits are pushed** — run 3's `harness_commit 9d27b4fe` now resolves off-box, verified by fetching into a fresh empty repo rather than trusting the push output. ⚠ **HTTPS push 403s for every gitea token including site-admin; SSH works.** Untracked `__pycache__` (`894fbe8`) because a tracked `.pyc` dirtied the tree and would have stamped `harness_dirty_at_launch: true`.
+
 - `[2026-08-26]` **Run 3's corpus is built and HELD — `creative-writing-multiturn` is a MEGAMIX containing bluemoon, PIPPA, LimaRP and stheno**, and the remix promoted two roots that overlap at median jaccard 0.873. Dedup direction reversed to keep the primary source. F1 does not do what the recipe says; F2 does. → `persistent-memory.d/2026-08-26-run3-corpus-and-the-megamix-containment.md`
 - `[2026-08-26]` **No conversation admission threshold for run 3** (brokkr delegated, then endorsed). Masked context costs the transformer body but NOT the LM head — the harness drops `IGNORE_INDEX` before the 262k-vocab head — so a 20% trim buys ~35 min against a 3-hour run, and it would be a fifth filter confounding a run whose purpose is testing the filters alone. Tracked in the run-3 detail file.
 
@@ -232,30 +238,22 @@ _As of 2026-08-26 ~09:45 PDT — **run 2 is done, gated (FAIL), and serving. Run
 
 - `[2026-08-15]` **Uncensored gen seat: JonathanColetti/Qwen3.8-27B-Uncensored deployed as `gen-seat`/`vllm-gen` (NVFP4 W4A16 + grafted MTP, 262K); 7 aliases repointed; the definitive `re:^mtp.*`-ignore fix.** 0%-MTP-on-quant (twice) was NOT the abliteration/scheme — the grafted bf16 MTP was missing from `quantization_config.ignore` (vLLM loaded it as quantized → uninitialized). Full arc, the working pipeline, VRAM budget, unsloth speed decomposition, modelopt dead-end. → `persistent-memory.d/2026-08-15-uncensored-gen-seat.md`
 
-- `[2026-08-12]` **eRP dual-seat overhaul: MeroMero-v2 (`char-rp`) + Dark-Scarlett (`char-rp-reasoning`), both NVFP4A16 @ 256K on ana-ml2; granite retired.** Replaced the GGUF/heretic2 RP seats with two home-quantized vLLM seats. The DS blocker (an `AutoModelForCausalLM` save wrote a flat `Qwen3_5TextConfig` that **both vLLM AND SGLang reject**) was fixed by re-quanting via the `Qwen3_5ForConditionalGeneration` **wrapper class**; ModelOpt was a version deadlock, SGLang lacked the impl (but revealed the fix). MeroMero vision reconstructed by extracting `preprocessor_config.json` from `processor_config.json`. Both models KV-efficient (Gemma-4 sliding-window / Qwen3.6 hybrid linear-attn) → full 256K; GPU-swapped for headroom; compose-ified + committed `f08b6cb`. granite downed + LiteLLM `summarizer`/`classifier`→gen. Full arc, lessons, dead-ends → `persistent-memory.d/2026-08-12-erp-dual-seat-overhaul.md`
 
 
-- `[2026-08-12]` **infra-ops now holds an all-zones Cloudflare DNS-edit token (vaulted) + wgtunnel Phase-0 DNS landed.** Operator handed over a `Zone·DNS·Edit` (all zones) CF token → `secret put nh3-dev/.config/cloudflare/infra-ops-dns-token` (round-trip verified; /tmp drop shredded). Fleet DNS is now self-serve for infra-ops (⚠ HIGH blast radius — all zones). First use: created `boring.phasefinal.com` CNAME → `ana-srv1.phasefinal.com`, **DNS-only** (proxied:false), verified resolving to 38.120.12.44 on both authoritative NS (louis/wren) + 1.1.1.1 — NOT Cloudflare-proxied. Unblocks wgtunnel's wstunnel ACME cert. phasefinal.com zone id `f812ba74ed9a75cf21bbe7ce9188db50`. auto-memory `reference_infra_ops_cloudflare_dns_token`. (Earlier gap: the only prior vaulted CF token, jackdaw's, had `zone:read`+`worker:edit` but no `dns_records:edit`.)
 
 
-- `[2026-08-12]` **wgtunnel stood up as its own repo (`vh/wgtunnel`, private) after a live endpoint-verification pass.** Operator directed own-repo (mirrors stonehenge-park/tts-stack). Verified off the fleet before seeding: `ana-wg` WG server = **UDP/31337** (not 51820), subnet 10.30.10.0/24, MTU 1420, active roaming peer proves the public UDP DNAT works; traefik on ana-docker **terminates TLS :443** (ACME `anaprod` http-challenge, docker+file providers, CrowdSec bouncer) → confirms the clean design (wstunnel container on `traefik-net`, Host-routed, WS→UDP to `ana-wg:31337`); edge `38.120.12.44` direct-A, `tunnel.phasefinal.com` free (⚠ must be **direct**, NOT Cloudflare-proxied like vaultwarden). Repo pre-seeded (README/CLAUDE/persistent-memory/ROADMAP + `docs/verified-infrastructure.md` = ground truth) + pushed; commit `9584d38`, Vuong-attributed. vh gitea token pulled from the vault (`secret get`), not persisted to `.git/config`. **NEXT = `/vor-plan` or `/vor` (operator's call, interactive).** Deps to line up in the plan: DNS A-record, FortiGate :443 host-routing, a new ana-wg peer for the laptop, client tooling.
 
 - `[2026-08-10→12]` **secrets-broker: per-box Vaultwarden credential store SHIPPED + consumer-confirmed.** `secret` CLI (`put/get/list/rm/backfill`, bw-backed) on `~/.local/bin`; 25 nh3-dev secrets backfilled + round-trip-verified; `rm` + new-namespace warning added post-launch; standing "vault is the credential source of truth" directive now global. → `persistent-memory.d/2026-08-12-secrets-broker.md`
 
 
-- `[2026-08-11]` **stonehenge-park: new fleet `/park` service repo stood up + designed (`/vor-plan` + `/vor-ui`).** Self-contained SQLite+FastAPI idea-parking service that actively resurfaces (statusline + althing) so nothing dies in a cold repo; `vh/stonehenge-park` pushed + pre-seeded for a fresh agent; build starts at the U1 tracer contract. → `persistent-memory.d/2026-08-11-stonehenge-park.md`
 
 
-- `[2026-08-12]` **Global `~/.claude/CLAUDE.md`: `secret`/vault tool entry + "store in AND pull from the vault" standing directive** (dotfiles `9db703b`, pushed); statusline reset-countdowns + a latent tab-collapse parse-bug fix, now tracked in the dotfiles stow tree. Dogfooded the directive: created `vh/stonehenge-park` pulling the gitea token via `secret get`. (dotfiles + global config, not eshpfi.)
 
 
-- `[2026-08-11]` **TTS stack extracted to its own repo (`tts-stack`) + eshpfi stood down on TTS dev.** Operator: hand all TTS tuning/dev to a separate agent with a self-contained repo (knowledge + infra access + a live knowledge list), and move the voice corpus in. New repo `~/development/tts-stack` (commit `9ee3288`) carries: dots-tts stack (canonical intent), `voices/` corpus (MOVED out of eshpfi), `KNOWLEDGE.md` (engine landscape + prosody findings + foot-guns), `docs/infrastructure.md` (irv-ml1 access + gated deploy runbook + rollback), CLAUDE/persistent-memory/ROADMAP, `tools/` (pause-probe + Booth render). Followed the **chatterbox-fast precedent**: eshpfi `stacks/dots-tts/` reduced to a POINTER README; the ~15 experimental TTS compose wrappers stay here as reference (catalogued in tts-stack KNOWLEDGE). Blast-radius check: no eshpfi playbook/script reads the canonical corpus (other `voices/` refs = unrelated host paths). **Reverses** the earlier "Corpus home = eshpfi `voices/` (keep-here)" call. ⚠ tts-stack is LOCAL-ONLY until pushed — needs a gitea remote (`vh/tts-stack`) + push before the separate agent can clone (operator's call — outward-facing + repo-create creds).
 
 
-- `[2026-08-10]` **dots-tts v3 — clause-break → period pause mapping.** Operator: v2 "sounds good" but donut won't pause at semicolons/dashes. ROOT CAUSE (measured via a pause-probe A/B — synth duration over N runs, non-determinism averaged out): dots' prosody honors a real pause **only for ellipsis (~+0.43s) and period (~+0.3s, capitalization-independent)**; comma/semicolon/colon/dash all run **flat (~+0.03s vs no-punct)**. Two distinct sub-causes: **dashes regressed in v2** (the `—`→`-` fold made em-dashes read as word-joiners), while **semicolons were NEVER a v2 change** — dots ignores them natively, only newly noticeable because v2 made everything else clean. Operator call: ellipsis "too much" → **map `;`, clause `:`, and em-dash `—` → period** in `_sanitize` (believable ~0.3s clause break). GUARDS (pinned by 11 unit tests, `stacks/dots-tts/test_sanitize.py`): digit-guarded colon `(?<!\d)\s*:\s*(?!\d)` so times `3:45` / ratios `2:1` survive; en-dash `–`→hyphen KEPT (numeric-range `10–20` safety — em-dash breaks, en-dash ranges, different jobs); genuine ellipsis left at full strength (author meant a long pause). Gated deploy (redeploy2 pattern → v3): build → throwaway :8199 test container + **pause-gate** (semicolon sentence must run ≥0.12s longer than baseline; measured **+0.427s**) → only then cut live over. LIVE + healthy `local/dots-tts:v3` on :8198. **rollback = `sed -i 's/^DOTS_TAG=.*/DOTS_TAG=v2/' .env + docker compose up -d dots-tts`** (v2 image retained). Booth `dots-pauses` (A=old-flat / C=ellipsis-too-much / D=live-v3). [[reference_chatterbox_fast_repo]]
 
 
-- `[2026-08-10]` **dots-tts v2 — contraction fix (curly-sanitize) + sentence-chunking + dependency-pin recovery.** Operator: donut read contractions wrong ("you're"→"you ree", "donut's"→"donut ess"). ROOT CAUSE (isolated via A/B booth): **curly/typographic apostrophes** (`’` U+2019 from ratatoskr's LLM) — dots' tokenizer mispronounces them; STRAIGHT apostrophes read clean under `normalize_text=True`. FIX (`app.py`): fold curly→ASCII (`str.maketrans`) before synth, **KEEP `normalize_text=True`** (operator call — retains number/date expansion). Also added **server-side sentence-chunking** (pack ≤280 chars): dots caps one `generate()` at ~500 patches/~40s, so long RP turns (the Zev monologue = 160s audio) truncated; chunking stitches them (verified full 160.3s, not 40s-cut). **⚠ BUILD FOOT-GUNS (both bit this redeploy):** (1) upstream dots.tts `constraints/recommended.txt` now pins **`gradio==6.17.0` — phantom, not on PyPI** → fresh `pip install dots.tts` unsatisfiable; FIX = pin `dots.tts==0.2.1` + **DROP** the `-c recommended.txt` constraints (0.2.1 pulls working gradio 6.17.3). (2) pinning only `torch==2.8.0` let **torchaudio float to 2.11.0 → dots.tts refuses to load** (minor-version match check); FIX = pin `torchaudio==2.8.0`. **⚠ DEPLOY LESSON:** `docker compose up -d` to a new tag swaps the LIVE container BEFORE any health check — a broken image crash-loops production (**ratatoskr TTS down ~1-2min this session**). NEW PATTERN = build → test in a THROWAWAY container on an alt port (:8199) → health+verify → only THEN cut live over (redeploy2.sh). v2 LIVE + healthy on irv-ml1:8198, **CONSUMER-CONFIRMED clean** (ratatoskr verified end-to-end on their :8765 — apostrophe string reads clean, /api/tts 200 @ 48kHz, no client change; the ~1-2min blip didn't hit them, their concurrent auto-audio issue was client-side localStorage). **rollback = `sed DOTS_TAG=v1 + docker compose up -d dots-tts`** (v1 image retained). Also: deployed container GPU crept ~6→13.9GB over 8h serving (cache accumulation; a redeploy resets it — watch item). [[reference_chatterbox_fast_repo]]
 
 - `[2026-08-09→10]` **dots.tts (rednote-hilab) TTS burn-in on irv-ml1 + canonical voice corpus built (`voices/`).** Operator-directed eval to potentially replace chatterbox-fast. **dots.tts VERIFIED real** (canonical HF ns `dots-studio/`, `rednote-hilab/dots.tts-*` redirects there; Apache-2.0; PyPI `dots.tts` 0.2.1; 2B continuous-AR = semantic enc + Qwen2.5-1.5B LLM + flow-matching acoustic head over 48kHz AudioVAE; zero-shot clone from wav+transcript). **Runs on Ampere 3090** (sm_86, bf16, no fp8 dep); **optimized RTF 0.22** at num_steps=10 (`from_pretrained(..., optimize=True)` CUDA graphs — raw unoptimized was 1.21), **~6GB VRAM**, 48kHz, streams (`generate_stream`). Venv+cache at `irv-ml1:/home/lkraven/dots-tts` (~10GB). **Operator design calls:** SGLang Omni serving (OpenAI `/v1/audio/speech`), transcribe-refs-first, `soar` variant. ⚠ Omni serves soar but its continuous-batching + streaming opts are **mf-only** (soar = single-request) — non-issue for ratatoskr's single-consumer RP surface. **KEY FINDING — dots is highly sensitive to an accurate AND sentence-bounded reference transcript:** mismatched transcript → 0.16s collapse; over-long/messy transcript → reference-audio BLEEDS as an output prefix; mid-clause trim → dangling-word leak (glados "we'll", emmie "And,"). RECIPE (baked into `voices/derive.py`): trim ref to a clean ~6–10s clip ending on a sentence boundary + accurate transcript of exactly that clip. **CANONICAL VOICE CORPUS** stood up in eshpfi `voices/` (operator idea): engine-agnostic `canonical/<v>.wav` + `transcripts/<v>.txt` → per-engine ref sets DERIVED by `derive.py` reading `engines.yaml` profiles (dots/chatterbox/zonos); canonical wavs git-tracked (small/curated), `derived/` gitignored. **4 voices optimized + verified CLEAN for dots: donut, glados, emmie, miranda** (glados canonical is low-SR 16kHz — flagged upgrade candidate). ⚠ GPU GOTCHA: irv-ml1 native CUDA orders **A6000=device0** (ComfyUI-full) — pin the 3090 with `CUDA_DEVICE_ORDER=PCI_BUS_ID CUDA_VISIBLE_DEVICES=0`; and `PYTORCH_CUDA_ALLOC_CONF=expandable_segments` CONFLICTS with `optimize=True` CUDA graphs (curr_block error). Booths: `dots-vs-chatterbox`, `dots-voices-optimized`. **SHIPPED 2026-08-10:** operator A/B verdict "dots is very good" → containerized as a **thin FastAPI wrapper over DotsTtsRuntime** (chosen over SGLang Omni — Omni's batching is mf-only, unneeded for ratatoskr's single consumer; wrapper is SERIALIZED one-gen-at-a-time via a threading.Lock, Omni+mf = parked API-compatible escalation if multi-consumer ever lands). **LIVE on irv-ml1:8198** (`local/dots-tts:v1`, OpenAI `/v1/audio/speech` + `/health` + `/v1/voices`, container healthy, both stream + non-stream verified CLEAN, 4 voices donut/glados/emmie/miranda) alongside chatterbox :8197 (nothing repointed). Stack = `stacks/dots-tts/` (Dockerfile/app.py/compose/.env.example/README). ⚠ CONTAINER GOTCHA: `optimize=True` (torch.compile/inductor/triton) needs a **C compiler at RUNTIME** — slim image must `apt install build-essential` or model-load dies "Failed to find C compiler" (host venv had gcc ambient, masking it); persist `TORCHINDUCTOR_CACHE_DIR` to a mounted dir or every restart re-JITs ~5min. Corpus home = eshpfi `voices/` (operator ruled keep-here). **REMAINING: ratatoskr client cutover** to :8198 `/v1/audio/speech` (Phase-2 tail, peer-coupled — draft the ask). [[reference_chatterbox_fast_repo]] [[reference_zonos_tts_stack]] [[reference_verify_hf_repo_ids_before_pull]]
 
@@ -273,7 +271,7 @@ _As of 2026-08-26 ~09:45 PDT — **run 2 is done, gated (FAIL), and serving. Run
 
 
 
-_214 older entries archived to archival-memory.md._
+_222 older entries archived to archival-memory.md._
 ## Tried and abandoned
 
 - `[2026-08-25]` **Four throughput levers measured and killed — do not re-chase.** (1) **Fused MoE / `grouped_mm`** — 0.9% *slower* than the Python loop and dense GEMM is only 7.9% of the step, capping the whole category near 10%. (2) **CUDA graphs / `torch.compile` over the expert loop** — the two-term scaling fit closed with residuals under 3ms and needed NO constant term, so there is no fixed per-batch cost to amortise; 3,840 expert-GEMM launches per forward are not what we pay for. (3) **`liger` fused linear CE** — the chunked CE measured **1.1% of the step** forward, ~3% with recompute. A tidy-up, not a lever. (4) **Selective gradient checkpointing** — ~2% of a post-fix step, real bug surface. Also: **token-budget batching is dead by the same fit** — with no constant term, total time over a fixed set of widths is invariant to how you group them; only the widths matter, which is exactly why bucketing works and repacking does not.