memory: snapshot — chatterbox-fast streaming build (adaptive buffer-ratchet chunking); native-turbo-streaming abandoned; fish reference_id fix + glados transcript + NFS/smithy + references-togglable; add design doc

docs/design/chatterbox-fast.md

@@ -0,0 +1,137 @@
+# Chatterbox-Fast — Streaming TTS Engine (design)
+
+**Status:** Draft / pre-contract design. Spike-validated 2026-06-02.
+**Owner:** infra-ops · **Workload (operator-confirmed):** single-stream interactive.
+
+## 1. Goal
+
+Make Chatterbox-Turbo our **primary interactive TTS engine** by cutting
+time-to-first-audio from today's ~2.5 s to ~0.3–0.5 s via **true
+incremental streaming**, while preserving turbo's quality, inline
+paralinguistic tags, and voice cloning.
+
+## 2. Why now — the proven result
+
+Spike (2026-06-02, turbo on the A6000), proper CUDA-sync timing:
+
+| | time-to-first-audio |
+|---|---|
+| today (devnen, no real streaming) | **~2.5 s** |
+| windowed generate, K=20 tokens | **0.31 s** (delivers ~0.96 s audio) |
+| K=30 | 0.41 s |
+| K=50 | 0.57 s |
+
+~**8× TTFB win.** Decode cost is ~constant **0.12 s** regardless of chunk
+size (turbo's 2-step decoder), so first-chunk time is dominated by
+generating the first K tokens — smaller K = faster first audio. Full
+generation runs at **4.2× realtime**, so once the first chunk plays the
+generator stays well ahead of playback and the stream never starves.
+
+## 3. Non-goals
+
+- High-concurrency **batch throughput** — that's a separate base-on-vLLM
+  lane; deferred (research: vLLM port doesn't support turbo).
+- **Multilingual** — turbo is EN-only; out of scope.
+- **Quantization** — research flags it high-risk (gibberish < Q8, Q8-CUDA
+  broken); deprioritized.
+
+## 4. Background — turbo internals (from the spike)
+
+- `generate()` = `t3.inference_turbo()` (AR loop, all tokens) →
+  `s3gen.inference(..., n_cfm_timesteps=2)` (flow + HiFT vocoder, all tokens).
+- `inference_turbo` is a **plain-Python `for` loop** with a KV cache, a
+  `max_gen_len` param, and a stop-token break — cleanly hookable.
+- `s3gen` decode is cheap per-window (~0.12 s constant).
+
+## 5. Architecture
+
+A **lean, purpose-built FastAPI server on the `chatterbox` library** —
+*not* a fork of devnen. devnen buffers the entire synthesis before
+emitting (proven: opus/mp3/wav all return first byte at full-synth time);
+we need to own the generate loop. Components:
+
+1. **Model holder** — `ChatterboxTurboTTS` loaded once at startup, warmed.
+2. **Streaming generate** — windowed wrapper around `inference_turbo`:
+   yields token windows; each window decoded via `s3gen` → audio chunk →
+   streamed. (~40 lines on top of the lib, per the spike.)
+3. **Voice management** — predefined voices (dir of wavs) + clone refs
+   (the `prepare_conditionals` path); reuse chatterbox's `Conditionals`.
+4. **HTTP API** — `POST /tts` (streaming + non-streaming), optional
+   OpenAI-compat `/v1/audio/speech`, `/health`.
+5. **Watermark** — Resemble PerTh (mandatory); applied per-chunk or post.
+
+### Decision 1 — streaming transport
+**Recommend: HTTP chunked transfer of raw PCM** (client plays chunks as
+they arrive). Lowest latency, trivial client. Offer opus for
+bandwidth-constrained callers. *Not* websocket (one-way; overkill).
+
+### Decision 2 — seam handling (the key productionization detail)
+Independent per-window decode (as in the spike) can leave faint **seams**
+at chunk boundaries because the vocoder has receptive-field context.
+**Approach: overlap-discard** — decode each window with a small lookback
+of the previous window's trailing tokens, discard that overlap's audio,
+keep only the new window's output. (The `davidbrowne17/chatterbox-streaming`
+fork uses this pattern.) Tune the overlap for inaudible seams vs latency.
+**Validate** by ear + a spectral seam check.
+
+### Decision 3 — chunk schedule
+First chunk **small** (K≈20–25 → ~0.3 s first audio); subsequent chunks
+**larger** (K≈50–100) for decode efficiency, since after chunk 1 we're
+ahead of playback. A simple ramp.
+
+## 6. Performance levers (fold in, measure each)
+
+- **bf16** (Ampere-safe), **TF32** (matmul), **SDPA/flash** backend on the
+  Llama backbone — low-risk, measure the delta.
+- **torch.compile** — **DEFER.** Research flags a real batch-1 regression
+  risk (documented 0.85× at batch-1). Benchmark separately; adopt only if
+  it beats eager on our hardware. Not on the critical path.
+
+## 7. Deployment
+
+- New stack **`chatterbox-fast` deployed ALONGSIDE** the existing
+  `chatterbox` (zero disruption; A/B then cut over).
+- **Port:** 8197 (next free on irv-ml1).
+- **GPU placement (decided 2026-06-02):** **3090 (device 0) if it fits,
+  else A6000 (device 1).** The GPU stack is a shared dev stack — workloads
+  float across cards, so the 20.5 GB-at-idle on the 3090 is expected
+  residency, not a blocker. Fit is borderline: turbo is ~2.5 GB but the
+  3090 currently shows ~3.5 GB free, so the deploy step **tries the 3090,
+  falls back to the A6000 (device 1, ~30 GB free, shares with Fish) on
+  OOM.** Pin via `device_ids` in compose per fleet convention.
+- **From-source Dockerfile** (chatterbox lib + our server), pinned.
+
+## 8. Benchmark / A-B gate (deploy guard, like the Fish reference_id gate)
+
+- **first-audio (TTFB)** under target (e.g. < 0.6 s on the deployment GPU).
+- **realtime factor** maintained (> 3×).
+- **quality parity** vs current chatterbox — ECAPA speaker-sim for clone
+  voices, listen test for predefined, spectral **seam** check.
+- Wire as a hard gate in the deploy playbook.
+
+## 9. Risks / open questions
+
+1. **Seam artifacts** — mitigation: overlap-discard decode; validate by ear + spectral.
+2. **torch.compile batch-1 regression** — mitigation: benchmark, optional.
+3. **3090's 20.5 GB-at-idle** — RESOLVED (non-issue): shared dev stack, expected residency. Placement decided (§7): 3090-if-fits-else-A6000.
+4. **PerTh watermark on short chunks** — confirm no artifacts per-chunk.
+5. **Paralinguistic tags across chunk boundaries** — confirm a tag split
+   across windows doesn't break delivery.
+
+## 10. Build plan (phases)
+
+0. **Spike** — DONE, proven (§2).
+1. **Streaming server MVP** — windowed generate + overlap-discard seam
+   handling + `/tts` streaming endpoint; bench first-audio + seam quality.
+2. **Parity + perf** — predefined + clone voice management; bf16/TF32/SDPA;
+   per-chunk watermark.
+3. **Containerize + deploy** — from-source Dockerfile; deploy `chatterbox-fast`
+   alongside; wire the A-B gate.
+4. **Cutover** — switch the catalog route; burn-in; deprecate the old stack.
+
+## 11. Open decisions for operator
+
+- ~~GPU placement~~ — **DECIDED (2026-06-02):** 3090 if it fits, else A6000 (§7).
+- ~~Cutover strategy~~ — **DECIDED (2026-06-02): parallel catalog entry**,
+  burn-in beside the live `chatterbox`, then flip the route once it earns
+  trust. Phases 1–3 are cutover-agnostic; the flip happens in Phase 4.

persistent-memory.md

@@ -1,6 +1,6 @@
 # Persistent memory — eshpfi-management
 
-_Last updated: 2026-06-01_
+_Last updated: 2026-06-02_
 
 ## Repo purpose
 
@@ -87,8 +87,34 @@ Sister repos (separate gitea repos, deployed by playbooks here):
 
 ## Current state / in-flight
 
-_As of 2026-06-01:_
+_As of 2026-06-02:_
 
+- **PRIMARY FOCUS — building `chatterbox-fast`, a custom streaming TTS
+  container; Chatterbox is becoming our MAIN TTS engine.** Operator-authorized
+  high-effort build (incl. custom container from source). **Plan-of-attack:
+  `/tmp/chatterbox-fast-plan.md`** (write/refresh before /clear — it carries
+  the full executable detail). Design doc: `docs/design/chatterbox-fast.md`.
+  - **Goal:** cut time-to-first-audio from ~2.5s → sub-second via streaming,
+    keep turbo quality. Workload = single-stream interactive.
+  - **Chosen approach = adaptive buffer-ratchet chunking** (operator's idea):
+    generate the FIRST sentence alone for instant response (~0.66s first-audio
+    measured), then while it plays, generate the largest sentence-aligned chunk
+    that fits in ~80% of the bought playback time; chunks grow ~3× each (driven
+    by Chatterbox's ~3.8× realtime headroom), so after 2-3 chunks the rest of
+    the paragraph is one big chunk with near-full context. Context loss confined
+    to 2-3 joins at natural sentence-pause boundaries. THIS is "sentence-level
+    done right" and supersedes naive per-sentence splitting (which loses
+    cross-sentence prosodic context = real quality loss, operator-corrected).
+    Only works because RTF>1 (Fish at <1× realtime starves — can't use this).
+  - **Native frame-level streaming on turbo = ABANDONED** (see Tried/abandoned).
+  - **Also to build for A/B:** base-chatterbox + davidbrowne17 streaming fork
+    (true frame-stream but base-model quality) — not yet installed.
+  - **A/B samples** (GLaDOS voice) on nh3-dev `~/chatterbox-ab/`: sentence-level
+    turbo, chunked-native (artifacty), chunked-oneshot.
+  - **Dev/test pattern:** one-off GPU-1 container from `local/chatterbox:v1`,
+    mount `/worktank/chatterbox/{cache,reference_audio}` + the test script;
+    `docker exec -i chatterbox python -` for lib introspection. lkraven is in
+    the `docker` group on irv-ml1 — NO sudo needed for docker.
 - **TTS fleet on irv-ml1 reworked this session; asset-engine catalog now
   17 services** (CSM removed). Canonical `docs/asset-engine/services.yaml`
   + vendored `vh/asset-engine` both pushed (eshpfi `38d9e3b`; asset-engine
@@ -109,15 +135,16 @@ _As of 2026-06-01:_
     API). Research/non-commercial license.
   - **zonos (:8203)** down (adapter built, not deployed). **ComfyUI :8188**
     catalog-deferred link.
-- **BLOCKED on asset-engine-dev — P1 "undefined" select bug** (default voice
-  picks send the literal string `"undefined"` → Chatterbox 404s, Fish falls to
-  default voice). ROOT-CAUSED: the form's Kokoro voice-blend widget
-  (`templates/partials/_fields.html`) runs `recompute()` at page load, reading
-  the Shoelace `<sl-select>.value` BEFORE hydration → writes "undefined" into
-  the submit field. Escalated with full root cause + 3-part fix (althing msg
-  `01KT2K2SY9N7AY69R9V0B4RXSW` → asset-engine-dev). **Workaround until fixed:
-  click the dropdown (fires change) or hit the API directly.** My half
-  (`blendable: false` catalog flag) is queued — see Recent decisions.
+- **"undefined" select bug — RESOLVED.** asset-engine-dev shipped form-select
+  hardening (v0.1.9/.10) + a durable **per-field enable-toggle** (`togglable`,
+  v0.1.14/.16) — the real fix for the "form submits untouched fields" family.
+  My catalog half: opted fish-s2 `references` into `togglable: true`
+  (`catalog_version 1→2`, schema gained the `togglable` boolean; `c98a12b`,
+  pushed to origin). The earlier `blendable` idea was superseded by `togglable`.
+  dia2/chatterbox clone fields deliberately NOT toggled (dia2 defaults clone-on
+  as its stable voice). asset-engine CI was briefly red because the commit was
+  local-only until pushed — lesson: push catalog commits promptly (their CI
+  drift-checks against the remote).
 - **Fish "not British" had TWO independent root causes — BOTH now fixed.**
   The ECAPA-TDNN re-test (2026-06-01) showed Fish's cloning *engine* works: an
   Imogen reference scores **~0.79 cosine vs real `Imogen.wav`** vs **~0.10 vs
@@ -147,9 +174,19 @@ _As of 2026-06-01:_
   (CC BY 4.0, accent-tagged speaker IDs), Unmute voice-donations (CC0), EARS +
   expresso (CC BY-NC). Source for future clone voices. British-female Southern
   England speakers p225/p228/p229 staged into Fish as Imogen/Eleanor/Beatrice.
-- **Parakeet ASR (:8765) is DOWN** — no container, but catalog says
-  `status: ready` (drift). Use Whisper-in-a-container (`faster-whisper`) for
-  transcripts instead; it's self-contained (CTranslate2, no torch).
+- **Parakeet ASR (:8765) is now UP** (brought online 2026-06-02 to transcribe
+  the glados clip; operator wants it kept online). CPU-only (`gpu_device_id:
+  null`), `restart=unless-stopped`, healthy. API: POST multipart `file` to
+  `/transcribe` → `{"text":...}`. Image `local/parakeet:sherpa-onnx-v2`
+  pre-built; `docker compose up -d` in `/opt/docker/compose/parakeet`.
+- **NFS share for Smithy — DONE.** `/volume1/smithy` on nh3-nas → `/mnt/smithy`
+  on nh3-dev (single export, RW, map-to-admin, scoped to nh3-dev only, fstab
+  `_netdev,nofail,x-systemd.automount`). Subdirs `datasets/{raw,manifest-store,
+  derived,holdout,quarantine}`; rename-atomicity verified. Smithy wired
+  storage-roots.yaml + E2E-verified. **Backup DEFERRED** per operator (datasets
+  regenerable; raw + manifest-store are the irreplaceable surface for when it's
+  wired — durable tier should land OFF nh3-nas/cross-site, since the NH3 restic
+  repo lives on the same NAS as the source).
 - **Worldtree healthy v0.29.13** (last-known); **Skaldsong v0.32.2**
   (ana-docker:8300, Kokoro SSE streaming); **artemis-31b-v1i** live on
   llama-swap + worldtree personal; **ttyd fleet driver seat** on nh3-dev
@@ -168,6 +205,29 @@ _As of 2026-06-01:_
 
 ## Recent decisions
 
+- `[2026-06-02]` **Chatterbox → main TTS engine; build custom `chatterbox-fast`
+  streaming container.** Workload = single-stream interactive. **GPU placement:
+  3090 (device 0) if it fits else A6000 (device 1)** — shared dev stack, 20.5 GB
+  3090-idle is expected residency, not a blocker. **Cutover: parallel catalog
+  entry**, burn in beside live `chatterbox`, then flip. **Streaming approach:
+  adaptive buffer-ratchet chunking** (see in-flight). Native frame-streaming
+  abandoned (Tried/abandoned). Tracked: `docs/design/chatterbox-fast.md` +
+  `/tmp/chatterbox-fast-plan.md`.
+
+- `[2026-06-02]` **Sentence-splitting loses quality (operator-corrected).** I
+  claimed naive sentence-level streaming has "zero quality loss" — WRONG. The
+  T3 AR backbone conditions prosody on the WHOLE text; splitting loses
+  cross-sentence prosodic context (contextual delivery, declination, affect
+  continuity) even though voice timbre stays (reference-conditioned). No
+  *artifacts* ≠ no *quality loss*. Hence the adaptive-chunk design (maximize
+  context per chunk subject to latency budget), not fixed per-sentence splits.
+
+- `[2026-06-01]` **Fish reference_id empty-dir fix shipped** (`c5bbb90`) — see
+  in-flight + Tried/abandoned. Populated `references/<name>/<name>.wav`+`.lab`
+  for all 32 voices; playbook gained normalize-step + A/B smoke gate. glados got
+  a real transcript (ASR'd via Parakeet): the Portal "Welcome to test chamber 4"
+  lines.
+
 - `[2026-06-01]` **Fish cloning VERIFIED competent (ECAPA-TDNN)** — retracting
   the earlier "weak cloner" call. Isolated test: Imogen-referenced clone ~0.79
   cosine to the real `Imogen.wav` vs ~0.10 for the no-reference default;
@@ -262,6 +322,32 @@ _25 older entries archived to archival-memory.md._
 
 ## Tried and abandoned
 
+- `[2026-06-02]` **Native frame-level streaming on Chatterbox-TURBO — ABANDONED
+  (turbo isn't built for streaming).** Long R&D arc; record so it's not
+  re-derived. (1) The model's flow is CosyVoice2-derived but `S3GenStreamer` is
+  referenced-in-docstring-only (not implemented). (2) The lib's
+  `flow_inference(finalize=False)` is BUGGY: the lookahead trim removes
+  `pre_lookahead_len(3)*token_mel_ratio(2)=6` frames from `h` but NOT from
+  `h_masks`/conds → decoder shape mismatch (e.g. 656 vs 662). A 1-line patch
+  (`h_masks = h_masks[:, :, :-pre*ratio]` after the `h` trim) + sizing the
+  meanflow noise to the trimmed length makes finalize=False RUN. (3) BUT the
+  flow encoder uses FULL-context attention (`static_chunk_size=0`), so
+  incremental/cumulative decode is **prefix-unstable** — adding tokens
+  re-attends and shifts earlier mel (maxdiff ~0.30-0.39 vs one-shot,
+  irrespective of fixed-noise slicing or emit-margin). (4) Forcing
+  `static_chunk_size>0` on the 2 modules that carry the attr did NOT stabilize
+  it (decoding_chunk_size is a forward-arg, not settable via attribute). Verdict:
+  true sub-second frame-streaming on turbo needs deep model-attention surgery
+  with quality risk — not worth it. Matches research ("turbo+streaming
+  unsolved"; vLLM-turbo outputs noise; davidbrowne17 streaming fork is
+  BASE-only). → Use adaptive-chunking instead.
+
+- `[2026-06-02]` **Naive cumulative re-decode for streaming** — decode the
+  growing token prefix each chunk + emit the delta, assuming the causal flow
+  gives a stable prefix. It does NOT (full-context attention, see above);
+  maxdiff 0.30. Also the high-level `s3gen.inference` re-applies `trim_fade` at
+  the start every call. Don't go this way.
+
 - `[2026-06-01]` **CSM bring-up** — upstream `phildougherty/sesame_csm_openai`
   Dockerfile pins NO `huggingface_hub` version, so it now resolves to 1.17.0
   where `huggingface-cli` is fully removed (replaced by `hf`) → the build dies