esh-pfi-infrastructure/docs/design/chatterbox-fast-plan.md

# Plan of Attack — `chatterbox-fast` streaming TTS engine

_Authored 2026-06-02 for a fresh-context build session. Self-contained: you should
not need the prior conversation. Cross-refs: `docs/design/chatterbox-fast.md`
(design), `persistent-memory.md` (durable state + the abandoned native-streaming
arc), repo `~/development/eshpfi-management` on host **nh3-dev**._

---

## 0. Mission

Chatterbox(-Turbo) is becoming our **main TTS engine**. Build `chatterbox-fast`:
a custom streaming server + container that delivers **sub-second time-to-first-
audio** while keeping **turbo's full quality**. Workload = **single-stream
interactive**. Operator authorized high effort incl. building the container from
source. Deploy as a **parallel** stack beside the live `chatterbox` (:8196), burn
in, then flip the catalog route.

## 1. THE design — adaptive buffer-ratchet chunking (operator's idea; chosen)

**Why not the alternatives** (settled this session, don't relitigate):
- **Whole-paragraph one-shot** = best quality but ~2.5s+ TTFB (no streaming).
- **Naive per-sentence split** = fast but **loses cross-sentence prosodic
  context** → real quality loss (the T3 AR backbone conditions prosody on the
  WHOLE text: contextual delivery, declination, affect continuity). "No
  artifacts" ≠ "no quality loss." Operator corrected this; don't claim otherwise.
- **Native frame-level streaming on turbo** = ABANDONED (turbo's flow uses
  full-context attention, `static_chunk_size=0` → prefix-unstable; see
  persistent-memory Tried/abandoned for the full dead-end map). Do NOT re-attempt
  without explicit operator direction.

**The adaptive-chunk algorithm:**
1. Split text into **sentences** (and fall back to clause/comma split for a very
   long FIRST sentence only, to protect first-audio latency).
2. **Chunk 1 = first sentence** — generate alone, emit immediately (~0.66s
   first-audio measured for a short sentence). Latency-critical.
3. **While chunk N plays, generate chunk N+1** = greedily accumulate WHOLE
   sentences until the *next* sentence would exceed the gen-time budget
   `margin × audio_buffered_remaining`. Never split mid-sentence (keeps each
   chunk prosodically self-coherent; joins land at natural sentence pauses).
4. Chunks grow ~**3× each** (Chatterbox runs ~3.8× realtime; each chunk's
   playback buys wall-clock for a ~3× bigger next chunk). So after 2-3 chunks,
   the rest of the paragraph is ONE big chunk with near-full context. Context
   loss confined to 2-3 joins at sentence boundaries.
5. **Drive off MEASURED realtime factor**, not a constant — track actual
   gen-speed live and self-correct. Start `margin=0.8`; be more conservative on
   the **first** transition (smallest buffer = highest starvation risk) — ~0.6-0.7
   there, then relax.
6. **Optional context-priming at joins (quality-max):** prepend the previous
   sentence as context to a chunk, generate, discard its audio → the chunk's
   first sentence gets backward context. Cheap on early small chunks; skip once
   chunks are large. Add this in Phase 2, measure if it's audibly worth it.

**Critical enabling fact:** this only works because **RTF > 1**. Fish (<1×
realtime) would starve no matter the chunking — that's why this is the
chatterbox-specific answer.

## 2. Validated API + facts (don't re-derive)

- Model: `from chatterbox.tts_turbo import ChatterboxTurboTTS`
  - `m = ChatterboxTurboTTS.from_pretrained(device="cuda")` (loads from HF cache)
  - `m.prepare_conditionals(wav_path, exaggeration=0.5, norm_loudness=True)`
  - `wav = m.generate(text, repetition_penalty=1.2, top_p=0.95, temperature=0.8,
    top_k=1000)` → returns **watermarked** wav tensor shape `[1, T]`, `m.sr=24000`.
    (CFG/exaggeration/min_p are ignored by turbo — warns but harmless.)
  - Paralinguistic tags work inline (`[laugh] [whispers] [sigh]` etc.).
- Architecture: T3 AR Llama 350M → S3Gen flow (2-step meanflow) → HiFTGenerator.
- Realtime: ~3.8× on A6000 (17.7s audio / 4.7s), ~3.4× on 3090.
- First-sentence latency: ~0.66s (short sentence, warm).
- Watermark (Resemble PerTh) is applied inside `m.generate` — mandatory, fine
  for internal use.

## 3. Dev/test pattern (host irv-ml1 = 10.100.79.3, ssh `lkraven@10.100.79.3`)

- lkraven is in the `docker` group on irv-ml1 → **NO sudo for docker**.
- Model weights cached at `/worktank/chatterbox/cache` (HF_HOME); reference wavs
  at `/worktank/chatterbox/reference_audio` (has `glados_25s.wav`, `Imogen.wav`).
- One-off GPU container (use **GPU 1 / A6000** for dev — 3090 is VRAM-tight):
  ```
  IMG=$(docker images --format '{{.Repository}}:{{.Tag}}' | grep -i chatterbox | grep -v '<none>' | head -1)  # local/chatterbox:v1
  docker run --rm --gpus '"device=1"' -e NVIDIA_VISIBLE_DEVICES=1 -e HF_HOME=/app/hf_cache \
    -v /worktank/chatterbox/cache:/app/hf_cache \
    -v /worktank/chatterbox/reference_audio:/refs \
    -v /tmp/yourscript.py:/test.py "$IMG" python /test.py
  ```
- Lib introspection: `docker exec -i chatterbox python - <<'PY' ... PY` against
  the running server container.
- Write A/B samples to `/refs/_*.wav`, then `scp lkraven@10.100.79.3:/worktank/
  chatterbox/reference_audio/_*.wav ~/chatterbox-ab/` for the operator to hear.

## 4. Build phases

**Phase 1 — streaming server MVP (the scheduler is the meat):**
- `stacks/chatterbox-fast/app.py` — FastAPI server:
  - Load model once at startup, warm it (one throwaway `generate`).
  - `POST /tts` → `StreamingResponse` of audio chunks. Body: text, voice
    (predefined name or clone ref), format (raw pcm s16le default for lowest
    latency; offer wav/opus), the sampling knobs.
  - The **adaptive-chunk scheduler** (§1): sentence-split → gen first sentence →
    emit → loop {measure RTF, accumulate sentences to budget, generate, emit}.
    Track `audio_emitted_seconds` and wall-clock to estimate buffer drain.
  - `GET /health`.
- Validate: first-audio latency, that the stream never starves (sim a player
  consuming at 1× realtime), and produce a sample for the operator vs the
  whole-paragraph one-shot.

**Phase 2 — parity + perf:**
- Predefined voices (dir of wavs) + clone refs (`prepare_conditionals`).
- bf16 (`TTS_BF16`-style, or set model dtype), TF32
  (`torch.backends.cuda.matmul.allow_tf32=True`), SDPA/flash backend.
- Optional context-priming at joins (§1.6) — measure if audibly worth it.
- torch.compile: DEFER (research flags batch-1 regression; bench separately).

**Phase 3 — containerize + deploy:**
- `stacks/chatterbox-fast/` : `compose.yaml`, `Dockerfile` (FROM the chatterbox
  base image / vendored chatterbox + our `app.py`), `.env.example`, `README.md`.
- Follow repo conventions (CLAUDE.md): `traefik-net`/`tnet`, named volumes,
  `restart: unless-stopped`, healthcheck, homepage labels, GPU pin via
  `device_ids`. **Port 8197** (next free on irv-ml1; reserved list in
  `stacks/chatterbox/.env.example`). **GPU: 3090 (device 0) if turbo fits in
  free VRAM, else A6000 (device 1)** — try 3090, fall back on OOM.
- `playbooks/deploy-chatterbox-fast.yaml` (model is HF-cached already; reuse
  `/worktank/chatterbox/cache`). Add an A/B smoke gate (first-audio < target).
- Deploy **alongside** the live `chatterbox` — do NOT disrupt :8196.

**Phase 4 — A/B + cutover:**
- Add a **parallel** `chatterbox-fast` catalog entry in
  `docs/asset-engine/services.yaml` (NOT replace `chatterbox` yet). If it needs a
  new schema field, that's a `catalog_version` bump — coordinate with
  **asset-engine-dev** via althing (and PUSH the commit promptly; their CI
  drift-checks against the remote — lesson learned this session).
- Burn-in + operator ear-A/B vs whole-paragraph. Then flip the route.

## 5. ALSO build for A/B (operator asked): base-chatterbox + streaming fork

- Install `davidbrowne17/chatterbox-streaming` (a fork with `generate_stream()`,
  measured first-chunk ~0.47s on a 4090) — **BASE chatterbox model, not turbo**.
  True frame-level streaming but base-model quality. Stand it up (own container /
  port), generate a sample with the SAME text + a comparable voice, drop in
  `~/chatterbox-ab/` for the 3-way A/B: adaptive-chunk-turbo vs base-fork-stream
  vs whole-paragraph-turbo. Operator judges by ear.

## 6. Acceptance / A/B

- **Latency:** first-audio < ~0.8s on the deployment GPU.
- **No starvation:** stream stays ahead of 1× playback (assert in a sim).
- **Quality:** operator ear-A/B the adaptive-chunk output vs whole-paragraph
  one-shot — the join-context loss should be ~imperceptible for multi-sentence
  text. Samples → `~/chatterbox-ab/`.

## 7. Existing A/B samples (this session, GLaDOS voice) on nh3-dev `~/chatterbox-ab/`
- `01_sentence_level_turbo.wav` — naive per-sentence (the baseline to BEAT).
- `02_chunked_native_streamed.wav` — abandoned native attempt (artifacty).
- `03_chunked_oneshot.wav` — chunked-attention one-shot.
(The adaptive-chunk output and the base-fork output are still to be generated.)

## 8. Watch-outs
- Don't claim sentence-splitting is lossless (it isn't — prosodic context).
- Don't re-attempt native turbo frame-streaming without operator say-so.
- Push catalog commits to origin promptly (asset-engine CI).
- Use `ssh -t` only when a remote needs sudo; docker on irv-ml1 needs no sudo.
- The 3090 shows ~20.5 GB used at idle (shared dev stack) — expect tight fit.