From 4b9bd109bf6fc41d1d553657cb97156a93f50b93 Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Mon, 1 Jun 2026 22:23:17 -0700 Subject: [PATCH] docs(chatterbox-fast): add executable plan-of-attack (durable, survives reboot vs /tmp) Self-contained build plan for the chatterbox-fast streaming engine: the adaptive buffer-ratchet chunking design, validated turbo API + facts, the GPU-1 dev/test container pattern, 4 build phases, the base-fork A/B, and watch-outs (incl. native-turbo-streaming is abandoned). Intended for a fresh-context session to execute at full strength. --- docs/design/chatterbox-fast-plan.md | 162 ++++++++++++++++++++++++++++ 1 file changed, 162 insertions(+) create mode 100644 docs/design/chatterbox-fast-plan.md diff --git a/docs/design/chatterbox-fast-plan.md b/docs/design/chatterbox-fast-plan.md new file mode 100644 index 0000000..c6f6623 --- /dev/null +++ b/docs/design/chatterbox-fast-plan.md @@ -0,0 +1,162 @@ +# Plan of Attack — `chatterbox-fast` streaming TTS engine + +_Authored 2026-06-02 for a fresh-context build session. Self-contained: you should +not need the prior conversation. Cross-refs: `docs/design/chatterbox-fast.md` +(design), `persistent-memory.md` (durable state + the abandoned native-streaming +arc), repo `~/development/eshpfi-management` on host **nh3-dev**._ + +--- + +## 0. Mission + +Chatterbox(-Turbo) is becoming our **main TTS engine**. Build `chatterbox-fast`: +a custom streaming server + container that delivers **sub-second time-to-first- +audio** while keeping **turbo's full quality**. Workload = **single-stream +interactive**. Operator authorized high effort incl. building the container from +source. Deploy as a **parallel** stack beside the live `chatterbox` (:8196), burn +in, then flip the catalog route. + +## 1. THE design — adaptive buffer-ratchet chunking (operator's idea; chosen) + +**Why not the alternatives** (settled this session, don't relitigate): +- **Whole-paragraph one-shot** = best quality but ~2.5s+ TTFB (no streaming). +- **Naive per-sentence split** = fast but **loses cross-sentence prosodic + context** → real quality loss (the T3 AR backbone conditions prosody on the + WHOLE text: contextual delivery, declination, affect continuity). "No + artifacts" ≠ "no quality loss." Operator corrected this; don't claim otherwise. +- **Native frame-level streaming on turbo** = ABANDONED (turbo's flow uses + full-context attention, `static_chunk_size=0` → prefix-unstable; see + persistent-memory Tried/abandoned for the full dead-end map). Do NOT re-attempt + without explicit operator direction. + +**The adaptive-chunk algorithm:** +1. Split text into **sentences** (and fall back to clause/comma split for a very + long FIRST sentence only, to protect first-audio latency). +2. **Chunk 1 = first sentence** — generate alone, emit immediately (~0.66s + first-audio measured for a short sentence). Latency-critical. +3. **While chunk N plays, generate chunk N+1** = greedily accumulate WHOLE + sentences until the *next* sentence would exceed the gen-time budget + `margin × audio_buffered_remaining`. Never split mid-sentence (keeps each + chunk prosodically self-coherent; joins land at natural sentence pauses). +4. Chunks grow ~**3× each** (Chatterbox runs ~3.8× realtime; each chunk's + playback buys wall-clock for a ~3× bigger next chunk). So after 2-3 chunks, + the rest of the paragraph is ONE big chunk with near-full context. Context + loss confined to 2-3 joins at sentence boundaries. +5. **Drive off MEASURED realtime factor**, not a constant — track actual + gen-speed live and self-correct. Start `margin=0.8`; be more conservative on + the **first** transition (smallest buffer = highest starvation risk) — ~0.6-0.7 + there, then relax. +6. **Optional context-priming at joins (quality-max):** prepend the previous + sentence as context to a chunk, generate, discard its audio → the chunk's + first sentence gets backward context. Cheap on early small chunks; skip once + chunks are large. Add this in Phase 2, measure if it's audibly worth it. + +**Critical enabling fact:** this only works because **RTF > 1**. Fish (<1× +realtime) would starve no matter the chunking — that's why this is the +chatterbox-specific answer. + +## 2. Validated API + facts (don't re-derive) + +- Model: `from chatterbox.tts_turbo import ChatterboxTurboTTS` + - `m = ChatterboxTurboTTS.from_pretrained(device="cuda")` (loads from HF cache) + - `m.prepare_conditionals(wav_path, exaggeration=0.5, norm_loudness=True)` + - `wav = m.generate(text, repetition_penalty=1.2, top_p=0.95, temperature=0.8, + top_k=1000)` → returns **watermarked** wav tensor shape `[1, T]`, `m.sr=24000`. + (CFG/exaggeration/min_p are ignored by turbo — warns but harmless.) + - Paralinguistic tags work inline (`[laugh] [whispers] [sigh]` etc.). +- Architecture: T3 AR Llama 350M → S3Gen flow (2-step meanflow) → HiFTGenerator. +- Realtime: ~3.8× on A6000 (17.7s audio / 4.7s), ~3.4× on 3090. +- First-sentence latency: ~0.66s (short sentence, warm). +- Watermark (Resemble PerTh) is applied inside `m.generate` — mandatory, fine + for internal use. + +## 3. Dev/test pattern (host irv-ml1 = 10.100.79.3, ssh `lkraven@10.100.79.3`) + +- lkraven is in the `docker` group on irv-ml1 → **NO sudo for docker**. +- Model weights cached at `/worktank/chatterbox/cache` (HF_HOME); reference wavs + at `/worktank/chatterbox/reference_audio` (has `glados_25s.wav`, `Imogen.wav`). +- One-off GPU container (use **GPU 1 / A6000** for dev — 3090 is VRAM-tight): + ``` + IMG=$(docker images --format '{{.Repository}}:{{.Tag}}' | grep -i chatterbox | grep -v '' | head -1) # local/chatterbox:v1 + docker run --rm --gpus '"device=1"' -e NVIDIA_VISIBLE_DEVICES=1 -e HF_HOME=/app/hf_cache \ + -v /worktank/chatterbox/cache:/app/hf_cache \ + -v /worktank/chatterbox/reference_audio:/refs \ + -v /tmp/yourscript.py:/test.py "$IMG" python /test.py + ``` +- Lib introspection: `docker exec -i chatterbox python - <<'PY' ... PY` against + the running server container. +- Write A/B samples to `/refs/_*.wav`, then `scp lkraven@10.100.79.3:/worktank/ + chatterbox/reference_audio/_*.wav ~/chatterbox-ab/` for the operator to hear. + +## 4. Build phases + +**Phase 1 — streaming server MVP (the scheduler is the meat):** +- `stacks/chatterbox-fast/app.py` — FastAPI server: + - Load model once at startup, warm it (one throwaway `generate`). + - `POST /tts` → `StreamingResponse` of audio chunks. Body: text, voice + (predefined name or clone ref), format (raw pcm s16le default for lowest + latency; offer wav/opus), the sampling knobs. + - The **adaptive-chunk scheduler** (§1): sentence-split → gen first sentence → + emit → loop {measure RTF, accumulate sentences to budget, generate, emit}. + Track `audio_emitted_seconds` and wall-clock to estimate buffer drain. + - `GET /health`. +- Validate: first-audio latency, that the stream never starves (sim a player + consuming at 1× realtime), and produce a sample for the operator vs the + whole-paragraph one-shot. + +**Phase 2 — parity + perf:** +- Predefined voices (dir of wavs) + clone refs (`prepare_conditionals`). +- bf16 (`TTS_BF16`-style, or set model dtype), TF32 + (`torch.backends.cuda.matmul.allow_tf32=True`), SDPA/flash backend. +- Optional context-priming at joins (§1.6) — measure if audibly worth it. +- torch.compile: DEFER (research flags batch-1 regression; bench separately). + +**Phase 3 — containerize + deploy:** +- `stacks/chatterbox-fast/` : `compose.yaml`, `Dockerfile` (FROM the chatterbox + base image / vendored chatterbox + our `app.py`), `.env.example`, `README.md`. +- Follow repo conventions (CLAUDE.md): `traefik-net`/`tnet`, named volumes, + `restart: unless-stopped`, healthcheck, homepage labels, GPU pin via + `device_ids`. **Port 8197** (next free on irv-ml1; reserved list in + `stacks/chatterbox/.env.example`). **GPU: 3090 (device 0) if turbo fits in + free VRAM, else A6000 (device 1)** — try 3090, fall back on OOM. +- `playbooks/deploy-chatterbox-fast.yaml` (model is HF-cached already; reuse + `/worktank/chatterbox/cache`). Add an A/B smoke gate (first-audio < target). +- Deploy **alongside** the live `chatterbox` — do NOT disrupt :8196. + +**Phase 4 — A/B + cutover:** +- Add a **parallel** `chatterbox-fast` catalog entry in + `docs/asset-engine/services.yaml` (NOT replace `chatterbox` yet). If it needs a + new schema field, that's a `catalog_version` bump — coordinate with + **asset-engine-dev** via althing (and PUSH the commit promptly; their CI + drift-checks against the remote — lesson learned this session). +- Burn-in + operator ear-A/B vs whole-paragraph. Then flip the route. + +## 5. ALSO build for A/B (operator asked): base-chatterbox + streaming fork + +- Install `davidbrowne17/chatterbox-streaming` (a fork with `generate_stream()`, + measured first-chunk ~0.47s on a 4090) — **BASE chatterbox model, not turbo**. + True frame-level streaming but base-model quality. Stand it up (own container / + port), generate a sample with the SAME text + a comparable voice, drop in + `~/chatterbox-ab/` for the 3-way A/B: adaptive-chunk-turbo vs base-fork-stream + vs whole-paragraph-turbo. Operator judges by ear. + +## 6. Acceptance / A/B + +- **Latency:** first-audio < ~0.8s on the deployment GPU. +- **No starvation:** stream stays ahead of 1× playback (assert in a sim). +- **Quality:** operator ear-A/B the adaptive-chunk output vs whole-paragraph + one-shot — the join-context loss should be ~imperceptible for multi-sentence + text. Samples → `~/chatterbox-ab/`. + +## 7. Existing A/B samples (this session, GLaDOS voice) on nh3-dev `~/chatterbox-ab/` +- `01_sentence_level_turbo.wav` — naive per-sentence (the baseline to BEAT). +- `02_chunked_native_streamed.wav` — abandoned native attempt (artifacty). +- `03_chunked_oneshot.wav` — chunked-attention one-shot. +(The adaptive-chunk output and the base-fork output are still to be generated.) + +## 8. Watch-outs +- Don't claim sentence-splitting is lossless (it isn't — prosodic context). +- Don't re-attempt native turbo frame-streaming without operator say-so. +- Push catalog commits to origin promptly (asset-engine CI). +- Use `ssh -t` only when a remote needs sudo; docker on irv-ml1 needs no sudo. +- The 3090 shows ~20.5 GB used at idle (shared dev stack) — expect tight fit.