esh-pfi-infrastructure/docs/design/chatterbox-fast-plan.md

Plan of Attack — chatterbox-fast streaming TTS engine

Authored 2026-06-02 for a fresh-context build session. Self-contained: you should not need the prior conversation. Cross-refs: docs/design/chatterbox-fast.md (design), persistent-memory.md (durable state + the abandoned native-streaming arc), repo ~/development/eshpfi-management on host nh3-dev.

---

0. Mission

Chatterbox(-Turbo) is becoming our main TTS engine. Build chatterbox-fast: a custom streaming server + container that delivers sub-second time-to-first- audio while keeping turbo's full quality. Workload = single-stream interactive. Operator authorized high effort incl. building the container from source. Deploy as a parallel stack beside the live chatterbox (:8196), burn in, then flip the catalog route.

1. THE design — adaptive buffer-ratchet chunking (operator's idea; chosen)

Why not the alternatives (settled this session, don't relitigate):

• Whole-paragraph one-shot = best quality but ~2.5s+ TTFB (no streaming).
• Naive per-sentence split = fast but loses cross-sentence prosodic context → real quality loss (the T3 AR backbone conditions prosody on the WHOLE text: contextual delivery, declination, affect continuity). "No artifacts" ≠ "no quality loss." Operator corrected this; don't claim otherwise.
• Native frame-level streaming on turbo = ABANDONED (turbo's flow uses full-context attention, static_chunk_size=0 → prefix-unstable; see persistent-memory Tried/abandoned for the full dead-end map). Do NOT re-attempt without explicit operator direction.

The adaptive-chunk algorithm:

1. Split text into sentences (and fall back to clause/comma split for a very long FIRST sentence only, to protect first-audio latency).
2. Chunk 1 = first sentence — generate alone, emit immediately (~0.66s first-audio measured for a short sentence). Latency-critical.
3. While chunk N plays, generate chunk N+1 = greedily accumulate WHOLE sentences until the next sentence would exceed the gen-time budget margin × audio_buffered_remaining. Never split mid-sentence (keeps each chunk prosodically self-coherent; joins land at natural sentence pauses).
4. Chunks grow ~3× each (Chatterbox runs ~3.8× realtime; each chunk's playback buys wall-clock for a ~3× bigger next chunk). So after 2-3 chunks, the rest of the paragraph is ONE big chunk with near-full context. Context loss confined to 2-3 joins at sentence boundaries.
5. Drive off MEASURED realtime factor, not a constant — track actual gen-speed live and self-correct. Start margin=0.8; be more conservative on the first transition (smallest buffer = highest starvation risk) — ~0.6-0.7 there, then relax.
6. Optional context-priming at joins (quality-max): prepend the previous sentence as context to a chunk, generate, discard its audio → the chunk's first sentence gets backward context. Cheap on early small chunks; skip once chunks are large. Add this in Phase 2, measure if it's audibly worth it.

Critical enabling fact: this only works because RTF > 1. Fish (<1× realtime) would starve no matter the chunking — that's why this is the chatterbox-specific answer.

2. Validated API + facts (don't re-derive)

• Model: from chatterbox.tts_turbo import ChatterboxTurboTTS
  • m = ChatterboxTurboTTS.from_pretrained(device="cuda") (loads from HF cache)
  • m.prepare_conditionals(wav_path, exaggeration=0.5, norm_loudness=True)
  • wav = m.generate(text, repetition_penalty=1.2, top_p=0.95, temperature=0.8, top_k=1000) → returns watermarked wav tensor shape [1, T], m.sr=24000. (CFG/exaggeration/min_p are ignored by turbo — warns but harmless.)
  • Paralinguistic tags work inline ([laugh] [whispers] [sigh] etc.).
• Architecture: T3 AR Llama 350M → S3Gen flow (2-step meanflow) → HiFTGenerator.
• Realtime: ~3.8× on A6000 (17.7s audio / 4.7s), ~3.4× on 3090.
• First-sentence latency: ~0.66s (short sentence, warm).
• Watermark (Resemble PerTh) is applied inside m.generate — mandatory, fine for internal use.

3. Dev/test pattern (host irv-ml1 = 10.100.79.3, ssh lkraven@10.100.79.3)

• lkraven is in the docker group on irv-ml1 → NO sudo for docker.
• Model weights cached at /worktank/chatterbox/cache (HF_HOME); reference wavs at /worktank/chatterbox/reference_audio (has glados_25s.wav, Imogen.wav).
• One-off GPU container (use GPU 1 / A6000 for dev — 3090 is VRAM-tight):

  IMG=$(docker images --format '{{.Repository}}:{{.Tag}}' | grep -i chatterbox | grep -v '<none>' | head -1)  # local/chatterbox:v1
  docker run --rm --gpus '"device=1"' -e NVIDIA_VISIBLE_DEVICES=1 -e HF_HOME=/app/hf_cache \
    -v /worktank/chatterbox/cache:/app/hf_cache \
    -v /worktank/chatterbox/reference_audio:/refs \
    -v /tmp/yourscript.py:/test.py "$IMG" python /test.py
  

• Lib introspection: docker exec -i chatterbox python - <<'PY' ... PY against the running server container.
• Write A/B samples to /refs/_*.wav, then scp lkraven@10.100.79.3:/worktank/ chatterbox/reference_audio/_*.wav ~/chatterbox-ab/ for the operator to hear.

4. Build phases

Phase 1 — streaming server MVP (the scheduler is the meat):

• stacks/chatterbox-fast/app.py — FastAPI server:
  • Load model once at startup, warm it (one throwaway generate).
  • POST /tts → StreamingResponse of audio chunks. Body: text, voice (predefined name or clone ref), format (raw pcm s16le default for lowest latency; offer wav/opus), the sampling knobs.
  • The adaptive-chunk scheduler (§1): sentence-split → gen first sentence → emit → loop {measure RTF, accumulate sentences to budget, generate, emit}. Track audio_emitted_seconds and wall-clock to estimate buffer drain.
  • GET /health.
• Validate: first-audio latency, that the stream never starves (sim a player consuming at 1× realtime), and produce a sample for the operator vs the whole-paragraph one-shot.

Phase 2 — parity + perf:

• Predefined voices (dir of wavs) + clone refs (prepare_conditionals).
• bf16 (TTS_BF16-style, or set model dtype), TF32 (torch.backends.cuda.matmul.allow_tf32=True), SDPA/flash backend.
• Optional context-priming at joins (§1.6) — measure if audibly worth it.
• torch.compile: DEFER (research flags batch-1 regression; bench separately).

Phase 3 — containerize + deploy:

• stacks/chatterbox-fast/ : compose.yaml, Dockerfile (FROM the chatterbox base image / vendored chatterbox + our app.py), .env.example, README.md.
• Follow repo conventions (CLAUDE.md): traefik-net/tnet, named volumes, restart: unless-stopped, healthcheck, homepage labels, GPU pin via device_ids. Port 8197 (next free on irv-ml1; reserved list in stacks/chatterbox/.env.example). GPU: 3090 (device 0) if turbo fits in free VRAM, else A6000 (device 1) — try 3090, fall back on OOM.
• playbooks/deploy-chatterbox-fast.yaml (model is HF-cached already; reuse /worktank/chatterbox/cache). Add an A/B smoke gate (first-audio < target).
• Deploy alongside the live chatterbox — do NOT disrupt :8196.

Phase 4 — A/B + cutover:

• Add a parallel chatterbox-fast catalog entry in docs/asset-engine/services.yaml (NOT replace chatterbox yet). If it needs a new schema field, that's a catalog_version bump — coordinate with asset-engine-dev via althing (and PUSH the commit promptly; their CI drift-checks against the remote — lesson learned this session).
• Burn-in + operator ear-A/B vs whole-paragraph. Then flip the route.

5. ALSO build for A/B (operator asked): base-chatterbox + streaming fork

• Install davidbrowne17/chatterbox-streaming (a fork with generate_stream(), measured first-chunk ~0.47s on a 4090) — BASE chatterbox model, not turbo. True frame-level streaming but base-model quality. Stand it up (own container / port), generate a sample with the SAME text + a comparable voice, drop in ~/chatterbox-ab/ for the 3-way A/B: adaptive-chunk-turbo vs base-fork-stream vs whole-paragraph-turbo. Operator judges by ear.

6. Acceptance / A/B

• Latency: first-audio < ~0.8s on the deployment GPU.
• No starvation: stream stays ahead of 1× playback (assert in a sim).
• Quality: operator ear-A/B the adaptive-chunk output vs whole-paragraph one-shot — the join-context loss should be ~imperceptible for multi-sentence text. Samples → ~/chatterbox-ab/.

7. Existing A/B samples (this session, GLaDOS voice) on nh3-dev ~/chatterbox-ab/

• 01_sentence_level_turbo.wav — naive per-sentence (the baseline to BEAT).
• 02_chunked_native_streamed.wav — abandoned native attempt (artifacty).
• 03_chunked_oneshot.wav — chunked-attention one-shot. (The adaptive-chunk output and the base-fork output are still to be generated.)

8. Watch-outs

• Don't claim sentence-splitting is lossless (it isn't — prosodic context).
• Don't re-attempt native turbo frame-streaming without operator say-so.
• Push catalog commits to origin promptly (asset-engine CI).
• Use ssh -t only when a remote needs sudo; docker on irv-ml1 needs no sudo.
• The 3090 shows ~20.5 GB used at idle (shared dev stack) — expect tight fit.