esh-pfi-infrastructure/docs/design/chatterbox-fast.md

# Chatterbox-Fast — Streaming TTS Engine (design)

**Status:** Draft / pre-contract design. Spike-validated 2026-06-02.
**Owner:** infra-ops · **Workload (operator-confirmed):** single-stream interactive.

## 1. Goal

Make Chatterbox-Turbo our **primary interactive TTS engine** by cutting
time-to-first-audio from today's ~2.5 s to ~0.3–0.5 s via **true
incremental streaming**, while preserving turbo's quality, inline
paralinguistic tags, and voice cloning.

## 2. Why now — the proven result

Spike (2026-06-02, turbo on the A6000), proper CUDA-sync timing:

| | time-to-first-audio |
|---|---|
| today (devnen, no real streaming) | **~2.5 s** |
| windowed generate, K=20 tokens | **0.31 s** (delivers ~0.96 s audio) |
| K=30 | 0.41 s |
| K=50 | 0.57 s |

~**8× TTFB win.** Decode cost is ~constant **0.12 s** regardless of chunk
size (turbo's 2-step decoder), so first-chunk time is dominated by
generating the first K tokens — smaller K = faster first audio. Full
generation runs at **4.2× realtime**, so once the first chunk plays the
generator stays well ahead of playback and the stream never starves.

## 3. Non-goals

- High-concurrency **batch throughput** — that's a separate base-on-vLLM
  lane; deferred (research: vLLM port doesn't support turbo).
- **Multilingual** — turbo is EN-only; out of scope.
- **Quantization** — research flags it high-risk (gibberish < Q8, Q8-CUDA
  broken); deprioritized.

## 4. Background — turbo internals (from the spike)

- `generate()` = `t3.inference_turbo()` (AR loop, all tokens) →
  `s3gen.inference(..., n_cfm_timesteps=2)` (flow + HiFT vocoder, all tokens).
- `inference_turbo` is a **plain-Python `for` loop** with a KV cache, a
  `max_gen_len` param, and a stop-token break — cleanly hookable.
- `s3gen` decode is cheap per-window (~0.12 s constant).

## 5. Architecture

A **lean, purpose-built FastAPI server on the `chatterbox` library** —
*not* a fork of devnen. devnen buffers the entire synthesis before
emitting (proven: opus/mp3/wav all return first byte at full-synth time);
we need to own the generate loop. Components:

1. **Model holder** — `ChatterboxTurboTTS` loaded once at startup, warmed.
2. **Streaming generate** — windowed wrapper around `inference_turbo`:
   yields token windows; each window decoded via `s3gen` → audio chunk →
   streamed. (~40 lines on top of the lib, per the spike.)
3. **Voice management** — predefined voices (dir of wavs) + clone refs
   (the `prepare_conditionals` path); reuse chatterbox's `Conditionals`.
4. **HTTP API** — `POST /tts` (streaming + non-streaming), optional
   OpenAI-compat `/v1/audio/speech`, `/health`.
5. **Watermark** — Resemble PerTh (mandatory); applied per-chunk or post.

### Decision 1 — streaming transport
**Recommend: HTTP chunked transfer of raw PCM** (client plays chunks as
they arrive). Lowest latency, trivial client. Offer opus for
bandwidth-constrained callers. *Not* websocket (one-way; overkill).

### Decision 2 — seam handling (the key productionization detail)
Independent per-window decode (as in the spike) can leave faint **seams**
at chunk boundaries because the vocoder has receptive-field context.
**Approach: overlap-discard** — decode each window with a small lookback
of the previous window's trailing tokens, discard that overlap's audio,
keep only the new window's output. (The `davidbrowne17/chatterbox-streaming`
fork uses this pattern.) Tune the overlap for inaudible seams vs latency.
**Validate** by ear + a spectral seam check.

### Decision 3 — chunk schedule
First chunk **small** (K≈20–25 → ~0.3 s first audio); subsequent chunks
**larger** (K≈50–100) for decode efficiency, since after chunk 1 we're
ahead of playback. A simple ramp.

## 6. Performance levers (fold in, measure each)

- **bf16** (Ampere-safe), **TF32** (matmul), **SDPA/flash** backend on the
  Llama backbone — low-risk, measure the delta.
- **torch.compile** — **DEFER.** Research flags a real batch-1 regression
  risk (documented 0.85× at batch-1). Benchmark separately; adopt only if
  it beats eager on our hardware. Not on the critical path.

## 7. Deployment

- New stack **`chatterbox-fast` deployed ALONGSIDE** the existing
  `chatterbox` (zero disruption; A/B then cut over).
- **Port:** 8197 (next free on irv-ml1).
- **GPU placement (decided 2026-06-02):** **3090 (device 0) if it fits,
  else A6000 (device 1).** The GPU stack is a shared dev stack — workloads
  float across cards, so the 20.5 GB-at-idle on the 3090 is expected
  residency, not a blocker. Fit is borderline: turbo is ~2.5 GB but the
  3090 currently shows ~3.5 GB free, so the deploy step **tries the 3090,
  falls back to the A6000 (device 1, ~30 GB free, shares with Fish) on
  OOM.** Pin via `device_ids` in compose per fleet convention.
- **From-source Dockerfile** (chatterbox lib + our server), pinned.

## 8. Benchmark / A-B gate (deploy guard, like the Fish reference_id gate)

- **first-audio (TTFB)** under target (e.g. < 0.6 s on the deployment GPU).
- **realtime factor** maintained (> 3×).
- **quality parity** vs current chatterbox — ECAPA speaker-sim for clone
  voices, listen test for predefined, spectral **seam** check.
- Wire as a hard gate in the deploy playbook.

## 9. Risks / open questions

1. **Seam artifacts** — mitigation: overlap-discard decode; validate by ear + spectral.
2. **torch.compile batch-1 regression** — mitigation: benchmark, optional.
3. **3090's 20.5 GB-at-idle** — RESOLVED (non-issue): shared dev stack, expected residency. Placement decided (§7): 3090-if-fits-else-A6000.
4. **PerTh watermark on short chunks** — confirm no artifacts per-chunk.
5. **Paralinguistic tags across chunk boundaries** — confirm a tag split
   across windows doesn't break delivery.

## 10. Build plan (phases)

0. **Spike** — DONE, proven (§2).
1. **Streaming server MVP** — windowed generate + overlap-discard seam
   handling + `/tts` streaming endpoint; bench first-audio + seam quality.
2. **Parity + perf** — predefined + clone voice management; bf16/TF32/SDPA;
   per-chunk watermark.
3. **Containerize + deploy** — from-source Dockerfile; deploy `chatterbox-fast`
   alongside; wire the A-B gate.
4. **Cutover** — switch the catalog route; burn-in; deprecate the old stack.

## 11. Open decisions for operator

- ~~GPU placement~~ — **DECIDED (2026-06-02):** 3090 if it fits, else A6000 (§7).
- ~~Cutover strategy~~ — **DECIDED (2026-06-02): parallel catalog entry**,
  burn-in beside the live `chatterbox`, then flip the route once it earns
  trust. Phases 1–3 are cutover-agnostic; the flip happens in Phase 4.