Chatterbox-Fast — Streaming TTS Engine (design)

Status: Draft / pre-contract design. Spike-validated 2026-06-02. Owner: infra-ops · Workload (operator-confirmed): single-stream interactive.

1. Goal

Make Chatterbox-Turbo our primary interactive TTS engine by cutting time-to-first-audio from today's ~2.5 s to ~0.3–0.5 s via true incremental streaming, while preserving turbo's quality, inline paralinguistic tags, and voice cloning.

2. Why now — the proven result

Spike (2026-06-02, turbo on the A6000), proper CUDA-sync timing:

	time-to-first-audio
today (devnen, no real streaming)	~2.5 s
windowed generate, K=20 tokens	0.31 s (delivers ~0.96 s audio)
K=30	0.41 s
K=50	0.57 s

~8× TTFB win. Decode cost is ~constant 0.12 s regardless of chunk size (turbo's 2-step decoder), so first-chunk time is dominated by generating the first K tokens — smaller K = faster first audio. Full generation runs at 4.2× realtime, so once the first chunk plays the generator stays well ahead of playback and the stream never starves.

3. Non-goals

High-concurrency batch throughput — that's a separate base-on-vLLM lane; deferred (research: vLLM port doesn't support turbo).
Multilingual — turbo is EN-only; out of scope.
Quantization — research flags it high-risk (gibberish < Q8, Q8-CUDA broken); deprioritized.

4. Background — turbo internals (from the spike)

generate() = t3.inference_turbo() (AR loop, all tokens) → s3gen.inference(..., n_cfm_timesteps=2) (flow + HiFT vocoder, all tokens).
inference_turbo is a plain-Python for loop with a KV cache, a max_gen_len param, and a stop-token break — cleanly hookable.
s3gen decode is cheap per-window (~0.12 s constant).

5. Architecture

A lean, purpose-built FastAPI server on the chatterbox library — not a fork of devnen. devnen buffers the entire synthesis before emitting (proven: opus/mp3/wav all return first byte at full-synth time); we need to own the generate loop. Components:

Model holder — ChatterboxTurboTTS loaded once at startup, warmed.
Streaming generate — windowed wrapper around inference_turbo: yields token windows; each window decoded via s3gen → audio chunk → streamed. (~40 lines on top of the lib, per the spike.)
Voice management — predefined voices (dir of wavs) + clone refs (the prepare_conditionals path); reuse chatterbox's Conditionals.
HTTP API — POST /tts (streaming + non-streaming), optional OpenAI-compat /v1/audio/speech, /health.
Watermark — Resemble PerTh (mandatory); applied per-chunk or post.

Decision 1 — streaming transport

Recommend: HTTP chunked transfer of raw PCM (client plays chunks as they arrive). Lowest latency, trivial client. Offer opus for bandwidth-constrained callers. Not websocket (one-way; overkill).

Decision 2 — seam handling (the key productionization detail)

Independent per-window decode (as in the spike) can leave faint seams at chunk boundaries because the vocoder has receptive-field context. Approach: overlap-discard — decode each window with a small lookback of the previous window's trailing tokens, discard that overlap's audio, keep only the new window's output. (The davidbrowne17/chatterbox-streaming fork uses this pattern.) Tune the overlap for inaudible seams vs latency. Validate by ear + a spectral seam check.

Decision 3 — chunk schedule

First chunk small (K≈20–25 → ~0.3 s first audio); subsequent chunks larger (K≈50–100) for decode efficiency, since after chunk 1 we're ahead of playback. A simple ramp.

6. Performance levers (fold in, measure each)

bf16 (Ampere-safe), TF32 (matmul), SDPA/flash backend on the Llama backbone — low-risk, measure the delta.
torch.compile — DEFER. Research flags a real batch-1 regression risk (documented 0.85× at batch-1). Benchmark separately; adopt only if it beats eager on our hardware. Not on the critical path.

7. Deployment

New stack chatterbox-fast deployed ALONGSIDE the existing chatterbox (zero disruption; A/B then cut over).
Port: 8197 (next free on irv-ml1).
GPU placement (decided 2026-06-02): 3090 (device 0) if it fits, else A6000 (device 1). The GPU stack is a shared dev stack — workloads float across cards, so the 20.5 GB-at-idle on the 3090 is expected residency, not a blocker. Fit is borderline: turbo is ~2.5 GB but the 3090 currently shows ~3.5 GB free, so the deploy step tries the 3090, falls back to the A6000 (device 1, ~30 GB free, shares with Fish) on OOM. Pin via device_ids in compose per fleet convention.
From-source Dockerfile (chatterbox lib + our server), pinned.

8. Benchmark / A-B gate (deploy guard, like the Fish reference_id gate)

first-audio (TTFB) under target (e.g. < 0.6 s on the deployment GPU).
realtime factor maintained (> 3×).
quality parity vs current chatterbox — ECAPA speaker-sim for clone voices, listen test for predefined, spectral seam check.
Wire as a hard gate in the deploy playbook.

9. Risks / open questions

Seam artifacts — mitigation: overlap-discard decode; validate by ear + spectral.
torch.compile batch-1 regression — mitigation: benchmark, optional.
3090's 20.5 GB-at-idle — RESOLVED (non-issue): shared dev stack, expected residency. Placement decided (§7): 3090-if-fits-else-A6000.
PerTh watermark on short chunks — confirm no artifacts per-chunk.
Paralinguistic tags across chunk boundaries — confirm a tag split across windows doesn't break delivery.

10. Build plan (phases)

Spike — DONE, proven (§2).
Streaming server MVP — windowed generate + overlap-discard seam handling + /tts streaming endpoint; bench first-audio + seam quality.
Parity + perf — predefined + clone voice management; bf16/TF32/SDPA; per-chunk watermark.
Containerize + deploy — from-source Dockerfile; deploy chatterbox-fast alongside; wire the A-B gate.
Cutover — switch the catalog route; burn-in; deprecate the old stack.

11. Open decisions for operator

~~GPU placement~~ — DECIDED (2026-06-02): 3090 if it fits, else A6000 (§7).
~~Cutover strategy~~ — DECIDED (2026-06-02): parallel catalog entry, burn-in beside the live chatterbox, then flip the route once it earns trust. Phases 1–3 are cutover-agnostic; the flip happens in Phase 4.

6.5 KiB Raw Blame History Unescape Escape