feat(chatterbox-fast): Phase 1 streaming server — adaptive-chunk scheduler
Build the streaming TTS server MVP per docs/design/chatterbox-fast-plan.md §4. - scheduler.py: adaptive buffer-ratchet chunker (the meat) — GPU-free pure logic. First sentence emitted alone for low TTFA, then chunks ratchet ~3x by packing whole sentences to margin x buffered-audio; drives off measured RTF + sec/char (EMA). relieve_leader() clause-splits a too-big mid-stream sentence to avoid starvation (joins land on commas); a long comma-less sentence is the one honored-but-flagged limitation. - test_scheduler.py: GPU-free simulation, 13 tests — asserts no-starvation (incl. overestimated RTF) and the ratchet. - app.py: FastAPI model holder + POST /tts StreamingResponse (raw PCM s16le default, wav optional, stream/oneshot) + GET /health. - bench.py: client — ground-truth TTFB + real 1x-consumer starvation check. Live test on irv-ml1 (turbo, A6000, GLaDOS voice): streaming TTFB 499ms vs oneshot 5230ms (~10x), stayed ahead of a 1x player (no starvation), ratchet 1.64->4.08->8.60->8.60s audio, measured RTF self-corrected 3.38->4.01. Kill the superseded docs/design/chatterbox-fast.md — its §5 windowed-token streaming was the abandoned native-frame-streaming arc; the adaptive-chunk plan supersedes it. Repoint persistent-memory + README at the canonical plan.
This commit is contained in:
@@ -1,137 +0,0 @@
|
||||
# Chatterbox-Fast — Streaming TTS Engine (design)
|
||||
|
||||
**Status:** Draft / pre-contract design. Spike-validated 2026-06-02.
|
||||
**Owner:** infra-ops · **Workload (operator-confirmed):** single-stream interactive.
|
||||
|
||||
## 1. Goal
|
||||
|
||||
Make Chatterbox-Turbo our **primary interactive TTS engine** by cutting
|
||||
time-to-first-audio from today's ~2.5 s to ~0.3–0.5 s via **true
|
||||
incremental streaming**, while preserving turbo's quality, inline
|
||||
paralinguistic tags, and voice cloning.
|
||||
|
||||
## 2. Why now — the proven result
|
||||
|
||||
Spike (2026-06-02, turbo on the A6000), proper CUDA-sync timing:
|
||||
|
||||
| | time-to-first-audio |
|
||||
|---|---|
|
||||
| today (devnen, no real streaming) | **~2.5 s** |
|
||||
| windowed generate, K=20 tokens | **0.31 s** (delivers ~0.96 s audio) |
|
||||
| K=30 | 0.41 s |
|
||||
| K=50 | 0.57 s |
|
||||
|
||||
~**8× TTFB win.** Decode cost is ~constant **0.12 s** regardless of chunk
|
||||
size (turbo's 2-step decoder), so first-chunk time is dominated by
|
||||
generating the first K tokens — smaller K = faster first audio. Full
|
||||
generation runs at **4.2× realtime**, so once the first chunk plays the
|
||||
generator stays well ahead of playback and the stream never starves.
|
||||
|
||||
## 3. Non-goals
|
||||
|
||||
- High-concurrency **batch throughput** — that's a separate base-on-vLLM
|
||||
lane; deferred (research: vLLM port doesn't support turbo).
|
||||
- **Multilingual** — turbo is EN-only; out of scope.
|
||||
- **Quantization** — research flags it high-risk (gibberish < Q8, Q8-CUDA
|
||||
broken); deprioritized.
|
||||
|
||||
## 4. Background — turbo internals (from the spike)
|
||||
|
||||
- `generate()` = `t3.inference_turbo()` (AR loop, all tokens) →
|
||||
`s3gen.inference(..., n_cfm_timesteps=2)` (flow + HiFT vocoder, all tokens).
|
||||
- `inference_turbo` is a **plain-Python `for` loop** with a KV cache, a
|
||||
`max_gen_len` param, and a stop-token break — cleanly hookable.
|
||||
- `s3gen` decode is cheap per-window (~0.12 s constant).
|
||||
|
||||
## 5. Architecture
|
||||
|
||||
A **lean, purpose-built FastAPI server on the `chatterbox` library** —
|
||||
*not* a fork of devnen. devnen buffers the entire synthesis before
|
||||
emitting (proven: opus/mp3/wav all return first byte at full-synth time);
|
||||
we need to own the generate loop. Components:
|
||||
|
||||
1. **Model holder** — `ChatterboxTurboTTS` loaded once at startup, warmed.
|
||||
2. **Streaming generate** — windowed wrapper around `inference_turbo`:
|
||||
yields token windows; each window decoded via `s3gen` → audio chunk →
|
||||
streamed. (~40 lines on top of the lib, per the spike.)
|
||||
3. **Voice management** — predefined voices (dir of wavs) + clone refs
|
||||
(the `prepare_conditionals` path); reuse chatterbox's `Conditionals`.
|
||||
4. **HTTP API** — `POST /tts` (streaming + non-streaming), optional
|
||||
OpenAI-compat `/v1/audio/speech`, `/health`.
|
||||
5. **Watermark** — Resemble PerTh (mandatory); applied per-chunk or post.
|
||||
|
||||
### Decision 1 — streaming transport
|
||||
**Recommend: HTTP chunked transfer of raw PCM** (client plays chunks as
|
||||
they arrive). Lowest latency, trivial client. Offer opus for
|
||||
bandwidth-constrained callers. *Not* websocket (one-way; overkill).
|
||||
|
||||
### Decision 2 — seam handling (the key productionization detail)
|
||||
Independent per-window decode (as in the spike) can leave faint **seams**
|
||||
at chunk boundaries because the vocoder has receptive-field context.
|
||||
**Approach: overlap-discard** — decode each window with a small lookback
|
||||
of the previous window's trailing tokens, discard that overlap's audio,
|
||||
keep only the new window's output. (The `davidbrowne17/chatterbox-streaming`
|
||||
fork uses this pattern.) Tune the overlap for inaudible seams vs latency.
|
||||
**Validate** by ear + a spectral seam check.
|
||||
|
||||
### Decision 3 — chunk schedule
|
||||
First chunk **small** (K≈20–25 → ~0.3 s first audio); subsequent chunks
|
||||
**larger** (K≈50–100) for decode efficiency, since after chunk 1 we're
|
||||
ahead of playback. A simple ramp.
|
||||
|
||||
## 6. Performance levers (fold in, measure each)
|
||||
|
||||
- **bf16** (Ampere-safe), **TF32** (matmul), **SDPA/flash** backend on the
|
||||
Llama backbone — low-risk, measure the delta.
|
||||
- **torch.compile** — **DEFER.** Research flags a real batch-1 regression
|
||||
risk (documented 0.85× at batch-1). Benchmark separately; adopt only if
|
||||
it beats eager on our hardware. Not on the critical path.
|
||||
|
||||
## 7. Deployment
|
||||
|
||||
- New stack **`chatterbox-fast` deployed ALONGSIDE** the existing
|
||||
`chatterbox` (zero disruption; A/B then cut over).
|
||||
- **Port:** 8197 (next free on irv-ml1).
|
||||
- **GPU placement (decided 2026-06-02):** **3090 (device 0) if it fits,
|
||||
else A6000 (device 1).** The GPU stack is a shared dev stack — workloads
|
||||
float across cards, so the 20.5 GB-at-idle on the 3090 is expected
|
||||
residency, not a blocker. Fit is borderline: turbo is ~2.5 GB but the
|
||||
3090 currently shows ~3.5 GB free, so the deploy step **tries the 3090,
|
||||
falls back to the A6000 (device 1, ~30 GB free, shares with Fish) on
|
||||
OOM.** Pin via `device_ids` in compose per fleet convention.
|
||||
- **From-source Dockerfile** (chatterbox lib + our server), pinned.
|
||||
|
||||
## 8. Benchmark / A-B gate (deploy guard, like the Fish reference_id gate)
|
||||
|
||||
- **first-audio (TTFB)** under target (e.g. < 0.6 s on the deployment GPU).
|
||||
- **realtime factor** maintained (> 3×).
|
||||
- **quality parity** vs current chatterbox — ECAPA speaker-sim for clone
|
||||
voices, listen test for predefined, spectral **seam** check.
|
||||
- Wire as a hard gate in the deploy playbook.
|
||||
|
||||
## 9. Risks / open questions
|
||||
|
||||
1. **Seam artifacts** — mitigation: overlap-discard decode; validate by ear + spectral.
|
||||
2. **torch.compile batch-1 regression** — mitigation: benchmark, optional.
|
||||
3. **3090's 20.5 GB-at-idle** — RESOLVED (non-issue): shared dev stack, expected residency. Placement decided (§7): 3090-if-fits-else-A6000.
|
||||
4. **PerTh watermark on short chunks** — confirm no artifacts per-chunk.
|
||||
5. **Paralinguistic tags across chunk boundaries** — confirm a tag split
|
||||
across windows doesn't break delivery.
|
||||
|
||||
## 10. Build plan (phases)
|
||||
|
||||
0. **Spike** — DONE, proven (§2).
|
||||
1. **Streaming server MVP** — windowed generate + overlap-discard seam
|
||||
handling + `/tts` streaming endpoint; bench first-audio + seam quality.
|
||||
2. **Parity + perf** — predefined + clone voice management; bf16/TF32/SDPA;
|
||||
per-chunk watermark.
|
||||
3. **Containerize + deploy** — from-source Dockerfile; deploy `chatterbox-fast`
|
||||
alongside; wire the A-B gate.
|
||||
4. **Cutover** — switch the catalog route; burn-in; deprecate the old stack.
|
||||
|
||||
## 11. Open decisions for operator
|
||||
|
||||
- ~~GPU placement~~ — **DECIDED (2026-06-02):** 3090 if it fits, else A6000 (§7).
|
||||
- ~~Cutover strategy~~ — **DECIDED (2026-06-02): parallel catalog entry**,
|
||||
burn-in beside the live `chatterbox`, then flip the route once it earns
|
||||
trust. Phases 1–3 are cutover-agnostic; the flip happens in Phase 4.
|
||||
Reference in New Issue
Block a user