chatterbox-fast/README.md

chatterbox-fast

Sub-second streaming text-to-speech on Chatterbox-Turbo, with full quality.

chatterbox-fast is a small, self-contained streaming server that delivers the first audio in ~0.5 seconds instead of waiting ~5 seconds for a whole paragraph to synthesize — without the quality loss of naive sentence-by-sentence splitting. It does this with a scheduling trick (adaptive buffer-ratchet chunking) rather than any model surgery, so it rides whatever quality and voice cloning Chatterbox-Turbo gives you.

                 time-to-first-audio
  one-shot   ████████████████████████  ~5.2 s
  fast       ██▌                        ~0.5 s   ← chatterbox-fast

• 🚀 ~0.5 s time-to-first-audio (vs ~5 s one-shot), measured on an RTX A6000
• 🎚️ No quality compromise — chunk joins land on natural sentence pauses, and the stream converges to one large, full-context chunk within 2-3 joins
• 🔌 Drop-in HTTP — POST /tts streams raw PCM or WAV; trivial to consume
• 🗣️ Voice cloning — any 5-30 s reference clip; ships with starter voices
• 🧪 The scheduler is GPU-free and unit-tested — the no-starvation guarantee is proven in a pure simulation
• 📦 Self-contained — one image, weights auto-download on first run
• ⚖️ MIT licensed

---

Why it works: adaptive buffer-ratchet chunking

The whole idea rests on one fact: Chatterbox-Turbo generates faster than realtime (~3.4–4× on a modern GPU). That headroom is the fuel.

1. Chunk 1 = the first sentence, generated alone and emitted immediately. This is the latency-critical part — keep it short, get audio out fast.
2. While chunk N plays, generate chunk N+1 by greedily packing whole sentences until the next one would take longer to generate than the audio you have buffered (times a safety margin). You never split mid-sentence, so every chunk stays prosodically coherent and joins fall on natural pauses.
3. Because generation outruns playback, each chunk's playback buys wall-clock for a ~3× bigger next chunk. After 2-3 joins the rest of the text is one big near-full-context chunk — so context loss is confined to a couple of sentence boundaries, not every sentence.
4. The scheduler measures the realtime factor live and self-corrects, so it adapts to your GPU instead of trusting a constant.

This only works because the engine is faster than realtime. A sub-realtime model would starve no matter how you chunk it — which is exactly why this is a Chatterbox-specific design. The no-starvation property is asserted in tests/test_scheduler.py, which simulates the whole stream without a GPU.

Quickstart

docker build -t chatterbox-fast .
docker run --rm --gpus all -p 8197:8197 \
  -e HF_TOKEN=hf_your_token_here \
  -v "$HOME/.cache/huggingface:/app/hf_cache" \
  chatterbox-fast

On first run it downloads the Chatterbox-Turbo weights (~6 GB) from Hugging Face into the mounted cache.

You need a (free) Hugging Face token. The Chatterbox-Turbo model is MIT-licensed and public, but the underlying chatterbox-tts package requires a token to be present when it downloads the weights. Any valid token works — read scope is enough; grab one at https://huggingface.co/settings/tokens. Once the weights are cached, later runs reuse them.

Then:

# stream raw PCM and play it as it arrives
curl -N -X POST http://localhost:8197/tts \
  -H 'Content-Type: application/json' \
  -d '{"text":"The cake is a lie. But the streaming is real.","format":"wav"}' \
  > out.wav

A reference streaming client that measures real time-to-first-byte and checks for starvation lives in bench.py:

python bench.py --host http://localhost:8197 --out out.wav

HTTP API

POST /tts → streamed audio

{
  "text": "Your text, with optional [laugh] [whispers] [sigh] tags.",
  "voice": "<name or absolute path to a reference wav>",
  "format": "pcm",            // "pcm" (raw s16le @24kHz, lowest latency) or "wav"
  "stream": true,             // false = whole-text one-shot
  "temperature": 0.8,
  "top_p": 0.95,
  "top_k": 1000,
  "repetition_penalty": 1.2,
  "seed": 0                   // 0 = random; a fixed seed repeats a one-shot take
}

The response is a chunked HTTP stream — read it incrementally to get the low-latency benefit. format: "pcm" is raw signed 16-bit little-endian mono at 24 kHz; format: "wav" adds a header (a complete header for one-shot, an open-ended one for streaming).

GET /voices → { "voices": [...], "default": "..." }

Lists the predefined voice names (the *.wav stems in the voices directory).

GET /health → readiness

{ "status": "ok", ... } once the model is loaded.

Voices

A voice is just a 5-30 s reference WAV. The server clones it on the fly. Point CBF_VOICES_DIR at a directory of *.wav files — the file stem becomes the voice name in /voices, and the first one (or CBF_DEFAULT_VOICE) is the default.

docker run --rm --gpus all -p 8197:8197 \
  -v "$HOME/.cache/huggingface:/app/hf_cache" \
  -v "$PWD/my-voices:/app/voices" \
  -e CBF_DEFAULT_VOICE=my_narrator \
  chatterbox-fast

The image ships with three public-domain starter voices — catharine, peter, and kara (LibriVox readings; see voices/ATTRIBUTION.md) — so a fresh container works immediately. Drop in your own clips to add voices — no restart needed for /voices discovery. A clone reference can also be passed per-request as an absolute path in the voice field.

Configuration

env var	default	meaning
CBF_MODEL_DEVICE	cuda	cuda / cuda:0 / cpu
CBF_VOICES_DIR	/app/voices	directory of predefined voice wavs
CBF_DEFAULT_VOICE	first wav in dir	default voice (name or path)
CBF_BIND / CBF_PORT	0.0.0.0 / 8197	server bind
CBF_TF32 / CBF_SDPA_FLASH	1 / 1	low-risk Ampere+ speed levers

Requirements

• An NVIDIA GPU that runs Chatterbox-Turbo faster than realtime (any recent card does; the design depends on it). ~6 GB VRAM for the fp32 model.
• The Docker NVIDIA runtime (--gpus).
• Workload assumption: single-stream interactive (one request at a time; generation is serialized under a lock).

Development

pip install -e ".[dev]"
pytest                       # the scheduler simulation — no GPU required

The engine is split so the interesting part is testable without hardware:

file	role
chatterbox_fast/scheduler.py	the adaptive-chunk scheduler — pure logic, no GPU
chatterbox_fast/app.py	FastAPI server + model holder
tests/test_scheduler.py	GPU-free simulation: asserts no-starvation + the ratchet
bench.py	streaming client: ground-truth TTFB + starvation check

Acknowledgements

Built on Resemble AI's Chatterbox (the chatterbox-tts package). Outputs carry Resemble's Perth watermark, applied by the model.

License

MIT © 2026 Vuong Hoang. See LICENSE.