chatterbox-fast/README.md

# chatterbox-fast

**Sub-second streaming text-to-speech on [Chatterbox-Turbo](https://github.com/resemble-ai/chatterbox), with full quality.**

`chatterbox-fast` is a small, self-contained streaming server that delivers the
**first audio in ~0.5 seconds** instead of waiting ~5 seconds for a whole
paragraph to synthesize — without the quality loss of naive sentence-by-sentence
splitting. It does this with a scheduling trick (**adaptive buffer-ratchet
chunking**) rather than any model surgery, so it rides whatever quality and voice
cloning Chatterbox-Turbo gives you.

```
                 time-to-first-audio
  one-shot   ████████████████████████  ~5.2 s
  fast       ██▌                        ~0.5 s   ← chatterbox-fast
```

- 🚀 **~0.5 s time-to-first-audio** (vs ~5 s one-shot), measured on an RTX A6000
- 🎚️ **No quality compromise** — chunk joins land on natural sentence pauses, and
  the stream converges to one large, full-context chunk within 2-3 joins
- 🔌 **Drop-in HTTP** — `POST /tts` streams raw PCM or WAV; trivial to consume
- 🗣️ **Voice cloning** — any 5-30 s reference clip; ships with starter voices
- 🧪 **The scheduler is GPU-free and unit-tested** — the no-starvation guarantee
  is proven in a pure simulation
- 📦 **Self-contained** — one image, weights auto-download on first run
- ⚖️ **MIT licensed**

---

## Why it works: adaptive buffer-ratchet chunking

The whole idea rests on one fact: **Chatterbox-Turbo generates faster than
realtime** (~3.4–4× on a modern GPU). That headroom is the fuel.

1. **Chunk 1 = the first sentence**, generated alone and emitted immediately. This
   is the latency-critical part — keep it short, get audio out fast.
2. **While chunk N plays, generate chunk N+1** by greedily packing whole sentences
   until the next one would take longer to generate than the audio you have
   buffered (times a safety margin). You never split mid-sentence, so every chunk
   stays prosodically coherent and joins fall on natural pauses.
3. Because generation outruns playback, **each chunk's playback buys wall-clock for
   a ~3× bigger next chunk.** After 2-3 joins the rest of the text is one big
   near-full-context chunk — so context loss is confined to a couple of sentence
   boundaries, not every sentence.
4. The scheduler **measures the realtime factor live** and self-corrects, so it
   adapts to your GPU instead of trusting a constant.

> **This only works because the engine is faster than realtime.** A sub-realtime
> model would starve no matter how you chunk it — which is exactly why this is a
> Chatterbox-specific design. The no-starvation property is asserted in
> `tests/test_scheduler.py`, which simulates the whole stream without a GPU.

## Quickstart

```bash
docker build -t chatterbox-fast .
docker run --rm --gpus all -p 8197:8197 \
  -e HF_TOKEN=hf_your_token_here \
  -v "$HOME/.cache/huggingface:/app/hf_cache" \
  chatterbox-fast
```

On first run it downloads the Chatterbox-Turbo weights (~6 GB) from Hugging Face
into the mounted cache.

> **You need a (free) Hugging Face token.** The Chatterbox-Turbo model is
> [MIT-licensed and public](https://huggingface.co/ResembleAI/chatterbox-turbo),
> but the underlying `chatterbox-tts` package requires a token to be present when
> it downloads the weights. Any valid token works — `read` scope is enough; grab
> one at <https://huggingface.co/settings/tokens>. Once the weights are cached,
> later runs reuse them.

Then:

```bash
# stream raw PCM and play it as it arrives
curl -N -X POST http://localhost:8197/tts \
  -H 'Content-Type: application/json' \
  -d '{"text":"The cake is a lie. But the streaming is real.","format":"wav"}' \
  > out.wav
```

A reference streaming client that measures real time-to-first-byte and checks for
starvation lives in [`bench.py`](bench.py):

```bash
python bench.py --host http://localhost:8197 --out out.wav
```

## HTTP API

### `POST /tts` → streamed audio

```json
{
  "text": "Your text, with optional [laugh] [whispers] [sigh] tags.",
  "voice": "<name or absolute path to a reference wav>",
  "format": "pcm",            // "pcm" (raw s16le @24kHz, lowest latency) or "wav"
  "stream": true,             // false = whole-text one-shot
  "temperature": 0.8,
  "top_p": 0.95,
  "top_k": 1000,
  "repetition_penalty": 1.2,
  "seed": 0                   // 0 = random; a fixed seed repeats a one-shot take
}
```

The response is a chunked HTTP stream — read it incrementally to get the
low-latency benefit. `format: "pcm"` is raw signed 16-bit little-endian mono at
24 kHz; `format: "wav"` adds a header (a complete header for one-shot, an
open-ended one for streaming).

### `GET /voices` → `{ "voices": [...], "default": "..." }`
Lists the predefined voice names (the `*.wav` stems in the voices directory).

### `GET /health` → readiness
`{ "status": "ok", ... }` once the model is loaded.

## Voices

A **voice** is just a 5-30 s reference WAV. The server clones it on the fly. Point
`CBF_VOICES_DIR` at a directory of `*.wav` files — the file stem becomes the voice
name in `/voices`, and the first one (or `CBF_DEFAULT_VOICE`) is the default.

```bash
docker run --rm --gpus all -p 8197:8197 \
  -v "$HOME/.cache/huggingface:/app/hf_cache" \
  -v "$PWD/my-voices:/app/voices" \
  -e CBF_DEFAULT_VOICE=my_narrator \
  chatterbox-fast
```

The image ships with three **public-domain** starter voices — `catharine`,
`peter`, and `kara` (LibriVox readings; see
[voices/ATTRIBUTION.md](voices/ATTRIBUTION.md)) — so a fresh container works
immediately. Drop in your own clips to add voices — no restart needed for
`/voices` discovery. A clone reference can also be passed per-request as an
absolute path in the `voice` field.

## Configuration

| env var | default | meaning |
|---|---|---|
| `CBF_MODEL_DEVICE` | `cuda` | `cuda` / `cuda:0` / `cpu` |
| `CBF_VOICES_DIR` | `/app/voices` | directory of predefined voice wavs |
| `CBF_DEFAULT_VOICE` | first wav in dir | default voice (name or path) |
| `CBF_BIND` / `CBF_PORT` | `0.0.0.0` / `8197` | server bind |
| `CBF_TF32` / `CBF_SDPA_FLASH` | `1` / `1` | low-risk Ampere+ speed levers |

## Requirements

- An NVIDIA GPU that runs Chatterbox-Turbo **faster than realtime** (any recent
  card does; the design depends on it). ~6 GB VRAM for the fp32 model.
- The Docker NVIDIA runtime (`--gpus`).
- Workload assumption: **single-stream interactive** (one request at a time;
  generation is serialized under a lock).

## Development

```bash
pip install -e ".[dev]"
pytest                       # the scheduler simulation — no GPU required
```

The engine is split so the interesting part is testable without hardware:

| file | role |
|---|---|
| `chatterbox_fast/scheduler.py` | the adaptive-chunk scheduler — pure logic, no GPU |
| `chatterbox_fast/app.py` | FastAPI server + model holder |
| `tests/test_scheduler.py` | GPU-free simulation: asserts no-starvation + the ratchet |
| `bench.py` | streaming client: ground-truth TTFB + starvation check |

## Acknowledgements

Built on Resemble AI's [Chatterbox](https://github.com/resemble-ai/chatterbox)
(the `chatterbox-tts` package). Outputs carry Resemble's Perth watermark, applied
by the model.

## License

MIT © 2026 Vuong Hoang. See [LICENSE](LICENSE).