esh-pfi-infrastructure/stacks/chatterbox

Chatterbox Turbo

Resemble AI's 350M-param low-latency English TTS with zero-shot voice cloning, served via devnen/Chatterbox-TTS-Server — the most actively-maintained OpenAI-compat wrapper supporting Turbo.

Model: ResembleAI/chatterbox-turbo — released April 2026, ~6× realtime, ~75 ms latency, MIT-licensed.

Why this stack exists

Fills the low-latency English voice-cloning slot none of the other TTS own cleanly: Kokoro is fast but fixed-voice; IndexTTS-2 clones beautifully but is slow; Qwen3-TTS-Base clones at a higher quality bar but isn't tuned for sub-second latency. Chatterbox Turbo trades some fidelity for 6× realtime + 5-second-reference cloning, ideal for real-time voice-agent use cases.

	use case
Chatterbox Turbo	low-latency English w/ voice cloning + paralinguistic tags
Kokoro	low-latency English, fixed voice library
IndexTTS-2	English voice cloning + emotion vector / text control
Qwen3-TTS-1.7B-Base	high-quality English voice cloning
CosyVoice 3	multilingual (Chinese-leaning)
VibeVoice 1.5B	long-form / multi-speaker dialogue

Headline features

• Zero-shot voice cloning from ~5 s reference (base Chatterbox needs ~10 s; Turbo cuts that in half).
• Native paralinguistic tags inline in text — drop these into your prompt and the model honors them:

  [laugh] [cough] [sigh] [gasp] [whisper] [breath]
  

  Different shape from IndexTTS-2's 8-vector emotion control: cleaner for "say it like this" markup directly in the prompt.
• Mandatory PerTh watermark on outputs (Resemble policy, cannot be disabled). Non-issue for internal use; mention it if you ever ship Chatterbox-generated audio externally.

API

OpenAI-compat at http://10.100.79.3:8196:

# Built-in voices.
curl http://10.100.79.3:8196/v1/audio/voices

# Single-shot synthesis with a built-in voice.
curl -fsS -X POST http://10.100.79.3:8196/v1/audio/speech \
  -H 'Content-Type: application/json' \
  -d '{"model":"chatterbox-turbo","input":"Hello there. [laugh] What a day.","voice":"alloy","response_format":"wav"}' \
  > out.wav

# Voice cloning — drop a 5 s reference WAV into
# /worktank/chatterbox/reference_audio/glados.wav, then:
curl -fsS -X POST http://10.100.79.3:8196/v1/audio/speech \
  -H 'Content-Type: application/json' \
  -d '{"model":"chatterbox-turbo","input":"I have all the time in the world.","voice":"glados"}' \
  > glados.wav

# Streaming (where supported by the wrapper).
curl -fsS -X POST http://10.100.79.3:8196/v1/audio/speech \
  -H 'Content-Type: application/json' \
  -d '{"model":"chatterbox-turbo","input":"long passage…","voice":"alloy","stream":true}' \
  | mpv --no-cache -

Native /tts endpoint (devnen wrapper extension) and OpenAPI at /docs. Healthcheck at /health.

Voice library

Drop reference WAV / MP3 / FLAC into /worktank/chatterbox/reference_audio/ on the host. The wrapper discovers new files on next request — no restart needed. Use clean ~5 s clips, single speaker.

Built-in OpenAI-style voice aliases (alloy, echo, fable, onyx, nova, shimmer) map to bundled presets — useful for OpenAI SDK clients that hardcode those names.

Deploy

scripts/elway irv-ml1 --playbook playbooks/deploy-chatterbox.yaml

Cold deploy budget:

• Image build: ~5-8 GB (CUDA + torch + Chatterbox deps)
• Model download: ~6 GB (Chatterbox Turbo weights, first run)
• Total: ~12 GB on /worktank/chatterbox/

First build: ~8-10 min. First synthesis: ~10-30 s warmup.

Switching the model

ssh irv-ml1 '
  cd /opt/docker/compose/chatterbox
  sed -i "s|^CHATTERBOX_MODEL_REPO=.*|CHATTERBOX_MODEL_REPO=ResembleAI/chatterbox|" .env
  docker compose up -d
'

Options for CHATTERBOX_MODEL_REPO:

• ResembleAI/chatterbox-turbo — flagship Turbo (default, fastest)
• ResembleAI/chatterbox — base Chatterbox, 500M, slower but with exaggeration / CFG-weight knobs Turbo doesn't expose
• ResembleAI/chatterbox-multilingual — 23 languages (slower than Turbo, useful if you need non-English on this stack vs CosyVoice 3)

Gotchas

• Python 3.10 only (devnen wrapper). Image bakes that in; not something you'd hit unless you fork the Dockerfile.
• PerTh watermark is unconditional. Can't disable.
• Turbo loses some knobs vs base Chatterbox — no exaggeration or CFG-weight tuning. If you need expressive amplitude control, flip to base Chatterbox via the config swap above.
• Repo is fresh (~weekly commits). Pin to a SHA in .env (CHATTERBOX_SHA=...) and rebuild monthly to ride upstream bug-fix progress.
• License: wrapper MIT; weights MIT (Resemble) — including the watermark requirement.