vh
16d018ff96
Adds the three premier 2026 TTS releases we missed during the original
fleet build-out (early April), all licensed for self-host:
* Fish Audio S2-Pro (port 8195, GPU 1 / A6000) — released 2026-03-09.
4B dual-AR (Slow + Fast) trained on 10M+ hours / 80+ languages.
Headline: 15,000+ paralinguistic / emotion tags via natural language
([laugh] [whispers] [super happy] etc.) — a step-function over
Chatterbox Turbo's 9 fixed tags. 91.61% paralinguistic win rate on
EmergentTTS-Eval. ~150 ms streaming TTFB, voice cloning, MIT-style
open. ~17 GB VRAM.
* Voxtral TTS (port 8197, GPU 1 / A6000) — Mistral, released 2026-03-28.
4B open-weight, 70 ms model latency, 9.7× realtime. 68.4% blind A/B
win rate vs ElevenLabs Flash v2.5 in cloning. 8 languages
(EN/FR/DE/ES/IT/PT/NL/HI). Served via vLLM-Omni (Mistral's partner
serving stack) — published Docker image, no local build. ~16 GB VRAM.
CC BY-NC license — personal/research use only; flagged in README.
* Kyutai TTS (port 8198, GPU 0 / 3090) — kyutai/tts-1.6b-en_fr.
Trained on 2.5M hours from the Moshi/Mimi team. Claimed 220 ms in
solo setup, 32 simultaneous streams under 350 ms on L40. Kyutai's
official deploy is Rust + websockets only; using NillPointer's
community OpenAI-compat wrapper to bridge to /v1/audio/speech so
it slots into the same bench harness. ~4-6 GB VRAM.
Each stack: compose.yaml (build context, env, volumes, healthcheck,
homepage label), .env.example (all tunables documented), README.md
(why it exists, headline numbers, API, deploy + hardware notes).
Playbooks at playbooks/deploy-{fish-s2,voxtral,kyutai-tts}.yaml are
idempotent in the same shape as the existing deploy-vibevoice /
deploy-chatterbox playbooks.
Port allocations on irv-ml1 after this lands: 8188 ComfyUI, 8190
CosyVoice, 8191 Qwen3-TTS, 8192 IndexTTS-2, 8193 Kokoro, 8194
VibeVoice, 8195 Fish, 8196 Chatterbox, 8197 Voxtral, 8198 Kyutai,
8765 Parakeet ASR.
84 lines
3.0 KiB
Markdown
84 lines
3.0 KiB
Markdown
# Kyutai TTS
|
|
|
|
[kyutai/tts-1.6b-en_fr](https://huggingface.co/kyutai/tts-1.6b-en_fr)
|
|
— Kyutai's flagship streaming TTS (1.6B params, EN/FR bilingual,
|
|
trained on 2.5M hours), served via the
|
|
[NillPointer/Kyutai-TTS-Server](https://github.com/NillPointer/Kyutai-TTS-Server)
|
|
community OpenAI-compatible wrapper.
|
|
|
|
## Why this stack exists
|
|
|
|
Kyutai's claim is the **lowest streaming latency in this size class**:
|
|
220 ms in solo setup; up to 32 simultaneous streams under 350 ms on
|
|
a single L40-class GPU. Worth bench-comparing against:
|
|
|
|
| | claimed latency | use case |
|
|
|---|---|---|
|
|
| **Kyutai TTS** | **~220 ms** | streaming EN/FR, conversational dialogue heritage |
|
|
| Kokoro | ~26 ms TTFB measured | low-latency English, fixed voice library |
|
|
| Chatterbox Turbo | ~1.2 s TTFB measured | English w/ cloning + 9 paralinguistic tags |
|
|
| Fish Audio S2-Pro | ~150 ms claimed | richest paralinguistic English |
|
|
| Voxtral | ~70 ms model latency | multilingual EN/FR/DE/ES/IT/PT/NL/HI |
|
|
|
|
## Deployment notes
|
|
|
|
Kyutai's official deployment path is **Rust + websockets only** (no
|
|
HTTP, no OpenAI-compat). That doesn't fit the OpenAI-`/v1/audio/speech`
|
|
contract the rest of our TTS fleet uses. The
|
|
NillPointer/Kyutai-TTS-Server community wrapper bridges Kyutai's
|
|
native streaming to the OpenAI HTTP shape, which lets us slot it
|
|
into the same bench harness as the others.
|
|
|
|
**Tradeoff**: the wrapper adds Python overhead on the request path,
|
|
so measured latency on this stack will be *higher* than Kyutai's
|
|
220 ms claim (which is for the bare Rust server). If we measure
|
|
~400-500 ms TTFB end-to-end, the wrapper is the floor — Kyutai itself
|
|
is hitting its target.
|
|
|
|
## Architecture heritage
|
|
|
|
Kyutai's TTS shares the **Mimi** neural codec + **Moshi** dialogue
|
|
modeling framework. Both target full-duplex conversational AI (Moshi
|
|
is their flagship speech-text foundation model). The TTS-only model
|
|
is the "synthesis half" of the stack, distilled for low-latency
|
|
streaming.
|
|
|
|
Trained on **2.5M hours** — a different scaling regime from the
|
|
others (CosyVoice 5k, Fish 10M, Voxtral undisclosed).
|
|
|
|
## API
|
|
|
|
OpenAI-compat at `http://10.100.79.3:8198`:
|
|
|
|
```bash
|
|
# Single-shot synthesis.
|
|
curl -fsS -X POST http://10.100.79.3:8198/v1/audio/speech \
|
|
-H 'Content-Type: application/json' \
|
|
-d '{"model":"tts-1.6b-en_fr","input":"Hello there.","voice":"default","response_format":"wav"}' \
|
|
> out.wav
|
|
|
|
# Streaming.
|
|
curl -fsS -X POST http://10.100.79.3:8198/v1/audio/speech \
|
|
-H 'Content-Type: application/json' \
|
|
-d '{"model":"tts-1.6b-en_fr","input":"long passage…","voice":"default","stream":true}' \
|
|
| mpv --no-cache -
|
|
|
|
# Built-in voices.
|
|
curl http://10.100.79.3:8198/v1/audio/voices
|
|
```
|
|
|
|
## Deploy
|
|
|
|
```bash
|
|
scripts/elway irv-ml1 --playbook playbooks/deploy-kyutai-tts.yaml
|
|
```
|
|
|
|
First boot pulls the kyutai/tts-1.6b-en_fr checkpoint (~3-6 GB) into
|
|
the HF cache.
|
|
|
|
## Hardware footprint
|
|
|
|
- **VRAM**: ~4-6 GB practical. Pinned to GPU 0 (RTX 3090) by default
|
|
— the A6000 is hosting the heavier Fish S2-Pro / Voxtral.
|
|
- **Disk**: ~6 GB for the checkpoint + HF cache.
|