vh
4549d241a7
Three TTS additions to round out coverage on irv-ml1, each filling a
distinct niche the existing slate doesn't own.
Final coverage matrix (all on irv-ml1):
Kokoro — low-latency English, fixed voice library, ~300ms TTFA
Chatterbox Turbo — low-latency English w/ voice cloning + paralinguistic tags
IndexTTS-2 — English voice cloning + emotion vector / text control
Qwen3-TTS-1.7B-Base — high-quality English voice cloning
CosyVoice 3 — multilingual (Chinese-leaning)
VibeVoice 1.5B — long-form / multi-speaker dialogue
stacks/kokoro:
- port 8193, GPU device 0 (3090)
- pulls ghcr.io/remsky/kokoro-fastapi-gpu:v0.2.4-master (no Dockerfile,
no first-run model download — models baked in)
- 60+ built-in voices, OpenAI-compat with stream=true over chunked HTTP
- Apache-2.0 weights + code, ~1 GB VRAM
stacks/vibevoice:
- port 8194, GPU device 1 (A6000 — for 7B headroom)
- builds groxaxo/VibeVoice-FastAPI1 (more current fork of ncoder-ai)
pinned to 7614c469a145
- default model microsoft/VibeVoice-1.5B (~7 GB bf16 VRAM); env var
swap to rsxdalv/VibeVoice-Large (7B) or FabioSarracino/VibeVoice-Large-Q8
- multi-speaker dialogue via /v1/vibevoice/generate with Speaker N: format
- long-form niche only — not low-latency
stacks/chatterbox:
- port 8196, GPU device 0 (3090)
- builds devnen/Chatterbox-TTS-Server (most active Turbo-supporting wrapper)
- default model ResembleAI/chatterbox-turbo (~2.5 GB fp16, ~75ms latency)
- paralinguistic tags inline ([laugh] [whisper] etc) — different shape
from IndexTTS-2's emotion vector; fills the speed+cloning niche
Kokoro/IndexTTS don't cover together
- mandatory PerTh watermark on outputs (Resemble policy)
Three matching playbooks under playbooks/deploy-{kokoro,vibevoice,
chatterbox}.yaml. All idempotent, creates-/when-gated.
Cold-deploy disk on /worktank/: ~7 GB Kokoro + ~19 GB VibeVoice 1.5B
+ ~12 GB Chatterbox = ~38 GB total. VRAM concurrent: ~10-11 GB across
both GPUs.
Skipped from the original four-stack proposal: VibeVoice Realtime
(overlaps Kokoro's niche; Kokoro wins on latency, license, and not
needing a build).
116 lines
4.1 KiB
Markdown
116 lines
4.1 KiB
Markdown
# VibeVoice
|
|
|
|
Microsoft's diffusion-based long-form TTS, served via
|
|
[groxaxo/VibeVoice-FastAPI1](https://github.com/groxaxo/VibeVoice-FastAPI1)
|
|
(a recent fork of [ncoder-ai/VibeVoice-FastAPI](https://github.com/ncoder-ai/VibeVoice-FastAPI)
|
|
which moves faster than upstream).
|
|
|
|
Model: [microsoft/VibeVoice-1.5B](https://huggingface.co/microsoft/VibeVoice-1.5B)
|
|
by default. Switch to the 7B variant via `.env` if you want the
|
|
bigger checkpoint.
|
|
|
|
## Why this stack exists
|
|
|
|
Long-form / podcast-quality TTS with native multi-speaker dialogue
|
|
support. Designed for one-shot generation of multi-minute scripts
|
|
where conversation flow matters. **Not** for low-latency single-line
|
|
synthesis — for that use Kokoro or Chatterbox Turbo.
|
|
|
|
| | use case |
|
|
|---|---|
|
|
| **VibeVoice 1.5B** | long-form, multi-speaker dialogue (this stack) |
|
|
| Kokoro | low-latency English, fixed voice library |
|
|
| Chatterbox Turbo | low-latency English w/ voice cloning |
|
|
| IndexTTS-2 | English voice cloning + emotion control |
|
|
| Qwen3-TTS-1.7B-Base | high-quality English voice cloning |
|
|
| CosyVoice 3 | multilingual (Chinese-leaning) |
|
|
|
|
## API
|
|
|
|
OpenAI-compat at `http://10.100.79.3:8194`:
|
|
|
|
```bash
|
|
# Single-speaker (OpenAI-style).
|
|
curl -fsS -X POST http://10.100.79.3:8194/v1/audio/speech \
|
|
-H 'Content-Type: application/json' \
|
|
-d '{"model":"vibevoice","input":"Hello there.","voice":"voice-name","response_format":"wav"}' \
|
|
> out.wav
|
|
|
|
# Multi-speaker dialogue — the headline feature. Format the input
|
|
# as a script with `Speaker N:` prefixes (0-indexed). The wrapper's
|
|
# extended /v1/vibevoice/generate endpoint handles voice switching.
|
|
curl -fsS -X POST http://10.100.79.3:8194/v1/vibevoice/generate \
|
|
-H 'Content-Type: application/json' \
|
|
-d '{
|
|
"script":"Speaker 0: Welcome to the show.\nSpeaker 1: Glad to be here.\nSpeaker 0: Today we discuss…",
|
|
"voices":["voice-host","voice-guest"],
|
|
"stream":true
|
|
}' > podcast.wav
|
|
|
|
# List available voices.
|
|
curl http://10.100.79.3:8194/v1/audio/voices
|
|
```
|
|
|
|
OpenAPI / docs at `/docs`. Healthcheck at `/health`.
|
|
|
|
`stream=true` is honored on the multi-speaker endpoint; the
|
|
single-shot OpenAI endpoint returns the full file in one go.
|
|
|
|
## Voices
|
|
|
|
Drop `.wav` / `.mp3` / `.flac` / `.m4a` into
|
|
`/worktank/vibevoice/voices/` on the host (mounted read-only into
|
|
the container). Restart the container after adding; the wrapper
|
|
scans the dir at init, not per-request:
|
|
|
|
```bash
|
|
ssh irv-ml1 'cd /opt/docker/compose/vibevoice && docker compose restart'
|
|
```
|
|
|
|
VibeVoice also has built-in voice presets (Carter, Davis, Emma,
|
|
Frank, Grace, Mike, Samuel) accessible by name. Microsoft has not
|
|
released the cloning tooling so you can't add new "trained" voices
|
|
— but the bundled ones already cover most podcast use cases.
|
|
|
|
## Deploy
|
|
|
|
```bash
|
|
scripts/elway irv-ml1 --playbook playbooks/deploy-vibevoice.yaml
|
|
```
|
|
|
|
Cold deploy budget:
|
|
- ~12 GB image build (CUDA 12.8 + torch 2.8 + flash-attn)
|
|
- ~7 GB model download (VibeVoice-1.5B) on first start
|
|
- **Total: ~19 GB on /worktank/vibevoice/**
|
|
|
|
First build: ~12 min. First generation: ~30-60 s warmup.
|
|
|
|
## Switching to the 7B variant
|
|
|
|
```bash
|
|
ssh irv-ml1 '
|
|
cd /opt/docker/compose/vibevoice
|
|
sed -i "s|^VIBEVOICE_MODEL=.*|VIBEVOICE_MODEL=rsxdalv/VibeVoice-Large|" .env
|
|
docker compose up -d
|
|
'
|
|
```
|
|
|
|
The 7B model auto-downloads on next start (~18 GB). VRAM jumps from
|
|
~7 GB to ~18 GB bf16 — keep `VIBEVOICE_GPU_DEVICES=1` (A6000) for it.
|
|
For lower VRAM at slight quality cost, set `VIBEVOICE_QUANT=int8_torchao`
|
|
which brings 7B down to ~10 GB.
|
|
|
|
## Gotchas
|
|
|
|
- **Not streaming-friendly for single-line use.** The diffusion head
|
|
has to denoise the whole latent before vocoding. Streaming on
|
|
`/v1/vibevoice/generate` works at script-segment granularity
|
|
(paragraph-ish), not token-by-token.
|
|
- **Voice cloning isn't published.** Microsoft released the inference
|
|
models but not the training pipeline. Use the built-in voices, or
|
|
pick another stack (IndexTTS-2 / Qwen3-TTS / Chatterbox Turbo).
|
|
- **`flash_attention_2`** is the upstream default; if your GPU/torch
|
|
combo doesn't have it built, set `VIBEVOICE_ATTN=sdpa` in `.env`
|
|
to fall back to PyTorch's scaled-dot-product attention.
|
|
- **License**: VibeVoice MIT (Microsoft); wrapper MIT.
|