esh-pfi-infrastructure/stacks/vibevoice

VibeVoice

Microsoft's diffusion-based long-form TTS, served via groxaxo/VibeVoice-FastAPI1 (a recent fork of ncoder-ai/VibeVoice-FastAPI which moves faster than upstream).

Model: microsoft/VibeVoice-1.5B by default. Switch to the 7B variant via .env if you want the bigger checkpoint.

Why this stack exists

Long-form / podcast-quality TTS with native multi-speaker dialogue support. Designed for one-shot generation of multi-minute scripts where conversation flow matters. Not for low-latency single-line synthesis — for that use Kokoro or Chatterbox Turbo.

	use case
VibeVoice 1.5B	long-form, multi-speaker dialogue (this stack)
Kokoro	low-latency English, fixed voice library
Chatterbox Turbo	low-latency English w/ voice cloning
IndexTTS-2	English voice cloning + emotion control
Qwen3-TTS-1.7B-Base	high-quality English voice cloning
CosyVoice 3	multilingual (Chinese-leaning)

API

OpenAI-compat at http://10.100.79.3:8194:

# Single-speaker (OpenAI-style).
curl -fsS -X POST http://10.100.79.3:8194/v1/audio/speech \
  -H 'Content-Type: application/json' \
  -d '{"model":"vibevoice","input":"Hello there.","voice":"voice-name","response_format":"wav"}' \
  > out.wav

# Multi-speaker dialogue — the headline feature. Format the input
# as a script with `Speaker N:` prefixes (0-indexed). The wrapper's
# extended /v1/vibevoice/generate endpoint handles voice switching.
curl -fsS -X POST http://10.100.79.3:8194/v1/vibevoice/generate \
  -H 'Content-Type: application/json' \
  -d '{
    "script":"Speaker 0: Welcome to the show.\nSpeaker 1: Glad to be here.\nSpeaker 0: Today we discuss…",
    "voices":["voice-host","voice-guest"],
    "stream":true
  }' > podcast.wav

# List available voices.
curl http://10.100.79.3:8194/v1/audio/voices

OpenAPI / docs at /docs. Healthcheck at /health.

stream=true is honored on the multi-speaker endpoint; the single-shot OpenAI endpoint returns the full file in one go.

Voices

Drop .wav / .mp3 / .flac / .m4a into /worktank/vibevoice/voices/ on the host (mounted read-only into the container). Restart the container after adding; the wrapper scans the dir at init, not per-request:

ssh irv-ml1 'cd /opt/docker/compose/vibevoice && docker compose restart'

VibeVoice also has built-in voice presets (Carter, Davis, Emma, Frank, Grace, Mike, Samuel) accessible by name. Microsoft has not released the cloning tooling so you can't add new "trained" voices — but the bundled ones already cover most podcast use cases.

Deploy

scripts/elway irv-ml1 --playbook playbooks/deploy-vibevoice.yaml

Cold deploy budget:

• ~12 GB image build (CUDA 12.8 + torch 2.8 + flash-attn)
• ~7 GB model download (VibeVoice-1.5B) on first start
• Total: ~19 GB on /worktank/vibevoice/

First build: ~12 min. First generation: ~30-60 s warmup.

Switching to the 7B variant

ssh irv-ml1 '
  cd /opt/docker/compose/vibevoice
  sed -i "s|^VIBEVOICE_MODEL=.*|VIBEVOICE_MODEL=rsxdalv/VibeVoice-Large|" .env
  docker compose up -d
'

The 7B model auto-downloads on next start (~18 GB). VRAM jumps from ~7 GB to ~18 GB bf16 — keep VIBEVOICE_GPU_DEVICES=1 (A6000) for it. For lower VRAM at slight quality cost, set VIBEVOICE_QUANT=int8_torchao which brings 7B down to ~10 GB.

Gotchas

• Not streaming-friendly for single-line use. The diffusion head has to denoise the whole latent before vocoding. Streaming on /v1/vibevoice/generate works at script-segment granularity (paragraph-ish), not token-by-token.
• Voice cloning isn't published. Microsoft released the inference models but not the training pipeline. Use the built-in voices, or pick another stack (IndexTTS-2 / Qwen3-TTS / Chatterbox Turbo).
• flash_attention_2 is the upstream default; if your GPU/torch combo doesn't have it built, set VIBEVOICE_ATTN=sdpa in .env to fall back to PyTorch's scaled-dot-product attention.
• License: VibeVoice MIT (Microsoft); wrapper MIT.