# VibeVoice

Microsoft's diffusion-based long-form TTS, served via
[groxaxo/VibeVoice-FastAPI1](https://github.com/groxaxo/VibeVoice-FastAPI1)
(a recent fork of [ncoder-ai/VibeVoice-FastAPI](https://github.com/ncoder-ai/VibeVoice-FastAPI)
which moves faster than upstream).

Model: [microsoft/VibeVoice-1.5B](https://huggingface.co/microsoft/VibeVoice-1.5B)
by default. Switch to the 7B variant via `.env` if you want the
bigger checkpoint.

## Why this stack exists

Long-form / podcast-quality TTS with native multi-speaker dialogue
support. Designed for one-shot generation of multi-minute scripts
where conversation flow matters. **Not** for low-latency single-line
synthesis — for that use Kokoro or Chatterbox Turbo.

| | use case |
|---|---|
| **VibeVoice 1.5B** | long-form, multi-speaker dialogue (this stack) |
| Kokoro | low-latency English, fixed voice library |
| Chatterbox Turbo | low-latency English w/ voice cloning |
| IndexTTS-2 | English voice cloning + emotion control |
| Qwen3-TTS-1.7B-Base | high-quality English voice cloning |
| CosyVoice 3 | multilingual (Chinese-leaning) |

## API

OpenAI-compat at `http://10.100.79.3:8194`:

```bash
# Single-speaker (OpenAI-style).
curl -fsS -X POST http://10.100.79.3:8194/v1/audio/speech \
  -H 'Content-Type: application/json' \
  -d '{"model":"vibevoice","input":"Hello there.","voice":"voice-name","response_format":"wav"}' \
  > out.wav

# Multi-speaker dialogue — the headline feature. Format the input
# as a script with `Speaker N:` prefixes (0-indexed). The wrapper's
# extended /v1/vibevoice/generate endpoint handles voice switching.
curl -fsS -X POST http://10.100.79.3:8194/v1/vibevoice/generate \
  -H 'Content-Type: application/json' \
  -d '{
    "script":"Speaker 0: Welcome to the show.\nSpeaker 1: Glad to be here.\nSpeaker 0: Today we discuss…",
    "voices":["voice-host","voice-guest"],
    "stream":true
  }' > podcast.wav

# List available voices.
curl http://10.100.79.3:8194/v1/audio/voices
```

OpenAPI / docs at `/docs`. Healthcheck at `/health`.

`stream=true` is honored on the multi-speaker endpoint; the
single-shot OpenAI endpoint returns the full file in one go.

## Voices

Drop `.wav` / `.mp3` / `.flac` / `.m4a` into
`/worktank/vibevoice/voices/` on the host (mounted read-only into
the container). Restart the container after adding; the wrapper
scans the dir at init, not per-request:

```bash
ssh irv-ml1 'cd /opt/docker/compose/vibevoice && docker compose restart'
```

VibeVoice also has built-in voice presets (Carter, Davis, Emma,
Frank, Grace, Mike, Samuel) accessible by name. Microsoft has not
released the cloning tooling so you can't add new "trained" voices
— but the bundled ones already cover most podcast use cases.

## Deploy

```bash
scripts/elway irv-ml1 --playbook playbooks/deploy-vibevoice.yaml
```

Cold deploy budget:
- ~12 GB image build (CUDA 12.8 + torch 2.8 + flash-attn)
- ~7 GB model download (VibeVoice-1.5B) on first start
- **Total: ~19 GB on /worktank/vibevoice/**

First build: ~12 min. First generation: ~30-60 s warmup.

## Switching to the 7B variant

```bash
ssh irv-ml1 '
  cd /opt/docker/compose/vibevoice
  sed -i "s|^VIBEVOICE_MODEL=.*|VIBEVOICE_MODEL=rsxdalv/VibeVoice-Large|" .env
  docker compose up -d
'
```

The 7B model auto-downloads on next start (~18 GB). VRAM jumps from
~7 GB to ~18 GB bf16 — keep `VIBEVOICE_GPU_DEVICES=1` (A6000) for it.
For lower VRAM at slight quality cost, set `VIBEVOICE_QUANT=int8_torchao`
which brings 7B down to ~10 GB.

## Gotchas

- **Not streaming-friendly for single-line use.** The diffusion head
  has to denoise the whole latent before vocoding. Streaming on
  `/v1/vibevoice/generate` works at script-segment granularity
  (paragraph-ish), not token-by-token.
- **Voice cloning isn't published.** Microsoft released the inference
  models but not the training pipeline. Use the built-in voices, or
  pick another stack (IndexTTS-2 / Qwen3-TTS / Chatterbox Turbo).
- **`flash_attention_2`** is the upstream default; if your GPU/torch
  combo doesn't have it built, set `VIBEVOICE_ATTN=sdpa` in `.env`
  to fall back to PyTorch's scaled-dot-product attention.
- **License**: VibeVoice MIT (Microsoft); wrapper MIT.