vh
cd92b85157
Empirical follow-up to the streaming /tts smoke test on the 3090. OmniVoice is diffusion: a ~fixed per-call overhead (~1.5s at 32 steps, ~0.7s at 16) dominates regardless of chunk length, so the upstream-claimed 40x RTF does NOT hold here (measured ~2.8x/32-step, ~5.6x/16-step) and the chatterbox- tuned scheduler over-chunks and starves. - Streaming /tts defaults to num_step=16 (TTFA ~1.5s -> ~0.7s); batch /v1/audio/speech stays num_step=32 for quality. Per-request override intact. - Scheduler prior raised to rtf_prior=20 (env OMNIVOICE_STREAM_RTF_PRIOR, wired through compose + .env.example) so it packs whole-text-minus-first- sentence into a few chunks: validated ~3 chunks, no starvation, total wall ~= one-shot, less per-chunk silence padding. - Docs corrected: the "sub-second / 40x" claims were wrong; streaming has a diffusion TTFA floor (~0.7s) and wins mainly on long replies. chatterbox- fast (autoregressive, ~0.5s TTFA) stays the lowest-latency front-end; OmniVoice is the multilingual / voice-design complement.
123 lines
6.1 KiB
Markdown
123 lines
6.1 KiB
Markdown
# OmniVoice
|
||
|
||
[k2-fsa/OmniVoice](https://github.com/k2-fsa/OmniVoice) — zero-shot,
|
||
massively-multilingual (**600+ languages**) voice-cloning + voice-design
|
||
TTS from the Next-gen Kaldi / k2-fsa team. Diffusion-LM architecture,
|
||
RTF as low as ~0.025 (≈40× real-time). **Apache-2.0** — commercially clean
|
||
(unlike Voxtral's CC BY-NC).
|
||
|
||
## What it does
|
||
|
||
| Capability | Notes |
|
||
|---|---|
|
||
| Zero-shot voice cloning | Clone from a short reference clip |
|
||
| Voice **design** | Synthesize a voice from attributes (gender, age, pitch, accent, whisper, …) — no reference needed |
|
||
| 600+ languages | Broadest coverage of any zero-shot TTS |
|
||
| Fine control | Non-verbal symbols + pronunciation correction |
|
||
|
||
## How it's served
|
||
|
||
Behind our own thin **FastAPI wrapper** ([`app.py`](app.py)) — upstream ships
|
||
only a Gradio demo, which we replaced (2026-06-19). The wrapper serves **two
|
||
consumption modes** on `http://10.100.79.3:8199`:
|
||
|
||
| Endpoint | Purpose |
|
||
|---|---|
|
||
| `POST /v1/audio/speech` | **Batch** OpenAI-style `{input, voice, instruct, language, …}` → one 24 kHz PCM_16 mono WAV. For the **asset-engine** (form-driven asset generation). |
|
||
| `POST /tts` | **Streaming** chunked 24 kHz mono `s16le` PCM (`format=pcm`, default) or open-ended WAV — for **live speech-to-speech chat engines**. Wire-compatible with chatterbox-fast's `/tts`. |
|
||
| `GET /v1/audio/voices` | `{"voices": [...]}` — the staged clone targets |
|
||
| `GET /v1/audio/languages` | `{"languages": ["Auto", …]}` — 600+ |
|
||
| `GET /v1/audio/instruct-items` | `{"instruct_items": [...]}` — controlled voice-DESIGN tags |
|
||
| `GET /healthz` | readiness (200 once model + ≥1 voice loaded) |
|
||
|
||
The wrapper loads OmniVoice + a Whisper ASR and **precomputes a voice-clone
|
||
prompt per staged reference WAV at startup** (Whisper auto-transcribes each
|
||
reference), so per-request latency is just generation. The full generation
|
||
surface is exposed: zero-shot **clone** (`voice`) and/or voice-**design**
|
||
(`instruct`), plus `language` / `speed` / `duration` and the diffusion knobs.
|
||
|
||
### Streaming — earlier first-audio (with a diffusion floor)
|
||
|
||
`POST /tts` (`stream=true`, default) runs the **adaptive buffer-ratchet
|
||
scheduler** vendored from chatterbox-fast ([`scheduler.py`](scheduler.py)): it
|
||
emits the first sentence immediately, then packs the rest into a few chunks so a
|
||
live consumer hears speech start sooner than waiting for the whole utterance.
|
||
`stream=false` is a whole-text one-shot for A/B.
|
||
|
||
**Measured reality (3090, not the upstream-claimed 40× RTF):** OmniVoice is a
|
||
diffusion model, so each `generate()` call has a **~fixed per-call overhead**
|
||
(~1.5 s at `num_step=32`, ~0.7 s at 16) that sets a **time-to-first-audio
|
||
floor** — short and long chunks cost nearly the same. Server-side TTFA is
|
||
therefore ~0.7 s (streaming default, 16 steps), **not** sub-second-at-full-
|
||
quality. Effective RTF is ~2.8× (32 steps) / ~5.6× (16 steps). The win over
|
||
one-shot is small for short replies and grows with length (one-shot TTFA scales
|
||
with the whole utterance; streaming stays ~flat at the first-sentence cost).
|
||
For absolute-lowest TTFA, **chatterbox-fast** (autoregressive, ~0.5 s) remains
|
||
the better front-end; OmniVoice is the multilingual / voice-design complement.
|
||
|
||
Defaults tuned for this: **streaming `num_step=16`** (batch `/v1/audio/speech`
|
||
stays 32 for quality), and an **aggressive packing prior** (`rtf_prior=20`, env
|
||
`OMNIVOICE_STREAM_RTF_PRIOR`) — diffusion's fixed overhead makes the chatterbox
|
||
default over-chunk and starve, so we pack whole-text-minus-first-sentence into a
|
||
few chunks (validated: ~3 chunks, no starvation, total ≈ one-shot). Scheduler
|
||
tunables (`margin`, `margin_first`, `rtf_prior`, `sec_per_char_prior`) and
|
||
`num_step` are per-request overrides.
|
||
|
||
`scheduler.py` is a **vendored byte-faithful copy** (not a dependency) of
|
||
chatterbox-fast's pure-Python, torch-free scheduler — see its header for the
|
||
pinned commit. It is reused without dragging chatterbox-fast's GPU dependency
|
||
tree into this image; re-vendor on upstream change rather than editing in place.
|
||
|
||
### Text sanitization
|
||
|
||
Both endpoints run `input` through a **language-safe sanitizer**
|
||
([`sanitize.py`](sanitize.py)) before synthesis: it strips markdown, LLM
|
||
artifacts (`<think>` blocks), HTML, and model control tokens, but deliberately
|
||
**skips** English-only number/phone/email normalization that would corrupt
|
||
OmniVoice's multilingual input. OmniVoice's own `[laughter]`-style symbols are
|
||
preserved.
|
||
|
||
### Voices — reused from chatterbox
|
||
|
||
The clone references are chatterbox-fast's `/refs/*.wav`, staged into
|
||
`/worktank/omnivoice/voices/` by the deploy playbook (33 named voices at deploy;
|
||
`_*.wav` test artifacts skipped). Add more by dropping WAVs there and restarting.
|
||
|
||
### asset-engine
|
||
|
||
Catalogued in [`docs/asset-engine/services.yaml`](../../docs/asset-engine/services.yaml)
|
||
(`id: omnivoice`, `lifecycle.stack: omnivoice`, `voice` field sourced live from
|
||
`/v1/audio/voices`). The compose **project name is pinned to `omnivoice`** so the
|
||
liveness probe (docker-ps project-name match) sees it online.
|
||
|
||
## Placement
|
||
|
||
- **irv-ml1, GPU 0 (RTX 3090)** — pinned via `OMNIVOICE_GPU_DEVICES=0`.
|
||
The A6000 (device 1) is ComfyUI-exclusive after the 2026-06-18 VRAM
|
||
consolidation. OmniVoice fits in <5 GB; the 3090 had ~18 GB free.
|
||
- Port **8199** (8001 inside the container).
|
||
|
||
## Deploy
|
||
|
||
```bash
|
||
scripts/elway irv-ml1 --playbook playbooks/deploy-omnivoice.yaml
|
||
```
|
||
|
||
Builds the image locally (CUDA 12.8 + torch 2.8.0 + `omnivoice` from PyPI),
|
||
stages the build context under `/opt/docker/compose/omnivoice/`, brings it
|
||
up, and waits for the Gradio UI on `:8199`. First boot is slow: ~5-10 min
|
||
docker build + a one-time HF weight pre-warm (`k2-fsa/OmniVoice`, entrypoint
|
||
pre-download into `${OMNIVOICE_CACHE_DIR}`).
|
||
|
||
## Tunables
|
||
|
||
All in `.env` (see `.env.example`): `OMNIVOICE_PORT`, `OMNIVOICE_GPU_DEVICES`,
|
||
`OMNIVOICE_TAG`, `OMNIVOICE_VERSION` (optional PyPI pin),
|
||
`OMNIVOICE_CACHE_DIR`, `OMNIVOICE_VOICES_DIR`. Drop reference WAV/FLAC into
|
||
`/worktank/omnivoice/voices/` to stage cloning sources.
|
||
|
||
## Footprint
|
||
|
||
- **Disk**: HF weight cache under `/worktank/omnivoice/hf_cache`.
|
||
- **VRAM**: <5 GB (docs cite 4 GB+ GPUs).
|