esh-pfi-infrastructure/stacks/omnivoice/README.md

OmniVoice

k2-fsa/OmniVoice — zero-shot, massively-multilingual (600+ languages) voice-cloning + voice-design TTS from the Next-gen Kaldi / k2-fsa team. Diffusion-LM architecture, RTF as low as ~0.025 (≈40× real-time). Apache-2.0 — commercially clean (unlike Voxtral's CC BY-NC).

What it does

Capability	Notes
Zero-shot voice cloning	Clone from a short reference clip
Voice design	Synthesize a voice from attributes (gender, age, pitch, accent, whisper, …) — no reference needed
600+ languages	Broadest coverage of any zero-shot TTS
Fine control	Non-verbal symbols + pronunciation correction

How it's served

Behind our own thin FastAPI wrapper (app.py) — upstream ships only a Gradio demo, which we replaced (2026-06-19). The wrapper serves two consumption modes on http://10.100.79.3:8199:

Endpoint	Purpose
POST /v1/audio/speech	Batch OpenAI-style {input, voice, instruct, language, …} → one 24 kHz PCM_16 mono WAV. For the asset-engine (form-driven asset generation).
POST /tts	Streaming chunked 24 kHz mono s16le PCM (format=pcm, default) or open-ended WAV — for live speech-to-speech chat engines. Wire-compatible with chatterbox-fast's /tts.
GET /v1/audio/voices	{"voices": [...]} — the staged clone targets
GET /v1/audio/languages	{"languages": ["Auto", …]} — 600+
GET /v1/audio/instruct-items	{"instruct_items": [...]} — controlled voice-DESIGN tags
GET /healthz	readiness (200 once model + ≥1 voice loaded)

The wrapper loads OmniVoice + a Whisper ASR and precomputes a voice-clone prompt per staged reference WAV at startup (Whisper auto-transcribes each reference), so per-request latency is just generation. The full generation surface is exposed: zero-shot clone (voice) and/or voice-design (instruct), plus language / speed / duration and the diffusion knobs.

Streaming — earlier first-audio (with a diffusion floor)

POST /tts (stream=true, default) runs the adaptive buffer-ratchet scheduler vendored from chatterbox-fast (scheduler.py): it emits the first sentence immediately, then packs the rest into a few chunks so a live consumer hears speech start sooner than waiting for the whole utterance. stream=false is a whole-text one-shot for A/B.

Measured reality (3090, not the upstream-claimed 40× RTF): OmniVoice is a diffusion model, so each generate() call has a ~fixed per-call overhead (~1.5 s at num_step=32, ~0.7 s at 16) that sets a time-to-first-audio floor — short and long chunks cost nearly the same. Server-side TTFA is therefore ~0.7 s (streaming default, 16 steps), not sub-second-at-full- quality. Effective RTF is ~2.8× (32 steps) / ~5.6× (16 steps). The win over one-shot is small for short replies and grows with length (one-shot TTFA scales with the whole utterance; streaming stays ~flat at the first-sentence cost). For absolute-lowest TTFA, chatterbox-fast (autoregressive, ~0.5 s) remains the better front-end; OmniVoice is the multilingual / voice-design complement.

Defaults tuned for this: streaming num_step=16 (batch /v1/audio/speech stays 32 for quality), and an aggressive packing prior (rtf_prior=20, env OMNIVOICE_STREAM_RTF_PRIOR) — diffusion's fixed overhead makes the chatterbox default over-chunk and starve, so we pack whole-text-minus-first-sentence into a few chunks (validated: ~3 chunks, no starvation, total ≈ one-shot). Scheduler tunables (margin, margin_first, rtf_prior, sec_per_char_prior) and num_step are per-request overrides.

scheduler.py is a vendored byte-faithful copy (not a dependency) of chatterbox-fast's pure-Python, torch-free scheduler — see its header for the pinned commit. It is reused without dragging chatterbox-fast's GPU dependency tree into this image; re-vendor on upstream change rather than editing in place.

Text sanitization

Both endpoints run input through a language-safe sanitizer (sanitize.py) before synthesis: it strips markdown, LLM artifacts (<think> blocks), HTML, and model control tokens, but deliberately skips English-only number/phone/email normalization that would corrupt OmniVoice's multilingual input. OmniVoice's own [laughter]-style symbols are preserved.

Voices — reused from chatterbox

The clone references are chatterbox-fast's /refs/*.wav, staged into /worktank/omnivoice/voices/ by the deploy playbook (33 named voices at deploy; _*.wav test artifacts skipped). Add more by dropping WAVs there and restarting.

asset-engine

Catalogued in docs/asset-engine/services.yaml (id: omnivoice, lifecycle.stack: omnivoice, voice field sourced live from /v1/audio/voices). The compose project name is pinned to omnivoice so the liveness probe (docker-ps project-name match) sees it online.

Placement

• irv-ml1, GPU 0 (RTX 3090) — pinned via OMNIVOICE_GPU_DEVICES=0. The A6000 (device 1) is ComfyUI-exclusive after the 2026-06-18 VRAM consolidation. OmniVoice fits in <5 GB; the 3090 had ~18 GB free.
• Port 8199 (8001 inside the container).

Deploy

scripts/elway irv-ml1 --playbook playbooks/deploy-omnivoice.yaml

Builds the image locally (CUDA 12.8 + torch 2.8.0 + omnivoice from PyPI), stages the build context under /opt/docker/compose/omnivoice/, brings it up, and waits for the Gradio UI on :8199. First boot is slow: ~5-10 min docker build + a one-time HF weight pre-warm (k2-fsa/OmniVoice, entrypoint pre-download into ${OMNIVOICE_CACHE_DIR}).

Tunables

All in .env (see .env.example): OMNIVOICE_PORT, OMNIVOICE_GPU_DEVICES, OMNIVOICE_TAG, OMNIVOICE_VERSION (optional PyPI pin), OMNIVOICE_CACHE_DIR, OMNIVOICE_VOICES_DIR. Drop reference WAV/FLAC into /worktank/omnivoice/voices/ to stage cloning sources.

Footprint

• Disk: HF weight cache under /worktank/omnivoice/hf_cache.
• VRAM: <5 GB (docs cite 4 GB+ GPUs).