vh
981ae4e6a1
Wrapper /v1/audio/speech now accepts OmniVoice's whole surface: - voice (clone, now OPTIONAL) and/or instruct (voice DESIGN). instruct is a CONTROLLED vocabulary (gender/age/pitch/accent/whisper tags, comma-separated), not free prose — discoverable at the new /v1/audio/instruct-items endpoint (23 items). - language (Auto + 647, new /v1/audio/languages endpoint), speed, duration. - diffusion controls: num_step, guidance_scale, denoise, preprocess_prompt, postprocess_output; plus a generation_overrides JSON passthrough for expert GenerationConfig knobs (t_shift, layer_penalty_factor, position/class temperature, audio_chunk_*). - at least one of voice/instruct required (else 400). Catalog (services.yaml): omnivoice v1 -> v2, 13 schema-valid fields; instruct as a controlled-vocab text field sourced from the items endpoint. Verified live on irv-ml1: clone, voice-design (instruct-only), and tuned-param synths all -> 24 kHz PCM_16 WAV; 647 languages; 23 instruct items.
OmniVoice
k2-fsa/OmniVoice — zero-shot, massively-multilingual (600+ languages) voice-cloning + voice-design TTS from the Next-gen Kaldi / k2-fsa team. Diffusion-LM architecture, RTF as low as ~0.025 (≈40× real-time). Apache-2.0 — commercially clean (unlike Voxtral's CC BY-NC).
What it does
| Capability | Notes |
|---|---|
| Zero-shot voice cloning | Clone from a short reference clip |
| Voice design | Synthesize a voice from attributes (gender, age, pitch, accent, whisper, …) — no reference needed |
| 600+ languages | Broadest coverage of any zero-shot TTS |
| Fine control | Non-verbal symbols + pronunciation correction |
How it's served
Behind our own thin FastAPI wrapper (app.py) — upstream ships
only a Gradio demo, which we replaced (2026-06-19) so the asset-engine can
consume it. Endpoints on http://10.100.79.3:8199:
| Endpoint | Purpose |
|---|---|
POST /v1/audio/speech |
OpenAI-style {input, voice, response_format=wav} → 24 kHz PCM_16 mono |
GET /v1/audio/voices |
{"voices": [...]} — the staged clone targets |
GET /healthz |
readiness (200 once model + ≥1 voice loaded) |
The wrapper loads OmniVoice + a Whisper ASR and precomputes a voice-clone prompt per staged reference WAV at startup (Whisper auto-transcribes each reference), so per-request latency is just generation. v1 is clone-only — OmniVoice's voice-design / language / instruct controls aren't exposed yet.
Voices — reused from chatterbox
The clone references are chatterbox-fast's /refs/*.wav, staged into
/worktank/omnivoice/voices/ by the deploy playbook (33 named voices at deploy;
_*.wav test artifacts skipped). Add more by dropping WAVs there and restarting.
asset-engine
Catalogued in docs/asset-engine/services.yaml
(id: omnivoice, lifecycle.stack: omnivoice, voice field sourced live from
/v1/audio/voices). The compose project name is pinned to omnivoice so the
liveness probe (docker-ps project-name match) sees it online.
Placement
- irv-ml1, GPU 0 (RTX 3090) — pinned via
OMNIVOICE_GPU_DEVICES=0. The A6000 (device 1) is ComfyUI-exclusive after the 2026-06-18 VRAM consolidation. OmniVoice fits in <5 GB; the 3090 had ~18 GB free. - Port 8199 (8001 inside the container).
Deploy
scripts/elway irv-ml1 --playbook playbooks/deploy-omnivoice.yaml
Builds the image locally (CUDA 12.8 + torch 2.8.0 + omnivoice from PyPI),
stages the build context under /opt/docker/compose/omnivoice/, brings it
up, and waits for the Gradio UI on :8199. First boot is slow: ~5-10 min
docker build + a one-time HF weight pre-warm (k2-fsa/OmniVoice, entrypoint
pre-download into ${OMNIVOICE_CACHE_DIR}).
Tunables
All in .env (see .env.example): OMNIVOICE_PORT, OMNIVOICE_GPU_DEVICES,
OMNIVOICE_TAG, OMNIVOICE_VERSION (optional PyPI pin),
OMNIVOICE_CACHE_DIR, OMNIVOICE_VOICES_DIR. Drop reference WAV/FLAC into
/worktank/omnivoice/voices/ to stage cloning sources.
Footprint
- Disk: HF weight cache under
/worktank/omnivoice/hf_cache. - VRAM: <5 GB (docs cite 4 GB+ GPUs).