vh
288d085236
Add a live-consumer streaming path and text sanitation to the OmniVoice wrapper, so it can front speech-to-speech chat engines (not just the asset-engine's batch WAV use). - POST /tts: chunked 24 kHz mono s16le PCM (or open-ended WAV), driven by the adaptive buffer-ratchet scheduler. Emits the first sentence immediately, then ratchets chunk size up on OmniVoice's ~40x realtime headroom -> sub-second time-to-first-audio. Wire-compatible with chatterbox-fast /tts (both 24 kHz mono PCM). Batch /v1/audio/speech is unchanged for asset/file callers. - scheduler.py: VENDORED byte-faithful copy of chatterbox-fast's pure- Python (torch-free) scheduler, pinned to commit 7631462 (v0.1.0/v0.1.1). Vendor-copy over a shared package (operator call 2026-06-19): the module has no GPU deps, so reuse it without dragging chatterbox-fast's torch tree into this image. Promote to a shared package only on a 3rd consumer or real drift. - sanitize.py: language-safe TTS sanitizer run on both endpoints. Strips markdown, <think> blocks, HTML, and model control tokens; deliberately SKIPS the fork's English-only number/phone normalization that would corrupt OmniVoice's 600-language input. Preserves [laughter]-style tags. - Refactor: shared GenParams base for SpeechRequest + TTSStreamRequest; single GEN_LOCK serializes generation (single-stream interactive). - Dockerfile/playbook: copy + upload the two new modules; build-time `import app` smoke; correct stale "Gradio demo / no FastAPI" comments.
OmniVoice
k2-fsa/OmniVoice — zero-shot, massively-multilingual (600+ languages) voice-cloning + voice-design TTS from the Next-gen Kaldi / k2-fsa team. Diffusion-LM architecture, RTF as low as ~0.025 (≈40× real-time). Apache-2.0 — commercially clean (unlike Voxtral's CC BY-NC).
What it does
| Capability | Notes |
|---|---|
| Zero-shot voice cloning | Clone from a short reference clip |
| Voice design | Synthesize a voice from attributes (gender, age, pitch, accent, whisper, …) — no reference needed |
| 600+ languages | Broadest coverage of any zero-shot TTS |
| Fine control | Non-verbal symbols + pronunciation correction |
How it's served
Behind our own thin FastAPI wrapper (app.py) — upstream ships
only a Gradio demo, which we replaced (2026-06-19). The wrapper serves two
consumption modes on http://10.100.79.3:8199:
| Endpoint | Purpose |
|---|---|
POST /v1/audio/speech |
Batch OpenAI-style {input, voice, instruct, language, …} → one 24 kHz PCM_16 mono WAV. For the asset-engine (form-driven asset generation). |
POST /tts |
Streaming chunked 24 kHz mono s16le PCM (format=pcm, default) or open-ended WAV — for live speech-to-speech chat engines. Wire-compatible with chatterbox-fast's /tts. |
GET /v1/audio/voices |
{"voices": [...]} — the staged clone targets |
GET /v1/audio/languages |
{"languages": ["Auto", …]} — 600+ |
GET /v1/audio/instruct-items |
{"instruct_items": [...]} — controlled voice-DESIGN tags |
GET /healthz |
readiness (200 once model + ≥1 voice loaded) |
The wrapper loads OmniVoice + a Whisper ASR and precomputes a voice-clone
prompt per staged reference WAV at startup (Whisper auto-transcribes each
reference), so per-request latency is just generation. The full generation
surface is exposed: zero-shot clone (voice) and/or voice-design
(instruct), plus language / speed / duration and the diffusion knobs.
Streaming — sub-second time-to-first-audio
POST /tts (stream=true, default) runs the adaptive buffer-ratchet
scheduler vendored from chatterbox-fast (scheduler.py): it
emits the first sentence immediately and ratchets chunk size up on OmniVoice's
~40× realtime headroom, so a live consumer hears speech start in ~tens of ms
instead of waiting for the whole utterance. stream=false is a whole-text
one-shot for A/B. Scheduler tunables (margin, margin_first, rtf_prior,
sec_per_char_prior) are per-request overrides.
scheduler.py is a vendored byte-faithful copy (not a dependency) of
chatterbox-fast's pure-Python, torch-free scheduler — see its header for the
pinned commit. It is reused without dragging chatterbox-fast's GPU dependency
tree into this image; re-vendor on upstream change rather than editing in place.
Text sanitization
Both endpoints run input through a language-safe sanitizer
(sanitize.py) before synthesis: it strips markdown, LLM
artifacts (<think> blocks), HTML, and model control tokens, but deliberately
skips English-only number/phone/email normalization that would corrupt
OmniVoice's multilingual input. OmniVoice's own [laughter]-style symbols are
preserved.
Voices — reused from chatterbox
The clone references are chatterbox-fast's /refs/*.wav, staged into
/worktank/omnivoice/voices/ by the deploy playbook (33 named voices at deploy;
_*.wav test artifacts skipped). Add more by dropping WAVs there and restarting.
asset-engine
Catalogued in docs/asset-engine/services.yaml
(id: omnivoice, lifecycle.stack: omnivoice, voice field sourced live from
/v1/audio/voices). The compose project name is pinned to omnivoice so the
liveness probe (docker-ps project-name match) sees it online.
Placement
- irv-ml1, GPU 0 (RTX 3090) — pinned via
OMNIVOICE_GPU_DEVICES=0. The A6000 (device 1) is ComfyUI-exclusive after the 2026-06-18 VRAM consolidation. OmniVoice fits in <5 GB; the 3090 had ~18 GB free. - Port 8199 (8001 inside the container).
Deploy
scripts/elway irv-ml1 --playbook playbooks/deploy-omnivoice.yaml
Builds the image locally (CUDA 12.8 + torch 2.8.0 + omnivoice from PyPI),
stages the build context under /opt/docker/compose/omnivoice/, brings it
up, and waits for the Gradio UI on :8199. First boot is slow: ~5-10 min
docker build + a one-time HF weight pre-warm (k2-fsa/OmniVoice, entrypoint
pre-download into ${OMNIVOICE_CACHE_DIR}).
Tunables
All in .env (see .env.example): OMNIVOICE_PORT, OMNIVOICE_GPU_DEVICES,
OMNIVOICE_TAG, OMNIVOICE_VERSION (optional PyPI pin),
OMNIVOICE_CACHE_DIR, OMNIVOICE_VOICES_DIR. Drop reference WAV/FLAC into
/worktank/omnivoice/voices/ to stage cloning sources.
Footprint
- Disk: HF weight cache under
/worktank/omnivoice/hf_cache. - VRAM: <5 GB (docs cite 4 GB+ GPUs).