# OmniVoice

[k2-fsa/OmniVoice](https://github.com/k2-fsa/OmniVoice) — zero-shot,
massively-multilingual (**600+ languages**) voice-cloning + voice-design
TTS from the Next-gen Kaldi / k2-fsa team. Diffusion-LM architecture,
RTF as low as ~0.025 (≈40× real-time). **Apache-2.0** — commercially clean
(unlike Voxtral's CC BY-NC).

## What it does

| Capability | Notes |
|---|---|
| Zero-shot voice cloning | Clone from a short reference clip |
| Voice **design** | Synthesize a voice from attributes (gender, age, pitch, accent, whisper, …) — no reference needed |
| 600+ languages | Broadest coverage of any zero-shot TTS |
| Fine control | Non-verbal symbols + pronunciation correction |

## How it's served

Behind our own thin **FastAPI wrapper** ([`app.py`](app.py)) — upstream ships
only a Gradio demo, which we replaced (2026-06-19) so the **asset-engine** can
consume it. Endpoints on `http://10.100.79.3:8199`:

| Endpoint | Purpose |
|---|---|
| `POST /v1/audio/speech` | OpenAI-style `{input, voice, response_format=wav}` → 24 kHz PCM_16 mono |
| `GET /v1/audio/voices` | `{"voices": [...]}` — the staged clone targets |
| `GET /healthz` | readiness (200 once model + ≥1 voice loaded) |

The wrapper loads OmniVoice + a Whisper ASR and **precomputes a voice-clone
prompt per staged reference WAV at startup** (Whisper auto-transcribes each
reference), so per-request latency is just generation. v1 is **clone-only** —
OmniVoice's voice-*design* / language / instruct controls aren't exposed yet.

### Voices — reused from chatterbox

The clone references are chatterbox-fast's `/refs/*.wav`, staged into
`/worktank/omnivoice/voices/` by the deploy playbook (33 named voices at deploy;
`_*.wav` test artifacts skipped). Add more by dropping WAVs there and restarting.

### asset-engine

Catalogued in [`docs/asset-engine/services.yaml`](../../docs/asset-engine/services.yaml)
(`id: omnivoice`, `lifecycle.stack: omnivoice`, `voice` field sourced live from
`/v1/audio/voices`). The compose **project name is pinned to `omnivoice`** so the
liveness probe (docker-ps project-name match) sees it online.

## Placement

- **irv-ml1, GPU 0 (RTX 3090)** — pinned via `OMNIVOICE_GPU_DEVICES=0`.
  The A6000 (device 1) is ComfyUI-exclusive after the 2026-06-18 VRAM
  consolidation. OmniVoice fits in <5 GB; the 3090 had ~18 GB free.
- Port **8199** (8001 inside the container).

## Deploy

```bash
scripts/elway irv-ml1 --playbook playbooks/deploy-omnivoice.yaml
```

Builds the image locally (CUDA 12.8 + torch 2.8.0 + `omnivoice` from PyPI),
stages the build context under `/opt/docker/compose/omnivoice/`, brings it
up, and waits for the Gradio UI on `:8199`. First boot is slow: ~5-10 min
docker build + a one-time HF weight pre-warm (`k2-fsa/OmniVoice`, entrypoint
pre-download into `${OMNIVOICE_CACHE_DIR}`).

## Tunables

All in `.env` (see `.env.example`): `OMNIVOICE_PORT`, `OMNIVOICE_GPU_DEVICES`,
`OMNIVOICE_TAG`, `OMNIVOICE_VERSION` (optional PyPI pin),
`OMNIVOICE_CACHE_DIR`, `OMNIVOICE_VOICES_DIR`. Drop reference WAV/FLAC into
`/worktank/omnivoice/voices/` to stage cloning sources.

## Footprint

- **Disk**: HF weight cache under `/worktank/omnivoice/hf_cache`.
- **VRAM**: <5 GB (docs cite 4 GB+ GPUs).