feat(omnivoice): wire to asset-engine via FastAPI wrapper + reuse chatterbox voices
- app.py: thin FastAPI wrapper exposing OpenAI /v1/audio/speech (+ /v1/audio/voices, /healthz) around OmniVoice's Python API; precomputes a voice-clone prompt per voice at startup (loaded Whisper auto-transcribes each reference). Replaces the Gradio demo. - Dockerfile/compose: run the uvicorn wrapper, /healthz healthcheck, project name pinned to "omnivoice" so the asset-engine liveness probe matches. - deploy-omnivoice.yaml: stage chatterbox /refs/*.wav as clone voices (skip _* artifacts) + verify the API surface. - services.yaml: catalog entry (id omnivoice, :8199/v1/audio/speech, voice list sourced live from /v1/audio/voices) + reproducibility_audit row. Verified live on irv-ml1: /healthz ok, 33 voices loaded, test synth -> 24kHz PCM_16 WAV.
This commit is contained in:
+11
-10
@@ -33,22 +33,23 @@ RUN pip install --no-cache-dir \
|
||||
ARG OMNIVOICE_VERSION=
|
||||
RUN pip install --no-cache-dir "omnivoice${OMNIVOICE_VERSION:+==${OMNIVOICE_VERSION}}" huggingface_hub
|
||||
|
||||
# Fail the build loudly if the console script name isn't what we expect,
|
||||
# rather than crash-loop at runtime. Logs the actual omni* entrypoints.
|
||||
RUN echo "omni console scripts:" && (ls /opt/venv/bin | grep -i omni || true) \
|
||||
&& command -v omnivoice-demo >/dev/null \
|
||||
|| { echo "ERROR: omnivoice-demo CLI not found after install"; exit 1; }
|
||||
# Our asset-engine wrapper deps (FastAPI stack + soundfile for WAV encoding).
|
||||
RUN pip install --no-cache-dir fastapi 'uvicorn[standard]' soundfile python-multipart
|
||||
|
||||
# Fail the build loudly if the runtime imports aren't satisfiable.
|
||||
RUN python -c "import omnivoice, fastapi, soundfile, uvicorn; print('omnivoice wrapper deps OK')"
|
||||
|
||||
WORKDIR /app
|
||||
COPY app.py /app/app.py
|
||||
COPY entrypoint.sh /usr/local/bin/entrypoint.sh
|
||||
RUN chmod +x /usr/local/bin/entrypoint.sh
|
||||
|
||||
EXPOSE 8001
|
||||
|
||||
# Gradio serves HTML at / — 200 once the UI is up (weights load lazily on
|
||||
# first synth; the entrypoint pre-warms them). Generous start period.
|
||||
HEALTHCHECK --interval=30s --timeout=10s --start-period=900s --retries=3 \
|
||||
CMD wget -q -O /dev/null http://127.0.0.1:8001/ || exit 1
|
||||
# Our wrapper exposes /healthz (200 once model + >=1 voice are loaded). Generous
|
||||
# start period: first boot pre-warms OmniVoice + Whisper ASR + clones every voice.
|
||||
HEALTHCHECK --interval=30s --timeout=10s --start-period=1200s --retries=3 \
|
||||
CMD wget -q -O /dev/null http://127.0.0.1:8001/healthz || exit 1
|
||||
|
||||
ENTRYPOINT ["/usr/local/bin/entrypoint.sh"]
|
||||
CMD ["omnivoice-demo", "--ip", "0.0.0.0", "--port", "8001"]
|
||||
CMD ["uvicorn", "app:app", "--host", "0.0.0.0", "--port", "8001"]
|
||||
|
||||
@@ -17,14 +17,33 @@ RTF as low as ~0.025 (≈40× real-time). **Apache-2.0** — commercially clean
|
||||
|
||||
## How it's served
|
||||
|
||||
Upstream ships **its own Gradio demo** (`omnivoice-demo`), so this stack
|
||||
just runs that — no custom wrapper. That means the surface is the **Gradio
|
||||
UI + Gradio API**, *not* an OpenAI-compatible `/v1/audio/speech` endpoint.
|
||||
Behind our own thin **FastAPI wrapper** ([`app.py`](app.py)) — upstream ships
|
||||
only a Gradio demo, which we replaced (2026-06-19) so the **asset-engine** can
|
||||
consume it. Endpoints on `http://10.100.79.3:8199`:
|
||||
|
||||
- UI: `http://10.100.79.3:8199/`
|
||||
- Programmatic: the Gradio API under `/gradio_api/` (or `/config` to
|
||||
introspect). If you later want OpenAI-compat for asset-engine, add a thin
|
||||
FastAPI wrapper like [`stacks/index-tts/app.py`](../index-tts/app.py).
|
||||
| Endpoint | Purpose |
|
||||
|---|---|
|
||||
| `POST /v1/audio/speech` | OpenAI-style `{input, voice, response_format=wav}` → 24 kHz PCM_16 mono |
|
||||
| `GET /v1/audio/voices` | `{"voices": [...]}` — the staged clone targets |
|
||||
| `GET /healthz` | readiness (200 once model + ≥1 voice loaded) |
|
||||
|
||||
The wrapper loads OmniVoice + a Whisper ASR and **precomputes a voice-clone
|
||||
prompt per staged reference WAV at startup** (Whisper auto-transcribes each
|
||||
reference), so per-request latency is just generation. v1 is **clone-only** —
|
||||
OmniVoice's voice-*design* / language / instruct controls aren't exposed yet.
|
||||
|
||||
### Voices — reused from chatterbox
|
||||
|
||||
The clone references are chatterbox-fast's `/refs/*.wav`, staged into
|
||||
`/worktank/omnivoice/voices/` by the deploy playbook (33 named voices at deploy;
|
||||
`_*.wav` test artifacts skipped). Add more by dropping WAVs there and restarting.
|
||||
|
||||
### asset-engine
|
||||
|
||||
Catalogued in [`docs/asset-engine/services.yaml`](../../docs/asset-engine/services.yaml)
|
||||
(`id: omnivoice`, `lifecycle.stack: omnivoice`, `voice` field sourced live from
|
||||
`/v1/audio/voices`). The compose **project name is pinned to `omnivoice`** so the
|
||||
liveness probe (docker-ps project-name match) sees it online.
|
||||
|
||||
## Placement
|
||||
|
||||
|
||||
@@ -0,0 +1,111 @@
|
||||
"""Thin FastAPI wrapper exposing OmniVoice (k2-fsa/OmniVoice) as an OpenAI-style
|
||||
TTS for the asset-engine.
|
||||
|
||||
Upstream ships only a Gradio demo; we own this wrapper (same pattern as
|
||||
stacks/index-tts/app.py). Voices are reference WAVs staged in
|
||||
${OMNIVOICE_VOICES_DIR} (the reused chatterbox /refs/*.wav). A voice-clone prompt
|
||||
is precomputed once per voice at startup (the loaded Whisper ASR auto-transcribes
|
||||
each reference) and cached, so per-request latency is just generation.
|
||||
|
||||
Endpoints:
|
||||
GET /healthz -> readiness (200 once model + >=1 voice are loaded)
|
||||
GET /v1/audio/voices -> {"voices": [<name>, ...]}
|
||||
POST /v1/audio/speech -> {input, voice, response_format=wav} -> audio/wav
|
||||
"""
|
||||
import glob
|
||||
import io
|
||||
import logging
|
||||
import os
|
||||
from pathlib import Path
|
||||
from typing import Optional
|
||||
|
||||
import numpy as np
|
||||
import soundfile as sf
|
||||
import torch
|
||||
from fastapi import FastAPI, HTTPException
|
||||
from fastapi.responses import Response
|
||||
from pydantic import BaseModel
|
||||
|
||||
from omnivoice import OmniVoice, OmniVoiceGenerationConfig
|
||||
|
||||
logging.basicConfig(level=os.environ.get("OMNIVOICE_LOG_LEVEL", "INFO"))
|
||||
log = logging.getLogger("omnivoice-api")
|
||||
|
||||
CKPT = os.environ.get("OMNIVOICE_CKPT", "k2-fsa/OmniVoice")
|
||||
VOICES_DIR = os.environ.get("OMNIVOICE_VOICES_DIR", "/app/voices")
|
||||
ASR_MODEL = os.environ.get("OMNIVOICE_ASR_MODEL", "openai/whisper-large-v3-turbo")
|
||||
NUM_STEP = int(os.environ.get("OMNIVOICE_NUM_STEP", "32"))
|
||||
GUIDANCE = float(os.environ.get("OMNIVOICE_GUIDANCE_SCALE", "2.0"))
|
||||
|
||||
app = FastAPI(title="OmniVoice TTS (asset-engine wrapper)")
|
||||
|
||||
MODEL: Optional[OmniVoice] = None
|
||||
PROMPTS: dict = {} # voice name -> VoiceClonePrompt
|
||||
SR: int = 24000
|
||||
|
||||
|
||||
class SpeechRequest(BaseModel):
|
||||
input: str
|
||||
voice: str
|
||||
response_format: str = "wav"
|
||||
model: Optional[str] = None # ignored (single model); OpenAI-compat field
|
||||
|
||||
|
||||
@app.on_event("startup")
|
||||
def _load() -> None:
|
||||
global MODEL, SR
|
||||
device = "cuda" if torch.cuda.is_available() else "cpu"
|
||||
log.info("loading OmniVoice %s on %s (asr=%s)", CKPT, device, ASR_MODEL)
|
||||
MODEL = OmniVoice.from_pretrained(
|
||||
CKPT, device_map=device, load_asr=True, asr_model_name=ASR_MODEL
|
||||
)
|
||||
SR = int(getattr(MODEL, "sampling_rate", 24000))
|
||||
|
||||
for wav in sorted(glob.glob(os.path.join(VOICES_DIR, "*.wav"))):
|
||||
name = Path(wav).stem
|
||||
if name.startswith("_"):
|
||||
continue # skip _*.wav (chatterbox test/deploy artifacts)
|
||||
try:
|
||||
PROMPTS[name] = MODEL.create_voice_clone_prompt(ref_audio=wav)
|
||||
log.info("voice ready: %s", name)
|
||||
except Exception as exc: # noqa: BLE001
|
||||
log.warning("voice %s failed to load: %s", name, exc)
|
||||
log.info("%d voices loaded: %s", len(PROMPTS), sorted(PROMPTS))
|
||||
|
||||
|
||||
@app.get("/healthz")
|
||||
def healthz():
|
||||
if MODEL is None or not PROMPTS:
|
||||
raise HTTPException(status_code=503, detail="not ready")
|
||||
return {"status": "ok", "voices": len(PROMPTS), "sampling_rate": SR}
|
||||
|
||||
|
||||
@app.get("/v1/audio/voices")
|
||||
def voices():
|
||||
return {"voices": sorted(PROMPTS.keys())}
|
||||
|
||||
|
||||
@app.post("/v1/audio/speech")
|
||||
def speech(req: SpeechRequest):
|
||||
if MODEL is None:
|
||||
raise HTTPException(status_code=503, detail="model not loaded")
|
||||
if not req.input or not req.input.strip():
|
||||
raise HTTPException(status_code=400, detail="input is empty")
|
||||
if req.response_format not in ("wav", "", None):
|
||||
raise HTTPException(status_code=400, detail="only response_format=wav is supported")
|
||||
|
||||
prompt = PROMPTS.get(req.voice)
|
||||
if prompt is None:
|
||||
raise HTTPException(
|
||||
status_code=404,
|
||||
detail=f"unknown voice '{req.voice}'; have {sorted(PROMPTS)}",
|
||||
)
|
||||
|
||||
gen = OmniVoiceGenerationConfig(num_step=NUM_STEP, guidance_scale=GUIDANCE, denoise=True)
|
||||
out = MODEL.generate(text=req.input.strip(), voice_clone_prompt=prompt, generation_config=gen)
|
||||
audio = out[0] if isinstance(out, (list, tuple)) else out
|
||||
audio = np.asarray(audio, dtype=np.float32)
|
||||
|
||||
buf = io.BytesIO()
|
||||
sf.write(buf, audio, SR, format="WAV", subtype="PCM_16")
|
||||
return Response(content=buf.getvalue(), media_type="audio/wav")
|
||||
@@ -1,9 +1,10 @@
|
||||
# OmniVoice (k2-fsa/OmniVoice) — zero-shot, massively-multilingual (600+
|
||||
# language) voice-cloning + voice-design TTS, diffusion-LM, Apache-2.0.
|
||||
# Served via upstream's own Gradio demo. NOTE: this exposes the Gradio UI
|
||||
# + Gradio API, NOT an OpenAI-compatible /v1/audio/speech endpoint — wrap
|
||||
# it later (à la stacks/index-tts/app.py) if asset-engine integration is
|
||||
# wanted. For now it's a "stand it up and try it" UI.
|
||||
# Served behind our OWN thin FastAPI wrapper (stacks/omnivoice/app.py) exposing
|
||||
# OpenAI-compatible /v1/audio/speech (+ /v1/audio/voices, /healthz) so the
|
||||
# asset-engine can consume it. Upstream ships only a Gradio demo; the wrapper
|
||||
# replaced it (2026-06-19). Voices are reference WAVs in ${OMNIVOICE_VOICES_DIR}
|
||||
# (the reused chatterbox /refs/*.wav); clone prompts are precomputed at startup.
|
||||
#
|
||||
# Build: local image from the Dockerfile in this dir. Weights download
|
||||
# from HF (k2-fsa/OmniVoice) on first boot into ${OMNIVOICE_CACHE_DIR}.
|
||||
@@ -14,6 +15,10 @@
|
||||
#
|
||||
# All tunables live in .env — edit that, not this file.
|
||||
|
||||
# Compose project name MUST equal the catalog lifecycle.stack ("omnivoice") or the
|
||||
# asset-engine liveness probe (docker ps project-name match) shows it OFFLINE.
|
||||
name: omnivoice
|
||||
|
||||
services:
|
||||
omnivoice:
|
||||
image: local/omnivoice:${OMNIVOICE_TAG:-latest}
|
||||
@@ -34,15 +39,15 @@ services:
|
||||
- ${OMNIVOICE_CACHE_DIR}:/app/hf_cache
|
||||
- ${OMNIVOICE_VOICES_DIR}:/app/voices
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "wget -q -O /dev/null http://localhost:8001/ || exit 1"]
|
||||
test: ["CMD-SHELL", "wget -q -O /dev/null http://localhost:8001/healthz || exit 1"]
|
||||
interval: 30s
|
||||
timeout: 10s
|
||||
retries: 3
|
||||
# First boot: weight pre-warm download (entrypoint) + CUDA warmup.
|
||||
start_period: 900s
|
||||
# First boot: OmniVoice + Whisper ASR pre-warm + cloning every staged voice.
|
||||
start_period: 1200s
|
||||
labels:
|
||||
- homepage.group=AI Systems
|
||||
- homepage.name=OmniVoice
|
||||
- homepage.icon=mdi-account-voice
|
||||
- homepage.description=Zero-shot multilingual voice-cloning TTS (irv-ml1, 3090)
|
||||
- homepage.href=http://10.100.79.3:${OMNIVOICE_PORT}
|
||||
- homepage.description=Zero-shot multilingual voice-cloning TTS, OpenAI /v1/audio/speech (irv-ml1, 3090)
|
||||
- homepage.href=http://10.100.79.3:${OMNIVOICE_PORT}/docs
|
||||
|
||||
Reference in New Issue
Block a user