feat(omnivoice): streaming /tts + language-safe sanitizer
Add a live-consumer streaming path and text sanitation to the OmniVoice wrapper, so it can front speech-to-speech chat engines (not just the asset-engine's batch WAV use). - POST /tts: chunked 24 kHz mono s16le PCM (or open-ended WAV), driven by the adaptive buffer-ratchet scheduler. Emits the first sentence immediately, then ratchets chunk size up on OmniVoice's ~40x realtime headroom -> sub-second time-to-first-audio. Wire-compatible with chatterbox-fast /tts (both 24 kHz mono PCM). Batch /v1/audio/speech is unchanged for asset/file callers. - scheduler.py: VENDORED byte-faithful copy of chatterbox-fast's pure- Python (torch-free) scheduler, pinned to commit 7631462 (v0.1.0/v0.1.1). Vendor-copy over a shared package (operator call 2026-06-19): the module has no GPU deps, so reuse it without dragging chatterbox-fast's torch tree into this image. Promote to a shared package only on a 3rd consumer or real drift. - sanitize.py: language-safe TTS sanitizer run on both endpoints. Strips markdown, <think> blocks, HTML, and model control tokens; deliberately SKIPS the fork's English-only number/phone normalization that would corrupt OmniVoice's 600-language input. Preserves [laughter]-style tags. - Refactor: shared GenParams base for SpeechRequest + TTSStreamRequest; single GEN_LOCK serializes generation (single-stream interactive). - Dockerfile/playbook: copy + upload the two new modules; build-time `import app` smoke; correct stale "Gradio demo / no FastAPI" comments.
This commit is contained in:
@@ -18,19 +18,47 @@ RTF as low as ~0.025 (≈40× real-time). **Apache-2.0** — commercially clean
|
||||
## How it's served
|
||||
|
||||
Behind our own thin **FastAPI wrapper** ([`app.py`](app.py)) — upstream ships
|
||||
only a Gradio demo, which we replaced (2026-06-19) so the **asset-engine** can
|
||||
consume it. Endpoints on `http://10.100.79.3:8199`:
|
||||
only a Gradio demo, which we replaced (2026-06-19). The wrapper serves **two
|
||||
consumption modes** on `http://10.100.79.3:8199`:
|
||||
|
||||
| Endpoint | Purpose |
|
||||
|---|---|
|
||||
| `POST /v1/audio/speech` | OpenAI-style `{input, voice, response_format=wav}` → 24 kHz PCM_16 mono |
|
||||
| `POST /v1/audio/speech` | **Batch** OpenAI-style `{input, voice, instruct, language, …}` → one 24 kHz PCM_16 mono WAV. For the **asset-engine** (form-driven asset generation). |
|
||||
| `POST /tts` | **Streaming** chunked 24 kHz mono `s16le` PCM (`format=pcm`, default) or open-ended WAV — for **live speech-to-speech chat engines**. Wire-compatible with chatterbox-fast's `/tts`. |
|
||||
| `GET /v1/audio/voices` | `{"voices": [...]}` — the staged clone targets |
|
||||
| `GET /v1/audio/languages` | `{"languages": ["Auto", …]}` — 600+ |
|
||||
| `GET /v1/audio/instruct-items` | `{"instruct_items": [...]}` — controlled voice-DESIGN tags |
|
||||
| `GET /healthz` | readiness (200 once model + ≥1 voice loaded) |
|
||||
|
||||
The wrapper loads OmniVoice + a Whisper ASR and **precomputes a voice-clone
|
||||
prompt per staged reference WAV at startup** (Whisper auto-transcribes each
|
||||
reference), so per-request latency is just generation. v1 is **clone-only** —
|
||||
OmniVoice's voice-*design* / language / instruct controls aren't exposed yet.
|
||||
reference), so per-request latency is just generation. The full generation
|
||||
surface is exposed: zero-shot **clone** (`voice`) and/or voice-**design**
|
||||
(`instruct`), plus `language` / `speed` / `duration` and the diffusion knobs.
|
||||
|
||||
### Streaming — sub-second time-to-first-audio
|
||||
|
||||
`POST /tts` (`stream=true`, default) runs the **adaptive buffer-ratchet
|
||||
scheduler** vendored from chatterbox-fast ([`scheduler.py`](scheduler.py)): it
|
||||
emits the first sentence immediately and ratchets chunk size up on OmniVoice's
|
||||
~40× realtime headroom, so a live consumer hears speech start in ~tens of ms
|
||||
instead of waiting for the whole utterance. `stream=false` is a whole-text
|
||||
one-shot for A/B. Scheduler tunables (`margin`, `margin_first`, `rtf_prior`,
|
||||
`sec_per_char_prior`) are per-request overrides.
|
||||
|
||||
`scheduler.py` is a **vendored byte-faithful copy** (not a dependency) of
|
||||
chatterbox-fast's pure-Python, torch-free scheduler — see its header for the
|
||||
pinned commit. It is reused without dragging chatterbox-fast's GPU dependency
|
||||
tree into this image; re-vendor on upstream change rather than editing in place.
|
||||
|
||||
### Text sanitization
|
||||
|
||||
Both endpoints run `input` through a **language-safe sanitizer**
|
||||
([`sanitize.py`](sanitize.py)) before synthesis: it strips markdown, LLM
|
||||
artifacts (`<think>` blocks), HTML, and model control tokens, but deliberately
|
||||
**skips** English-only number/phone/email normalization that would corrupt
|
||||
OmniVoice's multilingual input. OmniVoice's own `[laughter]`-style symbols are
|
||||
preserved.
|
||||
|
||||
### Voices — reused from chatterbox
|
||||
|
||||
|
||||
Reference in New Issue
Block a user