feat(omnivoice): streaming /tts + language-safe sanitizer

Add a live-consumer streaming path and text sanitation to the OmniVoice
wrapper, so it can front speech-to-speech chat engines (not just the
asset-engine's batch WAV use).

- POST /tts: chunked 24 kHz mono s16le PCM (or open-ended WAV), driven by
  the adaptive buffer-ratchet scheduler. Emits the first sentence
  immediately, then ratchets chunk size up on OmniVoice's ~40x realtime
  headroom -> sub-second time-to-first-audio. Wire-compatible with
  chatterbox-fast /tts (both 24 kHz mono PCM). Batch /v1/audio/speech is
  unchanged for asset/file callers.

- scheduler.py: VENDORED byte-faithful copy of chatterbox-fast's pure-
  Python (torch-free) scheduler, pinned to commit 7631462 (v0.1.0/v0.1.1).
  Vendor-copy over a shared package (operator call 2026-06-19): the module
  has no GPU deps, so reuse it without dragging chatterbox-fast's torch
  tree into this image. Promote to a shared package only on a 3rd consumer
  or real drift.

- sanitize.py: language-safe TTS sanitizer run on both endpoints. Strips
  markdown, <think> blocks, HTML, and model control tokens; deliberately
  SKIPS the fork's English-only number/phone normalization that would
  corrupt OmniVoice's 600-language input. Preserves [laughter]-style tags.

- Refactor: shared GenParams base for SpeechRequest + TTSStreamRequest;
  single GEN_LOCK serializes generation (single-stream interactive).

- Dockerfile/playbook: copy + upload the two new modules; build-time
  `import app` smoke; correct stale "Gradio demo / no FastAPI" comments.
This commit is contained in:
vh
2026-06-19 22:47:15 -07:00
parent 826c2a6a64
commit 288d085236
7 changed files with 671 additions and 67 deletions
+33 -5
View File
@@ -18,19 +18,47 @@ RTF as low as ~0.025 (≈40× real-time). **Apache-2.0** — commercially clean
## How it's served
Behind our own thin **FastAPI wrapper** ([`app.py`](app.py)) — upstream ships
only a Gradio demo, which we replaced (2026-06-19) so the **asset-engine** can
consume it. Endpoints on `http://10.100.79.3:8199`:
only a Gradio demo, which we replaced (2026-06-19). The wrapper serves **two
consumption modes** on `http://10.100.79.3:8199`:
| Endpoint | Purpose |
|---|---|
| `POST /v1/audio/speech` | OpenAI-style `{input, voice, response_format=wav}` → 24 kHz PCM_16 mono |
| `POST /v1/audio/speech` | **Batch** OpenAI-style `{input, voice, instruct, language, …}` → one 24 kHz PCM_16 mono WAV. For the **asset-engine** (form-driven asset generation). |
| `POST /tts` | **Streaming** chunked 24 kHz mono `s16le` PCM (`format=pcm`, default) or open-ended WAV — for **live speech-to-speech chat engines**. Wire-compatible with chatterbox-fast's `/tts`. |
| `GET /v1/audio/voices` | `{"voices": [...]}` — the staged clone targets |
| `GET /v1/audio/languages` | `{"languages": ["Auto", …]}` — 600+ |
| `GET /v1/audio/instruct-items` | `{"instruct_items": [...]}` — controlled voice-DESIGN tags |
| `GET /healthz` | readiness (200 once model + ≥1 voice loaded) |
The wrapper loads OmniVoice + a Whisper ASR and **precomputes a voice-clone
prompt per staged reference WAV at startup** (Whisper auto-transcribes each
reference), so per-request latency is just generation. v1 is **clone-only** —
OmniVoice's voice-*design* / language / instruct controls aren't exposed yet.
reference), so per-request latency is just generation. The full generation
surface is exposed: zero-shot **clone** (`voice`) and/or voice-**design**
(`instruct`), plus `language` / `speed` / `duration` and the diffusion knobs.
### Streaming — sub-second time-to-first-audio
`POST /tts` (`stream=true`, default) runs the **adaptive buffer-ratchet
scheduler** vendored from chatterbox-fast ([`scheduler.py`](scheduler.py)): it
emits the first sentence immediately and ratchets chunk size up on OmniVoice's
~40× realtime headroom, so a live consumer hears speech start in ~tens of ms
instead of waiting for the whole utterance. `stream=false` is a whole-text
one-shot for A/B. Scheduler tunables (`margin`, `margin_first`, `rtf_prior`,
`sec_per_char_prior`) are per-request overrides.
`scheduler.py` is a **vendored byte-faithful copy** (not a dependency) of
chatterbox-fast's pure-Python, torch-free scheduler — see its header for the
pinned commit. It is reused without dragging chatterbox-fast's GPU dependency
tree into this image; re-vendor on upstream change rather than editing in place.
### Text sanitization
Both endpoints run `input` through a **language-safe sanitizer**
([`sanitize.py`](sanitize.py)) before synthesis: it strips markdown, LLM
artifacts (`<think>` blocks), HTML, and model control tokens, but deliberately
**skips** English-only number/phone/email normalization that would corrupt
OmniVoice's multilingual input. OmniVoice's own `[laughter]`-style symbols are
preserved.
### Voices — reused from chatterbox