feat: stream Donut TTS play-as-it-arrives + autoplay unlock (supersedes buffered)
Operator: play-as-it-arrives, don't wait for the whole clip. infra-ops confirmed the
Zonos gateway ALREADY streams (chunked int16 WAV, TTFB ~0.44s vs ~7s total; placeholder
0xFFFFFFFF sizes are DESIGNED for progressive <audio src>). The buffering was entirely
in our proxy, and the _finalize_wav_header rewrite (6c3c08b) FORCED it — computing the
real sizes needs the whole clip.
The fix — pipe the chunks straight through:
- tts.py: buffered tts_synthesize + _finalize_wav_header REMOVED; tts_stream (an async
generator over the gateway's chunked response) + gateway_body added. Never buffer,
never rewrite the placeholder header.
- server.py: /api/tts is now GET (so a browser <audio src> plays it progressively) →
a chunked StreamingResponse piping the gateway; peeks the first chunk so a bad gateway
OPEN still returns 503; the serialize lock is held across the stream and released on
completion/abort; PAD rides p/a query floats.
- index.html: speakOnDone sets <audio src="/api/tts?..."> (streaming) instead of
fetch->blob; dropped the blob machinery. AUTOPLAY UNLOCK: _unlockTtsAudio() plays a
silent WAV within the toggle/submit gesture so the delayed play() isn't blocked — the
actual cause of "no audio" (play() fires ~15s after the keypress, past the browser's
transient-activation window).
Live-verified: GET /api/tts is transfer-encoding: chunked, TTFB 0.46s. Playwright with
--autoplay-policy=document-user-activation-required: the streaming <audio src> plays
progressively (currentTime advances, no decode error, no MSE fallback needed) 6.5s after
the gesture — proving the unlock's persistent element flag. 521 green.
DEC-2 amended (streaming supersedes "no streaming"); FN tts_stream / tts_endpoint updated.
This commit is contained in:
@@ -62,12 +62,18 @@ each independently shippable. Slice order is chosen for fastest visible result.
|
||||
swappable `ext-tts` LiteLLM alias. Rationale: the emotion dials (the whole
|
||||
point — affect-driven voice) don't pass through `ext-tts`. Accept the
|
||||
Zonos coupling; the `tts.py` client is the single swap seam if we ever move.
|
||||
- **DEC-2 — full-synth latency accepted (no true streaming).** The gateway
|
||||
buffers to a complete clip (~1.8s/sentence, scales). "Speak on done" gives the
|
||||
whole clip after a short delay. True first-audio-early is a future gateway
|
||||
enhancement (infra-ops to expose the native PCM stream); not in v1.
|
||||
- **DEC-3 — wav only.** `response_format:"wav"` (16-bit RIFF). `mp3`/`opus` are
|
||||
accepted but silently return mislabeled PCM — never request them.
|
||||
- **DEC-2 — STREAMING, play-as-it-arrives (amended 2026-08-02, operator-directed).**
|
||||
The gateway ALREADY streams: `POST /v1/audio/speech` relays a chunked int16 WAV
|
||||
(transfer-encoding: chunked, placeholder 0xFFFFFFFF RIFF/data sizes) as it synthesizes
|
||||
— TTFB ~0.44s vs ~7s total (infra-ops verified). So ratatoskr PROXIES THE CHUNKS
|
||||
STRAIGHT THROUGH (`tts_stream`, `GET /api/tts`) and the browser plays a progressive
|
||||
`<audio src>`; NEVER buffer, NEVER rewrite the placeholder header (a rewrite needs the
|
||||
whole clip and defeats streaming — the bug the original buffered `tts_synthesize` +
|
||||
`_finalize_wav_header` hit). The placeholder-size WAV is DESIGNED for `<audio src>`
|
||||
progressive playback (validated in Chromium: plays, currentTime advances, no MSE
|
||||
needed). Supersedes the original "full-synth latency accepted / no streaming."
|
||||
- **DEC-3 — wav only.** `response_format:"wav"` (streaming int16 RIFF/WAVE). `mp3`/`opus`
|
||||
are accepted but silently return mislabeled PCM — never request them.
|
||||
- **DEC-4 — server-side proxy.** Browser → `/api/tts` (nh3-dev) → gateway. The
|
||||
irv-ml1 host/URL never reaches the client (INV-TTS-1). No key exists, so
|
||||
INV-003 is trivially satisfied, but the proxy still stands (browser can't
|
||||
@@ -125,19 +131,17 @@ each independently shippable. Slice order is chosen for fastest visible result.
|
||||
|
||||
## FN blocks
|
||||
|
||||
### FN tts_synthesize
|
||||
### FN tts_stream (replaces the buffered tts_synthesize — DEC-2 streaming)
|
||||
```
|
||||
tts_synthesize(text: str, *, voice: str, dials: EmotionDials, client: httpx.AsyncClient, url=ZONOS_TTS_URL) -> bytes
|
||||
# POST {input:text, voice, response_format:"wav", **dials} to the Zonos gateway; return wav bytes.
|
||||
# `url` (added — heid-code-review F3) is the swap seam (DEC-1): the endpoint passes app.state.tts_url;
|
||||
# tests pass a respx-mocked URL. Defaults to ZONOS_TTS_URL so the parameter is inert for the common call.
|
||||
precondition: text non-empty. Voice membership in /v1/voices is GATEWAY-enforced, not client-asserted
|
||||
(an unknown voice surfaces as a gateway non-200 -> TtsUnavailable) — the contract + bundle
|
||||
carry no local voice catalog, so a client-side check would fork a source of truth.
|
||||
postcondition: returns a RIFF/WAVE container (magic 0:4 == "RIFF" AND form 8:12 == "WAVE"). The DEC-3
|
||||
guard is operational ("is this wav, not HTML / mislabeled-PCM"), NOT a full 16-bit-PCM
|
||||
fmt-chunk parse (heid-code-review: "16-bit" is aspirational; the guard is container-level).
|
||||
error: gateway non-200 / transport failure / non-WAVE body -> TtsUnavailable (caller degrades per INV-TTS-4).
|
||||
tts_stream(text, *, voice, dials, client: httpx.AsyncClient, url=ZONOS_TTS_URL) -> AsyncIterator[bytes]
|
||||
# Open the gateway's CHUNKED stream (client.stream("POST", url, json=gateway_body(...))) and YIELD wav
|
||||
# chunks as they synthesize. Pass through verbatim — never buffer, never rewrite the placeholder header.
|
||||
# gateway_body(text, voice, dials) = {input, voice, response_format:"wav", **dials.to_body()}.
|
||||
precondition: text non-empty. Voice membership in /v1/voices is GATEWAY-enforced, not client-asserted.
|
||||
postcondition: yields the gateway's chunked int16 streaming WAV bytes unmodified (0xFFFFFFFF placeholder
|
||||
sizes intact — the browser <audio src> plays them progressively).
|
||||
error: a non-200 OPEN or connect/transport failure -> TtsUnavailable BEFORE the first chunk (so the
|
||||
endpoint can still return 503); a mid-stream drop just ends the generator.
|
||||
invariant: response_format is ALWAYS "wav" (DEC-3); never mp3/opus.
|
||||
```
|
||||
|
||||
@@ -152,17 +156,19 @@ pad_to_dials(pad: PadState | None) -> EmotionDials
|
||||
invariant: total over any PAD input (finite/None/out-of-range) -> valid dials, never raises.
|
||||
```
|
||||
|
||||
### FN tts_endpoint (server.py, /api/tts)
|
||||
### FN tts_endpoint (server.py, GET /api/tts)
|
||||
```
|
||||
POST /api/tts {text, agent_id?, pad?} -> audio/wav
|
||||
GET /api/tts?text=&agent_id=&p=&a= -> audio/wav (chunked StreamingResponse)
|
||||
# GET (not POST) so a browser <audio src> plays it progressively (DEC-2 streaming). Params ride the query
|
||||
# string; text is capped ~2000 chars and truncated at a word boundary (URL-safe + bounds the shared-3090 hold).
|
||||
steps:
|
||||
- resolve voice (per-character map -> "donut"; default Cora; a non-str agent_id coerces to None).
|
||||
- dials = pad_to_dials(PadState.from_obj(pad)) — the PAD is BROWSER-SENT in the request body (per DEC-7:
|
||||
the console already holds live PAD from the affect_update SSE), NOT a server-side PAD lookup. (Clarified
|
||||
per heid-code-review F5 — the original "the agent's current PAD if known" wording read as a server fetch.)
|
||||
- reject text over the max-char budget with 413 (bug-hunt: bound before the lock); missing/non-str text -> 400.
|
||||
- tts_synthesize(...) behind the serialize guard (DEC-5); return wav with Content-Type audio/wav.
|
||||
- on TtsUnavailable -> 503 controlled envelope (client skips playback, INV-TTS-4).
|
||||
- missing text -> 400. resolve voice (per-character map -> "donut"; default Cora).
|
||||
- dials = pad_to_dials(PadState(p, a)) from the p/a query floats (DEC-7, BROWSER-SENT live PAD); malformed
|
||||
p/a -> neutral read, never a 500.
|
||||
- acquire the serialize lock (DEC-5, one stream at a time on the shared 3090); open tts_stream and PEEK the
|
||||
first chunk so a bad gateway OPEN surfaces as 503 (INV-TTS-4) before committing a 200.
|
||||
- return StreamingResponse piping the chunks; the generator's finally releases the lock + closes the client
|
||||
(incl. the browser-abort path: a new turn's <audio> load() drops the GET).
|
||||
```
|
||||
|
||||
### FN pin_kb_context (kb_bridge.py — RETIRE-READY, INV-KB-1)
|
||||
@@ -195,14 +201,19 @@ pin_kb_context(question: str, agent_id: str | None, *, client) -> list[dict] #
|
||||
searches in-voice natively.
|
||||
```
|
||||
|
||||
### FN client: speakOnDone (index.html)
|
||||
### FN client: speakOnDone (index.html — STREAMING, DEC-2)
|
||||
```
|
||||
on SSE `done`:
|
||||
if !ttsEnabled(): return # INV-TTS-2
|
||||
cancelInFlight() # INV-TTS-3
|
||||
const wav = await fetch('/api/tts', {text: assistantText, agent_id}) # non-stream; whole clip
|
||||
if !ok: return # INV-TTS-4 (silent skip)
|
||||
play(wav) in the <audio> sink; a new turn start -> cancelInFlight() + audio.pause()
|
||||
cancelTts() # INV-TTS-3: audio.pause()+removeAttribute(src)+load()
|
||||
audio.src = "/api/tts?text=&agent_id=&p=&a=" # GET streaming URL (text sliced to the 2000 cap)
|
||||
audio.play() -> ▶ voiced ; .catch -> "playback blocked" # progressive; failure non-fatal (INV-TTS-4)
|
||||
|
||||
AUTOPLAY UNLOCK (the load-bearing fix for "no audio"): speak fires play() in an async callback seconds after
|
||||
the keypress, past the browser's transient-activation window, so a bare play() is blocked. _unlockTtsAudio()
|
||||
plays a tiny silent WAV inside a REAL gesture (toggle-on + each prompt submit), which grants the <audio>
|
||||
element a PERSISTENT "may-play" flag so the later streaming play() isn't refused. Validated in Chromium with
|
||||
--autoplay-policy=document-user-activation-required: play succeeds 6.5s after the gesture, currentTime advances.
|
||||
```
|
||||
|
||||
## Slice plan
|
||||
|
||||
Reference in New Issue
Block a user