feat: stream Donut TTS play-as-it-arrives + autoplay unlock (supersedes buffered)

Operator: play-as-it-arrives, don't wait for the whole clip. infra-ops confirmed the
Zonos gateway ALREADY streams (chunked int16 WAV, TTFB ~0.44s vs ~7s total; placeholder
0xFFFFFFFF sizes are DESIGNED for progressive <audio src>). The buffering was entirely
in our proxy, and the _finalize_wav_header rewrite (6c3c08b) FORCED it — computing the
real sizes needs the whole clip.

The fix — pipe the chunks straight through:
- tts.py: buffered tts_synthesize + _finalize_wav_header REMOVED; tts_stream (an async
  generator over the gateway's chunked response) + gateway_body added. Never buffer,
  never rewrite the placeholder header.
- server.py: /api/tts is now GET (so a browser <audio src> plays it progressively) →
  a chunked StreamingResponse piping the gateway; peeks the first chunk so a bad gateway
  OPEN still returns 503; the serialize lock is held across the stream and released on
  completion/abort; PAD rides p/a query floats.
- index.html: speakOnDone sets <audio src="/api/tts?..."> (streaming) instead of
  fetch->blob; dropped the blob machinery. AUTOPLAY UNLOCK: _unlockTtsAudio() plays a
  silent WAV within the toggle/submit gesture so the delayed play() isn't blocked — the
  actual cause of "no audio" (play() fires ~15s after the keypress, past the browser's
  transient-activation window).

Live-verified: GET /api/tts is transfer-encoding: chunked, TTFB 0.46s. Playwright with
--autoplay-policy=document-user-activation-required: the streaming <audio src> plays
progressively (currentTime advances, no decode error, no MSE fallback needed) 6.5s after
the gesture — proving the unlock's persistent element flag. 521 green.

DEC-2 amended (streaming supersedes "no streaming"); FN tts_stream / tts_endpoint updated.
This commit is contained in:
vh
2026-08-01 23:59:35 -07:00
parent 608e9a54fd
commit 7856ec5438
6 changed files with 272 additions and 359 deletions
@@ -62,12 +62,18 @@ each independently shippable. Slice order is chosen for fastest visible result.
swappable `ext-tts` LiteLLM alias. Rationale: the emotion dials (the whole
point — affect-driven voice) don't pass through `ext-tts`. Accept the
Zonos coupling; the `tts.py` client is the single swap seam if we ever move.
- **DEC-2 — full-synth latency accepted (no true streaming).** The gateway
buffers to a complete clip (~1.8s/sentence, scales). "Speak on done" gives the
whole clip after a short delay. True first-audio-early is a future gateway
enhancement (infra-ops to expose the native PCM stream); not in v1.
- **DEC-3 — wav only.** `response_format:"wav"` (16-bit RIFF). `mp3`/`opus` are
accepted but silently return mislabeled PCM — never request them.
- **DEC-2 — STREAMING, play-as-it-arrives (amended 2026-08-02, operator-directed).**
The gateway ALREADY streams: `POST /v1/audio/speech` relays a chunked int16 WAV
(transfer-encoding: chunked, placeholder 0xFFFFFFFF RIFF/data sizes) as it synthesizes
— TTFB ~0.44s vs ~7s total (infra-ops verified). So ratatoskr PROXIES THE CHUNKS
STRAIGHT THROUGH (`tts_stream`, `GET /api/tts`) and the browser plays a progressive
`<audio src>`; NEVER buffer, NEVER rewrite the placeholder header (a rewrite needs the
whole clip and defeats streaming — the bug the original buffered `tts_synthesize` +
`_finalize_wav_header` hit). The placeholder-size WAV is DESIGNED for `<audio src>`
progressive playback (validated in Chromium: plays, currentTime advances, no MSE
needed). Supersedes the original "full-synth latency accepted / no streaming."
- **DEC-3 — wav only.** `response_format:"wav"` (streaming int16 RIFF/WAVE). `mp3`/`opus`
are accepted but silently return mislabeled PCM — never request them.
- **DEC-4 — server-side proxy.** Browser → `/api/tts` (nh3-dev) → gateway. The
irv-ml1 host/URL never reaches the client (INV-TTS-1). No key exists, so
INV-003 is trivially satisfied, but the proxy still stands (browser can't
@@ -125,19 +131,17 @@ each independently shippable. Slice order is chosen for fastest visible result.
## FN blocks
### FN tts_synthesize
### FN tts_stream (replaces the buffered tts_synthesize — DEC-2 streaming)
```
tts_synthesize(text: str, *, voice: str, dials: EmotionDials, client: httpx.AsyncClient, url=ZONOS_TTS_URL) -> bytes
# POST {input:text, voice, response_format:"wav", **dials} to the Zonos gateway; return wav bytes.
# `url` (added — heid-code-review F3) is the swap seam (DEC-1): the endpoint passes app.state.tts_url;
# tests pass a respx-mocked URL. Defaults to ZONOS_TTS_URL so the parameter is inert for the common call.
precondition: text non-empty. Voice membership in /v1/voices is GATEWAY-enforced, not client-asserted
(an unknown voice surfaces as a gateway non-200 -> TtsUnavailable) — the contract + bundle
carry no local voice catalog, so a client-side check would fork a source of truth.
postcondition: returns a RIFF/WAVE container (magic 0:4 == "RIFF" AND form 8:12 == "WAVE"). The DEC-3
guard is operational ("is this wav, not HTML / mislabeled-PCM"), NOT a full 16-bit-PCM
fmt-chunk parse (heid-code-review: "16-bit" is aspirational; the guard is container-level).
error: gateway non-200 / transport failure / non-WAVE body -> TtsUnavailable (caller degrades per INV-TTS-4).
tts_stream(text, *, voice, dials, client: httpx.AsyncClient, url=ZONOS_TTS_URL) -> AsyncIterator[bytes]
# Open the gateway's CHUNKED stream (client.stream("POST", url, json=gateway_body(...))) and YIELD wav
# chunks as they synthesize. Pass through verbatim — never buffer, never rewrite the placeholder header.
# gateway_body(text, voice, dials) = {input, voice, response_format:"wav", **dials.to_body()}.
precondition: text non-empty. Voice membership in /v1/voices is GATEWAY-enforced, not client-asserted.
postcondition: yields the gateway's chunked int16 streaming WAV bytes unmodified (0xFFFFFFFF placeholder
sizes intact — the browser <audio src> plays them progressively).
error: a non-200 OPEN or connect/transport failure -> TtsUnavailable BEFORE the first chunk (so the
endpoint can still return 503); a mid-stream drop just ends the generator.
invariant: response_format is ALWAYS "wav" (DEC-3); never mp3/opus.
```
@@ -152,17 +156,19 @@ pad_to_dials(pad: PadState | None) -> EmotionDials
invariant: total over any PAD input (finite/None/out-of-range) -> valid dials, never raises.
```
### FN tts_endpoint (server.py, /api/tts)
### FN tts_endpoint (server.py, GET /api/tts)
```
POST /api/tts {text, agent_id?, pad?} -> audio/wav
GET /api/tts?text=&agent_id=&p=&a= -> audio/wav (chunked StreamingResponse)
# GET (not POST) so a browser <audio src> plays it progressively (DEC-2 streaming). Params ride the query
# string; text is capped ~2000 chars and truncated at a word boundary (URL-safe + bounds the shared-3090 hold).
steps:
- resolve voice (per-character map -> "donut"; default Cora; a non-str agent_id coerces to None).
- dials = pad_to_dials(PadState.from_obj(pad)) — the PAD is BROWSER-SENT in the request body (per DEC-7:
the console already holds live PAD from the affect_update SSE), NOT a server-side PAD lookup. (Clarified
per heid-code-review F5 — the original "the agent's current PAD if known" wording read as a server fetch.)
- reject text over the max-char budget with 413 (bug-hunt: bound before the lock); missing/non-str text -> 400.
- tts_synthesize(...) behind the serialize guard (DEC-5); return wav with Content-Type audio/wav.
- on TtsUnavailable -> 503 controlled envelope (client skips playback, INV-TTS-4).
- missing text -> 400. resolve voice (per-character map -> "donut"; default Cora).
- dials = pad_to_dials(PadState(p, a)) from the p/a query floats (DEC-7, BROWSER-SENT live PAD); malformed
p/a -> neutral read, never a 500.
- acquire the serialize lock (DEC-5, one stream at a time on the shared 3090); open tts_stream and PEEK the
first chunk so a bad gateway OPEN surfaces as 503 (INV-TTS-4) before committing a 200.
- return StreamingResponse piping the chunks; the generator's finally releases the lock + closes the client
(incl. the browser-abort path: a new turn's <audio> load() drops the GET).
```
### FN pin_kb_context (kb_bridge.py — RETIRE-READY, INV-KB-1)
@@ -195,14 +201,19 @@ pin_kb_context(question: str, agent_id: str | None, *, client) -> list[dict] #
searches in-voice natively.
```
### FN client: speakOnDone (index.html)
### FN client: speakOnDone (index.html — STREAMING, DEC-2)
```
on SSE `done`:
if !ttsEnabled(): return # INV-TTS-2
cancelInFlight() # INV-TTS-3
const wav = await fetch('/api/tts', {text: assistantText, agent_id}) # non-stream; whole clip
if !ok: return # INV-TTS-4 (silent skip)
play(wav) in the <audio> sink; a new turn start -> cancelInFlight() + audio.pause()
cancelTts() # INV-TTS-3: audio.pause()+removeAttribute(src)+load()
audio.src = "/api/tts?text=&agent_id=&p=&a=" # GET streaming URL (text sliced to the 2000 cap)
audio.play() -> ▶ voiced ; .catch -> "playback blocked" # progressive; failure non-fatal (INV-TTS-4)
AUTOPLAY UNLOCK (the load-bearing fix for "no audio"): speak fires play() in an async callback seconds after
the keypress, past the browser's transient-activation window, so a bare play() is blocked. _unlockTtsAudio()
plays a tiny silent WAV inside a REAL gesture (toggle-on + each prompt submit), which grants the <audio>
element a PERSISTENT "may-play" flag so the later streaming play() isn't refused. Validated in Chromium with
--autoplay-policy=document-user-activation-required: play succeeds 6.5s after the gesture, currentTime advances.
```
## Slice plan