feat: stream Donut TTS play-as-it-arrives + autoplay unlock (supersedes buffered)

Operator: play-as-it-arrives, don't wait for the whole clip. infra-ops confirmed the
Zonos gateway ALREADY streams (chunked int16 WAV, TTFB ~0.44s vs ~7s total; placeholder
0xFFFFFFFF sizes are DESIGNED for progressive <audio src>). The buffering was entirely
in our proxy, and the _finalize_wav_header rewrite (6c3c08b) FORCED it — computing the
real sizes needs the whole clip.

The fix — pipe the chunks straight through:
- tts.py: buffered tts_synthesize + _finalize_wav_header REMOVED; tts_stream (an async
  generator over the gateway's chunked response) + gateway_body added. Never buffer,
  never rewrite the placeholder header.
- server.py: /api/tts is now GET (so a browser <audio src> plays it progressively) →
  a chunked StreamingResponse piping the gateway; peeks the first chunk so a bad gateway
  OPEN still returns 503; the serialize lock is held across the stream and released on
  completion/abort; PAD rides p/a query floats.
- index.html: speakOnDone sets <audio src="/api/tts?..."> (streaming) instead of
  fetch->blob; dropped the blob machinery. AUTOPLAY UNLOCK: _unlockTtsAudio() plays a
  silent WAV within the toggle/submit gesture so the delayed play() isn't blocked — the
  actual cause of "no audio" (play() fires ~15s after the keypress, past the browser's
  transient-activation window).

Live-verified: GET /api/tts is transfer-encoding: chunked, TTFB 0.46s. Playwright with
--autoplay-policy=document-user-activation-required: the streaming <audio src> plays
progressively (currentTime advances, no decode error, no MSE fallback needed) 6.5s after
the gesture — proving the unlock's persistent element flag. 521 green.

DEC-2 amended (streaming supersedes "no streaming"); FN tts_stream / tts_endpoint updated.
This commit is contained in:
2026-08-01 23:59:35 -07:00
parent 608e9a54fd
commit 7856ec5438
6 changed files with 272 additions and 359 deletions
@@ -62,12 +62,18 @@ each independently shippable. Slice order is chosen for fastest visible result.
swappable `ext-tts` LiteLLM alias. Rationale: the emotion dials (the whole
point — affect-driven voice) don't pass through `ext-tts`. Accept the
Zonos coupling; the `tts.py` client is the single swap seam if we ever move.
- **DEC-2 — full-synth latency accepted (no true streaming).** The gateway
buffers to a complete clip (~1.8s/sentence, scales). "Speak on done" gives the
whole clip after a short delay. True first-audio-early is a future gateway
enhancement (infra-ops to expose the native PCM stream); not in v1.
- **DEC-3 — wav only.** `response_format:"wav"` (16-bit RIFF). `mp3`/`opus` are
accepted but silently return mislabeled PCM — never request them.
- **DEC-2 — STREAMING, play-as-it-arrives (amended 2026-08-02, operator-directed).**
The gateway ALREADY streams: `POST /v1/audio/speech` relays a chunked int16 WAV
(transfer-encoding: chunked, placeholder 0xFFFFFFFF RIFF/data sizes) as it synthesizes
— TTFB ~0.44s vs ~7s total (infra-ops verified). So ratatoskr PROXIES THE CHUNKS
STRAIGHT THROUGH (`tts_stream`, `GET /api/tts`) and the browser plays a progressive
`<audio src>`; NEVER buffer, NEVER rewrite the placeholder header (a rewrite needs the
whole clip and defeats streaming — the bug the original buffered `tts_synthesize` +
`_finalize_wav_header` hit). The placeholder-size WAV is DESIGNED for `<audio src>`
progressive playback (validated in Chromium: plays, currentTime advances, no MSE
needed). Supersedes the original "full-synth latency accepted / no streaming."
- **DEC-3 — wav only.** `response_format:"wav"` (streaming int16 RIFF/WAVE). `mp3`/`opus`
are accepted but silently return mislabeled PCM — never request them.
- **DEC-4 — server-side proxy.** Browser → `/api/tts` (nh3-dev) → gateway. The
irv-ml1 host/URL never reaches the client (INV-TTS-1). No key exists, so
INV-003 is trivially satisfied, but the proxy still stands (browser can't
@@ -125,19 +131,17 @@ each independently shippable. Slice order is chosen for fastest visible result.
## FN blocks
### FN tts_synthesize
### FN tts_stream (replaces the buffered tts_synthesize — DEC-2 streaming)
```
tts_synthesize(text: str, *, voice: str, dials: EmotionDials, client: httpx.AsyncClient, url=ZONOS_TTS_URL) -> bytes
# POST {input:text, voice, response_format:"wav", **dials} to the Zonos gateway; return wav bytes.
# `url` (added — heid-code-review F3) is the swap seam (DEC-1): the endpoint passes app.state.tts_url;
# tests pass a respx-mocked URL. Defaults to ZONOS_TTS_URL so the parameter is inert for the common call.
precondition: text non-empty. Voice membership in /v1/voices is GATEWAY-enforced, not client-asserted
(an unknown voice surfaces as a gateway non-200 -> TtsUnavailable) — the contract + bundle
carry no local voice catalog, so a client-side check would fork a source of truth.
postcondition: returns a RIFF/WAVE container (magic 0:4 == "RIFF" AND form 8:12 == "WAVE"). The DEC-3
guard is operational ("is this wav, not HTML / mislabeled-PCM"), NOT a full 16-bit-PCM
fmt-chunk parse (heid-code-review: "16-bit" is aspirational; the guard is container-level).
error: gateway non-200 / transport failure / non-WAVE body -> TtsUnavailable (caller degrades per INV-TTS-4).
tts_stream(text, *, voice, dials, client: httpx.AsyncClient, url=ZONOS_TTS_URL) -> AsyncIterator[bytes]
# Open the gateway's CHUNKED stream (client.stream("POST", url, json=gateway_body(...))) and YIELD wav
# chunks as they synthesize. Pass through verbatim — never buffer, never rewrite the placeholder header.
# gateway_body(text, voice, dials) = {input, voice, response_format:"wav", **dials.to_body()}.
precondition: text non-empty. Voice membership in /v1/voices is GATEWAY-enforced, not client-asserted.
postcondition: yields the gateway's chunked int16 streaming WAV bytes unmodified (0xFFFFFFFF placeholder
sizes intact — the browser <audio src> plays them progressively).
error: a non-200 OPEN or connect/transport failure -> TtsUnavailable BEFORE the first chunk (so the
endpoint can still return 503); a mid-stream drop just ends the generator.
invariant: response_format is ALWAYS "wav" (DEC-3); never mp3/opus.
```
@@ -152,17 +156,19 @@ pad_to_dials(pad: PadState | None) -> EmotionDials
invariant: total over any PAD input (finite/None/out-of-range) -> valid dials, never raises.
```
### FN tts_endpoint (server.py, /api/tts)
### FN tts_endpoint (server.py, GET /api/tts)
```
POST /api/tts {text, agent_id?, pad?} -> audio/wav
GET /api/tts?text=&agent_id=&p=&a= -> audio/wav (chunked StreamingResponse)
# GET (not POST) so a browser <audio src> plays it progressively (DEC-2 streaming). Params ride the query
# string; text is capped ~2000 chars and truncated at a word boundary (URL-safe + bounds the shared-3090 hold).
steps:
- resolve voice (per-character map -> "donut"; default Cora; a non-str agent_id coerces to None).
- dials = pad_to_dials(PadState.from_obj(pad)) — the PAD is BROWSER-SENT in the request body (per DEC-7:
the console already holds live PAD from the affect_update SSE), NOT a server-side PAD lookup. (Clarified
per heid-code-review F5 — the original "the agent's current PAD if known" wording read as a server fetch.)
- reject text over the max-char budget with 413 (bug-hunt: bound before the lock); missing/non-str text -> 400.
- tts_synthesize(...) behind the serialize guard (DEC-5); return wav with Content-Type audio/wav.
- on TtsUnavailable -> 503 controlled envelope (client skips playback, INV-TTS-4).
- missing text -> 400. resolve voice (per-character map -> "donut"; default Cora).
- dials = pad_to_dials(PadState(p, a)) from the p/a query floats (DEC-7, BROWSER-SENT live PAD); malformed
p/a -> neutral read, never a 500.
- acquire the serialize lock (DEC-5, one stream at a time on the shared 3090); open tts_stream and PEEK the
first chunk so a bad gateway OPEN surfaces as 503 (INV-TTS-4) before committing a 200.
- return StreamingResponse piping the chunks; the generator's finally releases the lock + closes the client
(incl. the browser-abort path: a new turn's <audio> load() drops the GET).
```
### FN pin_kb_context (kb_bridge.py — RETIRE-READY, INV-KB-1)
@@ -195,14 +201,19 @@ pin_kb_context(question: str, agent_id: str | None, *, client) -> list[dict] #
searches in-voice natively.
```
### FN client: speakOnDone (index.html)
### FN client: speakOnDone (index.html — STREAMING, DEC-2)
```
on SSE `done`:
if !ttsEnabled(): return # INV-TTS-2
cancelInFlight() # INV-TTS-3
const wav = await fetch('/api/tts', {text: assistantText, agent_id}) # non-stream; whole clip
if !ok: return # INV-TTS-4 (silent skip)
play(wav) in the <audio> sink; a new turn start -> cancelInFlight() + audio.pause()
cancelTts() # INV-TTS-3: audio.pause()+removeAttribute(src)+load()
audio.src = "/api/tts?text=&agent_id=&p=&a=" # GET streaming URL (text sliced to the 2000 cap)
audio.play() -> ▶ voiced ; .catch -> "playback blocked" # progressive; failure non-fatal (INV-TTS-4)
AUTOPLAY UNLOCK (the load-bearing fix for "no audio"): speak fires play() in an async callback seconds after
the keypress, past the browser's transient-activation window, so a bare play() is blocked. _unlockTtsAudio()
plays a tiny silent WAV inside a REAL gesture (toggle-on + each prompt submit), which grants the <audio>
element a PERSISTENT "may-play" flag so the later streaming play() isn't refused. Validated in Chromium with
--autoplay-policy=document-user-activation-required: play succeeds 6.5s after the gesture, currentTime advances.
```
## Slice plan
+26 -46
View File
@@ -14,8 +14,7 @@ Foot-guns (verified live 2026-08-02):
from __future__ import annotations
import struct
from collections.abc import Mapping
from collections.abc import AsyncIterator, Mapping
from dataclasses import dataclass
import httpx
@@ -98,27 +97,6 @@ class EmotionDials:
}
def _finalize_wav_header(data: bytes) -> bytes:
"""Rewrite the RIFF + data chunk sizes with the real byte counts.
The Zonos gateway emits a STREAMING wav header — the RIFF chunk size (offset 4)
and the data chunk size are both 0xFFFFFFFF ("unknown length"), because it can
stream. A browser <audio> element playing a fully-downloaded blob needs a finite,
correctly-sized WAV to decode it; a 0xFFFFFFFF length reads as raw/streaming PCM
and won't play (operator-reported). Now that the whole clip is buffered we know the
real sizes, so patch them in. Idempotent — writing an already-correct size is a
no-op; safe if the data chunk isn't found (leaves those bytes untouched).
"""
if len(data) < 44: # shorter than a canonical PCM header — nothing to patch
return data
buf = bytearray(data)
struct.pack_into("<I", buf, 4, len(buf) - 8) # RIFF chunk size = file len - 8
data_pos = buf.find(b"data", 12) # first "data" after the fmt chunk = the data chunk
if data_pos != -1 and data_pos + 8 <= len(buf):
struct.pack_into("<I", buf, data_pos + 4, len(buf) - (data_pos + 8))
return bytes(buf)
def _clamp(x: float, lo: float, hi: float) -> float:
"""Clamp to [lo, hi], NaN-safe: a NaN axis (a dead/malformed signal) → 0.0
rather than propagating through max/min into the gateway."""
@@ -144,37 +122,39 @@ def pad_to_dials(pad: PadState | None) -> EmotionDials:
)
async def tts_synthesize(
def gateway_body(text: str, voice: str, dials: EmotionDials) -> dict:
"""The Zonos gateway POST body — response_format is ALWAYS "wav" (DEC-3)."""
return {"input": text, "voice": voice, "response_format": "wav", **dials.to_body()}
async def tts_stream(
text: str,
*,
voice: str,
dials: EmotionDials,
client: httpx.AsyncClient,
url: str = ZONOS_TTS_URL,
) -> bytes:
"""POST {input, voice, response_format:"wav", **dials} to the Zonos gateway;
return 16-bit RIFF/WAVE bytes (FN tts_synthesize).
) -> AsyncIterator[bytes]:
"""Open the gateway's CHUNKED stream and yield WAV bytes as they synthesize.
response_format is ALWAYS "wav" (DEC-3). Any non-200, transport failure, or
non-wav body → TtsUnavailable — the caller degrades (INV-TTS-4).
The gateway emits a streaming int16 WAV (RIFF/data sizes = 0xFFFFFFFF placeholders)
over `transfer-encoding: chunked`, TTFB ~0.44s vs ~7s total (infra-ops verified) —
designed to be played progressively by a browser <audio>. So we PROXY THE CHUNKS
STRAIGHT THROUGH: never buffer, never rewrite the header (that would force us to wait
for the whole clip and defeat the streaming — the bug this replaces).
Raises TtsUnavailable if the gateway open is a non-200 or the connect/transport
fails — BEFORE the first chunk, so the endpoint can still return a 503 (INV-TTS-4).
A drop mid-stream just ends the generator (the browser has already played the head).
"""
assert text, "tts_synthesize: text must be non-empty (the endpoint guards this)"
body = {"input": text, "voice": voice, "response_format": "wav", **dials.to_body()}
assert text, "tts_stream: text must be non-empty (the endpoint guards this)"
try:
resp = await client.post(url, json=body)
async with client.stream("POST", url, json=gateway_body(text, voice, dials)) as resp:
if resp.status_code != 200:
raise TtsUnavailable(
f"gateway status {resp.status_code}", status=resp.status_code
)
async for chunk in resp.aiter_bytes():
yield chunk
except httpx.RequestError as exc:
raise TtsUnavailable(f"gateway transport failure: {exc}") from exc
if resp.status_code != 200:
raise TtsUnavailable(
f"gateway status {resp.status_code}", status=resp.status_code
)
data = resp.content
# DEC-3 guard: the gateway MUST return a RIFF/WAVE container. A non-wav 200 (an
# error page, or the mislabeled-PCM mp3/opus trap) is treated as unavailable —
# played garbage is worse than silence. Check BOTH the RIFF magic (0:4) and the
# WAVE form tag (8:12); a `b"RIFF..."` body that isn't WAVE would still fail decode.
if data[:4] != b"RIFF" or data[8:12] != b"WAVE":
raise TtsUnavailable("gateway returned a non-wav body")
# The gateway's streaming header carries 0xFFFFFFFF sizes; rewrite them with the
# real byte counts so a browser <audio> element can decode the finite clip.
return _finalize_wav_header(data)
+66 -39
View File
@@ -69,7 +69,7 @@ from ratatoskr.tts import (
PadState,
TtsUnavailable,
pad_to_dials,
tts_synthesize,
tts_stream,
)
@@ -542,55 +542,82 @@ async def _memory_chunks_endpoint(request: Request) -> JSONResponse:
# Zonos voice; everything else falls to the gateway default. Case-folded gateway-side.
_TTS_VOICE_MAP = {"ratatoskr:donut": "donut"}
_TTS_DEFAULT_VOICE = "Cora"
# Cap the synth input before taking the process-global lock. An interview clip is a
# few sentences; a huge/hostile body would otherwise hold the shared 3090 for the full
# gateway timeout, starving every other turn's audio (heid bug-hunt: text-size DoS).
_TTS_MAX_TEXT_CHARS = 8000
# The streamed text rides the GET query string, so keep it URL-safe-short — a few
# sentences is plenty for a voiced turn, and a very long response is truncated at a word
# boundary (the full text still shows in the transcript). Also bounds the shared-3090 hold.
_TTS_MAX_TEXT_CHARS = 2000
def _truncate_at_boundary(text: str, limit: int) -> str:
"""Trim to <= limit chars, preferring the last space so we don't cut mid-word."""
if len(text) <= limit:
return text
head = text[:limit]
cut = head.rfind(" ")
return head[:cut] if cut > limit // 2 else head
async def _tts_endpoint(request: Request) -> Response:
"""POST /api/tts {text, agent_id?, pad?} → audio/wav (FN tts_endpoint, slice 2).
"""GET /api/tts?text=&agent_id=&p=&a= → audio/wav, STREAMED chunked from the Zonos
gateway (FN tts_endpoint). GET so a browser <audio src> plays it progressively; the
gateway already streams (TTFB ~0.44s vs ~7s total), so we pipe the chunks straight
through — never buffer, never rewrite the placeholder-size WAV header (that would
force buffering the whole clip and defeat the streaming).
Server-side proxy to the Zonos gateway (DEC-4 / INV-TTS-1: the gateway host
never reaches the browser). Voice resolves per-character (DEC-8); emotion dials
map the browser-sent live PAD (DEC-7 — the affect the persona pane already shows).
Serialized one-synth-at-a-time (DEC-5 / INV-TTS-3 — the gateway shares one 3090).
A gateway failure / non-wav body degrades to 503 (INV-TTS-4: the client skips
playback; the turn/transcript is unaffected)."""
try:
body = await request.json()
except (json.JSONDecodeError, ValueError, TypeError):
body = None
text = body.get("text") if isinstance(body, dict) else None
if not text or not isinstance(text, str):
Server-side proxy (DEC-4 / INV-TTS-1: the gateway host never reaches the browser).
Voice per-character (DEC-8); emotion dials from the browser-sent live PAD (DEC-7,
passed as p/a query floats). Serialized one-stream-at-a-time (DEC-5); a new turn
aborts the prior <audio> load → the GET drops → the generator's finally releases the
lock. A gateway open-failure 503 (INV-TTS-4: the client skips playback)."""
q = request.query_params
text = q.get("text")
if not text:
return JSONResponse({"error_code": "missing_text"}, status_code=400)
if len(text) > _TTS_MAX_TEXT_CHARS: # bound before the lock (bug-hunt: text-size DoS)
return JSONResponse({"error_code": "text_too_large"}, status_code=413)
agent_id = body.get("agent_id") if isinstance(body, dict) else None
if not isinstance(agent_id, str):
# Coerce a non-str (incl. an unhashable list/dict) to None BEFORE the map lookup:
# `_TTS_VOICE_MAP.get([])` would TypeError → uncaught 500, diverging from the
# submit path's identical guard (heid bug-hunt: open-world agent_id).
agent_id = None
text = _truncate_at_boundary(text, _TTS_MAX_TEXT_CHARS)
agent_id = q.get("agent_id")
voice = _TTS_VOICE_MAP.get(agent_id, _TTS_DEFAULT_VOICE)
pad_obj = body.get("pad") if isinstance(body, dict) else None
dials = pad_to_dials(PadState.from_obj(pad_obj))
pad: PadState | None = None
try:
if q.get("p") is not None and q.get("a") is not None:
pad = PadState(pleasure=float(q["p"]), arousal=float(q["a"]))
except (TypeError, ValueError):
pad = None # malformed p/a → neutral read, never a 500
dials = pad_to_dials(pad)
tts_url = request.app.state.tts_url
lock = request.app.state.tts_lock
# DEC-5: one stream at a time on the shared 3090. Held for the stream's duration and
# released in the generator's finally — including the browser-abort/cancel path.
await lock.acquire()
client = httpx.AsyncClient(
timeout=httpx.Timeout(connect=10.0, read=30.0, write=10.0, pool=10.0)
)
gen = tts_stream(text, voice=voice, dials=dials, client=client, url=tts_url)
try:
# DEC-5: serialize — a new turn's synth waits on any in-flight one (the
# client also aborts the prior request, cancelling the server task). 60s ceiling:
# generous for a multi-sentence clip, bounded so a stalled gateway can't pin the
# shared lock for two minutes (bug-hunt: lock-hold starvation).
async with lock:
async with httpx.AsyncClient(timeout=httpx.Timeout(60.0)) as client:
wav = await tts_synthesize(
text, voice=voice, dials=dials, client=client, url=tts_url
)
# Peek the first chunk so a bad gateway OPEN (non-200 / transport) surfaces as a
# 503 BEFORE we commit a 200 StreamingResponse. TTFB ~0.44s, so this is cheap.
first = await gen.__anext__()
except TtsUnavailable:
await gen.aclose()
await client.aclose()
lock.release()
return JSONResponse({"error_code": "tts_unavailable"}, status_code=503)
return Response(wav, media_type="audio/wav")
except StopAsyncIteration:
first = None
async def piped() -> AsyncIterator[bytes]:
try:
if first:
yield first
async for chunk in gen:
yield chunk
finally:
await gen.aclose() # unwinds tts_stream's `async with` → closes the gateway resp
await client.aclose()
lock.release()
return StreamingResponse(piped(), media_type="audio/wav")
async def _session_tools_endpoint(request: Request) -> JSONResponse:
@@ -804,7 +831,7 @@ def create_app(
Route("/api/sessions/{session_id}/messages", _session_messages_endpoint),
Route("/api/sessions/{session_id}/bifrost", _session_bifrost_endpoint),
Route("/api/admin/events", _admin_events_endpoint),
Route("/api/tts", _tts_endpoint, methods=["POST"]),
Route("/api/tts", _tts_endpoint, methods=["GET"]),
Route("/api/turns/{session_id}", _submit_turn_endpoint, methods=["POST"]),
Route("/api/turns/{session_id}/stream", _stream_turn_endpoint),
Route("/api/turns/{session_id}/cancel", _cancel_turn_endpoint, methods=["POST"]),
+53 -41
View File
@@ -1709,6 +1709,7 @@ async function submitPrompt() {
const content = input.value.trim();
if (!content || !state.sessionId || state.turnId) return;
cancelTts(); // INV-TTS-3: a new turn cancels prior voice
if (ttsEnabled()) _unlockTtsAudio(); // this keypress is a gesture — grant autoplay
input.value = ""; input.style.height = "20px";
let r;
@@ -1993,56 +1994,66 @@ async function cancelTurn() {
});
})();
// ---- auto-TTS (slice 2): voiced, affect-modulated playback on turn `done` -----
// Server-proxied to the Zonos gateway (/api/tts). Opt-in (INV-TTS-2), one clip at a
// time (INV-TTS-3: a new turn or a superseding synth cancels the prior fetch + audio),
// and non-blocking (INV-TTS-4: any failure logs to the ticker + skips — never the turn).
let _ttsAbort = null;
let _ttsUrl = null; // active blob object URL, tracked so cancel can revoke it
// ---- auto-TTS: voiced, affect-modulated STREAMING playback on turn `done` -----
// The <audio> element streams the chunked GET /api/tts response and plays as it arrives
// (TTFA ~0.5s), never waiting for the whole clip. Opt-in (INV-TTS-2), one stream at a
// time (INV-TTS-3: a new turn / toggle-off aborts the prior <audio> load, dropping the
// server stream), non-blocking (INV-TTS-4: any failure logs to the ticker, never the turn).
function ttsEnabled() {
try { return localStorage.getItem("ratatoskr-tts") === "1"; } catch (_) { return false; }
}
function _revokeTtsUrl() {
if (_ttsUrl) { try { URL.revokeObjectURL(_ttsUrl); } catch (_) {} _ttsUrl = null; }
// A short silent WAV (PCM/mono/44.1k/16-bit) used only to UNLOCK the <audio> element.
// speak-on-done fires play() in an async callback seconds after the user's keypress, by
// which point the browser's autoplay policy has revoked the activation and blocks it.
// Playing this once inside a real gesture (toggle-on, each prompt submit) grants the
// element a persistent "may play" flag so the later real playback isn't blocked.
const _TTS_SILENT = (() => {
const N = 256, data = 2 * N, buf = new Uint8Array(44 + data), dv = new DataView(buf.buffer);
buf.set([0x52, 0x49, 0x46, 0x46], 0); dv.setUint32(4, 36 + data, true);
buf.set([0x57, 0x41, 0x56, 0x45], 8); buf.set([0x66, 0x6d, 0x74, 0x20], 12);
dv.setUint32(16, 16, true); dv.setUint16(20, 1, true); dv.setUint16(22, 1, true);
dv.setUint32(24, 44100, true); dv.setUint32(28, 88200, true);
dv.setUint16(32, 2, true); dv.setUint16(34, 16, true);
buf.set([0x64, 0x61, 0x74, 0x61], 36); dv.setUint32(40, data, true);
let s = ""; for (const x of buf) s += String.fromCharCode(x);
return "data:audio/wav;base64," + btoa(s);
})();
let _ttsUnlocked = false;
function _unlockTtsAudio() {
if (_ttsUnlocked) return; // once per page is enough
const a = $("tts-audio"); if (!a) return;
try {
a.src = _TTS_SILENT;
const p = a.play();
if (p && p.then) p.then(() => {
_ttsUnlocked = true;
try { a.pause(); a.removeAttribute("src"); a.load(); } catch (_) {}
}).catch(() => {});
} catch (_) {}
}
function cancelTts() {
if (_ttsAbort) { try { _ttsAbort.abort(); } catch (_) {} _ttsAbort = null; }
// Stop + drop the current <audio> src; load() aborts the in-flight GET stream, which
// drops the server proxy (releasing its synth lock). No object URLs to revoke — the
// src is a streaming /api/tts URL, not a blob.
const a = $("tts-audio");
if (a) { try { a.pause(); a.removeAttribute("src"); a.load(); } catch (_) {} }
// removeAttribute("src")+load() fires NEITHER ended nor error, so the src's own
// revoke handler never runs — revoke the tracked URL here or the blob leaks per
// interrupted turn (heid bug-hunt).
_revokeTtsUrl();
}
async function speakOnDone(text, agentId, pad) {
function speakOnDone(text, agentId, pad) {
const clip = (text || "").trim();
if (!clip) return;
cancelTts(); // INV-TTS-3: stop any prior synth/playback
const ctrl = new AbortController();
_ttsAbort = ctrl;
let resp;
try {
resp = await fetch("/api/tts", {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({ text: clip, agent_id: agentId || null, pad: pad || null }),
signal: ctrl.signal,
});
} catch (_) { return; } // aborted / network → silent skip (INV-TTS-4)
if (!resp.ok) { tickerAdd("err", "tts", "unavailable " + resp.status); return; }
let url;
try { url = URL.createObjectURL(await resp.blob()); } catch (_) { return; }
if (_ttsAbort !== ctrl) { URL.revokeObjectURL(url); return; } // superseded mid-fetch
const a = $("tts-audio");
if (!a) { URL.revokeObjectURL(url); return; }
a.src = url;
_ttsUrl = url; // track for revoke on natural end OR cancel
a.onended = a.onerror = () => { _revokeTtsUrl(); };
// Playback can be refused by the browser autoplay policy until the page has an
// activation; the operator's toggle+prompt gesture generally satisfies it, and the
// catch keeps a refusal non-fatal (INV-TTS-4).
try { await a.play(); tickerAdd("ok", "tts", "▶ voiced"); }
catch (_) { _revokeTtsUrl(); tickerAdd("err", "tts", "playback blocked"); }
cancelTts(); // INV-TTS-3: stop any prior stream
const a = $("tts-audio"); if (!a) return;
const params = new URLSearchParams({ text: clip.slice(0, 2000) }); // matches the server cap
if (agentId) params.set("agent_id", agentId);
if (pad && typeof pad.pleasure === "number" && typeof pad.arousal === "number") {
params.set("p", pad.pleasure); params.set("a", pad.arousal); // affect dials (DEC-7)
}
// Progressive playback: point <audio> at the chunked stream; it plays as bytes arrive.
a.src = "/api/tts?" + params.toString();
// play() may be refused by the autoplay policy in this async callback; the prompt/toggle
// gesture unlock (_unlockTtsAudio) satisfies it, and the catch keeps a refusal non-fatal.
a.play().then(() => tickerAdd("ok", "tts", "▶ voiced"))
.catch(() => tickerAdd("err", "tts", "playback blocked"));
}
// ---- 🔊 toggle (mirrors theme / cot-toggle; default OFF, persisted) ----
(function () {
@@ -2057,7 +2068,8 @@ async function speakOnDone(text, agentId, pad) {
btn.classList.toggle("on", next);
btn.title = label(next);
try { localStorage.setItem("ratatoskr-tts", next ? "1" : "0"); } catch (_) {}
if (!next) cancelTts(); // turning off stops in-flight playback
if (next) _unlockTtsAudio(); // grant autoplay within this gesture
else cancelTts(); // turning off stops in-flight playback
});
})();
+54 -133
View File
@@ -1,17 +1,11 @@
"""Tests for ratatoskr.tts per docs/contracts/donut_voiced_interview.contract.md
(slice 2 — auto-TTS). Two units:
"""Tests for ratatoskr.tts — the STREAMING Zonos-gateway client + PAD→dial mapping.
- pad_to_dials / PadState / EmotionDials — pure, total, degrade-not-crash.
- tts_synthesize — the Zonos-gateway client (respx-mocked; no live network).
DEC-3: response_format is ALWAYS "wav". DEC-7: live PAD → emotion dials.
INV-TTS-4: a gateway failure / non-wav body → TtsUnavailable (caller degrades).
tts_stream proxies the gateway's chunked response verbatim (no buffering, no header
rewrite — the placeholder-size streaming WAV is meant to be played progressively).
pad_to_dials / PadState / EmotionDials are pure + total.
"""
import io
import math
import struct
import wave
import httpx
import pytest
@@ -21,24 +15,25 @@ from ratatoskr.tts import (
EmotionDials,
PadState,
TtsUnavailable,
gateway_body,
pad_to_dials,
tts_synthesize,
tts_stream,
)
_URL = "http://tts.example/v1/audio/speech"
# The gateway's streaming WAV bytes (placeholder 0xFFFFFFFF sizes). We pass them through
# untouched, so the content only has to round-trip.
_WAV = (
b"RIFF\xff\xff\xff\xffWAVEfmt \x10\x00\x00\x00" + b"\x00" * 20
+ b"data\xff\xff\xff\xff" + b"\x11\x22" * 64
)
def _wav(*, streaming: bool = False, nsamples: int = 8) -> bytes:
"""A canonical PCM mono/44.1k/16-bit WAV. `streaming=True` mimics the Zonos
gateway's placeholder header (RIFF + data sizes = 0xFFFFFFFF)."""
samples = b"\x11\x22" * nsamples
fmt = struct.pack("<HHIIHH", 1, 1, 44100, 88200, 2, 16)
body = b"WAVE" + b"fmt " + struct.pack("<I", 16) + fmt + b"data"
data_size = 0xFFFFFFFF if streaming else len(samples)
riff_size = 0xFFFFFFFF if streaming else (len(body) + 4 + len(samples))
return b"RIFF" + struct.pack("<I", riff_size) + body + struct.pack("<I", data_size) + samples
# A proper finite WAV — the header-finalize step is idempotent on it (sizes already real).
_WAV = _wav()
async def _drain(gen) -> bytes:
out = b""
async for chunk in gen:
out += chunk
return out
class TestPadState:
@@ -64,8 +59,6 @@ class TestPadState:
assert PadState.from_obj({"pleasure": "hot", "arousal": 0.1}) is None
def test_from_obj_huge_int_overflow_is_none(self) -> None:
# float() of a 400-digit JSON integer raises OverflowError (an ArithmeticError,
# not a ValueError); from_obj must still degrade to None, not propagate a 500.
huge = int("9" * 400)
assert PadState.from_obj({"pleasure": huge, "arousal": 0}) is None
@@ -74,7 +67,7 @@ class TestPadToDials:
def test_none_pad_is_neutral_disabled(self) -> None:
d = pad_to_dials(None)
assert d.emotion_enabled is False
assert d.to_body() == {} # a neutral read sends no emotion params
assert d.to_body() == {}
def test_maps_pleasure_and_arousal(self) -> None:
d = pad_to_dials(PadState(pleasure=0.4, arousal=0.6))
@@ -88,19 +81,14 @@ class TestPadToDials:
assert d.emotion_arousal == -1.0
def test_nan_degrades_to_zero_never_raises(self) -> None:
# INV FN pad_to_dials: total over ANY input (incl. NaN) → valid dials.
d = pad_to_dials(PadState(pleasure=math.nan, arousal=math.inf))
assert d.emotion_valence == 0.0
assert d.emotion_arousal == 1.0 # +inf clamps to the ceiling
assert not math.isnan(d.emotion_valence)
assert d.emotion_arousal == 1.0
def test_non_padstate_input_degrades_to_neutral(self) -> None:
# The declared surface is PadState | None, but the function must be total over
# ANY object (a raw dict / str / arbitrary object) → neutral, never raise.
for bad in ({}, "bad", [0.1, 0.2], object(), 42):
d = pad_to_dials(bad)
assert d.emotion_enabled is False
assert d.to_body() == {}
assert d.emotion_enabled is False and d.to_body() == {}
class TestEmotionDialsToBody:
@@ -117,118 +105,51 @@ class TestEmotionDialsToBody:
assert "emotion_strength" in body
class TestTtsSynthesize:
class TestGatewayBody:
def test_always_wav_with_dials(self) -> None:
b = gateway_body("hi", "donut", pad_to_dials(PadState(pleasure=0.5, arousal=0.2)))
assert b["input"] == "hi" and b["voice"] == "donut"
assert b["response_format"] == "wav" # DEC-3 — ALWAYS wav
assert b["emotion_valence"] == pytest.approx(0.5)
def test_neutral_omits_emotion(self) -> None:
b = gateway_body("hi", "Cora", pad_to_dials(None))
assert "emotion_valence" not in b and b["response_format"] == "wav"
class TestTtsStream:
@respx.mock
async def test_happy_posts_wav_request_and_returns_bytes(self) -> None:
route = respx.post("http://tts.example/v1/audio/speech").mock(
return_value=httpx.Response(200, content=_WAV)
)
async def test_streams_chunks_and_posts_wav_body(self) -> None:
route = respx.post(_URL).mock(return_value=httpx.Response(200, content=_WAV))
async with httpx.AsyncClient() as client:
out = await tts_synthesize(
"hello there",
voice="donut",
out = await _drain(tts_stream(
"hello there", voice="donut",
dials=pad_to_dials(PadState(pleasure=0.5, arousal=0.2)),
client=client,
url="http://tts.example/v1/audio/speech",
)
assert out == _WAV
sent = route.calls.last.request
import json as _json
body = _json.loads(sent.content)
assert body["input"] == "hello there"
assert body["voice"] == "donut"
assert body["response_format"] == "wav" # DEC-3 — ALWAYS wav
assert body["emotion_valence"] == pytest.approx(0.5)
@respx.mock
async def test_neutral_dials_send_no_emotion_params(self) -> None:
route = respx.post("http://tts.example/v1/audio/speech").mock(
return_value=httpx.Response(200, content=_WAV)
)
async with httpx.AsyncClient() as client:
await tts_synthesize(
"hi", voice="Cora", dials=pad_to_dials(None), client=client,
url="http://tts.example/v1/audio/speech",
)
client=client, url=_URL,
))
assert out == _WAV # passed through verbatim — no header rewrite
import json as _json
body = _json.loads(route.calls.last.request.content)
assert "emotion_valence" not in body
assert body["voice"] == "donut"
assert body["response_format"] == "wav"
assert body["emotion_valence"] == pytest.approx(0.5)
@respx.mock
async def test_non_200_raises_tts_unavailable(self) -> None:
respx.post("http://tts.example/v1/audio/speech").mock(
return_value=httpx.Response(500, content=b"boom")
)
async def test_non_200_open_raises_before_any_chunk(self) -> None:
respx.post(_URL).mock(return_value=httpx.Response(500, content=b"boom"))
async with httpx.AsyncClient() as client:
with pytest.raises(TtsUnavailable) as exc:
await tts_synthesize(
"hi", voice="Cora", dials=pad_to_dials(None), client=client,
url="http://tts.example/v1/audio/speech",
)
await _drain(tts_stream(
"hi", voice="Cora", dials=pad_to_dials(None), client=client, url=_URL
))
assert exc.value.status == 500
@respx.mock
async def test_transport_error_raises_tts_unavailable(self) -> None:
respx.post("http://tts.example/v1/audio/speech").mock(
side_effect=httpx.ConnectError("refused")
)
async def test_transport_error_raises(self) -> None:
respx.post(_URL).mock(side_effect=httpx.ConnectError("refused"))
async with httpx.AsyncClient() as client:
with pytest.raises(TtsUnavailable):
await tts_synthesize(
"hi", voice="Cora", dials=pad_to_dials(None), client=client,
url="http://tts.example/v1/audio/speech",
)
@respx.mock
async def test_non_wav_body_raises_tts_unavailable(self) -> None:
# DEC-3 guard: a 200 with a non-RIFF body (an error page, or the
# mislabeled-PCM mp3/opus trap) is unavailable, not played.
respx.post("http://tts.example/v1/audio/speech").mock(
return_value=httpx.Response(200, content=b"<html>error</html>")
)
async with httpx.AsyncClient() as client:
with pytest.raises(TtsUnavailable):
await tts_synthesize(
"hi", voice="Cora", dials=pad_to_dials(None), client=client,
url="http://tts.example/v1/audio/speech",
)
@respx.mock
async def test_finalizes_streaming_wav_header_to_real_sizes(self) -> None:
# The Zonos gateway returns a streaming header (RIFF + data sizes = 0xFFFFFFFF);
# the browser <audio> element can't play that. tts_synthesize must rewrite both
# to the real byte counts so the result is a decodable finite WAV.
streaming = _wav(streaming=True)
assert struct.unpack("<I", streaming[4:8])[0] == 0xFFFFFFFF # precondition
respx.post("http://tts.example/v1/audio/speech").mock(
return_value=httpx.Response(200, content=streaming)
)
async with httpx.AsyncClient() as client:
out = await tts_synthesize(
"hi", voice="Cora", dials=pad_to_dials(None), client=client,
url="http://tts.example/v1/audio/speech",
)
# RIFF size + data size are now real, not 0xFFFFFFFF.
assert struct.unpack("<I", out[4:8])[0] == len(out) - 8
dpos = out.find(b"data", 12)
assert struct.unpack("<I", out[dpos + 4:dpos + 8])[0] == len(out) - (dpos + 8)
# …and Python's wave module (a real decoder) can now open it as a finite clip.
with wave.open(io.BytesIO(out)) as w:
assert w.getnchannels() == 1 and w.getsampwidth() == 2 and w.getframerate() == 44100
@respx.mock
async def test_riff_but_not_wave_raises_tts_unavailable(self) -> None:
# A body with the RIFF magic but a non-WAVE form tag (e.g. RIFF/AVI, or a
# truncated/garbage container) is not playable audio — reject it too.
respx.post("http://tts.example/v1/audio/speech").mock(
return_value=httpx.Response(200, content=b"RIFF\x00\x00\x00\x00AVI LIST")
)
async with httpx.AsyncClient() as client:
with pytest.raises(TtsUnavailable):
await tts_synthesize(
"hi", voice="Cora", dials=pad_to_dials(None), client=client,
url="http://tts.example/v1/audio/speech",
)
await _drain(tts_stream(
"hi", voice="Cora", dials=pad_to_dials(None), client=client, url=_URL
))
+30 -68
View File
@@ -7,7 +7,6 @@ AsyncClient.
"""
import json
import struct
import httpx
import pytest
@@ -1459,68 +1458,55 @@ class TestMemoryChunksEndpoint:
class TestTtsEndpoint:
"""tts_endpoint FN — POST /api/tts → audio/wav via the Zonos gateway proxy
(slice 2, docs/contracts/donut_voiced_interview.contract.md). The gateway host
never reaches the browser (INV-TTS-1/DEC-4); voice resolves per-character
(DEC-8); emotion dials map the browser-sent live PAD (DEC-7); a gateway failure
degrades to 503 (INV-TTS-4)."""
"""tts_endpoint FN — GET /api/tts → audio/wav STREAMED (chunked) from the Zonos
gateway. Voice per-character (DEC-8), emotion dials from p/a query floats (DEC-7),
the gateway host never reaches the browser (INV-TTS-1), gateway open-failure → 503
(INV-TTS-4). GET so a browser <audio src> plays it progressively."""
# A proper finite PCM WAV (correct RIFF + data sizes) so the header-finalize step is
# idempotent — the endpoint returns these bytes unchanged.
_WAV = (
b"RIFF" + (36 + 16).to_bytes(4, "little") + b"WAVE"
+ b"fmt " + (16).to_bytes(4, "little")
+ struct.pack("<HHIIHH", 1, 1, 44100, 88200, 2, 16)
+ b"data" + (16).to_bytes(4, "little") + b"\x11\x22" * 8
)
# Streaming WAV bytes (placeholder 0xFFFFFFFF sizes) — proxied through verbatim.
_WAV = b"RIFF\xff\xff\xff\xffWAVEdata\xff\xff\xff\xff" + b"\x11\x22" * 64
_TTS = "http://tts.example/v1/audio/speech"
@respx.mock
def test_happy_returns_wav_and_resolves_donut_voice_and_pad(self) -> None:
def test_happy_streams_wav_resolves_donut_voice_and_pad(self) -> None:
from ratatoskr.web.server import create_app
route = respx.post(self._TTS).mock(
return_value=httpx.Response(200, content=self._WAV)
)
app = create_app(_mock_client_factory(), tts_url=self._TTS)
resp = TestClient(app).post(
resp = TestClient(app).get(
"/api/tts",
json={
"text": "Carl looks intimidating, but he's a softie.",
"agent_id": "ratatoskr:donut",
"pad": {"pleasure": 0.6, "arousal": 0.3, "dominance": 0.1},
},
params={"text": "Carl is a softie.", "agent_id": "ratatoskr:donut",
"p": "0.6", "a": "0.3"},
)
assert resp.status_code == 200
assert resp.headers["content-type"] == "audio/wav"
assert resp.content == self._WAV
# DEC-8 voice map + DEC-7 affect dials rode the gateway request.
assert resp.headers["content-type"].startswith("audio/wav")
assert resp.content == self._WAV # streamed through verbatim, no header rewrite
body = json.loads(route.calls.last.request.content)
assert body["voice"] == "donut"
assert body["response_format"] == "wav"
assert body["emotion_valence"] == pytest.approx(0.6)
@respx.mock
def test_unmapped_agent_uses_default_voice_and_no_pad_no_emotion(self) -> None:
def test_unmapped_agent_default_voice_no_pad_no_emotion(self) -> None:
from ratatoskr.web.server import create_app
route = respx.post(self._TTS).mock(
return_value=httpx.Response(200, content=self._WAV)
)
app = create_app(_mock_client_factory(), tts_url=self._TTS)
resp = TestClient(app).post(
"/api/tts", json={"text": "hello", "agent_id": "mimir"}
)
resp = TestClient(app).get("/api/tts", params={"text": "hello", "agent_id": "mimir"})
assert resp.status_code == 200
body = json.loads(route.calls.last.request.content)
assert body["voice"] == "Cora" # gateway default (DEC-8)
assert "emotion_valence" not in body # no PAD → neutral read (DEC-7)
assert body["voice"] == "Cora" # gateway default (DEC-8)
assert "emotion_valence" not in body # no p/a → neutral read (DEC-7)
def test_missing_text_returns_400(self) -> None:
from ratatoskr.web.server import create_app
app = create_app(_mock_client_factory(), tts_url=self._TTS)
resp = TestClient(app).post("/api/tts", json={"agent_id": "ratatoskr:donut"})
resp = TestClient(app).get("/api/tts", params={"agent_id": "ratatoskr:donut"})
assert resp.status_code == 400
assert resp.json()["error_code"] == "missing_text"
@@ -1530,56 +1516,32 @@ class TestTtsEndpoint:
respx.post(self._TTS).mock(return_value=httpx.Response(500, content=b"boom"))
app = create_app(_mock_client_factory(), tts_url=self._TTS)
resp = TestClient(app).post("/api/tts", json={"text": "hi"})
resp = TestClient(app).get("/api/tts", params={"text": "hi"})
assert resp.status_code == 503
assert resp.json()["error_code"] == "tts_unavailable"
@respx.mock
def test_non_wav_body_degrades_to_503(self) -> None:
from ratatoskr.web.server import create_app
def test_long_text_truncated_at_cap(self) -> None:
from ratatoskr.web.server import _TTS_MAX_TEXT_CHARS, create_app
respx.post(self._TTS).mock(
return_value=httpx.Response(200, content=b"<html>nope</html>")
route = respx.post(self._TTS).mock(
return_value=httpx.Response(200, content=self._WAV)
)
app = create_app(_mock_client_factory(), tts_url=self._TTS)
resp = TestClient(app).post("/api/tts", json={"text": "hi"})
assert resp.status_code == 503
resp = TestClient(app).get("/api/tts", params={"text": "word " * 1000})
assert resp.status_code == 200
body = json.loads(route.calls.last.request.content)
assert len(body["input"]) <= _TTS_MAX_TEXT_CHARS # truncated for the URL / lock
@respx.mock
def test_unhashable_agent_id_does_not_500(self) -> None:
# An unhashable agent_id ([] / {}) must not TypeError the voice-map lookup into
# a 500 — it coerces to None (default voice), matching the submit path's guard.
def test_malformed_pad_query_degrades_no_500(self) -> None:
from ratatoskr.web.server import create_app
route = respx.post(self._TTS).mock(
return_value=httpx.Response(200, content=self._WAV)
)
app = create_app(_mock_client_factory(), tts_url=self._TTS)
resp = TestClient(app).post("/api/tts", json={"text": "hi", "agent_id": []})
resp = TestClient(app).get("/api/tts", params={"text": "hi", "p": "notafloat", "a": "0.1"})
assert resp.status_code == 200
assert json.loads(route.calls.last.request.content)["voice"] == "Cora"
def test_text_too_large_returns_413(self) -> None:
from ratatoskr.web.server import _TTS_MAX_TEXT_CHARS, create_app
app = create_app(_mock_client_factory(), tts_url=self._TTS)
resp = TestClient(app).post(
"/api/tts", json={"text": "x" * (_TTS_MAX_TEXT_CHARS + 1)}
)
assert resp.status_code == 413
assert resp.json()["error_code"] == "text_too_large"
@respx.mock
def test_huge_int_pad_does_not_500(self) -> None:
# A malformed pad (huge-int → float() OverflowError in from_obj) must degrade to
# a neutral read, not 500 the endpoint.
from ratatoskr.web.server import create_app
respx.post(self._TTS).mock(return_value=httpx.Response(200, content=self._WAV))
app = create_app(_mock_client_factory(), tts_url=self._TTS)
resp = TestClient(app).post(
"/api/tts",
json={"text": "hi", "pad": {"pleasure": int("9" * 400), "arousal": 0}},
)
assert resp.status_code == 200
body = json.loads(route.calls.last.request.content)
assert "emotion_valence" not in body # malformed p → neutral read, not a 500