feat: stream Donut TTS play-as-it-arrives + autoplay unlock (supersedes buffered)
Operator: play-as-it-arrives, don't wait for the whole clip. infra-ops confirmed the
Zonos gateway ALREADY streams (chunked int16 WAV, TTFB ~0.44s vs ~7s total; placeholder
0xFFFFFFFF sizes are DESIGNED for progressive <audio src>). The buffering was entirely
in our proxy, and the _finalize_wav_header rewrite (6c3c08b) FORCED it — computing the
real sizes needs the whole clip.
The fix — pipe the chunks straight through:
- tts.py: buffered tts_synthesize + _finalize_wav_header REMOVED; tts_stream (an async
generator over the gateway's chunked response) + gateway_body added. Never buffer,
never rewrite the placeholder header.
- server.py: /api/tts is now GET (so a browser <audio src> plays it progressively) →
a chunked StreamingResponse piping the gateway; peeks the first chunk so a bad gateway
OPEN still returns 503; the serialize lock is held across the stream and released on
completion/abort; PAD rides p/a query floats.
- index.html: speakOnDone sets <audio src="/api/tts?..."> (streaming) instead of
fetch->blob; dropped the blob machinery. AUTOPLAY UNLOCK: _unlockTtsAudio() plays a
silent WAV within the toggle/submit gesture so the delayed play() isn't blocked — the
actual cause of "no audio" (play() fires ~15s after the keypress, past the browser's
transient-activation window).
Live-verified: GET /api/tts is transfer-encoding: chunked, TTFB 0.46s. Playwright with
--autoplay-policy=document-user-activation-required: the streaming <audio src> plays
progressively (currentTime advances, no decode error, no MSE fallback needed) 6.5s after
the gesture — proving the unlock's persistent element flag. 521 green.
DEC-2 amended (streaming supersedes "no streaming"); FN tts_stream / tts_endpoint updated.
This commit is contained in:
@@ -62,12 +62,18 @@ each independently shippable. Slice order is chosen for fastest visible result.
|
||||
swappable `ext-tts` LiteLLM alias. Rationale: the emotion dials (the whole
|
||||
point — affect-driven voice) don't pass through `ext-tts`. Accept the
|
||||
Zonos coupling; the `tts.py` client is the single swap seam if we ever move.
|
||||
- **DEC-2 — full-synth latency accepted (no true streaming).** The gateway
|
||||
buffers to a complete clip (~1.8s/sentence, scales). "Speak on done" gives the
|
||||
whole clip after a short delay. True first-audio-early is a future gateway
|
||||
enhancement (infra-ops to expose the native PCM stream); not in v1.
|
||||
- **DEC-3 — wav only.** `response_format:"wav"` (16-bit RIFF). `mp3`/`opus` are
|
||||
accepted but silently return mislabeled PCM — never request them.
|
||||
- **DEC-2 — STREAMING, play-as-it-arrives (amended 2026-08-02, operator-directed).**
|
||||
The gateway ALREADY streams: `POST /v1/audio/speech` relays a chunked int16 WAV
|
||||
(transfer-encoding: chunked, placeholder 0xFFFFFFFF RIFF/data sizes) as it synthesizes
|
||||
— TTFB ~0.44s vs ~7s total (infra-ops verified). So ratatoskr PROXIES THE CHUNKS
|
||||
STRAIGHT THROUGH (`tts_stream`, `GET /api/tts`) and the browser plays a progressive
|
||||
`<audio src>`; NEVER buffer, NEVER rewrite the placeholder header (a rewrite needs the
|
||||
whole clip and defeats streaming — the bug the original buffered `tts_synthesize` +
|
||||
`_finalize_wav_header` hit). The placeholder-size WAV is DESIGNED for `<audio src>`
|
||||
progressive playback (validated in Chromium: plays, currentTime advances, no MSE
|
||||
needed). Supersedes the original "full-synth latency accepted / no streaming."
|
||||
- **DEC-3 — wav only.** `response_format:"wav"` (streaming int16 RIFF/WAVE). `mp3`/`opus`
|
||||
are accepted but silently return mislabeled PCM — never request them.
|
||||
- **DEC-4 — server-side proxy.** Browser → `/api/tts` (nh3-dev) → gateway. The
|
||||
irv-ml1 host/URL never reaches the client (INV-TTS-1). No key exists, so
|
||||
INV-003 is trivially satisfied, but the proxy still stands (browser can't
|
||||
@@ -125,19 +131,17 @@ each independently shippable. Slice order is chosen for fastest visible result.
|
||||
|
||||
## FN blocks
|
||||
|
||||
### FN tts_synthesize
|
||||
### FN tts_stream (replaces the buffered tts_synthesize — DEC-2 streaming)
|
||||
```
|
||||
tts_synthesize(text: str, *, voice: str, dials: EmotionDials, client: httpx.AsyncClient, url=ZONOS_TTS_URL) -> bytes
|
||||
# POST {input:text, voice, response_format:"wav", **dials} to the Zonos gateway; return wav bytes.
|
||||
# `url` (added — heid-code-review F3) is the swap seam (DEC-1): the endpoint passes app.state.tts_url;
|
||||
# tests pass a respx-mocked URL. Defaults to ZONOS_TTS_URL so the parameter is inert for the common call.
|
||||
precondition: text non-empty. Voice membership in /v1/voices is GATEWAY-enforced, not client-asserted
|
||||
(an unknown voice surfaces as a gateway non-200 -> TtsUnavailable) — the contract + bundle
|
||||
carry no local voice catalog, so a client-side check would fork a source of truth.
|
||||
postcondition: returns a RIFF/WAVE container (magic 0:4 == "RIFF" AND form 8:12 == "WAVE"). The DEC-3
|
||||
guard is operational ("is this wav, not HTML / mislabeled-PCM"), NOT a full 16-bit-PCM
|
||||
fmt-chunk parse (heid-code-review: "16-bit" is aspirational; the guard is container-level).
|
||||
error: gateway non-200 / transport failure / non-WAVE body -> TtsUnavailable (caller degrades per INV-TTS-4).
|
||||
tts_stream(text, *, voice, dials, client: httpx.AsyncClient, url=ZONOS_TTS_URL) -> AsyncIterator[bytes]
|
||||
# Open the gateway's CHUNKED stream (client.stream("POST", url, json=gateway_body(...))) and YIELD wav
|
||||
# chunks as they synthesize. Pass through verbatim — never buffer, never rewrite the placeholder header.
|
||||
# gateway_body(text, voice, dials) = {input, voice, response_format:"wav", **dials.to_body()}.
|
||||
precondition: text non-empty. Voice membership in /v1/voices is GATEWAY-enforced, not client-asserted.
|
||||
postcondition: yields the gateway's chunked int16 streaming WAV bytes unmodified (0xFFFFFFFF placeholder
|
||||
sizes intact — the browser <audio src> plays them progressively).
|
||||
error: a non-200 OPEN or connect/transport failure -> TtsUnavailable BEFORE the first chunk (so the
|
||||
endpoint can still return 503); a mid-stream drop just ends the generator.
|
||||
invariant: response_format is ALWAYS "wav" (DEC-3); never mp3/opus.
|
||||
```
|
||||
|
||||
@@ -152,17 +156,19 @@ pad_to_dials(pad: PadState | None) -> EmotionDials
|
||||
invariant: total over any PAD input (finite/None/out-of-range) -> valid dials, never raises.
|
||||
```
|
||||
|
||||
### FN tts_endpoint (server.py, /api/tts)
|
||||
### FN tts_endpoint (server.py, GET /api/tts)
|
||||
```
|
||||
POST /api/tts {text, agent_id?, pad?} -> audio/wav
|
||||
GET /api/tts?text=&agent_id=&p=&a= -> audio/wav (chunked StreamingResponse)
|
||||
# GET (not POST) so a browser <audio src> plays it progressively (DEC-2 streaming). Params ride the query
|
||||
# string; text is capped ~2000 chars and truncated at a word boundary (URL-safe + bounds the shared-3090 hold).
|
||||
steps:
|
||||
- resolve voice (per-character map -> "donut"; default Cora; a non-str agent_id coerces to None).
|
||||
- dials = pad_to_dials(PadState.from_obj(pad)) — the PAD is BROWSER-SENT in the request body (per DEC-7:
|
||||
the console already holds live PAD from the affect_update SSE), NOT a server-side PAD lookup. (Clarified
|
||||
per heid-code-review F5 — the original "the agent's current PAD if known" wording read as a server fetch.)
|
||||
- reject text over the max-char budget with 413 (bug-hunt: bound before the lock); missing/non-str text -> 400.
|
||||
- tts_synthesize(...) behind the serialize guard (DEC-5); return wav with Content-Type audio/wav.
|
||||
- on TtsUnavailable -> 503 controlled envelope (client skips playback, INV-TTS-4).
|
||||
- missing text -> 400. resolve voice (per-character map -> "donut"; default Cora).
|
||||
- dials = pad_to_dials(PadState(p, a)) from the p/a query floats (DEC-7, BROWSER-SENT live PAD); malformed
|
||||
p/a -> neutral read, never a 500.
|
||||
- acquire the serialize lock (DEC-5, one stream at a time on the shared 3090); open tts_stream and PEEK the
|
||||
first chunk so a bad gateway OPEN surfaces as 503 (INV-TTS-4) before committing a 200.
|
||||
- return StreamingResponse piping the chunks; the generator's finally releases the lock + closes the client
|
||||
(incl. the browser-abort path: a new turn's <audio> load() drops the GET).
|
||||
```
|
||||
|
||||
### FN pin_kb_context (kb_bridge.py — RETIRE-READY, INV-KB-1)
|
||||
@@ -195,14 +201,19 @@ pin_kb_context(question: str, agent_id: str | None, *, client) -> list[dict] #
|
||||
searches in-voice natively.
|
||||
```
|
||||
|
||||
### FN client: speakOnDone (index.html)
|
||||
### FN client: speakOnDone (index.html — STREAMING, DEC-2)
|
||||
```
|
||||
on SSE `done`:
|
||||
if !ttsEnabled(): return # INV-TTS-2
|
||||
cancelInFlight() # INV-TTS-3
|
||||
const wav = await fetch('/api/tts', {text: assistantText, agent_id}) # non-stream; whole clip
|
||||
if !ok: return # INV-TTS-4 (silent skip)
|
||||
play(wav) in the <audio> sink; a new turn start -> cancelInFlight() + audio.pause()
|
||||
cancelTts() # INV-TTS-3: audio.pause()+removeAttribute(src)+load()
|
||||
audio.src = "/api/tts?text=&agent_id=&p=&a=" # GET streaming URL (text sliced to the 2000 cap)
|
||||
audio.play() -> ▶ voiced ; .catch -> "playback blocked" # progressive; failure non-fatal (INV-TTS-4)
|
||||
|
||||
AUTOPLAY UNLOCK (the load-bearing fix for "no audio"): speak fires play() in an async callback seconds after
|
||||
the keypress, past the browser's transient-activation window, so a bare play() is blocked. _unlockTtsAudio()
|
||||
plays a tiny silent WAV inside a REAL gesture (toggle-on + each prompt submit), which grants the <audio>
|
||||
element a PERSISTENT "may-play" flag so the later streaming play() isn't refused. Validated in Chromium with
|
||||
--autoplay-policy=document-user-activation-required: play succeeds 6.5s after the gesture, currentTime advances.
|
||||
```
|
||||
|
||||
## Slice plan
|
||||
|
||||
+26
-46
@@ -14,8 +14,7 @@ Foot-guns (verified live 2026-08-02):
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import struct
|
||||
from collections.abc import Mapping
|
||||
from collections.abc import AsyncIterator, Mapping
|
||||
from dataclasses import dataclass
|
||||
|
||||
import httpx
|
||||
@@ -98,27 +97,6 @@ class EmotionDials:
|
||||
}
|
||||
|
||||
|
||||
def _finalize_wav_header(data: bytes) -> bytes:
|
||||
"""Rewrite the RIFF + data chunk sizes with the real byte counts.
|
||||
|
||||
The Zonos gateway emits a STREAMING wav header — the RIFF chunk size (offset 4)
|
||||
and the data chunk size are both 0xFFFFFFFF ("unknown length"), because it can
|
||||
stream. A browser <audio> element playing a fully-downloaded blob needs a finite,
|
||||
correctly-sized WAV to decode it; a 0xFFFFFFFF length reads as raw/streaming PCM
|
||||
and won't play (operator-reported). Now that the whole clip is buffered we know the
|
||||
real sizes, so patch them in. Idempotent — writing an already-correct size is a
|
||||
no-op; safe if the data chunk isn't found (leaves those bytes untouched).
|
||||
"""
|
||||
if len(data) < 44: # shorter than a canonical PCM header — nothing to patch
|
||||
return data
|
||||
buf = bytearray(data)
|
||||
struct.pack_into("<I", buf, 4, len(buf) - 8) # RIFF chunk size = file len - 8
|
||||
data_pos = buf.find(b"data", 12) # first "data" after the fmt chunk = the data chunk
|
||||
if data_pos != -1 and data_pos + 8 <= len(buf):
|
||||
struct.pack_into("<I", buf, data_pos + 4, len(buf) - (data_pos + 8))
|
||||
return bytes(buf)
|
||||
|
||||
|
||||
def _clamp(x: float, lo: float, hi: float) -> float:
|
||||
"""Clamp to [lo, hi], NaN-safe: a NaN axis (a dead/malformed signal) → 0.0
|
||||
rather than propagating through max/min into the gateway."""
|
||||
@@ -144,37 +122,39 @@ def pad_to_dials(pad: PadState | None) -> EmotionDials:
|
||||
)
|
||||
|
||||
|
||||
async def tts_synthesize(
|
||||
def gateway_body(text: str, voice: str, dials: EmotionDials) -> dict:
|
||||
"""The Zonos gateway POST body — response_format is ALWAYS "wav" (DEC-3)."""
|
||||
return {"input": text, "voice": voice, "response_format": "wav", **dials.to_body()}
|
||||
|
||||
|
||||
async def tts_stream(
|
||||
text: str,
|
||||
*,
|
||||
voice: str,
|
||||
dials: EmotionDials,
|
||||
client: httpx.AsyncClient,
|
||||
url: str = ZONOS_TTS_URL,
|
||||
) -> bytes:
|
||||
"""POST {input, voice, response_format:"wav", **dials} to the Zonos gateway;
|
||||
return 16-bit RIFF/WAVE bytes (FN tts_synthesize).
|
||||
) -> AsyncIterator[bytes]:
|
||||
"""Open the gateway's CHUNKED stream and yield WAV bytes as they synthesize.
|
||||
|
||||
response_format is ALWAYS "wav" (DEC-3). Any non-200, transport failure, or
|
||||
non-wav body → TtsUnavailable — the caller degrades (INV-TTS-4).
|
||||
The gateway emits a streaming int16 WAV (RIFF/data sizes = 0xFFFFFFFF placeholders)
|
||||
over `transfer-encoding: chunked`, TTFB ~0.44s vs ~7s total (infra-ops verified) —
|
||||
designed to be played progressively by a browser <audio>. So we PROXY THE CHUNKS
|
||||
STRAIGHT THROUGH: never buffer, never rewrite the header (that would force us to wait
|
||||
for the whole clip and defeat the streaming — the bug this replaces).
|
||||
|
||||
Raises TtsUnavailable if the gateway open is a non-200 or the connect/transport
|
||||
fails — BEFORE the first chunk, so the endpoint can still return a 503 (INV-TTS-4).
|
||||
A drop mid-stream just ends the generator (the browser has already played the head).
|
||||
"""
|
||||
assert text, "tts_synthesize: text must be non-empty (the endpoint guards this)"
|
||||
body = {"input": text, "voice": voice, "response_format": "wav", **dials.to_body()}
|
||||
assert text, "tts_stream: text must be non-empty (the endpoint guards this)"
|
||||
try:
|
||||
resp = await client.post(url, json=body)
|
||||
async with client.stream("POST", url, json=gateway_body(text, voice, dials)) as resp:
|
||||
if resp.status_code != 200:
|
||||
raise TtsUnavailable(
|
||||
f"gateway status {resp.status_code}", status=resp.status_code
|
||||
)
|
||||
async for chunk in resp.aiter_bytes():
|
||||
yield chunk
|
||||
except httpx.RequestError as exc:
|
||||
raise TtsUnavailable(f"gateway transport failure: {exc}") from exc
|
||||
if resp.status_code != 200:
|
||||
raise TtsUnavailable(
|
||||
f"gateway status {resp.status_code}", status=resp.status_code
|
||||
)
|
||||
data = resp.content
|
||||
# DEC-3 guard: the gateway MUST return a RIFF/WAVE container. A non-wav 200 (an
|
||||
# error page, or the mislabeled-PCM mp3/opus trap) is treated as unavailable —
|
||||
# played garbage is worse than silence. Check BOTH the RIFF magic (0:4) and the
|
||||
# WAVE form tag (8:12); a `b"RIFF..."` body that isn't WAVE would still fail decode.
|
||||
if data[:4] != b"RIFF" or data[8:12] != b"WAVE":
|
||||
raise TtsUnavailable("gateway returned a non-wav body")
|
||||
# The gateway's streaming header carries 0xFFFFFFFF sizes; rewrite them with the
|
||||
# real byte counts so a browser <audio> element can decode the finite clip.
|
||||
return _finalize_wav_header(data)
|
||||
|
||||
+66
-39
@@ -69,7 +69,7 @@ from ratatoskr.tts import (
|
||||
PadState,
|
||||
TtsUnavailable,
|
||||
pad_to_dials,
|
||||
tts_synthesize,
|
||||
tts_stream,
|
||||
)
|
||||
|
||||
|
||||
@@ -542,55 +542,82 @@ async def _memory_chunks_endpoint(request: Request) -> JSONResponse:
|
||||
# Zonos voice; everything else falls to the gateway default. Case-folded gateway-side.
|
||||
_TTS_VOICE_MAP = {"ratatoskr:donut": "donut"}
|
||||
_TTS_DEFAULT_VOICE = "Cora"
|
||||
# Cap the synth input before taking the process-global lock. An interview clip is a
|
||||
# few sentences; a huge/hostile body would otherwise hold the shared 3090 for the full
|
||||
# gateway timeout, starving every other turn's audio (heid bug-hunt: text-size DoS).
|
||||
_TTS_MAX_TEXT_CHARS = 8000
|
||||
# The streamed text rides the GET query string, so keep it URL-safe-short — a few
|
||||
# sentences is plenty for a voiced turn, and a very long response is truncated at a word
|
||||
# boundary (the full text still shows in the transcript). Also bounds the shared-3090 hold.
|
||||
_TTS_MAX_TEXT_CHARS = 2000
|
||||
|
||||
|
||||
def _truncate_at_boundary(text: str, limit: int) -> str:
|
||||
"""Trim to <= limit chars, preferring the last space so we don't cut mid-word."""
|
||||
if len(text) <= limit:
|
||||
return text
|
||||
head = text[:limit]
|
||||
cut = head.rfind(" ")
|
||||
return head[:cut] if cut > limit // 2 else head
|
||||
|
||||
|
||||
async def _tts_endpoint(request: Request) -> Response:
|
||||
"""POST /api/tts {text, agent_id?, pad?} → audio/wav (FN tts_endpoint, slice 2).
|
||||
"""GET /api/tts?text=&agent_id=&p=&a= → audio/wav, STREAMED chunked from the Zonos
|
||||
gateway (FN tts_endpoint). GET so a browser <audio src> plays it progressively; the
|
||||
gateway already streams (TTFB ~0.44s vs ~7s total), so we pipe the chunks straight
|
||||
through — never buffer, never rewrite the placeholder-size WAV header (that would
|
||||
force buffering the whole clip and defeat the streaming).
|
||||
|
||||
Server-side proxy to the Zonos gateway (DEC-4 / INV-TTS-1: the gateway host
|
||||
never reaches the browser). Voice resolves per-character (DEC-8); emotion dials
|
||||
map the browser-sent live PAD (DEC-7 — the affect the persona pane already shows).
|
||||
Serialized one-synth-at-a-time (DEC-5 / INV-TTS-3 — the gateway shares one 3090).
|
||||
A gateway failure / non-wav body degrades to 503 (INV-TTS-4: the client skips
|
||||
playback; the turn/transcript is unaffected)."""
|
||||
try:
|
||||
body = await request.json()
|
||||
except (json.JSONDecodeError, ValueError, TypeError):
|
||||
body = None
|
||||
text = body.get("text") if isinstance(body, dict) else None
|
||||
if not text or not isinstance(text, str):
|
||||
Server-side proxy (DEC-4 / INV-TTS-1: the gateway host never reaches the browser).
|
||||
Voice per-character (DEC-8); emotion dials from the browser-sent live PAD (DEC-7,
|
||||
passed as p/a query floats). Serialized one-stream-at-a-time (DEC-5); a new turn
|
||||
aborts the prior <audio> load → the GET drops → the generator's finally releases the
|
||||
lock. A gateway open-failure → 503 (INV-TTS-4: the client skips playback)."""
|
||||
q = request.query_params
|
||||
text = q.get("text")
|
||||
if not text:
|
||||
return JSONResponse({"error_code": "missing_text"}, status_code=400)
|
||||
if len(text) > _TTS_MAX_TEXT_CHARS: # bound before the lock (bug-hunt: text-size DoS)
|
||||
return JSONResponse({"error_code": "text_too_large"}, status_code=413)
|
||||
agent_id = body.get("agent_id") if isinstance(body, dict) else None
|
||||
if not isinstance(agent_id, str):
|
||||
# Coerce a non-str (incl. an unhashable list/dict) to None BEFORE the map lookup:
|
||||
# `_TTS_VOICE_MAP.get([])` would TypeError → uncaught 500, diverging from the
|
||||
# submit path's identical guard (heid bug-hunt: open-world agent_id).
|
||||
agent_id = None
|
||||
text = _truncate_at_boundary(text, _TTS_MAX_TEXT_CHARS)
|
||||
agent_id = q.get("agent_id")
|
||||
voice = _TTS_VOICE_MAP.get(agent_id, _TTS_DEFAULT_VOICE)
|
||||
pad_obj = body.get("pad") if isinstance(body, dict) else None
|
||||
dials = pad_to_dials(PadState.from_obj(pad_obj))
|
||||
pad: PadState | None = None
|
||||
try:
|
||||
if q.get("p") is not None and q.get("a") is not None:
|
||||
pad = PadState(pleasure=float(q["p"]), arousal=float(q["a"]))
|
||||
except (TypeError, ValueError):
|
||||
pad = None # malformed p/a → neutral read, never a 500
|
||||
dials = pad_to_dials(pad)
|
||||
|
||||
tts_url = request.app.state.tts_url
|
||||
lock = request.app.state.tts_lock
|
||||
|
||||
# DEC-5: one stream at a time on the shared 3090. Held for the stream's duration and
|
||||
# released in the generator's finally — including the browser-abort/cancel path.
|
||||
await lock.acquire()
|
||||
client = httpx.AsyncClient(
|
||||
timeout=httpx.Timeout(connect=10.0, read=30.0, write=10.0, pool=10.0)
|
||||
)
|
||||
gen = tts_stream(text, voice=voice, dials=dials, client=client, url=tts_url)
|
||||
try:
|
||||
# DEC-5: serialize — a new turn's synth waits on any in-flight one (the
|
||||
# client also aborts the prior request, cancelling the server task). 60s ceiling:
|
||||
# generous for a multi-sentence clip, bounded so a stalled gateway can't pin the
|
||||
# shared lock for two minutes (bug-hunt: lock-hold starvation).
|
||||
async with lock:
|
||||
async with httpx.AsyncClient(timeout=httpx.Timeout(60.0)) as client:
|
||||
wav = await tts_synthesize(
|
||||
text, voice=voice, dials=dials, client=client, url=tts_url
|
||||
)
|
||||
# Peek the first chunk so a bad gateway OPEN (non-200 / transport) surfaces as a
|
||||
# 503 BEFORE we commit a 200 StreamingResponse. TTFB ~0.44s, so this is cheap.
|
||||
first = await gen.__anext__()
|
||||
except TtsUnavailable:
|
||||
await gen.aclose()
|
||||
await client.aclose()
|
||||
lock.release()
|
||||
return JSONResponse({"error_code": "tts_unavailable"}, status_code=503)
|
||||
return Response(wav, media_type="audio/wav")
|
||||
except StopAsyncIteration:
|
||||
first = None
|
||||
|
||||
async def piped() -> AsyncIterator[bytes]:
|
||||
try:
|
||||
if first:
|
||||
yield first
|
||||
async for chunk in gen:
|
||||
yield chunk
|
||||
finally:
|
||||
await gen.aclose() # unwinds tts_stream's `async with` → closes the gateway resp
|
||||
await client.aclose()
|
||||
lock.release()
|
||||
|
||||
return StreamingResponse(piped(), media_type="audio/wav")
|
||||
|
||||
|
||||
async def _session_tools_endpoint(request: Request) -> JSONResponse:
|
||||
@@ -804,7 +831,7 @@ def create_app(
|
||||
Route("/api/sessions/{session_id}/messages", _session_messages_endpoint),
|
||||
Route("/api/sessions/{session_id}/bifrost", _session_bifrost_endpoint),
|
||||
Route("/api/admin/events", _admin_events_endpoint),
|
||||
Route("/api/tts", _tts_endpoint, methods=["POST"]),
|
||||
Route("/api/tts", _tts_endpoint, methods=["GET"]),
|
||||
Route("/api/turns/{session_id}", _submit_turn_endpoint, methods=["POST"]),
|
||||
Route("/api/turns/{session_id}/stream", _stream_turn_endpoint),
|
||||
Route("/api/turns/{session_id}/cancel", _cancel_turn_endpoint, methods=["POST"]),
|
||||
|
||||
@@ -1709,6 +1709,7 @@ async function submitPrompt() {
|
||||
const content = input.value.trim();
|
||||
if (!content || !state.sessionId || state.turnId) return;
|
||||
cancelTts(); // INV-TTS-3: a new turn cancels prior voice
|
||||
if (ttsEnabled()) _unlockTtsAudio(); // this keypress is a gesture — grant autoplay
|
||||
input.value = ""; input.style.height = "20px";
|
||||
|
||||
let r;
|
||||
@@ -1993,56 +1994,66 @@ async function cancelTurn() {
|
||||
});
|
||||
})();
|
||||
|
||||
// ---- auto-TTS (slice 2): voiced, affect-modulated playback on turn `done` -----
|
||||
// Server-proxied to the Zonos gateway (/api/tts). Opt-in (INV-TTS-2), one clip at a
|
||||
// time (INV-TTS-3: a new turn or a superseding synth cancels the prior fetch + audio),
|
||||
// and non-blocking (INV-TTS-4: any failure logs to the ticker + skips — never the turn).
|
||||
let _ttsAbort = null;
|
||||
let _ttsUrl = null; // active blob object URL, tracked so cancel can revoke it
|
||||
// ---- auto-TTS: voiced, affect-modulated STREAMING playback on turn `done` -----
|
||||
// The <audio> element streams the chunked GET /api/tts response and plays as it arrives
|
||||
// (TTFA ~0.5s), never waiting for the whole clip. Opt-in (INV-TTS-2), one stream at a
|
||||
// time (INV-TTS-3: a new turn / toggle-off aborts the prior <audio> load, dropping the
|
||||
// server stream), non-blocking (INV-TTS-4: any failure logs to the ticker, never the turn).
|
||||
function ttsEnabled() {
|
||||
try { return localStorage.getItem("ratatoskr-tts") === "1"; } catch (_) { return false; }
|
||||
}
|
||||
function _revokeTtsUrl() {
|
||||
if (_ttsUrl) { try { URL.revokeObjectURL(_ttsUrl); } catch (_) {} _ttsUrl = null; }
|
||||
// A short silent WAV (PCM/mono/44.1k/16-bit) used only to UNLOCK the <audio> element.
|
||||
// speak-on-done fires play() in an async callback seconds after the user's keypress, by
|
||||
// which point the browser's autoplay policy has revoked the activation and blocks it.
|
||||
// Playing this once inside a real gesture (toggle-on, each prompt submit) grants the
|
||||
// element a persistent "may play" flag so the later real playback isn't blocked.
|
||||
const _TTS_SILENT = (() => {
|
||||
const N = 256, data = 2 * N, buf = new Uint8Array(44 + data), dv = new DataView(buf.buffer);
|
||||
buf.set([0x52, 0x49, 0x46, 0x46], 0); dv.setUint32(4, 36 + data, true);
|
||||
buf.set([0x57, 0x41, 0x56, 0x45], 8); buf.set([0x66, 0x6d, 0x74, 0x20], 12);
|
||||
dv.setUint32(16, 16, true); dv.setUint16(20, 1, true); dv.setUint16(22, 1, true);
|
||||
dv.setUint32(24, 44100, true); dv.setUint32(28, 88200, true);
|
||||
dv.setUint16(32, 2, true); dv.setUint16(34, 16, true);
|
||||
buf.set([0x64, 0x61, 0x74, 0x61], 36); dv.setUint32(40, data, true);
|
||||
let s = ""; for (const x of buf) s += String.fromCharCode(x);
|
||||
return "data:audio/wav;base64," + btoa(s);
|
||||
})();
|
||||
let _ttsUnlocked = false;
|
||||
function _unlockTtsAudio() {
|
||||
if (_ttsUnlocked) return; // once per page is enough
|
||||
const a = $("tts-audio"); if (!a) return;
|
||||
try {
|
||||
a.src = _TTS_SILENT;
|
||||
const p = a.play();
|
||||
if (p && p.then) p.then(() => {
|
||||
_ttsUnlocked = true;
|
||||
try { a.pause(); a.removeAttribute("src"); a.load(); } catch (_) {}
|
||||
}).catch(() => {});
|
||||
} catch (_) {}
|
||||
}
|
||||
function cancelTts() {
|
||||
if (_ttsAbort) { try { _ttsAbort.abort(); } catch (_) {} _ttsAbort = null; }
|
||||
// Stop + drop the current <audio> src; load() aborts the in-flight GET stream, which
|
||||
// drops the server proxy (releasing its synth lock). No object URLs to revoke — the
|
||||
// src is a streaming /api/tts URL, not a blob.
|
||||
const a = $("tts-audio");
|
||||
if (a) { try { a.pause(); a.removeAttribute("src"); a.load(); } catch (_) {} }
|
||||
// removeAttribute("src")+load() fires NEITHER ended nor error, so the src's own
|
||||
// revoke handler never runs — revoke the tracked URL here or the blob leaks per
|
||||
// interrupted turn (heid bug-hunt).
|
||||
_revokeTtsUrl();
|
||||
}
|
||||
async function speakOnDone(text, agentId, pad) {
|
||||
function speakOnDone(text, agentId, pad) {
|
||||
const clip = (text || "").trim();
|
||||
if (!clip) return;
|
||||
cancelTts(); // INV-TTS-3: stop any prior synth/playback
|
||||
const ctrl = new AbortController();
|
||||
_ttsAbort = ctrl;
|
||||
let resp;
|
||||
try {
|
||||
resp = await fetch("/api/tts", {
|
||||
method: "POST",
|
||||
headers: { "Content-Type": "application/json" },
|
||||
body: JSON.stringify({ text: clip, agent_id: agentId || null, pad: pad || null }),
|
||||
signal: ctrl.signal,
|
||||
});
|
||||
} catch (_) { return; } // aborted / network → silent skip (INV-TTS-4)
|
||||
if (!resp.ok) { tickerAdd("err", "tts", "unavailable " + resp.status); return; }
|
||||
let url;
|
||||
try { url = URL.createObjectURL(await resp.blob()); } catch (_) { return; }
|
||||
if (_ttsAbort !== ctrl) { URL.revokeObjectURL(url); return; } // superseded mid-fetch
|
||||
const a = $("tts-audio");
|
||||
if (!a) { URL.revokeObjectURL(url); return; }
|
||||
a.src = url;
|
||||
_ttsUrl = url; // track for revoke on natural end OR cancel
|
||||
a.onended = a.onerror = () => { _revokeTtsUrl(); };
|
||||
// Playback can be refused by the browser autoplay policy until the page has an
|
||||
// activation; the operator's toggle+prompt gesture generally satisfies it, and the
|
||||
// catch keeps a refusal non-fatal (INV-TTS-4).
|
||||
try { await a.play(); tickerAdd("ok", "tts", "▶ voiced"); }
|
||||
catch (_) { _revokeTtsUrl(); tickerAdd("err", "tts", "playback blocked"); }
|
||||
cancelTts(); // INV-TTS-3: stop any prior stream
|
||||
const a = $("tts-audio"); if (!a) return;
|
||||
const params = new URLSearchParams({ text: clip.slice(0, 2000) }); // matches the server cap
|
||||
if (agentId) params.set("agent_id", agentId);
|
||||
if (pad && typeof pad.pleasure === "number" && typeof pad.arousal === "number") {
|
||||
params.set("p", pad.pleasure); params.set("a", pad.arousal); // affect dials (DEC-7)
|
||||
}
|
||||
// Progressive playback: point <audio> at the chunked stream; it plays as bytes arrive.
|
||||
a.src = "/api/tts?" + params.toString();
|
||||
// play() may be refused by the autoplay policy in this async callback; the prompt/toggle
|
||||
// gesture unlock (_unlockTtsAudio) satisfies it, and the catch keeps a refusal non-fatal.
|
||||
a.play().then(() => tickerAdd("ok", "tts", "▶ voiced"))
|
||||
.catch(() => tickerAdd("err", "tts", "playback blocked"));
|
||||
}
|
||||
// ---- 🔊 toggle (mirrors theme / cot-toggle; default OFF, persisted) ----
|
||||
(function () {
|
||||
@@ -2057,7 +2068,8 @@ async function speakOnDone(text, agentId, pad) {
|
||||
btn.classList.toggle("on", next);
|
||||
btn.title = label(next);
|
||||
try { localStorage.setItem("ratatoskr-tts", next ? "1" : "0"); } catch (_) {}
|
||||
if (!next) cancelTts(); // turning off stops in-flight playback
|
||||
if (next) _unlockTtsAudio(); // grant autoplay within this gesture
|
||||
else cancelTts(); // turning off stops in-flight playback
|
||||
});
|
||||
})();
|
||||
|
||||
|
||||
+54
-133
@@ -1,17 +1,11 @@
|
||||
"""Tests for ratatoskr.tts per docs/contracts/donut_voiced_interview.contract.md
|
||||
(slice 2 — auto-TTS). Two units:
|
||||
"""Tests for ratatoskr.tts — the STREAMING Zonos-gateway client + PAD→dial mapping.
|
||||
|
||||
- pad_to_dials / PadState / EmotionDials — pure, total, degrade-not-crash.
|
||||
- tts_synthesize — the Zonos-gateway client (respx-mocked; no live network).
|
||||
|
||||
DEC-3: response_format is ALWAYS "wav". DEC-7: live PAD → emotion dials.
|
||||
INV-TTS-4: a gateway failure / non-wav body → TtsUnavailable (caller degrades).
|
||||
tts_stream proxies the gateway's chunked response verbatim (no buffering, no header
|
||||
rewrite — the placeholder-size streaming WAV is meant to be played progressively).
|
||||
pad_to_dials / PadState / EmotionDials are pure + total.
|
||||
"""
|
||||
|
||||
import io
|
||||
import math
|
||||
import struct
|
||||
import wave
|
||||
|
||||
import httpx
|
||||
import pytest
|
||||
@@ -21,24 +15,25 @@ from ratatoskr.tts import (
|
||||
EmotionDials,
|
||||
PadState,
|
||||
TtsUnavailable,
|
||||
gateway_body,
|
||||
pad_to_dials,
|
||||
tts_synthesize,
|
||||
tts_stream,
|
||||
)
|
||||
|
||||
_URL = "http://tts.example/v1/audio/speech"
|
||||
# The gateway's streaming WAV bytes (placeholder 0xFFFFFFFF sizes). We pass them through
|
||||
# untouched, so the content only has to round-trip.
|
||||
_WAV = (
|
||||
b"RIFF\xff\xff\xff\xffWAVEfmt \x10\x00\x00\x00" + b"\x00" * 20
|
||||
+ b"data\xff\xff\xff\xff" + b"\x11\x22" * 64
|
||||
)
|
||||
|
||||
|
||||
def _wav(*, streaming: bool = False, nsamples: int = 8) -> bytes:
|
||||
"""A canonical PCM mono/44.1k/16-bit WAV. `streaming=True` mimics the Zonos
|
||||
gateway's placeholder header (RIFF + data sizes = 0xFFFFFFFF)."""
|
||||
samples = b"\x11\x22" * nsamples
|
||||
fmt = struct.pack("<HHIIHH", 1, 1, 44100, 88200, 2, 16)
|
||||
body = b"WAVE" + b"fmt " + struct.pack("<I", 16) + fmt + b"data"
|
||||
data_size = 0xFFFFFFFF if streaming else len(samples)
|
||||
riff_size = 0xFFFFFFFF if streaming else (len(body) + 4 + len(samples))
|
||||
return b"RIFF" + struct.pack("<I", riff_size) + body + struct.pack("<I", data_size) + samples
|
||||
|
||||
|
||||
# A proper finite WAV — the header-finalize step is idempotent on it (sizes already real).
|
||||
_WAV = _wav()
|
||||
async def _drain(gen) -> bytes:
|
||||
out = b""
|
||||
async for chunk in gen:
|
||||
out += chunk
|
||||
return out
|
||||
|
||||
|
||||
class TestPadState:
|
||||
@@ -64,8 +59,6 @@ class TestPadState:
|
||||
assert PadState.from_obj({"pleasure": "hot", "arousal": 0.1}) is None
|
||||
|
||||
def test_from_obj_huge_int_overflow_is_none(self) -> None:
|
||||
# float() of a 400-digit JSON integer raises OverflowError (an ArithmeticError,
|
||||
# not a ValueError); from_obj must still degrade to None, not propagate a 500.
|
||||
huge = int("9" * 400)
|
||||
assert PadState.from_obj({"pleasure": huge, "arousal": 0}) is None
|
||||
|
||||
@@ -74,7 +67,7 @@ class TestPadToDials:
|
||||
def test_none_pad_is_neutral_disabled(self) -> None:
|
||||
d = pad_to_dials(None)
|
||||
assert d.emotion_enabled is False
|
||||
assert d.to_body() == {} # a neutral read sends no emotion params
|
||||
assert d.to_body() == {}
|
||||
|
||||
def test_maps_pleasure_and_arousal(self) -> None:
|
||||
d = pad_to_dials(PadState(pleasure=0.4, arousal=0.6))
|
||||
@@ -88,19 +81,14 @@ class TestPadToDials:
|
||||
assert d.emotion_arousal == -1.0
|
||||
|
||||
def test_nan_degrades_to_zero_never_raises(self) -> None:
|
||||
# INV FN pad_to_dials: total over ANY input (incl. NaN) → valid dials.
|
||||
d = pad_to_dials(PadState(pleasure=math.nan, arousal=math.inf))
|
||||
assert d.emotion_valence == 0.0
|
||||
assert d.emotion_arousal == 1.0 # +inf clamps to the ceiling
|
||||
assert not math.isnan(d.emotion_valence)
|
||||
assert d.emotion_arousal == 1.0
|
||||
|
||||
def test_non_padstate_input_degrades_to_neutral(self) -> None:
|
||||
# The declared surface is PadState | None, but the function must be total over
|
||||
# ANY object (a raw dict / str / arbitrary object) → neutral, never raise.
|
||||
for bad in ({}, "bad", [0.1, 0.2], object(), 42):
|
||||
d = pad_to_dials(bad)
|
||||
assert d.emotion_enabled is False
|
||||
assert d.to_body() == {}
|
||||
assert d.emotion_enabled is False and d.to_body() == {}
|
||||
|
||||
|
||||
class TestEmotionDialsToBody:
|
||||
@@ -117,118 +105,51 @@ class TestEmotionDialsToBody:
|
||||
assert "emotion_strength" in body
|
||||
|
||||
|
||||
class TestTtsSynthesize:
|
||||
class TestGatewayBody:
|
||||
def test_always_wav_with_dials(self) -> None:
|
||||
b = gateway_body("hi", "donut", pad_to_dials(PadState(pleasure=0.5, arousal=0.2)))
|
||||
assert b["input"] == "hi" and b["voice"] == "donut"
|
||||
assert b["response_format"] == "wav" # DEC-3 — ALWAYS wav
|
||||
assert b["emotion_valence"] == pytest.approx(0.5)
|
||||
|
||||
def test_neutral_omits_emotion(self) -> None:
|
||||
b = gateway_body("hi", "Cora", pad_to_dials(None))
|
||||
assert "emotion_valence" not in b and b["response_format"] == "wav"
|
||||
|
||||
|
||||
class TestTtsStream:
|
||||
@respx.mock
|
||||
async def test_happy_posts_wav_request_and_returns_bytes(self) -> None:
|
||||
route = respx.post("http://tts.example/v1/audio/speech").mock(
|
||||
return_value=httpx.Response(200, content=_WAV)
|
||||
)
|
||||
async def test_streams_chunks_and_posts_wav_body(self) -> None:
|
||||
route = respx.post(_URL).mock(return_value=httpx.Response(200, content=_WAV))
|
||||
async with httpx.AsyncClient() as client:
|
||||
out = await tts_synthesize(
|
||||
"hello there",
|
||||
voice="donut",
|
||||
out = await _drain(tts_stream(
|
||||
"hello there", voice="donut",
|
||||
dials=pad_to_dials(PadState(pleasure=0.5, arousal=0.2)),
|
||||
client=client,
|
||||
url="http://tts.example/v1/audio/speech",
|
||||
)
|
||||
assert out == _WAV
|
||||
sent = route.calls.last.request
|
||||
import json as _json
|
||||
|
||||
body = _json.loads(sent.content)
|
||||
assert body["input"] == "hello there"
|
||||
assert body["voice"] == "donut"
|
||||
assert body["response_format"] == "wav" # DEC-3 — ALWAYS wav
|
||||
assert body["emotion_valence"] == pytest.approx(0.5)
|
||||
|
||||
@respx.mock
|
||||
async def test_neutral_dials_send_no_emotion_params(self) -> None:
|
||||
route = respx.post("http://tts.example/v1/audio/speech").mock(
|
||||
return_value=httpx.Response(200, content=_WAV)
|
||||
)
|
||||
async with httpx.AsyncClient() as client:
|
||||
await tts_synthesize(
|
||||
"hi", voice="Cora", dials=pad_to_dials(None), client=client,
|
||||
url="http://tts.example/v1/audio/speech",
|
||||
)
|
||||
client=client, url=_URL,
|
||||
))
|
||||
assert out == _WAV # passed through verbatim — no header rewrite
|
||||
import json as _json
|
||||
|
||||
body = _json.loads(route.calls.last.request.content)
|
||||
assert "emotion_valence" not in body
|
||||
assert body["voice"] == "donut"
|
||||
assert body["response_format"] == "wav"
|
||||
assert body["emotion_valence"] == pytest.approx(0.5)
|
||||
|
||||
@respx.mock
|
||||
async def test_non_200_raises_tts_unavailable(self) -> None:
|
||||
respx.post("http://tts.example/v1/audio/speech").mock(
|
||||
return_value=httpx.Response(500, content=b"boom")
|
||||
)
|
||||
async def test_non_200_open_raises_before_any_chunk(self) -> None:
|
||||
respx.post(_URL).mock(return_value=httpx.Response(500, content=b"boom"))
|
||||
async with httpx.AsyncClient() as client:
|
||||
with pytest.raises(TtsUnavailable) as exc:
|
||||
await tts_synthesize(
|
||||
"hi", voice="Cora", dials=pad_to_dials(None), client=client,
|
||||
url="http://tts.example/v1/audio/speech",
|
||||
)
|
||||
await _drain(tts_stream(
|
||||
"hi", voice="Cora", dials=pad_to_dials(None), client=client, url=_URL
|
||||
))
|
||||
assert exc.value.status == 500
|
||||
|
||||
@respx.mock
|
||||
async def test_transport_error_raises_tts_unavailable(self) -> None:
|
||||
respx.post("http://tts.example/v1/audio/speech").mock(
|
||||
side_effect=httpx.ConnectError("refused")
|
||||
)
|
||||
async def test_transport_error_raises(self) -> None:
|
||||
respx.post(_URL).mock(side_effect=httpx.ConnectError("refused"))
|
||||
async with httpx.AsyncClient() as client:
|
||||
with pytest.raises(TtsUnavailable):
|
||||
await tts_synthesize(
|
||||
"hi", voice="Cora", dials=pad_to_dials(None), client=client,
|
||||
url="http://tts.example/v1/audio/speech",
|
||||
)
|
||||
|
||||
@respx.mock
|
||||
async def test_non_wav_body_raises_tts_unavailable(self) -> None:
|
||||
# DEC-3 guard: a 200 with a non-RIFF body (an error page, or the
|
||||
# mislabeled-PCM mp3/opus trap) is unavailable, not played.
|
||||
respx.post("http://tts.example/v1/audio/speech").mock(
|
||||
return_value=httpx.Response(200, content=b"<html>error</html>")
|
||||
)
|
||||
async with httpx.AsyncClient() as client:
|
||||
with pytest.raises(TtsUnavailable):
|
||||
await tts_synthesize(
|
||||
"hi", voice="Cora", dials=pad_to_dials(None), client=client,
|
||||
url="http://tts.example/v1/audio/speech",
|
||||
)
|
||||
|
||||
@respx.mock
|
||||
async def test_finalizes_streaming_wav_header_to_real_sizes(self) -> None:
|
||||
# The Zonos gateway returns a streaming header (RIFF + data sizes = 0xFFFFFFFF);
|
||||
# the browser <audio> element can't play that. tts_synthesize must rewrite both
|
||||
# to the real byte counts so the result is a decodable finite WAV.
|
||||
streaming = _wav(streaming=True)
|
||||
assert struct.unpack("<I", streaming[4:8])[0] == 0xFFFFFFFF # precondition
|
||||
respx.post("http://tts.example/v1/audio/speech").mock(
|
||||
return_value=httpx.Response(200, content=streaming)
|
||||
)
|
||||
async with httpx.AsyncClient() as client:
|
||||
out = await tts_synthesize(
|
||||
"hi", voice="Cora", dials=pad_to_dials(None), client=client,
|
||||
url="http://tts.example/v1/audio/speech",
|
||||
)
|
||||
# RIFF size + data size are now real, not 0xFFFFFFFF.
|
||||
assert struct.unpack("<I", out[4:8])[0] == len(out) - 8
|
||||
dpos = out.find(b"data", 12)
|
||||
assert struct.unpack("<I", out[dpos + 4:dpos + 8])[0] == len(out) - (dpos + 8)
|
||||
# …and Python's wave module (a real decoder) can now open it as a finite clip.
|
||||
with wave.open(io.BytesIO(out)) as w:
|
||||
assert w.getnchannels() == 1 and w.getsampwidth() == 2 and w.getframerate() == 44100
|
||||
|
||||
@respx.mock
|
||||
async def test_riff_but_not_wave_raises_tts_unavailable(self) -> None:
|
||||
# A body with the RIFF magic but a non-WAVE form tag (e.g. RIFF/AVI, or a
|
||||
# truncated/garbage container) is not playable audio — reject it too.
|
||||
respx.post("http://tts.example/v1/audio/speech").mock(
|
||||
return_value=httpx.Response(200, content=b"RIFF\x00\x00\x00\x00AVI LIST")
|
||||
)
|
||||
async with httpx.AsyncClient() as client:
|
||||
with pytest.raises(TtsUnavailable):
|
||||
await tts_synthesize(
|
||||
"hi", voice="Cora", dials=pad_to_dials(None), client=client,
|
||||
url="http://tts.example/v1/audio/speech",
|
||||
)
|
||||
await _drain(tts_stream(
|
||||
"hi", voice="Cora", dials=pad_to_dials(None), client=client, url=_URL
|
||||
))
|
||||
|
||||
+30
-68
@@ -7,7 +7,6 @@ AsyncClient.
|
||||
"""
|
||||
|
||||
import json
|
||||
import struct
|
||||
|
||||
import httpx
|
||||
import pytest
|
||||
@@ -1459,68 +1458,55 @@ class TestMemoryChunksEndpoint:
|
||||
|
||||
|
||||
class TestTtsEndpoint:
|
||||
"""tts_endpoint FN — POST /api/tts → audio/wav via the Zonos gateway proxy
|
||||
(slice 2, docs/contracts/donut_voiced_interview.contract.md). The gateway host
|
||||
never reaches the browser (INV-TTS-1/DEC-4); voice resolves per-character
|
||||
(DEC-8); emotion dials map the browser-sent live PAD (DEC-7); a gateway failure
|
||||
degrades to 503 (INV-TTS-4)."""
|
||||
"""tts_endpoint FN — GET /api/tts → audio/wav STREAMED (chunked) from the Zonos
|
||||
gateway. Voice per-character (DEC-8), emotion dials from p/a query floats (DEC-7),
|
||||
the gateway host never reaches the browser (INV-TTS-1), gateway open-failure → 503
|
||||
(INV-TTS-4). GET so a browser <audio src> plays it progressively."""
|
||||
|
||||
# A proper finite PCM WAV (correct RIFF + data sizes) so the header-finalize step is
|
||||
# idempotent — the endpoint returns these bytes unchanged.
|
||||
_WAV = (
|
||||
b"RIFF" + (36 + 16).to_bytes(4, "little") + b"WAVE"
|
||||
+ b"fmt " + (16).to_bytes(4, "little")
|
||||
+ struct.pack("<HHIIHH", 1, 1, 44100, 88200, 2, 16)
|
||||
+ b"data" + (16).to_bytes(4, "little") + b"\x11\x22" * 8
|
||||
)
|
||||
# Streaming WAV bytes (placeholder 0xFFFFFFFF sizes) — proxied through verbatim.
|
||||
_WAV = b"RIFF\xff\xff\xff\xffWAVEdata\xff\xff\xff\xff" + b"\x11\x22" * 64
|
||||
_TTS = "http://tts.example/v1/audio/speech"
|
||||
|
||||
@respx.mock
|
||||
def test_happy_returns_wav_and_resolves_donut_voice_and_pad(self) -> None:
|
||||
def test_happy_streams_wav_resolves_donut_voice_and_pad(self) -> None:
|
||||
from ratatoskr.web.server import create_app
|
||||
|
||||
route = respx.post(self._TTS).mock(
|
||||
return_value=httpx.Response(200, content=self._WAV)
|
||||
)
|
||||
app = create_app(_mock_client_factory(), tts_url=self._TTS)
|
||||
resp = TestClient(app).post(
|
||||
resp = TestClient(app).get(
|
||||
"/api/tts",
|
||||
json={
|
||||
"text": "Carl looks intimidating, but he's a softie.",
|
||||
"agent_id": "ratatoskr:donut",
|
||||
"pad": {"pleasure": 0.6, "arousal": 0.3, "dominance": 0.1},
|
||||
},
|
||||
params={"text": "Carl is a softie.", "agent_id": "ratatoskr:donut",
|
||||
"p": "0.6", "a": "0.3"},
|
||||
)
|
||||
assert resp.status_code == 200
|
||||
assert resp.headers["content-type"] == "audio/wav"
|
||||
assert resp.content == self._WAV
|
||||
# DEC-8 voice map + DEC-7 affect dials rode the gateway request.
|
||||
assert resp.headers["content-type"].startswith("audio/wav")
|
||||
assert resp.content == self._WAV # streamed through verbatim, no header rewrite
|
||||
body = json.loads(route.calls.last.request.content)
|
||||
assert body["voice"] == "donut"
|
||||
assert body["response_format"] == "wav"
|
||||
assert body["emotion_valence"] == pytest.approx(0.6)
|
||||
|
||||
@respx.mock
|
||||
def test_unmapped_agent_uses_default_voice_and_no_pad_no_emotion(self) -> None:
|
||||
def test_unmapped_agent_default_voice_no_pad_no_emotion(self) -> None:
|
||||
from ratatoskr.web.server import create_app
|
||||
|
||||
route = respx.post(self._TTS).mock(
|
||||
return_value=httpx.Response(200, content=self._WAV)
|
||||
)
|
||||
app = create_app(_mock_client_factory(), tts_url=self._TTS)
|
||||
resp = TestClient(app).post(
|
||||
"/api/tts", json={"text": "hello", "agent_id": "mimir"}
|
||||
)
|
||||
resp = TestClient(app).get("/api/tts", params={"text": "hello", "agent_id": "mimir"})
|
||||
assert resp.status_code == 200
|
||||
body = json.loads(route.calls.last.request.content)
|
||||
assert body["voice"] == "Cora" # gateway default (DEC-8)
|
||||
assert "emotion_valence" not in body # no PAD → neutral read (DEC-7)
|
||||
assert body["voice"] == "Cora" # gateway default (DEC-8)
|
||||
assert "emotion_valence" not in body # no p/a → neutral read (DEC-7)
|
||||
|
||||
def test_missing_text_returns_400(self) -> None:
|
||||
from ratatoskr.web.server import create_app
|
||||
|
||||
app = create_app(_mock_client_factory(), tts_url=self._TTS)
|
||||
resp = TestClient(app).post("/api/tts", json={"agent_id": "ratatoskr:donut"})
|
||||
resp = TestClient(app).get("/api/tts", params={"agent_id": "ratatoskr:donut"})
|
||||
assert resp.status_code == 400
|
||||
assert resp.json()["error_code"] == "missing_text"
|
||||
|
||||
@@ -1530,56 +1516,32 @@ class TestTtsEndpoint:
|
||||
|
||||
respx.post(self._TTS).mock(return_value=httpx.Response(500, content=b"boom"))
|
||||
app = create_app(_mock_client_factory(), tts_url=self._TTS)
|
||||
resp = TestClient(app).post("/api/tts", json={"text": "hi"})
|
||||
resp = TestClient(app).get("/api/tts", params={"text": "hi"})
|
||||
assert resp.status_code == 503
|
||||
assert resp.json()["error_code"] == "tts_unavailable"
|
||||
|
||||
@respx.mock
|
||||
def test_non_wav_body_degrades_to_503(self) -> None:
|
||||
from ratatoskr.web.server import create_app
|
||||
def test_long_text_truncated_at_cap(self) -> None:
|
||||
from ratatoskr.web.server import _TTS_MAX_TEXT_CHARS, create_app
|
||||
|
||||
respx.post(self._TTS).mock(
|
||||
return_value=httpx.Response(200, content=b"<html>nope</html>")
|
||||
route = respx.post(self._TTS).mock(
|
||||
return_value=httpx.Response(200, content=self._WAV)
|
||||
)
|
||||
app = create_app(_mock_client_factory(), tts_url=self._TTS)
|
||||
resp = TestClient(app).post("/api/tts", json={"text": "hi"})
|
||||
assert resp.status_code == 503
|
||||
resp = TestClient(app).get("/api/tts", params={"text": "word " * 1000})
|
||||
assert resp.status_code == 200
|
||||
body = json.loads(route.calls.last.request.content)
|
||||
assert len(body["input"]) <= _TTS_MAX_TEXT_CHARS # truncated for the URL / lock
|
||||
|
||||
@respx.mock
|
||||
def test_unhashable_agent_id_does_not_500(self) -> None:
|
||||
# An unhashable agent_id ([] / {}) must not TypeError the voice-map lookup into
|
||||
# a 500 — it coerces to None (default voice), matching the submit path's guard.
|
||||
def test_malformed_pad_query_degrades_no_500(self) -> None:
|
||||
from ratatoskr.web.server import create_app
|
||||
|
||||
route = respx.post(self._TTS).mock(
|
||||
return_value=httpx.Response(200, content=self._WAV)
|
||||
)
|
||||
app = create_app(_mock_client_factory(), tts_url=self._TTS)
|
||||
resp = TestClient(app).post("/api/tts", json={"text": "hi", "agent_id": []})
|
||||
resp = TestClient(app).get("/api/tts", params={"text": "hi", "p": "notafloat", "a": "0.1"})
|
||||
assert resp.status_code == 200
|
||||
assert json.loads(route.calls.last.request.content)["voice"] == "Cora"
|
||||
|
||||
def test_text_too_large_returns_413(self) -> None:
|
||||
from ratatoskr.web.server import _TTS_MAX_TEXT_CHARS, create_app
|
||||
|
||||
app = create_app(_mock_client_factory(), tts_url=self._TTS)
|
||||
resp = TestClient(app).post(
|
||||
"/api/tts", json={"text": "x" * (_TTS_MAX_TEXT_CHARS + 1)}
|
||||
)
|
||||
assert resp.status_code == 413
|
||||
assert resp.json()["error_code"] == "text_too_large"
|
||||
|
||||
@respx.mock
|
||||
def test_huge_int_pad_does_not_500(self) -> None:
|
||||
# A malformed pad (huge-int → float() OverflowError in from_obj) must degrade to
|
||||
# a neutral read, not 500 the endpoint.
|
||||
from ratatoskr.web.server import create_app
|
||||
|
||||
respx.post(self._TTS).mock(return_value=httpx.Response(200, content=self._WAV))
|
||||
app = create_app(_mock_client_factory(), tts_url=self._TTS)
|
||||
resp = TestClient(app).post(
|
||||
"/api/tts",
|
||||
json={"text": "hi", "pad": {"pleasure": int("9" * 400), "arousal": 0}},
|
||||
)
|
||||
assert resp.status_code == 200
|
||||
|
||||
body = json.loads(route.calls.last.request.content)
|
||||
assert "emotion_valence" not in body # malformed p → neutral read, not a 500
|
||||
|
||||
Reference in New Issue
Block a user