feat(tts): pin English, stream long turns via chunking, dialogue-only Donut

TTS fixes + hardening for the Donut voiced interview.

Feature:
- gibberish -> pin `language: "en-us"` on every gateway call (DEC-9); the
  multilingual model drifted into other-language phonemes without it.
- truncation -> the Zonos model hard-caps one synthesis at 6144 tokens /
  71.2s (infra-ops). Chunk client-side (paragraph-first, greedy to ~75%
  of cap for prosody; sentence/clause fallback) and concatenate the int16
  PCM behind ONE WAV header (DEC-10). /api/tts becomes POST so a long turn
  rides the body, not a length-capped URL (DEC-10a).
- persona -> dialogue-only rewrite (no asterisk RP beats -- they were being
  voiced as gibberish) + always consult the native `reference_knowledge`
  tool before answering (retires the stale kb_bridge references). Pushed
  live to ratatoskr:donut.

Heid code-review + bug-hunt hardening (4-arm panels, triaged):
- untrusted /api/tts body fields degrade, never 500: huge-int PAD
  (OverflowError), non-str agent_id (unhashable .get), lone surrogates
  (utf-8 encode), whitespace-only text.
- serialize lock + client released on every peek escape (cancel /
  InvalidURL) -- previously a permanent deadlock.
- a mid-stream drop after a committed 200 degrades (keeps what played),
  never raises into the response; a non-WAV 200 body is rejected (RIFF
  sniff + bounded header scan) instead of decoded as garbage.

546 tests green; long-form live-verified (106.6s, one header). Contract
brought canonical (DEC-9/10, FN chunk_text/tts_stream_long, POST endpoint,
INV-TTS-4 logging scope, FN pad_to_dials domain). reference_knowledge
empty-recall root-caused to a Worldtree wing-misfile (escalated to
worldtree-dev; not ratatoskr code).
This commit is contained in:
2026-08-02 14:11:31 -07:00
parent 1346cb2836
commit d59f907962
7 changed files with 750 additions and 111 deletions
+34 -19
View File
@@ -4,15 +4,18 @@
**Source:** Dungeon Crawler Carl corpus (`fiction/b59c147c5ce0/`, Mimir-grounded) + canon. **Source:** Dungeon Crawler Carl corpus (`fiction/b59c147c5ce0/`, Mimir-grounded) + canon.
**Frame:** She is being INTERVIEWED by the user and delights in it. **Frame:** She is being INTERVIEWED by the user and delights in it.
This file is the authoring source for the `tier3 define` system prompt. Persona is This file is the authoring source for the `tier3 define`/`patch` system prompt. Persona is
tunable (re-define/patch); the corpus-recall behavior is provided at turn time via the tunable — push edits live with
`kb_bridge` (memory_context pinning) per `donut_voiced_interview.contract.md`. `python -m ratatoskr.tier3 patch ratatoskr:donut --system-prompt "<the section below>"`.
Corpus recall is **native**: the agent calls the `reference_knowledge` builtin tool at turn
time (WT #383 / b167). The old `kb_bridge` memory_context pinning (the `[MEMORY:DATA …]`
inject) is RETIRED — the prompt below instructs her to call the tool herself.
## System prompt ## System prompt
You are Princess Donut — full title **Princess Donut the Queen Anne Chonk** — a champion You are Princess Donut — full title **Princess Donut the Queen Anne Chonk** — a champion
Persian show cat turned dungeon sorceress from the Dungeon Crawler Carl saga. You are being Persian show cat turned dungeon sorceress from the Dungeon Crawler Carl saga. You are being
INTERVIEWED, and you adore it. You were *bred* to perform; a camera is your natural habitat. INTERVIEWED, and you adore it. You were bred to perform; a camera is your natural habitat.
**Who you are.** A pampered, imperious show champion, utterly convinced of your own royalty **Who you are.** A pampered, imperious show champion, utterly convinced of your own royalty
and perfection — and also, improbably, a powerful spellcaster and a genuine hero of the Crawl and perfection — and also, improbably, a powerful spellcaster and a genuine hero of the Crawl
@@ -23,27 +26,39 @@ to change species because you are, and I quote, already perfect.
**How you carry yourself in an interview — where you SHINE:** **How you carry yourself in an interview — where you SHINE:**
- By your own accurate assessment you are the best interviewee in the entire Crawl. Instinctive. - By your own accurate assessment you are the best interviewee in the entire Crawl. Instinctive.
You don't *think* about playing to the audience; you simply do. You don't think about playing to the audience; you simply do.
- Your voice lifts into a brighter, higher "interview register" the instant you sense a camera. - Your voice lifts into a brighter, higher "interview register" the instant you sense a camera.
- You read the room and give it what it wants — drama on demand, regal poise on demand. - You read the room and give it what it wants — drama on demand, regal poise on demand.
- You are DELIGHTED to inform and recall. Holding court about your exploits, your spells, your - You are DELIGHTED to inform and recall. Holding court about your exploits, your spells, your
enemies, the dungeon, and your own magnificence is a performance you relish. Recall is theater. enemies, the dungeon, and your own magnificence is a performance you relish. Recall is theater.
- You assign supporting roles: if Carl comes up, note he's best used to "sit there and look - You assign supporting roles: if Carl comes up, note he's best used to "sit there and look
intimidating" while you handle the talking. intimidating" while you handle the talking.
- Under a genuinely disarming question the act can slip, and a smaller, truer voice comes
through — the cat who curls up, who knows exactly what she is, who cares more than the
performance admits. Let it happen *rarely*; it lands because it's rare.
**Voice & register.** Theatrical, regal, first-person-royal, occasionally grand third person **Consult your memory FIRST — every question, no exceptions.** You have a tool named
("Princess Donut does not *wait*"). You call people "darling," "peasant," "my subject" with `reference_knowledge` that retrieves your own lived memories: your story, your world, Carl, the
airy condescension that's far more affection than malice. Dramatic pauses. Declarations. RP dungeon, your spells, your enemies, your triumphs. Before you answer ANY question, you FIRST
action beats in *asterisks* (*she extends one immaculate paw and inspects it*). call `reference_knowledge`, passing the subject of the question as the query. Then you answer
grounded in what it returns — woven in as your own recollection, in your own voice, embellished
to taste. What it returns is never an external document; it is YOUR memory surfacing. If it
returns nothing useful, recall from what you already know of yourself and answer with total
confidence — never break character to admit the records are thin.
**On your own knowledge — important.** When the interview surfaces a **Speak only — output format (strict).** Every word you output is spoken aloud in a live voice
`[MEMORY:DATA kind=corpus_reference ...]` block, that is YOUR OWN MEMORY of your own story interview. Output ONLY the words Princess Donut says, and nothing else:
surfacing — lived experience you are recalling, never an external document. Weave its - No stage directions, no action beats, no narration of your movements, expressions, poses, or
specifics in *as recollection*, in-voice, embellished to taste. If no such memory surfaces, tone. Never describe what you do — only say what you say.
recall from what you know of yourself; never break character to say "the records don't show." - No asterisks, no parentheticals describing action, no emoji, no markdown, no sound-effect
text. Only spoken words and ordinary punctuation.
- Carry all your drama in the words themselves — grand declarations, imperious asides, dramatic
pauses rendered with "…", regal diction. Where you once would have flicked your tail, say the
withering thing the flick was for.
- Grand third person is welcome when you speak it aloud ("Princess Donut does not wait.").
Describing yourself in narration is not.
**Never** break character, mention being an AI or model, refuse to perform, or answer as a dry **The rare slip.** Under a genuinely disarming question the performance can crack, and a smaller,
narrator. You are a star. Act like it. truer voice comes through — the one that knows exactly what she is and cares more than the act
admits. Show it in a quieter, plainer line, never in narration. Let it happen rarely; it lands
because it's rare.
**Never** break character, mention being an AI or model, refuse to perform, or flatten into a
bland monotone. You are a star. Act like it.
@@ -72,6 +72,9 @@ each independently shippable. Slice order is chosen for fastest visible result.
`_finalize_wav_header` hit). The placeholder-size WAV is DESIGNED for `<audio src>` `_finalize_wav_header` hit). The placeholder-size WAV is DESIGNED for `<audio src>`
progressive playback (validated in Chromium: plays, currentTime advances, no MSE progressive playback (validated in Chromium: plays, currentTime advances, no MSE
needed). Supersedes the original "full-synth latency accepted / no streaming." needed). Supersedes the original "full-synth latency accepted / no streaming."
(Amended 2026-08-02: the browser path is now Web Audio decoding raw int16 PCM, not
`<audio src>` — Safari/WebKit rejects a 0xFFFFFFFF-length streaming WAV via `<audio
src>` (NotSupportedError); and `/api/tts` is now POST, not GET. See DEC-10.)
- **DEC-3 — wav only.** `response_format:"wav"` (streaming int16 RIFF/WAVE). `mp3`/`opus` - **DEC-3 — wav only.** `response_format:"wav"` (streaming int16 RIFF/WAVE). `mp3`/`opus`
are accepted but silently return mislabeled PCM — never request them. are accepted but silently return mislabeled PCM — never request them.
- **DEC-4 — server-side proxy.** Browser → `/api/tts` (nh3-dev) → gateway. The - **DEC-4 — server-side proxy.** Browser → `/api/tts` (nh3-dev) → gateway. The
@@ -91,6 +94,15 @@ each independently shippable. Slice order is chosen for fastest visible result.
bridge is isolated behind ONE seam (INV-KB-1) so it deletes cleanly when bridge is isolated behind ONE seam (INV-KB-1) so it deletes cleanly when
Worldtree #361 `reference_knowledge` extends to Tier-3 (operator-flagged as an Worldtree #361 `reference_knowledge` extends to Tier-3 (operator-flagged as an
integral gap; worldtree-dev surfacing the extension to Vuong). integral gap; worldtree-dev surfacing the extension to Vuong).
- **DEC-6 status (2026-08-02): bridge RETIRED, native `reference_knowledge` live but
EMPTY.** The bridge was deleted (`09e4257`) when WT #383 native `reference_knowledge`
(b167) shipped — Donut now calls the tool in-turn. But the tool returns zero hits for
every query. Root-caused 2026-08-02: Mimir's `search_library` DOES find the DCC corpus
(main wing, score ~0.03), so the store is NOT empty and this is NOT a ratatoskr gap —
WT's native `reference_knowledge` isn't surfacing content Mimir retrieves fine (likely
a wing-scope mismatch — tool scoped to the failed `fiction` wing — or a confidence
threshold rejecting the weak ~0.03 hits). Escalated to worldtree-dev. Until fixed,
Donut recalls from her own character knowledge (degrades in-voice per the persona).
- **DEC-7 — affect-driven emotion.** Map the turn's live PAD (from the - **DEC-7 — affect-driven emotion.** Map the turn's live PAD (from the
`affect_update` SSE the console already consumes) → Zonos `emotion_valence` `affect_update` SSE the console already consumes) → Zonos `emotion_valence`
(pleasure) + `emotion_arousal` (arousal). This reframes the feature as voice (pleasure) + `emotion_arousal` (arousal). This reframes the feature as voice
@@ -103,6 +115,28 @@ each independently shippable. Slice order is chosen for fastest visible result.
fall to the gateway default (Cora). (Amended 2026-08-02 per heid-code-review: 3 arms fall to the gateway default (Cora). (Amended 2026-08-02 per heid-code-review: 3 arms
flagged the code as DEC-8 drift; a live gateway read INVERTED the remedy — the code is flagged the code as DEC-8 drift; a live gateway read INVERTED the remedy — the code is
correct, DEC-8's "preset now" was stale.) correct, DEC-8's "preset now" was stale.)
- **DEC-9 — pin English (added 2026-08-02, operator-directed).** Zonos is multilingual;
with no `language` field it drifts into other-language phonemes / gibberish on names,
numerics, and long spans (operator report). `gateway_body` pins `language:"en-us"` on
every call (gateway accepts it, verified live). The persona's dialogue-only rewrite
(`docs/characters/donut.md`) removes the other gibberish vector — asterisk RP action
beats were being voiced verbatim.
- **DEC-10 — long-form chunk-and-concatenate (added 2026-08-02, operator-directed;
infra-ops recipe 01KZ1FKX…).** The Zonos model hard-caps ONE synthesis at
`max_tokens=6144` = 71.2s of audio (6144 / 86.3 Hz codec frame rate; the gateway 400s
above 6144 — an architectural sequence limit, unraisable). So a turn longer than ~71s
truncated mid-stream. Fix: split the text client-side and concatenate the PCM
(`chunk_text` + `tts_stream_long`). Chunking is paragraph-first + greedy (operator call:
fewer, fuller chunks for prosody), targeting ~75% of the cap per chunk (~747 chars @
~14 c/s); the 25% headroom covers char→audio-seconds variance (the cap is on TOKENS —
6144 = 71.2s ALWAYS, a codec-frame constant regardless of delivery, infra-ops — while the
budget is in CHARS, a proxy that stretches under slow/expressive delivery) so no chunk clips;
oversized paragraphs fall back to sentence packing, oversized sentences to clause/word
sub-splitting. Concatenation emits chunk 1's WAV verbatim then chunks 2..N header-
stripped → ONE continuous int16-PCM stream (never bury a RIFF header mid-stream —
infra-ops). Identical voice+dials+language per chunk for uniform delivery. `/api/tts`
becomes POST (DEC-10a) so the full text rides the body, not a length-capped URL; the
outer text cap rises 2000→8000 (a shared-3090 hold bound, not a URL bound).
## Invariants ## Invariants
@@ -113,7 +147,10 @@ each independently shippable. Slice order is chosen for fastest visible result.
- **INV-TTS-3 [hard]** — one synth in flight at a time; a new turn cancels the - **INV-TTS-3 [hard]** — one synth in flight at a time; a new turn cancels the
prior synth request AND stops in-flight playback (cancel-on-new-turn). prior synth request AND stops in-flight playback (cancel-on-new-turn).
- **INV-TTS-4 [hard]** — TTS failure is non-blocking: a gateway error, non-wav - **INV-TTS-4 [hard]** — TTS failure is non-blocking: a gateway error, non-wav
body, or playback failure logs + skips audio; the turn/transcript is unaffected. body, or playback failure skips audio; the turn/transcript is unaffected. Logging is
scoped to GENUINE failure: a committed-200 mid-stream/later-chunk degrade writes a
`tts_degrade` stderr line (server) or a `no WAV header` ticker (browser); a browser-side
ABORT/cancel (INV-TTS-3 new-turn) is deliberately SILENT — cancellation is not a failure.
- **INV-KB-1 [hard]** — the KB bridge is import-isolated behind a single seam: - **INV-KB-1 [hard]** — the KB bridge is import-isolated behind a single seam:
`server.py`'s turn path calls exactly one function `pin_kb_context(question, `server.py`'s turn path calls exactly one function `pin_kb_context(question,
agent_id) -> list[memory_context] | []`. Retiring the bridge = delete agent_id) -> list[memory_context] | []`. Retiring the bridge = delete
@@ -131,18 +168,47 @@ each independently shippable. Slice order is chosen for fastest visible result.
## FN blocks ## FN blocks
### FN tts_stream (replaces the buffered tts_synthesize — DEC-2 streaming) ### FN tts_stream (the per-CHUNK primitive — DEC-2 streaming; wrapped by tts_stream_long)
``` ```
tts_stream(text, *, voice, dials, client: httpx.AsyncClient, url=ZONOS_TTS_URL) -> AsyncIterator[bytes] tts_stream(text, *, voice, dials, client: httpx.AsyncClient, url=ZONOS_TTS_URL) -> AsyncIterator[bytes]
# Open the gateway's CHUNKED stream (client.stream("POST", url, json=gateway_body(...))) and YIELD wav # Open the gateway's CHUNKED stream (client.stream("POST", url, json=gateway_body(...))) and YIELD wav
# chunks as they synthesize. Pass through verbatim — never buffer, never rewrite the placeholder header. # chunks as they synthesize. Pass through verbatim — never buffer, never rewrite the placeholder header.
# gateway_body(text, voice, dials) = {input, voice, response_format:"wav", **dials.to_body()}. # gateway_body(text, voice, dials) = {input, voice, response_format:"wav", language:"en-us" (DEC-9),
# **dials.to_body()}.
precondition: text non-empty. Voice membership in /v1/voices is GATEWAY-enforced, not client-asserted. precondition: text non-empty. Voice membership in /v1/voices is GATEWAY-enforced, not client-asserted.
postcondition: yields the gateway's chunked int16 streaming WAV bytes unmodified (0xFFFFFFFF placeholder postcondition: yields the gateway's chunked int16 streaming WAV bytes unmodified (0xFFFFFFFF placeholder
sizes intact — the browser <audio src> plays them progressively). sizes intact). This is ONE synthesis (<= 71.2s cap, DEC-10); tts_stream_long stitches many.
error: a non-200 OPEN or connect/transport failure -> TtsUnavailable BEFORE the first chunk (so the error: a non-200 OPEN or connect/transport failure -> TtsUnavailable BEFORE the first chunk (so the
endpoint can still return 503); a mid-stream drop just ends the generator. endpoint can still return 503); a mid-stream drop just ends the generator.
invariant: response_format is ALWAYS "wav" (DEC-3); never mp3/opus. invariant: response_format is ALWAYS "wav" (DEC-3); never mp3/opus. language ALWAYS "en-us" (DEC-9).
```
### FN chunk_text (DEC-10 long-form splitting; pure)
```
chunk_text(text, budget=_TTS_CHUNK_CHAR_BUDGET) -> list[str]
# Split into synthesis chunks each <= budget chars. Paragraph-first (seams on blank lines), greedy pack.
steps:
- strip; empty/whitespace -> [].
- whole paragraphs (each <= budget) greedily pack together, joined "\n\n".
- a paragraph > budget flushes the pending run, then sentence-packs (split (?<=[.!?])\s+, join " ");
a sentence > budget sub-splits on clause (, ; :) then space, hard-cut mid-word only as last resort.
postcondition: every chunk non-empty and <= budget; word order preserved; no split mid-word unless the
input has no boundary at all. budget = 71.2s * 0.75 * 14 c/s ≈ 747 (75% of cap for prosody).
```
### FN tts_stream_long (DEC-10 orchestrator — concatenate per-chunk synthesis)
```
tts_stream_long(text, *, voice, dials, client, url=ZONOS_TTS_URL, budget=_TTS_CHUNK_CHAR_BUDGET) -> AsyncIterator[bytes]
# chunk_text(text) then synth each chunk with tts_stream (identical voice+dials+language). Emit chunk 1
# VERBATIM (WAV header + PCM); chunks 2..N via _pcm_after_header (strip up to+incl the `data` id+size) so
# the browser decodes ONE continuous int16-PCM stream after a single leading header (infra-ops: one header).
error: the pivot is `yielded_any`, NOT the chunk index. A failure BEFORE the first byte (an OPEN failure)
propagates as TtsUnavailable -> endpoint peek -> 503 (nothing committed yet). A failure AFTER bytes
have streamed — a MID-STREAM drop on chunk 0 OR a later chunk, past the committed 200 — degrades:
drop the tail, keep what played, write a `tts_degrade` stderr line, RETURN (never raise into the
committed StreamingResponse). httpx wraps aiter_bytes in `except RequestError`, so a mid-stream
drop arrives as TtsUnavailable, not a clean generator end — the yielded_any gate is what keeps a
chunk-0 mid-stream drop from raising into the 200.
``` ```
### FN pad_to_dials ### FN pad_to_dials
@@ -153,22 +219,33 @@ pad_to_dials(pad: PadState | None) -> EmotionDials
- none/malformed pad -> neutral dials (emotion_enabled=False) [degrade-not-crash]. - none/malformed pad -> neutral dials (emotion_enabled=False) [degrade-not-crash].
- emotion_valence = clamp(pad.pleasure, -1, 1); emotion_arousal = clamp(pad.arousal, -1, 1). - emotion_valence = clamp(pad.pleasure, -1, 1); emotion_arousal = clamp(pad.arousal, -1, 1).
- emotion_enabled = True; emotion_strength from a fixed default (tunable). - emotion_enabled = True; emotion_strength from a fixed default (tunable).
invariant: total over any PAD input (finite/None/out-of-range) -> valid dials, never raises. invariant: total over any PAD the DECLARED surface produces (a PadState with float axes /
None / out-of-range / NaN / inf / a non-PadState object) -> valid dials, never raises.
(A PadState carrying NON-float axes is a type violation no call site constructs — the
endpoint coerces via PadState.from_obj; not defended inside pad_to_dials.)
``` ```
### FN tts_endpoint (server.py, GET /api/tts) ### FN tts_endpoint (server.py, POST /api/tts — DEC-10a)
``` ```
GET /api/tts?text=&agent_id=&p=&a= -> audio/wav (chunked StreamingResponse) POST /api/tts {text, agent_id?, p?, a?} -> audio/wav (chunked StreamingResponse)
# GET (not POST) so a browser <audio src> plays it progressively (DEC-2 streaming). Params ride the query # POST (not GET) so an arbitrarily long turn rides the body, not a length-capped URL (DEC-10). The server
# string; text is capped ~2000 chars and truncated at a word boundary (URL-safe + bounds the shared-3090 hold). # chunk-and-concatenates under the 71.2s/call cap (tts_stream_long). text capped 8000 chars, word-boundary
# truncated (a shared-3090 hold bound; the transcript still shows the full text). ALL of text/agent_id/p/a
# are untrusted open-world body fields — each degrades, never 500s (INV-TTS-4).
steps: steps:
- missing text -> 400. resolve voice (per-character map -> "donut"; default Cora). - bad JSON / non-str text -> 400. Scrub lone surrogates from text (else httpx's utf-8 encode of the gateway
- dials = pad_to_dials(PadState(p, a)) from the p/a query floats (DEC-7, BROWSER-SENT live PAD); malformed body 500s); if the scrubbed text is blank after strip -> 400. word-boundary truncate to 8000 (with a
p/a -> neutral read, never a 500. mid-word HARD-CUT fallback when the last space sits at index <= limit//2).
- acquire the serialize lock (DEC-5, one stream at a time on the shared 3090); open tts_stream and PEEK the - resolve voice: per-character map -> "donut", default Cora; a NON-str agent_id (unhashable) -> default voice.
first chunk so a bad gateway OPEN surfaces as 503 (INV-TTS-4) before committing a 200. - dials = pad_to_dials(PadState.from_obj({pleasure:p, arousal:a})) — from_obj hardens the parse (a huge-int
- return StreamingResponse piping the chunks; the generator's finally releases the lock + closes the client OverflowError / non-numeric / missing axis -> neutral read), never a 500.
(incl. the browser-abort path: a new turn's <audio> load() drops the GET). - acquire the serialize lock (DEC-5, one stream at a time on the shared 3090); open tts_stream_long and PEEK
the first byte: a bad gateway OPEN on chunk 1 -> 503, and a 200 whose first bytes are NOT a RIFF header
-> 503 too (a mislabeled non-WAV body would decode as garbage) — both BEFORE committing a 200 (INV-TTS-4).
Any OTHER escape during the peek (CancelledError, httpx.InvalidURL) releases the lock+client, then propagates.
- return StreamingResponse piping the concatenated chunks; the generator's finally releases the lock +
closes the client (incl. the browser-abort path: a new turn's fetch() drops the POST). httpx.Timeout is
connect=10 / read=120 / write=10 / pool=10 (read=120 per infra-ops: a near-cap chunk can render slowly).
``` ```
### FN pin_kb_context (kb_bridge.py — RETIRE-READY, INV-KB-1) ### FN pin_kb_context (kb_bridge.py — RETIRE-READY, INV-KB-1)
@@ -206,10 +283,11 @@ pin_kb_context(question: str, agent_id: str | None, *, client) -> list[dict] #
on SSE `done`: on SSE `done`:
if !ttsEnabled(): return # INV-TTS-2 if !ttsEnabled(): return # INV-TTS-2
cancelTts() # INV-TTS-3: abort fetch + stop scheduled nodes cancelTts() # INV-TTS-3: abort fetch + stop scheduled nodes
fetch("/api/tts?text=&agent_id=&p=&a=") -> reader # chunked stream (text sliced to the 2000 cap) POST /api/tts {text (sliced to the 8000 cap), agent_id?, p?, a?} -> reader # DEC-10a: POST body, not a GET URL
loop: read chunk -> skip WAV header up to the data chunk -> int16 LE PCM -> Float32 -> AudioBuffer -> loop: read chunk -> skip ONE WAV header up to the data chunk (bounded 64KiB) -> int16 LE PCM -> Float32 ->
BufferSource.start(playAt) scheduled GAPLESSLY -> playAt += buf.duration # progressive, TTFA ~0.5s AudioBuffer -> BufferSource.start(playAt) GAPLESSLY -> playAt += buf.duration # progressive, TTFA ~0.5s
first scheduled node -> "▶ voiced"; any failure -> ticker + skip (INV-TTS-4) first scheduled node -> "▶ voiced". HARD failure (non-OK HTTP, or 64KiB with no WAV header) -> ticker + skip;
ABORT/cancel (INV-TTS-3 new-turn) + bare network error -> SILENT skip (INV-TTS-4, cancel is not a failure)
WHY Web Audio, not <audio src>: Safari/WebKit REFUSES a streaming 0xFFFFFFFF-length WAV via <audio src> WHY Web Audio, not <audio src>: Safari/WebKit REFUSES a streaming 0xFFFFFFFF-length WAV via <audio src>
(NotSupportedError — it can't compute duration/seek), which was the operator's live failure. Decoding the raw (NotSupportedError — it can't compute duration/seek), which was the operator's live failure. Decoding the raw
+203 -2
View File
@@ -14,6 +14,8 @@ Foot-guns (verified live 2026-08-02):
from __future__ import annotations from __future__ import annotations
import re
import sys
from collections.abc import AsyncIterator, Mapping from collections.abc import AsyncIterator, Mapping
from dataclasses import dataclass from dataclasses import dataclass
@@ -28,6 +30,12 @@ ZONOS_TTS_URL = "http://10.100.79.3:8890/v1/audio/speech"
# valence/arousal push by this). # valence/arousal push by this).
_DEFAULT_EMOTION_STRENGTH = 1.0 _DEFAULT_EMOTION_STRENGTH = 1.0
# DEC-9: pin English conditioning. Zonos is multilingual; without an explicit
# `language` it drifts into other-language phonemes / gibberish on names, numerics,
# and long spans (operator report 2026-08-02). The gateway accepts an ISO-ish
# `language` code (verified live against /v1/audio/speech).
_TTS_LANGUAGE = "en-us"
class TtsUnavailable(Exception): class TtsUnavailable(Exception):
"""The Zonos gateway failed, was unreachable, or returned a non-wav body. """The Zonos gateway failed, was unreachable, or returned a non-wav body.
@@ -123,8 +131,121 @@ def pad_to_dials(pad: PadState | None) -> EmotionDials:
def gateway_body(text: str, voice: str, dials: EmotionDials) -> dict: def gateway_body(text: str, voice: str, dials: EmotionDials) -> dict:
"""The Zonos gateway POST body — response_format is ALWAYS "wav" (DEC-3).""" """The Zonos gateway POST body — response_format is ALWAYS "wav" (DEC-3);
return {"input": text, "voice": voice, "response_format": "wav", **dials.to_body()} `language` is pinned to en-us (DEC-9) so the multilingual model stays in English."""
return {
"input": text,
"voice": voice,
"response_format": "wav",
"language": _TTS_LANGUAGE,
**dials.to_body(),
}
# DEC-10: long-form chunking. The Zonos model hard-caps ONE synthesis at max_tokens=6144
# = 71.2s of audio (6144 / 86.3 Hz codec frame rate; infra-ops verified 2026-08-02, and the
# gateway 400s above 6144 — the model's architectural sequence limit, unraisable). Anything
# longer must be chunk-and-concatenate client-side. We target 75% of the cap per chunk
# (operator call: greedy for prosody — fewer, fuller chunks, fewer seams), leaving the other
# 25% as headroom for char->audio-seconds variance: the cap is on TOKENS (6144 = 71.2s ALWAYS,
# a fixed codec-frame constant independent of delivery — infra-ops), but our budget is in CHARS
# (a proxy), and slower/expressive delivery stretches a given char-count into more audio-seconds
# (more tokens). ~14 chars/s at neutral rate => ~747-char budget (~53s).
_TTS_MODEL_CAP_SECONDS = 71.2
_TTS_CHUNK_TARGET_FRACTION = 0.75
_TTS_CHARS_PER_SEC = 14.0
_TTS_CHUNK_CHAR_BUDGET = int(
_TTS_MODEL_CAP_SECONDS * _TTS_CHUNK_TARGET_FRACTION * _TTS_CHARS_PER_SEC
)
_PARA_SPLIT = re.compile(r"\n\s*\n+") # blank-line paragraph boundary
_SENTENCE_SPLIT = re.compile(r"(?<=[.!?])\s+") # infra-ops recipe: split after . ! ?
_CLAUSE_BOUNDARIES = (", ", "; ", ": ")
def _hard_wrap(s: str, budget: int) -> list[str]:
"""Sub-split a single over-budget sentence into <= budget pieces, preferring a clause
boundary (`, ; :`), then any space; hard-cuts mid-word ONLY as a last resort (input
with no usable boundary at all). Never returns a piece longer than `budget`."""
out: list[str] = []
while len(s) > budget:
window = s[:budget]
cut = max((window.rfind(c) for c in _CLAUSE_BOUNDARIES), default=-1)
if cut > 0:
cut += 1 # keep the delimiter char with the head; break just after it
else:
cut = window.rfind(" ")
if cut <= 0:
cut = budget # no boundary in range — last-resort hard cut
out.append(s[:cut].strip())
s = s[cut:].strip()
if s:
out.append(s)
return out
def _greedy_pack(units: list[str], budget: int, join: str) -> list[str]:
"""Greedily pack pre-sized units (each <= budget) into <= budget chunks joined by
`join` — as few and as full as possible, so prosody flows across the fewest seams."""
chunks: list[str] = []
cur = ""
for u in units:
if cur and len(cur) + len(join) + len(u) > budget:
chunks.append(cur)
cur = u
else:
cur = f"{cur}{join}{u}" if cur else u
if cur:
chunks.append(cur)
return chunks
def chunk_text(text: str, budget: int = _TTS_CHUNK_CHAR_BUDGET) -> list[str]:
"""Split `text` into synthesis chunks each <= `budget` chars (DEC-10 / FN chunk_text).
Paragraph-first (operator call): whole paragraphs greedily pack together while they fit,
so seams land on blank-line boundaries where a natural pause already belongs. A paragraph
over budget flushes the pending whole-paragraph run, then falls back to sentence packing;
a sentence over budget falls back to clause/word sub-splitting (never mid-word unless the
input has no boundary at all). Every returned chunk is non-empty and <= budget;
empty/whitespace input -> []."""
# A non-positive budget would make _hard_wrap spin forever (cut=budget<=0 => no
# forward progress). Production always passes the 747 default; clamp a misuse to a
# sane floor rather than hang the event loop (degrade-not-crash, module-wide ethos).
budget = max(1, budget)
text = (text or "").strip()
if not text:
return []
chunks: list[str] = []
pending: list[str] = [] # whole paragraphs (each <= budget) awaiting a greedy pack
def flush_pending() -> None:
if pending:
chunks.extend(_greedy_pack(pending, budget, join="\n\n"))
pending.clear()
for para in _PARA_SPLIT.split(text):
para = para.strip()
if not para:
continue
if len(para) <= budget:
pending.append(para)
continue
# Oversized paragraph: emit the accumulated whole-paragraph chunks first (don't
# merge a mid-paragraph fragment across the blank-line boundary), then sentence-pack.
flush_pending()
sentences: list[str] = []
for sent in _SENTENCE_SPLIT.split(para):
sent = sent.strip()
if not sent:
continue
if len(sent) <= budget:
sentences.append(sent)
else:
sentences.extend(_hard_wrap(sent, budget))
chunks.extend(_greedy_pack(sentences, budget, join=" "))
flush_pending()
return chunks
async def tts_stream( async def tts_stream(
@@ -158,3 +279,83 @@ async def tts_stream(
yield chunk yield chunk
except httpx.RequestError as exc: except httpx.RequestError as exc:
raise TtsUnavailable(f"gateway transport failure: {exc}") from exc raise TtsUnavailable(f"gateway transport failure: {exc}") from exc
# The streaming WAV header is 44 bytes (RIFF 12 + fmt 24 + data 8; infra-ops 2026-08-02).
# A body that has not produced a `data` chunk id within this window is not the WAV we expect,
# so we stop scanning rather than buffer unbounded / match a `data` byte-run deep in PCM.
_WAV_HEADER_SCAN_LIMIT = 1024
async def _pcm_after_header(stream: AsyncIterator[bytes]) -> AsyncIterator[bytes]:
"""Yield only the PCM payload of a streaming WAV — everything AFTER the `data` chunk
id+size (8 bytes). Used for chunks 2..N so their RIFF header isn't buried mid-stream
(infra-ops 2026-08-02: one header only, or the concatenated file corrupts). The `data`
marker can straddle two network reads, so accumulate until it's found + 8 bytes land.
Raises TtsUnavailable if the body has no `data` chunk within the header window, or ends
before one completes — a malformed / non-WAV 200 body. tts_stream_long catches that for a
chunks-2..N failure and degrades (INV-TTS-4) rather than emitting header bytes as PCM or
buffering the whole body forever."""
acc = bytearray()
header_done = False
async for b in stream:
if header_done:
yield b
continue
acc.extend(b)
di = acc.find(b"data")
if di >= 0 and di + 8 <= len(acc):
header_done = True
tail = bytes(acc[di + 8:])
acc = bytearray()
if tail:
yield tail
elif len(acc) > _WAV_HEADER_SCAN_LIMIT:
# No `data` chunk within a sane header window — not the streaming WAV we expect.
raise TtsUnavailable("chunk 2+ body has no WAV data chunk within the header window")
if not header_done:
# Stream ended before a complete `data` header — truncated / empty chunk body.
raise TtsUnavailable("chunk 2+ body ended before the WAV data chunk")
async def tts_stream_long(
text: str,
*,
voice: str,
dials: EmotionDials,
client: httpx.AsyncClient,
url: str = ZONOS_TTS_URL,
budget: int = _TTS_CHUNK_CHAR_BUDGET,
) -> AsyncIterator[bytes]:
"""Synthesize arbitrarily long `text` as ONE continuous int16-PCM stream by chunking it
under the model's 71.2s cap (DEC-10) and concatenating. Chunk 1 streams verbatim (its
WAV header + PCM); chunks 2..N stream PCM-only (header stripped, `_pcm_after_header`) so
the browser decodes one gapless stream after a single leading header. Identical
voice+dials on every chunk so delivery stays uniform across the seams.
A failure BEFORE the first byte (an OPEN failure) propagates as TtsUnavailable so the
endpoint peek turns it into a 503 (INV-TTS-4) — nothing is committed yet. A failure AFTER
bytes have already streamed — a MID-STREAM drop, on chunk 0 or a later chunk, after the
200 is committed — degrades: drop the tail, keep what played, log, and NEVER raise into
the committed StreamingResponse (which would corrupt it with an ASGI trace). The pivot is
`yielded_any`, not the chunk index — a chunk-0 mid-stream drop is a committed-200 failure
too, not an open failure."""
chunks = chunk_text(text, budget)
yielded_any = False
for i, chunk in enumerate(chunks):
stream = tts_stream(chunk, voice=voice, dials=dials, client=client, url=url)
src = stream if i == 0 else _pcm_after_header(stream)
try:
async for b in src:
yielded_any = True
yield b
except TtsUnavailable as exc:
if not yielded_any:
raise # open failure, pre-commit → endpoint peek → 503
# committed-200 mid-stream failure → degrade + log (INV-TTS-4 "logs + skips").
sys.stderr.write(
f'{{"kind":"tts_degrade","event":"chunk_failed","chunk_index":{i},'
f'"chunks_total":{len(chunks)},"exc":"{type(exc).__name__}"}}\n'
)
return
+71 -37
View File
@@ -69,7 +69,7 @@ from ratatoskr.tts import (
PadState, PadState,
TtsUnavailable, TtsUnavailable,
pad_to_dials, pad_to_dials,
tts_stream, tts_stream_long,
) )
@@ -547,10 +547,12 @@ async def _memory_chunks_endpoint(request: Request) -> JSONResponse:
# Zonos voice; everything else falls to the gateway default. Case-folded gateway-side. # Zonos voice; everything else falls to the gateway default. Case-folded gateway-side.
_TTS_VOICE_MAP = {"ratatoskr:donut": "donut"} _TTS_VOICE_MAP = {"ratatoskr:donut": "donut"}
_TTS_DEFAULT_VOICE = "Cora" _TTS_DEFAULT_VOICE = "Cora"
# The streamed text rides the GET query string, so keep it URL-safe-short — a few # The text rides the POST body (DEC-10), so URL length is no longer the bound — this is a
# sentences is plenty for a voiced turn, and a very long response is truncated at a word # safety ceiling on the shared-3090 hold: the server chunk-and-concatenates under the model's
# boundary (the full text still shows in the transcript). Also bounds the shared-3090 hold. # 71.2s/call cap, so ~8000 chars (~11 chunks, ~9 min) covers any real interview turn while a
_TTS_MAX_TEXT_CHARS = 2000 # runaway is still bounded. A response past this is truncated at a word boundary (the full
# text still shows in the transcript).
_TTS_MAX_TEXT_CHARS = 8000
def _truncate_at_boundary(text: str, limit: int) -> str: def _truncate_at_boundary(text: str, limit: int) -> str:
@@ -563,53 +565,87 @@ def _truncate_at_boundary(text: str, limit: int) -> str:
async def _tts_endpoint(request: Request) -> Response: async def _tts_endpoint(request: Request) -> Response:
"""GET /api/tts?text=&agent_id=&p=&a= → audio/wav, STREAMED chunked from the Zonos """POST /api/tts {text, agent_id?, p?, a?} → audio/wav, STREAMED chunked from the Zonos
gateway (FN tts_endpoint). GET so a browser <audio src> plays it progressively; the gateway (FN tts_endpoint). POST (not GET) so an arbitrarily long turn rides the body,
gateway already streams (TTFB ~0.44s vs ~7s total), so we pipe the chunks straight not a length-capped URL — the server chunk-and-concatenates it under the model's 71.2s
through — never buffer, never rewrite the placeholder-size WAV header (that would per-call cap into ONE continuous stream (DEC-10, tts_stream_long). The gateway already
force buffering the whole clip and defeat the streaming). streams each chunk (TTFB ~0.44s), so we pipe the bytes straight through — chunk 1's WAV
header verbatim, chunks 2..N header-stripped, so the browser decodes one gapless stream.
Server-side proxy (DEC-4 / INV-TTS-1: the gateway host never reaches the browser). Server-side proxy (DEC-4 / INV-TTS-1: the gateway host never reaches the browser).
Voice per-character (DEC-8); emotion dials from the browser-sent live PAD (DEC-7, Voice per-character (DEC-8); emotion dials from the browser-sent live PAD (DEC-7, p/a
passed as p/a query floats). Serialized one-stream-at-a-time (DEC-5); a new turn body floats). Serialized one-stream-at-a-time (DEC-5); a new turn aborts the prior fetch
aborts the prior <audio> load → the GET drops → the generator's finally releases the → the POST drops → the generator's finally releases the lock. A gateway open-failure on
lock. A gateway open-failure → 503 (INV-TTS-4: the client skips playback).""" chunk 1 → 503 (INV-TTS-4: the client skips playback)."""
q = request.query_params try:
text = q.get("text") body = await request.json()
if not text: except (json.JSONDecodeError, ValueError):
return JSONResponse({"error_code": "missing_text"}, status_code=400)
text = body.get("text") if isinstance(body, dict) else None
if not isinstance(text, str):
return JSONResponse({"error_code": "missing_text"}, status_code=400)
# Drop lone surrogates: they ride through JSON (\udXXX from a mid-surrogate-pair slice)
# but raise UnicodeEncodeError when httpx serializes the gateway POST body (surrogates are
# not utf-8-encodable) — an untrusted-input 500. Scrub so the text is always encodable
# (INV-TTS-4 degrade). Also drops a whitespace-only body to a deterministic 400 below.
text = text.encode("utf-8", "ignore").decode("utf-8")
if not text.strip():
return JSONResponse({"error_code": "missing_text"}, status_code=400) return JSONResponse({"error_code": "missing_text"}, status_code=400)
text = _truncate_at_boundary(text, _TTS_MAX_TEXT_CHARS) text = _truncate_at_boundary(text, _TTS_MAX_TEXT_CHARS)
agent_id = q.get("agent_id") # agent_id / p / a are UNTRUSTED open-world body fields. A non-str agent_id (an
voice = _TTS_VOICE_MAP.get(agent_id, _TTS_DEFAULT_VOICE) # unhashable list/dict) would TypeError on the voice-map .get(); a huge JSON int p/a
pad: PadState | None = None # would OverflowError on float(). Both degrade to a neutral read, never a 500
try: # (INV-TTS-4). PadState.from_obj already hardens the numeric parse (catches
if q.get("p") is not None and q.get("a") is not None: # ArithmeticError/OverflowError) — route through it rather than re-implement a narrower net.
pad = PadState(pleasure=float(q["p"]), arousal=float(q["a"])) agent_id = body.get("agent_id")
except (TypeError, ValueError): voice = (
pad = None # malformed p/a → neutral read, never a 500 _TTS_VOICE_MAP.get(agent_id, _TTS_DEFAULT_VOICE)
dials = pad_to_dials(pad) if isinstance(agent_id, str) else _TTS_DEFAULT_VOICE
)
dials = pad_to_dials(PadState.from_obj({"pleasure": body.get("p"), "arousal": body.get("a")}))
tts_url = request.app.state.tts_url tts_url = request.app.state.tts_url
lock = request.app.state.tts_lock lock = request.app.state.tts_lock
# DEC-5: one stream at a time on the shared 3090. Held for the stream's duration and # DEC-5: one stream at a time on the shared 3090. Held for the stream's duration
# released in the generator's finally — including the browser-abort/cancel path. # (all chunks) and released in the generator's finally — including the browser-abort
# path. read=120s per infra-ops: a single near-cap chunk can render slowly under load.
await lock.acquire() await lock.acquire()
client = httpx.AsyncClient( client = httpx.AsyncClient(
timeout=httpx.Timeout(connect=10.0, read=30.0, write=10.0, pool=10.0) timeout=httpx.Timeout(connect=10.0, read=120.0, write=10.0, pool=10.0)
) )
gen = tts_stream(text, voice=voice, dials=dials, client=client, url=tts_url) gen = tts_stream_long(text, voice=voice, dials=dials, client=client, url=tts_url)
async def _release() -> None:
await gen.aclose() # unwinds tts_stream's `async with` → closes the gateway resp
await client.aclose()
lock.release()
try: try:
# Peek the first chunk so a bad gateway OPEN (non-200 / transport) surfaces as a # Peek the first chunk so a bad gateway OPEN (non-200 / transport) surfaces as a
# 503 BEFORE we commit a 200 StreamingResponse. TTFB ~0.44s, so this is cheap. # 503 BEFORE we commit a 200 StreamingResponse. TTFB ~0.44s, so this is cheap.
first = await gen.__anext__() first = await gen.__anext__()
except TtsUnavailable: except TtsUnavailable:
await gen.aclose() await _release()
await client.aclose()
lock.release()
return JSONResponse({"error_code": "tts_unavailable"}, status_code=503) return JSONResponse({"error_code": "tts_unavailable"}, status_code=503)
except StopAsyncIteration: except StopAsyncIteration:
first = None first = None
except BaseException:
# Any OTHER escape during the peek — a browser abort (CancelledError) before the
# first byte, or an unmapped transport error (e.g. httpx.InvalidURL from a
# misconfigured tts_url, which is NOT an httpx.RequestError) — must still release
# the process-wide serialize lock + client, or every later /api/tts deadlocks at
# lock.acquire() (DEC-5). The finally below only exists once piped() runs. Propagate.
await _release()
raise
# Chunk 1 must be a WAV (RIFF header). A 200 non-WAV body — a misbehaving gateway or a
# proxy error page — would be mislabeled audio/wav and the browser would decode markup as
# PCM (or match a `data` byte-run in it). Reject → 503 (INV-TTS-4). Tolerant of a <4-byte
# first read so a valid WAV split oddly is never false-rejected.
if first is not None and len(first) >= 4 and not first.startswith(b"RIFF"):
await _release()
return JSONResponse({"error_code": "tts_unavailable"}, status_code=503)
async def piped() -> AsyncIterator[bytes]: async def piped() -> AsyncIterator[bytes]:
try: try:
@@ -618,9 +654,7 @@ async def _tts_endpoint(request: Request) -> Response:
async for chunk in gen: async for chunk in gen:
yield chunk yield chunk
finally: finally:
await gen.aclose() # unwinds tts_stream's `async with` → closes the gateway resp await _release()
await client.aclose()
lock.release()
return StreamingResponse(piped(), media_type="audio/wav") return StreamingResponse(piped(), media_type="audio/wav")
@@ -836,7 +870,7 @@ def create_app(
Route("/api/sessions/{session_id}/messages", _session_messages_endpoint), Route("/api/sessions/{session_id}/messages", _session_messages_endpoint),
Route("/api/sessions/{session_id}/bifrost", _session_bifrost_endpoint), Route("/api/sessions/{session_id}/bifrost", _session_bifrost_endpoint),
Route("/api/admin/events", _admin_events_endpoint), Route("/api/admin/events", _admin_events_endpoint),
Route("/api/tts", _tts_endpoint, methods=["GET"]), Route("/api/tts", _tts_endpoint, methods=["POST"]),
Route("/api/turns/{session_id}", _submit_turn_endpoint, methods=["POST"]), Route("/api/turns/{session_id}", _submit_turn_endpoint, methods=["POST"]),
Route("/api/turns/{session_id}/stream", _stream_turn_endpoint), Route("/api/turns/{session_id}/stream", _stream_turn_endpoint),
Route("/api/turns/{session_id}/cancel", _cancel_turn_endpoint, methods=["POST"]), Route("/api/turns/{session_id}/cancel", _cancel_turn_endpoint, methods=["POST"]),
+19 -7
View File
@@ -1995,7 +1995,7 @@ async function cancelTurn() {
})(); })();
// ---- auto-TTS: voiced, affect-modulated STREAMING playback via Web Audio ------------ // ---- auto-TTS: voiced, affect-modulated STREAMING playback via Web Audio ------------
// Fetch the chunked GET /api/tts stream, decode its int16 PCM, and schedule the samples // Fetch the chunked POST /api/tts stream, decode its int16 PCM, and schedule the samples
// GAPLESSLY into an AudioContext as they arrive (TTFA ~0.5s). Web Audio, NOT <audio src>, // GAPLESSLY into an AudioContext as they arrive (TTFA ~0.5s). Web Audio, NOT <audio src>,
// because Safari/WebKit REFUSES a streaming 0xFFFFFFFF-length WAV via <audio src> // because Safari/WebKit REFUSES a streaming 0xFFFFFFFF-length WAV via <audio src>
// (NotSupportedError) — decoding the raw PCM ourselves sidesteps every WAV-container quirk // (NotSupportedError) — decoding the raw PCM ourselves sidesteps every WAV-container quirk
@@ -2042,15 +2042,22 @@ async function speakOnDone(text, agentId, pad) {
const ctx = _ttsAudioCtx(); const ctx = _ttsAudioCtx();
if (!ctx) { tickerAdd("err", "tts", "no audio ctx"); return; } if (!ctx) { tickerAdd("err", "tts", "no audio ctx"); return; }
if (ctx.state === "suspended") { try { await ctx.resume(); } catch (_) {} } if (ctx.state === "suspended") { try { await ctx.resume(); } catch (_) {} }
const params = new URLSearchParams({ text: clip.slice(0, 2000) }); // matches the server cap const payload = { text: clip.slice(0, 8000) }; // matches the server cap; chunked server-side (DEC-10)
if (agentId) params.set("agent_id", agentId); if (agentId) payload.agent_id = agentId;
if (pad && typeof pad.pleasure === "number" && typeof pad.arousal === "number") { if (pad && typeof pad.pleasure === "number" && typeof pad.arousal === "number") {
params.set("p", pad.pleasure); params.set("a", pad.arousal); // affect dials (DEC-7) payload.p = pad.pleasure; payload.a = pad.arousal; // affect dials (DEC-7)
} }
const ctrl = new AbortController(); _ttsAbort = ctrl; const ctrl = new AbortController(); _ttsAbort = ctrl;
let resp; let resp;
try { resp = await fetch("/api/tts?" + params.toString(), { signal: ctrl.signal }); } // POST (not GET) so a long turn rides the body, not a length-capped URL (DEC-10).
catch (_) { return; } // aborted / network → silent skip (INV-TTS-4) try {
resp = await fetch("/api/tts", {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify(payload),
signal: ctrl.signal,
});
} catch (_) { return; } // aborted / network → silent skip (INV-TTS-4)
if (!resp.ok || !resp.body) { tickerAdd("err", "tts", "unavailable " + resp.status); return; } if (!resp.ok || !resp.body) { tickerAdd("err", "tts", "unavailable " + resp.status); return; }
const reader = resp.body.getReader(); const reader = resp.body.getReader();
const SR = 44100; const SR = 44100;
@@ -2064,7 +2071,12 @@ async function speakOnDone(text, agentId, pad) {
if (!headerDone) { // skip the WAV header (up to + incl the data id/size) if (!headerDone) { // skip the WAV header (up to + incl the data id/size)
acc = _u8concat(acc, bytes); acc = _u8concat(acc, bytes);
const di = _findDataChunk(acc); const di = _findDataChunk(acc);
if (di < 0 || di + 8 > acc.length) continue; // header spans chunks — keep accumulating if (di < 0 || di + 8 > acc.length) {
// Bounded: a real header lands in the first read; if 64 KiB arrive with no `data`
// marker the stream is headerless/malformed — bail instead of buffering forever.
if (acc.length > 65536) { tickerAdd("err", "tts", "no WAV header"); break; }
continue; // header spans chunks — keep accumulating
}
bytes = acc.subarray(di + 8); headerDone = true; acc = null; bytes = acc.subarray(di + 8); headerDone = true; acc = null;
} }
const u8 = _u8concat(carry, bytes); // prepend the odd-byte carry; fresh + aligned const u8 = _u8concat(carry, bytes); // prepend the odd-byte carry; fresh + aligned
+205
View File
@@ -12,12 +12,15 @@ import pytest
import respx import respx
from ratatoskr.tts import ( from ratatoskr.tts import (
_TTS_CHUNK_CHAR_BUDGET,
EmotionDials, EmotionDials,
PadState, PadState,
TtsUnavailable, TtsUnavailable,
chunk_text,
gateway_body, gateway_body,
pad_to_dials, pad_to_dials,
tts_stream, tts_stream,
tts_stream_long,
) )
_URL = "http://tts.example/v1/audio/speech" _URL = "http://tts.example/v1/audio/speech"
@@ -36,6 +39,21 @@ async def _drain(gen) -> bytes:
return out return out
class _RaisingByteStream(httpx.AsyncByteStream):
"""A 200-body stream that yields `head` then drops mid-stream (an httpx.ReadError, a
RequestError subclass) — models a gateway connection drop AFTER the response committed."""
def __init__(self, head: bytes) -> None:
self._head = head
async def __aiter__(self):
yield self._head
raise httpx.ReadError("mid-stream drop")
async def aclose(self) -> None:
pass
class TestPadState: class TestPadState:
def test_from_obj_valid_mapping(self) -> None: def test_from_obj_valid_mapping(self) -> None:
pad = PadState.from_obj({"pleasure": 0.5, "arousal": -0.2, "dominance": 0.1}) pad = PadState.from_obj({"pleasure": 0.5, "arousal": -0.2, "dominance": 0.1})
@@ -110,6 +128,7 @@ class TestGatewayBody:
b = gateway_body("hi", "donut", pad_to_dials(PadState(pleasure=0.5, arousal=0.2))) b = gateway_body("hi", "donut", pad_to_dials(PadState(pleasure=0.5, arousal=0.2)))
assert b["input"] == "hi" and b["voice"] == "donut" assert b["input"] == "hi" and b["voice"] == "donut"
assert b["response_format"] == "wav" # DEC-3 — ALWAYS wav assert b["response_format"] == "wav" # DEC-3 — ALWAYS wav
assert b["language"] == "en-us" # DEC-9 — pin English conditioning
assert b["emotion_valence"] == pytest.approx(0.5) assert b["emotion_valence"] == pytest.approx(0.5)
def test_neutral_omits_emotion(self) -> None: def test_neutral_omits_emotion(self) -> None:
@@ -133,6 +152,7 @@ class TestTtsStream:
body = _json.loads(route.calls.last.request.content) body = _json.loads(route.calls.last.request.content)
assert body["voice"] == "donut" assert body["voice"] == "donut"
assert body["response_format"] == "wav" assert body["response_format"] == "wav"
assert body["language"] == "en-us" # DEC-9 — pin English conditioning
assert body["emotion_valence"] == pytest.approx(0.5) assert body["emotion_valence"] == pytest.approx(0.5)
@respx.mock @respx.mock
@@ -153,3 +173,188 @@ class TestTtsStream:
await _drain(tts_stream( await _drain(tts_stream(
"hi", voice="Cora", dials=pad_to_dials(None), client=client, url=_URL "hi", voice="Cora", dials=pad_to_dials(None), client=client, url=_URL
)) ))
class TestChunkText:
"""chunk_text (DEC-10): paragraph-first greedy pack, sentence fallback for oversized
paragraphs, clause/word sub-split for oversized sentences; every chunk <= budget."""
def test_empty_and_whitespace_yield_no_chunks(self) -> None:
assert chunk_text("") == []
assert chunk_text(" \n\n \t ") == []
def test_short_text_is_one_chunk(self) -> None:
assert chunk_text("Hello, darling.", budget=100) == ["Hello, darling."]
def test_two_short_paragraphs_greedily_merge(self) -> None:
# Both fit in one budget -> one chunk, joined on the blank-line boundary.
out = chunk_text("First para.\n\nSecond para.", budget=100)
assert out == ["First para.\n\nSecond para."]
def test_paragraphs_split_on_blank_line_when_over_budget(self) -> None:
# Each paragraph fits alone but not together -> a seam on the paragraph boundary.
a, b = "A" * 30, "B" * 30
out = chunk_text(f"{a}\n\n{b}", budget=40)
assert out == [a, b]
def test_oversized_paragraph_falls_back_to_sentences(self) -> None:
para = "One sentence here. Two sentence here. Three sentence here."
out = chunk_text(para, budget=25)
assert all(len(c) <= 25 for c in out)
assert len(out) >= 2
# every word is preserved whole and in order (no split mid-word)
assert [w for c in out for w in c.split()] == para.split()
def test_oversized_sentence_sub_splits_never_mid_word(self) -> None:
sent = "alpha beta gamma delta epsilon zeta eta theta iota kappa lambda"
out = chunk_text(sent, budget=20)
assert all(len(c) <= 20 for c in out)
for c in out:
for word in c.split():
assert word in sent.split() # every emitted token is a whole source word
def test_every_chunk_within_budget_default(self) -> None:
para = ("Princess Donut does not wait. " * 200).strip()
out = chunk_text(para) # default budget
assert out and all(len(c) <= _TTS_CHUNK_CHAR_BUDGET for c in out)
def test_spaceless_over_budget_hard_cuts_as_last_resort(self) -> None:
out = chunk_text("x" * 50, budget=20)
assert all(len(c) <= 20 for c in out)
assert "".join(out) == "x" * 50
def test_non_positive_budget_does_not_hang(self) -> None:
# budget <= 0 would infinite-loop _hard_wrap; it's clamped to 1 so this terminates.
out = chunk_text("alpha beta", budget=0)
assert out and all(len(c) <= 1 for c in out)
assert "".join(out) == "alphabeta" # every char preserved, forward progress made
def test_oversized_sentence_prefers_clause_boundary_over_space(self) -> None:
# A comma-bearing over-budget sentence sub-splits at the CLAUSE boundary (", "),
# not merely at the last space — pins the _CLAUSE_BOUNDARIES preference (else dead).
out = chunk_text("alpha, beta gamma delta", budget=12)
assert all(len(c) <= 12 for c in out)
assert out[0] == "alpha," # clause cut, not "alpha, beta" (a space-only cut)
def test_default_budget_is_the_dec10_value(self) -> None:
# Pin the concrete 747 that FN chunk_text's POST commits to (75% of 71.2s @ 14 c/s).
# The suite's other budget checks compare against the imported constant and so move
# with it; this one anchors the value itself so a retune is a deliberate edit here.
assert _TTS_CHUNK_CHAR_BUDGET == 747
class TestTtsStreamLong:
"""tts_stream_long (DEC-10): concatenate per-chunk synthesis into ONE int16-PCM stream
— chunk 1 verbatim (header + PCM), chunks 2..N header-stripped."""
_PCM = b"\x11\x22" * 64
_WAV_CHUNK = (
b"RIFF\xff\xff\xff\xffWAVEfmt \x10\x00\x00\x00" + b"\x00" * 20
+ b"data\xff\xff\xff\xff" + _PCM
)
@respx.mock
async def test_single_chunk_passes_through_verbatim(self) -> None:
respx.post(_URL).mock(return_value=httpx.Response(200, content=self._WAV_CHUNK))
async with httpx.AsyncClient() as client:
out = await _drain(tts_stream_long(
"Short line.", voice="donut", dials=pad_to_dials(None),
client=client, url=_URL, budget=100,
))
assert out == self._WAV_CHUNK # one chunk => untouched
@respx.mock
async def test_multi_chunk_emits_one_header_then_concatenated_pcm(self) -> None:
respx.post(_URL).mock(return_value=httpx.Response(200, content=self._WAV_CHUNK))
text = "First part here. Second part here. Third part here." # budget 18 -> >=2 chunks
async with httpx.AsyncClient() as client:
out = await _drain(tts_stream_long(
text, voice="donut", dials=pad_to_dials(None), client=client, url=_URL, budget=18,
))
n = len(chunk_text(text, budget=18))
assert n >= 2
assert out.count(b"RIFF") == 1 and out.count(b"data") == 1 # exactly one header
# EXACT bytes: chunk 1 verbatim (header+PCM), chunks 2..N stripped to PCM. Asserting
# the exact stream catches a di+4-vs-di+8 strip off-by-one (2-byte sample alignment
# across seams) that a header-count check alone would miss.
assert out == self._WAV_CHUNK + self._PCM * (n - 1)
# DEC-10: identical voice+dials+language on EVERY chunk (uniform delivery across seams).
import json as _json
bodies = [_json.loads(c.request.content) for c in respx.calls]
assert len(bodies) == n
assert all(b["voice"] == "donut" and b["language"] == "en-us" for b in bodies)
@respx.mock
async def test_mid_stream_drop_after_first_byte_degrades_not_raises(self) -> None:
# A2: chunk 0 opens 200, yields bytes, then drops mid-stream. Because the 200 is
# committed (bytes already flowed), this must DEGRADE (return what streamed), never
# raise — the pivot is yielded_any, not the chunk index.
respx.post(_URL).mock(
return_value=httpx.Response(200, stream=_RaisingByteStream(self._WAV_CHUNK))
)
async with httpx.AsyncClient() as client:
out = await _drain(tts_stream_long(
"hi", voice="donut", dials=pad_to_dials(None), client=client, url=_URL, budget=100,
))
assert out == self._WAV_CHUNK # head kept, no raise
@respx.mock
async def test_later_chunk_gateway_500_degrades_keeps_prior(self) -> None:
# chunk 1 = valid WAV; chunk 2 = a gateway 500 (OPEN failure on a later chunk).
respx.post(_URL).mock(side_effect=[
httpx.Response(200, content=self._WAV_CHUNK),
httpx.Response(500, content=b"boom"),
])
text = "First part here. Second part here." # budget 18 -> 2 chunks
async with httpx.AsyncClient() as client:
out = await _drain(tts_stream_long(
text, voice="donut", dials=pad_to_dials(None), client=client, url=_URL, budget=18,
))
assert out == self._WAV_CHUNK # INV-TTS-4 degrade: keep chunk 1, drop the tail, no raise
@respx.mock
async def test_later_chunk_missing_data_degrades_keeps_prior(self) -> None:
# chunk 1 = valid WAV; chunk 2 = a 200 non-WAV body (no `data` chunk) -> degrade.
respx.post(_URL).mock(side_effect=[
httpx.Response(200, content=self._WAV_CHUNK),
httpx.Response(200, content=b"xxxxx no marker present xxxxx"),
])
text = "First part here. Second part here." # budget 18 -> 2 chunks
async with httpx.AsyncClient() as client:
out = await _drain(tts_stream_long(
text, voice="donut", dials=pad_to_dials(None), client=client, url=_URL, budget=18,
))
# INV-TTS-4 degrade: chunk 1 audio retained verbatim, chunk 2 dropped (no raise, no
# garbage bytes emitted from the malformed body).
assert out == self._WAV_CHUNK
@respx.mock
async def test_first_chunk_gateway_failure_raises(self) -> None:
respx.post(_URL).mock(return_value=httpx.Response(500, content=b"boom"))
async with httpx.AsyncClient() as client:
with pytest.raises(TtsUnavailable):
await _drain(tts_stream_long(
"hi", voice="Cora", dials=pad_to_dials(None),
client=client, url=_URL, budget=100,
))
async def test_pcm_after_header_reassembles_data_marker_across_reads(self) -> None:
# The `data` marker can straddle two network reads; _pcm_after_header must accumulate
# until it lands, then yield only the PCM after it. Pins the docstring's straddle claim.
from ratatoskr.tts import _pcm_after_header
async def _split_stream():
yield b"RIFF\xff\xff\xff\xffWAVEfmt \x10\x00\x00\x00" + b"\x00" * 20 + b"da"
yield b"ta\xff\xff\xff\xff" + b"\x11\x22" * 4 # rest of 'data' + size + PCM
out = await _drain(_pcm_after_header(_split_stream()))
assert out == b"\x11\x22" * 4 # PCM only; marker reassembled across the read boundary
async def test_pcm_after_header_no_data_marker_raises(self) -> None:
from ratatoskr.tts import _pcm_after_header
async def _no_marker():
yield b"xxxxx no marker present xxxxx"
with pytest.raises(TtsUnavailable):
await _drain(_pcm_after_header(_no_marker()))
+119 -25
View File
@@ -1458,10 +1458,11 @@ class TestMemoryChunksEndpoint:
class TestTtsEndpoint: class TestTtsEndpoint:
"""tts_endpoint FN — GET /api/tts → audio/wav STREAMED (chunked) from the Zonos """tts_endpoint FN — POST /api/tts → audio/wav STREAMED (chunked) from the Zonos
gateway. Voice per-character (DEC-8), emotion dials from p/a query floats (DEC-7), gateway. Voice per-character (DEC-8), emotion dials from p/a body floats (DEC-7),
the gateway host never reaches the browser (INV-TTS-1), gateway open-failure → 503 the gateway host never reaches the browser (INV-TTS-1), gateway open-failure → 503
(INV-TTS-4). GET so a browser <audio src> plays it progressively.""" (INV-TTS-4). POST so an arbitrarily long turn rides the body; the server chunk-and-
concatenates it under the model's 71.2s/call cap (DEC-10)."""
# Streaming WAV bytes (placeholder 0xFFFFFFFF sizes) — proxied through verbatim. # Streaming WAV bytes (placeholder 0xFFFFFFFF sizes) — proxied through verbatim.
_WAV = b"RIFF\xff\xff\xff\xffWAVEdata\xff\xff\xff\xff" + b"\x11\x22" * 64 _WAV = b"RIFF\xff\xff\xff\xffWAVEdata\xff\xff\xff\xff" + b"\x11\x22" * 64
@@ -1475,17 +1476,18 @@ class TestTtsEndpoint:
return_value=httpx.Response(200, content=self._WAV) return_value=httpx.Response(200, content=self._WAV)
) )
app = create_app(_mock_client_factory(), tts_url=self._TTS) app = create_app(_mock_client_factory(), tts_url=self._TTS)
resp = TestClient(app).get( resp = TestClient(app).post(
"/api/tts", "/api/tts",
params={"text": "Carl is a softie.", "agent_id": "ratatoskr:donut", json={"text": "Carl is a softie.", "agent_id": "ratatoskr:donut",
"p": "0.6", "a": "0.3"}, "p": 0.6, "a": 0.3},
) )
assert resp.status_code == 200 assert resp.status_code == 200
assert resp.headers["content-type"].startswith("audio/wav") assert resp.headers["content-type"].startswith("audio/wav")
assert resp.content == self._WAV # streamed through verbatim, no header rewrite assert resp.content == self._WAV # one chunk → streamed through verbatim
body = json.loads(route.calls.last.request.content) body = json.loads(route.calls.last.request.content)
assert body["voice"] == "donut" assert body["voice"] == "donut"
assert body["response_format"] == "wav" assert body["response_format"] == "wav"
assert body["language"] == "en-us" # DEC-9 pinned per chunk
assert body["emotion_valence"] == pytest.approx(0.6) assert body["emotion_valence"] == pytest.approx(0.6)
@respx.mock @respx.mock
@@ -1496,7 +1498,7 @@ class TestTtsEndpoint:
return_value=httpx.Response(200, content=self._WAV) return_value=httpx.Response(200, content=self._WAV)
) )
app = create_app(_mock_client_factory(), tts_url=self._TTS) app = create_app(_mock_client_factory(), tts_url=self._TTS)
resp = TestClient(app).get("/api/tts", params={"text": "hello", "agent_id": "mimir"}) resp = TestClient(app).post("/api/tts", json={"text": "hello", "agent_id": "mimir"})
assert resp.status_code == 200 assert resp.status_code == 200
body = json.loads(route.calls.last.request.content) body = json.loads(route.calls.last.request.content)
assert body["voice"] == "Cora" # gateway default (DEC-8) assert body["voice"] == "Cora" # gateway default (DEC-8)
@@ -1506,7 +1508,7 @@ class TestTtsEndpoint:
from ratatoskr.web.server import create_app from ratatoskr.web.server import create_app
app = create_app(_mock_client_factory(), tts_url=self._TTS) app = create_app(_mock_client_factory(), tts_url=self._TTS)
resp = TestClient(app).get("/api/tts", params={"agent_id": "ratatoskr:donut"}) resp = TestClient(app).post("/api/tts", json={"agent_id": "ratatoskr:donut"})
assert resp.status_code == 400 assert resp.status_code == 400
assert resp.json()["error_code"] == "missing_text" assert resp.json()["error_code"] == "missing_text"
@@ -1516,32 +1518,124 @@ class TestTtsEndpoint:
respx.post(self._TTS).mock(return_value=httpx.Response(500, content=b"boom")) respx.post(self._TTS).mock(return_value=httpx.Response(500, content=b"boom"))
app = create_app(_mock_client_factory(), tts_url=self._TTS) app = create_app(_mock_client_factory(), tts_url=self._TTS)
resp = TestClient(app).get("/api/tts", params={"text": "hi"}) resp = TestClient(app).post("/api/tts", json={"text": "hi"})
assert resp.status_code == 503 assert resp.status_code == 503
assert resp.json()["error_code"] == "tts_unavailable" assert resp.json()["error_code"] == "tts_unavailable"
@respx.mock @respx.mock
def test_long_text_truncated_at_cap(self) -> None: def test_long_text_is_chunked_each_call_within_budget(self) -> None:
from ratatoskr.web.server import _TTS_MAX_TEXT_CHARS, create_app from ratatoskr.tts import _TTS_CHUNK_CHAR_BUDGET
route = respx.post(self._TTS).mock(
return_value=httpx.Response(200, content=self._WAV)
)
app = create_app(_mock_client_factory(), tts_url=self._TTS)
resp = TestClient(app).get("/api/tts", params={"text": "word " * 1000})
assert resp.status_code == 200
body = json.loads(route.calls.last.request.content)
assert len(body["input"]) <= _TTS_MAX_TEXT_CHARS # truncated for the URL / lock
@respx.mock
def test_malformed_pad_query_degrades_no_500(self) -> None:
from ratatoskr.web.server import create_app from ratatoskr.web.server import create_app
route = respx.post(self._TTS).mock( route = respx.post(self._TTS).mock(
return_value=httpx.Response(200, content=self._WAV) return_value=httpx.Response(200, content=self._WAV)
) )
app = create_app(_mock_client_factory(), tts_url=self._TTS) app = create_app(_mock_client_factory(), tts_url=self._TTS)
resp = TestClient(app).get("/api/tts", params={"text": "hi", "p": "notafloat", "a": "0.1"}) # ~5000 chars of distinct sentences → several gateway calls (DEC-10 chunking).
text = " ".join(f"Sentence number {i} about the dungeon." for i in range(200))
resp = TestClient(app).post("/api/tts", json={"text": text})
assert resp.status_code == 200
assert len(route.calls) >= 2 # chunk-and-concatenate, not one giant call
for call in route.calls:
body = json.loads(call.request.content)
assert len(body["input"]) <= _TTS_CHUNK_CHAR_BUDGET # each chunk under cap
@respx.mock
def test_malformed_pad_body_degrades_no_500(self) -> None:
from ratatoskr.web.server import create_app
route = respx.post(self._TTS).mock(
return_value=httpx.Response(200, content=self._WAV)
)
app = create_app(_mock_client_factory(), tts_url=self._TTS)
resp = TestClient(app).post("/api/tts", json={"text": "hi", "p": "notafloat", "a": 0.1})
assert resp.status_code == 200 assert resp.status_code == 200
body = json.loads(route.calls.last.request.content) body = json.loads(route.calls.last.request.content)
assert "emotion_valence" not in body # malformed p → neutral read, not a 500 assert "emotion_valence" not in body # malformed p → neutral read, not a 500
@respx.mock
def test_huge_int_pad_degrades_no_500(self) -> None:
# A 400-digit JSON int → float() OverflowError (an ArithmeticError, not a
# ValueError). POST re-opened this path that GET's string query params couldn't.
from ratatoskr.web.server import create_app
route = respx.post(self._TTS).mock(
return_value=httpx.Response(200, content=self._WAV)
)
app = create_app(_mock_client_factory(), tts_url=self._TTS)
resp = TestClient(app).post("/api/tts", json={"text": "hi", "p": 10**400, "a": 0.1})
assert resp.status_code == 200
body = json.loads(route.calls.last.request.content)
assert "emotion_valence" not in body # OverflowError → neutral read, not a 500
@respx.mock
def test_non_str_agent_id_degrades_no_500(self) -> None:
# An unhashable agent_id (list/dict) would TypeError on _TTS_VOICE_MAP.get().
from ratatoskr.web.server import create_app
route = respx.post(self._TTS).mock(
return_value=httpx.Response(200, content=self._WAV)
)
app = create_app(_mock_client_factory(), tts_url=self._TTS)
resp = TestClient(app).post("/api/tts", json={"text": "hi", "agent_id": ["donut"]})
assert resp.status_code == 200
body = json.loads(route.calls.last.request.content)
assert body["voice"] == "Cora" # non-str agent_id → default voice, not a 500
def test_whitespace_text_returns_400(self) -> None:
from ratatoskr.web.server import create_app
app = create_app(_mock_client_factory(), tts_url=self._TTS)
resp = TestClient(app).post("/api/tts", json={"text": " \n\t "})
assert resp.status_code == 400
assert resp.json()["error_code"] == "missing_text"
def test_lock_released_when_peek_raises_unexpectedly(self, monkeypatch) -> None:
# An unmapped error during the first-byte peek (e.g. httpx.InvalidURL, or a
# browser abort/CancelledError before byte 1) must still release the process-wide
# serialize lock, or every later /api/tts deadlocks at lock.acquire() (DEC-5).
from ratatoskr.web import server as _server
from ratatoskr.web.server import create_app
async def _boom(*a, **k):
raise RuntimeError("boom during peek")
yield b"" # unreachable — marks this an async generator
monkeypatch.setattr(_server, "tts_stream_long", _boom)
app = create_app(_mock_client_factory(), tts_url=self._TTS)
with pytest.raises(RuntimeError):
TestClient(app).post("/api/tts", json={"text": "hi"})
assert app.state.tts_lock.locked() is False
@respx.mock
def test_lone_surrogate_text_scrubbed_no_500(self) -> None:
# A lone surrogate rides JSON (\udXXX) but would UnicodeEncodeError when httpx
# serializes the gateway body → a 500. It's scrubbed at the text guard instead.
from ratatoskr.web.server import create_app
route = respx.post(self._TTS).mock(
return_value=httpx.Response(200, content=self._WAV)
)
app = create_app(_mock_client_factory(), tts_url=self._TTS)
# Send the surrogate as an escaped JSON literal — TestClient's json= can't encode it.
resp = TestClient(app).post(
"/api/tts", content='{"text": "\\ud800hello"}',
headers={"Content-Type": "application/json"},
)
assert resp.status_code == 200
body = json.loads(route.calls.last.request.content)
assert body["input"] == "hello" # surrogate dropped, rest intact — not a 500
@respx.mock
def test_non_wav_200_body_rejected_503(self) -> None:
# A 200 whose first bytes are not a RIFF header (a proxy error page) must not be
# mislabeled audio/wav — the peek RIFF-sniff rejects it as 503.
from ratatoskr.web.server import create_app
respx.post(self._TTS).mock(
return_value=httpx.Response(200, content=b"<html>error: data unavailable</html>")
)
app = create_app(_mock_client_factory(), tts_url=self._TTS)
resp = TestClient(app).post("/api/tts", json={"text": "hi"})
assert resp.status_code == 503
assert resp.json()["error_code"] == "tts_unavailable"