vh/ratatoskr
1
0
Fork 0
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity

feat(tts): pin English, stream long turns via chunking, dialogue-only Donut

Browse Source 
TTS fixes + hardening for the Donut voiced interview.

Feature:
- gibberish -> pin `language: "en-us"` on every gateway call (DEC-9); the
  multilingual model drifted into other-language phonemes without it.
- truncation -> the Zonos model hard-caps one synthesis at 6144 tokens /
  71.2s (infra-ops). Chunk client-side (paragraph-first, greedy to ~75%
  of cap for prosody; sentence/clause fallback) and concatenate the int16
  PCM behind ONE WAV header (DEC-10). /api/tts becomes POST so a long turn
  rides the body, not a length-capped URL (DEC-10a).
- persona -> dialogue-only rewrite (no asterisk RP beats -- they were being
  voiced as gibberish) + always consult the native `reference_knowledge`
  tool before answering (retires the stale kb_bridge references). Pushed
  live to ratatoskr:donut.

Heid code-review + bug-hunt hardening (4-arm panels, triaged):
- untrusted /api/tts body fields degrade, never 500: huge-int PAD
  (OverflowError), non-str agent_id (unhashable .get), lone surrogates
  (utf-8 encode), whitespace-only text.
- serialize lock + client released on every peek escape (cancel /
  InvalidURL) -- previously a permanent deadlock.
- a mid-stream drop after a committed 200 degrades (keeps what played),
  never raises into the response; a non-WAV 200 body is rejected (RIFF
  sniff + bounded header scan) instead of decoded as garbage.

546 tests green; long-form live-verified (106.6s, one header). Contract
brought canonical (DEC-9/10, FN chunk_text/tts_stream_long, POST endpoint,
INV-TTS-4 logging scope, FN pad_to_dials domain). reference_knowledge
empty-recall root-caused to a Worldtree wing-misfile (escalated to
worldtree-dev; not ratatoskr code).
This commit is contained in:
vh
2026-08-02 14:11:31 -07:00
parent 1346cb2836
commit d59f907962
7 changed files with 750 additions and 111 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/contracts/donut_voiced_interview.contract.md
+99 -21
View File
@@ -72,6 +72,9 @@ each independently shippable. Slice order is chosen for fastest visible result.
`_finalize_wav_header` hit). The placeholder-size WAV is DESIGNED for `<audio src>`
progressive playback (validated in Chromium: plays, currentTime advances, no MSE
needed). Supersedes the original "full-synth latency accepted / no streaming."
(Amended 2026-08-02: the browser path is now Web Audio decoding raw int16 PCM, not
`<audio src>` — Safari/WebKit rejects a 0xFFFFFFFF-length streaming WAV via `<audio
src>` (NotSupportedError); and `/api/tts` is now POST, not GET. See DEC-10.)
- **DEC-3 — wav only.** `response_format:"wav"` (streaming int16 RIFF/WAVE). `mp3`/`opus`
are accepted but silently return mislabeled PCM — never request them.
- **DEC-4 — server-side proxy.** Browser → `/api/tts` (nh3-dev) → gateway. The
@@ -91,6 +94,15 @@ each independently shippable. Slice order is chosen for fastest visible result.
bridge is isolated behind ONE seam (INV-KB-1) so it deletes cleanly when
Worldtree #361 `reference_knowledge` extends to Tier-3 (operator-flagged as an
integral gap; worldtree-dev surfacing the extension to Vuong).
- **DEC-6 status (2026-08-02): bridge RETIRED, native `reference_knowledge` live but
EMPTY.** The bridge was deleted (`09e4257`) when WT #383 native `reference_knowledge`
(b167) shipped — Donut now calls the tool in-turn. But the tool returns zero hits for
every query. Root-caused 2026-08-02: Mimir's `search_library` DOES find the DCC corpus
(main wing, score ~0.03), so the store is NOT empty and this is NOT a ratatoskr gap —
WT's native `reference_knowledge` isn't surfacing content Mimir retrieves fine (likely
a wing-scope mismatch — tool scoped to the failed `fiction` wing — or a confidence
threshold rejecting the weak ~0.03 hits). Escalated to worldtree-dev. Until fixed,
Donut recalls from her own character knowledge (degrades in-voice per the persona).
- **DEC-7 — affect-driven emotion.** Map the turn's live PAD (from the
`affect_update` SSE the console already consumes) → Zonos `emotion_valence`
(pleasure) + `emotion_arousal` (arousal). This reframes the feature as voice
@@ -103,6 +115,28 @@ each independently shippable. Slice order is chosen for fastest visible result.
fall to the gateway default (Cora). (Amended 2026-08-02 per heid-code-review: 3 arms
flagged the code as DEC-8 drift; a live gateway read INVERTED the remedy — the code is
correct, DEC-8's "preset now" was stale.)
- **DEC-9 — pin English (added 2026-08-02, operator-directed).** Zonos is multilingual;
with no `language` field it drifts into other-language phonemes / gibberish on names,
numerics, and long spans (operator report). `gateway_body` pins `language:"en-us"` on
every call (gateway accepts it, verified live). The persona's dialogue-only rewrite
(`docs/characters/donut.md`) removes the other gibberish vector — asterisk RP action
beats were being voiced verbatim.
- **DEC-10 — long-form chunk-and-concatenate (added 2026-08-02, operator-directed;
infra-ops recipe 01KZ1FKX…).** The Zonos model hard-caps ONE synthesis at
`max_tokens=6144` = 71.2s of audio (6144 / 86.3 Hz codec frame rate; the gateway 400s
above 6144 — an architectural sequence limit, unraisable). So a turn longer than ~71s
truncated mid-stream. Fix: split the text client-side and concatenate the PCM
(`chunk_text` + `tts_stream_long`). Chunking is paragraph-first + greedy (operator call:
fewer, fuller chunks for prosody), targeting ~75% of the cap per chunk (~747 chars @
~14 c/s); the 25% headroom covers char→audio-seconds variance (the cap is on TOKENS —
6144 = 71.2s ALWAYS, a codec-frame constant regardless of delivery, infra-ops — while the
budget is in CHARS, a proxy that stretches under slow/expressive delivery) so no chunk clips;
oversized paragraphs fall back to sentence packing, oversized sentences to clause/word
sub-splitting. Concatenation emits chunk 1's WAV verbatim then chunks 2..N header-
stripped → ONE continuous int16-PCM stream (never bury a RIFF header mid-stream —
infra-ops). Identical voice+dials+language per chunk for uniform delivery. `/api/tts`
becomes POST (DEC-10a) so the full text rides the body, not a length-capped URL; the
outer text cap rises 2000→8000 (a shared-3090 hold bound, not a URL bound).
## Invariants
@@ -113,7 +147,10 @@ each independently shippable. Slice order is chosen for fastest visible result.
- **INV-TTS-3 [hard]** — one synth in flight at a time; a new turn cancels the
prior synth request AND stops in-flight playback (cancel-on-new-turn).
- **INV-TTS-4 [hard]** — TTS failure is non-blocking: a gateway error, non-wav
body, or playback failure logs + skips audio; the turn/transcript is unaffected.
body, or playback failure skips audio; the turn/transcript is unaffected. Logging is
scoped to GENUINE failure: a committed-200 mid-stream/later-chunk degrade writes a
`tts_degrade` stderr line (server) or a `no WAV header` ticker (browser); a browser-side
ABORT/cancel (INV-TTS-3 new-turn) is deliberately SILENT — cancellation is not a failure.
- **INV-KB-1 [hard]** — the KB bridge is import-isolated behind a single seam:
`server.py`'s turn path calls exactly one function `pin_kb_context(question,
agent_id) -> list[memory_context] | []`. Retiring the bridge = delete
@@ -131,18 +168,47 @@ each independently shippable. Slice order is chosen for fastest visible result.
## FN blocks
### FN tts_stream (replaces the buffered tts_synthesize — DEC-2 streaming)
### FN tts_stream (the per-CHUNK primitive — DEC-2 streaming; wrapped by tts_stream_long)
```
tts_stream(text, *, voice, dials, client: httpx.AsyncClient, url=ZONOS_TTS_URL) -> AsyncIterator[bytes]
# Open the gateway's CHUNKED stream (client.stream("POST", url, json=gateway_body(...))) and YIELD wav
# chunks as they synthesize. Pass through verbatim — never buffer, never rewrite the placeholder header.
# gateway_body(text, voice, dials) = {input, voice, response_format:"wav", **dials.to_body()}.
# gateway_body(text, voice, dials) = {input, voice, response_format:"wav", language:"en-us" (DEC-9),
# **dials.to_body()}.
precondition: text non-empty. Voice membership in /v1/voices is GATEWAY-enforced, not client-asserted.
postcondition: yields the gateway's chunked int16 streaming WAV bytes unmodified (0xFFFFFFFF placeholder
sizes intact — the browser <audio src> plays them progressively).
sizes intact). This is ONE synthesis (<= 71.2s cap, DEC-10); tts_stream_long stitches many.
error: a non-200 OPEN or connect/transport failure -> TtsUnavailable BEFORE the first chunk (so the
endpoint can still return 503); a mid-stream drop just ends the generator.
invariant: response_format is ALWAYS "wav" (DEC-3); never mp3/opus.
invariant: response_format is ALWAYS "wav" (DEC-3); never mp3/opus. language ALWAYS "en-us" (DEC-9).
```
### FN chunk_text (DEC-10 long-form splitting; pure)
```
chunk_text(text, budget=_TTS_CHUNK_CHAR_BUDGET) -> list[str]
# Split into synthesis chunks each <= budget chars. Paragraph-first (seams on blank lines), greedy pack.
steps:
- strip; empty/whitespace -> [].
- whole paragraphs (each <= budget) greedily pack together, joined "\n\n".
- a paragraph > budget flushes the pending run, then sentence-packs (split (?<=[.!?])\s+, join " ");
a sentence > budget sub-splits on clause (, ; :) then space, hard-cut mid-word only as last resort.
postcondition: every chunk non-empty and <= budget; word order preserved; no split mid-word unless the
input has no boundary at all. budget = 71.2s * 0.75 * 14 c/s ≈ 747 (75% of cap for prosody).
```
### FN tts_stream_long (DEC-10 orchestrator — concatenate per-chunk synthesis)
```
tts_stream_long(text, *, voice, dials, client, url=ZONOS_TTS_URL, budget=_TTS_CHUNK_CHAR_BUDGET) -> AsyncIterator[bytes]
# chunk_text(text) then synth each chunk with tts_stream (identical voice+dials+language). Emit chunk 1
# VERBATIM (WAV header + PCM); chunks 2..N via _pcm_after_header (strip up to+incl the `data` id+size) so
# the browser decodes ONE continuous int16-PCM stream after a single leading header (infra-ops: one header).
error: the pivot is `yielded_any`, NOT the chunk index. A failure BEFORE the first byte (an OPEN failure)
propagates as TtsUnavailable -> endpoint peek -> 503 (nothing committed yet). A failure AFTER bytes
have streamed — a MID-STREAM drop on chunk 0 OR a later chunk, past the committed 200 — degrades:
drop the tail, keep what played, write a `tts_degrade` stderr line, RETURN (never raise into the
committed StreamingResponse). httpx wraps aiter_bytes in `except RequestError`, so a mid-stream
drop arrives as TtsUnavailable, not a clean generator end — the yielded_any gate is what keeps a
chunk-0 mid-stream drop from raising into the 200.
```
### FN pad_to_dials
@@ -153,22 +219,33 @@ pad_to_dials(pad: PadState | None) -> EmotionDials
- none/malformed pad -> neutral dials (emotion_enabled=False) [degrade-not-crash].
- emotion_valence = clamp(pad.pleasure, -1, 1); emotion_arousal = clamp(pad.arousal, -1, 1).
- emotion_enabled = True; emotion_strength from a fixed default (tunable).
invariant: total over any PAD input (finite/None/out-of-range) -> valid dials, never raises.
invariant: total over any PAD the DECLARED surface produces (a PadState with float axes /
None / out-of-range / NaN / inf / a non-PadState object) -> valid dials, never raises.
(A PadState carrying NON-float axes is a type violation no call site constructs — the
endpoint coerces via PadState.from_obj; not defended inside pad_to_dials.)
```
### FN tts_endpoint (server.py, GET /api/tts)
### FN tts_endpoint (server.py, POST /api/tts — DEC-10a)
```
GET /api/tts?text=&agent_id=&p=&a= -> audio/wav (chunked StreamingResponse)
# GET (not POST) so a browser <audio src> plays it progressively (DEC-2 streaming). Params ride the query
# string; text is capped ~2000 chars and truncated at a word boundary (URL-safe + bounds the shared-3090 hold).
POST /api/tts {text, agent_id?, p?, a?} -> audio/wav (chunked StreamingResponse)
# POST (not GET) so an arbitrarily long turn rides the body, not a length-capped URL (DEC-10). The server
# chunk-and-concatenates under the 71.2s/call cap (tts_stream_long). text capped 8000 chars, word-boundary
# truncated (a shared-3090 hold bound; the transcript still shows the full text). ALL of text/agent_id/p/a
# are untrusted open-world body fields — each degrades, never 500s (INV-TTS-4).
steps:
- missing text -> 400. resolve voice (per-character map -> "donut"; default Cora).
- dials = pad_to_dials(PadState(p, a)) from the p/a query floats (DEC-7, BROWSER-SENT live PAD); malformed
p/a -> neutral read, never a 500.
- acquire the serialize lock (DEC-5, one stream at a time on the shared 3090); open tts_stream and PEEK the
first chunk so a bad gateway OPEN surfaces as 503 (INV-TTS-4) before committing a 200.
- return StreamingResponse piping the chunks; the generator's finally releases the lock + closes the client
(incl. the browser-abort path: a new turn's <audio> load() drops the GET).
- bad JSON / non-str text -> 400. Scrub lone surrogates from text (else httpx's utf-8 encode of the gateway
body 500s); if the scrubbed text is blank after strip -> 400. word-boundary truncate to 8000 (with a
mid-word HARD-CUT fallback when the last space sits at index <= limit//2).
- resolve voice: per-character map -> "donut", default Cora; a NON-str agent_id (unhashable) -> default voice.
- dials = pad_to_dials(PadState.from_obj({pleasure:p, arousal:a})) — from_obj hardens the parse (a huge-int
OverflowError / non-numeric / missing axis -> neutral read), never a 500.
- acquire the serialize lock (DEC-5, one stream at a time on the shared 3090); open tts_stream_long and PEEK
the first byte: a bad gateway OPEN on chunk 1 -> 503, and a 200 whose first bytes are NOT a RIFF header
-> 503 too (a mislabeled non-WAV body would decode as garbage) — both BEFORE committing a 200 (INV-TTS-4).
Any OTHER escape during the peek (CancelledError, httpx.InvalidURL) releases the lock+client, then propagates.
- return StreamingResponse piping the concatenated chunks; the generator's finally releases the lock +
closes the client (incl. the browser-abort path: a new turn's fetch() drops the POST). httpx.Timeout is
connect=10 / read=120 / write=10 / pool=10 (read=120 per infra-ops: a near-cap chunk can render slowly).
```
### FN pin_kb_context (kb_bridge.py — RETIRE-READY, INV-KB-1)
@@ -206,10 +283,11 @@ pin_kb_context(question: str, agent_id: str | None, *, client) -> list[dict] #
on SSE `done`:
if !ttsEnabled(): return # INV-TTS-2
cancelTts() # INV-TTS-3: abort fetch + stop scheduled nodes
fetch("/api/tts?text=&agent_id=&p=&a=") -> reader # chunked stream (text sliced to the 2000 cap)
loop: read chunk -> skip WAV header up to the data chunk -> int16 LE PCM -> Float32 -> AudioBuffer ->
BufferSource.start(playAt) scheduled GAPLESSLY -> playAt += buf.duration # progressive, TTFA ~0.5s
first scheduled node -> "▶ voiced"; any failure -> ticker + skip (INV-TTS-4)
POST /api/tts {text (sliced to the 8000 cap), agent_id?, p?, a?} -> reader # DEC-10a: POST body, not a GET URL
loop: read chunk -> skip ONE WAV header up to the data chunk (bounded 64KiB) -> int16 LE PCM -> Float32 ->
AudioBuffer -> BufferSource.start(playAt) GAPLESSLY -> playAt += buf.duration # progressive, TTFA ~0.5s
first scheduled node -> "▶ voiced". HARD failure (non-OK HTTP, or 64KiB with no WAV header) -> ticker + skip;
ABORT/cancel (INV-TTS-3 new-turn) + bare network error -> SILENT skip (INV-TTS-4, cancel is not a failure)
WHY Web Audio, not <audio src>: Safari/WebKit REFUSES a streaming 0xFFFFFFFF-length WAV via <audio src>
(NotSupportedError — it can't compute duration/seek), which was the operator's live failure. Decoding the raw