vh/ratatoskr
1
0
Fork 0
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity

feat(tts): migrate off Zonos to chatterbox-fast; drop affect, hold English

Browse Source 
Repoint the TTS client from the Zonos gateway (:8890 /v1/audio/speech) to
chatterbox-fast (:8197 /tts — bespoke non-OpenAI {text,voice,format,stream}
schema, no auth, 24kHz, infra-ops-verified). tts.py stays the single swap seam.

Dropped, no backward-compat (pre-v1):
- Affect (DEC-7): the Turbo checkpoint has no emotion knob, so PadState,
  EmotionDials, pad_to_dials, the /api/tts p/a fields, and the browser pad
  argument are deleted. Voice is now flat.
- Client-side chunking (DEC-10): chatterbox has no per-synth cap and chunks
  internally, so chunk_text/tts_stream_long/_pcm_after_header are deleted; a
  single tts_stream call voices a whole turn, the mid-stream yielded_any degrade
  folded into it.
- Language pin (DEC-9): no language field; re-purposed to sampling curbs (below).

Fixed / added:
- Browser Web Audio sample rate 44100 -> 24000 (the chatterbox rate).
- Default voice Cora -> glados_25s; donut registered lowercase at /refs/donut.wav.
- English-drift curb: Turbo is multilingual-leaky and wanders off English on a
  long generation (the gateway scheduler ratchets chunk size unbounded). Tighten
  sampling in gateway_body: top_k 1000->80, top_p 0.95->0.85, temperature
  0.8->0.5. These reduce drift probability; the guaranteed fix is a server-side
  max-chunk cap (infra-ops, greenlit).
- OOM guard (DEC-9a): a long generation can OOM the shared 3090, returning 200
  with a 0-byte body; /api/tts surfaces an empty 200 as 503 rather than
  committing silent audio.

Contract donut_voiced_interview.contract.md amended: migration banner, DEC-1/3/8
amended, DEC-7/9/10 retired with historical notes, DEC-9a added.

Tests rewritten to the new wire; 520 green. Live-smoked against the gateway
(24kHz synth + endpoint proxy + web console). persistent-memory.md committed
alongside (commit-along).
This commit is contained in:
vh
2026-08-07 10:23:13 -07:00
parent 2cc670e4a1
commit 19b499ab50
8 changed files with 393 additions and 760 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/contracts/donut_voiced_interview.contract.md
+145 -66
View File
@@ -3,9 +3,10 @@ contract_version: "2.1"
module: "ratatoskr.web.tts_kb"
purpose: >
A voiced, corpus-grounded Tier-3 interview character in the ratatoskr web
console. Two capabilities plus one character: (a) auto-TTS via the Zonos
gateway, spoken on SSE `done`, with emotion driven by the character's LIVE
PAD affect; (b) a consumer-side KB-retrieval + `memory_context` pinning
console. Two capabilities plus one character: (a) auto-TTS via the
chatterbox-fast gateway, spoken on SSE `done` (migrated off Zonos 2026-08-07;
no affect modulation — chatterbox Turbo has no emotion knob); (b) a
consumer-side KB-retrieval + `memory_context` pinning
BRIDGE that grounds the character's recall in the ingested corpus while she
stays in-voice; (c) Princess Donut (Dungeon Crawler Carl) as the first
instance. The bridge is a deliberate, retire-ready workaround for a Worldtree
@@ -21,12 +22,12 @@ touches:
- src/ratatoskr/web/server.py # /api/tts route + the retrieval-pinning seam on the turn path
- src/ratatoskr/web/static/index.html # speak-on-done playback, 🔊 toggle, <audio> sink; turn POST carries agent_id
- src/ratatoskr/web/entrypoint.py # RATATOSKR_TTS_URL override (the tts swap seam)
- src/ratatoskr/tts.py # NEW — Zonos gateway client + PAD->emotion-dial mapping
- src/ratatoskr/tts.py # chatterbox-fast gateway client (was Zonos + PAD->dial; migrated 2026-08-07)
- src/ratatoskr/kb_bridge.py # NEW, RETIRE-READY — consumer-side retrieval + memory_context pinning
- src/ratatoskr/wt.py # stream_turn gains a memory_context passthrough (seam-review: the contract's original touch list undercounted this by one file; the param defaults None so the bridge's RETIREMENT stays inert — deleting kb_bridge.py + the one call-site leaves wt.stream_turn's SDK-parity param harmless)
- docs/characters/donut.md # NEW — Princess Donut persona (content; the tier3 define source)
depends_on:
- "Zonos gateway: POST http://10.100.79.3:8890/v1/audio/speech (infra-ops; WG-internal, no auth; wav; verified 2026-08-02)"
- "chatterbox-fast gateway: POST http://10.100.79.3:8197/tts (infra-ops; WG-internal, no auth; bespoke non-OpenAI schema {text,voice,format,stream}; streaming placeholder-header wav @ 24000 Hz; no per-synth cap; NO affect controls; verified 2026-08-07 against image local/chatterbox-fast:v1)"
- "Worldtree turn stream: memory_context[] passthrough (SDK stream_turn already forwards it verbatim)"
- "Worldtree agents.define (Tier-3) for Donut; Mimir (search_kb) for the out-of-band retrieval consult"
used_by:
@@ -40,6 +41,29 @@ confidence: 0.8
# Contract: Donut voiced interview (auto-TTS + KB-recall bridge)
> **⚠ TTS MIGRATED OFF ZONOS → chatterbox-fast 2026-08-07 (operator-directed).**
> Slice 2's synthesis backend moved from the Zonos gateway (:8890
> `/v1/audio/speech`) to chatterbox-fast (:8197 `/tts`). Three architecture deltas,
> all infra-ops-verified against image `local/chatterbox-fast:v1`:
> - **Affect dropped (DEC-7 RETIRED).** chatterbox serves the Chatterbox TURBO
> checkpoint, which has NO valence/arousal/emotion knob (exaggeration is exposed
> but inert on Turbo). The whole PAD→emotion-dial path — `PadState`,
> `EmotionDials`, `pad_to_dials`, and the browser `p`/`a` body fields — is
> deleted. Voice is now flat (high-quality but unmodulated). Operator's call: if
> live affect ever becomes load-bearing again, Zonos (:8890) remains the only
> fleet TTS with real emotion steering.
> - **Client-side chunking dropped (DEC-10 RETIRED).** chatterbox has no per-synth
> token/duration cap (Zonos capped at 6144 tok / 71.2s) and chunks arbitrary-length
> text internally, so `chunk_text` + `tts_stream_long` + `_pcm_after_header` are
> deleted; a single `tts_stream` call voices a whole turn. The mid-stream degrade
> policy (`yielded_any`) folds INTO `tts_stream`.
> - **Sample rate 44100 → 24000 Hz.** The browser Web Audio decode MUST use 24000
> or the voice plays ~1.8× too fast.
> `/api/tts` stays POST; the streaming placeholder-header WAV shape (DEC-2/DEC-3) and
> the browser Web-Audio PCM decode path (DEC-2) are UNCHANGED except the sample rate.
> The `tts.py` client remains the single swap seam (DEC-1). DEC-7/9/10 below are
> retained as historical record of the Zonos build.
> **⚠ SLICE 3 (KB-recall bridge) RETIRED 2026-08-02.** The `kb_bridge.py` module +
> its single `web/server.py` call-site were deleted per INV-KB-1 when Worldtree #383
> shipped native Tier-3 `reference_knowledge` (v1.0.0b167, live on :8081 + demo).
@@ -58,10 +82,14 @@ each independently shippable. Slice order is chosen for fastest visible result.
## Decisions (DEC)
- **DEC-1 — direct :8890 coupling.** Proxy straight to the Zonos gateway, not the
swappable `ext-tts` LiteLLM alias. Rationale: the emotion dials (the whole
point — affect-driven voice) don't pass through `ext-tts`. Accept the
Zonos coupling; the `tts.py` client is the single swap seam if we ever move.
- **DEC-1 — direct :8197 coupling (amended 2026-08-07).** Proxy straight to the
chatterbox-fast gateway. Original Zonos rationale was the emotion dials (which
`ext-tts` dropped); that rationale is retired with affect (DEC-7). The coupling
STANDS regardless: chatterbox-fast is a bespoke, non-OpenAI `/tts` schema
(`{text,voice,format,stream}`, verified with infra-ops), NOT reachable through the
OpenAI-shaped `ext-tts` LiteLLM alias — so `tts.py` remains the single swap seam,
now translating that bespoke schema. (Considered routing via the generic alias
once affect was dropped; the non-OpenAI wire ruled it out.)
- **DEC-2 — STREAMING, play-as-it-arrives (amended 2026-08-02, operator-directed).**
The gateway ALREADY streams: `POST /v1/audio/speech` relays a chunked int16 WAV
(transfer-encoding: chunked, placeholder 0xFFFFFFFF RIFF/data sizes) as it synthesizes
@@ -75,8 +103,14 @@ each independently shippable. Slice order is chosen for fastest visible result.
(Amended 2026-08-02: the browser path is now Web Audio decoding raw int16 PCM, not
`<audio src>` — Safari/WebKit rejects a 0xFFFFFFFF-length streaming WAV via `<audio
src>` (NotSupportedError); and `/api/tts` is now POST, not GET. See DEC-10.)
- **DEC-3 — wav only.** `response_format:"wav"` (streaming int16 RIFF/WAVE). `mp3`/`opus`
are accepted but silently return mislabeled PCM — never request them.
- **DEC-3 — wav streaming (amended 2026-08-07).** `format:"wav"`, `stream:true` →
a streaming int16 RIFF/WAVE with placeholder (0xFFFFFFFF) RIFF/data sizes, one
header, then s16le PCM to EOF — the shape the browser's one-header-strip decoder
expects. chatterbox also offers `format:"pcm"` (headerless raw s16le, leaner);
kept on `wav` so the streaming shape stays byte-identical to the Zonos path and the
endpoint's RIFF-sniff (non-WAV-200 → 503) and the browser header-strip both stand
unchanged. (pcm is a deferred lean-up — it would drop the header-strip + the
RIFF-sniff, net a few lines.)
- **DEC-4 — server-side proxy.** Browser → `/api/tts` (nh3-dev) → gateway. The
irv-ml1 host/URL never reaches the client (INV-TTS-1). No key exists, so
INV-003 is trivially satisfied, but the proxy still stands (browser can't
@@ -103,26 +137,64 @@ each independently shippable. Slice order is chosen for fastest visible result.
a wing-scope mismatch — tool scoped to the failed `fiction` wing — or a confidence
threshold rejecting the weak ~0.03 hits). Escalated to worldtree-dev. Until fixed,
Donut recalls from her own character knowledge (degrades in-voice per the persona).
- **DEC-7 — affect-driven emotion.** Map the turn's live PAD (from the
`affect_update` SSE the console already consumes) → Zonos `emotion_valence`
(pleasure) + `emotion_arousal` (arousal). This reframes the feature as voice
OBSERVABILITY (hear the affect the persona pane shows), not chat-app TTS.
- **DEC-8 — voice: custom "donut" is REGISTERED (superseded the preset-first plan).**
The original plan was a theatrical Zonos preset first (Miranda/Penny/Emmie), custom
"donut" later. But infra-ops registered + verified the custom `voice:"donut"` before
slice-2 build (GET /v1/voices returns Donut; case-folded), so `_TTS_VOICE_MAP` maps
`ratatoskr:donut → "donut"` directly — no interim preset. Non-interview agents still
fall to the gateway default (Cora). (Amended 2026-08-02 per heid-code-review: 3 arms
flagged the code as DEC-8 drift; a live gateway read INVERTED the remedy — the code is
correct, DEC-8's "preset now" was stale.)
- **DEC-9 — pin English (added 2026-08-02, operator-directed).** Zonos is multilingual;
with no `language` field it drifts into other-language phonemes / gibberish on names,
numerics, and long spans (operator report). `gateway_body` pins `language:"en-us"` on
every call (gateway accepts it, verified live). The persona's dialogue-only rewrite
(`docs/characters/donut.md`) removes the other gibberish vector — asterisk RP action
beats were being voiced verbatim.
- **DEC-10 — long-form chunk-and-concatenate (added 2026-08-02, operator-directed;
infra-ops recipe 01KZ1FKX…).** The Zonos model hard-caps ONE synthesis at
- **DEC-7 — affect-driven emotion. RETIRED 2026-08-07 (chatterbox migration).**
chatterbox Turbo has no valence/arousal/emotion control (infra-ops-verified:
exaggeration is exposed but inert; cfg_weight/min_p not exposed; only generic
sampling knobs move output, and those change timbre/variance not emotion). There
is no coupling point for a live-PAD driver, so the entire path is deleted —
`PadState`, `EmotionDials`, `pad_to_dials`, the `/api/tts` `p`/`a` body fields, and
the browser's `pad` argument. Voice is now flat. Retained below as historical
record of the Zonos build. (Original: map live PAD from the `affect_update` SSE →
Zonos `emotion_valence`/`emotion_arousal`, reframing the feature as voice
OBSERVABILITY. The observability framing dies with the knob.)
- **DEC-8 — voice: custom "donut" is REGISTERED (amended 2026-08-07 for chatterbox).**
chatterbox voices are `*.wav` reference clips in `/refs` (GET /voices lists the
stems). infra-ops registered `/refs/donut.wav` (the same reference clip behind the
Zonos Donut voice) at operator direction, so `_TTS_VOICE_MAP` maps
`ratatoskr:donut → "donut"` directly. NOTE the case: chatterbox wants lowercase
`"donut"` (Zonos used `"Donut"`). Non-interview agents fall to the chatterbox
default `"glados_25s"` (was Zonos `"Cora"`, which does not exist on chatterbox).
- **DEC-9 — hold English (amended 2026-08-07; the "nothing to drift" call was WRONG).**
The Zonos `language:"en-us"` pin is dropped — chatterbox has no `language` field. BUT the
initial "English-only, nothing to drift" rationale was FALSIFIED by an operator report the
same day: the Turbo checkpoint drifts into German partway through a long turn. infra-ops's
authoritative root-cause (source-read, thread 01KZEDMJ…): **Turbo has latent multilingual
capacity that leaks**, and the drift is **length-driven** — the gateway's adaptive scheduler
ratchets chunk size upward with NO cap, so a long turn collapses into essentially ONE long
generation after the first 2-3 sentences, and the sampler wanders off English on that single
long decode (generation state DOES reset per chunk — cross-chunk carry is not the mechanism;
it's the unbounded per-chunk length). Two-layer response:
- **Sampling curbs (shipped, no redeploy) — REDUCE drift probability, do NOT guarantee it.**
`gateway_body` tightens below the gateway defaults: `_TTS_TOP_K = 80` (from 1000 — the
highest-leverage knob; the huge default admits off-language tokens), `_TTS_TOP_P = 0.85`
(from 0.95), `_TTS_TEMPERATURE = 0.5` (from 0.8). Escalation if still drifting: temp
0.3-0.4, rep_penalty 1.2→1.3.
- **Length-bounding (the ROBUST fix) — GUARANTEES English by keeping each generation short.**
Two paths, operator's call: (a) return to short CLIENT-side chunking (~1-2 sentences per
/tts call, each a fresh re-anchored generation; partially reverses DEC-10; works today, no
redeploy; loses the gateway's seamless internal streaming); (b) infra-ops adds a SERVER-side
max-chunk cap to the scheduler (keeps seamless streaming AND holds English; needs a
chatterbox-fast redeploy + operator greenlight on the card-shared fleet service — then the
consumer reverts to sending full text).
The persona's dialogue-only rewrite in `docs/characters/donut.md` still stands (removes the
asterisk-RP-voiced-verbatim vector regardless of engine). (Original Zonos DEC-9 below.)
- **DEC-9a — OOM on long single generations → empty 200 (infra-ops 2026-08-07).** chatterbox-fast
shares the RTX 3090 with Zonos2 (~1 GB headroom). A long single generation can OOM the card;
the gateway then returns HTTP **200 with a 0-byte body** (not a 5xx). `tts_endpoint` treats an
empty 200 body as a synthesis failure → 503 (INV-TTS-4 visible skip), never a silent empty
audio/wav stream. Length-bounding (DEC-9 above) fixes the OOM too — small chunks don't OOM.
- **DEC-10 — long-form chunk-and-concatenate. RETIRED 2026-08-07 (chatterbox
migration).** chatterbox-fast has NO per-synth token/duration cap (Zonos capped at
6144 tok / 71.2s) and chunks arbitrary-length text INTERNALLY via its adaptive
scheduler, streaming seamlessly — so client-side chunk-and-concatenate is deleted:
`chunk_text`, `tts_stream_long`, `_pcm_after_header`, and the `_TTS_CHUNK_CHAR_BUDGET`
constants are gone; a single `tts_stream` call voices a whole turn. `/api/tts` STAYS
POST (DEC-10a) — a long turn still rides the body, not a length-capped URL — and the
8000-char outer cap stays as a shared-GPU hold ceiling (no longer a chunk-count
bound). The `yielded_any` mid-stream degrade that lived in `tts_stream_long` folds
into `tts_stream`. Retained below as historical record of the Zonos build. (Original:)
The Zonos model hard-caps ONE synthesis at
`max_tokens=6144` = 71.2s of audio (6144 / 86.3 Hz codec frame rate; the gateway 400s
above 6144 — an architectural sequence limit, unraisable). So a turn longer than ~71s
truncated mid-stream. Fix: split the text client-side and concatenate the PCM
@@ -140,8 +212,8 @@ each independently shippable. Slice order is chosen for fastest visible result.
## Invariants
- **INV-TTS-1 [hard]** — the Zonos gateway host/URL never reaches the browser;
all synthesis goes through `/api/tts`.
- **INV-TTS-1 [hard]** — the TTS gateway host/URL (chatterbox-fast :8197) never
reaches the browser; all synthesis goes through `/api/tts`.
- **INV-TTS-2 [hard]** — TTS is opt-in: a 🔊 toggle (default OFF), persisted to
localStorage (mirrors the theme/cot-toggle pattern). No speech without it.
- **INV-TTS-3 [hard]** — one synth in flight at a time; a new turn cancels the
@@ -168,22 +240,27 @@ each independently shippable. Slice order is chosen for fastest visible result.
## FN blocks
### FN tts_stream (the per-CHUNK primitive — DEC-2 streaming; wrapped by tts_stream_long)
### FN tts_stream (the sole synthesis primitive — DEC-2 streaming; amended 2026-08-07)
```
tts_stream(text, *, voice, dials, client: httpx.AsyncClient, url=ZONOS_TTS_URL) -> AsyncIterator[bytes]
# Open the gateway's CHUNKED stream (client.stream("POST", url, json=gateway_body(...))) and YIELD wav
# chunks as they synthesize. Pass through verbatim — never buffer, never rewrite the placeholder header.
# gateway_body(text, voice, dials) = {input, voice, response_format:"wav", language:"en-us" (DEC-9),
# **dials.to_body()}.
precondition: text non-empty. Voice membership in /v1/voices is GATEWAY-enforced, not client-asserted.
tts_stream(text, *, voice, client: httpx.AsyncClient, url=CHATTERBOX_TTS_URL) -> AsyncIterator[bytes]
# Open the gateway's CHUNKED stream (client.stream("POST", url, json=gateway_body(text, voice))) and
# YIELD wav chunks as they synthesize. Pass through verbatim — never buffer, never rewrite the placeholder
# header. chatterbox chunks arbitrary-length text INTERNALLY (no per-synth cap, DEC-10 RETIRED), so this
# SINGLE call voices a whole turn — no client-side chunk-and-concatenate wrapper.
# gateway_body(text, voice) = {text, voice, format:"wav", stream:true, temperature:_TTS_TEMPERATURE}.
# temperature < gateway-default 0.8 holds English across a long turn (DEC-9). NO dials, NO language (RETIRED).
precondition: text non-empty. Voice membership in GET /voices is GATEWAY-enforced, not client-asserted.
postcondition: yields the gateway's chunked int16 streaming WAV bytes unmodified (0xFFFFFFFF placeholder
sizes intact). This is ONE synthesis (<= 71.2s cap, DEC-10); tts_stream_long stitches many.
error: a non-200 OPEN or connect/transport failure -> TtsUnavailable BEFORE the first chunk (so the
endpoint can still return 503); a mid-stream drop just ends the generator.
invariant: response_format is ALWAYS "wav" (DEC-3); never mp3/opus. language ALWAYS "en-us" (DEC-9).
sizes intact), one leading header then s16le PCM @ 24000 Hz to EOF.
error (the yielded_any pivot, folded in from the retired tts_stream_long):
- a non-200 OPEN or a connect/transport failure BEFORE the first byte -> TtsUnavailable (so the endpoint
peek can still return 503; nothing committed yet).
- a transport drop AFTER >= 1 byte has streamed (the 200 is committed) -> DEGRADE: write a `tts_degrade`
stderr line, END the generator, keep what played. NEVER raise into the committed StreamingResponse.
invariant: format is ALWAYS "wav" (DEC-3); never mp3/opus/pcm from this seam.
```
### FN chunk_text (DEC-10 long-form splitting; pure)
### FN chunk_text (DEC-10 long-form splitting; pure) — RETIRED 2026-08-07 (chatterbox chunks internally; deleted). Historical:
```
chunk_text(text, budget=_TTS_CHUNK_CHAR_BUDGET) -> list[str]
# Split into synthesis chunks each <= budget chars. Paragraph-first (seams on blank lines), greedy pack.
@@ -196,7 +273,7 @@ chunk_text(text, budget=_TTS_CHUNK_CHAR_BUDGET) -> list[str]
input has no boundary at all. budget = 71.2s * 0.75 * 14 c/s ≈ 747 (75% of cap for prosody).
```
### FN tts_stream_long (DEC-10 orchestrator — concatenate per-chunk synthesis)
### FN tts_stream_long (DEC-10 orchestrator) — RETIRED 2026-08-07 (no per-synth cap; deleted, its yielded_any degrade folded into tts_stream). Historical:
```
tts_stream_long(text, *, voice, dials, client, url=ZONOS_TTS_URL, budget=_TTS_CHUNK_CHAR_BUDGET) -> AsyncIterator[bytes]
# chunk_text(text) then synth each chunk with tts_stream (identical voice+dials+language). Emit chunk 1
@@ -211,7 +288,7 @@ tts_stream_long(text, *, voice, dials, client, url=ZONOS_TTS_URL, budget=_TTS_CH
chunk-0 mid-stream drop from raising into the 200.
```
### FN pad_to_dials
### FN pad_to_dials — RETIRED 2026-08-07 (DEC-7 affect dropped; PadState/EmotionDials/pad_to_dials all deleted). Historical:
```
pad_to_dials(pad: PadState | None) -> EmotionDials
# Map live PAD -> Zonos emotion dials (DEC-7).
@@ -225,27 +302,28 @@ pad_to_dials(pad: PadState | None) -> EmotionDials
endpoint coerces via PadState.from_obj; not defended inside pad_to_dials.)
```
### FN tts_endpoint (server.py, POST /api/tts — DEC-10a)
### FN tts_endpoint (server.py, POST /api/tts — DEC-10a; amended 2026-08-07)
```
POST /api/tts {text, agent_id?, p?, a?} -> audio/wav (chunked StreamingResponse)
# POST (not GET) so an arbitrarily long turn rides the body, not a length-capped URL (DEC-10). The server
# chunk-and-concatenates under the 71.2s/call cap (tts_stream_long). text capped 8000 chars, word-boundary
# truncated (a shared-3090 hold bound; the transcript still shows the full text). ALL of text/agent_id/p/a
# are untrusted open-world body fields — each degrades, never 500s (INV-TTS-4).
POST /api/tts {text, agent_id?} -> audio/wav (chunked StreamingResponse)
# POST (not GET) so an arbitrarily long turn rides the body, not a length-capped URL. The gateway chunks
# arbitrary-length text internally (DEC-10 RETIRED — no client concat); a single tts_stream call proxies it.
# text capped 8000 chars, word-boundary truncated (a shared-GPU hold bound; the transcript still shows the
# full text). text/agent_id are untrusted open-world body fields — each degrades, never 500s (INV-TTS-4).
# (The `p`/`a` PAD body fields are GONE — DEC-7 affect retired.)
steps:
- bad JSON / non-str text -> 400. Scrub lone surrogates from text (else httpx's utf-8 encode of the gateway
body 500s); if the scrubbed text is blank after strip -> 400. word-boundary truncate to 8000 (with a
mid-word HARD-CUT fallback when the last space sits at index <= limit//2).
- resolve voice: per-character map -> "donut", default Cora; a NON-str agent_id (unhashable) -> default voice.
- dials = pad_to_dials(PadState.from_obj({pleasure:p, arousal:a})) — from_obj hardens the parse (a huge-int
OverflowError / non-numeric / missing axis -> neutral read), never a 500.
- acquire the serialize lock (DEC-5, one stream at a time on the shared 3090); open tts_stream_long and PEEK
the first byte: a bad gateway OPEN on chunk 1 -> 503, and a 200 whose first bytes are NOT a RIFF header
-> 503 too (a mislabeled non-WAV body would decode as garbage) — both BEFORE committing a 200 (INV-TTS-4).
Any OTHER escape during the peek (CancelledError, httpx.InvalidURL) releases the lock+client, then propagates.
- return StreamingResponse piping the concatenated chunks; the generator's finally releases the lock +
closes the client (incl. the browser-abort path: a new turn's fetch() drops the POST). httpx.Timeout is
connect=10 / read=120 / write=10 / pool=10 (read=120 per infra-ops: a near-cap chunk can render slowly).
- resolve voice: per-character map -> "donut", default "glados_25s"; a NON-str agent_id (unhashable) -> default.
- acquire the serialize lock (DEC-5, one stream at a time on the shared GPU); open tts_stream and PEEK the
first byte: a bad gateway OPEN -> 503; an EMPTY 200 body (no bytes — an OOM synth, DEC-9a) -> 503; and a
200 whose first bytes are NOT a RIFF header -> 503 too (a mislabeled non-WAV body would decode as
garbage) — all BEFORE committing a 200 (INV-TTS-4). Any OTHER escape during the peek (CancelledError,
httpx.InvalidURL) releases the lock+client, then propagates.
- return StreamingResponse piping tts_stream; the generator's finally releases the lock + closes the client
(incl. the browser-abort path: a new turn's fetch() drops the POST) and, on a committed mid-stream drop,
tts_stream degrades internally (ends the generator, no raise). httpx.Timeout is connect=10 / read=120 /
write=10 / pool=10 (read=120 per infra-ops: a long synth can render slowly).
```
### FN pin_kb_context (kb_bridge.py — RETIRE-READY, INV-KB-1)
@@ -278,14 +356,15 @@ pin_kb_context(question: str, agent_id: str | None, *, client) -> list[dict] #
searches in-voice natively.
```
### FN client: speakOnDone (index.html — Web Audio STREAMING, DEC-2)
### FN client: speakOnDone (index.html — Web Audio STREAMING, DEC-2; amended 2026-08-07)
```
on SSE `done`:
if !ttsEnabled(): return # INV-TTS-2
cancelTts() # INV-TTS-3: abort fetch + stop scheduled nodes
POST /api/tts {text (sliced to the 8000 cap), agent_id?, p?, a?} -> reader # DEC-10a: POST body, not a GET URL
POST /api/tts {text (sliced to the 8000 cap), agent_id?} -> reader # DEC-10a: POST body. NO p/a (DEC-7 retired).
loop: read chunk -> skip ONE WAV header up to the data chunk (bounded 64KiB) -> int16 LE PCM -> Float32 ->
AudioBuffer -> BufferSource.start(playAt) GAPLESSLY -> playAt += buf.duration # progressive, TTFA ~0.5s
AudioBuffer(sampleRate=24000) -> BufferSource.start(playAt) GAPLESSLY -> playAt += buf.duration
# SR = 24000 (chatterbox; was 44100 for Zonos — MUST match or the voice plays ~1.8x too fast). TTFA ~0.5s.
first scheduled node -> "▶ voiced". HARD failure (non-OK HTTP, or 64KiB with no WAV header) -> ticker + skip;
ABORT/cancel (INV-TTS-3 new-turn) + bare network error -> SILENT skip (INV-TTS-4, cancel is not a failure)
persistent-memory.md
+29 -2
View File
@@ -1,6 +1,6 @@
# Persistent memory — ratatoskr
_Last updated: 2026-08-03_
_Last updated: 2026-08-07_
> **Always check for `/tmp/ratatoskr-dev-handoff.md`** — if it exists and its
> `Written:` stamp is under an hour old, read it (it carries the in-flight
@@ -44,7 +44,27 @@ upstream API key stays server-side (INV-003).
## Current state / in-flight
_As of 2026-08-02:_
_As of 2026-08-07:_
**✅ TTS MIGRATED off Zonos → chatterbox-fast (this session; COMMITTED, not pushed).** `tts.py` repointed from the Zonos gateway (:8890 `/v1/audio/speech`) to **chatterbox-fast** (`http://10.100.79.3:8197/tts` — bespoke non-OpenAI `{text,voice,format,stream}` schema, no auth, 24kHz, infra-ops-verified against image `local/chatterbox-fast:v1`). Three subsystems DELETED: (1) **affect** — Turbo has no emotion knob, so `PadState`/`EmotionDials`/`pad_to_dials` + `/api/tts` `p`/`a` fields + browser `pad` arg are gone (DEC-7 retired; operator-directed "drop it for chatterbox"); (2) **client-side chunking** — no per-synth cap (gateway chunks internally), so `chunk_text`/`tts_stream_long`/`_pcm_after_header` gone, one `tts_stream` call voices a whole turn (DEC-10 retired, the mid-stream `yielded_any` degrade folded into `tts_stream`); (3) **language pin** — English-only, no `language` field (DEC-9 re-purposed, below). **Browser SR 44100→24000** (load-bearing correctness fix). Default voice `Cora`→`glados_25s`; `donut` registered lowercase at `/refs/donut.wav`. 518 suite green, live-smoked (real 24kHz synth + endpoint proxy + bounced `ratatoskr-web`). Contract `donut_voiced_interview.contract.md` amended (migration banner; DEC-1/3/8 amended; DEC-7/9/10 retired w/ historical notes). `tts.py` is the single swap seam; `RATATOSKR_TTS_URL` overrides (no env pin, uses the code default).
**⚠️ ENGLISH-DRIFT curb + PENDING cap verify (this session).** Operator: Donut "swaps to German halfway through." Root cause (infra-ops source-read, thread `01KZEDMJ…`): **Turbo is multilingual-leaky AND the drift is LENGTH-driven** — the gateway's adaptive scheduler ratchets chunk size UPWARD with no cap, so a long turn collapses into one giant generation that (a) wanders off English and (b) OOMs the shared 3090 → HTTP **200 with a 0-byte body**. SHIPPED consumer-side (no redeploy): sampling curbs in `gateway_body` — `top_k 1000→80` (the prime knob, never pulled before), `top_p 0.95→0.85`, `temp 0.8→0.5` (`_TTS_TOP_K`/`_TTS_TOP_P`/`_TTS_TEMPERATURE`, DEC-9 re-purposed); + `/api/tts` empty-200→503 OOM guard (DEC-9a). Knobs REDUCE drift probability, do NOT GUARANTEE it on a long single generation. **REAL FIX = (b) infra-ops SERVER-SIDE max-chunk cap — GREENLIT by operator 2026-08-07; infra-ops implementing + redeploying chatterbox-fast.** Went straight to (b) (never added interim client chunking), so ratatoskr is ALREADY sending full text — nothing to revert. **PENDING (next session), on infra-ops's "cap deployed" ping: (1) verify a long donut turn holds English end-to-end; (2) optionally relax the sampling knobs toward defaults now that length is bounded server-side.**
**✅ Donut interview character — voice + memory + honesty + query-formulation ALL DONE, on origin.** Deleted+redefined on Worldtree's **canonical recall branch (personal WT v1.0.0b183)**. Persona (`docs/characters/donut.md`) carries three layered behaviors, all committed+pushed: (1) **anti-fabrication** — her memory IS what `reference_knowledge` returns; LOW/no-on-target → deflect in character, never confabulate; (2) **near-miss LEAD rule** (`37b67a5`) — a weak-but-named candidate → offer the NAME the tool returned ("do you mean The Juicer?"), never invented detail; (3) **expand-don't-distill query formulation** (`2cc670e`) — pass the FULL descriptive phrase, enrich toward entity vocab, never boil to bare keywords. Push live edits: `python -m ratatoskr.tier3 patch ratatoskr:donut --system-prompt "$(awk '/^## System prompt/{f=1;next} f' docs/characters/donut.md)"`.
**✅ #393 (descriptive-query subject binding) — FULL LIFECYCLE DONE + CLOSED this session.** ratatoskr's b172 evidence → filed as **Worldtree #393** → our **`docs/diagnostics/descriptive_query_binding.py` = the canonical fixture** on the issue → persona-expand lever landed (roid-rage **4/10→9/10** binds, mis-binds **1/6→0/10**, anti-fab held) → tool-side directive (b182/b183 `reference_knowledge` query-desc + bge reranker) **stabilizes not lifts** (persona owns the niche) → **closed**. Two-mechanism split: cross-wing ranking dilution (mimir/all-wing) vs fiction-scope subject-selection (Donut fiction-only). Cross-instrument method corrected three of my confident mid-arc reads via the fixture (parametric-leak→mis-grounding, deployment-gap→wing-scope, subject-selection→reformulation-distillation).
**✅ Harness fix (`2111b1e`) — `fiction_wing_probe.py` fresh-session-per-query (WT #391 workaround) + NFKC/quote-fold on_target.** Surfaced Worldtree **#391** (reused mimir session → empty after turn 1) — CLOSED upstream (b171/b172 first-iteration tool_choice forcing). Fixture q=capture + wing-scope fold in `4f4b5ad`.
**⏳ order_by=chapter flag → FILED as Worldtree #397 (DEFERRED to next session's contract pass; our fixture is the measurement instrument).** New axis vs #393: narrative/temporal queries ("what was your first encounter in the dungeon") fail — `reference_knowledge` sorts by RELEVANCE not chronology, and Donut can't reorder her own results (native recall, kb_bridge retired WT #383). `provenance.chapter` IS on every chunk but the consumer never holds the result set. Operator ruled a `order_by=chapter` tool flag the clean fix (upstream OK for this one); worldtree-dev accepted → **WT #397** runs the full contract pass next session (design fork on the decision list: sort-top-k vs earliest-matching fetch, cross-generation chapter comparability, non-fiction fallback, tool-description discoverability). Expect the contract to reach our review threads. If it ships → point the fixture at it + add a chronological-query persona instruction; if the persona-only fallback is ever needed → honest-floor redirect (deflect-with-a-lead on ungroundable temporal Qs).
**🔧 TTS "unavailable" FIXED — stale web server.** `ratatoskr-web` (started Aug 2) predated the POST `/api/tts` endpoint → 405 → UI read unavailable. Restarted (new PID, detached), POST /api/tts verified 200 + WAV; gateway `http://10.100.79.3:8890` healthy. **DURABILITY GAP:** `ratatoskr-web` is a bare `nohup` process, NO supervisor → will re-stale on reboot/crash. A systemd user unit was offered — NOT done, operator's call.
**Substrate (this session):** branch `main`, **HEAD `2cc670e` PUSHED**; 5 commits all on origin — `2111b1e` (harness fresh-session+quote-fold), `37b67a5` (near-miss LEAD), `6c83a3b` (#393 fixture), `4f4b5ad` (fixture q=capture+wing-scope), `2cc670e` (expand-don't-distill). Personal WT **v1.0.0b183** (`http://10.250.50.152:8081`, both instances). Web on `:8765` (restarted). New auto-memory `feedback_review_provenance_stays_out_of_artifacts` (Heid provenance-leak note — never embed cold-review arm-names/vote-counts in artifacts; forward habit, operator declined retro-sweep).
---
_Prior in-flight (2026-08-02, historical — superseded by the block above):_
**✅ Donut voiced-interview + long-form TTS COMPLETE + PUSHED (`d59f907`).** The whole arc — auto-TTS, KB-bridge →
native `reference_knowledge`, then this session's delivery fixes — is on origin. This session (`d59f907`, 546 green,
@@ -177,6 +197,13 @@ relational-dynamics verify (bind `--bifrost-url :8392`); WT #356 resume-durabili
Chronological log of decisions with `[YYYY-MM-DD]` prefix. One line per
decision. Captures rationale that won't be obvious from code alone.
- `[2026-08-07]` **TTS migrated Zonos→chatterbox-fast (`:8197` bespoke schema); affect DROPPED (Turbo has no emotion knob, operator "drop it for chatterbox"), client-chunking DROPPED (no per-synth cap), language pin DROPPED, browser SR 44100→24000.** English drift ("swaps to German halfway") root-caused by infra-ops as Turbo multilingual-leak + LENGTH-driven scheduler ratchet (+ shared-3090 OOM → empty 200); curbed via `top_k 1000→80`/`top_p 0.95→0.85`/`temp 0.8→0.5` + `/api/tts` empty-200→503 guard. **Real fix = infra-ops server-side max-chunk cap, GREENLIT by operator — awaiting redeploy ping (then verify long-turn English + maybe relax knobs).** Contract `donut_voiced_interview.contract.md` amended. Committed, not pushed.
- `[2026-08-07]` **order_by=chapter tool flag → FILED as Worldtree #397 (DEFERRED to next session's contract pass).** Narrative/temporal-query gap ("first encounter in the dungeon"): `reference_knowledge` sorts by relevance not chronology; `provenance.chapter` is on every chunk but the consumer can't reorder native results (kb_bridge retired). Operator ruled the upstream sort flag the clean fix; worldtree-dev accepted, our fixture is the measurement instrument. Tracked at **Worldtree #397** (+ althing thread `01KZED2T3XHJ2WMS5NCYK42W6R`).
- `[2026-08-07]` **#393 (descriptive-query subject binding) CLOSED — persona-expand lever the win (4/10→9/10), tool directive the fleet floor.** `docs/diagnostics/descriptive_query_binding.py` is the canonical #393 fixture; two-mechanism split (cross-wing dilution vs fiction-scope selection). Commits `6c83a3b`/`4f4b5ad`/`2cc670e`.
- `[2026-08-07]` **Donut expand-don't-distill persona lever (`2cc670e`)** — keep full descriptive phrase + enrich toward entity vocab, never distill to bare keywords; measured 4/10→9/10 roid-rage binds persona-alone. Composes with near-miss LEAD (`37b67a5`).
- `[2026-08-07]` **TTS "unavailable" = stale web-server process** (predated POST /api/tts → 405); restarted + verified 200+WAV. Durability gap: bare nohup, no supervisor → systemd unit offered, deferred (operator's call).
- `[2026-08-03]` **fiction_wing_probe harness fix (`2111b1e`)** — fresh-session-per-query (WT #391 workaround) + NFKC/quote-fold on_target; surfaced+closed WT #391.
- `[2026-06-19]` **#18 D2 SHIPPED (`v0.17.14`, `39eebd1`) and the full #17+#18 arc PUSHED to origin** → `persistent-memory.d/2026-06-19-18-d2-shipped-v0-17-14-39eebd1-and-the-full-1.md`
- `[2026-06-19]` **bifrost repinned 0.8.0→0.10.0; `affect.fetch` became MANDATORY (strong-or-absent)** → `persistent-memory.d/2026-06-19-bifrost-repinned-0-8-0-0-10-0-affect-fetch-be.md`
src/ratatoskr/tts.py
+80 -309
View File
@@ -1,44 +1,62 @@
"""Zonos-gateway TTS client + PAD→emotion-dial mapping.
"""chatterbox-fast gateway TTS client.
Slice 2 of docs/contracts/donut_voiced_interview.contract.md. This module is the
SINGLE swap seam for voice synthesis: the `/api/tts` route in web/server.py is
its only caller. Direct coupling to the Zonos gateway (DEC-1) buys the emotion
dials that the swappable `ext-tts` LiteLLM alias drops — the whole point is
affect-driven voice (DEC-7). If we ever move off Zonos, this is the swap point.
Migrated off the Zonos gateway 2026-08-07 (operator-directed). chatterbox-fast
(the Chatterbox TURBO checkpoint, irv-ml1 :8197) is a bespoke, non-OpenAI `/tts`
gateway. The migration dropped two whole Zonos-era subsystems:
- the PAD->emotion-dial path (Turbo has NO valence/arousal/emotion knob;
infra-ops-verified), so voice is now flat; and
- the client-side chunk-and-concatenate (chatterbox has no per-synthesis cap
and chunks arbitrary-length text internally), so a single `tts_stream` call
voices a whole turn.
This module is the SINGLE swap seam for voice synthesis: the `/api/tts` route in
web/server.py is its only caller.
Foot-guns (verified live 2026-08-02):
- DEC-3: response_format is ALWAYS "wav". `mp3`/`opus` are accepted but the
gateway silently returns mislabeled PCM (no encoder wired) — never request them.
- Use the gateway :8890, NOT the engine :1920 (rep-penalty bug pads silence).
Foot-guns (infra-ops-verified 2026-08-07 against image local/chatterbox-fast:v1):
- Sample rate is 24000 Hz (Zonos was 44100). The browser Web Audio decode MUST
use 24000 or the voice plays ~1.8x too fast.
- format:"wav", stream:true emits a streaming placeholder-header WAV (0xFFFFFFFF
RIFF/data sizes, one header, then s16le PCM) — the shape the browser's
one-header-strip decoder expects. format:"pcm" (headerless raw s16le) is leaner
but would need the browser to drop the header strip; kept on wav so the
streaming shape stays byte-identical to the Zonos path.
- Body field names are the live pydantic schema: `text` (NOT `input`), `format`
(NOT `response_format`). No `model` field, no `language` field (English-only).
- English is NOT guaranteed by the model alone: the Turbo checkpoint drifts into
foreign-sounding phonemes partway through a long generation (operator report
2026-08-07: "swaps to German halfway through"). There is no `language` pin to
stop it — the only English-stability lever the wire exposes is a LOWER sampling
`temperature` (below the gateway default 0.8), which curbs the wander onto
off-distribution tokens. See `_TTS_TEMPERATURE`.
"""
from __future__ import annotations
import re
import sys
from collections.abc import AsyncIterator, Mapping
from dataclasses import dataclass
from collections.abc import AsyncIterator
import httpx
# DEC-1: the Zonos gateway (irv-ml1 :8890) — NOT the engine :1920, NOT the
# swappable ext-tts alias (which drops the emotion dials). Overridable per
# deployment via app.state.tts_url (RATATOSKR_TTS_URL) — the swap seam + tests.
ZONOS_TTS_URL = "http://10.100.79.3:8890/v1/audio/speech"
# The chatterbox-fast gateway (irv-ml1 :8197). Bespoke `/tts` schema, not OpenAI-shaped,
# not the swappable ext-tts alias (which is OpenAI-shaped and can't reach this wire).
# Overridable per deployment via app.state.tts_url (RATATOSKR_TTS_URL) — the swap seam + tests.
CHATTERBOX_TTS_URL = "http://10.100.79.3:8197/tts"
# DEC-7: fixed emotion strength when PAD-driven (tunable; the gateway scales the
# valence/arousal push by this).
_DEFAULT_EMOTION_STRENGTH = 1.0
# DEC-9: pin English conditioning. Zonos is multilingual; without an explicit
# `language` it drifts into other-language phonemes / gibberish on names, numerics,
# and long spans (operator report 2026-08-02). The gateway accepts an ISO-ish
# `language` code (verified live against /v1/audio/speech).
_TTS_LANGUAGE = "en-us"
# English-stability sampling curbs (DEC-9; infra-ops-authoritative 2026-08-07, thread 01KZEDMJ…).
# The Turbo checkpoint has latent multilingual capacity that LEAKS under high-entropy sampling on
# a long generation (operator: "swaps to German halfway through"). chatterbox has no `language`
# field, so these tighten the sample below the gateway defaults (temp 0.8 / top_p 0.95 / top_k
# 1000) to hold the decode on the English manifold. top_k is the highest-leverage knob — the
# default 1000 admits very-low-probability off-language tokens; infra-ops named it the prime
# suspect. These REDUCE drift probability but do NOT GUARANTEE English on an arbitrarily long
# single generation — the robust fix is bounding generation LENGTH (infra-ops's server-side
# max-chunk cap, or a return to short client-side chunking). See DEC-9.
_TTS_TEMPERATURE = 0.5
_TTS_TOP_P = 0.85
_TTS_TOP_K = 80
class TtsUnavailable(Exception):
"""The Zonos gateway failed, was unreachable, or returned a non-wav body.
"""The gateway failed, was unreachable, or returned a non-wav body.
The caller degrades (INV-TTS-4): logs + skips audio; the turn/transcript is
never blocked or failed on a synthesis error.
@@ -50,312 +68,65 @@ class TtsUnavailable(Exception):
self.message = message
@dataclass(frozen=True)
class PadState:
"""Live PAD read off the affect_update SSE `current` snapshot (the console
already consumes it, DEC-7). `dominance` is carried for completeness but
unused by the emotion dials — Zonos exposes valence + arousal only."""
pleasure: float
arousal: float
dominance: float = 0.0
@classmethod
def from_obj(cls, obj: object) -> PadState | None:
"""Open-world parse of the browser-sent `pad`. Returns None on a missing
or malformed value (pad_to_dials then degrades to a neutral read). Never
raises — the wire is untrusted (INV-TTS-4 / degrade-not-crash)."""
if not isinstance(obj, Mapping):
return None
try:
return cls(
pleasure=float(obj["pleasure"]),
arousal=float(obj["arousal"]),
dominance=float(obj.get("dominance", 0.0)),
)
except (KeyError, TypeError, ValueError, ArithmeticError):
# ArithmeticError covers OverflowError — float() of a huge JSON integer
# literal (e.g. a 400-digit number) raises it; the wire is untrusted, so a
# malformed pad degrades to a neutral read rather than 500ing /api/tts.
return None
@dataclass(frozen=True)
class EmotionDials:
"""Zonos emotion dials (DEC-7). Fields mirror the gateway's /v1/dials surface
(verified live 2026-08-02). `emotion_enabled=False` is a neutral read: the
other fields are omitted from the POST body, so the gateway uses its default
voice emotion."""
emotion_enabled: bool = False
emotion_valence: float = 0.0
emotion_arousal: float = 0.0
emotion_strength: float = _DEFAULT_EMOTION_STRENGTH
def to_body(self) -> dict:
"""The dial fields for the gateway POST body. Emitted ONLY when enabled;
a neutral read contributes nothing (the gateway falls to its default)."""
if not self.emotion_enabled:
return {}
return {
"emotion_enabled": True,
"emotion_valence": self.emotion_valence,
"emotion_arousal": self.emotion_arousal,
"emotion_strength": self.emotion_strength,
}
def _clamp(x: float, lo: float, hi: float) -> float:
"""Clamp to [lo, hi], NaN-safe: a NaN axis (a dead/malformed signal) → 0.0
rather than propagating through max/min into the gateway."""
if x != x: # NaN
return 0.0
return max(lo, min(hi, x))
def pad_to_dials(pad: PadState | None) -> EmotionDials:
"""Map live PAD → Zonos emotion dials (DEC-7 / FN pad_to_dials).
Total: any input (None / finite / out-of-range / NaN / inf / a non-PadState
object) → valid dials, never raises. None/absent/malformed PAD → a neutral read
(emotion_enabled=False)."""
if not isinstance(pad, PadState):
# None, or any non-PadState the declared surface doesn't cover — neutral read.
return EmotionDials(emotion_enabled=False)
return EmotionDials(
emotion_enabled=True,
emotion_valence=_clamp(pad.pleasure, -1.0, 1.0),
emotion_arousal=_clamp(pad.arousal, -1.0, 1.0),
emotion_strength=_DEFAULT_EMOTION_STRENGTH,
)
def gateway_body(text: str, voice: str, dials: EmotionDials) -> dict:
"""The Zonos gateway POST body — response_format is ALWAYS "wav" (DEC-3);
`language` is pinned to en-us (DEC-9) so the multilingual model stays in English."""
def gateway_body(text: str, voice: str) -> dict:
"""The chatterbox-fast POST body. `format:"wav"` (DEC-3) so the streaming shape is a
placeholder-header WAV the browser decoder strips; `stream:true` for play-as-it-arrives
(DEC-2); tightened `temperature`/`top_p`/`top_k` (DEC-9) to hold English across a long turn.
Field names are the live pydantic schema: `text` (not `input`), `format` (not
`response_format`); no `language` (English-only) and no affect dials (DEC-7 retired)."""
return {
"input": text,
"text": text,
"voice": voice,
"response_format": "wav",
"language": _TTS_LANGUAGE,
**dials.to_body(),
"format": "wav",
"stream": True,
"temperature": _TTS_TEMPERATURE,
"top_p": _TTS_TOP_P,
"top_k": _TTS_TOP_K,
}
# DEC-10: long-form chunking. The Zonos model hard-caps ONE synthesis at max_tokens=6144
# = 71.2s of audio (6144 / 86.3 Hz codec frame rate; infra-ops verified 2026-08-02, and the
# gateway 400s above 6144 — the model's architectural sequence limit, unraisable). Anything
# longer must be chunk-and-concatenate client-side. We target 75% of the cap per chunk
# (operator call: greedy for prosody — fewer, fuller chunks, fewer seams), leaving the other
# 25% as headroom for char->audio-seconds variance: the cap is on TOKENS (6144 = 71.2s ALWAYS,
# a fixed codec-frame constant independent of delivery — infra-ops), but our budget is in CHARS
# (a proxy), and slower/expressive delivery stretches a given char-count into more audio-seconds
# (more tokens). ~14 chars/s at neutral rate => ~747-char budget (~53s).
_TTS_MODEL_CAP_SECONDS = 71.2
_TTS_CHUNK_TARGET_FRACTION = 0.75
_TTS_CHARS_PER_SEC = 14.0
_TTS_CHUNK_CHAR_BUDGET = int(
_TTS_MODEL_CAP_SECONDS * _TTS_CHUNK_TARGET_FRACTION * _TTS_CHARS_PER_SEC
)
_PARA_SPLIT = re.compile(r"\n\s*\n+") # blank-line paragraph boundary
_SENTENCE_SPLIT = re.compile(r"(?<=[.!?])\s+") # infra-ops recipe: split after . ! ?
_CLAUSE_BOUNDARIES = (", ", "; ", ": ")
def _hard_wrap(s: str, budget: int) -> list[str]:
"""Sub-split a single over-budget sentence into <= budget pieces, preferring a clause
boundary (`, ; :`), then any space; hard-cuts mid-word ONLY as a last resort (input
with no usable boundary at all). Never returns a piece longer than `budget`."""
out: list[str] = []
while len(s) > budget:
window = s[:budget]
cut = max((window.rfind(c) for c in _CLAUSE_BOUNDARIES), default=-1)
if cut > 0:
cut += 1 # keep the delimiter char with the head; break just after it
else:
cut = window.rfind(" ")
if cut <= 0:
cut = budget # no boundary in range — last-resort hard cut
out.append(s[:cut].strip())
s = s[cut:].strip()
if s:
out.append(s)
return out
def _greedy_pack(units: list[str], budget: int, join: str) -> list[str]:
"""Greedily pack pre-sized units (each <= budget) into <= budget chunks joined by
`join` — as few and as full as possible, so prosody flows across the fewest seams."""
chunks: list[str] = []
cur = ""
for u in units:
if cur and len(cur) + len(join) + len(u) > budget:
chunks.append(cur)
cur = u
else:
cur = f"{cur}{join}{u}" if cur else u
if cur:
chunks.append(cur)
return chunks
def chunk_text(text: str, budget: int = _TTS_CHUNK_CHAR_BUDGET) -> list[str]:
"""Split `text` into synthesis chunks each <= `budget` chars (DEC-10 / FN chunk_text).
Paragraph-first (operator call): whole paragraphs greedily pack together while they fit,
so seams land on blank-line boundaries where a natural pause already belongs. A paragraph
over budget flushes the pending whole-paragraph run, then falls back to sentence packing;
a sentence over budget falls back to clause/word sub-splitting (never mid-word unless the
input has no boundary at all). Every returned chunk is non-empty and <= budget;
empty/whitespace input -> []."""
# A non-positive budget would make _hard_wrap spin forever (cut=budget<=0 => no
# forward progress). Production always passes the 747 default; clamp a misuse to a
# sane floor rather than hang the event loop (degrade-not-crash, module-wide ethos).
budget = max(1, budget)
text = (text or "").strip()
if not text:
return []
chunks: list[str] = []
pending: list[str] = [] # whole paragraphs (each <= budget) awaiting a greedy pack
def flush_pending() -> None:
if pending:
chunks.extend(_greedy_pack(pending, budget, join="\n\n"))
pending.clear()
for para in _PARA_SPLIT.split(text):
para = para.strip()
if not para:
continue
if len(para) <= budget:
pending.append(para)
continue
# Oversized paragraph: emit the accumulated whole-paragraph chunks first (don't
# merge a mid-paragraph fragment across the blank-line boundary), then sentence-pack.
flush_pending()
sentences: list[str] = []
for sent in _SENTENCE_SPLIT.split(para):
sent = sent.strip()
if not sent:
continue
if len(sent) <= budget:
sentences.append(sent)
else:
sentences.extend(_hard_wrap(sent, budget))
chunks.extend(_greedy_pack(sentences, budget, join=" "))
flush_pending()
return chunks
async def tts_stream(
text: str,
*,
voice: str,
dials: EmotionDials,
client: httpx.AsyncClient,
url: str = ZONOS_TTS_URL,
url: str = CHATTERBOX_TTS_URL,
) -> AsyncIterator[bytes]:
"""Open the gateway's CHUNKED stream and yield WAV bytes as they synthesize.
The gateway emits a streaming int16 WAV (RIFF/data sizes = 0xFFFFFFFF placeholders)
over `transfer-encoding: chunked`, TTFB ~0.44s vs ~7s total (infra-ops verified) —
designed to be played progressively by a browser <audio>. So we PROXY THE CHUNKS
STRAIGHT THROUGH: never buffer, never rewrite the header (that would force us to wait
for the whole clip and defeat the streaming — the bug this replaces).
format:"wav"/stream:true emits a streaming int16 WAV (RIFF/data sizes = 0xFFFFFFFF
placeholders) over `transfer-encoding: chunked`, first byte well under 1s (infra-ops) —
designed to be played progressively by the browser Web Audio path. So we PROXY THE CHUNKS
STRAIGHT THROUGH: never buffer, never rewrite the header. chatterbox chunks arbitrary-length
text internally (no per-synthesis cap), so this SINGLE call voices a whole turn — there is
no client-side chunk-and-concatenate wrapper.
Raises TtsUnavailable if the gateway open is a non-200 or the connect/transport
fails — BEFORE the first chunk, so the endpoint can still return a 503 (INV-TTS-4).
A drop mid-stream just ends the generator (the browser has already played the head).
The error policy (the `yielded_any` pivot, folded in from the retired tts_stream_long):
- a non-200 OPEN, or a connect/transport failure BEFORE the first byte, raises
TtsUnavailable so the endpoint peek can still return a 503 (INV-TTS-4) — nothing
committed yet.
- a transport drop AFTER >= 1 byte has already streamed (the 200 is committed) DEGRADES:
log a `tts_degrade` line, end the generator, keep what played. It NEVER raises into the
committed StreamingResponse (which would corrupt it with an ASGI trace).
"""
assert text, "tts_stream: text must be non-empty (the endpoint guards this)"
yielded_any = False
try:
async with client.stream("POST", url, json=gateway_body(text, voice, dials)) as resp:
async with client.stream("POST", url, json=gateway_body(text, voice)) as resp:
if resp.status_code != 200:
raise TtsUnavailable(
f"gateway status {resp.status_code}", status=resp.status_code
)
async for chunk in resp.aiter_bytes():
yielded_any = True
yield chunk
except httpx.RequestError as exc:
raise TtsUnavailable(f"gateway transport failure: {exc}") from exc
# The streaming WAV header is 44 bytes (RIFF 12 + fmt 24 + data 8; infra-ops 2026-08-02).
# A body that has not produced a `data` chunk id within this window is not the WAV we expect,
# so we stop scanning rather than buffer unbounded / match a `data` byte-run deep in PCM.
_WAV_HEADER_SCAN_LIMIT = 1024
async def _pcm_after_header(stream: AsyncIterator[bytes]) -> AsyncIterator[bytes]:
"""Yield only the PCM payload of a streaming WAV — everything AFTER the `data` chunk
id+size (8 bytes). Used for chunks 2..N so their RIFF header isn't buried mid-stream
(infra-ops 2026-08-02: one header only, or the concatenated file corrupts). The `data`
marker can straddle two network reads, so accumulate until it's found + 8 bytes land.
Raises TtsUnavailable if the body has no `data` chunk within the header window, or ends
before one completes — a malformed / non-WAV 200 body. tts_stream_long catches that for a
chunks-2..N failure and degrades (INV-TTS-4) rather than emitting header bytes as PCM or
buffering the whole body forever."""
acc = bytearray()
header_done = False
async for b in stream:
if header_done:
yield b
continue
acc.extend(b)
di = acc.find(b"data")
if di >= 0 and di + 8 <= len(acc):
header_done = True
tail = bytes(acc[di + 8:])
acc = bytearray()
if tail:
yield tail
elif len(acc) > _WAV_HEADER_SCAN_LIMIT:
# No `data` chunk within a sane header window — not the streaming WAV we expect.
raise TtsUnavailable("chunk 2+ body has no WAV data chunk within the header window")
if not header_done:
# Stream ended before a complete `data` header — truncated / empty chunk body.
raise TtsUnavailable("chunk 2+ body ended before the WAV data chunk")
async def tts_stream_long(
text: str,
*,
voice: str,
dials: EmotionDials,
client: httpx.AsyncClient,
url: str = ZONOS_TTS_URL,
budget: int = _TTS_CHUNK_CHAR_BUDGET,
) -> AsyncIterator[bytes]:
"""Synthesize arbitrarily long `text` as ONE continuous int16-PCM stream by chunking it
under the model's 71.2s cap (DEC-10) and concatenating. Chunk 1 streams verbatim (its
WAV header + PCM); chunks 2..N stream PCM-only (header stripped, `_pcm_after_header`) so
the browser decodes one gapless stream after a single leading header. Identical
voice+dials on every chunk so delivery stays uniform across the seams.
A failure BEFORE the first byte (an OPEN failure) propagates as TtsUnavailable so the
endpoint peek turns it into a 503 (INV-TTS-4) — nothing is committed yet. A failure AFTER
bytes have already streamed — a MID-STREAM drop, on chunk 0 or a later chunk, after the
200 is committed — degrades: drop the tail, keep what played, log, and NEVER raise into
the committed StreamingResponse (which would corrupt it with an ASGI trace). The pivot is
`yielded_any`, not the chunk index — a chunk-0 mid-stream drop is a committed-200 failure
too, not an open failure."""
chunks = chunk_text(text, budget)
yielded_any = False
for i, chunk in enumerate(chunks):
stream = tts_stream(chunk, voice=voice, dials=dials, client=client, url=url)
src = stream if i == 0 else _pcm_after_header(stream)
try:
async for b in src:
yielded_any = True
yield b
except TtsUnavailable as exc:
if not yielded_any:
raise # open failure, pre-commit → endpoint peek → 503
# committed-200 mid-stream failure → degrade + log (INV-TTS-4 "logs + skips").
if yielded_any:
# committed-200 mid-stream drop → degrade + log (INV-TTS-4 "logs + skips").
sys.stderr.write(
f'{{"kind":"tts_degrade","event":"chunk_failed","chunk_index":{i},'
f'"chunks_total":{len(chunks)},"exc":"{type(exc).__name__}"}}\n'
f'{{"kind":"tts_degrade","event":"stream_dropped",'
f'"exc":"{type(exc).__name__}"}}\n'
)
return
# open failure, pre-commit → the endpoint peek turns this into a 503.
raise TtsUnavailable(f"gateway transport failure: {exc}") from exc
src/ratatoskr/web/entrypoint.py
+3 -3
View File
@@ -80,9 +80,9 @@ def main(argv: list[str] | None = None) -> int:
# never receives the key, only the session-filtered result.
admin_key = os.environ.get("RATATOSKR_ADMIN_API_KEY")
# Auto-TTS (slice 2): the Zonos gateway URL. Defaults to the direct gateway
# (DEC-1) inside the server; override here only to point at a different synth
# host (the swap seam). None → the server's ZONOS_TTS_URL default.
# Auto-TTS (slice 2): the chatterbox-fast gateway URL. Defaults to the direct
# gateway (DEC-1) inside the server; override here only to point at a different
# synth host (the swap seam). None → the server's CHATTERBOX_TTS_URL default.
tts_url = os.environ.get("RATATOSKR_TTS_URL")
# INV-001: lazy import. Users without [web] extras get a clean hint
src/ratatoskr/web/server.py
+36 -33
View File
@@ -66,11 +66,9 @@ from ratatoskr.sse_client import (
TurnIdFlip,
)
from ratatoskr.tts import (
ZONOS_TTS_URL,
PadState,
CHATTERBOX_TTS_URL,
TtsUnavailable,
pad_to_dials,
tts_stream_long,
tts_stream,
)
@@ -545,14 +543,15 @@ async def _memory_chunks_endpoint(request: Request) -> JSONResponse:
# Per-character voice map (DEC-8): interview characters resolve to their registered
# Zonos voice; everything else falls to the gateway default. Case-folded gateway-side.
# chatterbox reference clip (/refs/<name>.wav); everything else falls to the gateway default.
# NOTE the case: chatterbox wants lowercase "donut" (Zonos used "Donut").
_TTS_VOICE_MAP = {"ratatoskr:donut": "donut"}
_TTS_DEFAULT_VOICE = "Cora"
# The text rides the POST body (DEC-10), so URL length is no longer the bound — this is a
# safety ceiling on the shared-3090 hold: the server chunk-and-concatenates under the model's
# 71.2s/call cap, so ~8000 chars (~11 chunks, ~9 min) covers any real interview turn while a
# runaway is still bounded. A response past this is truncated at a word boundary (the full
# text still shows in the transcript).
_TTS_DEFAULT_VOICE = "glados_25s" # the chatterbox default (Zonos "Cora" does not exist here)
# The text rides the POST body (DEC-10a), so URL length is not the bound — this is a safety
# ceiling on the shared-GPU hold. chatterbox has no per-synth cap and chunks arbitrary-length
# text internally, so a single call voices the whole turn; ~8000 chars still covers any real
# interview turn while bounding a runaway. A response past this is truncated at a word boundary
# (the full text still shows in the transcript).
_TTS_MAX_TEXT_CHARS = 8000
@@ -566,18 +565,17 @@ def _truncate_at_boundary(text: str, limit: int) -> str:
async def _tts_endpoint(request: Request) -> Response:
"""POST /api/tts {text, agent_id?, p?, a?} → audio/wav, STREAMED chunked from the Zonos
gateway (FN tts_endpoint). POST (not GET) so an arbitrarily long turn rides the body,
not a length-capped URL — the server chunk-and-concatenates it under the model's 71.2s
per-call cap into ONE continuous stream (DEC-10, tts_stream_long). The gateway already
streams each chunk (TTFB ~0.44s), so we pipe the bytes straight through — chunk 1's WAV
header verbatim, chunks 2..N header-stripped, so the browser decodes one gapless stream.
"""POST /api/tts {text, agent_id?} → audio/wav, STREAMED chunked from the chatterbox-fast
gateway (FN tts_endpoint). POST (not GET) so an arbitrarily long turn rides the body, not a
length-capped URL. chatterbox chunks arbitrary-length text internally (no per-synth cap), so
a single tts_stream call proxies the whole turn — bytes straight through (one leading WAV
header + s16le PCM), and the browser decodes one gapless stream.
Server-side proxy (DEC-4 / INV-TTS-1: the gateway host never reaches the browser).
Voice per-character (DEC-8); emotion dials from the browser-sent live PAD (DEC-7, p/a
body floats). Serialized one-stream-at-a-time (DEC-5); a new turn aborts the prior fetch
→ the POST drops → the generator's finally releases the lock. A gateway open-failure on
chunk 1 → 503 (INV-TTS-4: the client skips playback)."""
Voice per-character (DEC-8). No affect modulation — DEC-7 retired with the Zonos migration.
Serialized one-stream-at-a-time (DEC-5); a new turn aborts the prior fetch → the POST drops →
the generator's finally releases the lock. A gateway open-failure → 503 (INV-TTS-4: the
client skips playback)."""
try:
body = await request.json()
except (json.JSONDecodeError, ValueError):
@@ -593,29 +591,26 @@ async def _tts_endpoint(request: Request) -> Response:
if not text.strip():
return JSONResponse({"error_code": "missing_text"}, status_code=400)
text = _truncate_at_boundary(text, _TTS_MAX_TEXT_CHARS)
# agent_id / p / a are UNTRUSTED open-world body fields. A non-str agent_id (an
# unhashable list/dict) would TypeError on the voice-map .get(); a huge JSON int p/a
# would OverflowError on float(). Both degrade to a neutral read, never a 500
# (INV-TTS-4). PadState.from_obj already hardens the numeric parse (catches
# ArithmeticError/OverflowError) — route through it rather than re-implement a narrower net.
# agent_id is an UNTRUSTED open-world body field: a non-str (unhashable list/dict) would
# TypeError on the voice-map .get(), so guard the type and degrade to the default voice
# rather than 500 (INV-TTS-4). (The Zonos-era p/a PAD fields are gone — DEC-7 retired.)
agent_id = body.get("agent_id")
voice = (
_TTS_VOICE_MAP.get(agent_id, _TTS_DEFAULT_VOICE)
if isinstance(agent_id, str) else _TTS_DEFAULT_VOICE
)
dials = pad_to_dials(PadState.from_obj({"pleasure": body.get("p"), "arousal": body.get("a")}))
tts_url = request.app.state.tts_url
lock = request.app.state.tts_lock
# DEC-5: one stream at a time on the shared 3090. Held for the stream's duration
# (all chunks) and released in the generator's finally — including the browser-abort
# path. read=120s per infra-ops: a single near-cap chunk can render slowly under load.
# DEC-5: one stream at a time on the shared GPU. Held for the stream's duration and
# released in the generator's finally — including the browser-abort path. read=120s per
# infra-ops: a long synth can render slowly under load.
await lock.acquire()
client = httpx.AsyncClient(
timeout=httpx.Timeout(connect=10.0, read=120.0, write=10.0, pool=10.0)
)
gen = tts_stream_long(text, voice=voice, dials=dials, client=client, url=tts_url)
gen = tts_stream(text, voice=voice, client=client, url=tts_url)
async def _release() -> None:
await gen.aclose() # unwinds tts_stream's `async with` → closes the gateway resp
@@ -640,6 +635,14 @@ async def _tts_endpoint(request: Request) -> Response:
await _release()
raise
# An empty 200 body (no bytes at all) is a synthesis failure, not audio: infra-ops
# (2026-08-07) — chatterbox returns 200 with a 0-byte body when a long single generation
# OOMs the shared 3090. Surface it as a 503 (INV-TTS-4 visible skip) rather than committing
# a silent, empty audio/wav StreamingResponse the browser would play as nothing.
if first is None:
await _release()
return JSONResponse({"error_code": "tts_unavailable"}, status_code=503)
# Chunk 1 must be a WAV (RIFF header). A 200 non-WAV body — a misbehaving gateway or a
# proxy error page — would be mislabeled audio/wav and the browser would decode markup as
# PCM (or match a `data` byte-run in it). Reject → 503 (INV-TTS-4). Tolerant of a <4-byte
@@ -895,11 +898,11 @@ def create_app(
# SERVER-HELD (RATATOSKR_ADMIN_API_KEY) and never reaches the browser — the
# server proxies admin-scoped reads and forwards only the session-filtered result.
app.state.admin_key = admin_key
# Auto-TTS (slice 2): the Zonos gateway URL is SERVER-HELD config — the host
# Auto-TTS (slice 2): the chatterbox-fast gateway URL is SERVER-HELD config — the host
# never reaches the browser (DEC-4 / INV-TTS-1). Defaults to the direct gateway
# (DEC-1); overridable via RATATOSKR_TTS_URL (the swap seam). The lock serializes
# one synth at a time so concurrent turns don't contend the shared 3090 (DEC-5).
app.state.tts_url = tts_url or ZONOS_TTS_URL
app.state.tts_url = tts_url or CHATTERBOX_TTS_URL
app.state.tts_lock = asyncio.Lock()
# INV-002: turn registry is in-process memory, keyed (session_id, turn_id)
app.state.turn_registry = {}
src/ratatoskr/web/static/index.html
+6 -9
View File
@@ -1826,7 +1826,7 @@ async function submitPrompt() {
// Auto-TTS (slice 2, DEC-7): speak the completed response, emotion-modulated by the
// live PAD the persona pane shows. Only on `done`; opt-in; failures are non-blocking.
if (cls === "done" && ttsEnabled()) {
speakOnDone(LIVE.resp, state.agentId, state.lastSnap && state.lastSnap.pad);
speakOnDone(LIVE.resp, state.agentId);
}
es.close();
state.eventSource = null; state.turnId = null; state.curTurnEl = null;
@@ -1994,7 +1994,7 @@ async function cancelTurn() {
});
})();
// ---- auto-TTS: voiced, affect-modulated STREAMING playback via Web Audio ------------
// ---- auto-TTS: voiced STREAMING playback via Web Audio (chatterbox-fast, 24kHz) ------
// Fetch the chunked POST /api/tts stream, decode its int16 PCM, and schedule the samples
// GAPLESSLY into an AudioContext as they arrive (TTFA ~0.5s). Web Audio, NOT <audio src>,
// because Safari/WebKit REFUSES a streaming 0xFFFFFFFF-length WAV via <audio src>
@@ -2035,18 +2035,15 @@ function _findDataChunk(u8) { // offset of the "data" chunk id in a WAV header,
function _u8concat(a, b) { // always returns a FRESH array (byteOffset 0) so Int16Array aligns
const out = new Uint8Array(a.length + b.length); out.set(a, 0); out.set(b, a.length); return out;
}
async function speakOnDone(text, agentId, pad) {
async function speakOnDone(text, agentId) {
const clip = (text || "").trim();
if (!clip) return;
cancelTts(); // INV-TTS-3: stop any prior stream
const ctx = _ttsAudioCtx();
if (!ctx) { tickerAdd("err", "tts", "no audio ctx"); return; }
if (ctx.state === "suspended") { try { await ctx.resume(); } catch (_) {} }
const payload = { text: clip.slice(0, 8000) }; // matches the server cap; chunked server-side (DEC-10)
if (agentId) payload.agent_id = agentId;
if (pad && typeof pad.pleasure === "number" && typeof pad.arousal === "number") {
payload.p = pad.pleasure; payload.a = pad.arousal; // affect dials (DEC-7)
}
const payload = { text: clip.slice(0, 8000) }; // matches the server cap; chunked server-side by the gateway
if (agentId) payload.agent_id = agentId; // no affect fields — DEC-7 retired (chatterbox has no emotion knob)
const ctrl = new AbortController(); _ttsAbort = ctrl;
let resp;
// POST (not GET) so a long turn rides the body, not a length-capped URL (DEC-10).
@@ -2060,7 +2057,7 @@ async function speakOnDone(text, agentId, pad) {
} catch (_) { return; } // aborted / network → silent skip (INV-TTS-4)
if (!resp.ok || !resp.body) { tickerAdd("err", "tts", "unavailable " + resp.status); return; }
const reader = resp.body.getReader();
const SR = 44100;
const SR = 24000; // chatterbox-fast sample rate (was 44100 for Zonos — MUST match or the voice plays ~1.8x too fast)
let playAt = ctx.currentTime + 0.06, started = false, headerDone = false;
let acc = new Uint8Array(0), carry = new Uint8Array(0);
try {
tests/test_tts.py
+56 -285
View File
@@ -1,29 +1,27 @@
"""Tests for ratatoskr.tts — the STREAMING Zonos-gateway client + PAD→dial mapping.
"""Tests for ratatoskr.tts — the STREAMING chatterbox-fast gateway client.
tts_stream proxies the gateway's chunked response verbatim (no buffering, no header
rewrite — the placeholder-size streaming WAV is meant to be played progressively).
pad_to_dials / PadState / EmotionDials are pure + total.
rewrite — the placeholder-size streaming WAV is meant to be played progressively). It
is the sole synthesis primitive: chatterbox chunks arbitrary-length text internally, so
there is no client-side chunk-and-concatenate (retired with the Zonos migration), and no
affect dials (Turbo has no emotion knob). The mid-stream degrade policy is folded in.
"""
import math
import httpx
import pytest
import respx
from ratatoskr.tts import (
_TTS_CHUNK_CHAR_BUDGET,
EmotionDials,
PadState,
_TTS_TEMPERATURE,
_TTS_TOP_K,
_TTS_TOP_P,
CHATTERBOX_TTS_URL,
TtsUnavailable,
chunk_text,
gateway_body,
pad_to_dials,
tts_stream,
tts_stream_long,
)
_URL = "http://tts.example/v1/audio/speech"
_URL = "http://tts.example/tts"
# The gateway's streaming WAV bytes (placeholder 0xFFFFFFFF sizes). We pass them through
# untouched, so the content only has to round-trip.
_WAV = (
@@ -54,307 +52,80 @@ class _RaisingByteStream(httpx.AsyncByteStream):
pass
class TestPadState:
def test_from_obj_valid_mapping(self) -> None:
pad = PadState.from_obj({"pleasure": 0.5, "arousal": -0.2, "dominance": 0.1})
assert pad == PadState(pleasure=0.5, arousal=-0.2, dominance=0.1)
def test_from_obj_dominance_optional(self) -> None:
pad = PadState.from_obj({"pleasure": 0.5, "arousal": -0.2})
assert pad is not None and pad.dominance == 0.0
def test_from_obj_none_is_none(self) -> None:
assert PadState.from_obj(None) is None
def test_from_obj_non_mapping_is_none(self) -> None:
assert PadState.from_obj("not a mapping") is None
assert PadState.from_obj([0.1, 0.2]) is None
def test_from_obj_missing_key_is_none(self) -> None:
assert PadState.from_obj({"pleasure": 0.5}) is None # no arousal
def test_from_obj_non_numeric_is_none(self) -> None:
assert PadState.from_obj({"pleasure": "hot", "arousal": 0.1}) is None
def test_from_obj_huge_int_overflow_is_none(self) -> None:
huge = int("9" * 400)
assert PadState.from_obj({"pleasure": huge, "arousal": 0}) is None
class TestPadToDials:
def test_none_pad_is_neutral_disabled(self) -> None:
d = pad_to_dials(None)
assert d.emotion_enabled is False
assert d.to_body() == {}
def test_maps_pleasure_and_arousal(self) -> None:
d = pad_to_dials(PadState(pleasure=0.4, arousal=0.6))
assert d.emotion_enabled is True
assert d.emotion_valence == pytest.approx(0.4)
assert d.emotion_arousal == pytest.approx(0.6)
def test_clamps_out_of_range(self) -> None:
d = pad_to_dials(PadState(pleasure=5.0, arousal=-9.0))
assert d.emotion_valence == 1.0
assert d.emotion_arousal == -1.0
def test_nan_degrades_to_zero_never_raises(self) -> None:
d = pad_to_dials(PadState(pleasure=math.nan, arousal=math.inf))
assert d.emotion_valence == 0.0
assert d.emotion_arousal == 1.0
def test_non_padstate_input_degrades_to_neutral(self) -> None:
for bad in ({}, "bad", [0.1, 0.2], object(), 42):
d = pad_to_dials(bad)
assert d.emotion_enabled is False and d.to_body() == {}
class TestEmotionDialsToBody:
def test_disabled_emits_no_params(self) -> None:
assert EmotionDials(emotion_enabled=False).to_body() == {}
def test_enabled_emits_valence_arousal_strength(self) -> None:
body = EmotionDials(
emotion_enabled=True, emotion_valence=0.3, emotion_arousal=-0.1
).to_body()
assert body["emotion_enabled"] is True
assert body["emotion_valence"] == 0.3
assert body["emotion_arousal"] == -0.1
assert "emotion_strength" in body
class TestGatewayBody:
def test_always_wav_with_dials(self) -> None:
b = gateway_body("hi", "donut", pad_to_dials(PadState(pleasure=0.5, arousal=0.2)))
assert b["input"] == "hi" and b["voice"] == "donut"
assert b["response_format"] == "wav" # DEC-3 — ALWAYS wav
assert b["language"] == "en-us" # DEC-9 — pin English conditioning
assert b["emotion_valence"] == pytest.approx(0.5)
def test_bespoke_chatterbox_schema(self) -> None:
b = gateway_body("hi", "donut")
assert b["text"] == "hi" # "text", not "input"
assert b["voice"] == "donut"
assert b["format"] == "wav" # DEC-3 — "format", not "response_format"
assert b["stream"] is True # DEC-2 — play-as-it-arrives
def test_neutral_omits_emotion(self) -> None:
b = gateway_body("hi", "Cora", pad_to_dials(None))
assert "emotion_valence" not in b and b["response_format"] == "wav"
def test_sampling_curbs_below_gateway_defaults_hold_english(self) -> None:
# DEC-9: the model has no `language` pin and Turbo's multilingual capacity leaks under
# high-entropy sampling on long turns. gateway_body tightens temperature/top_p/top_k
# below the gateway defaults (0.8 / 0.95 / 1000) to hold English — top_k the highest-
# leverage. Pin presence + the below-default relationship (infra-ops-authoritative).
b = gateway_body("a long turn", "donut")
assert b["temperature"] == _TTS_TEMPERATURE and _TTS_TEMPERATURE < 0.8
assert b["top_p"] == _TTS_TOP_P and _TTS_TOP_P < 0.95
assert b["top_k"] == _TTS_TOP_K and _TTS_TOP_K < 1000
def test_no_zonos_era_fields(self) -> None:
# The Zonos body fields are gone: no OpenAI `input`/`response_format`, no
# `language` pin (DEC-9 retired), no affect dials (DEC-7 retired).
b = gateway_body("hi", "Cora")
for dead in ("input", "response_format", "language", "emotion_valence",
"emotion_arousal", "emotion_enabled", "emotion_strength"):
assert dead not in b
class TestTtsStream:
@respx.mock
async def test_streams_chunks_and_posts_wav_body(self) -> None:
async def test_streams_chunks_and_posts_bespoke_body(self) -> None:
route = respx.post(_URL).mock(return_value=httpx.Response(200, content=_WAV))
async with httpx.AsyncClient() as client:
out = await _drain(tts_stream(
"hello there", voice="donut",
dials=pad_to_dials(PadState(pleasure=0.5, arousal=0.2)),
client=client, url=_URL,
))
out = await _drain(tts_stream("hello there", voice="donut", client=client, url=_URL))
assert out == _WAV # passed through verbatim — no header rewrite
import json as _json
body = _json.loads(route.calls.last.request.content)
assert body["text"] == "hello there"
assert body["voice"] == "donut"
assert body["response_format"] == "wav"
assert body["language"] == "en-us" # DEC-9 — pin English conditioning
assert body["emotion_valence"] == pytest.approx(0.5)
assert body["format"] == "wav"
assert body["stream"] is True
@respx.mock
async def test_default_url_is_chatterbox(self) -> None:
route = respx.post(CHATTERBOX_TTS_URL).mock(
return_value=httpx.Response(200, content=_WAV)
)
async with httpx.AsyncClient() as client:
await _drain(tts_stream("hi", voice="donut", client=client))
assert route.called # the module default points at the chatterbox gateway
@respx.mock
async def test_non_200_open_raises_before_any_chunk(self) -> None:
respx.post(_URL).mock(return_value=httpx.Response(500, content=b"boom"))
async with httpx.AsyncClient() as client:
with pytest.raises(TtsUnavailable) as exc:
await _drain(tts_stream(
"hi", voice="Cora", dials=pad_to_dials(None), client=client, url=_URL
))
await _drain(tts_stream("hi", voice="Cora", client=client, url=_URL))
assert exc.value.status == 500
@respx.mock
async def test_transport_error_raises(self) -> None:
async def test_transport_error_on_open_raises(self) -> None:
respx.post(_URL).mock(side_effect=httpx.ConnectError("refused"))
async with httpx.AsyncClient() as client:
with pytest.raises(TtsUnavailable):
await _drain(tts_stream(
"hi", voice="Cora", dials=pad_to_dials(None), client=client, url=_URL
))
class TestChunkText:
"""chunk_text (DEC-10): paragraph-first greedy pack, sentence fallback for oversized
paragraphs, clause/word sub-split for oversized sentences; every chunk <= budget."""
def test_empty_and_whitespace_yield_no_chunks(self) -> None:
assert chunk_text("") == []
assert chunk_text(" \n\n \t ") == []
def test_short_text_is_one_chunk(self) -> None:
assert chunk_text("Hello, darling.", budget=100) == ["Hello, darling."]
def test_two_short_paragraphs_greedily_merge(self) -> None:
# Both fit in one budget -> one chunk, joined on the blank-line boundary.
out = chunk_text("First para.\n\nSecond para.", budget=100)
assert out == ["First para.\n\nSecond para."]
def test_paragraphs_split_on_blank_line_when_over_budget(self) -> None:
# Each paragraph fits alone but not together -> a seam on the paragraph boundary.
a, b = "A" * 30, "B" * 30
out = chunk_text(f"{a}\n\n{b}", budget=40)
assert out == [a, b]
def test_oversized_paragraph_falls_back_to_sentences(self) -> None:
para = "One sentence here. Two sentence here. Three sentence here."
out = chunk_text(para, budget=25)
assert all(len(c) <= 25 for c in out)
assert len(out) >= 2
# every word is preserved whole and in order (no split mid-word)
assert [w for c in out for w in c.split()] == para.split()
def test_oversized_sentence_sub_splits_never_mid_word(self) -> None:
sent = "alpha beta gamma delta epsilon zeta eta theta iota kappa lambda"
out = chunk_text(sent, budget=20)
assert all(len(c) <= 20 for c in out)
for c in out:
for word in c.split():
assert word in sent.split() # every emitted token is a whole source word
def test_every_chunk_within_budget_default(self) -> None:
para = ("Princess Donut does not wait. " * 200).strip()
out = chunk_text(para) # default budget
assert out and all(len(c) <= _TTS_CHUNK_CHAR_BUDGET for c in out)
def test_spaceless_over_budget_hard_cuts_as_last_resort(self) -> None:
out = chunk_text("x" * 50, budget=20)
assert all(len(c) <= 20 for c in out)
assert "".join(out) == "x" * 50
def test_non_positive_budget_does_not_hang(self) -> None:
# budget <= 0 would infinite-loop _hard_wrap; it's clamped to 1 so this terminates.
out = chunk_text("alpha beta", budget=0)
assert out and all(len(c) <= 1 for c in out)
assert "".join(out) == "alphabeta" # every char preserved, forward progress made
def test_oversized_sentence_prefers_clause_boundary_over_space(self) -> None:
# A comma-bearing over-budget sentence sub-splits at the CLAUSE boundary (", "),
# not merely at the last space — pins the _CLAUSE_BOUNDARIES preference (else dead).
out = chunk_text("alpha, beta gamma delta", budget=12)
assert all(len(c) <= 12 for c in out)
assert out[0] == "alpha," # clause cut, not "alpha, beta" (a space-only cut)
def test_default_budget_is_the_dec10_value(self) -> None:
# Pin the concrete 747 that FN chunk_text's POST commits to (75% of 71.2s @ 14 c/s).
# The suite's other budget checks compare against the imported constant and so move
# with it; this one anchors the value itself so a retune is a deliberate edit here.
assert _TTS_CHUNK_CHAR_BUDGET == 747
class TestTtsStreamLong:
"""tts_stream_long (DEC-10): concatenate per-chunk synthesis into ONE int16-PCM stream
— chunk 1 verbatim (header + PCM), chunks 2..N header-stripped."""
_PCM = b"\x11\x22" * 64
_WAV_CHUNK = (
b"RIFF\xff\xff\xff\xffWAVEfmt \x10\x00\x00\x00" + b"\x00" * 20
+ b"data\xff\xff\xff\xff" + _PCM
)
@respx.mock
async def test_single_chunk_passes_through_verbatim(self) -> None:
respx.post(_URL).mock(return_value=httpx.Response(200, content=self._WAV_CHUNK))
async with httpx.AsyncClient() as client:
out = await _drain(tts_stream_long(
"Short line.", voice="donut", dials=pad_to_dials(None),
client=client, url=_URL, budget=100,
))
assert out == self._WAV_CHUNK # one chunk => untouched
@respx.mock
async def test_multi_chunk_emits_one_header_then_concatenated_pcm(self) -> None:
respx.post(_URL).mock(return_value=httpx.Response(200, content=self._WAV_CHUNK))
text = "First part here. Second part here. Third part here." # budget 18 -> >=2 chunks
async with httpx.AsyncClient() as client:
out = await _drain(tts_stream_long(
text, voice="donut", dials=pad_to_dials(None), client=client, url=_URL, budget=18,
))
n = len(chunk_text(text, budget=18))
assert n >= 2
assert out.count(b"RIFF") == 1 and out.count(b"data") == 1 # exactly one header
# EXACT bytes: chunk 1 verbatim (header+PCM), chunks 2..N stripped to PCM. Asserting
# the exact stream catches a di+4-vs-di+8 strip off-by-one (2-byte sample alignment
# across seams) that a header-count check alone would miss.
assert out == self._WAV_CHUNK + self._PCM * (n - 1)
# DEC-10: identical voice+dials+language on EVERY chunk (uniform delivery across seams).
import json as _json
bodies = [_json.loads(c.request.content) for c in respx.calls]
assert len(bodies) == n
assert all(b["voice"] == "donut" and b["language"] == "en-us" for b in bodies)
await _drain(tts_stream("hi", voice="Cora", client=client, url=_URL))
@respx.mock
async def test_mid_stream_drop_after_first_byte_degrades_not_raises(self) -> None:
# A2: chunk 0 opens 200, yields bytes, then drops mid-stream. Because the 200 is
# committed (bytes already flowed), this must DEGRADE (return what streamed), never
# raise — the pivot is yielded_any, not the chunk index.
# The 200 is committed once bytes flow; a later transport drop must DEGRADE
# (return what streamed), never raise — the pivot is yielded_any, folded in from
# the retired tts_stream_long. Keeps a committed StreamingResponse from an ASGI trace.
respx.post(_URL).mock(
return_value=httpx.Response(200, stream=_RaisingByteStream(self._WAV_CHUNK))
return_value=httpx.Response(200, stream=_RaisingByteStream(_WAV))
)
async with httpx.AsyncClient() as client:
out = await _drain(tts_stream_long(
"hi", voice="donut", dials=pad_to_dials(None), client=client, url=_URL, budget=100,
))
assert out == self._WAV_CHUNK # head kept, no raise
@respx.mock
async def test_later_chunk_gateway_500_degrades_keeps_prior(self) -> None:
# chunk 1 = valid WAV; chunk 2 = a gateway 500 (OPEN failure on a later chunk).
respx.post(_URL).mock(side_effect=[
httpx.Response(200, content=self._WAV_CHUNK),
httpx.Response(500, content=b"boom"),
])
text = "First part here. Second part here." # budget 18 -> 2 chunks
async with httpx.AsyncClient() as client:
out = await _drain(tts_stream_long(
text, voice="donut", dials=pad_to_dials(None), client=client, url=_URL, budget=18,
))
assert out == self._WAV_CHUNK # INV-TTS-4 degrade: keep chunk 1, drop the tail, no raise
@respx.mock
async def test_later_chunk_missing_data_degrades_keeps_prior(self) -> None:
# chunk 1 = valid WAV; chunk 2 = a 200 non-WAV body (no `data` chunk) -> degrade.
respx.post(_URL).mock(side_effect=[
httpx.Response(200, content=self._WAV_CHUNK),
httpx.Response(200, content=b"xxxxx no marker present xxxxx"),
])
text = "First part here. Second part here." # budget 18 -> 2 chunks
async with httpx.AsyncClient() as client:
out = await _drain(tts_stream_long(
text, voice="donut", dials=pad_to_dials(None), client=client, url=_URL, budget=18,
))
# INV-TTS-4 degrade: chunk 1 audio retained verbatim, chunk 2 dropped (no raise, no
# garbage bytes emitted from the malformed body).
assert out == self._WAV_CHUNK
@respx.mock
async def test_first_chunk_gateway_failure_raises(self) -> None:
respx.post(_URL).mock(return_value=httpx.Response(500, content=b"boom"))
async with httpx.AsyncClient() as client:
with pytest.raises(TtsUnavailable):
await _drain(tts_stream_long(
"hi", voice="Cora", dials=pad_to_dials(None),
client=client, url=_URL, budget=100,
))
async def test_pcm_after_header_reassembles_data_marker_across_reads(self) -> None:
# The `data` marker can straddle two network reads; _pcm_after_header must accumulate
# until it lands, then yield only the PCM after it. Pins the docstring's straddle claim.
from ratatoskr.tts import _pcm_after_header
async def _split_stream():
yield b"RIFF\xff\xff\xff\xffWAVEfmt \x10\x00\x00\x00" + b"\x00" * 20 + b"da"
yield b"ta\xff\xff\xff\xff" + b"\x11\x22" * 4 # rest of 'data' + size + PCM
out = await _drain(_pcm_after_header(_split_stream()))
assert out == b"\x11\x22" * 4 # PCM only; marker reassembled across the read boundary
async def test_pcm_after_header_no_data_marker_raises(self) -> None:
from ratatoskr.tts import _pcm_after_header
async def _no_marker():
yield b"xxxxx no marker present xxxxx"
with pytest.raises(TtsUnavailable):
await _drain(_pcm_after_header(_no_marker()))
out = await _drain(tts_stream("hi", voice="donut", client=client, url=_URL))
assert out == _WAV # head kept, no raise
tests/test_web_server.py
+38 -53
View File
@@ -1458,18 +1458,18 @@ class TestMemoryChunksEndpoint:
class TestTtsEndpoint:
"""tts_endpoint FN — POST /api/tts → audio/wav STREAMED (chunked) from the Zonos
gateway. Voice per-character (DEC-8), emotion dials from p/a body floats (DEC-7),
the gateway host never reaches the browser (INV-TTS-1), gateway open-failure → 503
(INV-TTS-4). POST so an arbitrarily long turn rides the body; the server chunk-and-
concatenates it under the model's 71.2s/call cap (DEC-10)."""
"""tts_endpoint FN — POST /api/tts → audio/wav STREAMED (chunked) from the
chatterbox-fast gateway. Voice per-character (DEC-8), the gateway host never reaches
the browser (INV-TTS-1), gateway open-failure → 503 (INV-TTS-4). POST so an
arbitrarily long turn rides the body; the gateway chunks internally (DEC-10 retired —
no client concat). No affect: p/a body fields are gone (DEC-7 retired)."""
# Streaming WAV bytes (placeholder 0xFFFFFFFF sizes) — proxied through verbatim.
_WAV = b"RIFF\xff\xff\xff\xffWAVEdata\xff\xff\xff\xff" + b"\x11\x22" * 64
_TTS = "http://tts.example/v1/audio/speech"
_TTS = "http://tts.example/tts"
@respx.mock
def test_happy_streams_wav_resolves_donut_voice_and_pad(self) -> None:
def test_happy_streams_wav_resolves_donut_voice(self) -> None:
from ratatoskr.web.server import create_app
route = respx.post(self._TTS).mock(
@@ -1478,20 +1478,22 @@ class TestTtsEndpoint:
app = create_app(_mock_client_factory(), tts_url=self._TTS)
resp = TestClient(app).post(
"/api/tts",
json={"text": "Carl is a softie.", "agent_id": "ratatoskr:donut",
"p": 0.6, "a": 0.3},
json={"text": "Carl is a softie.", "agent_id": "ratatoskr:donut"},
)
assert resp.status_code == 200
assert resp.headers["content-type"].startswith("audio/wav")
assert resp.content == self._WAV # one chunk → streamed through verbatim
assert resp.content == self._WAV # streamed through verbatim
body = json.loads(route.calls.last.request.content)
assert body["text"] == "Carl is a softie." # "text", not "input"
assert body["voice"] == "donut"
assert body["response_format"] == "wav"
assert body["language"] == "en-us" # DEC-9 pinned per chunk
assert body["emotion_valence"] == pytest.approx(0.6)
assert body["format"] == "wav" # "format", not "response_format"
assert body["stream"] is True
# No Zonos-era fields ride the body.
for dead in ("input", "response_format", "language", "emotion_valence"):
assert dead not in body
@respx.mock
def test_unmapped_agent_default_voice_no_pad_no_emotion(self) -> None:
def test_unmapped_agent_falls_to_default_voice(self) -> None:
from ratatoskr.web.server import create_app
route = respx.post(self._TTS).mock(
@@ -1501,8 +1503,7 @@ class TestTtsEndpoint:
resp = TestClient(app).post("/api/tts", json={"text": "hello", "agent_id": "mimir"})
assert resp.status_code == 200
body = json.loads(route.calls.last.request.content)
assert body["voice"] == "Cora" # gateway default (DEC-8)
assert "emotion_valence" not in body # no p/a → neutral read (DEC-7)
assert body["voice"] == "glados_25s" # chatterbox default (DEC-8)
def test_missing_text_returns_400(self) -> None:
from ratatoskr.web.server import create_app
@@ -1523,50 +1524,21 @@ class TestTtsEndpoint:
assert resp.json()["error_code"] == "tts_unavailable"
@respx.mock
def test_long_text_is_chunked_each_call_within_budget(self) -> None:
from ratatoskr.tts import _TTS_CHUNK_CHAR_BUDGET
def test_long_text_is_one_gateway_call_not_chunked(self) -> None:
# DEC-10 RETIRED: chatterbox chunks internally, so a long turn is ONE gateway call
# with the full text — no client-side chunk-and-concatenate.
from ratatoskr.web.server import create_app
route = respx.post(self._TTS).mock(
return_value=httpx.Response(200, content=self._WAV)
)
app = create_app(_mock_client_factory(), tts_url=self._TTS)
# ~5000 chars of distinct sentences → several gateway calls (DEC-10 chunking).
text = " ".join(f"Sentence number {i} about the dungeon." for i in range(200))
resp = TestClient(app).post("/api/tts", json={"text": text})
assert resp.status_code == 200
assert len(route.calls) >= 2 # chunk-and-concatenate, not one giant call
for call in route.calls:
body = json.loads(call.request.content)
assert len(body["input"]) <= _TTS_CHUNK_CHAR_BUDGET # each chunk under cap
@respx.mock
def test_malformed_pad_body_degrades_no_500(self) -> None:
from ratatoskr.web.server import create_app
route = respx.post(self._TTS).mock(
return_value=httpx.Response(200, content=self._WAV)
)
app = create_app(_mock_client_factory(), tts_url=self._TTS)
resp = TestClient(app).post("/api/tts", json={"text": "hi", "p": "notafloat", "a": 0.1})
assert resp.status_code == 200
assert len(route.calls) == 1 # one call, not chunk-and-concatenate
body = json.loads(route.calls.last.request.content)
assert "emotion_valence" not in body # malformed p → neutral read, not a 500
@respx.mock
def test_huge_int_pad_degrades_no_500(self) -> None:
# A 400-digit JSON int → float() OverflowError (an ArithmeticError, not a
# ValueError). POST re-opened this path that GET's string query params couldn't.
from ratatoskr.web.server import create_app
route = respx.post(self._TTS).mock(
return_value=httpx.Response(200, content=self._WAV)
)
app = create_app(_mock_client_factory(), tts_url=self._TTS)
resp = TestClient(app).post("/api/tts", json={"text": "hi", "p": 10**400, "a": 0.1})
assert resp.status_code == 200
body = json.loads(route.calls.last.request.content)
assert "emotion_valence" not in body # OverflowError → neutral read, not a 500
assert body["text"] == text # full text in one call, unsplit
@respx.mock
def test_non_str_agent_id_degrades_no_500(self) -> None:
@@ -1580,7 +1552,7 @@ class TestTtsEndpoint:
resp = TestClient(app).post("/api/tts", json={"text": "hi", "agent_id": ["donut"]})
assert resp.status_code == 200
body = json.loads(route.calls.last.request.content)
assert body["voice"] == "Cora" # non-str agent_id → default voice, not a 500
assert body["voice"] == "glados_25s" # non-str agent_id → default voice, not a 500
def test_whitespace_text_returns_400(self) -> None:
from ratatoskr.web.server import create_app
@@ -1601,7 +1573,7 @@ class TestTtsEndpoint:
raise RuntimeError("boom during peek")
yield b"" # unreachable — marks this an async generator
monkeypatch.setattr(_server, "tts_stream_long", _boom)
monkeypatch.setattr(_server, "tts_stream", _boom)
app = create_app(_mock_client_factory(), tts_url=self._TTS)
with pytest.raises(RuntimeError):
TestClient(app).post("/api/tts", json={"text": "hi"})
@@ -1624,7 +1596,20 @@ class TestTtsEndpoint:
)
assert resp.status_code == 200
body = json.loads(route.calls.last.request.content)
assert body["input"] == "hello" # surrogate dropped, rest intact — not a 500
assert body["text"] == "hello" # surrogate dropped, rest intact — not a 500
@respx.mock
def test_empty_200_body_rejected_503(self) -> None:
# infra-ops (2026-08-07): chatterbox returns 200 with a 0-byte body when a long single
# generation OOMs the shared 3090. An empty 200 is a synthesis failure, not silent audio
# — surface it as 503 (INV-TTS-4) rather than committing an empty audio/wav stream.
from ratatoskr.web.server import create_app
respx.post(self._TTS).mock(return_value=httpx.Response(200, content=b""))
app = create_app(_mock_client_factory(), tts_url=self._TTS)
resp = TestClient(app).post("/api/tts", json={"text": "a very long turn"})
assert resp.status_code == 503
assert resp.json()["error_code"] == "tts_unavailable"
@respx.mock
def test_non_wav_200_body_rejected_503(self) -> None: