Voice assignment moves from the hardcoded server map to ~/.config/ratatoskr/
voices.json (per-agent voice + optional narration_voice). An agent with a
narration_voice gets a two-voice split: quoted speech in `voice`, narration in
`narration_voice`, synthesized per-span and stitched under one WAV header.
- new src/ratatoskr/voices.py: load_voice_config (degrade-not-crash),
segment_dialogue (quote-based, straight + curly), resolve_voice_spans
- tts.py: tts_stream_stitched replaces tts_stream — serial per-span synth,
span 0 verbatim, spans 1..N header-stripped -> one gapless 48kHz stream;
a single-span list is a byte-identical passthrough (no single-voice regression)
- server.py: _tts_endpoint resolves spans from app.state.voice_config; the
hardcoded _TTS_VOICE_MAP is retired; create_app gains a voice_config param
- entrypoint.py: loads voices.json at startup
- contract DEC-11 + INV-TTS-5/6/7; initial config donut->donut,
sindra->miranda (dialogue) / emmie (narration)
Live-verified on :8765: Sindra mixed turn -> 2 dots calls (emmie+miranda)
stitched into one 48kHz WAV with a single RIFF header; Donut single-voice
unchanged. 545 tests green (incl. new test_voices.py).
The long-turn "swaps to German" garble was NOT a language leak (infra-ops's
initial framing) and NOT the sampling entropy my interim curb targeted. The real
cause, signal-measured by infra-ops: the Chatterbox Turbo model over-runs its
generation TAIL — a long single generation degrades into garble/dead-air in its
final ~2-3s (voiced-tail zero-crossing rate 1.58x the middle). The gateway's
unbounded chunk-size ratchet built 300-600 char mega-chunks that landed in that
zone, and streaming concatenated each bad tail.
My interim curb (top_k 1000->80, top_p 0.95->0.85, temp 0.8->0.5) made it WORSE:
tight sampling pulls the degradation onset to a shorter length (~200 chars vs
~300 at defaults), so it fights the server-side fix rather than helping.
Fixed server-side (infra-ops, chatterbox-fast image :v2): a max_chunk_chars=250
cap bounds each generation below the ~300-char onset -> clean prosodic chunks
(verified ZCR 1.58x -> 0.64x; operator ear-confirmed clean audio + clean joins).
Consumer side, this commit:
- Revert the sampling knobs: gateway_body back to {text, voice, format, stream},
send full text with the gateway's default sampling. The server chunks at 250.
- Keep the /api/tts empty-200 -> 503 guard as hygiene (DEC-9a; the shared-3090
OOM that produced empty 200s is also resolved — Zonos moved off the card).
Contract DEC-9 rewritten with the resolved root cause + the two wrong hypotheses;
DEC-9a marked kept-as-hygiene. 520 green.
Repoint the TTS client from the Zonos gateway (:8890 /v1/audio/speech) to
chatterbox-fast (:8197 /tts — bespoke non-OpenAI {text,voice,format,stream}
schema, no auth, 24kHz, infra-ops-verified). tts.py stays the single swap seam.
Dropped, no backward-compat (pre-v1):
- Affect (DEC-7): the Turbo checkpoint has no emotion knob, so PadState,
EmotionDials, pad_to_dials, the /api/tts p/a fields, and the browser pad
argument are deleted. Voice is now flat.
- Client-side chunking (DEC-10): chatterbox has no per-synth cap and chunks
internally, so chunk_text/tts_stream_long/_pcm_after_header are deleted; a
single tts_stream call voices a whole turn, the mid-stream yielded_any degrade
folded into it.
- Language pin (DEC-9): no language field; re-purposed to sampling curbs (below).
Fixed / added:
- Browser Web Audio sample rate 44100 -> 24000 (the chatterbox rate).
- Default voice Cora -> glados_25s; donut registered lowercase at /refs/donut.wav.
- English-drift curb: Turbo is multilingual-leaky and wanders off English on a
long generation (the gateway scheduler ratchets chunk size unbounded). Tighten
sampling in gateway_body: top_k 1000->80, top_p 0.95->0.85, temperature
0.8->0.5. These reduce drift probability; the guaranteed fix is a server-side
max-chunk cap (infra-ops, greenlit).
- OOM guard (DEC-9a): a long generation can OOM the shared 3090, returning 200
with a 0-byte body; /api/tts surfaces an empty 200 as 503 rather than
committing silent audio.
Contract donut_voiced_interview.contract.md amended: migration banner, DEC-1/3/8
amended, DEC-7/9/10 retired with historical notes, DEC-9a added.
Tests rewritten to the new wire; 520 green. Live-smoked against the gateway
(24kHz synth + endpoint proxy + web console). persistent-memory.md committed
alongside (commit-along).
TTS fixes + hardening for the Donut voiced interview.
Feature:
- gibberish -> pin `language: "en-us"` on every gateway call (DEC-9); the
multilingual model drifted into other-language phonemes without it.
- truncation -> the Zonos model hard-caps one synthesis at 6144 tokens /
71.2s (infra-ops). Chunk client-side (paragraph-first, greedy to ~75%
of cap for prosody; sentence/clause fallback) and concatenate the int16
PCM behind ONE WAV header (DEC-10). /api/tts becomes POST so a long turn
rides the body, not a length-capped URL (DEC-10a).
- persona -> dialogue-only rewrite (no asterisk RP beats -- they were being
voiced as gibberish) + always consult the native `reference_knowledge`
tool before answering (retires the stale kb_bridge references). Pushed
live to ratatoskr:donut.
Heid code-review + bug-hunt hardening (4-arm panels, triaged):
- untrusted /api/tts body fields degrade, never 500: huge-int PAD
(OverflowError), non-str agent_id (unhashable .get), lone surrogates
(utf-8 encode), whitespace-only text.
- serialize lock + client released on every peek escape (cancel /
InvalidURL) -- previously a permanent deadlock.
- a mid-stream drop after a committed 200 degrades (keeps what played),
never raises into the response; a non-WAV 200 body is rejected (RIFF
sniff + bounded header scan) instead of decoded as garbage.
546 tests green; long-form live-verified (106.6s, one header). Contract
brought canonical (DEC-9/10, FN chunk_text/tts_stream_long, POST endpoint,
INV-TTS-4 logging scope, FN pad_to_dials domain). reference_knowledge
empty-recall root-caused to a Worldtree wing-misfile (escalated to
worldtree-dev; not ratatoskr code).
Operator: play-as-it-arrives, don't wait for the whole clip. infra-ops confirmed the
Zonos gateway ALREADY streams (chunked int16 WAV, TTFB ~0.44s vs ~7s total; placeholder
0xFFFFFFFF sizes are DESIGNED for progressive <audio src>). The buffering was entirely
in our proxy, and the _finalize_wav_header rewrite (6c3c08b) FORCED it — computing the
real sizes needs the whole clip.
The fix — pipe the chunks straight through:
- tts.py: buffered tts_synthesize + _finalize_wav_header REMOVED; tts_stream (an async
generator over the gateway's chunked response) + gateway_body added. Never buffer,
never rewrite the placeholder header.
- server.py: /api/tts is now GET (so a browser <audio src> plays it progressively) →
a chunked StreamingResponse piping the gateway; peeks the first chunk so a bad gateway
OPEN still returns 503; the serialize lock is held across the stream and released on
completion/abort; PAD rides p/a query floats.
- index.html: speakOnDone sets <audio src="/api/tts?..."> (streaming) instead of
fetch->blob; dropped the blob machinery. AUTOPLAY UNLOCK: _unlockTtsAudio() plays a
silent WAV within the toggle/submit gesture so the delayed play() isn't blocked — the
actual cause of "no audio" (play() fires ~15s after the keypress, past the browser's
transient-activation window).
Live-verified: GET /api/tts is transfer-encoding: chunked, TTFB 0.46s. Playwright with
--autoplay-policy=document-user-activation-required: the streaming <audio src> plays
progressively (currentTime advances, no decode error, no MSE fallback needed) 6.5s after
the gesture — proving the unlock's persistent element flag. 521 green.
DEC-2 amended (streaming supersedes "no streaming"); FN tts_stream / tts_endpoint updated.
The Zonos gateway returns a STREAMING wav header — the RIFF chunk size (offset 4)
and the data chunk size are both 0xFFFFFFFF ("unknown length"), because it can
stream. A browser <audio> element playing a fully-downloaded blob needs a finite,
correctly-sized WAV; a 0xFFFFFFFF length reads as raw/streaming PCM and won't play
(operator-reported: "zonos sends pcm by default, but the browser wants wav").
tts_synthesize now rewrites both size fields with the real byte counts — the whole
clip is buffered server-side, so the sizes are known. Idempotent on an already-
correct header; no-op-safe if the data chunk isn't found. Live-verified: /api/tts
output now opens as a valid finite WAV (wave.open: 1ch/16bit/44.1kHz), RIFF + data
sizes correct where they were 0xFFFFFFFF before.
TDD: +1 test (streaming 0xFFFFFFFF header -> real sizes, wave-module-decodable); the
_WAV fixtures upgraded from bare RIFF stubs to proper finite WAVs. 525 green.
Triaged the heid-bug-hunt panel (Gróa+Hulda+Regin+Kimi, 11 distinct findings).
Fixed the real ones; the 3-arm "memory_context unverifiable" alarm was refuted
(tests + live SDK verify), and caller-supplied agent_id is accepted under the
LAN/no-auth debug-tool trust model (documented, not fixed).
Constructible crashes (were uncaught HTTP 500s from wire input):
- _tts_endpoint: coerce non-str / unhashable agent_id -> None before the voice-map
lookup (matches the submit path's guard); an unhashable {} / [] TypeError'd -> 500.
- PadState.from_obj: catch ArithmeticError — float() of a huge-int JSON literal
raises OverflowError, absent from the except tuple -> 500; now a neutral read.
- _submit_turn_endpoint: require a non-blank STR content — a truthy non-str crashed
pin_kb_context's question.strip() mid-stream instead of a deterministic 400.
pin_kb_context also isinstance-guards the question defensively.
Robustness:
- kb_bridge: delete the throwaway Mimir consult session (SDK sessions.delete) on
success/error/timeout via a caller-owned holder so cleanup survives a mid-stream
timeout — consults no longer accumulate server-side under the fixed partition.
- _stream_turn_endpoint: emit a ": keepalive" SSE comment BEFORE the (<=20s) KB
consult so a reverse proxy / EventSource doesn't drop the silent connection into
a false "WIRE LOST" before the turn starts.
- _tts_endpoint: cap text at 8000 chars (413) before the process-global lock;
gateway timeout 120s->60s — one huge/stalled body can't starve all TTS.
- tts_synthesize: validate the WAVE form tag (bytes 8:12), not just the RIFF magic.
- index.html: revoke the audio blob URL in cancelTts (removeAttribute+load fires
neither ended nor error, so the src's own revoke never ran -> per-turn blob leak).
TDD: +11 tests (543 green). Live-smoked on :8765: all five constructible adversarial
inputs now return 200/413/400, never 500.
Adds affect-modulated voice to the web console: the completed assistant
response is spoken on SSE `done`, emotion-modulated by the live PAD the persona
pane already shows (DEC-7 — voice as affect OBSERVABILITY, not chat-app TTS).
- src/ratatoskr/tts.py (new): Zonos-gateway client + PAD→emotion-dial mapping.
tts_synthesize POSTs {input, voice, response_format:"wav", **dials}; wav-only
(DEC-3 — mp3/opus silently return mislabeled PCM). pad_to_dials is total
(None/NaN/out-of-range → valid dials, never raises). TtsUnavailable on any
gateway failure; the single swap seam if we ever move off Zonos.
- web/server.py: POST /api/tts proxy (DEC-4/INV-TTS-1 — the gateway host never
reaches the browser). Per-character voice map (DEC-8: ratatoskr:donut→donut),
serialize lock (DEC-5 — shared 3090), 503 degrade (INV-TTS-4).
- web/static/index.html: 🔊 toggle (opt-in, localStorage, default off,
INV-TTS-2), speak-on-done, AbortController cancel-on-new-turn (INV-TTS-3),
hidden <audio> sink; PAD read off the pane's current snapshot.
- web/entrypoint.py: RATATOSKR_TTS_URL override (the swap seam).
TDD: 17 tts unit tests + 5 endpoint tests (516 green). Live-smoked end-to-end
against the Zonos gateway (:8890): Donut voice + affect dials → 44.1kHz wav,
missing-text→400, neutral→200, gateway-fail→503.
Per docs/contracts/donut_voiced_interview.contract.md (slice 2 of 3).
vh