feat(tts): migrate RP-surface TTS chatterbox-fast → dots-tts
Swap the voice synthesis backend from chatterbox-fast (:8197 bespoke /tts)
to dots-tts (rednote-hilab dots.tts-soar, :8198 OpenAI-shaped
/v1/audio/speech), operator-directed after an A/B win. tts.py stays the
single swap seam.
- gateway body OpenAI-shaped: {input, voice, response_format, stream}
(was chatterbox {text, voice, format, stream})
- sample rate 24000 -> 48000 Hz (browser Web Audio SR)
- default voice glados_25s -> glados; donut voice carries over
- serialized single-consumer (satisfied by the existing DEC-5 lock)
- affect stays dropped (dots has no emotion knob, same as chatterbox)
DOTS_TTS_URL replaces CHATTERBOX_TTS_URL; RATATOSKR_TTS_URL override
unchanged. chatterbox-fast :8197 kept up as rollback. Contract amended
(donut_voiced_interview.contract.md). Live-verified end-to-end on :8765
(RIFF/WAVE 48kHz mono s16le through /api/tts). 520 tests green.
This commit is contained in:
@@ -4,8 +4,9 @@ module: "ratatoskr.web.tts_kb"
|
||||
purpose: >
|
||||
A voiced, corpus-grounded Tier-3 interview character in the ratatoskr web
|
||||
console. Two capabilities plus one character: (a) auto-TTS via the
|
||||
chatterbox-fast gateway, spoken on SSE `done` (migrated off Zonos 2026-08-07;
|
||||
no affect modulation — chatterbox Turbo has no emotion knob); (b) a
|
||||
dots-tts gateway, spoken on SSE `done` (migrated Zonos→chatterbox-fast
|
||||
2026-08-07, then chatterbox-fast→dots-tts 2026-08-10; no affect modulation —
|
||||
dots has no emotion knob); (b) a
|
||||
consumer-side KB-retrieval + `memory_context` pinning
|
||||
BRIDGE that grounds the character's recall in the ingested corpus while she
|
||||
stays in-voice; (c) Princess Donut (Dungeon Crawler Carl) as the first
|
||||
@@ -20,14 +21,14 @@ scope: >
|
||||
consult (an existing agent turn).
|
||||
touches:
|
||||
- src/ratatoskr/web/server.py # /api/tts route + the retrieval-pinning seam on the turn path
|
||||
- src/ratatoskr/web/static/index.html # speak-on-done playback, 🔊 toggle, <audio> sink; turn POST carries agent_id
|
||||
- src/ratatoskr/web/static/index.html # speak-on-done playback (SR 48000), 🔊 toggle, <audio> sink; turn POST carries agent_id
|
||||
- src/ratatoskr/web/entrypoint.py # RATATOSKR_TTS_URL override (the tts swap seam)
|
||||
- src/ratatoskr/tts.py # chatterbox-fast gateway client (was Zonos + PAD->dial; migrated 2026-08-07)
|
||||
- src/ratatoskr/tts.py # dots-tts gateway client (Zonos→chatterbox 2026-08-07→dots 2026-08-10; OpenAI-shaped)
|
||||
- src/ratatoskr/kb_bridge.py # NEW, RETIRE-READY — consumer-side retrieval + memory_context pinning
|
||||
- src/ratatoskr/wt.py # stream_turn gains a memory_context passthrough (seam-review: the contract's original touch list undercounted this by one file; the param defaults None so the bridge's RETIREMENT stays inert — deleting kb_bridge.py + the one call-site leaves wt.stream_turn's SDK-parity param harmless)
|
||||
- docs/characters/donut.md # NEW — Princess Donut persona (content; the tier3 define source)
|
||||
depends_on:
|
||||
- "chatterbox-fast gateway: POST http://10.100.79.3:8197/tts (infra-ops; WG-internal, no auth; bespoke non-OpenAI schema {text,voice,format,stream}; streaming placeholder-header wav @ 24000 Hz; no per-synth cap; NO affect controls; verified 2026-08-07 against image local/chatterbox-fast:v1)"
|
||||
- "dots-tts gateway: POST http://10.100.79.3:8198/v1/audio/speech (infra-ops; WG-internal, no auth; OpenAI-shaped schema {input,voice,response_format,stream}; streaming placeholder-header wav @ 48000 Hz mono s16le; dots streams a whole turn from one call; SERIALIZED single-consumer; zero-shot voice cloning, voices donut/glados/emmie/miranda; NO affect controls; verified 2026-08-10 against dots-studio/dots.tts-soar). chatterbox-fast :8197 kept up as rollback."
|
||||
- "Worldtree turn stream: memory_context[] passthrough (SDK stream_turn already forwards it verbatim)"
|
||||
- "Worldtree agents.define (Tier-3) for Donut; Mimir (search_kb) for the out-of-band retrieval consult"
|
||||
used_by:
|
||||
@@ -41,6 +42,28 @@ confidence: 0.8
|
||||
|
||||
# Contract: Donut voiced interview (auto-TTS + KB-recall bridge)
|
||||
|
||||
> **⚠ TTS MIGRATED chatterbox-fast → dots-tts 2026-08-10 (operator-directed, after an
|
||||
> A/B win).** The synthesis backend moved from chatterbox-fast (:8197 bespoke `/tts`)
|
||||
> to dots-tts (rednote-hilab `dots.tts-soar`, :8198 OpenAI-shaped `/v1/audio/speech`),
|
||||
> verified live. Four deltas; everything else (the streaming placeholder-header WAV
|
||||
> shape, the browser Web-Audio PCM decode path, POST `/api/tts`, the serialize lock,
|
||||
> INV-TTS-1..4) is UNCHANGED:
|
||||
> - **Gateway body OpenAI-shaped.** `{input, voice, response_format:"wav", stream:true}`
|
||||
> — `input` (not chatterbox's `text`), `response_format` (not `format`). Closer to the
|
||||
> Zonos-era client. `tts.py` stays the single swap seam (DEC-1), now translating the
|
||||
> OpenAI schema; `DOTS_TTS_URL` replaces `CHATTERBOX_TTS_URL`.
|
||||
> - **Sample rate 24000 → 48000 Hz.** The browser Web Audio decode MUST use 48000 or the
|
||||
> voice plays ~2× too fast (`index.html` `SR = 48000`).
|
||||
> - **Default voice `glados_25s` → `glados`.** dots voices are donut/glados/emmie/miranda
|
||||
> (GET /v1/voices); `donut` carries over. Non-interview agents fall to `glados`.
|
||||
> - **Serialized single-consumer.** dots renders one generation at a time — satisfied by
|
||||
> the existing DEC-5 lock (no code change). If concurrent streams are ever needed,
|
||||
> infra-ops escalates the backend behind the same API (client unchanged).
|
||||
> Affect stays dropped (DEC-7): dots has no emotion knob, same as chatterbox — NOT a fresh
|
||||
> regression. chatterbox-fast :8197 is kept up as the rollback until dots is confirmed
|
||||
> solid. The 2026-08-07 chatterbox banner + DEC-7/9/9a/10 below are retained as historical
|
||||
> record.
|
||||
|
||||
> **⚠ TTS MIGRATED OFF ZONOS → chatterbox-fast 2026-08-07 (operator-directed).**
|
||||
> Slice 2's synthesis backend moved from the Zonos gateway (:8890
|
||||
> `/v1/audio/speech`) to chatterbox-fast (:8197 `/tts`). Three architecture deltas,
|
||||
@@ -147,13 +170,14 @@ each independently shippable. Slice order is chosen for fastest visible result.
|
||||
record of the Zonos build. (Original: map live PAD from the `affect_update` SSE →
|
||||
Zonos `emotion_valence`/`emotion_arousal`, reframing the feature as voice
|
||||
OBSERVABILITY. The observability framing dies with the knob.)
|
||||
- **DEC-8 — voice: custom "donut" is REGISTERED (amended 2026-08-07 for chatterbox).**
|
||||
chatterbox voices are `*.wav` reference clips in `/refs` (GET /voices lists the
|
||||
stems). infra-ops registered `/refs/donut.wav` (the same reference clip behind the
|
||||
Zonos Donut voice) at operator direction, so `_TTS_VOICE_MAP` maps
|
||||
`ratatoskr:donut → "donut"` directly. NOTE the case: chatterbox wants lowercase
|
||||
`"donut"` (Zonos used `"Donut"`). Non-interview agents fall to the chatterbox
|
||||
default `"glados_25s"` (was Zonos `"Cora"`, which does not exist on chatterbox).
|
||||
- **DEC-8 — voice: custom "donut" is REGISTERED (amended 2026-08-10 for dots).**
|
||||
dots clones a voice server-side from a reference clip + transcript; the client just
|
||||
passes a voice NAME (GET /v1/voices lists them: donut/glados/emmie/miranda). The
|
||||
`donut` voice carries over from chatterbox, so `_TTS_VOICE_MAP` maps
|
||||
`ratatoskr:donut → "donut"` unchanged. NOTE the case: lowercase `"donut"` (Zonos used
|
||||
`"Donut"`). Non-interview agents fall to the dots default `"glados"` (was chatterbox
|
||||
`"glados_25s"` / Zonos `"Cora"`, neither of which exists on dots). New voices are a
|
||||
one-line request to infra-ops (derived from the canonical voice corpus).
|
||||
- **DEC-9 — hold English: RESOLVED SERVER-SIDE 2026-08-07 (client sends full text, default
|
||||
sampling).** The Zonos `language:"en-us"` pin is dropped — chatterbox has no `language` field.
|
||||
The long-turn garble ("swaps to German halfway through") went through two WRONG hypotheses
|
||||
@@ -241,19 +265,19 @@ each independently shippable. Slice order is chosen for fastest visible result.
|
||||
|
||||
## FN blocks
|
||||
|
||||
### FN tts_stream (the sole synthesis primitive — DEC-2 streaming; amended 2026-08-07)
|
||||
### FN tts_stream (the sole synthesis primitive — DEC-2 streaming; amended 2026-08-10 dots)
|
||||
```
|
||||
tts_stream(text, *, voice, client: httpx.AsyncClient, url=CHATTERBOX_TTS_URL) -> AsyncIterator[bytes]
|
||||
tts_stream(text, *, voice, client: httpx.AsyncClient, url=DOTS_TTS_URL) -> AsyncIterator[bytes]
|
||||
# Open the gateway's CHUNKED stream (client.stream("POST", url, json=gateway_body(text, voice))) and
|
||||
# YIELD wav chunks as they synthesize. Pass through verbatim — never buffer, never rewrite the placeholder
|
||||
# header. chatterbox chunks arbitrary-length text INTERNALLY (no per-synth cap, DEC-10 RETIRED), so this
|
||||
# SINGLE call voices a whole turn — no client-side chunk-and-concatenate wrapper.
|
||||
# gateway_body(text, voice) = {text, voice, format:"wav", stream:true}. Full text, DEFAULT sampling — the
|
||||
# :v2 server caps each generation at 250 chars, which fixes the long-turn tail garble (DEC-9). NO dials,
|
||||
# NO language, NO client sampling curbs (a curb was counterproductive — it pulled the garble onset earlier).
|
||||
precondition: text non-empty. Voice membership in GET /voices is GATEWAY-enforced, not client-asserted.
|
||||
# header. dots streams a whole turn from this SINGLE call (DEC-10 RETIRED) — no client-side
|
||||
# chunk-and-concatenate wrapper.
|
||||
# gateway_body(text, voice) = {input, voice, response_format:"wav", stream:true} (OpenAI-shaped: `input`
|
||||
# not `text`, `response_format` not `format`). Full text, DEFAULT sampling. NO dials, NO language,
|
||||
# NO client sampling curbs.
|
||||
precondition: text non-empty. Voice membership in GET /v1/voices is GATEWAY-enforced, not client-asserted.
|
||||
postcondition: yields the gateway's chunked int16 streaming WAV bytes unmodified (0xFFFFFFFF placeholder
|
||||
sizes intact), one leading header then s16le PCM @ 24000 Hz to EOF.
|
||||
sizes intact), one leading header then mono s16le PCM @ 48000 Hz to EOF.
|
||||
error (the yielded_any pivot, folded in from the retired tts_stream_long):
|
||||
- a non-200 OPEN or a connect/transport failure BEFORE the first byte -> TtsUnavailable (so the endpoint
|
||||
peek can still return 503; nothing committed yet).
|
||||
@@ -365,8 +389,8 @@ on SSE `done`:
|
||||
cancelTts() # INV-TTS-3: abort fetch + stop scheduled nodes
|
||||
POST /api/tts {text (sliced to the 8000 cap), agent_id?} -> reader # DEC-10a: POST body. NO p/a (DEC-7 retired).
|
||||
loop: read chunk -> skip ONE WAV header up to the data chunk (bounded 64KiB) -> int16 LE PCM -> Float32 ->
|
||||
AudioBuffer(sampleRate=24000) -> BufferSource.start(playAt) GAPLESSLY -> playAt += buf.duration
|
||||
# SR = 24000 (chatterbox; was 44100 for Zonos — MUST match or the voice plays ~1.8x too fast). TTFA ~0.5s.
|
||||
AudioBuffer(sampleRate=48000) -> BufferSource.start(playAt) GAPLESSLY -> playAt += buf.duration
|
||||
# SR = 48000 (dots; was 24000 for chatterbox — MUST match or the voice plays ~2x too fast). TTFA ~0.5s.
|
||||
first scheduled node -> "▶ voiced". HARD failure (non-OK HTTP, or 64KiB with no WAV header) -> ticker + skip;
|
||||
ABORT/cancel (INV-TTS-3 new-turn) + bare network error -> SILENT skip (INV-TTS-4, cancel is not a failure)
|
||||
|
||||
|
||||
+1
-1
@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
|
||||
|
||||
[project]
|
||||
name = "ratatoskr"
|
||||
version = "0.22.1"
|
||||
version = "0.22.2"
|
||||
description = "Worldtree Conversation API debug console (web + headless CLI) — multi-pane observability"
|
||||
readme = "README.md"
|
||||
requires-python = ">=3.12"
|
||||
|
||||
+39
-46
@@ -1,35 +1,30 @@
|
||||
"""chatterbox-fast gateway TTS client.
|
||||
"""dots-tts gateway TTS client.
|
||||
|
||||
Migrated off the Zonos gateway 2026-08-07 (operator-directed). chatterbox-fast
|
||||
(the Chatterbox TURBO checkpoint, irv-ml1 :8197) is a bespoke, non-OpenAI `/tts`
|
||||
gateway. The migration dropped two whole Zonos-era subsystems:
|
||||
- the PAD->emotion-dial path (Turbo has NO valence/arousal/emotion knob;
|
||||
infra-ops-verified), so voice is now flat; and
|
||||
- the client-side chunk-and-concatenate (chatterbox has no per-synthesis cap
|
||||
and chunks arbitrary-length text internally), so a single `tts_stream` call
|
||||
voices a whole turn.
|
||||
Migrated off chatterbox-fast 2026-08-10 (operator-directed, after an A/B win —
|
||||
"very good"). dots-tts (rednote-hilab `dots.tts-soar`, irv-ml1 :8198) is an
|
||||
OpenAI-shaped `/v1/audio/speech` gateway — closer to the Zonos-era client than
|
||||
chatterbox's bespoke `/tts`. It carries over the two Zonos-era subsystem drops:
|
||||
- NO affect/emotion knob (dots, like chatterbox Turbo, exposes no
|
||||
valence/arousal/emotion dial), so voice stays flat (DEC-7 stays retired); and
|
||||
- NO client-side chunk-and-concatenate — dots streams a whole turn from one call.
|
||||
This module is the SINGLE swap seam for voice synthesis: the `/api/tts` route in
|
||||
web/server.py is its only caller.
|
||||
|
||||
Foot-guns (infra-ops-verified 2026-08-07 against image local/chatterbox-fast:v1):
|
||||
- Sample rate is 24000 Hz (Zonos was 44100). The browser Web Audio decode MUST
|
||||
use 24000 or the voice plays ~1.8x too fast.
|
||||
- format:"wav", stream:true emits a streaming placeholder-header WAV (0xFFFFFFFF
|
||||
RIFF/data sizes, one header, then s16le PCM) — the shape the browser's
|
||||
one-header-strip decoder expects. format:"pcm" (headerless raw s16le) is leaner
|
||||
but would need the browser to drop the header strip; kept on wav so the
|
||||
streaming shape stays byte-identical to the Zonos path.
|
||||
- Body field names are the live pydantic schema: `text` (NOT `input`), `format`
|
||||
(NOT `response_format`). No `model` field, no `language` field (English-only).
|
||||
- Long-turn garble ("swaps to German halfway through", operator 2026-08-07) was NOT a
|
||||
language leak — it was the Turbo model OVER-RUNNING its generation tail (garble/dead-air
|
||||
in the final ~2-3s of a long single generation; infra-ops measured voiced-tail
|
||||
zero-crossing rate at 1.58x the middle). FIXED SERVER-SIDE (image :v2): a
|
||||
`max_chunk_chars=250` cap on the gateway's scheduler bounds each generation below the
|
||||
~300-char degradation onset → clean prosodic chunks. So the client SENDS FULL TEXT and
|
||||
uses the gateway's DEFAULT sampling — an earlier client-side temperature/top_k curb was
|
||||
counterproductive (tight sampling pulls the garble onset to a SHORTER length). Per-request
|
||||
`max_chunk_chars` override is available if per-call tuning is ever wanted.
|
||||
Foot-guns (infra-ops-verified 2026-08-10 against dots-studio/dots.tts-soar :8198):
|
||||
- Sample rate is 48000 Hz (chatterbox was 24000). The browser Web Audio decode
|
||||
MUST use 48000 or the voice plays ~2x too fast.
|
||||
- response_format:"wav", stream:true emits a streaming placeholder-header WAV
|
||||
(RIFF/data sizes as placeholders, one header, then 48kHz mono s16le PCM) — the
|
||||
same shape the browser's one-header-strip decoder already expects. "pcm" (raw
|
||||
s16le, headerless) is leaner but would need the browser to drop the header
|
||||
strip; kept on wav so the streaming shape stays byte-identical to the prior path.
|
||||
- Body field names are OpenAI-shaped: `input` (NOT chatterbox's `text`),
|
||||
`response_format` (NOT chatterbox's `format`). No `model` field, no `language`
|
||||
field, no affect dials.
|
||||
- SERIALIZED single-consumer: dots renders one generation at a time (no concurrent
|
||||
requests). Harmless here — the endpoint already serializes on app.state.tts_lock
|
||||
(DEC-5, one synth per turn on the shared GPU). If concurrent streams are ever
|
||||
needed, infra-ops escalates the backend behind this same API (client unchanged).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
@@ -39,10 +34,9 @@ from collections.abc import AsyncIterator
|
||||
|
||||
import httpx
|
||||
|
||||
# The chatterbox-fast gateway (irv-ml1 :8197). Bespoke `/tts` schema, not OpenAI-shaped,
|
||||
# not the swappable ext-tts alias (which is OpenAI-shaped and can't reach this wire).
|
||||
# The dots-tts gateway (irv-ml1 :8198). OpenAI-shaped `/v1/audio/speech`.
|
||||
# Overridable per deployment via app.state.tts_url (RATATOSKR_TTS_URL) — the swap seam + tests.
|
||||
CHATTERBOX_TTS_URL = "http://10.100.79.3:8197/tts"
|
||||
DOTS_TTS_URL = "http://10.100.79.3:8198/v1/audio/speech"
|
||||
|
||||
|
||||
class TtsUnavailable(Exception):
|
||||
@@ -59,13 +53,12 @@ class TtsUnavailable(Exception):
|
||||
|
||||
|
||||
def gateway_body(text: str, voice: str) -> dict:
|
||||
"""The chatterbox-fast POST body. `format:"wav"` (DEC-3) so the streaming shape is a
|
||||
placeholder-header WAV the browser decoder strips; `stream:true` for play-as-it-arrives
|
||||
(DEC-2). Field names are the live pydantic schema: `text` (not `input`), `format` (not
|
||||
`response_format`). Send the FULL text with the gateway's DEFAULT sampling: the :v2 server
|
||||
caps each generation at 250 chars, which fixes the long-turn tail garble (DEC-9); a client
|
||||
sampling curb was counterproductive. No `language` (English-only), no affect dials (DEC-7)."""
|
||||
return {"text": text, "voice": voice, "format": "wav", "stream": True}
|
||||
"""The dots-tts POST body (OpenAI-shaped). `response_format:"wav"` (DEC-3) so the
|
||||
streaming shape is a placeholder-header WAV the browser decoder strips; `stream:true`
|
||||
for play-as-it-arrives (DEC-2). Field names are OpenAI: `input` (not `text`),
|
||||
`response_format` (not `format`). Send the FULL text — dots streams a whole turn from
|
||||
one call. No `language`, no affect dials (DEC-7)."""
|
||||
return {"input": text, "voice": voice, "response_format": "wav", "stream": True}
|
||||
|
||||
|
||||
async def tts_stream(
|
||||
@@ -73,18 +66,18 @@ async def tts_stream(
|
||||
*,
|
||||
voice: str,
|
||||
client: httpx.AsyncClient,
|
||||
url: str = CHATTERBOX_TTS_URL,
|
||||
url: str = DOTS_TTS_URL,
|
||||
) -> AsyncIterator[bytes]:
|
||||
"""Open the gateway's CHUNKED stream and yield WAV bytes as they synthesize.
|
||||
|
||||
format:"wav"/stream:true emits a streaming int16 WAV (RIFF/data sizes = 0xFFFFFFFF
|
||||
placeholders) over `transfer-encoding: chunked`, first byte well under 1s (infra-ops) —
|
||||
designed to be played progressively by the browser Web Audio path. So we PROXY THE CHUNKS
|
||||
STRAIGHT THROUGH: never buffer, never rewrite the header. chatterbox chunks arbitrary-length
|
||||
text internally (no per-synthesis cap), so this SINGLE call voices a whole turn — there is
|
||||
no client-side chunk-and-concatenate wrapper.
|
||||
response_format:"wav"/stream:true emits a streaming int16 WAV (RIFF/data sizes =
|
||||
placeholders) over `transfer-encoding: chunked`, streaming ahead of real-time
|
||||
(RTF ~0.22, infra-ops) — designed to be played progressively by the browser Web Audio
|
||||
path. So we PROXY THE CHUNKS STRAIGHT THROUGH: never buffer, never rewrite the header.
|
||||
dots streams a whole turn from this SINGLE call — there is no client-side
|
||||
chunk-and-concatenate wrapper.
|
||||
|
||||
The error policy (the `yielded_any` pivot, folded in from the retired tts_stream_long):
|
||||
The error policy (the `yielded_any` pivot):
|
||||
- a non-200 OPEN, or a connect/transport failure BEFORE the first byte, raises
|
||||
TtsUnavailable so the endpoint peek can still return a 503 (INV-TTS-4) — nothing
|
||||
committed yet.
|
||||
|
||||
@@ -80,9 +80,9 @@ def main(argv: list[str] | None = None) -> int:
|
||||
# never receives the key, only the session-filtered result.
|
||||
admin_key = os.environ.get("RATATOSKR_ADMIN_API_KEY")
|
||||
|
||||
# Auto-TTS (slice 2): the chatterbox-fast gateway URL. Defaults to the direct
|
||||
# Auto-TTS (slice 2): the dots-tts gateway URL. Defaults to the direct
|
||||
# gateway (DEC-1) inside the server; override here only to point at a different
|
||||
# synth host (the swap seam). None → the server's CHATTERBOX_TTS_URL default.
|
||||
# synth host (the swap seam). None → the server's DOTS_TTS_URL default.
|
||||
tts_url = os.environ.get("RATATOSKR_TTS_URL")
|
||||
|
||||
# INV-001: lazy import. Users without [web] extras get a clean hint
|
||||
|
||||
+20
-19
@@ -66,7 +66,7 @@ from ratatoskr.sse_client import (
|
||||
TurnIdFlip,
|
||||
)
|
||||
from ratatoskr.tts import (
|
||||
CHATTERBOX_TTS_URL,
|
||||
DOTS_TTS_URL,
|
||||
TtsUnavailable,
|
||||
tts_stream,
|
||||
)
|
||||
@@ -543,15 +543,15 @@ async def _memory_chunks_endpoint(request: Request) -> JSONResponse:
|
||||
|
||||
|
||||
# Per-character voice map (DEC-8): interview characters resolve to their registered
|
||||
# chatterbox reference clip (/refs/<name>.wav); everything else falls to the gateway default.
|
||||
# NOTE the case: chatterbox wants lowercase "donut" (Zonos used "Donut").
|
||||
# dots voice name (GET /v1/voices lists them); everything else falls to the gateway default.
|
||||
# NOTE the case: dots wants lowercase "donut" (Zonos used "Donut").
|
||||
_TTS_VOICE_MAP = {"ratatoskr:donut": "donut"}
|
||||
_TTS_DEFAULT_VOICE = "glados_25s" # the chatterbox default (Zonos "Cora" does not exist here)
|
||||
_TTS_DEFAULT_VOICE = "glados" # a dots voice (chatterbox "glados_25s" / Zonos "Cora" do not exist here)
|
||||
# The text rides the POST body (DEC-10a), so URL length is not the bound — this is a safety
|
||||
# ceiling on the shared-GPU hold. chatterbox has no per-synth cap and chunks arbitrary-length
|
||||
# text internally, so a single call voices the whole turn; ~8000 chars still covers any real
|
||||
# interview turn while bounding a runaway. A response past this is truncated at a word boundary
|
||||
# (the full text still shows in the transcript).
|
||||
# ceiling on the shared-GPU hold. dots streams a whole turn from one call (no client concat),
|
||||
# so a single call voices the whole turn; ~8000 chars still covers any real interview turn
|
||||
# while bounding a runaway. A response past this is truncated at a word boundary (the full
|
||||
# text still shows in the transcript).
|
||||
_TTS_MAX_TEXT_CHARS = 8000
|
||||
|
||||
|
||||
@@ -565,11 +565,11 @@ def _truncate_at_boundary(text: str, limit: int) -> str:
|
||||
|
||||
|
||||
async def _tts_endpoint(request: Request) -> Response:
|
||||
"""POST /api/tts {text, agent_id?} → audio/wav, STREAMED chunked from the chatterbox-fast
|
||||
"""POST /api/tts {text, agent_id?} → audio/wav, STREAMED chunked from the dots-tts
|
||||
gateway (FN tts_endpoint). POST (not GET) so an arbitrarily long turn rides the body, not a
|
||||
length-capped URL. chatterbox chunks arbitrary-length text internally (no per-synth cap), so
|
||||
a single tts_stream call proxies the whole turn — bytes straight through (one leading WAV
|
||||
header + s16le PCM), and the browser decodes one gapless stream.
|
||||
length-capped URL. dots streams a whole turn from one call, so a single tts_stream call
|
||||
proxies it — bytes straight through (one leading WAV header + s16le PCM @ 48kHz), and the
|
||||
browser decodes one gapless stream.
|
||||
|
||||
Server-side proxy (DEC-4 / INV-TTS-1: the gateway host never reaches the browser).
|
||||
Voice per-character (DEC-8). No affect modulation — DEC-7 retired with the Zonos migration.
|
||||
@@ -635,10 +635,10 @@ async def _tts_endpoint(request: Request) -> Response:
|
||||
await _release()
|
||||
raise
|
||||
|
||||
# An empty 200 body (no bytes at all) is a synthesis failure, not audio: infra-ops
|
||||
# (2026-08-07) — chatterbox returns 200 with a 0-byte body when a long single generation
|
||||
# OOMs the shared 3090. Surface it as a 503 (INV-TTS-4 visible skip) rather than committing
|
||||
# a silent, empty audio/wav StreamingResponse the browser would play as nothing.
|
||||
# An empty 200 body (no bytes at all) is a synthesis failure, not audio: a gateway that
|
||||
# returns 200 with a 0-byte body (a stalled/failed generation) must not commit a silent,
|
||||
# empty audio/wav StreamingResponse the browser would play as nothing. Surface it as a 503
|
||||
# (INV-TTS-4 visible skip). Kept as hygiene regardless of backend (infra-ops-endorsed).
|
||||
if first is None:
|
||||
await _release()
|
||||
return JSONResponse({"error_code": "tts_unavailable"}, status_code=503)
|
||||
@@ -898,11 +898,12 @@ def create_app(
|
||||
# SERVER-HELD (RATATOSKR_ADMIN_API_KEY) and never reaches the browser — the
|
||||
# server proxies admin-scoped reads and forwards only the session-filtered result.
|
||||
app.state.admin_key = admin_key
|
||||
# Auto-TTS (slice 2): the chatterbox-fast gateway URL is SERVER-HELD config — the host
|
||||
# Auto-TTS (slice 2): the dots-tts gateway URL is SERVER-HELD config — the host
|
||||
# never reaches the browser (DEC-4 / INV-TTS-1). Defaults to the direct gateway
|
||||
# (DEC-1); overridable via RATATOSKR_TTS_URL (the swap seam). The lock serializes
|
||||
# one synth at a time so concurrent turns don't contend the shared 3090 (DEC-5).
|
||||
app.state.tts_url = tts_url or CHATTERBOX_TTS_URL
|
||||
# one synth at a time — required by dots (single-consumer) and so concurrent turns
|
||||
# don't contend the shared GPU (DEC-5).
|
||||
app.state.tts_url = tts_url or DOTS_TTS_URL
|
||||
app.state.tts_lock = asyncio.Lock()
|
||||
# INV-002: turn registry is in-process memory, keyed (session_id, turn_id)
|
||||
app.state.turn_registry = {}
|
||||
|
||||
@@ -1994,7 +1994,7 @@ async function cancelTurn() {
|
||||
});
|
||||
})();
|
||||
|
||||
// ---- auto-TTS: voiced STREAMING playback via Web Audio (chatterbox-fast, 24kHz) ------
|
||||
// ---- auto-TTS: voiced STREAMING playback via Web Audio (dots-tts, 48kHz) ------
|
||||
// Fetch the chunked POST /api/tts stream, decode its int16 PCM, and schedule the samples
|
||||
// GAPLESSLY into an AudioContext as they arrive (TTFA ~0.5s). Web Audio, NOT <audio src>,
|
||||
// because Safari/WebKit REFUSES a streaming 0xFFFFFFFF-length WAV via <audio src>
|
||||
@@ -2043,7 +2043,7 @@ async function speakOnDone(text, agentId) {
|
||||
if (!ctx) { tickerAdd("err", "tts", "no audio ctx"); return; }
|
||||
if (ctx.state === "suspended") { try { await ctx.resume(); } catch (_) {} }
|
||||
const payload = { text: clip.slice(0, 8000) }; // matches the server cap; chunked server-side by the gateway
|
||||
if (agentId) payload.agent_id = agentId; // no affect fields — DEC-7 retired (chatterbox has no emotion knob)
|
||||
if (agentId) payload.agent_id = agentId; // no affect fields — DEC-7 retired (dots has no emotion knob)
|
||||
const ctrl = new AbortController(); _ttsAbort = ctrl;
|
||||
let resp;
|
||||
// POST (not GET) so a long turn rides the body, not a length-capped URL (DEC-10).
|
||||
@@ -2057,7 +2057,7 @@ async function speakOnDone(text, agentId) {
|
||||
} catch (_) { return; } // aborted / network → silent skip (INV-TTS-4)
|
||||
if (!resp.ok || !resp.body) { tickerAdd("err", "tts", "unavailable " + resp.status); return; }
|
||||
const reader = resp.body.getReader();
|
||||
const SR = 24000; // chatterbox-fast sample rate (was 44100 for Zonos — MUST match or the voice plays ~1.8x too fast)
|
||||
const SR = 48000; // dots-tts sample rate (was 24000 for chatterbox — MUST match or the voice plays ~2x too fast)
|
||||
let playAt = ctx.currentTime + 0.06, started = false, headerDone = false;
|
||||
let acc = new Uint8Array(0), carry = new Uint8Array(0);
|
||||
try {
|
||||
|
||||
+23
-25
@@ -1,10 +1,10 @@
|
||||
"""Tests for ratatoskr.tts — the STREAMING chatterbox-fast gateway client.
|
||||
"""Tests for ratatoskr.tts — the STREAMING dots-tts gateway client.
|
||||
|
||||
tts_stream proxies the gateway's chunked response verbatim (no buffering, no header
|
||||
rewrite — the placeholder-size streaming WAV is meant to be played progressively). It
|
||||
is the sole synthesis primitive: chatterbox chunks arbitrary-length text internally, so
|
||||
there is no client-side chunk-and-concatenate (retired with the Zonos migration), and no
|
||||
affect dials (Turbo has no emotion knob). The mid-stream degrade policy is folded in.
|
||||
is the sole synthesis primitive: dots streams a whole turn from one call, so there is
|
||||
no client-side chunk-and-concatenate, and no affect dials (dots has no emotion knob).
|
||||
The mid-stream degrade policy is folded in.
|
||||
"""
|
||||
|
||||
import httpx
|
||||
@@ -12,13 +12,13 @@ import pytest
|
||||
import respx
|
||||
|
||||
from ratatoskr.tts import (
|
||||
CHATTERBOX_TTS_URL,
|
||||
DOTS_TTS_URL,
|
||||
TtsUnavailable,
|
||||
gateway_body,
|
||||
tts_stream,
|
||||
)
|
||||
|
||||
_URL = "http://tts.example/tts"
|
||||
_URL = "http://tts.example/v1/audio/speech"
|
||||
# The gateway's streaming WAV bytes (placeholder 0xFFFFFFFF sizes). We pass them through
|
||||
# untouched, so the content only has to round-trip.
|
||||
_WAV = (
|
||||
@@ -50,34 +50,32 @@ class _RaisingByteStream(httpx.AsyncByteStream):
|
||||
|
||||
|
||||
class TestGatewayBody:
|
||||
def test_bespoke_chatterbox_schema(self) -> None:
|
||||
def test_openai_dots_schema(self) -> None:
|
||||
b = gateway_body("hi", "donut")
|
||||
assert b["text"] == "hi" # "text", not "input"
|
||||
assert b["input"] == "hi" # OpenAI "input", not chatterbox "text"
|
||||
assert b["voice"] == "donut"
|
||||
assert b["format"] == "wav" # DEC-3 — "format", not "response_format"
|
||||
assert b["stream"] is True # DEC-2 — play-as-it-arrives
|
||||
assert b["response_format"] == "wav" # DEC-3 — "response_format", not "format"
|
||||
assert b["stream"] is True # DEC-2 — play-as-it-arrives
|
||||
|
||||
def test_default_sampling_no_client_side_curbs(self) -> None:
|
||||
# DEC-9 (real cause): the long-turn garble was Turbo over-running its GENERATION TAIL,
|
||||
# fixed SERVER-SIDE (:v2 max_chunk_chars=250). A client sampling curb was
|
||||
# counterproductive (tight sampling pulls the garble onset earlier), so gateway_body
|
||||
# sends NO temperature/top_p/top_k — the gateway's defaults govern.
|
||||
# No client sampling curbs — the gateway's defaults govern (a client-side curb
|
||||
# was counterproductive on the prior backend and dots exposes no such need).
|
||||
b = gateway_body("a long turn", "donut")
|
||||
for knob in ("temperature", "top_p", "top_k"):
|
||||
assert knob not in b
|
||||
|
||||
def test_no_zonos_era_fields(self) -> None:
|
||||
# The Zonos body fields are gone: no OpenAI `input`/`response_format`, no
|
||||
# `language` pin (DEC-9 retired), no affect dials (DEC-7 retired).
|
||||
b = gateway_body("hi", "Cora")
|
||||
for dead in ("input", "response_format", "language", "emotion_valence",
|
||||
def test_no_chatterbox_or_zonos_era_fields(self) -> None:
|
||||
# The chatterbox bespoke names + Zonos-era fields are gone: no `text`/`format`
|
||||
# (chatterbox), no `language` pin, no affect dials (DEC-7 retired).
|
||||
b = gateway_body("hi", "glados")
|
||||
for dead in ("text", "format", "language", "emotion_valence",
|
||||
"emotion_arousal", "emotion_enabled", "emotion_strength"):
|
||||
assert dead not in b
|
||||
|
||||
|
||||
class TestTtsStream:
|
||||
@respx.mock
|
||||
async def test_streams_chunks_and_posts_bespoke_body(self) -> None:
|
||||
async def test_streams_chunks_and_posts_openai_body(self) -> None:
|
||||
route = respx.post(_URL).mock(return_value=httpx.Response(200, content=_WAV))
|
||||
async with httpx.AsyncClient() as client:
|
||||
out = await _drain(tts_stream("hello there", voice="donut", client=client, url=_URL))
|
||||
@@ -85,19 +83,19 @@ class TestTtsStream:
|
||||
import json as _json
|
||||
|
||||
body = _json.loads(route.calls.last.request.content)
|
||||
assert body["text"] == "hello there"
|
||||
assert body["input"] == "hello there"
|
||||
assert body["voice"] == "donut"
|
||||
assert body["format"] == "wav"
|
||||
assert body["response_format"] == "wav"
|
||||
assert body["stream"] is True
|
||||
|
||||
@respx.mock
|
||||
async def test_default_url_is_chatterbox(self) -> None:
|
||||
route = respx.post(CHATTERBOX_TTS_URL).mock(
|
||||
async def test_default_url_is_dots(self) -> None:
|
||||
route = respx.post(DOTS_TTS_URL).mock(
|
||||
return_value=httpx.Response(200, content=_WAV)
|
||||
)
|
||||
async with httpx.AsyncClient() as client:
|
||||
await _drain(tts_stream("hi", voice="donut", client=client))
|
||||
assert route.called # the module default points at the chatterbox gateway
|
||||
assert route.called # the module default points at the dots-tts gateway
|
||||
|
||||
@respx.mock
|
||||
async def test_non_200_open_raises_before_any_chunk(self) -> None:
|
||||
|
||||
+17
-17
@@ -1459,14 +1459,14 @@ class TestMemoryChunksEndpoint:
|
||||
|
||||
class TestTtsEndpoint:
|
||||
"""tts_endpoint FN — POST /api/tts → audio/wav STREAMED (chunked) from the
|
||||
chatterbox-fast gateway. Voice per-character (DEC-8), the gateway host never reaches
|
||||
dots-tts gateway. Voice per-character (DEC-8), the gateway host never reaches
|
||||
the browser (INV-TTS-1), gateway open-failure → 503 (INV-TTS-4). POST so an
|
||||
arbitrarily long turn rides the body; the gateway chunks internally (DEC-10 retired —
|
||||
no client concat). No affect: p/a body fields are gone (DEC-7 retired)."""
|
||||
arbitrarily long turn rides the body; dots streams a whole turn from one call (no
|
||||
client concat). No affect: p/a body fields are gone (DEC-7 retired)."""
|
||||
|
||||
# Streaming WAV bytes (placeholder 0xFFFFFFFF sizes) — proxied through verbatim.
|
||||
_WAV = b"RIFF\xff\xff\xff\xffWAVEdata\xff\xff\xff\xff" + b"\x11\x22" * 64
|
||||
_TTS = "http://tts.example/tts"
|
||||
_TTS = "http://tts.example/v1/audio/speech"
|
||||
|
||||
@respx.mock
|
||||
def test_happy_streams_wav_resolves_donut_voice(self) -> None:
|
||||
@@ -1484,12 +1484,12 @@ class TestTtsEndpoint:
|
||||
assert resp.headers["content-type"].startswith("audio/wav")
|
||||
assert resp.content == self._WAV # streamed through verbatim
|
||||
body = json.loads(route.calls.last.request.content)
|
||||
assert body["text"] == "Carl is a softie." # "text", not "input"
|
||||
assert body["input"] == "Carl is a softie." # OpenAI "input", not chatterbox "text"
|
||||
assert body["voice"] == "donut"
|
||||
assert body["format"] == "wav" # "format", not "response_format"
|
||||
assert body["response_format"] == "wav" # OpenAI "response_format", not "format"
|
||||
assert body["stream"] is True
|
||||
# No Zonos-era fields ride the body.
|
||||
for dead in ("input", "response_format", "language", "emotion_valence"):
|
||||
# No chatterbox-bespoke or Zonos-era fields ride the body.
|
||||
for dead in ("text", "format", "language", "emotion_valence"):
|
||||
assert dead not in body
|
||||
|
||||
@respx.mock
|
||||
@@ -1503,7 +1503,7 @@ class TestTtsEndpoint:
|
||||
resp = TestClient(app).post("/api/tts", json={"text": "hello", "agent_id": "mimir"})
|
||||
assert resp.status_code == 200
|
||||
body = json.loads(route.calls.last.request.content)
|
||||
assert body["voice"] == "glados_25s" # chatterbox default (DEC-8)
|
||||
assert body["voice"] == "glados" # dots default voice (DEC-8)
|
||||
|
||||
def test_missing_text_returns_400(self) -> None:
|
||||
from ratatoskr.web.server import create_app
|
||||
@@ -1525,8 +1525,8 @@ class TestTtsEndpoint:
|
||||
|
||||
@respx.mock
|
||||
def test_long_text_is_one_gateway_call_not_chunked(self) -> None:
|
||||
# DEC-10 RETIRED: chatterbox chunks internally, so a long turn is ONE gateway call
|
||||
# with the full text — no client-side chunk-and-concatenate.
|
||||
# DEC-10 RETIRED: dots streams a whole turn from one call, so a long turn is ONE
|
||||
# gateway call with the full text — no client-side chunk-and-concatenate.
|
||||
from ratatoskr.web.server import create_app
|
||||
|
||||
route = respx.post(self._TTS).mock(
|
||||
@@ -1538,7 +1538,7 @@ class TestTtsEndpoint:
|
||||
assert resp.status_code == 200
|
||||
assert len(route.calls) == 1 # one call, not chunk-and-concatenate
|
||||
body = json.loads(route.calls.last.request.content)
|
||||
assert body["text"] == text # full text in one call, unsplit
|
||||
assert body["input"] == text # full text in one call, unsplit
|
||||
|
||||
@respx.mock
|
||||
def test_non_str_agent_id_degrades_no_500(self) -> None:
|
||||
@@ -1552,7 +1552,7 @@ class TestTtsEndpoint:
|
||||
resp = TestClient(app).post("/api/tts", json={"text": "hi", "agent_id": ["donut"]})
|
||||
assert resp.status_code == 200
|
||||
body = json.loads(route.calls.last.request.content)
|
||||
assert body["voice"] == "glados_25s" # non-str agent_id → default voice, not a 500
|
||||
assert body["voice"] == "glados" # non-str agent_id → default voice, not a 500
|
||||
|
||||
def test_whitespace_text_returns_400(self) -> None:
|
||||
from ratatoskr.web.server import create_app
|
||||
@@ -1596,13 +1596,13 @@ class TestTtsEndpoint:
|
||||
)
|
||||
assert resp.status_code == 200
|
||||
body = json.loads(route.calls.last.request.content)
|
||||
assert body["text"] == "hello" # surrogate dropped, rest intact — not a 500
|
||||
assert body["input"] == "hello" # surrogate dropped, rest intact — not a 500
|
||||
|
||||
@respx.mock
|
||||
def test_empty_200_body_rejected_503(self) -> None:
|
||||
# infra-ops (2026-08-07): chatterbox returns 200 with a 0-byte body when a long single
|
||||
# generation OOMs the shared 3090. An empty 200 is a synthesis failure, not silent audio
|
||||
# — surface it as 503 (INV-TTS-4) rather than committing an empty audio/wav stream.
|
||||
# A gateway that returns 200 with a 0-byte body (a stalled/failed generation) is a
|
||||
# synthesis failure, not silent audio — surface it as 503 (INV-TTS-4) rather than
|
||||
# committing an empty audio/wav stream. Backend-agnostic hygiene (infra-ops-endorsed).
|
||||
from ratatoskr.web.server import create_app
|
||||
|
||||
respx.post(self._TTS).mock(return_value=httpx.Response(200, content=b""))
|
||||
|
||||
Reference in New Issue
Block a user