feat: Donut voiced-interview slice-1 (contract + persona + define) + /snapshot
Slice 1 of the auto-TTS/voiced-KB-character build (operator ask "add auto-tts to the web gui"): the donut_voiced_interview contract (validated), the Princess Donut persona (corpus-grounded from a Mimir DCC pull), and ratatoskr:donut defined on :8081 (server-side; in the picker). Slices 2 (Zonos auto-TTS) + 3 (retire-ready KB-bridge) are TO BUILD. Snapshot captures the full build state + design (Zonos gateway :8890, voice "donut" registered, affect-driven emotion dials; the worldtree-dev-ruled consumer-side retrieval + memory_context pinning bridge, retire-ready) for the post-clear resume, plus the arcs since v0.22.0 (SDK 1.1.2 repin, bifrost 1.1.5, canonical sync, release-only versioning, the Sindra saga + local-index schema-burial foot-gun, the Mimir #382 reference-consumer finding). Handoff at /tmp/ratatoskr-dev-handoff.md. Release-only cadence: no tag.
This commit is contained in:
@@ -0,0 +1,175 @@
|
||||
---
|
||||
contract_version: "2.1"
|
||||
module: "ratatoskr.web.tts_kb"
|
||||
purpose: >
|
||||
A voiced, corpus-grounded Tier-3 interview character in the ratatoskr web
|
||||
console. Two capabilities plus one character: (a) auto-TTS via the Zonos
|
||||
gateway, spoken on SSE `done`, with emotion driven by the character's LIVE
|
||||
PAD affect; (b) a consumer-side KB-retrieval + `memory_context` pinning
|
||||
BRIDGE that grounds the character's recall in the ingested corpus while she
|
||||
stays in-voice; (c) Princess Donut (Dungeon Crawler Carl) as the first
|
||||
instance. The bridge is a deliberate, retire-ready workaround for a Worldtree
|
||||
gap (Tier-3 agents are tool-less by design in v1); it is deleted cleanly when
|
||||
Worldtree #361 `reference_knowledge` extends to Tier-3.
|
||||
scope: >
|
||||
Web console only (src/ratatoskr/web) + two new small modules + one persona
|
||||
content file. The Bifrost provider planes and the ratatoskr.wt SDK-adapter
|
||||
core are UNTOUCHED. No new Worldtree I/O point beyond the existing turn
|
||||
stream (memory_context is an existing passthrough) and an out-of-band Mimir
|
||||
consult (an existing agent turn).
|
||||
touches:
|
||||
- src/ratatoskr/web/server.py # /api/tts route + the retrieval-pinning seam on the turn path
|
||||
- src/ratatoskr/web/static/index.html # speak-on-done playback, 🔊 toggle, <audio> sink
|
||||
- src/ratatoskr/tts.py # NEW — Zonos gateway client + PAD->emotion-dial mapping
|
||||
- src/ratatoskr/kb_bridge.py # NEW, RETIRE-READY — consumer-side retrieval + memory_context pinning
|
||||
- docs/characters/donut.md # NEW — Princess Donut persona (content; the tier3 define source)
|
||||
depends_on:
|
||||
- "Zonos gateway: POST http://10.100.79.3:8890/v1/audio/speech (infra-ops; WG-internal, no auth; wav; verified 2026-08-02)"
|
||||
- "Worldtree turn stream: memory_context[] passthrough (SDK stream_turn already forwards it verbatim)"
|
||||
- "Worldtree agents.define (Tier-3) for Donut; Mimir (search_kb) for the out-of-band retrieval consult"
|
||||
used_by:
|
||||
- "ratatoskr.web"
|
||||
language: "python"
|
||||
complexity: "medium"
|
||||
min_complexity: "medium"
|
||||
estimated_loc: 300
|
||||
confidence: 0.8
|
||||
---
|
||||
|
||||
# Contract: Donut voiced interview (auto-TTS + KB-recall bridge)
|
||||
|
||||
Migration-style contract: three separable slices (persona / TTS / KB-bridge),
|
||||
each independently shippable. Slice order is chosen for fastest visible result.
|
||||
|
||||
## Decisions (DEC)
|
||||
|
||||
- **DEC-1 — direct :8890 coupling.** Proxy straight to the Zonos gateway, not the
|
||||
swappable `ext-tts` LiteLLM alias. Rationale: the emotion dials (the whole
|
||||
point — affect-driven voice) don't pass through `ext-tts`. Accept the
|
||||
Zonos coupling; the `tts.py` client is the single swap seam if we ever move.
|
||||
- **DEC-2 — full-synth latency accepted (no true streaming).** The gateway
|
||||
buffers to a complete clip (~1.8s/sentence, scales). "Speak on done" gives the
|
||||
whole clip after a short delay. True first-audio-early is a future gateway
|
||||
enhancement (infra-ops to expose the native PCM stream); not in v1.
|
||||
- **DEC-3 — wav only.** `response_format:"wav"` (16-bit RIFF). `mp3`/`opus` are
|
||||
accepted but silently return mislabeled PCM — never request them.
|
||||
- **DEC-4 — server-side proxy.** Browser → `/api/tts` (nh3-dev) → gateway. The
|
||||
irv-ml1 host/URL never reaches the client (INV-TTS-1). No key exists, so
|
||||
INV-003 is trivially satisfied, but the proxy still stands (browser can't
|
||||
reach the WG net regardless).
|
||||
- **DEC-5 — serialize one synth per turn.** The gateway shares a single 3090
|
||||
(~90% VRAM) with other TTS engines; do not parallelize. A new turn cancels any
|
||||
in-flight synth + playback.
|
||||
- **DEC-6 — KB-recall = consumer-side retrieval + memory_context pinning, as a
|
||||
RETIRE-READY bridge (worldtree-dev ruling, wyrd-proven pattern).** Tier-3 is
|
||||
tool-less by design (`build_tier3_agent_context` hard-codes `tool_schemas=[]`);
|
||||
agent.call:mimir is rejected (recursion). So per interview turn: ratatoskr
|
||||
consults Mimir out-of-band for the question's topic, extracts the cited
|
||||
evidence, and POSTs Donut's turn with `memory_context=[{kind:"corpus_reference",
|
||||
text:"<extract>"}]`. She answers in-voice, framing it as her own memory. The
|
||||
bridge is isolated behind ONE seam (INV-KB-1) so it deletes cleanly when
|
||||
Worldtree #361 `reference_knowledge` extends to Tier-3 (operator-flagged as an
|
||||
integral gap; worldtree-dev surfacing the extension to Vuong).
|
||||
- **DEC-7 — affect-driven emotion.** Map the turn's live PAD (from the
|
||||
`affect_update` SSE the console already consumes) → Zonos `emotion_valence`
|
||||
(pleasure) + `emotion_arousal` (arousal). This reframes the feature as voice
|
||||
OBSERVABILITY (hear the affect the persona pane shows), not chat-app TTS.
|
||||
- **DEC-8 — voice: preset now, custom later.** Donut starts on a theatrical
|
||||
Zonos preset (of the 8; pick expressive — Miranda/Penny/Emmie). A custom
|
||||
"donut" voice registers server-side via infra-ops later; adoption is a single
|
||||
`voice:` string swap — no code change.
|
||||
|
||||
## Invariants
|
||||
|
||||
- **INV-TTS-1 [hard]** — the Zonos gateway host/URL never reaches the browser;
|
||||
all synthesis goes through `/api/tts`.
|
||||
- **INV-TTS-2 [hard]** — TTS is opt-in: a 🔊 toggle (default OFF), persisted to
|
||||
localStorage (mirrors the theme/cot-toggle pattern). No speech without it.
|
||||
- **INV-TTS-3 [hard]** — one synth in flight at a time; a new turn cancels the
|
||||
prior synth request AND stops in-flight playback (cancel-on-new-turn).
|
||||
- **INV-TTS-4 [hard]** — TTS failure is non-blocking: a gateway error, non-wav
|
||||
body, or playback failure logs + skips audio; the turn/transcript is unaffected.
|
||||
- **INV-KB-1 [hard]** — the KB bridge is import-isolated behind a single seam:
|
||||
`server.py`'s turn path calls exactly one function `pin_kb_context(question,
|
||||
agent_id) -> list[memory_context] | []`. Retiring the bridge = delete
|
||||
`kb_bridge.py` + that one call-site; no other file changes. `kb_bridge` imports
|
||||
nothing from the TTS or SDK-adapter core.
|
||||
- **INV-KB-2 [hard]** — the retrieval channel is never persisted: `memory_context`
|
||||
is per-turn, `kind="corpus_reference"` (a free label), and rides the turn POST
|
||||
body only.
|
||||
- **INV-KB-3 [hard]** — KB-retrieval failure is non-blocking: a failed/empty
|
||||
Mimir consult pins nothing (`[]`) and Donut answers without the extract; never
|
||||
block or fail her turn on retrieval.
|
||||
- **INV-KB-4** — the bridge is scoped to characters that opt in (a per-agent
|
||||
allowlist / config), not applied to every turn — only interview characters
|
||||
that need corpus recall pay the retrieval round-trip.
|
||||
|
||||
## FN blocks
|
||||
|
||||
### FN tts_synthesize
|
||||
```
|
||||
tts_synthesize(text: str, *, voice: str, dials: EmotionDials, client: httpx.AsyncClient) -> bytes
|
||||
# POST {input:text, voice, response_format:"wav", **dials} to the Zonos gateway; return wav bytes.
|
||||
precondition: text non-empty; voice in the gateway's /v1/voices set.
|
||||
postcondition: returns 16-bit RIFF/WAVE bytes.
|
||||
error: gateway non-200 / transport failure -> TtsUnavailable (caller degrades per INV-TTS-4).
|
||||
invariant: response_format is ALWAYS "wav" (DEC-3); never mp3/opus.
|
||||
```
|
||||
|
||||
### FN pad_to_dials
|
||||
```
|
||||
pad_to_dials(pad: PadState | None) -> EmotionDials
|
||||
# Map live PAD -> Zonos emotion dials (DEC-7).
|
||||
steps:
|
||||
- none/malformed pad -> neutral dials (emotion_enabled=False) [degrade-not-crash].
|
||||
- emotion_valence = clamp(pad.pleasure, -1, 1); emotion_arousal = clamp(pad.arousal, -1, 1).
|
||||
- emotion_enabled = True; emotion_strength from a fixed default (tunable).
|
||||
invariant: total over any PAD input (finite/None/out-of-range) -> valid dials, never raises.
|
||||
```
|
||||
|
||||
### FN tts_endpoint (server.py, /api/tts)
|
||||
```
|
||||
POST /api/tts {text, agent_id?} -> audio/wav
|
||||
steps:
|
||||
- resolve voice (per-character map -> preset; default Cora) + dials (pad_to_dials of the agent's current PAD if known).
|
||||
- tts_synthesize(...) behind the serialize guard (DEC-5); return wav with Content-Type audio/wav.
|
||||
- on TtsUnavailable -> 503 controlled envelope (client skips playback, INV-TTS-4).
|
||||
```
|
||||
|
||||
### FN pin_kb_context (kb_bridge.py — RETIRE-READY, INV-KB-1)
|
||||
```
|
||||
pin_kb_context(question: str, agent_id: str, *, client) -> list[dict] # memory_context items, or []
|
||||
# The bridge. Consumer-side retrieval + pinning (DEC-6).
|
||||
steps:
|
||||
- gate on the interview-character allowlist (INV-KB-4); not listed -> [].
|
||||
- out-of-band Mimir consult: a one-shot turn "search the well for <question topic> (<char's corpus>)".
|
||||
- extract cited evidence text (bounded length).
|
||||
- return [{"kind":"corpus_reference","text":<extract>}].
|
||||
error/empty: any failure or no hits -> [] (INV-KB-3; never raises to the turn path).
|
||||
RETIREMENT: when Worldtree #361 reference_knowledge reaches Tier-3, delete this module + the single
|
||||
server.py call-site; Donut then searches in-voice natively.
|
||||
```
|
||||
|
||||
### FN client: speakOnDone (index.html)
|
||||
```
|
||||
on SSE `done`:
|
||||
if !ttsEnabled(): return # INV-TTS-2
|
||||
cancelInFlight() # INV-TTS-3
|
||||
const wav = await fetch('/api/tts', {text: assistantText, agent_id}) # non-stream; whole clip
|
||||
if !ok: return # INV-TTS-4 (silent skip)
|
||||
play(wav) in the <audio> sink; a new turn start -> cancelInFlight() + audio.pause()
|
||||
```
|
||||
|
||||
## Slice plan
|
||||
|
||||
1. **Persona** — author `docs/characters/donut.md` (corpus-grounded, interview frame,
|
||||
markdown-RP register) + `tier3 define` Princess Donut. Visible: she's in the picker,
|
||||
chattable (answers from training until slice 3). No code.
|
||||
2. **Auto-TTS** — `tts.py` (gateway client + pad_to_dials) + `/api/tts` proxy + the
|
||||
index.html 🔊 toggle + speak-on-done playback + serialize/cancel. Visible: she's voiced,
|
||||
affect-modulated. TDD (respx-mock the gateway; live smoke on the console).
|
||||
3. **KB-bridge** — `kb_bridge.py` + the one server.py seam. Visible: she recalls the DCC
|
||||
corpus in-voice. TDD. RETIRE-READY per INV-KB-1.
|
||||
|
||||
Out of scope: true audio streaming (DEC-2), custom voice registration (DEC-8, infra-ops
|
||||
follow-up), extending the bridge to non-interview agents.
|
||||
Reference in New Issue
Block a user