From 0a9fb85a520f6818b69dc6542d4fcc5ec4d9996f Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Fri, 17 Jul 2026 18:33:26 -0700 Subject: [PATCH] docs(zonos-gateway): dials-first emotion spec (operator ruling) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Canonical direction: emotion set by twisting raw dials per-utterance, not preset selection. Presets demoted to optional examples. Spec covers the dial vocabulary + ranges, the emotion_cfg_scale 'deaf by 1.5' rule (NO cap — documented ceiling, explicit over implicit), measured RTF cost, starting-point dial-sets, an LLM client system-prompt snippet, and clone reference guidance (~15-24s, no transcript). Follow-ups flagged: align dials.py cfg help/metadata, trim in-code PRESETS pending usage check, stand up vh/zonos-gateway for version control. --- stacks/zonos-gateway/EMOTION-DIALS-SPEC.md | 140 +++++++++++++++++++++ stacks/zonos-gateway/README.md | 12 +- 2 files changed, 151 insertions(+), 1 deletion(-) create mode 100644 stacks/zonos-gateway/EMOTION-DIALS-SPEC.md diff --git a/stacks/zonos-gateway/EMOTION-DIALS-SPEC.md b/stacks/zonos-gateway/EMOTION-DIALS-SPEC.md new file mode 100644 index 0000000..1e60b90 --- /dev/null +++ b/stacks/zonos-gateway/EMOTION-DIALS-SPEC.md @@ -0,0 +1,140 @@ +# Zonos Emotion Control — Dials-First Spec + +**Status:** canonical direction (operator ruling 2026-07-17). **Applies to:** +`zonos-gateway` (`:8890`) + its consumers (asset-engine, gateway-chat, any +future client). **Supersedes:** the preset-centric usage pattern. + +## 1. Philosophy — twist the dials, don't pick a mood + +Emotion is set **per-utterance by twisting the raw dials**, not by choosing from +a fixed menu of named presets. The dials ARE the engine's canonical interface +(Zyphra/ZONOS2 exposes exactly these; presets were a PFI-only convenience layer, +not upstream). Rationale: + +- **Expressiveness.** A scene needs "wry, tired, a little fond" — not `sad`. An + LLM client picks fine-grained affect per line far better than N fixed moods. +- **Explicit over implicit.** A dial-set states the actual emotion in the call; + a preset name hides it behind an opaque label we alone define. +- **No curation burden.** Presets are a layer we'd have to keep tuning; they + already drifted overwrought (`excited`/`intense` cfg 1.6–1.8 chipmunked with + nobody re-tuning them). Dials have no drift and no maintenance surface. + +**Presets are demoted to optional copy-and-tweak examples** (see §5), not the +primary interface and not a maintained product. + +## 2. The dials (`POST /v1/audio/speech`) + +All emotion dials require `emotion_enabled: true` — it is the master switch; +without it every `emotion_*` field is inert. + +| Dial | Range | Default | What it does | Guidance | +|---|---|---|---|---| +| `emotion_enabled` | bool | `false` | Master switch for emotion conditioning. | Set `true` to use any dial below. | +| `emotion_sliders` | `{happy\|sad\|angry\|surprised: w}` | `null` | Per-emotion weights (the 4 named directions). Blend by setting several. | Weights ~`0.4–0.8`. **`surprised` raises pitch — the chipmunk driver; use sparingly or omit.** | +| `emotion_valence` | `−1.0 … +1.0` | `0.0` | Pleasant ↔ unpleasant axis. | Continuous affect for states between the 4 named emotions. | +| `emotion_arousal` | `−1.0 … +1.0` | `0.0` | Calm ↔ energized axis. | `+` = excited/anxious, `−` = tired/subdued. High arousal + `surprised` = chipmunk. | +| `emotion_strength` | float | `1.0` | Multiplier on the **per-speaker-calibrated** direction. `1.0` = calibrated; `>1` exaggerates. | Keep `~0.7–1.1`. Above `~1.3` strains the voice. | +| `emotion_cfg_scale` | `≥ 1.0` | `1.0` | Classifier-free guidance on emotion. `1.0` = off. `>1` amplifies. | **See §3 — the "deaf by 1.5" dial.** | +| `accurate_mode` | bool | `true` | `true` = faithful to the reference voice; `false` = looser / more expressive. | Leave `true` for identity-preserving; flip `false` only when you want the voice to bend. | +| `speaking_rate_enabled` | bool | `false` | Master switch for rate conditioning. | Needed for `speaking_rate` / `speaking_rate_bucket` / `speed` to bite. | +| `speaking_rate_bucket` | int `0–7` | `null` | Exact rate bucket (`0`=slowest … `7`=fastest; wps bands from `/tts/capabilities`). | Fast = excited/anxious; slow = sad/tired. Pair with the emotion, don't overdrive. | + +Discovery is live: `GET /v1/dials` self-describes emotions, axes, rate buckets, +and per-dial ranges. + +## 3. `emotion_cfg_scale` — the knob goes to 30, but you're deaf by 1.5 + +**There is deliberately NO enforced cap.** The dial will take large values — set +it to 30 if you want. You will regret it. Emotional realism tops out around +**1.5**; past ~1.4 the voice is overwrought and distorted, and by 1.5 you're +"deaf" — pushing harder only adds artifact, not feeling. + +- **Usable range: `1.0`–`1.5`.** Recommended ceiling **`1.4`**. **Never above 1.5.** +- `1.0` = off: the additive emotion directions alone (already calibrated). Plenty + expressive for most lines. +- `~1.25–1.4` = the sweet spot when you want a line to land harder. +- `> 1.5` = do not. It's the amp that goes to 11 — the number is bigger, the + sound is worse. + +This is documented, not clamped, on purpose (explicit over implicit): the client +owns the choice; the spec owns the warning. + +## 4. Cost / real-time — a second reason to stay ≤ 1.5 + +Measured on the 3090 (irv-ml1): + +| `emotion_cfg_scale` | RTF | Note | +|---|---|---| +| `1.0` (off) | ~0.52 | **free** — same as neutral; additive vectors, no extra forward pass | +| `1.5` | ~0.625 | ~+20% wall (CFG doubles the *decode* pass), still comfortably real-time | +| `> 1.5` | worse | more compute AND worse sound — strictly dominated | + +So cfg above 1.5 costs more *and* sounds worse. Stay ≤ 1.4. + +## 5. Starting-point dial-sets (examples — copy and tweak, not presets) + +Tuned by ear on BritishFemale. Treat these as departure points, not a menu. + +```jsonc +// warm — friendly, unhurried +{ "emotion_enabled": true, "emotion_sliders": {"happy": 0.4}, + "emotion_valence": 0.4, "emotion_strength": 0.8, "emotion_cfg_scale": 1.3 } + +// excited — bright, brisk (NO 'surprised' — that chipmunks it) +{ "emotion_enabled": true, "emotion_sliders": {"happy": 0.55}, + "emotion_valence": 0.5, "emotion_arousal": 0.35, "emotion_strength": 0.75, + "emotion_cfg_scale": 1.25, "speaking_rate_enabled": true, "speaking_rate_bucket": 4 } + +// sad — slow, quiet +{ "emotion_enabled": true, "emotion_sliders": {"sad": 0.8}, + "emotion_valence": -0.6, "emotion_arousal": -0.5, "emotion_strength": 1.0, + "emotion_cfg_scale": 1.4, "speaking_rate_enabled": true, "speaking_rate_bucket": 1 } + +// intense — emphatic, tense (angry only; no 'surprised') +{ "emotion_enabled": true, "emotion_sliders": {"angry": 0.55}, + "emotion_valence": -0.25, "emotion_arousal": 0.45, "emotion_strength": 0.85, + "emotion_cfg_scale": 1.3 } + +// whisper — hushed, breathy +{ "emotion_enabled": true, "emotion_sliders": {"sad": 0.2}, + "emotion_arousal": -0.7, "emotion_strength": 0.9, "emotion_cfg_scale": 1.4, + "speaking_rate_enabled": true, "speaking_rate_bucket": 2 } +``` + +## 6. LLM client system-prompt snippet (the real deliverable) + +Paste into an LLM client that drives TTS, so it emits a dial-set per utterance: + +> You control speech emotion by emitting a JSON dial-set alongside each line. +> Set `emotion_enabled: true`, then shape the delivery: +> - `emotion_sliders`: weights (0–0.8) over `happy` / `sad` / `angry` / +> `surprised`. Blend them. **Avoid `surprised` unless you truly want a gasp — +> it pitches the voice up unnaturally.** +> - `emotion_valence` (−1..+1): unpleasant → pleasant. `emotion_arousal` +> (−1..+1): calm → energized. Use these for nuance between the four emotions. +> - `emotion_strength` (~0.7–1.1): how hard to push. `emotion_cfg_scale` +> (**1.0–1.4, never above 1.5**): amplification — 1.0 is already expressive; +> 1.25–1.4 lands a line harder; above 1.5 is distorted garbage, do not. +> - `speaking_rate_enabled: true` + `speaking_rate_bucket` (0 slow … 7 fast) for +> pacing: fast when excited/anxious, slow when sad/weary. +> Match the dial-set to the line's actual feeling — subtle by default, bold only +> when the moment earns it. + +## 7. Voice cloning (reference audio) + +Cloning is text-independent (Qwen3 speaker embedding) — **no transcript needed.** +Give it **~15–24s of clean, single-speaker audio** (Zyphra's own default voices +run 8.7–23.9s). Shorter clips (≤5s) under-condition the embedding and audibly +degrade, worse under emotion steering. Register a reusable voice via +`POST /tts/speakers` (caches the embedding; reference by `speaker_embedding_id`). + +## 8. Follow-ups (not done here) + +- **Gateway code:** align `dials.py` `emotion_cfg_scale` help/`max` metadata with + §3 (carry the "deaf by 1.5" warning; no clamp added — no cap by ruling). Trim + the in-code `PRESETS` to a couple of examples (or drop) once current `preset:` + usage in asset-engine / gateway-chat is confirmed. +- **Version control:** the gateway source lives only on irv-ml1 (`~/zonos-gateway`, + not in gitea) — stand up `vh/zonos-gateway` so this spec + the code are tracked + together. This spec currently lives in the eshpfi mirror as the version-controlled + home until then. diff --git a/stacks/zonos-gateway/README.md b/stacks/zonos-gateway/README.md index 84c4f31..a7fb730 100644 --- a/stacks/zonos-gateway/README.md +++ b/stacks/zonos-gateway/README.md @@ -41,7 +41,17 @@ one-line alias repoint, invisible to every consumer. This repo stays Zonos-specific (voice registry, correct native defaults, presets); the swap point is the alias, not the code. -## Emotion control — the simple caller path (presets) +## Emotion control + +> **Direction (operator ruling 2026-07-17): dials-first.** Emotion is set by +> twisting the raw dials per-utterance; presets are demoted to optional +> copy-and-tweak examples, not the primary interface. See +> [`EMOTION-DIALS-SPEC.md`](EMOTION-DIALS-SPEC.md) for the dial vocabulary, the +> `emotion_cfg_scale` "deaf by 1.5" rule (no cap, documented ceiling), the +> LLM-client system-prompt snippet, and reference-voice guidance. The +> preset-centric description below is retained as background. + +### (background) the preset path Emotion is exposed as **named presets** that bundle the engine's steering knobs (`emotion_sliders` / `emotion_valence` / `emotion_arousal` / `emotion_strength`