docs(zonos-gateway): dials-first emotion spec (operator ruling)
Canonical direction: emotion set by twisting raw dials per-utterance, not preset selection. Presets demoted to optional examples. Spec covers the dial vocabulary + ranges, the emotion_cfg_scale 'deaf by 1.5' rule (NO cap — documented ceiling, explicit over implicit), measured RTF cost, starting-point dial-sets, an LLM client system-prompt snippet, and clone reference guidance (~15-24s, no transcript). Follow-ups flagged: align dials.py cfg help/metadata, trim in-code PRESETS pending usage check, stand up vh/zonos-gateway for version control.
This commit is contained in:
@@ -0,0 +1,140 @@
|
||||
# Zonos Emotion Control — Dials-First Spec
|
||||
|
||||
**Status:** canonical direction (operator ruling 2026-07-17). **Applies to:**
|
||||
`zonos-gateway` (`:8890`) + its consumers (asset-engine, gateway-chat, any
|
||||
future client). **Supersedes:** the preset-centric usage pattern.
|
||||
|
||||
## 1. Philosophy — twist the dials, don't pick a mood
|
||||
|
||||
Emotion is set **per-utterance by twisting the raw dials**, not by choosing from
|
||||
a fixed menu of named presets. The dials ARE the engine's canonical interface
|
||||
(Zyphra/ZONOS2 exposes exactly these; presets were a PFI-only convenience layer,
|
||||
not upstream). Rationale:
|
||||
|
||||
- **Expressiveness.** A scene needs "wry, tired, a little fond" — not `sad`. An
|
||||
LLM client picks fine-grained affect per line far better than N fixed moods.
|
||||
- **Explicit over implicit.** A dial-set states the actual emotion in the call;
|
||||
a preset name hides it behind an opaque label we alone define.
|
||||
- **No curation burden.** Presets are a layer we'd have to keep tuning; they
|
||||
already drifted overwrought (`excited`/`intense` cfg 1.6–1.8 chipmunked with
|
||||
nobody re-tuning them). Dials have no drift and no maintenance surface.
|
||||
|
||||
**Presets are demoted to optional copy-and-tweak examples** (see §5), not the
|
||||
primary interface and not a maintained product.
|
||||
|
||||
## 2. The dials (`POST /v1/audio/speech`)
|
||||
|
||||
All emotion dials require `emotion_enabled: true` — it is the master switch;
|
||||
without it every `emotion_*` field is inert.
|
||||
|
||||
| Dial | Range | Default | What it does | Guidance |
|
||||
|---|---|---|---|---|
|
||||
| `emotion_enabled` | bool | `false` | Master switch for emotion conditioning. | Set `true` to use any dial below. |
|
||||
| `emotion_sliders` | `{happy\|sad\|angry\|surprised: w}` | `null` | Per-emotion weights (the 4 named directions). Blend by setting several. | Weights ~`0.4–0.8`. **`surprised` raises pitch — the chipmunk driver; use sparingly or omit.** |
|
||||
| `emotion_valence` | `−1.0 … +1.0` | `0.0` | Pleasant ↔ unpleasant axis. | Continuous affect for states between the 4 named emotions. |
|
||||
| `emotion_arousal` | `−1.0 … +1.0` | `0.0` | Calm ↔ energized axis. | `+` = excited/anxious, `−` = tired/subdued. High arousal + `surprised` = chipmunk. |
|
||||
| `emotion_strength` | float | `1.0` | Multiplier on the **per-speaker-calibrated** direction. `1.0` = calibrated; `>1` exaggerates. | Keep `~0.7–1.1`. Above `~1.3` strains the voice. |
|
||||
| `emotion_cfg_scale` | `≥ 1.0` | `1.0` | Classifier-free guidance on emotion. `1.0` = off. `>1` amplifies. | **See §3 — the "deaf by 1.5" dial.** |
|
||||
| `accurate_mode` | bool | `true` | `true` = faithful to the reference voice; `false` = looser / more expressive. | Leave `true` for identity-preserving; flip `false` only when you want the voice to bend. |
|
||||
| `speaking_rate_enabled` | bool | `false` | Master switch for rate conditioning. | Needed for `speaking_rate` / `speaking_rate_bucket` / `speed` to bite. |
|
||||
| `speaking_rate_bucket` | int `0–7` | `null` | Exact rate bucket (`0`=slowest … `7`=fastest; wps bands from `/tts/capabilities`). | Fast = excited/anxious; slow = sad/tired. Pair with the emotion, don't overdrive. |
|
||||
|
||||
Discovery is live: `GET /v1/dials` self-describes emotions, axes, rate buckets,
|
||||
and per-dial ranges.
|
||||
|
||||
## 3. `emotion_cfg_scale` — the knob goes to 30, but you're deaf by 1.5
|
||||
|
||||
**There is deliberately NO enforced cap.** The dial will take large values — set
|
||||
it to 30 if you want. You will regret it. Emotional realism tops out around
|
||||
**1.5**; past ~1.4 the voice is overwrought and distorted, and by 1.5 you're
|
||||
"deaf" — pushing harder only adds artifact, not feeling.
|
||||
|
||||
- **Usable range: `1.0`–`1.5`.** Recommended ceiling **`1.4`**. **Never above 1.5.**
|
||||
- `1.0` = off: the additive emotion directions alone (already calibrated). Plenty
|
||||
expressive for most lines.
|
||||
- `~1.25–1.4` = the sweet spot when you want a line to land harder.
|
||||
- `> 1.5` = do not. It's the amp that goes to 11 — the number is bigger, the
|
||||
sound is worse.
|
||||
|
||||
This is documented, not clamped, on purpose (explicit over implicit): the client
|
||||
owns the choice; the spec owns the warning.
|
||||
|
||||
## 4. Cost / real-time — a second reason to stay ≤ 1.5
|
||||
|
||||
Measured on the 3090 (irv-ml1):
|
||||
|
||||
| `emotion_cfg_scale` | RTF | Note |
|
||||
|---|---|---|
|
||||
| `1.0` (off) | ~0.52 | **free** — same as neutral; additive vectors, no extra forward pass |
|
||||
| `1.5` | ~0.625 | ~+20% wall (CFG doubles the *decode* pass), still comfortably real-time |
|
||||
| `> 1.5` | worse | more compute AND worse sound — strictly dominated |
|
||||
|
||||
So cfg above 1.5 costs more *and* sounds worse. Stay ≤ 1.4.
|
||||
|
||||
## 5. Starting-point dial-sets (examples — copy and tweak, not presets)
|
||||
|
||||
Tuned by ear on BritishFemale. Treat these as departure points, not a menu.
|
||||
|
||||
```jsonc
|
||||
// warm — friendly, unhurried
|
||||
{ "emotion_enabled": true, "emotion_sliders": {"happy": 0.4},
|
||||
"emotion_valence": 0.4, "emotion_strength": 0.8, "emotion_cfg_scale": 1.3 }
|
||||
|
||||
// excited — bright, brisk (NO 'surprised' — that chipmunks it)
|
||||
{ "emotion_enabled": true, "emotion_sliders": {"happy": 0.55},
|
||||
"emotion_valence": 0.5, "emotion_arousal": 0.35, "emotion_strength": 0.75,
|
||||
"emotion_cfg_scale": 1.25, "speaking_rate_enabled": true, "speaking_rate_bucket": 4 }
|
||||
|
||||
// sad — slow, quiet
|
||||
{ "emotion_enabled": true, "emotion_sliders": {"sad": 0.8},
|
||||
"emotion_valence": -0.6, "emotion_arousal": -0.5, "emotion_strength": 1.0,
|
||||
"emotion_cfg_scale": 1.4, "speaking_rate_enabled": true, "speaking_rate_bucket": 1 }
|
||||
|
||||
// intense — emphatic, tense (angry only; no 'surprised')
|
||||
{ "emotion_enabled": true, "emotion_sliders": {"angry": 0.55},
|
||||
"emotion_valence": -0.25, "emotion_arousal": 0.45, "emotion_strength": 0.85,
|
||||
"emotion_cfg_scale": 1.3 }
|
||||
|
||||
// whisper — hushed, breathy
|
||||
{ "emotion_enabled": true, "emotion_sliders": {"sad": 0.2},
|
||||
"emotion_arousal": -0.7, "emotion_strength": 0.9, "emotion_cfg_scale": 1.4,
|
||||
"speaking_rate_enabled": true, "speaking_rate_bucket": 2 }
|
||||
```
|
||||
|
||||
## 6. LLM client system-prompt snippet (the real deliverable)
|
||||
|
||||
Paste into an LLM client that drives TTS, so it emits a dial-set per utterance:
|
||||
|
||||
> You control speech emotion by emitting a JSON dial-set alongside each line.
|
||||
> Set `emotion_enabled: true`, then shape the delivery:
|
||||
> - `emotion_sliders`: weights (0–0.8) over `happy` / `sad` / `angry` /
|
||||
> `surprised`. Blend them. **Avoid `surprised` unless you truly want a gasp —
|
||||
> it pitches the voice up unnaturally.**
|
||||
> - `emotion_valence` (−1..+1): unpleasant → pleasant. `emotion_arousal`
|
||||
> (−1..+1): calm → energized. Use these for nuance between the four emotions.
|
||||
> - `emotion_strength` (~0.7–1.1): how hard to push. `emotion_cfg_scale`
|
||||
> (**1.0–1.4, never above 1.5**): amplification — 1.0 is already expressive;
|
||||
> 1.25–1.4 lands a line harder; above 1.5 is distorted garbage, do not.
|
||||
> - `speaking_rate_enabled: true` + `speaking_rate_bucket` (0 slow … 7 fast) for
|
||||
> pacing: fast when excited/anxious, slow when sad/weary.
|
||||
> Match the dial-set to the line's actual feeling — subtle by default, bold only
|
||||
> when the moment earns it.
|
||||
|
||||
## 7. Voice cloning (reference audio)
|
||||
|
||||
Cloning is text-independent (Qwen3 speaker embedding) — **no transcript needed.**
|
||||
Give it **~15–24s of clean, single-speaker audio** (Zyphra's own default voices
|
||||
run 8.7–23.9s). Shorter clips (≤5s) under-condition the embedding and audibly
|
||||
degrade, worse under emotion steering. Register a reusable voice via
|
||||
`POST /tts/speakers` (caches the embedding; reference by `speaker_embedding_id`).
|
||||
|
||||
## 8. Follow-ups (not done here)
|
||||
|
||||
- **Gateway code:** align `dials.py` `emotion_cfg_scale` help/`max` metadata with
|
||||
§3 (carry the "deaf by 1.5" warning; no clamp added — no cap by ruling). Trim
|
||||
the in-code `PRESETS` to a couple of examples (or drop) once current `preset:`
|
||||
usage in asset-engine / gateway-chat is confirmed.
|
||||
- **Version control:** the gateway source lives only on irv-ml1 (`~/zonos-gateway`,
|
||||
not in gitea) — stand up `vh/zonos-gateway` so this spec + the code are tracked
|
||||
together. This spec currently lives in the eshpfi mirror as the version-controlled
|
||||
home until then.
|
||||
@@ -41,7 +41,17 @@ one-line alias repoint, invisible to every consumer. This repo stays
|
||||
Zonos-specific (voice registry, correct native defaults, presets); the swap
|
||||
point is the alias, not the code.
|
||||
|
||||
## Emotion control — the simple caller path (presets)
|
||||
## Emotion control
|
||||
|
||||
> **Direction (operator ruling 2026-07-17): dials-first.** Emotion is set by
|
||||
> twisting the raw dials per-utterance; presets are demoted to optional
|
||||
> copy-and-tweak examples, not the primary interface. See
|
||||
> [`EMOTION-DIALS-SPEC.md`](EMOTION-DIALS-SPEC.md) for the dial vocabulary, the
|
||||
> `emotion_cfg_scale` "deaf by 1.5" rule (no cap, documented ceiling), the
|
||||
> LLM-client system-prompt snippet, and reference-voice guidance. The
|
||||
> preset-centric description below is retained as background.
|
||||
|
||||
### (background) the preset path
|
||||
|
||||
Emotion is exposed as **named presets** that bundle the engine's steering knobs
|
||||
(`emotion_sliders` / `emotion_valence` / `emotion_arousal` / `emotion_strength`
|
||||
|
||||
Reference in New Issue
Block a user