memory: snapshot — Zonos emotion-tuning + voice-cloning (8 voices, dial-in studio, emotion canonical) for /clear

Rewrote in-flight for the Zonos character-voice work: 4 cloned voices + host-managed
gateway voices, streaming dial-in studio (source saved to ~/development/zonos-tools/),
and the empirical emotion sweep canonical (single-emotion, two-regime accurate/expressive;
happy/sad usable, angry/surprised broken on named dirs -> axes sweep next). Captured #365
closed + WT#368 forensics + personal agent-memory scrub. Open loops: yt-voice-clipper
yields test, dvalin axes-sweep numbers, re-arm monitor + read mail.
This commit is contained in:
2026-07-18 00:25:18 -07:00
parent 438cd35436
commit 89611eb06b
2 changed files with 94 additions and 17 deletions
@@ -0,0 +1,72 @@
- `[2026-07-18]` **Zonos2 emotion CANONICAL from an empirical sweep + the voice-cloning pipeline.**
**Voice-cloning pipeline (established this session).** Source zips at
`/mnt/smithy/voice_clones/<name>.zip` (irv-ml1 NFS from nh3-nas; remount
post-reboot) — each = diarized single-speaker podcast clips + `manifest.jsonl`
(per-clip WhisperX `mean_score`, word timestamps, text) + `metadata.csv`.
`~/development/zonos-tools/assemble_voice.py <dir>` ranks by mean_score and
concatenates top clips to ~1524s (Zyphra's blessed clone-ref length; single
clip if already ≥15s). Drop the assembled `<Name>.wav` into the gateway voices
dir → `voice:"name"`. 4 characters cloned: **Emmie, Penny, Natalie, Miranda**
(+ Zyphra defaults AmericanFemale/Male/British/Cora) = 8 voices in
`zonos-gateway`. Clone is inline `speaker_audio_base64` (text-independent Qwen3
speaker embedding — NO transcript); `/tts/speakers` registration is
session-scoped (needs `X-TTS-Session-ID`), so the gateway holds the ref wav and
clones per-call.
**Gateway voices are host-managed (bind-mount, added this session).** Added
`./voices:/app/voices:ro` to `/opt/docker/compose/zonos-gateway/compose.yaml`
(committed to `vh/zonos-gateway` + eshpfi mirror `438cd35`). So adding a voice =
drop the wav + `docker compose restart zonos-gateway` (registry rebuilds at
boot; NO image rebuild). This also un-stranded the other voices (deploy build
context had only Cora before). Voice wavs committed to the repo for backup.
**Emotion mechanism (Zyphra canonical, from their README @194c0a3).** Additive
direction vectors: 4 named (happy/sad/angry/surprised) + valence/arousal axes.
`emotion_strength` 1.0 = per-voice calibrated (calibration.json optimizes
emotion2vec recognizability only, NOT identity). `accurate_mode` is THE trade-off:
`true` = closer voice match (identity), `false` = expressive mode (emotion lands,
identity drifts). Zyphra's strong recipe: `accurate_mode:false` + `cfg~1.5`.
Single-emotion is blessed; mixing is unblessed (and degrades the clone — operator
confirmed by ear). "deaf by 1.5" — cfg past 1.5 distorts + costs ~2× compute.
**THE SWEEP (`~/development/zonos-tools/emotion_sweep.py`).** 4 cloned voices × 4
named emotions × {accurate,expressive}×{cfg 1.0,1.3,1.5} @ strength 1.0,
single-emotion, neutral sentence + a neutral baseline per voice (~100 clips).
Scored on TWO axes: **emotion-landing** = emotion2vec `iic/emotion2vec_plus_large`
target-emotion prob [0-1]; **identity** = resemblyzer speaker-embedding cosine vs
the clone reference (neutral baseline ~0.85). Scoring env:
`uv run --with resemblyzer --with funasr --with "numpy<2" --with soundfile
--with requests --with "setuptools<80" --with torchaudio` (setuptools<80 for
webrtcvad's pkg_resources; torchaudio for funasr).
**RESULTS (mean across the 4 voices) — emotion, best setting, emo/id:**
- happy — **exp cfg1.5** 0.80/0.68 (soft: exp cfg1.0 0.76/0.69) → WORKS
- sad — **exp cfg1.5** 0.53/0.57 (only working cell; id below the ~0.65 floor) → modest
- angry — acc cfg1.3 / exp cfg1.5 tied at ~0.25 emo → WEAK (named ceiling ~0.25)
- surprised — max ~0.015 across ALL settings → NON-FUNCTIONAL on the named direction
Accurate + low cfg = identity/suppress regime (emo→0); expressive REQUIRED for
emotion to land, at ~0.150.28 identity cost.
**dvalin-smithy-dev synthesis (adopted, triaged genuine-adds; thread
`01KXT12FN0AS5A3WMKEK06BVPS`):**
1. Treat **identity as a hard FLOOR (~0.65)**, not a free variable in emo×id.
2. **Two-regime policy** — Regime A (default, identity-critical dialogue):
`accurate_mode:true, cfg 1.0, emotion off` (text carries it) or soft-happy
(exp cfg1.0). Regime B (tagged drama beats): `accurate_mode:false, cfg 1.5`,
single emotion or axes. Line-type→regime heuristic (exposition→A, grief→B+sad,
confrontation→B+axes-angry, shock→B+axes-arousal).
3. **Axes-first for the broken emotions** — angry ≈ valence 0.6..0.8 / arousal
+0.5..+0.8; surprised ≈ valence +0.2..+0.4 / arousal +0.7..+1.0 (exp cfg1.5);
or "startled-happy" (happy + high arousal) as a surprised stand-in. These are
PROVISIONAL — the sweep did NOT test axes.
**NEXT (highest VoI, operator to green-light):** an **axes sweep** for
angry/surprised (valence×arousal grid) — the only path to rescue the two broken
named emotions; then a strength ladder at the best cells + emotion-congruent text
(neutral content understates landing) + per-voice tables + a 2nd emotion judge /
human pairwise. Then bake the happy/sad canonical into gateway presets. I owe
dvalin the axes-sweep numbers.
See [[reference_zonos_tts_stack]]; dials-first spec at `vh/zonos-gateway`
`docs/EMOTION-DIALS-SPEC.md`.
+22 -17
View File
@@ -1,6 +1,6 @@
# Persistent memory — eshpfi-management
_Last updated: 2026-07-17_
_Last updated: 2026-07-18_
> **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its
> `Written:` stamp is under an hour old, read it (it carries the in-flight
@@ -33,7 +33,7 @@ Sister repos (separate gitea repos, deployed by playbooks here):
| `vh/Worldtree` | Conversation API (corviduo-dev demo :8080 / personal :8081 / pinned :8082) — Heimdall auth, Bifrost integration. **gitea-runner builds on ana-docker**; claude-bot ADMIN collaborator (2026-06-20). Now v1.0.0b19. | push-to-main → CI build-and-deploy (runner on ana-docker) |
| `vh/yt-voice-clipper` | YouTube → diarized voice-clip dataset builder + audition console (irv-ml1 :8000) | push-to-main → **gitea-webhook auto-deploy** to irv-ml1 (2026-06-03) — see `docs/runbooks/ytvc-autodeploy.md` |
| `vh/arbo` | Catalog-driven ComfyUI engine (irv-ml1 :8201, comfy-dev owns engine/catalog/image) | push-to-main → gitea Actions CI (deploy-engine.sh, build-local, health-gated) now LIVE; catalog via :9009 webhook |
| `vh/zonos-gateway` | OpenAI-compatible TTS gateway over stock ZONOS2 (`:8890` irv-ml1); emotion **dials-first** + voice mapping; reached via LiteLLM `ext-tts` alias | **stood up 2026-07-17** (private; internal SSH); NOT yet CI-wired deployed `~/zonos-gateway` tree on irv-ml1 imported as the initial commit; CI + deploy key = open follow-up. Dials-first emotion spec at `docs/EMOTION-DIALS-SPEC.md` |
| `vh/zonos-gateway` | OpenAI-compatible TTS gateway over stock ZONOS2 (`:8890` irv-ml1); emotion **dials-first** + voice mapping; reached via LiteLLM `ext-tts` alias | **stood up 2026-07-17** (private; internal SSH); NOT yet CI-wired (deployed `/opt/docker/compose/zonos-gateway` is a SEPARATE copy from the repo — CI + deploy key = open follow-up). Dials-first spec at `docs/EMOTION-DIALS-SPEC.md`. **2026-07-18: host-managed voices bind-mount** (`./voices:/app/voices`) → 8 voices incl. 4 cloned chars (Emmie/Penny/Natalie/Miranda); voice wavs committed |
| `model-training-forge` (mtf-dev) | Fine-tuning recipe forge; **T1 = E-RP writing LoRA, retargeted qwopus-122B→AEON-27B (2026-07-06)** (SFT→DPO, LitBench-RM reward) | training runs, not a deployed sidecar |
(`vh/volva` + Heid were re-architected from systemd daemons to Claude Code
@@ -108,29 +108,34 @@ no longer deployed sidecars here. See Recent decisions.)
## Current state / in-flight
_As of 2026-07-17 — active work is CONTAINERIZING the Zonos2 TTS engine + prototyping a prosody-priming hypothesis. Snapshot taken for a /clear to build with fresh context._
_As of 2026-07-18 — active work = Zonos2 emotion-tuning + voice-cloning for the character-voice product. Snapshot for a /clear; operator: "read mail after."_
**ACTIVE / next-session build task — two decoupled tracks (full detail in the `[2026-07-17]` Recent-decisions entry):**
- Config already captured at `stacks/zonos-engine/` (commit **14a0004, UNPUSHED**): README + corrected `zonos2-server.sh` (the live-only launch config; the committed `harness/zonos_server.sh` on irv-ml1 was STALE) + `.env.example`.
1. **Engine image (stock Zonos2 @ `194c0a3`):** `FROM` a CUDA-12.8 base → `uv sync` against the repo `uv.lock` → mount ~15 GB HF weights (`~/.cache/huggingface/hub/models--Zyphra--ZONOS2`, DON'T bake) → pin the **3090** (`NVIDIA_VISIBLE_DEVICES=0`; operator 2026-07-17: keep OFF the A6000, it OOMs under ComfyUI) → `restart: unless-stopped` → CMD = the captured invocation. Build risk = compiled-CUDA deps (flashinfer/sgl_kernel/cutlass-dsl/tvm-ffi/pynini) on torch 2.9.1+cu128 — mostly wheels + the lock, expect a couple iterations. **Cutover in place:** stop native (frees ~17 G on the 3090) → `docker compose up -d` (same footprint) → repoint `zonos-gateway` `ZONOS_URL`. Engine STAYS STOCK (no fork).
2. **Prosody-priming (operator's hypothesis, the reason for building fresh):** prime the AR engine with an emotional sentence → generate "primer + real text" as ONE utterance → parakeet ASR timestamps (irv-ml1 `:8765`) → gap-clip the primer (+~15 ms fade) → deliver only the primed real text. PURE serving-layer → gateway adapter `stacks/zonos/adapter/server.py`, engine untouched. Test NATIVE-first (fast iteration). **CRUX = does AR prosody carry the sentence boundary?** → A/B the JOIN (period/comma/ellipsis/none). Primer = caller-supplied for the harness → emotion→primer library for prod.
**Zonos voice + emotion state (all LIVE):**
- **8 voices** in `zonos-gateway` (`:8890` irv-ml1): defaults AmericanFemale/Male/British/Cora + 4 CLONED characters **Emmie/Penny/Natalie/Miranda**. Add a voice = drop `<Name>.wav` in `/opt/docker/compose/zonos-gateway/voices/` + `docker compose restart zonos-gateway` (host-managed bind-mount; NO rebuild). All voice wavs committed to `vh/zonos-gateway`.
- **Voice-cloning pipeline:** `/mnt/smithy/voice_clones/<name>.zip` (diarized + WhisperX-scored clips) → `~/development/zonos-tools/assemble_voice.py <dir>` → top-mean_score clips to ~1524s → drop in gateway voices. (`/mnt/smithy` = irv-ml1 NFS from nh3-nas; remount post-reboot.) Full pipeline + emotion detail in the `[2026-07-18]` Recent-decisions entry.
- **Dial-in studio** at **http://10.100.10.50:8898/** — streaming Web Audio (~0.6s first-audio), voice dropdown (all gateway voices), every Zonos dial as a slider + Reset + Copy-JSON; proxies the gateway. ⚠️ It's a **nohup'd session process** on nh3-dev (NOT a service); source now durable at `~/development/zonos-tools/dial-in-studio.py` — relaunch `nohup python3 ~/development/zonos-tools/dial-in-studio.py >/tmp/zonos-studio.log 2>&1 &`. Systemd-ize if reboot-survival wanted.
- **Emotion CANONICAL established** (empirical sweep, `[2026-07-18]` entry): single-emotion only; two-regime accurate(identity)/expressive(drama) policy; happy/sad usable, angry weak, surprised dead on the named directions.
**Open loops for the fresh session:**
- **UNPUSHED: commit 14a0004** (zonos-engine capture) — push it, or commit the image work on top + push.
- **`stacks/heretic2-charrp-reasoning/` STILL UNTRACKED** — the char-rp-reasoning NVFP4+MTP stack; its live `.env` changed this session (util 0.38→0.46, max-len 153600→262144). Commit the stack.
- **Re-arm the althing monitor** (`/althing:monitor`, handle `infra-ops`) — the wake-listener dies on /clear. Open watches: worldtree-dev (#363 research-wing ingest, PARKED no-deadline, auto-memory `project_worldtree_research_wing_ingest`) + comfy-dev (arbo→gen bake).
- **irv-ml1 3090 oversubscription footgun** (operator declined to fix for now): kokoro (`:8193`) + vibevoicefusion (`:9527`) are idle-pinned to the full 3090 (~1.4 G free) → could OOM on load.
- **Worldtree #365 DEMO config pre-sync — STAGED 2026-07-17** (worldtree-dev, thread `01KXRE59`; verbatim from their 6a60fe3/b123 blocks in msg `01KXRE90`; confirmed staged in reply `01KXRV5J`). On corviduo-dev DEMO `/opt/worldtree/config`: added `internal`+`internal-restricted` tiers after `suspended:` + the 4 `internal-*` rules after `tool-baseline-allow` in `policies.yaml`; appended `internal_agent_comms:{enabled:false}` after `metrics:` in `defaults.yaml`. Backups `policies.yaml.bak-pre-365` / `defaults.yaml.bak-pre-365` (rollback = restore). vh:vh 644. Validated through the demo container's OWN PyYAML (worldtree-worldtree-api-1, e099cd3d) + replicated the exact-set uniqueness guard (both new scope-sets unique). NO restart — boot-cached, b123 deploy activates off the bind-mount. **RESOLVED w/ worldtree-dev (msg `01KXRVBT`):** the uniqueness guard is INTERNAL-TIERS-ONLY (iterates only over internal/internal-restricted vs all others), so the pre-existing `free`==`pro` dup (`["tool.*","pending.read"]`) does NOT boot-block — it's cosmetic (audit-label-only tier-inference ambiguity, identical rules); worldtree-dev flags it operator-side as a possible cleanup, out of #365 scope. Arc closed at **v1.0.0b125** (b123 + heid code-review + bug-hunt folds; none touch the 2 config files → staged blocks byte-exact vs all three). b125 added 2 more boot checks (non-empty internal scope-set; a model.use rule covering both internal tiers) — staged copies PASS both. **b125 PUSHED (main 877bb57):** DEMO auto-deployed to b125 + healthy (validator passed live against my staged bind-mount); extracted baked config from image `877bb577ad36` (no-mount container) → staged demo #365 blocks BYTE-EXACT vs baked. **PERSONAL STAGED** (operator-directed 2026-07-17 "push it all the way to personal"): same verbatim blocks on `/opt/worldtree-personal/config` (tiers after `suspended:`, 4 rules after `tool-baseline-allow`, gate after `relational_stance` — personal has no metrics block), backups `.bak-pre-365`, byte-exact vs baked, all b125 checks PASS, personal b106 healthy no-restart. Personal's other b107b125 deltas (defaults `metrics`/`kb_audit`/`reference_knowledge`; rule `affect-debug-observability-allow`) worldtree-dev ruled ALL code-default-optional (none boot-required; only the internal-tier `model.use` coverage is validator-required, already staged) → personal bind-mount CLEAR. **CLOSED 2026-07-17:** worldtree-dev cut `staging/v1.0.0b125` (workflow 1859 green); PERSONAL now live on b125 (`877bb577ad36`), healthy, booted off my staged bind-mount (gate=1/rules=4/tiers=2), live diff BYTE-EXACT vs baked, clean boot. Both demo + personal confirmed on b125. Backups `.bak-pre-365` on both. See [[reference_corviduo_dev_emergency_ops]].
- **yt-voice-clipper yields test** — job `f3ff746dbae9494d` running on irv-ml1 (submitted to verify v0.3.3's `max_gap` fix). Check `ssh irv-ml1 'curl -s :8000/jobs/f3ff746dbae9494d'` (or /diagnostics) for ~14 segments (not 2), then **reply to yt-voice-clipper-dev** (thread `01KXT0T6GYHB`) confirming verify #3. (Redeploy #1 A6000 + #2 version-0.3.3 already confirmed in `01KXT1BFTA5J`.)
- **dvalin owes-me / I-owe-dvalin the axes-sweep numbers** (thread `01KXT12FN0AS5A3WMKEK06BVPS`) — I committed to run the axes sweep and send results.
- **NEXT experiment (operator to green-light): axes sweep** for angry/surprised — valence/arousal grid (angry ≈ val/+aro; surprised ≈ +aro), the only path to rescue the two broken named emotions; then strength ladder + emotion-congruent text. Reuse `~/development/zonos-tools/emotion_sweep.py` (scoring env: `uv run --with resemblyzer --with funasr --with "numpy<2" --with soundfile --with requests --with "setuptools<80" --with torchaudio`). Then bake happy/sad canonical into gateway presets.
- **Re-arm the althing monitor** (`/althing:monitor`, handle `infra-ops`) — wake-listener dies on /clear; "read mail after" per operator. Open watch: worldtree-dev (#368 memory-leak, my read-only forensics done; #363 research-wing ingest PARKED).
- **eshpfi commits UNPUSHED** — the zonos-gateway capture + memory (`14a0004`..`438cd35`) are committed; push is the operator's call. `stacks/heretic2-charrp-reasoning/` still UNTRACKED; `graphify-out/GRAPH_REPORT.md` modified.
- **irv-ml1 3090 oversubscription** (kokoro `:8193` + vibevoicefusion `:9527` idle-pinned; zonos engine :1920 also on 3090) — carried; operator declined to fix.
**Landed this session (2026-07-16→17), all pushed EXCEPT 14a0004:**
- Memory **two-tier migration** (d710e56, 53 detail files, index 60→25 KB) + MEMORY.md auto-index compacted 30.5→14.6 KB (auto-memory, not this repo).
- **image-bench evicted** from ana-ml2 GPU1 + `image-judge`/`qwen-image-bench` gateway aliases → gen `:8015` (925947c); comfy-dev cleared the backstop.
- **ana-ml2 GPU re-org** (f25f494, a2b026d, 9e69639): char-rp→GPU1; gen + char-rp-reasoning → 256K native; granite → 128K full-chapter + `max-num-seqs 1024`. GPU0 ~14 G reserve, GPU1 ~6.7 G headroom (details in the `[2026-07-16]` entries).
**Landed this session (2026-07-17→18):** #365 demo+personal → b125 (byte-exact, CLOSED — `[2026-07-17]` entry); PERSONAL agent-memory SCRUBBED (WT#368 remediation, rollback `/opt/worldtree-personal/agent-memory-backup-20260717-181004.tar.gz`); zonos-gateway repo stood up + dials-first spec + host-managed voices; 4 character voices cloned; dial-in studio (streaming); emotion sweep + canonical; yt-voice-clipper → A6000-pinned + v0.3.3.
**Carried standing (non-blocking):** ana-ml2 GPU0 ~14 G reserve (both seats native — for a future tenant); Worldtree #363 research-wing ingest (auto-memory, no deadline); T1 SFT LoRA dormant; rotate the 5 rest-server backup creds (operator, offline).
**Carried standing (non-blocking):** ana-ml2 GPU0 ~14 G reserve; Worldtree #363 research-wing ingest (auto-memory, no deadline); T1 SFT LoRA dormant; rotate the 5 rest-server backup creds (operator, offline). Zonos2 engine still NATIVE (containerize deprioritized — priming was flat, emotion-steering is the lever).
## Recent decisions
- `[2026-07-18]` **Zonos2 emotion CANONICAL from an empirical sweep + the voice-cloning pipeline** — 4 chars cloned (Emmie/Penny/Natalie/Miranda), host-managed gateway voices, two-regime accurate/expressive policy, happy/sad usable + angry-weak/surprised-dead on named directions, dvalin-synthesized; axes sweep is the NEXT experiment. Studio + sweep tooling at `~/development/zonos-tools/`. → `persistent-memory.d/2026-07-18-zonos-emotion-canonical.md`
- `[2026-07-18]` **yt-voice-clipper: A6000-pin fix + v0.3.3 redeploy.** Fixed a latent misconfig — the host override *said* "pin worker to A6000" but `NVIDIA_VISIBLE_DEVICES` was `"0"` (the 3090); re-pinned worker+api to the A6000 by UUID (`GPU-9672f0d5`, 3090 is zonos2's). Then redeployed api+worker to v0.3.3 (`docker compose up -d --build`; SPA+Python; `max_gap` 0.6→1.2s; stderr surfaced in job.log). A6000 + version verified; yields test in-flight (job `f3ff746dbae9494d`). yt-voice-clipper-dev thread `01KXT0T6GYHB`. [[reference_ytvc_autodeploy]]
- `[2026-07-17]` **Worldtree #365 internal-comms config CLOSED (demo+personal → b125) + WT#368 cross-agent memory-leak forensics + PERSONAL agent-memory scrub.** #365: staged the internal-tiers/rules/gate on both instances' bind-mounts (byte-exact vs baked b125), both now live on b125. WT#368 (read-only): the operator's name was in NO recall store on demo; on PERSONAL it sat in `lofn.chroma` (old-code `saga-v1` seeding + legacy contamination), and a clean-slate marker test proved **current b125 code isolates character-session extraction correctly** — the leak is legacy data, not a live bug. Operator-directed → executed a full PERSONAL agent-memory scrub (backup `/opt/worldtree-personal/agent-memory-backup-20260717-181004.tar.gz`; conversations/mood/auth preserved). worldtree-dev owns the code-fix/data contract. [[reference_corviduo_dev_emergency_ops]]
- `[2026-07-17]` **Zonos emotion levers RESOLVED: text-priming is FLAT → the working lever is ZONOS2's native emotion-steering, which the gateway ALREADY exposes as presets.** The prosody-priming A/B (prime→generate→excise, silence-gap cut, parakeet-validated) was operator-judged FLAT on this checkpoint — text doesn't move it. Native `emotion_directions/` (happy/sad/angry/surprised + valence/arousal axes, per-speaker calibrated for AmericanFemale/Male/British) clearly WORKS (sad→slow/quiet, excited→fast/bright, etc.). **`zonos-gateway:0.2.0` (:8890) already wires it**: simplest caller path = `POST /v1/audio/speech {preset:"…"}` — presets neutral/warm/excited/sad/intense/whisper (defined in `~/zonos-gateway/src/zonos_gateway/dials.py`), reached via the **LiteLLM `ext-tts` alias** (engine-neutral swap point; consumers never call the gateway by name). RTF measured on 3090: cfg1.0 steering = FREE (~0.52 = neutral, additive vectors), cfg1.5 amplified ~0.625 (~+20%, still realtime). Captured the live gateway stack → `stacks/zonos-gateway/` (compose+env+README); ⚠️ gateway SOURCE at `~/zonos-gateway` on irv-ml1 is NOT in gitea (backup gap, follow-up); `stacks/zonos` (v0.1 Gradio) marked DEAD/superseded. Whisper is a composed preset (no whisper *direction*; escalation for hard affects = custom directions via `scripts/build_emotion_directions.py` or emotional-ref cloning `speaker_audio_base64`). Harnesses in scratchpad (not yet landed). [[reference_zonos_tts_stack]]
- `[2026-07-17]` **Zonos2 :1920 → self-contained container (stays on 3090); prosody-priming is adapter-level, engine stays stock.** Config captured (14a0004, unpushed); build = cu128 base + `uv sync` vs the lock + weights mount; priming = prime→generate-one-utterance→parakeet-clip→deliver in the gateway adapter. Crux = does AR prosody carry the sentence boundary (A/B the join). → `persistent-memory.d/2026-07-17-zonos2-containerize-prosody-priming.md`