From 60813197438a14ed282b0ee1facefb890a3f9f1b Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Thu, 9 Jul 2026 10:48:30 -0700 Subject: [PATCH] =?UTF-8?q?memory:=20snapshot=20=E2=80=94=20mOrpheus=20TTS?= =?UTF-8?q?=20voice=20pipeline=20shipped=20(irv-ml1=20stack=20+=20gateway-?= =?UTF-8?q?chat=20voicing)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Captured the session's mOrpheus arc: permanent 2-container stack (vLLM bf16 v0.23.0 on the 3090 + CPU SNAC/FastAPI wrapper), gateway-chat auto-voicing of quoted dialogue (streaming, per-quote chunking, clone voices baddy/beatrice/whisper), and the load-bearing lessons (FP8 breaks audio, latest-vLLM crashes Ampere, per-frame decode too slow, sentence-chunking kills prosody, HF whisper datasets not whispered → kokoro af_nicole). Plus the granite→gen memory_extractor bind green-lit for worldtree-dev (#335 Slice 4). --- persistent-memory.md | 60 ++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 58 insertions(+), 2 deletions(-) diff --git a/persistent-memory.md b/persistent-memory.md index 0dfe5fd..95335f2 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -1,6 +1,6 @@ # Persistent memory — eshpfi-management -_Last updated: 2026-07-08_ +_Last updated: 2026-07-09_ ## Repo purpose @@ -102,7 +102,29 @@ no longer deployed sidecars here. See Recent decisions.) ## Current state / in-flight -_As of 2026-07-08 — 3-SEAT OFF-THE-SHELF INFERENCE STACK COMPLETE + SETTLED (gen + char-rp + char-rp-reasoning all LIVE + canonical; RP-seat build campaign closed this session). Home-training dormant. Serving-topology detail below is the live state:_ +_As of 2026-07-09 — RP-seat 3-seat inference stack still COMPLETE + SETTLED (gen + char-rp + +char-rp-reasoning, detail below). NEW this session: a **mOrpheus TTS voice pipeline** shipped +end-to-end (deployed on irv-ml1, voicing gateway-chat). Home-training dormant. Live state:_ + +- **🗣️ mOrpheus TTS — PERMANENT STACK DEPLOYED (irv-ml1), voicing gateway-chat.** Off-the-shelf + uncensored Orpheus TTS (`MrDragonFox/mOrpheus_3B-1Base_early_preview-v1-25000`; Llama-3.2-3B LLM + → SNAC 24kHz audio). Canonical `stacks/mOrpheus/`; deployed at `/home/lkraven/morpheus/` on irv-ml1. + **Two containers:** `vllm-morpheus` (engine — **bf16 REQUIRED** (FP8 breaks audio); **image PINNED + `vllm/vllm-openai:v0.23.0`** (`latest` crashes on Ampere import); on the **3090** (A6000 is + comfy-contended, spikes to ~41GB); `--enforce-eager` + util 0.32 ≈ 7.9GB) + `morpheus-tts` (CPU + FastAPI — tokenizer + SNAC decode; `/tts` whole-clip WAV, `/tts/stream` windowed-chunk PCM stream, + zero-shot cloning; CORS-enabled). **RTF ~1.0 end-to-end** (enforce-eager caps it; vLLM single-3090 + concurrency is near-linear to `--max-num-seqs 8` = 8× at 707 tok/s). **gateway-chat** (ana-docker:8091, + the zero-dep `tools/gateway-chat.html`) auto-voices assistant **quoted** dialogue: chunk by QUOTED + SECTION (prosody), Web-Audio streaming player (TTFA ~0.8s, shared clock, resume-on-gesture), 🔊 sidebar + toggle + endpoint/voice fields. **Clone voices** (drop `.wav` + `.txt` in the voices dir → + `voice=""` clones it): `baddy` (native speaker), `beatrice`, `whisper` (kokoro **af_nicole**, a + genuine whisper voiced 0.24; mOrpheus partially re-voices it → clone voiced 0.42, breathy not pure). + Honored tags = learned text-patterns (canonical Orpheus laugh/chuckle/giggle/sigh/groan/gasp/yawn/cough/ + sniffle + NVV pant/grunt/kiss/moan); agent voicing prompt at `stacks/mOrpheus/agent-voice-prompt.md`. + **OPEN:** whisper re-voices (checkpoint limit, not the reference); throwaway audition `:8210` + + `/home/lkraven/orpheus-audition/` still up on irv-ml1 (teardown pending, ~6.3G weights moved out already); + production concurrent-gen + a SNAC-decode worker pool NOT built (CPU decode is the scale bottleneck, not gen). - **SERVING TOPOLOGY, ana-ml2 GPU0 (co-resident), served via the `qwen36-27b-aeon` + `ms32-24b-angel` compose stacks:** - **gen (:8015) = `qwen36-35b-a3b-heretic-nvfp4` — LIVE + GOOD.** Off-the-shelf NVFP4 of @@ -280,6 +302,23 @@ _Earlier (2026-07-07):_ ## Recent decisions +- `[2026-07-09]` **mOrpheus TTS off-the-shelf voice pipeline SHIPPED end-to-end (irv-ml1) + wired into + gateway-chat.** Full arc (commits): gen served-name honesty rename aeon→qwen3.6-35b-a3b-heretic (99a4a17, + vLLM served-name + litellm refs, so /v1/models + spend-logs name the real model); permanent 2-container stack + (01eedd8); gateway-chat auto-voice quoted dialogue (c948013); streaming decode TTFA 4.5s→0.8s (da76829); + max_tokens 1200→2400→3500 with a context-clamp (f363fe6, 0655a37 — long lines were clipping at 14.6s, and + `repetition_penalty` 1.1 is LOAD-BEARING: at 1.0 the model never stops); AudioContext resume-on-gesture + no-sound fix (033f368); pre-chunk by QUOTED SECTION not sentence for prosody (a1f3023→f295cc1); staged clone + voices baddy/beatrice/whisper (0655a37 + runtime .wav/.txt in the voices dir); agent voicing prompt (a573514). + **Load-bearing config, all encoded in stacks/mOrpheus/: bf16 not FP8, image v0.23.0 not latest, GPU=3090 not + A6000, rep_penalty 1.1.** Serving-viability confirmed: vLLM concurrency near-linear to 8× (707 tok/s). +- `[2026-07-09]` **granite→gen memory_extractor bind GREEN-lit for worldtree-dev (Worldtree #335 Slice 4).** + Answered their VRAM/concurrency headroom check: gen (qwen 35B-A3B heretic) has ample headroom for ~2 bursty + idle-triggered extractor calls (fixed 0.37 util; KV pool runs 0-2%; --max-num-seqs 16, near-linear batching). + Corrected their stale "gen = Mistral Small 4 / 119B-6B" belief (gen IS the qwen 35B-A3B heretic since 2026-07-08). + **This bind is INDEPENDENT of the full granite RETIRE** (reclaim ~32GB on ana-ml2 GPU1) — that stays the + operator's call, pending brokkr R33 portfolio + production-concurrency due-diligence. Tracked: althing thread + 01KX3SGH… (worldtree-dev) + brokkr's gen-absorbs-granite consult (thread 01KX2V32…) + [[reference_litellm_gateway]]. - `[2026-07-08]` **RP-SEAT CAMPAIGN CLOSED — char-rp = Magidonia-24B-v4.3 (128K), char-rp-reasoning = Deckard-PKD Qwen3.5-27B (256K); both GGUF/llama.cpp on ana-ml2 GPU0 alongside gen (35B-A3B, util 0.37), ~4G GPU0 margin.** Arc: (1) replaced broken Angel NVFP4 with Magidonia prose + QwQ-RpR-v4 reasoning (b268f93); (2) max-context via q8_0 @@ -434,6 +473,23 @@ _125 older entries archived to archival-memory.md._ ## Tried and abandoned +- `[2026-07-09]` **FP8 breaks mOrpheus audio-token generation.** `--quantization fp8` on the 3B → 0 valid SNAC + frames even at GREEDY (degenerate audio+text mix, no start-of-speech); bf16 is clean (28/28 frames). Quant-breaks- + TTS, same class as the Angel-NVFP4 lesson below. bf16 is REQUIRED (so the operator's "util 0.1" wish is moot — the + bf16 weights alone are 6.6GB). NB the raw-token benchmark RTF 0.50 was fp8+graphs = never real. +- `[2026-07-09]` **`vllm/vllm-openai:latest` crashes on Ampere IMPORT** — Blackwell-only kernels (oink/aiter, + `has_device_capability(100)`) die during import on the 3090/A6000. Pin **v0.23.0** on irv-ml1's Ampere GPUs. + (`vllm/vllm-omni:v0.18.0` has a different entrypoint — don't use it either.) +- `[2026-07-09]` **Per-frame CPU SNAC decode is too slow for streaming** — per-call overhead × ~60 frames serialized + → RTF 2.2 (WORSE than whole-clip's 1.0). Fix = **windowed chunk decode** (every 6 frames decode a [2 ctx | 6 | 2 ctx] + window, emit the middle 6 → seamless, O(1)/frame, RTF ~0.97, TTFA ~0.8s). +- `[2026-07-09]` **Sentence-chunking TTS loses prosody** — generating each sentence cold flattens the intonation that + spans a line. Chunk by QUOTED SECTION (whole quote = one gen call). Also: `repetition_penalty` >1.1 BREAKS cloning + (penalizes the ~1100 in-context reference audio tokens; keep ≤1.1 on the clone path). +- `[2026-07-09]` **HF whisper datasets aren't actually whispered.** Claris-Whispered-English measures voiced 0.8 + (not a whisper) + IPA transcripts; `datasets` audio decode needs torchcodec (wants CUDA-13, incompatible w/ the + cu124 venv). LPC DSP-whisperize went unstable (NaN). **kokoro `af_nicole` IS a genuine whisper** (voiced 0.24) — + that (operator's pointer) is the working whisper reference source, not TTS-voice screening or dataset-hunting. - `[2026-07-08]` **Angel (allura-org/MS3.2-24b-Angel) self-quanted to NVFP4 = GARBAGE.** llm-compressor W4A4 NVFP4 (compressed-tensors, MLP-quantized, attn/vision bf16) of the Mistral3 dense 24B produces gibberish EVEN AT GREEDY (temp 0) → the quant itself is broken, not the tokenizer or sampler. Same recipe worked on the qwen models.