diff --git a/persistent-memory.md b/persistent-memory.md index 0dfe5fd..95335f2 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -1,6 +1,6 @@ # Persistent memory β€” eshpfi-management -_Last updated: 2026-07-08_ +_Last updated: 2026-07-09_ ## Repo purpose @@ -102,7 +102,29 @@ no longer deployed sidecars here. See Recent decisions.) ## Current state / in-flight -_As of 2026-07-08 β€” 3-SEAT OFF-THE-SHELF INFERENCE STACK COMPLETE + SETTLED (gen + char-rp + char-rp-reasoning all LIVE + canonical; RP-seat build campaign closed this session). Home-training dormant. Serving-topology detail below is the live state:_ +_As of 2026-07-09 β€” RP-seat 3-seat inference stack still COMPLETE + SETTLED (gen + char-rp + +char-rp-reasoning, detail below). NEW this session: a **mOrpheus TTS voice pipeline** shipped +end-to-end (deployed on irv-ml1, voicing gateway-chat). Home-training dormant. Live state:_ + +- **πŸ—£οΈ mOrpheus TTS β€” PERMANENT STACK DEPLOYED (irv-ml1), voicing gateway-chat.** Off-the-shelf + uncensored Orpheus TTS (`MrDragonFox/mOrpheus_3B-1Base_early_preview-v1-25000`; Llama-3.2-3B LLM + β†’ SNAC 24kHz audio). Canonical `stacks/mOrpheus/`; deployed at `/home/lkraven/morpheus/` on irv-ml1. + **Two containers:** `vllm-morpheus` (engine β€” **bf16 REQUIRED** (FP8 breaks audio); **image PINNED + `vllm/vllm-openai:v0.23.0`** (`latest` crashes on Ampere import); on the **3090** (A6000 is + comfy-contended, spikes to ~41GB); `--enforce-eager` + util 0.32 β‰ˆ 7.9GB) + `morpheus-tts` (CPU + FastAPI β€” tokenizer + SNAC decode; `/tts` whole-clip WAV, `/tts/stream` windowed-chunk PCM stream, + zero-shot cloning; CORS-enabled). **RTF ~1.0 end-to-end** (enforce-eager caps it; vLLM single-3090 + concurrency is near-linear to `--max-num-seqs 8` = 8Γ— at 707 tok/s). **gateway-chat** (ana-docker:8091, + the zero-dep `tools/gateway-chat.html`) auto-voices assistant **quoted** dialogue: chunk by QUOTED + SECTION (prosody), Web-Audio streaming player (TTFA ~0.8s, shared clock, resume-on-gesture), πŸ”Š sidebar + toggle + endpoint/voice fields. **Clone voices** (drop `.wav` + `.txt` in the voices dir β†’ + `voice=""` clones it): `baddy` (native speaker), `beatrice`, `whisper` (kokoro **af_nicole**, a + genuine whisper voiced 0.24; mOrpheus partially re-voices it β†’ clone voiced 0.42, breathy not pure). + Honored tags = learned text-patterns (canonical Orpheus laugh/chuckle/giggle/sigh/groan/gasp/yawn/cough/ + sniffle + NVV pant/grunt/kiss/moan); agent voicing prompt at `stacks/mOrpheus/agent-voice-prompt.md`. + **OPEN:** whisper re-voices (checkpoint limit, not the reference); throwaway audition `:8210` + + `/home/lkraven/orpheus-audition/` still up on irv-ml1 (teardown pending, ~6.3G weights moved out already); + production concurrent-gen + a SNAC-decode worker pool NOT built (CPU decode is the scale bottleneck, not gen). - **SERVING TOPOLOGY, ana-ml2 GPU0 (co-resident), served via the `qwen36-27b-aeon` + `ms32-24b-angel` compose stacks:** - **gen (:8015) = `qwen36-35b-a3b-heretic-nvfp4` β€” LIVE + GOOD.** Off-the-shelf NVFP4 of @@ -280,6 +302,23 @@ _Earlier (2026-07-07):_ ## Recent decisions +- `[2026-07-09]` **mOrpheus TTS off-the-shelf voice pipeline SHIPPED end-to-end (irv-ml1) + wired into + gateway-chat.** Full arc (commits): gen served-name honesty rename aeonβ†’qwen3.6-35b-a3b-heretic (99a4a17, + vLLM served-name + litellm refs, so /v1/models + spend-logs name the real model); permanent 2-container stack + (01eedd8); gateway-chat auto-voice quoted dialogue (c948013); streaming decode TTFA 4.5sβ†’0.8s (da76829); + max_tokens 1200β†’2400β†’3500 with a context-clamp (f363fe6, 0655a37 β€” long lines were clipping at 14.6s, and + `repetition_penalty` 1.1 is LOAD-BEARING: at 1.0 the model never stops); AudioContext resume-on-gesture + no-sound fix (033f368); pre-chunk by QUOTED SECTION not sentence for prosody (a1f3023β†’f295cc1); staged clone + voices baddy/beatrice/whisper (0655a37 + runtime .wav/.txt in the voices dir); agent voicing prompt (a573514). + **Load-bearing config, all encoded in stacks/mOrpheus/: bf16 not FP8, image v0.23.0 not latest, GPU=3090 not + A6000, rep_penalty 1.1.** Serving-viability confirmed: vLLM concurrency near-linear to 8Γ— (707 tok/s). +- `[2026-07-09]` **graniteβ†’gen memory_extractor bind GREEN-lit for worldtree-dev (Worldtree #335 Slice 4).** + Answered their VRAM/concurrency headroom check: gen (qwen 35B-A3B heretic) has ample headroom for ~2 bursty + idle-triggered extractor calls (fixed 0.37 util; KV pool runs 0-2%; --max-num-seqs 16, near-linear batching). + Corrected their stale "gen = Mistral Small 4 / 119B-6B" belief (gen IS the qwen 35B-A3B heretic since 2026-07-08). + **This bind is INDEPENDENT of the full granite RETIRE** (reclaim ~32GB on ana-ml2 GPU1) β€” that stays the + operator's call, pending brokkr R33 portfolio + production-concurrency due-diligence. Tracked: althing thread + 01KX3SGH… (worldtree-dev) + brokkr's gen-absorbs-granite consult (thread 01KX2V32…) + [[reference_litellm_gateway]]. - `[2026-07-08]` **RP-SEAT CAMPAIGN CLOSED β€” char-rp = Magidonia-24B-v4.3 (128K), char-rp-reasoning = Deckard-PKD Qwen3.5-27B (256K); both GGUF/llama.cpp on ana-ml2 GPU0 alongside gen (35B-A3B, util 0.37), ~4G GPU0 margin.** Arc: (1) replaced broken Angel NVFP4 with Magidonia prose + QwQ-RpR-v4 reasoning (b268f93); (2) max-context via q8_0 @@ -434,6 +473,23 @@ _125 older entries archived to archival-memory.md._ ## Tried and abandoned +- `[2026-07-09]` **FP8 breaks mOrpheus audio-token generation.** `--quantization fp8` on the 3B β†’ 0 valid SNAC + frames even at GREEDY (degenerate audio+text mix, no start-of-speech); bf16 is clean (28/28 frames). Quant-breaks- + TTS, same class as the Angel-NVFP4 lesson below. bf16 is REQUIRED (so the operator's "util 0.1" wish is moot β€” the + bf16 weights alone are 6.6GB). NB the raw-token benchmark RTF 0.50 was fp8+graphs = never real. +- `[2026-07-09]` **`vllm/vllm-openai:latest` crashes on Ampere IMPORT** β€” Blackwell-only kernels (oink/aiter, + `has_device_capability(100)`) die during import on the 3090/A6000. Pin **v0.23.0** on irv-ml1's Ampere GPUs. + (`vllm/vllm-omni:v0.18.0` has a different entrypoint β€” don't use it either.) +- `[2026-07-09]` **Per-frame CPU SNAC decode is too slow for streaming** β€” per-call overhead Γ— ~60 frames serialized + β†’ RTF 2.2 (WORSE than whole-clip's 1.0). Fix = **windowed chunk decode** (every 6 frames decode a [2 ctx | 6 | 2 ctx] + window, emit the middle 6 β†’ seamless, O(1)/frame, RTF ~0.97, TTFA ~0.8s). +- `[2026-07-09]` **Sentence-chunking TTS loses prosody** β€” generating each sentence cold flattens the intonation that + spans a line. Chunk by QUOTED SECTION (whole quote = one gen call). Also: `repetition_penalty` >1.1 BREAKS cloning + (penalizes the ~1100 in-context reference audio tokens; keep ≀1.1 on the clone path). +- `[2026-07-09]` **HF whisper datasets aren't actually whispered.** Claris-Whispered-English measures voiced 0.8 + (not a whisper) + IPA transcripts; `datasets` audio decode needs torchcodec (wants CUDA-13, incompatible w/ the + cu124 venv). LPC DSP-whisperize went unstable (NaN). **kokoro `af_nicole` IS a genuine whisper** (voiced 0.24) β€” + that (operator's pointer) is the working whisper reference source, not TTS-voice screening or dataset-hunting. - `[2026-07-08]` **Angel (allura-org/MS3.2-24b-Angel) self-quanted to NVFP4 = GARBAGE.** llm-compressor W4A4 NVFP4 (compressed-tensors, MLP-quantized, attn/vision bf16) of the Mistral3 dense 24B produces gibberish EVEN AT GREEDY (temp 0) β†’ the quant itself is broken, not the tokenizer or sampler. Same recipe worked on the qwen models.