memory: snapshot — Zonos2 containerize + prosody-priming (for /clear to build fresh)
Current state rewritten to the active build task: containerize the production
Zonos2 :1920 TTS engine as a self-contained image (config captured in 14a0004)
and prototype the operator's prosody-priming hypothesis (prime→generate-one-
utterance→parakeet-clip→deliver, adapter-level, engine stays stock). New detail
file + Recent-decisions pointer capture the full plan + the crux (does AR prosody
carry the sentence boundary — A/B the join). /tmp handoff written for the fresh session.
This commit is contained in:
+19
-16
@@ -1,6 +1,6 @@
|
||||
# Persistent memory — eshpfi-management
|
||||
|
||||
_Last updated: 2026-07-15_
|
||||
_Last updated: 2026-07-17_
|
||||
|
||||
> **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its
|
||||
> `Written:` stamp is under an hour old, read it (it carries the in-flight
|
||||
@@ -107,27 +107,30 @@ no longer deployed sidecars here. See Recent decisions.)
|
||||
|
||||
## Current state / in-flight
|
||||
|
||||
_As of 2026-07-15 — a long fleet-ops + infra session. Everything below either LANDED (committed/pushed) or is PARKED on a clean async watch. No blocking work in flight._
|
||||
_As of 2026-07-17 — active work is CONTAINERIZING the Zonos2 TTS engine + prototyping a prosody-priming hypothesis. Snapshot taken for a /clear to build with fresh context._
|
||||
|
||||
**Landed this session (2026-07-14 -> 15):**
|
||||
- **Homepage revamp** — the flat ~22-service "AI Systems" group split into a dedicated **AI tab** with 6 role-based groups (Inference / Eval & Retrieval / Gateways & Chat / Speech (TTS) / Audio Tools / Image & Media) + an **AI - Dormant** group for the 7 stopped stacks. Relabeled 22 live compose files (`homepage.group=`) via surgical live `sed` + canonical backfill; GPU0 seats recreated in the operator's VRAM order (char-rp -> gen -> char-rp-reasoning). Committed `569e1af`, pushed. Homepage runs on esh-docker-vm `:5100` (config `configs/homepage/`).
|
||||
- **esh-docker-vm rescued + NFS hardened** — uptime-kuma + paperless-web were wedged in the D-state/phantom class; **only a host REBOOT cleared it** (infra-ops sudo). **NFS fstab properly fixed** (`21d9a07`, pushed): added `x-systemd.before=docker.service,x-systemd.mount-timeout=30` to the 4 `10.0.50.50:` lines — the prior `_netdev,nofail`+`After=remote-fs.target` was silently defeated by `nofail`. Playbook `playbooks/fix-esh-nfs-boot-ordering.yaml` updated. Also: **mOrpheus killed** (irv-ml1 `docker compose down`; backend OOM-looping on the packed 3090) and a **1.36 GB HA coredump cleared**.
|
||||
- **Home Assistant config repo** — `vh/home-assistant-config` (private gitea) for the HA Container at `10.0.50.46:8123` (= `homeassistant` on esh-docker-vm, macvlan, config named-volume). **git-in-place in `/config`** (root/infra-ops) + scoped deploy key; **local clone `~/development/home-assistant-config`**. UI-managed HA -> allowlist model (YAML + curated secret-free `.storage` subset; secrets/DB/coredump gitignored).
|
||||
- **soong-lab library persistence** (corviduo-dev) — `SOONG_LAB_LIBRARY_DIR=/home/infra-ops/soong-lab-data/library` added to soong-lab.env (mirrors PORTRAIT_DIR), `soong-lab-studio.service` restarted. Unblocked the operator (Sindra import -> library save survives redeploys).
|
||||
- **char-rp-reasoning OOM rescue + tune** — the NVFP4+MTP seat (GPU0) crash-looped on a solo restart (GPU0 too tight). Fixed: `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True` + `REASONING_GPU_MEM_UTIL 0.39->0.38` + `REASONING_MAX_MODEL_LEN 196608->153600` (150K). Now ~4.5 GB GPU0 headroom, healthy; brokkr's text battery passed.
|
||||
- **Statusline overhaul** (`~/.claude/statusline-command.sh`, NOT this repo) — git state (branch/dirty/unpushed), 🔔/🔕 althing-monitor-armed, project tag, absolute tokens, **per-session cost** (`.cost.total_cost_usd`), threshold-colored ctx% + rate %s (green<60 / yellow60-90 / red>90).
|
||||
|
||||
**PARKED (operator-directed holds):**
|
||||
- **image-bench (qwen-image-bench, GPU1) eviction — DONE 2026-07-15 (operator-directed, same-day as the switch).** Stopped `vllm-qwen-image-bench` on ana-ml2 (Exited(0), revert via `docker compose start`) → GPU1 freed ~32 GB (95→63 GB used). LiteLLM `image-judge` + `qwen-image-bench` aliases repointed to gen (:8015, qwen3.6-35b-a3b-heretic), deterministic judge sampling (temp0/top_k1/rep_pen1.05) + `enable_thinking:false` kept; verified live with :8014 down. comfy-dev pinged as regression watcher (NSFW-anatomy + tight-grounding; thread 01KXMRDJ). Backstop = the stopped stack (revert ~90 s) + live config backup `config.yaml.bak-pre-imagejudge-20260715`. Full record: auto-memory `project_arbo_gen_switch_imagebench_evict`.
|
||||
- **GPU re-org COMPLETE 2026-07-16 — char-rp relocated GPU0→GPU1 + both cards re-optimized for max context.** Full arc: image-bench evicted (GPU1 −32 G) → granite right-sized 0.34→0.18/64K → **char-rp** (Magidonia-24B, `llama-charrp`) moved to GPU1 (`CHARRP_GPU_ID` 0→1, ~29.9 G; recreate llama-charrp ONLY — shared var w/ retired GGUF reasoning svc) → then MAXED context: **char-rp-reasoning 150K→256K** (heretic2 stack, util 0.38→0.46 → 1.56x@256K), **gen 256K** (qwen36-27b-aeon stack — live model is 35B-A3B-heretic; util 0.30→0.42 + seqs 16→32 → 5.43x@256K), **granite 64K→128K** full-chapter (util 0.18→0.27 → 1.50x@128K). FINAL: **GPU0** 83.1/97.9 G (gen + char-rp-reasoning, both 256K native, ~14 G reserve — both native so no more ctx to add; reserve is for a future tenant), **GPU1** 90.5/97.9 G (char-rp 128K + granite 128K + selene 32K + reward/embed/rerank, ~6.7 G headroom). All seats healthy + gateway-verified. Backups (all `-20260716`): char-rp-gguf `.env.bak-pre-gpu1move`, heretic2 `.env.bak-preopt`, qwen36-aeon `.env.bak-preopt`, vllm `.env.bak-pre-granite-rightsize`. LESSON: KV pool must hold ≥1× max-model-len (util-floor → startup crash, e.g. granite 0.15); slope ~950 MiB/0.01 util but per-model KV cost varies ~8× (gen MoE ~99K tok/GiB, granite dense ~13K tok/GiB) so tune util empirically per seat off the `Maximum concurrency for N tokens` log line.
|
||||
**ACTIVE / next-session build task — two decoupled tracks (full detail in the `[2026-07-17]` Recent-decisions entry):**
|
||||
- Config already captured at `stacks/zonos-engine/` (commit **14a0004, UNPUSHED**): README + corrected `zonos2-server.sh` (the live-only launch config; the committed `harness/zonos_server.sh` on irv-ml1 was STALE) + `.env.example`.
|
||||
1. **Engine image (stock Zonos2 @ `194c0a3`):** `FROM` a CUDA-12.8 base → `uv sync` against the repo `uv.lock` → mount ~15 GB HF weights (`~/.cache/huggingface/hub/models--Zyphra--ZONOS2`, DON'T bake) → pin the **3090** (`NVIDIA_VISIBLE_DEVICES=0`; operator 2026-07-17: keep OFF the A6000, it OOMs under ComfyUI) → `restart: unless-stopped` → CMD = the captured invocation. Build risk = compiled-CUDA deps (flashinfer/sgl_kernel/cutlass-dsl/tvm-ffi/pynini) on torch 2.9.1+cu128 — mostly wheels + the lock, expect a couple iterations. **Cutover in place:** stop native (frees ~17 G on the 3090) → `docker compose up -d` (same footprint) → repoint `zonos-gateway` `ZONOS_URL`. Engine STAYS STOCK (no fork).
|
||||
2. **Prosody-priming (operator's hypothesis, the reason for building fresh):** prime the AR engine with an emotional sentence → generate "primer + real text" as ONE utterance → parakeet ASR timestamps (irv-ml1 `:8765`) → gap-clip the primer (+~15 ms fade) → deliver only the primed real text. PURE serving-layer → gateway adapter `stacks/zonos/adapter/server.py`, engine untouched. Test NATIVE-first (fast iteration). **CRUX = does AR prosody carry the sentence boundary?** → A/B the JOIN (period/comma/ellipsis/none). Primer = caller-supplied for the harness → emotion→primer library for prod.
|
||||
|
||||
**Open loops for the fresh session:**
|
||||
- **`stacks/heretic2-charrp-reasoning/` is UNTRACKED in git** (`?? stacks/heretic2-charrp-reasoning/`) — the char-rp-reasoning compose stack is deployed but never committed; commit it (canonical mirror of the live seat).
|
||||
- **althing monitor** — re-arm `/althing:monitor` in the fresh session (the wake-listener dies on /clear). Open watch: comfy-dev on the arbo bake (no news = clean).
|
||||
- **Carried standing items** (non-blocking): T1 SFT LoRA dormant (operator pivoted off-the-shelf); rotate the 5 rest-server backup creds (operator, offline); brokkr P00 handoff for the NVFP4 char-rp-reasoning seat (in live traffic ahead of it).
|
||||
- **UNPUSHED: commit 14a0004** (zonos-engine capture) — push it, or commit the image work on top + push.
|
||||
- **`stacks/heretic2-charrp-reasoning/` STILL UNTRACKED** — the char-rp-reasoning NVFP4+MTP stack; its live `.env` changed this session (util 0.38→0.46, max-len 153600→262144). Commit the stack.
|
||||
- **Re-arm the althing monitor** (`/althing:monitor`, handle `infra-ops`) — the wake-listener dies on /clear. Open watches: worldtree-dev (#363 research-wing ingest, PARKED no-deadline, auto-memory `project_worldtree_research_wing_ingest`) + comfy-dev (arbo→gen bake).
|
||||
- **irv-ml1 3090 oversubscription footgun** (operator declined to fix for now): kokoro (`:8193`) + vibevoicefusion (`:9527`) are idle-pinned to the full 3090 (~1.4 G free) → could OOM on load.
|
||||
|
||||
**Landed this session (2026-07-16→17), all pushed EXCEPT 14a0004:**
|
||||
- Memory **two-tier migration** (d710e56, 53 detail files, index 60→25 KB) + MEMORY.md auto-index compacted 30.5→14.6 KB (auto-memory, not this repo).
|
||||
- **image-bench evicted** from ana-ml2 GPU1 + `image-judge`/`qwen-image-bench` gateway aliases → gen `:8015` (925947c); comfy-dev cleared the backstop.
|
||||
- **ana-ml2 GPU re-org** (f25f494, a2b026d, 9e69639): char-rp→GPU1; gen + char-rp-reasoning → 256K native; granite → 128K full-chapter + `max-num-seqs 1024`. GPU0 ~14 G reserve, GPU1 ~6.7 G headroom (details in the `[2026-07-16]` entries).
|
||||
|
||||
**Carried standing (non-blocking):** ana-ml2 GPU0 ~14 G reserve (both seats native — for a future tenant); Worldtree #363 research-wing ingest (auto-memory, no deadline); T1 SFT LoRA dormant; rotate the 5 rest-server backup creds (operator, offline).
|
||||
|
||||
## Recent decisions
|
||||
|
||||
- `[2026-07-17]` **Zonos2 :1920 → self-contained container (stays on 3090); prosody-priming is adapter-level, engine stays stock.** Config captured (14a0004, unpushed); build = cu128 base + `uv sync` vs the lock + weights mount; priming = prime→generate-one-utterance→parakeet-clip→deliver in the gateway adapter. Crux = does AR prosody carry the sentence boundary (A/B the join). → `persistent-memory.d/2026-07-17-zonos2-containerize-prosody-priming.md`
|
||||
|
||||
- `[2026-07-16]` **GPU re-org: char-rp→GPU1 + both cards re-optimized for max context.** Moved char-rp (Magidonia-24B) GPU0→GPU1, then maxed context: char-rp-reasoning 150K→256K (util 0.46, 1.56x), gen→256K + seqs 16→32 (util 0.42, 5.43x), granite 64K→**128K full-chapter** (util 0.27, 1.50x). FINAL: GPU0 ~14 G reserve (both seats 256K native), GPU1 ~6.7 G headroom. All healthy. LESSON: KV must hold ≥1× max-len (util-floor crashes) + per-model KV cost varies ~8× (MoE cheap, dense pricey) → tune util empirically. See Current state for the full layout + backups.
|
||||
|
||||
- `[2026-07-16]` **granite right-sized → ~10.5 GB freed on GPU1** (util 0.34→0.18 + max-len 131072→65536; KV 6.45 GiB / 1.29x@65536, summarizer healthy). GPU1 now ~45 GB free to relocate a GPU0 model. LESSON: ~950 MiB KV per 0.01 util here + KV must hold ≥1× max-len — util 0.15 crash-looped (est max-len 47184<65536, ~2-3 min summarizer blip) before 0.18 landed. `.env`-only, recreate `vllm-granite` alone (shared stack).
|
||||
|
||||
Reference in New Issue
Block a user