diff --git a/persistent-memory.md b/persistent-memory.md index 8c5d984..556462a 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -1,6 +1,6 @@ # Persistent memory — eshpfi-management -_Last updated: 2026-07-09_ +_Last updated: 2026-07-10_ ## Repo purpose @@ -102,6 +102,51 @@ no longer deployed sidecars here. See Recent decisions.) ## Current state / in-flight +_As of 2026-07-10 — session ran three Vuong-authorized peer tasks: brokkr TTS audition (DELEGATED, in-flight), +comfy-dev ComfyUI bump (attempted → ROLLED BACK, blocked on comfy-dev), biweekly scan cron (SETTLED). Full detail +in the 2026-07-10 Recent-decisions entries. Immediate fresh-context pickup:_ + +- **🔊 TTS AUDITION (research-slot) — env+harness build DELEGATED to a background agent (IN-FLIGHT).** Stand up + Higgs TTS 3 (`bosonai/higgs-tts-3-4b`, ungated despite NC) + ZONOS2 (`Zyphra/ZONOS2`, Apache) + Chatterbox + baseline (already running `chatterbox-fast` :8197) on irv-ml1's **A6000** (~30G free; 3090 full w/ mOrpheus), + SEQUENTIAL load. Harness = text→WAV per model + **TTFA/RTF** logging. Protocol (READ IT): + `~/development/brokkr-smithy/research/R17-nvv-extraction/erp-voice-nvv-corpus/tts-audition-protocol.md`. Decider = + NVV/expressive naturalness; deliver blind-A/B web-listen + latency table → **operator ranks** (not me). brokkr + delivers the ~15-20 prompt set AFTER the env's up. **⚠️ FRESH-CONTEXT PICKUP: a background general-purpose agent + (spawned 2026-07-10) is doing the build; its WORK persists on irv-ml1 but its completion-ping/agent-ID likely + won't route post-/clear → on resume VERIFY the build state DIRECTLY on irv-ml1 (`ssh irv-ml1`; look for a + `tts-audition` work area under /worktank or /home/lkraven, pulled models, the harness) instead of waiting for the + agent.** irv-ml1 = lkraven, docker-group, NO sudo; A6000 = docker device 1. + +- **🖼️ COMFYUI BUMP → v0.27.1 — comfy-dev GAVE THE GO + FULL RECIPE (2026-07-10); re-attempt on fresh context.** + Target CORRECTED to **v0.27.1** (the ACTUAL latest; operator said "v25" but wants latest + Krea-2 needs ≥0.25 so + 0.27.1 covers it; braced for breakage). irv-ml1 `comfyui` currently healthy on rolled-back 0.24.1. Prior attempt + crash-looped (SageAttention ABI vs torch 2.13; DISABLE_UPGRADES bumped the venv but NOT the ComfyUI git checkout + — see the 2026-07-10 Recent-decisions entry). **comfy-dev's atomic recipe (althing thread 01KX655V — run "your + way", bundle to avoid a broken intermediate):** (1) git-advance `/comfy/mnt/ComfyUI` → tag **v0.27.1** + + `pip install -r requirements.txt` **as uid 1000** for new deps; (2) torch 2.13.0 rides along (expected, for LTX); + (3) **rebuild SageAttention vs torch 2.13.0** so `--use-sage-attention` works — IF the recompile fights, DROP the + flag as fallback + **FLAG comfy-dev** (SageAttn is load-bearing for WAN speed + tied to their per-workflow + AttentionSelector workaround for the Qwen-black bug, so they re-validate); (4) re-pin `DISABLE_UPGRADES=true`. Net + = `/worktank/comfyui/venv-snapshot-comfyui-0.24.1-20260710.tar` (crash-loop → roll back like last time + ping). + **PING comfy-dev when healthy on 0.27.1** → they run their 3-tier validation (node-schema diff vs their saved + 105-node baseline → validate_graph_templates.py → smoke renders across families) + own the workflow-breakage + triage. Optional: add `decord` (uid 1000) for the pre-existing ComfyUI-RMBG SAM3Segment warning if trivial, else + skip. **⚠️ comfy-dev's GO is currently UNACKED; it's a RISKY re-attempt (crash-looped once) and the operator said + "wait for comfy… before WE go back" → the fresh context should SURFACE to the operator (comfy-dev GO'd + v0.27.1 + recipe ready) and confirm before executing, NOT silently auto-run.** + +- **📅 BIWEEKLY LANDSCAPE-SCAN CRON — SETTLED.** Live systemd --user timer on nh3-dev (next auto-run 2026-07-15 + 09:00 PT), dedicated `brokkr-scan-dev` handle, notify = `post --to brokkr-smithy-dev` (Vuong ruled inbox-post, + no phone-push). Fully validated end-to-end. Nothing owed; watch for a brokkr-scan-dev first-dwarf-arm routing + hiccup flag on the 15th. + +- **📬 althing:** a harness-tracked wake-listener was armed for comfy-dev's reply. **A /clear kills the monitor + loop → the fresh session must re-arm (`/althing:monitor`)**; the 📬 banner surfaces new peer mail on next prompt + regardless. + +_Earlier (2026-07-09) context preserved below:_ + _As of 2026-07-09 — RP-seat 3-seat inference stack still COMPLETE + SETTLED (gen + char-rp + char-rp-reasoning, detail below). NEW this session: a **mOrpheus TTS voice pipeline** shipped end-to-end (deployed on irv-ml1, voicing gateway-chat). Home-training dormant. Live state:_ @@ -301,6 +346,44 @@ _Earlier (2026-07-07):_ ## Recent decisions +- `[2026-07-10]` **ComfyUI 0.25.x bump on irv-ml1 ATTEMPTED → FAILED → ROLLED BACK (snapshot saved it).** comfy-dev + requested (Vuong-authorized) bumping the irv-ml1 `comfyui` stack (mmartial image, `/opt/docker/compose/comfyui/`, + 0.24.1) to 0.25.x for Krea-2 + LTXV 2.3. **TWO FINDINGS: (1) `DISABLE_UPGRADES=false`/USE_PIPUPGRADE bumps the + VENV (torch 2.12.1→2.13.0 + deps) but does NOT advance the ComfyUI CODE checkout** (`/comfy/mnt/ComfyUI` = + `/worktank/comfyui/run/ComfyUI` stayed 0.24.1 — pinned/detached git, comfy-dev's domain). **(2) the torch bump + broke SageAttention** (2.2.0 `_fused.so` undefined-symbol `c10::impl::cow::materialize_cow_storage` vs torch + 2.13.0) → `--use-sage-attention` (REQUIRED launch flag in COMFY_CMDLINE_EXTRA) crash-looped ComfyUI. Net: broke + the working state, zero 0.25.x payoff. **ROLLBACK WORKED**: pre-bump 16G venv snapshot + `/worktank/comfyui/venv-snapshot-comfyui-0.24.1-20260710.tar` restored (torch 2.12.1 + working SageAttention), + re-pinned DISABLE_UPGRADES=true, recreated → healthy on 0.24.1, serving :8188. Broken venv parked at + `/worktank/comfyui/run/venv.broken-torch213-20260710`. **CORRECTED PATH (sent comfy-dev, thread 01KX655V):** + comfy-dev git-advances the ComfyUI checkout to 0.25.x + reqs → I handle the torch bump + SageAttention + rebuild-against-2.13.0 + re-pin (snapshot stays as the net). **LESSON: mmartial `DISABLE_UPGRADES` gates ONLY + the venv pip-upgrades, NOT the ComfyUI git checkout; a torch bump breaks compiled exts (SageAttention) → + rebuild-after is mandatory.** Bump BLOCKED pending comfy-dev's git-advance. Stack: A6000 (NVIDIA_VISIBLE_DEVICES=1), + lkraven-owned compose+venv (uid 1000, no sudo needed), COMFY_CMDLINE_EXTRA OOM flags preserved. + +- `[2026-07-10]` **Biweekly open-weight-releases scan cron set up for brokkr-smithy (Vuong-authorized).** Durable + systemd **--user** timer on nh3-dev (`brokkr-landscape-scan.timer`, OnCalendar `*-*-01,15 09:00:00` + America/Los_Angeles, Persistent=true; linger on) → `.service` → wrapper `~/.local/bin/brokkr-landscape-scan.sh` + runs headless `claude -p "$(cat ~/.config/brokkr-landscape-scan/prompt.txt)" --dangerously-skip-permissions` in + `~/development/brokkr-smithy` (ALTHING_HANDLE=brokkr-smithy-dev; **explicit PATH** — the --user minimal-PATH + footgun; per-run logs `~/.local/state/brokkr-landscape-scan/`). Prompt = brokkr's payload verbatim (LLM/image/TTS + new-release sweep → ranked synthesis → commit+push+notify). VALIDATED: git-push non-interactive (BatchMode + ls-remote to gitea, passphraseless key — no agent), headless claude auth (READY smoke). VALIDATED END-TO-END 2026-07-10 (manual run, exit 0): + web-sweep→synthesis→commit `2ed2f29`→PUSH of scan #2 (open-weight-releases-2026-07-24.md); triaged dwarf input + + caught baseline errors, quality strong. **HANDLE-COLLISION caught+FIXED** — the headless scan shared handle + brokkr-smithy-dev with the LIVE session + raced its inbox (eitri's dwarf-reply got stolen by the live monitor); + registered a dedicated **brokkr-scan-dev** handle (`add-handle`, driver=none) + repointed the wrapper + rewired + step-5 notify → `althing-cli post --to brokkr-smithy-dev` (NO vuong althing handle exists — confirmed). model=default + + `--max-turns 80`. First run under the new handle = 7/15. Off-cycle 07-24 doc is a validation artifact (scheduled + 1st/15th runs date to their own run-date, no collision) — operator naming-convention call pending. + **NEXT AUTO-RUN 2026-07-15 09:00 PDT.** Manual validation/first run = `systemctl --user start + brokkr-landscape-scan.service`. Open w/ brokkr (thread 01KX63G6): confirm notify-Vuong handle/mechanism + session + handle + model/turn-cap. **NEXT brokkr task (operator-sequenced after this): TTS audition env** — Higgs-TTS-3 + + ZONOS2 + Chatterbox baseline, TTFA/RTF + blind-A/B web-listen (thread 01KX6371; needs GPU-placement + HF-token + feasibility pass first; brokkr delivers the prompt set after the env's up; protocol doc in brokkr-smithy repo). + - `[2026-07-09]` **Two parked items closed: phantom `qwen3.6-35b-a3b` alias VERIFIED already-gone; ana-docker docker log-cap SOLVED no-bounce.** (1) **Phantom**: absent from `/v1/models` + `/model/info` (config+DB registry), zero litellm log refs — the parked "400s in /v1/models" note was STALE (already cleaned in the @@ -344,8 +427,25 @@ _Earlier (2026-07-07):_ character RP off the GENERAL qwen onto the dedicated Magidonia/Deckard seats. Relayed to wt-dev w/ the mapping principle + a SAMPLER-OVERRIDE warning (DROP character-rp's old temp0.75/top_p0.85 overrides — carried onto the dedicated seats they'd clobber the canonical RP tuning DOWNWARD) + ratatoskr-reach note (role call is transparent - but Magidonia/Deckard quality/latency differs from gen). PENDING wt-dev's repo re-point → I pair the host-sync + - user_pass block. Threads `01KX3SGH`/`01KX48QP` (worldtree-dev), + but Magidonia/Deckard quality/latency differs from gen). DONE 2026-07-09: wt-dev committed 5d4fa4a (v1.0.0b44, + UNPUSHED — operator drives push); operator directed host-ahead-of-push, so I sourced BOTH config files directly + from that unpushed commit (local `~/development/Worldtree` checkout — capital W; `git show 5d4fa4a:config/…`) + + mirrored VERBATIM to `/opt/worldtree{,-personal}/config` on demo+personal, paired-pre-flighted via the app's + `load_model_roles()` (no DanglingBinding), restarted both → LIVE: character→char-rp, thoughtful-character + + character-rp→char-rp-reasoning, memory_extractor→gen preserved. Backups `*.bak-prerp-20260709`. context_window + VERIFIED (llama.cpp /props + char-rp-gguf `.env`): char-rp **131072**, char-rp-reasoning **262144** (gave wt-dev + to patch the repo from its interim 32768). **HOST AHEAD of repo-remote until the operator pushes 5d4fa4a** (baked + config converges with the bind-mount on push+deploy). GOTCHA: demo≠personal — PERSONAL was already partly + re-pointed (2026-07-06 AEON-era character→char-rp) so its delta was mostly stale-AEON-descriptions→Magidonia/Deckard + + character-rp + stripping personal's char-rp `default_params` temp0.7/top_p0.8 that CLOBBERED the gateway RP tuning + downward; DEMO had no char-rp catalog entries at all (b44 adds them). Diffed each instance vs b44 before applying + (both deltas = expected changeset only, nothing instance-specific clobbered). wt-dev PATCHED the context_window in **b45/3384a37** + (char-rp 131072, char-rp-reasoning 262144). BUMPED HOST-AHEAD on both instances (operator-directed 2026-07-09): + mirrored b45's providers.yaml → `/opt/worldtree{,-personal}/config`, restarted, verified LIVE (char-rp ctx + 131072, char-rp-reasoning 262144, bindings intact); backups `.bak-b44interim-20260709`. HOST now = **b45 + canonical** (providers.yaml) + b44 (model_roles unchanged b44→b45). STILL PENDING: (1) operator's batched push + of **b44+b45** (`5d4fa4a`+`3384a37`) to converge the repo-remote — host is ahead, no fork; (2) user_pass parity + block (defaults.yaml — NOT in either, separate). Threads `01KX3SGH`/`01KX48QP` (worldtree-dev), monitor armed. See [[reference_corviduo_dev_emergency_ops]]. - `[2026-07-09]` **mOrpheus TTS off-the-shelf voice pipeline SHIPPED end-to-end (irv-ml1) + wired into