diff --git a/persistent-memory.md b/persistent-memory.md index f509648..f48b5cf 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -1,6 +1,6 @@ # Persistent memory — eshpfi-management -_Last updated: 2026-07-11_ +_Last updated: 2026-07-13_ ## Repo purpose @@ -102,268 +102,41 @@ no longer deployed sidecars here. See Recent decisions.) ## Current state / in-flight -_As of 2026-07-11 — the R17 TTS audition CONCLUDED and **Zonos was productionized** (won the expressive tier → -shipped as the `zonos-gateway` stack, below). Fleet changes this session: **omnivoice RETIRED**, **vllm-morpheus -KILLED** (operator: unimpressive), **ComfyUI TAKEN DOWN** (A6000 freed for the audition; comfy-dev PARKED — the -v0.27.1 bump it validated stays LANDED, just not running). **Higgs** auditioned well + streams (room in the zoo) -but is PARKING-LOT, not shipped. Immediate fresh-context pickup:_ +_As of 2026-07-13 — two active tasks (per /snapshot args): the LiteLLM gateway #355-residual investigation and wiring the runner-side persistent BuildKit cache. The big WT #355 validation saga + Ledger provisioning + the assistant-role adds all LANDED this session; the concluded TTS/Zonos/RP-seat/ComfyUI/T1 in-flight was rolled out of this section (it lives in git + auto-memory)._ -- **🗣️ ZONOS-GATEWAY TTS — SHIPPED + ONLINE (irv-ml1), the R17 expressive winner.** Two pieces. (1) **Zonos - engine** — native zonos2 server (`Zyphra/ZONOS2`, Apache), bare `python3` on irv-ml1 **:1920**, on the **3090** - (operator moved it here off the retired omnivoice); streams float32 PCM @44.1kHz via `POST /tts/generate`; - emotion = **embedding-direction conditioning (NOT tags)** + **clone-gated** (no-ops without a speaker ref); - vendored copy kept **STOCK**. (2) **zonos-gateway wrapper** — OpenAI `/v1/audio/speech` facade over the native - endpoint, **containerized** at `/opt/docker/compose/zonos-gateway/` (image `zonos-gateway:0.2.0`, - `restart: unless-stopped`, reaches host Zonos via `host.docker.internal:1920`), **:8890**, default voice **Cora** - (voices = `*.wav` baked into the image). Full **24-dial** expressive surface + presets - (neutral/warm/excited/sad/intense/whisper) via `GET /v1/dials`; `response_format` pcm|wav. Repo - `~/development/zonos-gateway` (v0.2.0, LOCAL git only). **In asset-engine catalog** - (`docs/asset-engine/services.yaml`, id `zonos-gateway`, status experimental) → shows **online** (background poller - auto-adopts the running container, NO VRAM gate, NO `/start`) → experiment at - `http://10.250.50.70:8200/ui/stream-audition/zonos-gateway`. **gateway-chat** (ana-docker:8091) repointed DIRECT - to it (OpenAI body, Cora, streamed, quotes joined not chunked). ⚠️ **NEVER expose the rep-penalty dials** - (50/1.2/8 — LOAD-BEARING; OFF → 70s-silence footgun). Auto-memory `reference_zonos_tts_stack`. **PENDING - close-outs:** (a) repoint LiteLLM `ext-tts` alias → the wrapper (generic hot-swap point; NOTE LiteLLM buffers - audio → streaming needs a DIRECT wrapper hit, not through `ext-tts`); (b) revert vendored Zonos `api_server.py` - OpenAI-endpoint patches to STOCK (rep-penalty + `_voice_to_speaker_id` + Cora reg; `.bak-*-20260710` on irv-ml1 — - the wrapper made them redundant); (c) cosmetic: catalog `model.image` reads `local/zonos-gateway:0.1.0`, reality - `zonos-gateway:0.2.0` — reconcile on next catalog touch (a resync bounces the live lab). Old bare-deploy dir - `/home/lkraven/zonos-gateway` on irv-ml1 = idle cruft. +### 🔺 ACTIVE 1 — LiteLLM gateway #355-residual investigation (our domain) +- **The data-confirmed finding:** WT #355's residual 300s hang is NOT the seat and NOT WT-pre-send — it's OUR **LiteLLM gateway** (ana-docker `10.250.50.70:4000`) holding two char-rp-reasoning requests **~21 min** (spend_logs dur: **1255.9s + 1357.4s**, startTimes 16:54:44 + 16:56:26Z = the two wedge onsets) while the **Deckard seat sat IDLE**. The seat (ana-ml2:8018 llama.cpp, `--reasoning-budget 400` — forecloses an indefinite mid-thinking hang) completes every request **≤72s** and received no request at the wedge onset → **EXONERATED**. This CONTRADICTS worldtree-dev's "seat wedging mid-thinking" premise, which I'd relayed unconfirmed until the operator challenged it. +- **DECISIVE next step (brokkr's dwarf-panel, read-only pcap analysis):** the **FIN-check** — in the WT↔gateway pcap for the WEDGED streams, did WT send FIN/RST to the gateway at its 302s stall-cancel? + - WT closed but gateway held anyway → **pure LiteLLM bug** (brokkr's lead: a concurrency-slot LEAK — cancelled requests don't free LiteLLM's internal slot → new invokes queue ~21 min behind leaked slots while the seat idles). Our fix. + - WT did NOT close → WT's un-propagated cancel (earlier httpcore shielded-`aclose` can't-kill bug) → worldtree's already-planned force-close fix also releases the hold. +- **Then:** pull LiteLLM router / `max_parallel_requests` config for deployment `hosted_vllm/deckard-pkd-27b`; confirm whether the seat EVER `launch_slot`'d the 1255s request (never-served vs delayed). pcap-only tools: no tshark on corviduo — use `tcpdump -r`. +- **Data on hand:** WT-netns pcaps + py-spy dumps at `corviduo-dev:~infra-ops/py-spy-355/wedge_20260713_095626/` (+ 100039/100341/100543); the 095626 capture overlaps turn-2064's wedge (16:54:46–16:59:46Z). **py-spy + tcpdump are installed on corviduo-dev** (`~/.local/bin/py-spy`, hand-carried musl binary; no pip/uv on that box). LiteLLM spend_logs = `litellm-db` on ana-docker (`docker exec litellm printenv DATABASE_URL`; SpendLogs cols are camelCase → double-quote them). +- **Status:** WT-code panel DONE (seat-innocent accepted); residual is LiteLLM-forensic (mine, brokkr reasons alongside). **Awaiting operator go on the FIN-check.** -- **🔊 R17 TTS AUDITION — CONCLUDED (operator ranked live).** 3-model expressive audition - (Chatterbox-fast / ZONOS2 / Higgs-v3) on irv-ml1's A6000 (ComfyUI taken down to free it). **Latency smoke** - (`~/tts-audition/records/latency.csv`): Chatterbox TTFA 0.47–0.80s / RTF 0.28–0.38 (24kHz streamed — fastest, - stays the low-latency front-end); ZONOS2 warm TTFA ~8s / RTF 2.79 (44.1kHz, clone-gated emotion); Higgs whole-file - TTFA≈gen 2.6–2.8s / RTF 0.98 (NOT streamed — footnote, not comparable to the streamed two). **Outcome: Zonos - productionized** for the expressive tier (see zonos-gateway above); **Higgs = parking-lot** (streams, room in the - zoo, not shipped — Path-A in-process build preserved at `~/tts-audition/`: venv `venvs/higgs`, transformers 5.5.0, - `harness/higgs_native.py`, Qwen3-4B backbone + baked codec; revisit if wanted). **Chatterbox-fast** stays the - low-latency front-end. Protocol: brokkr-smithy R17 `tts-audition-protocol.md`. +### 🔺 ACTIVE 2 — Worldtree deploy-speed: wire the runner-side persistent BuildKit cache +- **Measured (gitea Actions API, `vh/worldtree`, claude-bot token):** full code deploy ≈**12 min**; bottleneck = `build-and-deploy` **~11 min** (685/614/659s recent). `lint-test` ~7.5 min runs PARALLEL (off critical path). `deploy`/recreate ≈50s (fine — not the problem). +- **Root cause (Dockerfile at repo root):** the dep install `RUN --mount=type=secret,... uv sync --frozen --no-dev --no-install-project --no-cache` uses `--no-cache` AND has **no BuildKit `--mount=type=cache`** → re-downloads+rebuilds ALL Python deps cold every build. Layer ordering is CORRECT (`COPY pyproject.toml uv.lock` + `packages/` before `uv sync`, app code after) — the problem is purely caching. +- **Fix (split by domain):** (a) worldtree-dev's Dockerfile diff — add `--mount=type=cache,target=/root/.cache/uv` + drop `--no-cache`; (b) **MINE, runner-side** — persist the BuildKit layer cache between CI runs (`--cache-to/--cache-from` registry or runner-local). Config-only changes already skip the pipeline via a pinned recreate. +- **Confirm first:** pull the build-STEP log to verify the dep-install is THE dominant sub-step (vs the apt layer or the registry push) before handing worldtree-dev the diff — don't assume. **Awaiting operator go.** -- **🖼️ COMFYUI — v0.27.1 LANDED + comfy-dev-VALIDATED, now TAKEN DOWN (A6000 freed; comfy-dev PARKED).** The - v0.27.1 bump stays LANDED (validated clean by comfy-dev, thread 01KX6D3C — 3-tier sweep 189 graphs / 105 - node-types, ZERO breakage, all 9 templates + smoke renders OK: Z-Image / FLUX.1-dev / Illustrious-SDXL / - FLUX.2-GGUF / Qwen-edit sage-AttentionSelector / LTX i2v). ComfyUI is just **not RUNNING** — operator took it down - to give the R17 audition the A6000. **Restart when comfy-dev un-parks.** Landed on the EXISTING **torch - 2.12.1+cu129, SageAttention 2.2.0 intact** (deliberately did NOT bump torch — the 2.13 bump was the mmartial - boot-upgrade's ABI crash-loop cause, never an LTX requirement; **SETTLED = don't bump**). Mechanics: code - `/worktank/comfyui/run/ComfyUI` (=/comfy/mnt/ComfyUI in-container), `docker exec` lands as uid **1025** → use - `-u 1000` + venv python `/comfy/mnt/venv/bin/python` (venv owned lkraven:1000), `DISABLE_UPGRADES=true` pinned, - NVIDIA_VISIBLE_DEVICES=1 (A6000). Rollback net `/worktank/comfyui/venv-snapshot-comfyui-0.24.1-20260710.tar`. LTX - Phase-2 node-install = its own future thread (installs uid-1000). +### Landed this session (2026-07-13) — context for the fresh session +- **WT #355 turn-lifecycle fix VALIDATED** (worldtree b60): wedged turns self-terminate cancelled/stalled at the 300s watchdog (turns 2064/2065 = 302s/360s) vs pre-b60 turn 2061's 16-min no-terminal silence. The residual (ACTIVE 1) is a separate LiteLLM matter. +- **Ledger tier-3 provisioned** (`ledger:miranda`) on personal `:8081`: key minted (user `ledger`, key_id `b38932f5`), GPG-delivered + verified + cleartext shredded; scope confirmed (user tier has `bifrost:invoke`+`agents.define`); allowlist `10.100.10.50:8770` activated by the b60 recreate. +- **`assistant`+`thoughtful-assistant` roles** added (bind `gen`/`gen-reasoning`, kind capability, grants `[foundational]`) on personal+demo; canonicalized by worldtree-dev (`d8bd497`). +- **3 zombie soong sessions retired** (2ad34ec6+7d23052f+0775c9b1 via `POST /admin/sessions//retire`); pre-b60 stale-active backlog HELD (worldtree's #356 rehydrate fix reclaims that class). -- **📅 BIWEEKLY LANDSCAPE-SCAN CRON — SETTLED.** Live systemd --user timer on nh3-dev (next auto-run 2026-07-15 - 09:00 PT), dedicated `brokkr-scan-dev` handle, notify = `post --to brokkr-smithy-dev` (Vuong ruled inbox-post, - no phone-push). Fully validated end-to-end. Nothing owed; watch for a brokkr-scan-dev first-dwarf-arm routing - hiccup flag on the 15th. - -- **📬 althing:** a harness-tracked wake-listener was armed for comfy-dev's reply. **A /clear kills the monitor - loop → the fresh session must re-arm (`/althing:monitor`)**; the 📬 banner surfaces new peer mail on next prompt - regardless. - -_Earlier (2026-07-09) context preserved below:_ - -_As of 2026-07-09 — RP-seat 3-seat inference stack still COMPLETE + SETTLED (gen + char-rp + -char-rp-reasoning, detail below). NEW this session: a **mOrpheus TTS voice pipeline** shipped -end-to-end (deployed on irv-ml1, voicing gateway-chat). Home-training dormant. Live state:_ - -- **⚠️ SUPERSEDED 2026-07-11: vllm-morpheus KILLED (operator: unimpressive); gateway-chat is now voiced by - zonos-gateway (above), NOT mOrpheus. Block kept for the SNAC/clone-voice detail only.** -- **🗣️ mOrpheus TTS — PERMANENT STACK DEPLOYED (irv-ml1), voicing gateway-chat.** Off-the-shelf - uncensored Orpheus TTS (`MrDragonFox/mOrpheus_3B-1Base_early_preview-v1-25000`; Llama-3.2-3B LLM - → SNAC 24kHz audio). Canonical `stacks/mOrpheus/`; deployed at `/home/lkraven/morpheus/` on irv-ml1. - **Two containers:** `vllm-morpheus` (engine — **bf16 REQUIRED** (FP8 breaks audio); **image PINNED - `vllm/vllm-openai:v0.23.0`** (`latest` crashes on Ampere import); on the **3090** (A6000 is - comfy-contended, spikes to ~41GB); `--enforce-eager` + util 0.32 ≈ 7.9GB) + `morpheus-tts` (CPU - FastAPI — tokenizer + SNAC decode; `/tts` whole-clip WAV, `/tts/stream` windowed-chunk PCM stream, - zero-shot cloning; CORS-enabled). **RTF ~1.0 end-to-end** (enforce-eager caps it; vLLM single-3090 - concurrency is near-linear to `--max-num-seqs 8` = 8× at 707 tok/s). **gateway-chat** (ana-docker:8091, - the zero-dep `tools/gateway-chat.html`) auto-voices assistant **quoted** dialogue: chunk by QUOTED - SECTION (prosody), Web-Audio streaming player (TTFA ~0.8s, shared clock, resume-on-gesture), 🔊 sidebar - toggle + endpoint/voice fields. **Clone voices** (drop `.wav` + `.txt` in the voices dir → - `voice=""` clones it): `baddy` (native speaker), `beatrice`, `whisper` (kokoro **af_nicole**, a - genuine whisper voiced 0.24; mOrpheus partially re-voices it → clone voiced 0.42, breathy not pure). - Honored tags = learned text-patterns (canonical Orpheus laugh/chuckle/giggle/sigh/groan/gasp/yawn/cough/ - sniffle + NVV pant/grunt/kiss/moan); agent voicing prompt at `stacks/mOrpheus/agent-voice-prompt.md`. - **OPEN:** whisper re-voices (checkpoint limit, not the reference); throwaway audition `:8210` + - `/home/lkraven/orpheus-audition/` still up on irv-ml1 (teardown pending, ~6.3G weights moved out already); - production concurrent-gen + a SNAC-decode worker pool NOT built (CPU decode is the scale bottleneck, not gen). - -- **SERVING TOPOLOGY, ana-ml2 GPU0 (co-resident), served via the `qwen36-27b-aeon` + `ms32-24b-angel` compose stacks:** - - **gen (:8015) = `qwen36-35b-a3b-heretic-nvfp4` — LIVE + GOOD.** Off-the-shelf NVFP4 of - `llmfan46/Qwen3.6-35B-A3B-uncensored-heretic-Native-MTP-Preserved-NVFP4-Experts-Only` (MoE 256e/8-active = - qwen3_5_moe, vision, GDN-hybrid; **modelopt** format → `--quantization modelopt`; vision bf16; 24G at - `/tank/aimodels/qwen36-35b-a3b-heretic-nvfp4`). UNCENSORED (writes brutal fiction think-off, no system prompt). - "Blathers" → needs adequate max_tokens (thinking-on looks EMPTY under ~600 tok, fine at 3k — NOT broken). - General-assistant seat. Served by `/opt/docker/compose/qwen36-27b-aeon/` (`.env` repointed: `AEON_GEN_MODEL`, - `AEON_GEN_QUANT=modelopt`, `AEON_GEN_GPU_MEM_UTIL=0.40`; the `--quantization` line is now - `${AEON_GEN_QUANT}`-parameterized). Served-names `qwen3.6-27b-aeon`/`-thinking` (gateway gen/gen-reasoning stay - transparent). **util 0.37 (~35G; dropped from 0.40 on 2026-07-08 to fund char-rp 128K — the RP seats are now co-resident, ~4.4G GPU0 margin).** AEON + the intermediate Pantheon-27B - serves are RETIRED. - - **char-rp (:8016) = `TheDrummer_Magidonia-24B-v4.3-Q6_K` GGUF — LIVE + EXCELLENT.** Magistral (Mistral) - dark-romantasy RP tune via llama.cpp (`char-rp-gguf` stack, container `llama-charrp`). NON-thinking PROSE seat: - elite literary prose, zero refusal on dark/explicit scenes, ~65 tok/s, tight POV/instruction adherence - (live-tested). Replaced the broken Angel NVFP4. Alt prose model (`.env` swap `CHARRP_MODEL`): - `MS3.2-PaintedFantasy-v4.1-24B` (more literary flair, looser POV). Both GGUFs pre-pulled at `/tank/aimodels/llm/rp/`. - - **char-rp-reasoning (:8018) = `Qwen3.5-27B-Deckard-PKD-Heretic-Uncensored-Thinking.i1-Q5_K_M` GGUF — LIVE - (A/B WINNER 2026-07-08, REPLACED RpR-v4).** DavidAU creative (PKD) tune on Qwen3.5-27B (container - `llama-charrp-reasoning`). Native Qwen3.5 template → `--reasoning on` managed CoT SURFACES in `reasoning_content`, - content stays clean prose, `--reasoning-budget 400` caps it; **DRY server-side** (sampler order = dry-after-temp) - tames looping. A/B: **0/30 loops + 0/30 refusals**, ~57 tok/s. Runs on the base llama-swap b8840 image (Qwen3.5). - Best-of-breed-per-seat (NOT the same model as char-rp). - -- **🦄 RP-SEAT UNICORN — RESOLVED + DEPLOYED (2026-07-08).** char-rp = **Magidonia-24B-v4.3** (Magistral prose, - 65tps); char-rp-reasoning = **Deckard-PKD (Qwen3.5-27B)** (managed reasoning + DRY, ~57tps; A/B winner over RpR-v4 — see re-A/B note below). Both GGUF via - llama.cpp (`char-rp-gguf` stack, ana-ml2 GPU0, ~86/97G co-resident with gen, ~11G margin). Canonical stack in repo - `stacks/char-rp-gguf/`; gateway rewired (`char-rp`→:8016, `char-rp-reasoning`→:8018, Mistral/QwQ samplers, dropped - the Qwen `enable_thinking` kwarg). **KEY FINDINGS:** (a) no single dense 24-32B is BOTH an elite non-thinking prose - seat AND a clean *managed*-reasoning seat on llama.cpp → best-of-breed per seat; (b) Magidonia's Magistral `[THINK]` - discipline is loose (won't close `[/THINK]` on substantive reasoning → prose bleeds into reasoning, content empties); - Cydonia-R1's `` is emergent (llama.cpp can't manage/cap → runaway CoT never reaches prose); QwQ's template - opens `` natively → llama.cpp manages+caps it; (c) **Pantheon-Reasoning-27B stays REJECTED** (re-censors in - ``; RpR-v4's abliterated-reasoning is the fix); (d) reasoning now SURFACES in `reasoning_content` (fixes the - old AEON trace-not-surfacing gap). One-model fallback (Magidonia both, lighter reasoning) documented in the stack - header/README. Requirements met: prose#1, ≥50tps#2, low-refusal#3, dense#4, GGUF-not-Ollama#5, fit-GPU0#6, thinking#7. - Candidate GGUFs also on disk for A/B: Cydonia-R1-24B-v4.1, PaintedFantasy-v4.1-24B, RpR-v4 Q6_K. - **REASONING-SEAT RE-A/B (2026-07-08, later — RpR→Deckard):** operator wanted a reasoning model that TAKES DRY - (RpR forbids rep/DRY → 1/30 loop tail). Full A/B on brokkr's 30-prompt D1 suite (content-only, slop-scored): - **Deckard-PKD (Qwen3.5-27B) WON** (0/30 loops, 0/30 refusals, clean) → NOW the char-rp-reasoning seat. - RpR-v4: 0 refusals but 1/30 loop (no-DRY). **Pantheon-Reasoning-27B: 7/30 explicit refusals** (DeepSeek-distilled - re-censor IS real across the batch, milder than feared; NOT wholesale-rejected) — kept on disk as alt. - Snowdrop-v0.5-Type-S + Gembrain-31B (Gemma-4): FLOORED — llama.cpp can't do MANAGED reasoning on them (Snowdrop's - ChatML template has no ``/`enable_thinking` hook; Gemma-4's reasoning parser splits content wrong). GATE for a - llama.cpp reasoning seat = STOCK template natively opens `` or has `enable_thinking` (Qwen3.x/QwQ do; ChatML + - Gemma-4 don't). **INFRA:** llama-swap b8840 image can't load Qwen3.6/Gemma-4 archs → use - `ghcr.io/ggml-org/llama.cpp:server-cuda` (newer, pulled on ana-ml2) for those; Deckard (Qwen3.5) runs on b8840. - **brokkr frozen-scorer CONFIRMED (median composite, content-only; lower=cleaner):** Deckard **2.176** (0 loop/0 - refuse — DEPLOYED, big win over RpR **3.716** + its loop-tail); Pantheon-Reasoning **1.383** (cleanest prose of the - field — beats even Magidonia char-rp 1.774 — BUT 7/30 explicit refusals → rejected: clean-but-refuses loses to - clean-and-uncensored). **MEMORY-LAYOUT note:** Deckard (Qwen3.5-27B) is GDN-hybrid → only **16 KV-caching layers** → - KV is TINY (1.4G @ 40K vs Magidonia's 10.9G @ 128K, 40 dense layers); Deckard native ctx = 256K, so the reasoning - seat's ctx scales nearly free. **Deckard BUMPED to full 256K (262144) 2026-07-08** (KV 8.7G q8_0). GPU0 layout now: - gen ~35G (256K/16-seq/util0.37), char-rp Magidonia ~30G (128K/4-slot), char-rp-reasoning Deckard ~28G (256K/4-slot), - **~4G free** (all static allocations → stable, no OOM). **Canonical Deckard RP-reasoning samplers CONFIRMED by dvalin 2026-07-08** - (thread 01KX1Y7P): the live empirical set IS canonical — temp1.0/top_p0.95/top_k40/min_p0.05, NO presence/rep - penalty, DRY0.8 server-side (base1.75/len2, dry-after-temp), reasoning-budget400. Delta from card base-thinking: - top_k20→40, min_p0→0.05, presence1.5→0, DRY-on. Tuning ladder: flat prose→min_p0.08; loops→DRY0.9; over-damped→ - DRY0.6/off. Do NOT import RpR/QwQ rules (QwQ hated DRY; Qwen3.5 benefits from DRY over rep_penalty). - **MAX CONTEXT (2026-07-08):** char-rp **128K** (Magidonia FULL native 131072), char-rp-reasoning **40K** (QwQ - native 40960, YaRN-free max), **q8_0 KV cache both** (near-lossless, ~2× ctx/GB; verified coherent, no Qwen - gibberish). **Funded by gen util 0.40→0.37** (freed ~2.9G of gen's IDLE KV headroom — gen KV usage runs 0-2%, - so its 4.74×→3.66× concurrency-at-256K cut is invisible). GPU0 ~93/97G, **~4.4G margin** (gen fixed-util + - static KV → stable, won't OOM). 64/50 tps. `kv_unified` → a single conversation gets the FULL n_ctx (4 slots - share the pool, not a /4 split). Tunables: CHARRP_CTX / CHARRP_REASONING_CTX / *_KV_TYPE / AEON_GEN_GPU_MEM_UTIL. - -- **✅ LiteLLM gateway sampling defaults WIRED for all 4 seats (2026-07-08)** — dvalin-smithy canonical, infra-ops - triaged + char-rp A/B-validated. **gen** (+summarizer-large twin, shared :8015 deployment): temp 0.7 / top_p 0.8 / - top_k 20 / presence_penalty **1.5** (non-thinking). **gen-reasoning**: temp **1.0** / top_p 0.95 / top_k 20 / - presence_penalty **1.5** (Qwen general-thinking profile — the old temp 0.6 was the CODING sub-profile by mistake; - verbose ~9k-char CoT → callers need big max_tokens). **char-rp**: temp **1.1** / top_p 0.95 / min_p **0.10** / - top_k 0, NO rep (A/B: min_p 0.10 richened imagery vs 0.03; rep 1.05 REJECTED — injected a markdown title in a grief - scene, matches Drummer-no-rep). **char-rp-reasoning**: temp 1.0 / top_p **0.95** / top_k 40 / min_p 0.02, no rep/DRY. - Canonical doc `docs/pfi/model-sampler-defaults.md`; dvalin derivation `dvalin-smithy/hoard-drafts/pfi-gateway-sampler-defaults-20260708.md`. - -- **NVFP4/GGUF quant tooling (proven, on ana-ml2):** `/tank/aimodels/quant-work/` — `nvfp4_quant.py` - (llm-compressor, MoE-aware `load_quantizable_moe`, Mistral-tuned ignore) + `.venv`; `qwen35_vision_nvfp4.py` - (llm-compressor + qwen ignore + `trust_remote_code` + a torchvision-gate bypass). ModelOpt path: - `/home/lkraven/nvfp4-quant/.venv-mopt` (modelopt 0.44) + `mopt_w4a16.py` (= `mtq.quantize` + `export_hf_checkpoint` - → modelopt format w/ `hf_quant_config.json`). Downloads via a detached vLLM container running - `huggingface_hub.snapshot_download` to `/tank/aimodels/-{bf16,nvfp4}`. **Gotchas:** llm-compressor - `save_compressed` DROPS tokenizer/chat/preprocessor files → stage from the bf16 source; the qwen ignore keeps - `re:.*visual.*` + `re:.*linear_attn.*`(GDN) + lm_head + embeds + `mlp.gate`(router) bf16. - -- **🪦 T1 / home-trained LoRA — DORMANT (pivoted to off-the-shelf).** DPO came back slop → operator killed - home-training (2026-07-08). ONLY this project dormant; LEAVE arbo's LoRA-worker (irv-ml1 :8203), ALL /tank models, - and every other project ("ritual seppuku" otherwise). **Revival map:** adapters `data/spike/{sft,dpo}_adapter`, - harness canonical v0.0.44, full saga in Recent decisions + mtf-dev auto-memory; aeon-trainer image left (28G). - mtf-dev stood down. (Detailed 2026-07-07 hot-swap findings preserved below + in Recent decisions.) - -_Earlier (2026-07-07):_ - -- **T1 SFT COMPLETE (AEON-27B E-RP LoRA) — adapter banked, but BLOCKED on hot-swap serving.** SFT ran - clean on ana-ml2 GPU0 (all-TRL torch-GDN, batch1/accum8/GC-off, 1 epoch, ~9h @ 0.25 s/s steady, final - loss 2.445); adapter at `ana-ml2:~/model-training-forge/data/spike/sft_adapter` (124MB, r16 α32, targets - `self_attn.q/k/v/o_proj` + `linear_attn.in_proj_qkv/out_proj`, NO `in_proj_z`). DPO leg deferred (needs - mergekit resolved + on-policy `rejected` regen). **THE WALL: the adapter is INERT on the vLLM NVFP4 - hot-load** — loads HTTP 200 but zero deltas reach inference (vLLM 0.24.0 qwen3_5 LoRA-APPLICATION silent - no-op; upstream #47639 open / fix PR #47640 open / regression from #37912). Proven quant-agnostic (NVFP4 - AND FP8 both inert) AND adapter-format-agnostic (separate-key `sft_adapter_zc` loads clean via vLLM's - allowlist but inert; the fused-key rekey rejected outright). The #47640 guard-patch overlay did NOT fix - it. **HARD REQUIREMENT (operator): hot-swappable or ABANDON — merge is NOT a solution now** (static; - reserved for a later RP-quant track only). - -- **HOT-SWAP — SGLang zc smoke + 2-line patch RAN (2026-07-07). PARTIAL PROGRESS: patch cleared startup - validation + self-attn LoRA loads end-to-end, but GDN in_proj hit 2 MORE name-routing gaps → NOT a - one-line fix. MERGE RECOMMENDED (operator's call, pending).** Ran `sft_adapter_zc` on SGLang 0.5.13 bf16 - (`--lora-target-modules all --max-lora-rank 16 --lora-paths t1=/adapters/sft_adapter_zc`, GPU0, gen+rp - stopped for the smoke then restored). Base weights loaded clean; self-attn q/k/v/o normalized + accepted - fine. **Died at `lora_manager.init_lora_shapes` with a NAME-VALIDATION `ValueError` (NOT the feared - in_proj_qkvz shape-assert):** adapter's `in_proj_qkv`+`in_proj_z` "not included in --lora-target-modules - [...in_proj_qkvz...]". **ROOT CAUSE (pinned):** `srt/lora/utils.py:235 get_normalized_target_modules` uses - a HARDCODED `params_mapping` dict that maps `q/k/v_proj→qkv_proj` (why self-attn works) but OMITS - `in_proj_qkv→in_proj_qkvz` + `in_proj_z→in_proj_qkvz`; it does NOT consult the model's - `packed_modules_mapping` (which DOES declare `"in_proj_qkvz":["in_proj_qkv","in_proj_z"]` at - `models/qwen3_5.py:1010`). Everything else is ALREADY in place: `get_hidden_dim("in_proj_qkvz")` IS - implemented in the model class (`:1040`, → `(hidden_size, key_dim*2+value_dim*2)`; the "utils NotImplementedError" - was only the DEFAULT); `get_stacked_multiply("in_proj_qkvz")=4` in the utils default dict; base-weight - `stacked_params_mapping` fuses `in_proj_qkv→shards(0,1,2)`,`in_proj_z→shard 3`. **PATCH APPLIED + RE-SMOKED - (bind-mounted a patched utils.py overlay into the container):** the 2-line add cleared `init_lora_shapes`; - server came up "fired up and ready", LoRA kernels compiled (K=16384 in_proj_qkvz expand + K=6144 in_proj_z - shrink — B-side math right per mtf-dev), **self-attn loaded end-to-end** ("loaded weights for [o_proj,out_proj, - qkv_proj]"). **BUT GDN in_proj died at TWO MORE name-resolution points (name-routing, NOT shape/intractable):** - (1) adapter-load SKIPPED 192 in_proj_qkv/in_proj_z weights ("did not match any target module [...in_proj_qkvz...]"); - (2) inference buffer-fill `utils.py:300 get_target_module_name` RAISED — SUBSTRING match, `in_proj_qkvz` ⊄ - `...in_proj_qkv.` and `qkv_proj` ⊄ it → "Cannot find target module name for ...in_proj_qkv.lora_A.weight" → - HTTP 500 + scheduler killed. So SGLang's GDN LoRA name-routing is incomplete in ≥3 code paths (fixed 1/3). - **FINISH LINE (if ever revived):** teach `get_target_module_name` + the load-consolidation the - `in_proj_qkv/in_proj_z→in_proj_qkvz` map (mirror qkv_proj, which works E2E), THEN mtf-dev's buffer - shard-placement (0,1,2)+3 question goes live again (still UNPROVEN, deepest step). Per operator's "one - targeted fix else merge" rule → this is multi-point, NOT one fix → MERGE recommended. mtf-dev replied + - loop closed (thread 01KWZDZFYN…). (SGLang GENERIC image still can't - LOAD our NVFP4 checkpoint — bf16-only smoke.) Adapter variants: `sft_adapter` (separate), `sft_adapter_zc` - (separate + zero `in_proj_z`, the canonical one — THIS smoke), `sft_adapter_fused` (fused — wrong), plus - `sft_adapter_attnonly`. All at `/home/lkraven/model-training-forge/data/spike/`. - -- **MERGE FALLBACK — primed (~2-3h) if hot-swap is fully abandoned.** peft merge_and_unload (~20min, LOW - wall — separate keys merge directly, no rekey) → sanity greedy-diff (finally reveals if the ~10h training - is any good — we've NEVER seen the LoRA apply) → ModelOpt NVFP4 re-quant of the merged bf16 (~1-2h, - MEDIUM-HIGH wall = the GDN-hybrid requant + calibration + verify GDN modules quantize) → serve as plain - base (no `--enable-lora`). Static, NOT hot-swappable. - -- **comfy-dev in-arbo LoRA-training worker — DONE (all phases, this session).** Built/deployed the worker - (`services/lora-training-worker/`, irv-ml1 `0.0.0.0:8203`, runs as llmuser, fixed-invocation sd-scripts, - systemd, elway deploy playbook; commits 888ba6a→b617a8b). Phase 1 first real train SUCCEEDED (Sindra - fast/lean). Phase 2 publish-step (`published_lora_name` + optional `train_id` field). Phase 2.5 GC - contract (shape A: `GET /published-loras` + `POST /gc/published {train_ids}`) CONFIRMED — worker impl - DEFERRED to after the T1 hot-swap work. auto-memory `reference_lora_training_worker`. - -- **Serving state (SUPERSEDED — see top; AEON retired 2026-07-08):** during the T1 work, AEON gen (:8015) + - char-rp (:8016) NVFP4 were HEALTHY, stopped/restored for hot-swap smokes via `docker start vllm-aeon-gen - vllm-aeon-rp`. Now replaced: gen = 35B-A3B heretic, char-rp = (broken Angel, being replaced). - -- **worldtree-dev vh/pypi read token** delivered (bifrost==1.1.0 pin, #349); dedicated read:package token - at `nh3-dev:~/.config/gitea-tokens/vh-pypi-read.txt`. The over-scoped write:package token the operator - pasted was revoked + verified dead. - -- **Backups — STILL OPEN:** rotate the 5 disclosed rest-server creds (operator, offline); confirm - esh-vm-db's resticprofile includes DB dumps. `docs/runbooks/backups.md`. - -- **Standing / parked:** inert `mood.decay_rate`/`stale_hours` keys in deployed WT config; Mac Pro - migration (hw-gated); R22 gateway-key at `/home/lkraven/.r22-gateway-key` (mode 600, paid GLM, do NOT - delete); Deckard staged on ana-ml2 as T1's writing benchmark; `/books` transient NFS mount on nh3-dev; - **dvalin sampler re-tune pass** for char-rp / char-rp-reasoning once they accumulate live Worldtree/ST - character-role traffic (dvalin offered 2026-07-08 to do a follow-up after more live sessions; the current - paper defaults + 2-prompt A/B are the starting point — real long-session usage is where repetition/drift - surfaces; re-open thread `01KX1DS6…` to take them up on it). - -- **Worldtree config-propagation (reference):** demo+personal bind-mount config from - `/opt/worldtree{,-personal}/config` (infra-ops-deployable, byte-identical from canonical); reload via - `docker restart `, NEVER `compose up` (stale-`:latest` footgun). Role registry loaded ONCE + - CACHED at startup → a bind-mount `model_roles.yaml` change needs a restart; pre-stage BEFORE the - activating deploy's restart. config REMOVALS are NOT backward-compatible with the running image. +### Still-open standing items (carried) +- **T1 SFT LoRA (AEON-27B E-RP)** — adapter banked but BLOCKED on hot-swap serving; DORMANT (operator pivoted to off-the-shelf). Merge fallback primed (~2-3h) if hot-swap abandoned. +- **Backups** — rotate the 5 disclosed rest-server creds (operator, offline); confirm cross-site restic mirror. +- **althing monitor dies on /clear** — re-arm `/althing:monitor` in the fresh session (open await-reply loops: brokkr on the #355-residual, worldtree-dev on the deploy-resilience follow-ons #356/#357/#358). ## Recent decisions +- `[2026-07-13]` WT #355 residual 300s hang localized to OUR LiteLLM gateway (holds 2 char-rp-reasoning requests ~21 min while the seat idles), NOT the seat — Deckard seat EXONERATED (completes ≤72s; `--reasoning-budget 400` forecloses a mid-thinking hang). Corrects worldtree-dev's "seat wedging" diagnosis. Decisive next = the FIN-check (pcap on corviduo). See in-flight ACTIVE 1. +- `[2026-07-13]` WT #355 turn-lifecycle fix VALIDATED on worldtree b60 — wedged turns self-terminate cancelled/stalled at the 300s stall-watchdog (turns 2064/2065 vs pre-b60 2061's 16-min no-terminal). worldtree-dev filed follow-ons #356 (rehydrate Tier-3 ctx on resume — the recreate-durability gap), #357 (reclaim orphaned active-turn locks), #358 (LLM-provider read-timeout audit); surfacing to Vuong to prioritize. +- `[2026-07-13]` Worldtree deploy bottleneck = the image build (~11 min of a ~12 min deploy), root cause the Dockerfile `uv sync ... --no-cache` + no BuildKit cache-mount (re-downloads all deps cold every build). Fix split: worldtree-dev Dockerfile cache-mount diff + infra-ops runner-side persistent BuildKit cache. Config-only changes skip the build entirely (pinned recreate). +- `[2026-07-13]` Ledger tier-3 consumer `ledger:miranda` provisioned on personal :8081 (key b38932f5, GPG-delivered+shredded, allowlist 10.100.10.50:8770 live); `assistant`+`thoughtful-assistant` capability roles added (gen/gen-reasoning) on personal+demo, canonical d8bd497. Chosen instance = personal (the tier-3-consumer instance, ratatoskr+soong-lab colocated). + - `[2026-07-10]` **Heimdall grant: ratatoskr `affect.full` on PERSONAL Worldtree (operator-approved, worldtree-dev R34-v1 request).** Added allow-rule `ratatoskr-affect-full-allow` to `/opt/worldtree-personal/config/policies.yaml` (`principal.user_ids:["ratatoskr"]`, action `affect.full`, resource `*`, effect allow), mirroring the #347 @@ -666,6 +439,9 @@ _125 older entries archived to archival-memory.md._ ## Tried and abandoned +- `[2026-07-13]` Relaying a peer's diagnosis as fact without confirming it against raw data. worldtree-dev diagnosed the WT #355 residual as "our llama.cpp seat wedging," which I echoed in a wrap-up; the operator challenged it and the seat logs DISPROVED it (seat completes ≤72s, idle at the wedge onset — the hang is the LiteLLM gateway). Lesson: CONFIRM peer diagnoses (esp. cross-domain ones) before acting/relaying — same discipline that caught the earlier char-rp-reasoning red-herring via a live `registry.resolve` reproduction. +- `[2026-07-13]` `althing-cli reply ` (thread id, not a MESSAGE id) → "unknown message_id"; and `reply` to your OWN message self-addresses to your handle ("replying to your own message"). Reply to a PEER's message id, or use `post --to `. Bit me several times this session. + - `[2026-07-09]` **FP8 breaks mOrpheus audio-token generation.** `--quantization fp8` on the 3B → 0 valid SNAC frames even at GREEDY (degenerate audio+text mix, no start-of-speech); bf16 is clean (28/28 frames). Quant-breaks- TTS, same class as the Angel-NVFP4 lesson below. bf16 is REQUIRED (so the operator's "util 0.1" wish is moot — the