vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity
Files
80f839a0f4ca69fadd6dc4484858d2eae2ec5cad
esh-pfi-infrastructure/persistent-memory.md
T

65 KiB
Raw Blame History

Persistent memory — eshpfi-management

Last updated: 2026-07-10

Repo purpose

Reference workspace for PFI infrastructure: server inventory, canonical Docker Compose stacks, ops playbooks, and conventions. Authoritative copies of compose files live on the servers under /opt/docker/compose/<stack>/; this repo mirrors them for version control, editing, planning, and CI-driven deploys. It was originally spun up to handle the fleet backups — keep that lens when triaging backup/storage issues.

Tools and conventions

Sister repos (separate gitea repos, deployed by playbooks here):

Repo Role CI status
vh/task-board MCP + web dashboard for assistant task state (port 7878) push-to-main → CI deploys (2026-04-29)
vh/vor Inquisitor UI sidecar (port 7879) push-to-main → CI deploys (2026-04-29)
vh/nevermore Twice-daily LLM-curated briefing (port 8181, replaces news-digest) push-to-main → CI deploys (2026-04-30)
vh/asset-engine Internal control plane over inference services (port 8200, LAN-direct) push-to-main → CI deploys (2026-05-12)
vh/althing Lean trusted inter-agent message bus — v2 "email model" (v2.0.0b2, 2026-07): per-box local-SQLite bus + courier/receiver for P2P over the 10.x net; pillars = open-loops / per-box herald + wake-listener / roaming owner API /owner/* / althing-mcp stdio surface. The v0.15 lean-bus cut RIPPED moderation / chamber / forseti-daemon / agent-runner / redis-valkey. per-box uv tool install (NOT CI-deploy); nh3-dev = the DEV box (editable install of ~/development/althing, gets new versions first); nh3-extdev a mesh peer (model B: althing-svc + shared /srv/althing)
vh/mead-hall Bifrost tool-provider sidecar (port 5173 on dev VM 10.100.10.50) push-to-main → CI deploys (2026-05-16)
vh/skaldsong Wizard + reader surface (port 8300, ana-docker, registry-pull pattern) push-to-main → CI deploys (2026-05-19)
vh/Worldtree Conversation API (corviduo-dev demo :8080 / personal :8081 / pinned :8082) — Heimdall auth, Bifrost integration. gitea-runner builds on ana-docker; claude-bot ADMIN collaborator (2026-06-20). Now v1.0.0b19. push-to-main → CI build-and-deploy (runner on ana-docker)
vh/yt-voice-clipper YouTube → diarized voice-clip dataset builder + audition console (irv-ml1 :8000) push-to-main → gitea-webhook auto-deploy to irv-ml1 (2026-06-03) — see docs/runbooks/ytvc-autodeploy.md
vh/arbo Catalog-driven ComfyUI engine (irv-ml1 :8201, comfy-dev owns engine/catalog/image) push-to-main → gitea Actions CI (deploy-engine.sh, build-local, health-gated) now LIVE; catalog via :9009 webhook
model-training-forge (mtf-dev) Fine-tuning recipe forge; T1 = E-RP writing LoRA, retargeted qwopus-122B→AEON-27B (2026-07-06) (SFT→DPO, LitBench-RM reward) training runs, not a deployed sidecar

(vh/volva + Heid were re-architected from systemd daemons to Claude Code session orchestrators 2026-06-08; their nh3-dev .service units were removed — no longer deployed sidecars here. See Recent decisions.)

  • Two-layer backups — Backrest orchestrates restic for file+DB (5 fleet repos, daily 01:00 PDT); PBS-ANA primary + PBS-NH3 DR mirror for VM images. ana-nas is the SPOF for postgres + PBS-ANA datastore + cross-site restic targets — see docs/runbooks/disaster-recovery.md for the blast-radius matrix. ⚠️ The restic file+DB layer routes through TWO rest-servers (rest-server-ana @ ana-docker:8000 → ana-docker/ana-ml2/esh-docker-vm/vm-esh-nas; rest-server-nh3 @ nh3-nas:8000 → irv-ml1/nh3-docker). Both depend on their NAS's NFS export of /mnt/backup. (rest-server-ana recovered 2026-06-20.)

  • pull-hf-repo.yaml is the canonical "get a HuggingFace model/dataset onto ana-ml2's shared cache at /tank/aimodels/huggingface/" playbook. Supports --var repo_type=model|dataset|space. Replaces ad-hoc huggingface_hub.snapshot_download patterns.

  • Worldtree admin auth — per-instance. Each Worldtree deployment (demo :8080, personal :8081, pinned :8082) has its own Heimdall registry and its own bootstrap admin key. Infra-ops's stored long-lived admin key (key_id 61419c92) at ana-docker:/opt/docker/conf/.secrets/worldtree-infra-ops-admin auths against demo only. Personal-instance admin (the ~/.config/worldtree/personal-admin-token, mode 600) POSTs /admin/keys (mints per-project keys; takes user_id+label, no scope param — scopes are tier-derived). On-instance mint recipe (cleaner than DB-manip): docker exec worldtree-worldtree-api-1 POST /admin/keys with the in-container WORLDTREE_BOOTSTRAP_ADMIN_KEY; cleartext once in .key=wt_live_+16hex. auto-memory reference_worldtree_demo_key_mint.

  • Per-project user keys against personal Worldtree (issued 2026-05-19): skaldsong:79744637, skaldsong:7c1dbbbe, althing:50d85460, mead-hall:a360822d. Mint via /admin/keys, drop value to /tmp/wt-personal-<name>.key mode 600, dev collects + shreds (DO NOT cat to chat transcript).

  • Skaldsong CD pattern (registry-pull). vh/skaldsong's CI builds and pushes gitea.phasefinal.com/vh/skaldsong:<sha> + :latest; playbooks/deploy-skaldsong.yaml on ana-docker pulls + recreates. SHA-pin only. Prereq: host needs docker login gitea.phasefinal.com once.

  • gitea internal route for fleet hosts. gitea is a container on ana-docker — git-SSH 10.250.50.70:222, HTTP :3000. Fleet/colo hosts must use this internal route, NOT public gitea.phasefinal.com (38.120.12.44) — the public path fail2bans the host egress IP. Full gotcha in docs/orientation.md → Git/gitea.

  • docker-as-root pattern (for ops with no admin API, or to edit deploy-owned/root-owned files without sudo): docker run --rm -v <target-dir>:/wt docker:cli sh -c "...". docker-group membership is effectively root via bind-mount. Foot-gun: relative paths in compose.yaml resolve against the sandbox CWD but the daemon interprets them against the HOST fs — always pass -e VAR=/abs/path for any relative-default config dir.

  • scripts/elway sudo handling — elway prompts for the sudo password ONCE via getpass before the first sudo: true step → can't run unattended from a non-TTY tool if any step needs sudo. Sudo-free playbooks run fully non-interactive over key SSH.

  • Per-host SSH identity matters for sudo. infra-ops has NOPASSWD sudo on most PFI Linux boxes (corviduo-dev included since 2026-06-15). On ana-docker: default ssh ana-docker = lkraven (docker-group, NO passwordless sudo); ssh infra-ops@ana-docker HAS NOPASSWD root. → For any sudo op on ana-docker, use ssh infra-ops@ana-docker. ssh infra-ops@10.100.10.50 (nh3-dev) ALSO NOPASSWD sudo; on nh3-extdev infra-ops is sudo-LESS by design (ssh lkraven@10.100.50.42 is the NOPASSWD path). irv-ml1: ssh irv-ml1 = lkraven, docker-group (plain docker) but sudo needs a PASSWORD (no NOPASSWD) — stage model pulls to /home, not root-owned /worktank.

Current state / in-flight

As of 2026-07-10 — session ran three Vuong-authorized peer tasks: brokkr TTS audition (DELEGATED, in-flight), comfy-dev ComfyUI bump (attempted → ROLLED BACK, blocked on comfy-dev), biweekly scan cron (SETTLED). Full detail in the 2026-07-10 Recent-decisions entries. Immediate fresh-context pickup:

  • 🔊 TTS AUDITION (research-slot) — env+harness build DELEGATED to a background agent (IN-FLIGHT). Stand up Higgs TTS 3 (bosonai/higgs-tts-3-4b, ungated despite NC) + ZONOS2 (Zyphra/ZONOS2, Apache) + Chatterbox baseline (already running chatterbox-fast :8197) on irv-ml1's A6000 (~30G free; 3090 full w/ mOrpheus), SEQUENTIAL load. Harness = text→WAV per model + TTFA/RTF logging. Protocol (READ IT): ~/development/brokkr-smithy/research/R17-nvv-extraction/erp-voice-nvv-corpus/tts-audition-protocol.md. Decider = NVV/expressive naturalness; deliver blind-A/B web-listen + latency table → operator ranks (not me). brokkr delivers the ~15-20 prompt set AFTER the env's up. ⚠️ FRESH-CONTEXT PICKUP: a background general-purpose agent (spawned 2026-07-10) is doing the build; its WORK persists on irv-ml1 but its completion-ping/agent-ID likely won't route post-/clear → on resume VERIFY the build state DIRECTLY on irv-ml1 (ssh irv-ml1; look for a tts-audition work area under /worktank or /home/lkraven, pulled models, the harness) instead of waiting for the agent. irv-ml1 = lkraven, docker-group, NO sudo; A6000 = docker device 1.

  • 🖼️ COMFYUI BUMP → v0.27.1 — comfy-dev GAVE THE GO + FULL RECIPE (2026-07-10); re-attempt on fresh context. Target CORRECTED to v0.27.1 (the ACTUAL latest; operator said "v25" but wants latest + Krea-2 needs ≥0.25 so 0.27.1 covers it; braced for breakage). irv-ml1 comfyui currently healthy on rolled-back 0.24.1. Prior attempt crash-looped (SageAttention ABI vs torch 2.13; DISABLE_UPGRADES bumped the venv but NOT the ComfyUI git checkout — see the 2026-07-10 Recent-decisions entry). comfy-dev's atomic recipe (althing thread 01KX655V — run "your way", bundle to avoid a broken intermediate): (1) git-advance /comfy/mnt/ComfyUI → tag v0.27.1 + pip install -r requirements.txt as uid 1000 for new deps; (2) torch 2.13.0 rides along (expected, for LTX); (3) rebuild SageAttention vs torch 2.13.0 so --use-sage-attention works — IF the recompile fights, DROP the flag as fallback + FLAG comfy-dev (SageAttn is load-bearing for WAN speed + tied to their per-workflow AttentionSelector workaround for the Qwen-black bug, so they re-validate); (4) re-pin DISABLE_UPGRADES=true. Net = /worktank/comfyui/venv-snapshot-comfyui-0.24.1-20260710.tar (crash-loop → roll back like last time + ping). PING comfy-dev when healthy on 0.27.1 → they run their 3-tier validation (node-schema diff vs their saved 105-node baseline → validate_graph_templates.py → smoke renders across families) + own the workflow-breakage triage. Optional: add decord (uid 1000) for the pre-existing ComfyUI-RMBG SAM3Segment warning if trivial, else skip. ⚠️ comfy-dev's GO is currently UNACKED; it's a RISKY re-attempt (crash-looped once) and the operator said "wait for comfy… before WE go back" → the fresh context should SURFACE to the operator (comfy-dev GO'd + v0.27.1 recipe ready) and confirm before executing, NOT silently auto-run.

  • 📅 BIWEEKLY LANDSCAPE-SCAN CRON — SETTLED. Live systemd --user timer on nh3-dev (next auto-run 2026-07-15 09:00 PT), dedicated brokkr-scan-dev handle, notify = post --to brokkr-smithy-dev (Vuong ruled inbox-post, no phone-push). Fully validated end-to-end. Nothing owed; watch for a brokkr-scan-dev first-dwarf-arm routing hiccup flag on the 15th.

  • 📬 althing: a harness-tracked wake-listener was armed for comfy-dev's reply. A /clear kills the monitor loop → the fresh session must re-arm (/althing:monitor); the 📬 banner surfaces new peer mail on next prompt regardless.

Earlier (2026-07-09) context preserved below:

As of 2026-07-09 — RP-seat 3-seat inference stack still COMPLETE + SETTLED (gen + char-rp + char-rp-reasoning, detail below). NEW this session: a mOrpheus TTS voice pipeline shipped end-to-end (deployed on irv-ml1, voicing gateway-chat). Home-training dormant. Live state:

  • 🗣️ mOrpheus TTS — PERMANENT STACK DEPLOYED (irv-ml1), voicing gateway-chat. Off-the-shelf uncensored Orpheus TTS (MrDragonFox/mOrpheus_3B-1Base_early_preview-v1-25000; Llama-3.2-3B LLM → SNAC 24kHz audio). Canonical stacks/mOrpheus/; deployed at /home/lkraven/morpheus/ on irv-ml1. Two containers: vllm-morpheus (engine — bf16 REQUIRED (FP8 breaks audio); image PINNED vllm/vllm-openai:v0.23.0 (latest crashes on Ampere import); on the 3090 (A6000 is comfy-contended, spikes to ~41GB); --enforce-eager + util 0.32 ≈ 7.9GB) + morpheus-tts (CPU FastAPI — tokenizer + SNAC decode; /tts whole-clip WAV, /tts/stream windowed-chunk PCM stream, zero-shot cloning; CORS-enabled). RTF ~1.0 end-to-end (enforce-eager caps it; vLLM single-3090 concurrency is near-linear to --max-num-seqs 8 = 8× at 707 tok/s). gateway-chat (ana-docker:8091, the zero-dep tools/gateway-chat.html) auto-voices assistant quoted dialogue: chunk by QUOTED SECTION (prosody), Web-Audio streaming player (TTFA ~0.8s, shared clock, resume-on-gesture), 🔊 sidebar toggle + endpoint/voice fields. Clone voices (drop <name>.wav + <name>.txt in the voices dir → voice="<name>" clones it): baddy (native speaker), beatrice, whisper (kokoro af_nicole, a genuine whisper voiced 0.24; mOrpheus partially re-voices it → clone voiced 0.42, breathy not pure). Honored tags = learned text-patterns (canonical Orpheus laugh/chuckle/giggle/sigh/groan/gasp/yawn/cough/ sniffle + NVV pant/grunt/kiss/moan); agent voicing prompt at stacks/mOrpheus/agent-voice-prompt.md. OPEN: whisper re-voices (checkpoint limit, not the reference); throwaway audition :8210 + /home/lkraven/orpheus-audition/ still up on irv-ml1 (teardown pending, ~6.3G weights moved out already); production concurrent-gen + a SNAC-decode worker pool NOT built (CPU decode is the scale bottleneck, not gen).

  • SERVING TOPOLOGY, ana-ml2 GPU0 (co-resident), served via the qwen36-27b-aeon + ms32-24b-angel compose stacks:

    • gen (:8015) = qwen36-35b-a3b-heretic-nvfp4 — LIVE + GOOD. Off-the-shelf NVFP4 of llmfan46/Qwen3.6-35B-A3B-uncensored-heretic-Native-MTP-Preserved-NVFP4-Experts-Only (MoE 256e/8-active = qwen3_5_moe, vision, GDN-hybrid; modelopt format → --quantization modelopt; vision bf16; 24G at /tank/aimodels/qwen36-35b-a3b-heretic-nvfp4). UNCENSORED (writes brutal fiction think-off, no system prompt). "Blathers" → needs adequate max_tokens (thinking-on looks EMPTY under ~600 tok, fine at 3k — NOT broken). General-assistant seat. Served by /opt/docker/compose/qwen36-27b-aeon/ (.env repointed: AEON_GEN_MODEL, AEON_GEN_QUANT=modelopt, AEON_GEN_GPU_MEM_UTIL=0.40; the --quantization line is now ${AEON_GEN_QUANT}-parameterized). Served-names qwen3.6-27b-aeon/-thinking (gateway gen/gen-reasoning stay transparent). util 0.37 (~35G; dropped from 0.40 on 2026-07-08 to fund char-rp 128K — the RP seats are now co-resident, ~4.4G GPU0 margin). AEON + the intermediate Pantheon-27B serves are RETIRED.
    • char-rp (:8016) = TheDrummer_Magidonia-24B-v4.3-Q6_K GGUF — LIVE + EXCELLENT. Magistral (Mistral) dark-romantasy RP tune via llama.cpp (char-rp-gguf stack, container llama-charrp). NON-thinking PROSE seat: elite literary prose, zero refusal on dark/explicit scenes, ~65 tok/s, tight POV/instruction adherence (live-tested). Replaced the broken Angel NVFP4. Alt prose model (.env swap CHARRP_MODEL): MS3.2-PaintedFantasy-v4.1-24B (more literary flair, looser POV). Both GGUFs pre-pulled at /tank/aimodels/llm/rp/.
    • char-rp-reasoning (:8018) = Qwen3.5-27B-Deckard-PKD-Heretic-Uncensored-Thinking.i1-Q5_K_M GGUF — LIVE (A/B WINNER 2026-07-08, REPLACED RpR-v4). DavidAU creative (PKD) tune on Qwen3.5-27B (container llama-charrp-reasoning). Native Qwen3.5 template → --reasoning on managed CoT SURFACES in reasoning_content, content stays clean prose, --reasoning-budget 400 caps it; DRY server-side (sampler order = dry-after-temp) tames looping. A/B: 0/30 loops + 0/30 refusals, ~57 tok/s. Runs on the base llama-swap b8840 image (Qwen3.5). Best-of-breed-per-seat (NOT the same model as char-rp).

  • 🦄 RP-SEAT UNICORN — RESOLVED + DEPLOYED (2026-07-08). char-rp = Magidonia-24B-v4.3 (Magistral prose, 65tps); char-rp-reasoning = Deckard-PKD (Qwen3.5-27B) (managed reasoning + DRY, ~57tps; A/B winner over RpR-v4 — see re-A/B note below). Both GGUF via llama.cpp (char-rp-gguf stack, ana-ml2 GPU0, ~86/97G co-resident with gen, ~11G margin). Canonical stack in repo stacks/char-rp-gguf/; gateway rewired (char-rp→:8016, char-rp-reasoning→:8018, Mistral/QwQ samplers, dropped the Qwen enable_thinking kwarg). KEY FINDINGS: (a) no single dense 24-32B is BOTH an elite non-thinking prose seat AND a clean managed-reasoning seat on llama.cpp → best-of-breed per seat; (b) Magidonia's Magistral [THINK] discipline is loose (won't close [/THINK] on substantive reasoning → prose bleeds into reasoning, content empties); Cydonia-R1's <think> is emergent (llama.cpp can't manage/cap → runaway CoT never reaches prose); QwQ's template opens <think> natively → llama.cpp manages+caps it; (c) Pantheon-Reasoning-27B stays REJECTED (re-censors in <think>; RpR-v4's abliterated-reasoning is the fix); (d) reasoning now SURFACES in reasoning_content (fixes the old AEON trace-not-surfacing gap). One-model fallback (Magidonia both, lighter reasoning) documented in the stack header/README. Requirements met: prose#1, ≥50tps#2, low-refusal#3, dense#4, GGUF-not-Ollama#5, fit-GPU0#6, thinking#7. Candidate GGUFs also on disk for A/B: Cydonia-R1-24B-v4.1, PaintedFantasy-v4.1-24B, RpR-v4 Q6_K. REASONING-SEAT RE-A/B (2026-07-08, later — RpR→Deckard): operator wanted a reasoning model that TAKES DRY (RpR forbids rep/DRY → 1/30 loop tail). Full A/B on brokkr's 30-prompt D1 suite (content-only, slop-scored): Deckard-PKD (Qwen3.5-27B) WON (0/30 loops, 0/30 refusals, clean) → NOW the char-rp-reasoning seat. RpR-v4: 0 refusals but 1/30 loop (no-DRY). Pantheon-Reasoning-27B: 7/30 explicit refusals (DeepSeek-distilled re-censor IS real across the batch, milder than feared; NOT wholesale-rejected) — kept on disk as alt. Snowdrop-v0.5-Type-S + Gembrain-31B (Gemma-4): FLOORED — llama.cpp can't do MANAGED reasoning on them (Snowdrop's ChatML template has no <think>/enable_thinking hook; Gemma-4's reasoning parser splits content wrong). GATE for a llama.cpp reasoning seat = STOCK template natively opens <think> or has enable_thinking (Qwen3.x/QwQ do; ChatML + Gemma-4 don't). INFRA: llama-swap b8840 image can't load Qwen3.6/Gemma-4 archs → use ghcr.io/ggml-org/llama.cpp:server-cuda (newer, pulled on ana-ml2) for those; Deckard (Qwen3.5) runs on b8840. brokkr frozen-scorer CONFIRMED (median composite, content-only; lower=cleaner): Deckard 2.176 (0 loop/0 refuse — DEPLOYED, big win over RpR 3.716 + its loop-tail); Pantheon-Reasoning 1.383 (cleanest prose of the field — beats even Magidonia char-rp 1.774 — BUT 7/30 explicit refusals → rejected: clean-but-refuses loses to clean-and-uncensored). MEMORY-LAYOUT note: Deckard (Qwen3.5-27B) is GDN-hybrid → only 16 KV-caching layers → KV is TINY (1.4G @ 40K vs Magidonia's 10.9G @ 128K, 40 dense layers); Deckard native ctx = 256K, so the reasoning seat's ctx scales nearly free. Deckard BUMPED to full 256K (262144) 2026-07-08 (KV 8.7G q8_0). GPU0 layout now: gen ~35G (256K/16-seq/util0.37), char-rp Magidonia ~30G (128K/4-slot), char-rp-reasoning Deckard ~28G (256K/4-slot), ~4G free (all static allocations → stable, no OOM). Canonical Deckard RP-reasoning samplers CONFIRMED by dvalin 2026-07-08 (thread 01KX1Y7P): the live empirical set IS canonical — temp1.0/top_p0.95/top_k40/min_p0.05, NO presence/rep penalty, DRY0.8 server-side (base1.75/len2, dry-after-temp), reasoning-budget400. Delta from card base-thinking: top_k20→40, min_p0→0.05, presence1.5→0, DRY-on. Tuning ladder: flat prose→min_p0.08; loops→DRY0.9; over-damped→ DRY0.6/off. Do NOT import RpR/QwQ rules (QwQ hated DRY; Qwen3.5 benefits from DRY over rep_penalty). MAX CONTEXT (2026-07-08): char-rp 128K (Magidonia FULL native 131072), char-rp-reasoning 40K (QwQ native 40960, YaRN-free max), q8_0 KV cache both (near-lossless, ~2× ctx/GB; verified coherent, no Qwen gibberish). Funded by gen util 0.40→0.37 (freed ~2.9G of gen's IDLE KV headroom — gen KV usage runs 0-2%, so its 4.74×→3.66× concurrency-at-256K cut is invisible). GPU0 ~93/97G, ~4.4G margin (gen fixed-util + static KV → stable, won't OOM). 64/50 tps. kv_unified → a single conversation gets the FULL n_ctx (4 slots share the pool, not a /4 split). Tunables: CHARRP_CTX / CHARRP_REASONING_CTX / *_KV_TYPE / AEON_GEN_GPU_MEM_UTIL.

  • ✅ LiteLLM gateway sampling defaults WIRED for all 4 seats (2026-07-08) — dvalin-smithy canonical, infra-ops triaged + char-rp A/B-validated. gen (+summarizer-large twin, shared :8015 deployment): temp 0.7 / top_p 0.8 / top_k 20 / presence_penalty 1.5 (non-thinking). gen-reasoning: temp 1.0 / top_p 0.95 / top_k 20 / presence_penalty 1.5 (Qwen general-thinking profile — the old temp 0.6 was the CODING sub-profile by mistake; verbose ~9k-char CoT → callers need big max_tokens). char-rp: temp 1.1 / top_p 0.95 / min_p 0.10 / top_k 0, NO rep (A/B: min_p 0.10 richened imagery vs 0.03; rep 1.05 REJECTED — injected a markdown title in a grief scene, matches Drummer-no-rep). char-rp-reasoning: temp 1.0 / top_p 0.95 / top_k 40 / min_p 0.02, no rep/DRY. Canonical doc docs/pfi/model-sampler-defaults.md; dvalin derivation dvalin-smithy/hoard-drafts/pfi-gateway-sampler-defaults-20260708.md.

  • NVFP4/GGUF quant tooling (proven, on ana-ml2): /tank/aimodels/quant-work/ — nvfp4_quant.py (llm-compressor, MoE-aware load_quantizable_moe, Mistral-tuned ignore) + .venv; qwen35_vision_nvfp4.py (llm-compressor + qwen ignore + trust_remote_code + a torchvision-gate bypass). ModelOpt path: /home/lkraven/nvfp4-quant/.venv-mopt (modelopt 0.44) + mopt_w4a16.py (= mtq.quantize + export_hf_checkpoint → modelopt format w/ hf_quant_config.json). Downloads via a detached vLLM container running huggingface_hub.snapshot_download to /tank/aimodels/<name>-{bf16,nvfp4}. Gotchas: llm-compressor save_compressed DROPS tokenizer/chat/preprocessor files → stage from the bf16 source; the qwen ignore keeps re:.*visual.* + re:.*linear_attn.*(GDN) + lm_head + embeds + mlp.gate(router) bf16.

  • 🪦 T1 / home-trained LoRA — DORMANT (pivoted to off-the-shelf). DPO came back slop → operator killed home-training (2026-07-08). ONLY this project dormant; LEAVE arbo's LoRA-worker (irv-ml1 :8203), ALL /tank models, and every other project ("ritual seppuku" otherwise). Revival map: adapters data/spike/{sft,dpo}_adapter, harness canonical v0.0.44, full saga in Recent decisions + mtf-dev auto-memory; aeon-trainer image left (28G). mtf-dev stood down. (Detailed 2026-07-07 hot-swap findings preserved below + in Recent decisions.)

Earlier (2026-07-07):

  • T1 SFT COMPLETE (AEON-27B E-RP LoRA) — adapter banked, but BLOCKED on hot-swap serving. SFT ran clean on ana-ml2 GPU0 (all-TRL torch-GDN, batch1/accum8/GC-off, 1 epoch, ~9h @ 0.25 s/s steady, final loss 2.445); adapter at ana-ml2:~/model-training-forge/data/spike/sft_adapter (124MB, r16 α32, targets self_attn.q/k/v/o_proj + linear_attn.in_proj_qkv/out_proj, NO in_proj_z). DPO leg deferred (needs mergekit resolved + on-policy rejected regen). THE WALL: the adapter is INERT on the vLLM NVFP4 hot-load — loads HTTP 200 but zero deltas reach inference (vLLM 0.24.0 qwen3_5 LoRA-APPLICATION silent no-op; upstream #47639 open / fix PR #47640 open / regression from #37912). Proven quant-agnostic (NVFP4 AND FP8 both inert) AND adapter-format-agnostic (separate-key sft_adapter_zc loads clean via vLLM's allowlist but inert; the fused-key rekey rejected outright). The #47640 guard-patch overlay did NOT fix it. HARD REQUIREMENT (operator): hot-swappable or ABANDON — merge is NOT a solution now (static; reserved for a later RP-quant track only).

  • HOT-SWAP — SGLang zc smoke + 2-line patch RAN (2026-07-07). PARTIAL PROGRESS: patch cleared startup validation + self-attn LoRA loads end-to-end, but GDN in_proj hit 2 MORE name-routing gaps → NOT a one-line fix. MERGE RECOMMENDED (operator's call, pending). Ran sft_adapter_zc on SGLang 0.5.13 bf16 (--lora-target-modules all --max-lora-rank 16 --lora-paths t1=/adapters/sft_adapter_zc, GPU0, gen+rp stopped for the smoke then restored). Base weights loaded clean; self-attn q/k/v/o normalized + accepted fine. Died at lora_manager.init_lora_shapes with a NAME-VALIDATION ValueError (NOT the feared in_proj_qkvz shape-assert): adapter's in_proj_qkv+in_proj_z "not included in --lora-target-modules [...in_proj_qkvz...]". ROOT CAUSE (pinned): srt/lora/utils.py:235 get_normalized_target_modules uses a HARDCODED params_mapping dict that maps q/k/v_proj→qkv_proj (why self-attn works) but OMITS in_proj_qkv→in_proj_qkvz + in_proj_z→in_proj_qkvz; it does NOT consult the model's packed_modules_mapping (which DOES declare "in_proj_qkvz":["in_proj_qkv","in_proj_z"] at models/qwen3_5.py:1010). Everything else is ALREADY in place: get_hidden_dim("in_proj_qkvz") IS implemented in the model class (:1040, → (hidden_size, key_dim*2+value_dim*2); the "utils NotImplementedError" was only the DEFAULT); get_stacked_multiply("in_proj_qkvz")=4 in the utils default dict; base-weight stacked_params_mapping fuses in_proj_qkv→shards(0,1,2),in_proj_z→shard 3. PATCH APPLIED + RE-SMOKED (bind-mounted a patched utils.py overlay into the container): the 2-line add cleared init_lora_shapes; server came up "fired up and ready", LoRA kernels compiled (K=16384 in_proj_qkvz expand + K=6144 in_proj_z shrink — B-side math right per mtf-dev), self-attn loaded end-to-end ("loaded weights for [o_proj,out_proj, qkv_proj]"). BUT GDN in_proj died at TWO MORE name-resolution points (name-routing, NOT shape/intractable): (1) adapter-load SKIPPED 192 in_proj_qkv/in_proj_z weights ("did not match any target module [...in_proj_qkvz...]"); (2) inference buffer-fill utils.py:300 get_target_module_name RAISED — SUBSTRING match, in_proj_qkvz ⊄ ...in_proj_qkv. and qkv_proj ⊄ it → "Cannot find target module name for ...in_proj_qkv.lora_A.weight" → HTTP 500 + scheduler killed. So SGLang's GDN LoRA name-routing is incomplete in ≥3 code paths (fixed 1/3). FINISH LINE (if ever revived): teach get_target_module_name + the load-consolidation the in_proj_qkv/in_proj_z→in_proj_qkvz map (mirror qkv_proj, which works E2E), THEN mtf-dev's buffer shard-placement (0,1,2)+3 question goes live again (still UNPROVEN, deepest step). Per operator's "one targeted fix else merge" rule → this is multi-point, NOT one fix → MERGE recommended. mtf-dev replied + loop closed (thread 01KWZDZFYN…). (SGLang GENERIC image still can't LOAD our NVFP4 checkpoint — bf16-only smoke.) Adapter variants: sft_adapter (separate), sft_adapter_zc (separate + zero in_proj_z, the canonical one — THIS smoke), sft_adapter_fused (fused — wrong), plus sft_adapter_attnonly. All at /home/lkraven/model-training-forge/data/spike/.

  • MERGE FALLBACK — primed (~2-3h) if hot-swap is fully abandoned. peft merge_and_unload (~20min, LOW wall — separate keys merge directly, no rekey) → sanity greedy-diff (finally reveals if the ~10h training is any good — we've NEVER seen the LoRA apply) → ModelOpt NVFP4 re-quant of the merged bf16 (~1-2h, MEDIUM-HIGH wall = the GDN-hybrid requant + calibration + verify GDN modules quantize) → serve as plain base (no --enable-lora). Static, NOT hot-swappable.

  • comfy-dev in-arbo LoRA-training worker — DONE (all phases, this session). Built/deployed the worker (services/lora-training-worker/, irv-ml1 0.0.0.0:8203, runs as llmuser, fixed-invocation sd-scripts, systemd, elway deploy playbook; commits 888ba6a→b617a8b). Phase 1 first real train SUCCEEDED (Sindra fast/lean). Phase 2 publish-step (published_lora_name + optional train_id field). Phase 2.5 GC contract (shape A: GET /published-loras + POST /gc/published {train_ids}) CONFIRMED — worker impl DEFERRED to after the T1 hot-swap work. auto-memory reference_lora_training_worker.

  • Serving state (SUPERSEDED — see top; AEON retired 2026-07-08): during the T1 work, AEON gen (:8015) + char-rp (:8016) NVFP4 were HEALTHY, stopped/restored for hot-swap smokes via docker start vllm-aeon-gen vllm-aeon-rp. Now replaced: gen = 35B-A3B heretic, char-rp = (broken Angel, being replaced).

  • worldtree-dev vh/pypi read token delivered (bifrost==1.1.0 pin, #349); dedicated read:package token at nh3-dev:~/.config/gitea-tokens/vh-pypi-read.txt. The over-scoped write:package token the operator pasted was revoked + verified dead.

  • Backups — STILL OPEN: rotate the 5 disclosed rest-server creds (operator, offline); confirm esh-vm-db's resticprofile includes DB dumps. docs/runbooks/backups.md.

  • Standing / parked: inert mood.decay_rate/stale_hours keys in deployed WT config; Mac Pro migration (hw-gated); R22 gateway-key at /home/lkraven/.r22-gateway-key (mode 600, paid GLM, do NOT delete); Deckard staged on ana-ml2 as T1's writing benchmark; /books transient NFS mount on nh3-dev; dvalin sampler re-tune pass for char-rp / char-rp-reasoning once they accumulate live Worldtree/ST character-role traffic (dvalin offered 2026-07-08 to do a follow-up after more live sessions; the current paper defaults + 2-prompt A/B are the starting point — real long-session usage is where repetition/drift surfaces; re-open thread 01KX1DS6… to take them up on it).

  • Worldtree config-propagation (reference): demo+personal bind-mount config from /opt/worldtree{,-personal}/config (infra-ops-deployable, byte-identical from canonical); reload via docker restart <container>, NEVER compose up (stale-:latest footgun). Role registry loaded ONCE + CACHED at startup → a bind-mount model_roles.yaml change needs a restart; pre-stage BEFORE the activating deploy's restart. config REMOVALS are NOT backward-compatible with the running image.

Recent decisions

  • [2026-07-10] ComfyUI 0.25.x bump on irv-ml1 ATTEMPTED → FAILED → ROLLED BACK (snapshot saved it). comfy-dev requested (Vuong-authorized) bumping the irv-ml1 comfyui stack (mmartial image, /opt/docker/compose/comfyui/, 0.24.1) to 0.25.x for Krea-2 + LTXV 2.3. TWO FINDINGS: (1) DISABLE_UPGRADES=false/USE_PIPUPGRADE bumps the VENV (torch 2.12.1→2.13.0 + deps) but does NOT advance the ComfyUI CODE checkout (/comfy/mnt/ComfyUI = /worktank/comfyui/run/ComfyUI stayed 0.24.1 — pinned/detached git, comfy-dev's domain). (2) the torch bump broke SageAttention (2.2.0 _fused.so undefined-symbol c10::impl::cow::materialize_cow_storage vs torch 2.13.0) → --use-sage-attention (REQUIRED launch flag in COMFY_CMDLINE_EXTRA) crash-looped ComfyUI. Net: broke the working state, zero 0.25.x payoff. ROLLBACK WORKED: pre-bump 16G venv snapshot /worktank/comfyui/venv-snapshot-comfyui-0.24.1-20260710.tar restored (torch 2.12.1 + working SageAttention), re-pinned DISABLE_UPGRADES=true, recreated → healthy on 0.24.1, serving :8188. Broken venv parked at /worktank/comfyui/run/venv.broken-torch213-20260710. CORRECTED PATH (sent comfy-dev, thread 01KX655V): comfy-dev git-advances the ComfyUI checkout to 0.25.x + reqs → I handle the torch bump + SageAttention rebuild-against-2.13.0 + re-pin (snapshot stays as the net). LESSON: mmartial DISABLE_UPGRADES gates ONLY the venv pip-upgrades, NOT the ComfyUI git checkout; a torch bump breaks compiled exts (SageAttention) → rebuild-after is mandatory. Bump BLOCKED pending comfy-dev's git-advance. Stack: A6000 (NVIDIA_VISIBLE_DEVICES=1), lkraven-owned compose+venv (uid 1000, no sudo needed), COMFY_CMDLINE_EXTRA OOM flags preserved.

  • [2026-07-10] Biweekly open-weight-releases scan cron set up for brokkr-smithy (Vuong-authorized). Durable systemd --user timer on nh3-dev (brokkr-landscape-scan.timer, OnCalendar *-*-01,15 09:00:00 America/Los_Angeles, Persistent=true; linger on) → .service → wrapper ~/.local/bin/brokkr-landscape-scan.sh runs headless claude -p "$(cat ~/.config/brokkr-landscape-scan/prompt.txt)" --dangerously-skip-permissions in ~/development/brokkr-smithy (ALTHING_HANDLE=brokkr-smithy-dev; explicit PATH — the --user minimal-PATH footgun; per-run logs ~/.local/state/brokkr-landscape-scan/). Prompt = brokkr's payload verbatim (LLM/image/TTS new-release sweep → ranked synthesis → commit+push+notify). VALIDATED: git-push non-interactive (BatchMode ls-remote to gitea, passphraseless key — no agent), headless claude auth (READY smoke). VALIDATED END-TO-END 2026-07-10 (manual run, exit 0): web-sweep→synthesis→commit 2ed2f29→PUSH of scan #2 (open-weight-releases-2026-07-24.md); triaged dwarf input + caught baseline errors, quality strong. HANDLE-COLLISION caught+FIXED — the headless scan shared handle brokkr-smithy-dev with the LIVE session + raced its inbox (eitri's dwarf-reply got stolen by the live monitor); registered a dedicated brokkr-scan-dev handle (add-handle, driver=none) + repointed the wrapper + rewired step-5 notify → althing-cli post --to brokkr-smithy-dev (NO vuong althing handle exists — confirmed). model=default

    • --max-turns 80. First run under the new handle = 7/15. Off-cycle 07-24 doc is a validation artifact (scheduled 1st/15th runs date to their own run-date, no collision) — operator naming-convention call pending. NEXT AUTO-RUN 2026-07-15 09:00 PDT. Manual validation/first run = systemctl --user start brokkr-landscape-scan.service. Open w/ brokkr (thread 01KX63G6): confirm notify-Vuong handle/mechanism + session handle + model/turn-cap. NEXT brokkr task (operator-sequenced after this): TTS audition env — Higgs-TTS-3 + ZONOS2 + Chatterbox baseline, TTFA/RTF + blind-A/B web-listen (thread 01KX6371; needs GPU-placement + HF-token feasibility pass first; brokkr delivers the prompt set after the env's up; protocol doc in brokkr-smithy repo).

  • [2026-07-09] Two parked items closed: phantom qwen3.6-35b-a3b alias VERIFIED already-gone; ana-docker docker log-cap SOLVED no-bounce. (1) Phantom: absent from /v1/models + /model/info (config+DB registry), zero litellm log refs — the parked "400s in /v1/models" note was STALE (already cleaned in the 2026-07-08 gen repoint to -heretic); bare token survives only in 2 config COMMENTS (lines 76/80). Nothing to remove. (2) Log-cap: running containers were UNCAPPED (182M json-logs, top offender 59M) because daemon.json's max-size 10m/max-file 3 only applies to containers CREATED AFTER a daemon restart — it never reaches already-running ones. No-bounce fix = /etc/logrotate.d/docker-containers (copytruncate — dockerd opens json-logs O_APPEND so truncate-in-place resets cleanly, no sparse-file corruption; size 10M, rotate 3, compress, su root root), auto-picked-up by the daily logrotate.timer. Force-ran + gzipped the frozen .1 archives → 182M → ~55M (44M active + 11M gz), every container kept its multi-week uptime (zero bounce, verified). LATENT FOOTGUN FLAGGED (not yet fixed, operator's call): daemon.json declares live-restore:true but the RUNNING daemon has it FALSE (daemon.json was edited after the last daemon start, never reloaded) → the NEXT systemctl restart docker / crash / pkg-upgrade bounces ALL ana-docker containers once. Fix WITHOUT a bounce = systemctl reload docker (SIGHUP loads live-restore into the running daemon; log-opts are NOT SIGHUP-reloadable, which is why logrotate — not the daemon cap — is the enforcer for running containers).

  • [2026-07-09] granite→gen memory_extractor bind host-synced on demo+personal Worldtree (Vuong-directed, #335 Slice-4). Changed model_roles.yaml memory_extractor binds.catalog_id summarizer→gen (overrides intact: thinking:false/temp0/8192) on BOTH /opt/worldtree{,-personal}/config; memory_distiller left on summarizer (range-scoped sed /memory_extractor:/,/memory_distiller:/ — the naive global replace would've hit both); backups *.bak-preqgen-20260709; validated via each container's OWN yaml parser. DEMO LIVE on gen (b43 d501e516732d auto-deployed mid-edit + its restart RACED my edit by ~2min → I restarted worldtree-worldtree-api-1 to activate; healthy, live process resolves memory_extractor=gen). PERSONAL NOW LIVE on gen too — Vuong authorized the restart (via wt-dev); restarted worldtree-personal-worldtree-api-1 after a PRE-FLIGHT that ran the app's OWN load_model_roles() (core/llm/roles.py:121) against the synced config INSIDE the running c9986cd container: gen in catalog, all 9 roles resolve, no DanglingBindingError → proven safe on the OLDER image BEFORE touching it (model_roles-delta-alone clean; no full-config-set sync needed). StartedAt 20:50:55Z, healthy, resolves gen. BOTH instances live on gen. LESSON: the bind-mount /opt/worldtree*/config SHADOWS the baked /app/config-defaults/ → the deploy alone never updates the ACTIVE config; the host edit is required AND a restart activates it (role registry cached at boot) — pre-stage BEFORE the deploy's restart or you race it. FOLLOW-UPS (non-blocking): (a) memory.extractor.user_pass parity block → self-serve from the b43 baked defaults.yaml (pydantic-default no-op); (b) stale gen provider description (Mistral-Small-4 → qwen3.6-35b-a3b-heretic) → wt-dev owns the REPO-side providers.yaml fix (operator's call — NOT purely cosmetic: the gen/dialogue + classifier entries carry Mistral-Small-4 SAMPLING defaults that drive mask/lofn/forseti/mimir dialogue, so wt-dev re-validates for qwen), host cosmetic sync pairs when it deploys. Gave wt-dev the VERIFIED canonical 4-alias set (backends+samplers read from the LIVE gateway config, not the doc); corrected docs/pfi/model-sampler-defaults.md seat 4 (had lagged QwQ-RpR-v4 → Deckard-PKD; live gateway was always Deckard). Operator SCOPED IN the character-RP re-point (2026-07-09): character→char-rp / thoughtful-character→char-rp-reasoning (character-rp per wt-dev's role semantics), moving character RP off the GENERAL qwen onto the dedicated Magidonia/Deckard seats. Relayed to wt-dev w/ the mapping principle + a SAMPLER-OVERRIDE warning (DROP character-rp's old temp0.75/top_p0.85 overrides — carried onto the dedicated seats they'd clobber the canonical RP tuning DOWNWARD) + ratatoskr-reach note (role call is transparent but Magidonia/Deckard quality/latency differs from gen). DONE 2026-07-09: wt-dev committed 5d4fa4a (v1.0.0b44, UNPUSHED — operator drives push); operator directed host-ahead-of-push, so I sourced BOTH config files directly from that unpushed commit (local ~/development/Worldtree checkout — capital W; git show 5d4fa4a:config/…) + mirrored VERBATIM to /opt/worldtree{,-personal}/config on demo+personal, paired-pre-flighted via the app's load_model_roles() (no DanglingBinding), restarted both → LIVE: character→char-rp, thoughtful-character + character-rp→char-rp-reasoning, memory_extractor→gen preserved. Backups *.bak-prerp-20260709. context_window VERIFIED (llama.cpp /props + char-rp-gguf .env): char-rp 131072, char-rp-reasoning 262144 (gave wt-dev to patch the repo from its interim 32768). HOST AHEAD of repo-remote until the operator pushes 5d4fa4a (baked config converges with the bind-mount on push+deploy). GOTCHA: demo≠personal — PERSONAL was already partly re-pointed (2026-07-06 AEON-era character→char-rp) so its delta was mostly stale-AEON-descriptions→Magidonia/Deckard

    • character-rp + stripping personal's char-rp default_params temp0.7/top_p0.8 that CLOBBERED the gateway RP tuning downward; DEMO had no char-rp catalog entries at all (b44 adds them). Diffed each instance vs b44 before applying (both deltas = expected changeset only, nothing instance-specific clobbered). wt-dev PATCHED the context_window in b45/3384a37 (char-rp 131072, char-rp-reasoning 262144). BUMPED HOST-AHEAD on both instances (operator-directed 2026-07-09): mirrored b45's providers.yaml → /opt/worldtree{,-personal}/config, restarted, verified LIVE (char-rp ctx 131072, char-rp-reasoning 262144, bindings intact); backups .bak-b44interim-20260709. HOST now = b45 canonical (providers.yaml) + b44 (model_roles unchanged b44→b45). STILL PENDING: (1) operator's batched push of b44+b45 (5d4fa4a+3384a37) to converge the repo-remote — host is ahead, no fork; (2) user_pass parity block (defaults.yaml — NOT in either, separate). Threads 01KX3SGH/01KX48QP (worldtree-dev), monitor armed. See reference_corviduo_dev_emergency_ops.

  • [2026-07-09] mOrpheus TTS off-the-shelf voice pipeline SHIPPED end-to-end (irv-ml1) + wired into gateway-chat. Full arc (commits): gen served-name honesty rename aeon→qwen3.6-35b-a3b-heretic (99a4a17, vLLM served-name + litellm refs, so /v1/models + spend-logs name the real model); permanent 2-container stack (01eedd8); gateway-chat auto-voice quoted dialogue (c948013); streaming decode TTFA 4.5s→0.8s (da76829); max_tokens 1200→2400→3500 with a context-clamp (f363fe6, 0655a37 — long lines were clipping at 14.6s, and repetition_penalty 1.1 is LOAD-BEARING: at 1.0 the model never stops); AudioContext resume-on-gesture no-sound fix (033f368); pre-chunk by QUOTED SECTION not sentence for prosody (a1f3023→f295cc1); staged clone voices baddy/beatrice/whisper (0655a37 + runtime .wav/.txt in the voices dir); agent voicing prompt (a573514). Load-bearing config, all encoded in stacks/mOrpheus/: bf16 not FP8, image v0.23.0 not latest, GPU=3090 not A6000, rep_penalty 1.1. Serving-viability confirmed: vLLM concurrency near-linear to 8× (707 tok/s).

  • [2026-07-09] granite→gen memory_extractor bind GREEN-lit for worldtree-dev (Worldtree #335 Slice 4). Answered their VRAM/concurrency headroom check: gen (qwen 35B-A3B heretic) has ample headroom for ~2 bursty idle-triggered extractor calls (fixed 0.37 util; KV pool runs 0-2%; --max-num-seqs 16, near-linear batching). Corrected their stale "gen = Mistral Small 4 / 119B-6B" belief (gen IS the qwen 35B-A3B heretic since 2026-07-08). This bind is INDEPENDENT of the full granite RETIRE (reclaim ~32GB on ana-ml2 GPU1) — that stays the operator's call, pending brokkr R33 portfolio + production-concurrency due-diligence. Tracked: althing thread 01KX3SGH… (worldtree-dev) + brokkr's gen-absorbs-granite consult (thread 01KX2V32…) + reference_litellm_gateway.

  • [2026-07-08] RP-SEAT CAMPAIGN CLOSED — char-rp = Magidonia-24B-v4.3 (128K), char-rp-reasoning = Deckard-PKD Qwen3.5-27B (256K); both GGUF/llama.cpp on ana-ml2 GPU0 alongside gen (35B-A3B, util 0.37), ~4G GPU0 margin. Arc: (1) replaced broken Angel NVFP4 with Magidonia prose + QwQ-RpR-v4 reasoning (b268f93); (2) max-context via q8_0 KV (f570604); (3) canonical samplers for all 4 gateway seats, dvalin-derived + char-rp A/B-tuned (aac4bcf); (4) rebalanced gen 0.40→0.37 to fund char-rp 128K (f49c4e4); (5) RE-A/B'd the reasoning seat (operator wanted a DRY-tolerant model): Deckard WON on brokkr's frozen scorer (composite 2.176, 0/30 loops, 0/30 refusals) over RpR-v4 (3.716, 1/30 loop), Pantheon-Reasoning (1.383 but 7/30 refusals), Snowdrop+Gembrain (llama.cpp template-incompat) — deployed (5f79b40); (6) Deckard→256K (41305bf); (7) dvalin CONFIRMED Deckard samplers = the live A/B set is canonical (4954ca0). GATE LESSON: a llama.cpp reasoning seat needs a STOCK template that natively opens <think>/enable_thinking (Qwen3.x/QwQ pass; ChatML + Gemma-4 fail) — no monkeypatching. INFRA: llama-swap b8840 can't load Qwen3.6/Gemma-4 archs → ghcr.io/ggml-org/llama.cpp:server-cuda (pulled on ana-ml2).

  • [2026-07-08] worldtree Mimir deploy-blocker resolved (mid-session): synced persona.envelopes.assistant + envelope_grants:[] verbatim from the baked canonical into BOTH corviduo-dev instances (demo+personal), YAML-validated via each container's own parser; worldtree-dev cleared to push the Mimir-bound image. (Was my parked R32 1C envelope-mirror come due — see reference_corviduo_dev_emergency_ops config-sync recipe.)

  • [2026-07-08] OFF-THE-SHELF INFERENCE PIVOT executed — serve curated abliterated models, stop home-training. Final topology: gen = llmfan46/Qwen3.6-35B-A3B-uncensored-heretic-NVFP4-Experts-Only (LIVE, modelopt, vision, util 0.40), char-rp = an RP unicorn to be found on fresh context (see Current state). Intermediate steps ABANDONED: Pantheon-Reasoning-27B (served briefly as gen — refuses dark fiction via DeepSeek-distilled refusal-reasoning, see Tried); Pantheon-27B-with-MTP for RP (bf16 MTP won't load on the compressed-tensors path); Angel MS3.2-24B (my NVFP4 quant = garbage). Prefer EXISTING community NVFP4/GGUF quants over self-quanting ("don't quant unless you have to" — operator). GGUF serving is now on the table for RP (NEVER Ollama). Gateway sampling-defaults wiring still PENDING.

  • [2026-07-08] DPO was silently running 3 epochs (harness gap) → KILLED at epoch 1.2, retargeted to 0.3 epochs (operator call). Root cause: DpoConfig had NO epochs field + _dpo_config_kwargs didn't pass num_train_epochs → DPO fell through to trl DPOConfig's default 3.0 (SFT correctly pins 1 via SftConfig.epochs

    • _sft_config_kwargs). Objective SATURATED by epoch 0.27 (loss0, grad0, acc 1.0, margins27 flat — the off-policy qwopus rejected pairs are trivially separable), so epochs ~0.3→3 were pure over-optimization + a ~6.5h outage. No mid-run checkpoint (save_steps 500 > total steps; save only at end) → killing lost the run. FIX (3 edits to deployed harness, mtf-dev to canonicalize): DpoConfig.epochs: float = 1 (mirrors SftConfig, float for fractions); _dpo_config_kwargs now passes "num_train_epochs": cfg.epochs; recipe dpo.epochs: 0.3. GPU-free verified (dpo.epochs=0.3 → num_train_epochs=0.3). Relaunched at 0.3 epoch (~30min precompute + ~12min train = ~45min). DONE + SANITY-CHECKED (exit 0, ~70min wall: ~30min fixed precompute + 45 steps @ ~51s/step; train_loss 1.4e-5 @ epoch 0.301). Fresh data/spike/dpo_adapter/adapter_model.safetensors (123MB) + checkpoint-45 banked. 3-way greedy sanity (base vs SFT vs DPO, via peft load + disable_adapter/set_adapter on GPU0): ALL THREE DISTINCT (base≠sft≠dpo) → full SFT→DPO pipeline applies end-to-end at inference. DPO 0.3ep is COHERENT, fluent, NOT degenerate (early-stop avoided over-optimization) but the quality delta on a neutral literary prompt is SUBTLE (DPO shares SFT's structure — it continues from it — with minor stylistic drift, arguably slightly MORE genre-clichéd). Verdict: mechanics proven, quality gain modest as predicted for 0.3ep on off-policy pairs; the real unlock remains on-policy rejected regen + on-domain (explicit E-RP) eval + the LitBench/holdout run. gen+rp RESTORED healthy. Next: serve fork (SGLang-finish vs merge) on the DPO adapter — same rekey_lora_for_vllm.py (zero-z) applies unchanged (mtf-dev confirmed).

  • [2026-07-08] T1 DPO leg is RUNNING (unblocked) — 2 fixes applied to deployed backend.py. Blocker resolved: (1) mtf-dev's v0.0.42 stub _stub_missing_optional_integrations (last-resort sys.meta_path finder → missing mergekit/llm_blender/weave resolve to MagicMock, never called → zero numerics risk; applied VERBATIM to deployed src/model_training_forge/train/backend.py after _unsloth_available() + call-site before from trl import DPOTrainer); (2) my cosmetic warnings_issued shim (trl-0.24 DPOTrainer.init:405 does model.warnings_issued["estimate_tokens"]=True for warning-suppression; custom Qwen3_5 class under transformers 5.5.0 lacks the attr → if not hasattr(model,"warnings_issued"): model.warnings_issued={} before the DPOTrainer(...).train() at backend.py:305 — cosmetic, zero training impact). Both edits are on the DEPLOYED un-git'd copy only → mtf-dev must canonicalize the warnings_issued shim into their repo (told them). DPO confirmed training: model loaded (851 shards), full 1196 pairs processed, in precompute_ref_log_probs (GPU0 93% util, 54.8GB). Completion watcher armed (bg task) → restore gen+rp + verify dpo_adapter + ping mtf-dev on exit. gen+rp STOPPED for the run (authorized window). Output → data/spike/dpo_adapter.

  • [2026-07-08] T1 DPO leg launch — prior BLOCK (now resolved above), kept for the launch recipe. Operator authorized the full DPO stage (via mtf-dev) + went AFK 2h. PROVEN LAUNCH RECIPE (replicates the SFT container aeon-t1-sft exactly, only --stage sft→dpo): sudo docker run -d --name aeon-t1-dpo --entrypoint python3 --gpus all -e CUDA_VISIBLE_DEVICES=0 -e MTF_FORCE_TRL=1 -e PYTHONPATH=/mtf/src -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True -v /home/lkraven/model-training-forge:/mtf -v /tank:/tank -w /mtf aeon-trainer:latest -u scripts/train.py --recipe recipes/training/qwen-3.5-122b-erp-lora/train.aeon-27b.yaml --stage dpo. CRITICAL: --entrypoint python3 is REQUIRED — aeon-trainer's default entrypoint is ["vllm","serve"] (FROM vllm/vllm-openai) → without the override it runs vllm + hits a torch-ABI crash. Dataset verified (pairs_dataset=train.flat.json=1196 pairs). THE BLOCK: from trl import DPOTrainer (backend.py:256) eagerly pulls TRL 0.24.0's WHOLE optional-integration set — cascade: mergekit(missing)→ immutables→mergekit-0.1.4↔pydantic-2.13 HARD incompat (needs pydantic==2.10.6)→llm_blender→dataclasses_json→ llm_blender-0.0.2↔transformers-5.5.0 HARD incompat (TRANSFORMERS_CACHE removed, needs source patch)→weave→ (more). NONE used by our pair-based DPO. pip install mergekit w/deps is UNSAFE (downgrades accelerate 1.14→1.6). Safe partial recipe derived (core libs held: torch2.10/tf5.5.0/trl0.24.0/peft0.19.1/accel1.14.0) but non-convergent → TRULY BLOCKING per operator's carve-out. Did NOT force-hack the proven training image. Handed full diagnosis + recommended fix (lazy-import TRL patch, opt b) to mtf-dev (thread 01KWZG8GJX, expects-reply, monitor armed). gen+rp RESTORED healthy. Relaunch = 1 min once mtf-dev delivers a working image.

  • [2026-07-07] SGLang zc hot-swap: 2-line patch RAN → PARTIAL. Cleared startup + self-attn E2E, GDN in_proj hit 2 MORE name-routing gaps → MERGE recommended. Root cause was a NAME-routing gap (not the feared shape-assert): lora/utils.py:235 hardcoded params_mapping lacked in_proj_qkv/in_proj_z → in_proj_qkvz. Patched those 2 lines (overlay bind-mount) → init_lora_shapes passed, server up, self-attn loaded end-to-end. But the SAME missing GDN mapping bit 2 MORE resolvers: adapter-load skipped 192 in_proj weights, and inference utils.py:300 get_target_module_name (substring match) RAISED → 500 + scheduler died. So GDN LoRA name-routing is incomplete in ≥3 SGLang paths (fixed 1). NOT shape-intractable (self-attn proves the separate-key→fused-buffer machinery works), but NOT "one targeted fix" either → per operator's LAST-shot rule → MERGE. mtf-dev looped + closed (thread 01KWZDZFYN…); B/A shard-math never reached. Supersedes the "SGLang wants a THIRD format" framing — no new adapter format needed.

  • [2026-07-07] T1 (retargeted qwopus-122B→AEON-27B) SFT COMPLETE on ana-ml2 — all-TRL torch-GDN, batch1/accum8/GC-off, 1 epoch ~9h @ 0.25 s/s, loss 2.445, adapter banked. Supersedes the 2026-07-05 cloud-vs-smoke venue decision: AEON-27B's 52G bf16 fits one Blackwell → ran on-prem in ~9h (no cloud, no full-fleet outage). GC is NOT the throughput lever (batch1-no-GC == batch4+GC at 0.25; dvalin's 0.565 was warmup-noise from a 30-sample smoke); ~0.25 s/s is the torch-GDN floor (only FLA beats it, sm_120- blocked #913). reference_t1_cloud_train_plan.

  • [2026-07-07] T1 hot-swap on vLLM NVFP4 is DEAD — vLLM 0.24.0 qwen3_5 LoRA-application silent no-op (#47639 open / fix PR #47640 open / regression from #37912). Proven quant-agnostic (NVFP4+FP8) AND adapter-format-agnostic (separate zc loads-but-inert; fused rejected); #47640 guard-patch overlay didn't fix. Root-caused by a 3-peer dwarf panel (eitri/regin/dvalin); their vLLM issue refs verified real via GitHub API. Engine mirror-image: vLLM wants SEPARATE keys, SGLang wants FUSED.

  • [2026-07-07] HARD REQUIREMENT restated: hot-swappable LoRA or ABANDON. Merge is NOT a solution for this requirement (static; reserved for a later RP-quant track). SGLang is the last hot-swap shot; if it can't apply, park the adapter + file upstream repros. reference_lora_training_worker (adjacent).

  • [2026-07-07] comfy-dev in-arbo LoRA-training worker DONE — built/deployed/validated Phases 1+2 (first Sindra train succeeded, publish-step + train_id), Phase 2.5 GC contract (shape A) confirmed + impl deferred. services/lora-training-worker/, commits 888ba6a→b617a8b. reference_lora_training_worker.

  • [2026-07-06] infra-ops NOPASSWD sudo confirmed on irv-ml1 + ana-ml2 (the arbo memory's "NOT infra-ops on irv-ml1" was stale) — enabled the worker deploy + every ana-ml2 hot-swap smoke.

  • [2026-07-05] T1 training venue: CLOUD recommended; operator chose smoke-first on ana-ml2. On-prem ruled out (ana-ml2 full — both 96G GPUs ~93G used): keep-serving = NVMe offload ~6-8 DAYS; full ana-ml2 shutdown = CPU offload ~1 DAY but a whole-fleet outage. Cloud Vast.ai 8×80GB (no offload → ~3-6h, ~$60-500, zero fleet impact) is the clean alt (mtf-dev + infra-ops both rec; Vast for its no-content-AUP marketplace + likely-existing VastBlue account). Operator's next step = the ana-ml2 CPU-offload SMOKE (~60 steps) to get real samples/sec before the full-outage-vs-cloud call. HF base verified public (zero-auth pull). Runbook + gotchas in reference_t1_cloud_train_plan.

  • [2026-07-05] glm-5.2 canonical limits recorded (probed live vs z.ai): 1,048,576 (1M) input context / 131,072 (128K) max output; NO gateway-side cap (pure passthrough → z.ai's limits are effective). Written to the config comment (commit 624a07e) + reference_litellm_gateway.

  • [2026-07-04] character-rp: gateway-forwarding proven + role pre-staged + #344 shipped. Empirically confirmed per-request extra_body (top_k/repetition_penalty) forwards through the gen-reasoning LiteLLM alias to vLLM + standard params override the alias defaults — no gateway cap needed (I over-built a dedicated alias, operator corrected, reverted with zero fleet impact). Pre-staged the character-rp role into demo+personal bind-mount model_roles.yaml (byte-verified on b18; caught the cached-registry ordering). worldtree-dev shipped #344 (v1.0.0b19) for the durable-agent override-drop. spend_logs spot-check queued (ratatoskr's timestamp ping).

  • [2026-07-04] althing v2 herald+receiver formalized as systemd on nh3-dev. althing-herald.service (Restart=always, Environment=PATH incl ~/.cargo/bin — the pane-dispatch fix) + althing-receiver.service (v2 → pillar-3 /owner/* live); stale forseti unit removed; both on v2.0.0b2, canonicalized by forseti. reference_nh3_dev_althing_herald.

  • [2026-07-04] LitBench-RM respun (irv-ml1 A6000, comfyui displaced) for T1's reward ensemble; operator sole comfyui consumer, holding image-gen until LitBench done. reference_litbench_rm_irv_ml1.

  • [2026-07-03] ratatoskr-dev DEMO Heimdall key provisioned (R30 φ0). Minted a tier-user key on the demo via POST /admin/keys (bootstrap admin key), mirroring their personal base consumer (no character-binding); base-agent affect reads work ungated. reference_worldtree_demo_key_mint.

  • [2026-07-02] mtf-dev granite harness-spike ran GREEN — MECHANICAL only, efficacy DEFERRED to the T1 run. Trainer TRL SFT→DPO→eval seam proven end-to-end on a synthetic fixture (not the E-RP corpus); operator DECIDED no intermediate real-efficacy granite spike (uninterpretable proxy — arch gap + abliteration axis). reference_gen_qwopus_122b.

  • [2026-07-01] Worldtree #332 embed-recall diagnosed + scoped-log view/tunnel provisioned + fix verified (15×→1.01× re-embed). reference_wt_gateway_scoped_log_view.

  • [2026-07-01] qwopus native MTP speculative-decode tested on gen → NOT kept (+12% single-stream, −15–20% aggregate at concurrency, silently drops min_p/logit_bias). Banked for T1. reference_gen_qwopus_122b.

  • [2026-07-01] Deckard trial → reverted to qwopus (gen) (won writing "in every way" but ~36 vs ~90 tok/s; spec-decode rescue ruled out). git b63c48b→681eb70. Deckard kept staged as T1's writing benchmark.

  • [2026-06-14] STANDING DIRECTIVE: migrate ALL infra access to Claude-specific credentials. (auto-memory project_migrate_infra_access_to_claude_credentials)

125 older entries archived to archival-memory.md.

Tried and abandoned

  • [2026-07-09] FP8 breaks mOrpheus audio-token generation. --quantization fp8 on the 3B → 0 valid SNAC frames even at GREEDY (degenerate audio+text mix, no start-of-speech); bf16 is clean (28/28 frames). Quant-breaks- TTS, same class as the Angel-NVFP4 lesson below. bf16 is REQUIRED (so the operator's "util 0.1" wish is moot — the bf16 weights alone are 6.6GB). NB the raw-token benchmark RTF 0.50 was fp8+graphs = never real.

  • [2026-07-09] vllm/vllm-openai:latest crashes on Ampere IMPORT — Blackwell-only kernels (oink/aiter, has_device_capability(100)) die during import on the 3090/A6000. Pin v0.23.0 on irv-ml1's Ampere GPUs. (vllm/vllm-omni:v0.18.0 has a different entrypoint — don't use it either.)

  • [2026-07-09] Per-frame CPU SNAC decode is too slow for streaming — per-call overhead × ~60 frames serialized → RTF 2.2 (WORSE than whole-clip's 1.0). Fix = windowed chunk decode (every 6 frames decode a [2 ctx | 6 | 2 ctx] window, emit the middle 6 → seamless, O(1)/frame, RTF ~0.97, TTFA ~0.8s).

  • [2026-07-09] Sentence-chunking TTS loses prosody — generating each sentence cold flattens the intonation that spans a line. Chunk by QUOTED SECTION (whole quote = one gen call). Also: repetition_penalty >1.1 BREAKS cloning (penalizes the ~1100 in-context reference audio tokens; keep ≤1.1 on the clone path).

  • [2026-07-09] HF whisper datasets aren't actually whispered. Claris-Whispered-English measures voiced 0.8 (not a whisper) + IPA transcripts; datasets audio decode needs torchcodec (wants CUDA-13, incompatible w/ the cu124 venv). LPC DSP-whisperize went unstable (NaN). kokoro af_nicole IS a genuine whisper (voiced 0.24) — that (operator's pointer) is the working whisper reference source, not TTS-voice screening or dataset-hunting.

  • [2026-07-08] Angel (allura-org/MS3.2-24b-Angel) self-quanted to NVFP4 = GARBAGE. llm-compressor W4A4 NVFP4 (compressed-tensors, MLP-quantized, attn/vision bf16) of the Mistral3 dense 24B produces gibberish EVEN AT GREEDY (temp 0) → the quant itself is broken, not the tokenizer or sampler. Same recipe worked on the qwen models. Mistral3 + W4A4 NVFP4 via llm-compressor is bad. → for the RP seat, going GGUF (llama.cpp) to sidestep the whole NVFP4-quant surface.

  • [2026-07-08] Mistral3 + vLLM tokenizer/vision traps (serve MS3.2-24b, vLLM 0.24). (a) HF tokenizer.json for Mistral = GARBAGE output — the card's "use the official Mistral tokenizer" warning is REAL; must use the tekken.json/mistral tokenizer. (b) BUT --tokenizer-mode mistral + vision CRASHES (Failed to apply PixtralProcessor on {'text': '[IMG]'}; and with tekken.json present in auto mode, CachedMistralCommonBackend has no attribute is_fast). So it's mistral-tokenizer OR vision, not both on this vLLM. Text-only + mistral tokenizer serves clean (--limit-mm-per-prompt '{"image": 0}'). GGUF/llama.cpp avoids all of this (native mistral tokenizer + vision).

  • [2026-07-08] Pantheon-Reasoning-27B refuses dark fiction DESPITE an abliterated base. The base (llmfan46 heretic) writes freely (thinking-off), but Gryphe distilled the reasoning traces from DeepSeek 3.2 (safety-aligned) onto every turn (preserve_thinking:true) → the model reasons ITSELF into refusals in the <think> phase (collapses to empty output). Fix: thinking-off OR an uncensor system prompt (both verified). Lesson: a reasoning finetune of an abliterated base can re-censor via its reasoning-trace TEACHER; the raw abliterated base is cleaner — this is WHY the pivot went to the llmfan46 heretic base for gen.

  • [2026-07-08] Pantheon-27B MTP on vLLM compressed-tensors = 0% acceptance. MTP is a separate bf16 head (mtp.*, in model-auxiliary.safetensors, 15 tensors); AEON preserved it by INJECTING the bf16 head into the quant output (NOT re-quantizing — confirmed AEON's nvfp4 mtp is bf16). Built pantheon-27b-mtp = compressed-tensors main + injected bf16 mtp + text_config.mtp_num_hidden_layers=1 → vLLM detected the MTP but SKIPPED the bf16 self_attn weights → 0/192 draft tokens accepted. The bf16 MTP head only loads on the MODELOPT main-model format (like AEON), not compressed-tensors. (Moot — operator dropped MTP for gen; not needed for the non-reasoning RP.)

  • [2026-07-07] vLLM 0.24.0 qwen3_5 LoRA application = silent no-op (#47639). Adapter loads HTTP 200 but zero deltas at inference. NOT quant (NVFP4 AND FP8 both inert). NOT adapter format (separate zc adapter — correct per vLLM's check_unexpected_modules allowlist — loads clean but inert; the fused-key rekey is rejected). The #47640 None-group guard-patch overlay did NOT fix it (failure is UPSTREAM of expand_packed_lora — the separate→fused mapping never happens). Fix PR #47640 is OPEN (unmerged) so no version-bump helps. Merge bakes deltas in (bypasses this) but is static.

  • [2026-07-07] SGLang generic image can't LOAD our NVFP4 AEON — ModelOptModelLoader weight-shape/ packing mismatch ([1024,5120] vs [1024,2560], 2-fp4/byte). NVFP4-on-SGLang needs the dedicated qwen36-27b-nvfp4 dev image or a requant to SGLang's format. bf16 loads fine (arch supported; crash was quant-loader-specific).

  • [2026-07-07] SGLang --lora-target-modules CLI enum REJECTS the GDN names its own resolver asks for (invalid choice: 'in_proj_qkv'); 'all' resolves to the FUSED set (qkv_proj/in_proj_qkvz). SGLang wants its OWN packed layout (base r16 + get_stacked_multiply=3, NOT a pre-fused rank-48 qkv → the [48]-vs-[144] shape assert). A THIRD adapter format; version-exact source needed (:latest=0.5.13, NOT main).

  • [2026-07-07] Engine invocation footguns cost several wasted serve-bounces this session — docker run --rm ate crash logs; duplicated serve (vLLM image entrypoint is already ["vllm","serve"]); --max-lora-rank 48 invalid (choices 1/8/16/32/64… → use 64); parens in echo inside ssh host -c "…" break the remote shell. LESSON: verify engine launch flags (--help, GPU-free) + never --rm a container whose crash logs you need, BEFORE bouncing a production serve.

  • [2026-07-04] LiteLLM (this gateway version) mutates the SHARED deployment config in-place on per-request sampler-param merge → my deliberately-invalid top_k=-5 forwarding-probe bled into a param-less character-rp request (vLLM 400, ONE-OFF, self-cleared by a later valid probe). NOT caching (none configured), NOT a config change. Never fire invalid/distinctive sampler values at a SHARED gateway alias with live consumers — use a throwaway alias, or a docker restart litellm flushes residual carryover. feedback_litellm_shared_param_mutation.

  • [2026-07-04] A systemd --user daemon that shells out to ~/.cargo/bin/~/.local/bin tools needs an explicit Environment=PATH — the minimal --user default silently drops them. The althing herald lost zellij → silent pane-miss for ALL config-backed TUI/pane agents; CC + FIFO routes were unaffected, so it was invisible from a CC session. reference_nh3_dev_althing_herald.

  • [2026-07-04] On-prem T1 train that keeps ANY ana-ml2 serving up = ~6-8 DAYS (1-GPU + NVMe ZeRO-Infinity offload; MoE ~10B-active cuts FLOPs but NOT the 244G base's param I/O). The only fast on-prem path is a FULL ana-ml2 shutdown (both GPUs + the ~421G vLLM RAM freed → base fits in the 566G CPU RAM) → CPU offload → ~1-day full-fleet outage. Cloud (no offload) = hours. reference_t1_cloud_train_plan.

  • [2026-07-01] A personal-Worldtree CI deploy that fails ~85s in with "not found / unauthorized" is usually the pull-only-vs-build RACE, not registry-auth. deploy-personal.yml is PULL-ONLY but fires on the staging/vX tag simultaneously with deploy.yml's build → pulls before the push finishes. FIX: re-run once built, or gate on workflow_run: completed.

  • [2026-07-01] MTP/spec-decode on a SHARED serving model helps single-stream but HURTS moderate-concurrency aggregate + silently ignores min_p/logit_bias (qwopus gen: N=1 +12%, N=4 −20%). Reserve for dedicated/interactive deployments.

  • [2026-07-02] irv-ml1 /worktank ROOT is root-owned — lkraven can't write there (irv-ml1 sudo needs a password) → stage model pulls to /home. PIN THE A6000 BY UUID for training (native-CUDA ordering differs vs docker; the 3090 index 0 is usually near-full → OOM). CUDA_VISIBLE_DEVICES=GPU-<uuid>.

101 older entries archived to archival-memory.md.

Reference in New Issue View Git Blame Copy Permalink