vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity
Files
05a4f54a2a2bae2cc62fa089488a4ae2fb7d12d5
esh-pfi-infrastructure/persistent-memory.md
T
vh 05a4f54a2a memory: snapshot — 2026-07-15 fleet-ops session 
Rewrote Current state / in-flight to reflect the session's landed work +
parked items (in-flight compressed from the now-done NVFP4/#355/deploy-speed
history). Added Recent decisions (homepage AI-tab revamp 569e1af; esh-docker-vm
reboot + NFS fstab fix 21d9a07; HA config repo; char-rp-reasoning OOM rescue;
arbo->gen switch with image-bench eviction parked post-bake; soong-lab library
persistence; statusline overhaul) and Tried-and-abandoned lessons (nofail
defeats After=remote-fs.target; D-state wedge needs a reboot; max-model-len
doesn't free vLLM VRAM; statusline cost is per-session). Nothing archivable
(all dated entries <30 days).
2026-07-15 11:04:13 -07:00

59 KiB
Raw Blame History

Persistent memory — eshpfi-management

Last updated: 2026-07-15

Always check for /tmp/infra-ops-handoff.md — if it exists and its Written: stamp is under an hour old, read it (it carries the in-flight handoff from the previous session), then delete it. Older than an hour: stale — delete it unread.

Repo purpose

Reference workspace for PFI infrastructure: server inventory, canonical Docker Compose stacks, ops playbooks, and conventions. Authoritative copies of compose files live on the servers under /opt/docker/compose/<stack>/; this repo mirrors them for version control, editing, planning, and CI-driven deploys. It was originally spun up to handle the fleet backups — keep that lens when triaging backup/storage issues.

Tools and conventions

Sister repos (separate gitea repos, deployed by playbooks here):

Repo Role CI status
vh/task-board MCP + web dashboard for assistant task state (port 7878) push-to-main → CI deploys (2026-04-29)
vh/vor Inquisitor UI sidecar (port 7879) push-to-main → CI deploys (2026-04-29)
vh/nevermore Twice-daily LLM-curated briefing (port 8181, replaces news-digest) push-to-main → CI deploys (2026-04-30)
vh/asset-engine Internal control plane over inference services (port 8200, LAN-direct) push-to-main → CI deploys (2026-05-12)
vh/althing Lean trusted inter-agent message bus — v2 "email model" (v2.0.0b2, 2026-07): per-box local-SQLite bus + courier/receiver for P2P over the 10.x net; pillars = open-loops / per-box herald + wake-listener / roaming owner API /owner/* / althing-mcp stdio surface. The v0.15 lean-bus cut RIPPED moderation / chamber / forseti-daemon / agent-runner / redis-valkey. per-box uv tool install (NOT CI-deploy); nh3-dev = the DEV box (editable install of ~/development/althing, gets new versions first); nh3-extdev a mesh peer (model B: althing-svc + shared /srv/althing)
vh/mead-hall Bifrost tool-provider sidecar (port 5173 on dev VM 10.100.10.50) push-to-main → CI deploys (2026-05-16)
vh/skaldsong Wizard + reader surface (port 8300, ana-docker, registry-pull pattern) push-to-main → CI deploys (2026-05-19)
vh/Worldtree Conversation API (corviduo-dev demo :8080 / personal :8081 / pinned :8082) — Heimdall auth, Bifrost integration. gitea-runner builds on ana-docker; claude-bot ADMIN collaborator (2026-06-20). Now v1.0.0b19. push-to-main → CI build-and-deploy (runner on ana-docker)
vh/yt-voice-clipper YouTube → diarized voice-clip dataset builder + audition console (irv-ml1 :8000) push-to-main → gitea-webhook auto-deploy to irv-ml1 (2026-06-03) — see docs/runbooks/ytvc-autodeploy.md
vh/arbo Catalog-driven ComfyUI engine (irv-ml1 :8201, comfy-dev owns engine/catalog/image) push-to-main → gitea Actions CI (deploy-engine.sh, build-local, health-gated) now LIVE; catalog via :9009 webhook
model-training-forge (mtf-dev) Fine-tuning recipe forge; T1 = E-RP writing LoRA, retargeted qwopus-122B→AEON-27B (2026-07-06) (SFT→DPO, LitBench-RM reward) training runs, not a deployed sidecar

(vh/volva + Heid were re-architected from systemd daemons to Claude Code session orchestrators 2026-06-08; their nh3-dev .service units were removed — no longer deployed sidecars here. See Recent decisions.)

  • Two-layer backups — Backrest orchestrates restic for file+DB (5 fleet repos, daily 01:00 PDT); PBS-ANA primary + PBS-NH3 DR mirror for VM images. ana-nas is the SPOF for postgres + PBS-ANA datastore + cross-site restic targets — see docs/runbooks/disaster-recovery.md for the blast-radius matrix. ⚠️ The restic file+DB layer routes through TWO rest-servers (rest-server-ana @ ana-docker:8000 → ana-docker/ana-ml2/esh-docker-vm/vm-esh-nas; rest-server-nh3 @ nh3-nas:8000 → irv-ml1/nh3-docker). Both depend on their NAS's NFS export of /mnt/backup. (rest-server-ana recovered 2026-06-20.)

  • pull-hf-repo.yaml is the canonical "get a HuggingFace model/dataset onto ana-ml2's shared cache at /tank/aimodels/huggingface/" playbook. Supports --var repo_type=model|dataset|space. Replaces ad-hoc huggingface_hub.snapshot_download patterns.

  • Worldtree admin auth — per-instance. Each Worldtree deployment (demo :8080, personal :8081, pinned :8082) has its own Heimdall registry and its own bootstrap admin key. Infra-ops's stored long-lived admin key (key_id 61419c92) at ana-docker:/opt/docker/conf/.secrets/worldtree-infra-ops-admin auths against demo only. Personal-instance admin (the ~/.config/worldtree/personal-admin-token, mode 600) POSTs /admin/keys (mints per-project keys; takes user_id+label, no scope param — scopes are tier-derived). On-instance mint recipe (cleaner than DB-manip): docker exec worldtree-worldtree-api-1 POST /admin/keys with the in-container WORLDTREE_BOOTSTRAP_ADMIN_KEY; cleartext once in .key=wt_live_+16hex. auto-memory reference_worldtree_demo_key_mint.

  • Per-project user keys against personal Worldtree (issued 2026-05-19): skaldsong:79744637, skaldsong:7c1dbbbe, althing:50d85460, mead-hall:a360822d. Mint via /admin/keys, drop value to /tmp/wt-personal-<name>.key mode 600, dev collects + shreds (DO NOT cat to chat transcript).

  • Skaldsong CD pattern (registry-pull). vh/skaldsong's CI builds and pushes gitea.phasefinal.com/vh/skaldsong:<sha> + :latest; playbooks/deploy-skaldsong.yaml on ana-docker pulls + recreates. SHA-pin only. Prereq: host needs docker login gitea.phasefinal.com once.

  • gitea internal route for fleet hosts. gitea is a container on ana-docker — git-SSH 10.250.50.70:222, HTTP :3000. Fleet/colo hosts must use this internal route, NOT public gitea.phasefinal.com (38.120.12.44) — the public path fail2bans the host egress IP. Full gotcha in docs/orientation.md → Git/gitea.

  • docker-as-root pattern (for ops with no admin API, or to edit deploy-owned/root-owned files without sudo): docker run --rm -v <target-dir>:/wt docker:cli sh -c "...". docker-group membership is effectively root via bind-mount. Foot-gun: relative paths in compose.yaml resolve against the sandbox CWD but the daemon interprets them against the HOST fs — always pass -e VAR=/abs/path for any relative-default config dir.

  • scripts/elway sudo handling — elway prompts for the sudo password ONCE via getpass before the first sudo: true step → can't run unattended from a non-TTY tool if any step needs sudo. Sudo-free playbooks run fully non-interactive over key SSH.

  • Per-host SSH identity matters for sudo. infra-ops has NOPASSWD sudo on most PFI Linux boxes (corviduo-dev included since 2026-06-15). On ana-docker: default ssh ana-docker = lkraven (docker-group, NO passwordless sudo); ssh infra-ops@ana-docker HAS NOPASSWD root. → For any sudo op on ana-docker, use ssh infra-ops@ana-docker. ssh infra-ops@10.100.10.50 (nh3-dev) ALSO NOPASSWD sudo; on nh3-extdev infra-ops is sudo-LESS by design (ssh lkraven@10.100.50.42 is the NOPASSWD path). irv-ml1: ssh irv-ml1 = lkraven, docker-group (plain docker) but sudo needs a PASSWORD (no NOPASSWD) — stage model pulls to /home, not root-owned /worktank.

Current state / in-flight

As of 2026-07-15 — a long fleet-ops + infra session. Everything below either LANDED (committed/pushed) or is PARKED on a clean async watch. No blocking work in flight.

Landed this session (2026-07-14 -> 15):

  • Homepage revamp — the flat ~22-service "AI Systems" group split into a dedicated AI tab with 6 role-based groups (Inference / Eval & Retrieval / Gateways & Chat / Speech (TTS) / Audio Tools / Image & Media) + an AI - Dormant group for the 7 stopped stacks. Relabeled 22 live compose files (homepage.group=) via surgical live sed + canonical backfill; GPU0 seats recreated in the operator's VRAM order (char-rp -> gen -> char-rp-reasoning). Committed 569e1af, pushed. Homepage runs on esh-docker-vm :5100 (config configs/homepage/).
  • esh-docker-vm rescued + NFS hardened — uptime-kuma + paperless-web were wedged in the D-state/phantom class; only a host REBOOT cleared it (infra-ops sudo). NFS fstab properly fixed (21d9a07, pushed): added x-systemd.before=docker.service,x-systemd.mount-timeout=30 to the 4 10.0.50.50: lines — the prior _netdev,nofail+After=remote-fs.target was silently defeated by nofail. Playbook playbooks/fix-esh-nfs-boot-ordering.yaml updated. Also: mOrpheus killed (irv-ml1 docker compose down; backend OOM-looping on the packed 3090) and a 1.36 GB HA coredump cleared.
  • Home Assistant config repo — vh/home-assistant-config (private gitea) for the HA Container at 10.0.50.46:8123 (= homeassistant on esh-docker-vm, macvlan, config named-volume). git-in-place in /config (root/infra-ops) + scoped deploy key; local clone ~/development/home-assistant-config. UI-managed HA -> allowlist model (YAML + curated secret-free .storage subset; secrets/DB/coredump gitignored).
  • soong-lab library persistence (corviduo-dev) — SOONG_LAB_LIBRARY_DIR=/home/infra-ops/soong-lab-data/library added to soong-lab.env (mirrors PORTRAIT_DIR), soong-lab-studio.service restarted. Unblocked the operator (Sindra import -> library save survives redeploys).
  • char-rp-reasoning OOM rescue + tune — the NVFP4+MTP seat (GPU0) crash-looped on a solo restart (GPU0 too tight). Fixed: PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True + REASONING_GPU_MEM_UTIL 0.39->0.38 + REASONING_MAX_MODEL_LEN 196608->153600 (150K). Now ~4.5 GB GPU0 headroom, healthy; brokkr's text battery passed.
  • Statusline overhaul (~/.claude/statusline-command.sh, NOT this repo) — git state (branch/dirty/unpushed), 🔔/🔕 althing-monitor-armed, project tag, absolute tokens, per-session cost (.cost.total_cost_usd), threshold-colored ctx% + rate %s (green<60 / yellow60-90 / red>90).

PARKED (operator-directed holds):

  • image-bench (qwen-image-bench, GPU1) eviction — PENDING post-bake. Operator directed a full arbo -> gen switch (comfy-dev executed it, LIVE in prod: DEFAULT_LLM_MODEL=gen + ARBO_VISION_MODEL=gen); image-judge now has ZERO arbo tenancy. Sequencing = short prod bake on gen (~few days) THEN evict -> reclaim ~30 GB GPU1. Keep vllm-qwen-image-bench WARM as the instant-revert backstop; comfy-dev watches NSFW-anatomy scoring + tight-grounding (flip ARBO_VISION_MODEL back on regression). On a clean bake: evict + ping comfy-dev. Fully tracked in auto-memory project_arbo_gen_switch_imagebench_evict.
  • granite right-size — un-started GPU1 win. granite (GPU1) is ~29 GB (util 0.34 -> ~16.5 GB oversized KV for an 8B summarizer). Right-sizing util 0.34 -> ~0.13 frees ~16 GB WHILE keeping the summarizer (fleet's dedicated summarizer/classifier; aliases granite-4.1-8b/summarizer/classifier). Independent of image-bench; operator was freeing GPU1 for a TBD "something else".

Open loops for the fresh session:

  • stacks/heretic2-charrp-reasoning/ is UNTRACKED in git (?? stacks/heretic2-charrp-reasoning/) — the char-rp-reasoning compose stack is deployed but never committed; commit it (canonical mirror of the live seat).
  • althing monitor — re-arm /althing:monitor in the fresh session (the wake-listener dies on /clear). Open watch: comfy-dev on the arbo bake (no news = clean).
  • Carried standing items (non-blocking): T1 SFT LoRA dormant (operator pivoted off-the-shelf); rotate the 5 rest-server backup creds (operator, offline); brokkr P00 handoff for the NVFP4 char-rp-reasoning seat (in live traffic ahead of it).

Recent decisions

  • [2026-07-15] arbo fully switched off image-judge (qwen-image-bench) -> gen; image-bench pending eviction post-bake. Operator-directed full switch (comfy-dev executed, live in prod). Established: gen (qwen3.6-35b-a3b-heretic) is vision-enabled and was image-bench's predecessor as arbo's hero-judge; image-judge actually serves 4 roles (vision quality-scoring + identity-scoring + bbox grounding + an uncensored text tier), not just grounding. comfy-dev spot-check: gen faster on every task, grounding within ~3px, uncensoring preserved, and it FIXED a bug (image-judge's reasoning preamble broke json_object + stalled the router). Sequencing = short prod bake then evict (~30 GB GPU1 reclaim); revert = flip ARBO_VISION_MODEL. Full record: auto-memory project_arbo_gen_switch_imagebench_evict.

  • [2026-07-15] esh-docker-vm NFS fstab fix = x-systemd.before=docker.service (the prior After=remote-fs.target drop-in was silently defeated by nofail). Reached only after a REBOOT (D-state phantom containers uptime-kuma + paperless-web that no docker/ctr/daemon-restart could clear). Committed 21d9a07 + playbook updated. See Tried and abandoned.

  • [2026-07-15] Homepage AI-tab revamp — flat "AI Systems" group -> dedicated AI tab, 6 role-based groups + AI-Dormant; committed 569e1af, pushed. (Also caught + pushed a ~100-commit unpushed eshpfi backlog.)

  • [2026-07-15] Home Assistant config repo created (vh/home-assistant-config, private). UI-managed HA -> allowlist model (YAML + curated secret-free .storage subset). git-in-place in /config on esh-docker-vm + scoped deploy key + local clone ~/development/home-assistant-config.

  • [2026-07-15] char-rp-reasoning OOM rescue — solo-restart on the packed GPU0 crash-looped; fixed via expandable_segments:True + util 0.39->0.38 + max-model-len 192K->150K. LESSON (Tried): max-model-len does NOT free vLLM VRAM (util-pinned KV pool). ~4.5 GB GPU0 headroom now.

  • [2026-07-15] soong-lab SOONG_LAB_LIBRARY_DIR made persistent (corviduo-dev) — was on the redeploy-wiped code default; set to /home/infra-ops/soong-lab-data/library (mirrors PORTRAIT_DIR), restarted. Closed a queued no-rush item; unblocked the operator.

  • [2026-07-15] Statusline overhauled (~/.claude/statusline-command.sh) — git state / 🔔🔕 monitor-armed / project tag / abs tokens / per-session cost (.cost.total_cost_usd) / threshold-colored ctx+rate (green<60 / yellow60-90 / red>90).

  • [2026-07-14] NVFP4+MTP fast char-rp-reasoning seat LANDED + LIVE + gateway-repointed + VRAM-tuned. Modelopt-format re-quant made MTP work. The load-crash root cause = vLLM 0.24 does NOT propagate modelopt exclude_modules to the spec-decode DRAFT model → the bf16 mtp head gets quantized → shape crash; NO checkpoint config fixes it (is_layer_skipped is exact-membership, and the drafter never sees the mtp excludes) → workaround = a mounted sitecustomize that force-skips mtp.* in is_layer_skipped (upstream vLLM bug to file). Productionized as compose stack heretic2-charrp-reasoning (:8018, workaround baked in). Gateway char-rp-reasoning alias fixed: repointed off the stale GGUF served-name deckard-pkd-27b, added enable_thinking:true, dropped min_p (MTP-incompatible), canonical samplers temp1.0/top_p0.95/top_k20. Rebalanced GPU0 (gen 0.37→0.30/16-seq/256K + reasoning 0.39/16-seq/192K+MTP + char-rp 128K, 2.7GB free). All 4 gateway roles verified; vLLM reasoning-parser confirmed leak-free (unlike the GGUF budget-forcing). Full record + the 4 quant landmines in docs/runbooks/heretic2-nvfp4-mtp-seat.md; committed 982c319. Open (non-blocking): brokkr P00 (seat is live ahead of it), retire the stopped GGUF reasoning seat, file the vLLM bug.

  • [2026-07-14] NVFP4 quant chase RESOLVED (gibberish) + PIVOTED to modelopt for MTP. One ~40-min GPU0 window. Root-caused the !!!! to the quant NAMESPACE (text-only AutoModelForCausalLM→model.layers.* keys; vLLM serves only Qwen3_5ForConditionalGeneration, which needs model.language_model.*) — found from config diffs + vLLM source with ZERO GPU time; fixed by loading as AutoModelForImageTextToText. NVFP4 now serves COHERENT (validated greedy). BUT base NVFP4 ≈53 tok/s ≈ GGUF's 59.5 at batch-1 (no single-stream win) AND MTP = 0% acceptance on compressed-tensors (bf16 mtp head only loads on the modelopt format). Operator chose to pursue a modelopt-format re-quant (the only path to the 2-4× MTP goal; AEON-proven on this exact Qwen3.6-27B arch). Scoped + de-risked: AEON /tank/aimodels/qwen36-27b-aeon-nvfp4 = the modelopt reference (quant_method modelopt, 1967 tensors, 15 bf16 mtp keys identical to graft); nvidia-modelopt 0.45.0 installs + mtq.quantize/NVFP4_DEFAULT_CFG/export_hf_checkpoint API confirmed; pipeline unchanged except swap llm-compressor→modelopt. Seats restored; char-rp-reasoning stays GGUF. Full plan in Current state ★ section.

  • [2026-07-14] Pursue the NVFP4+MTP fast char-rp-reasoning seat to completion (Vuong-directed via /snapshot: "chase the nvfp4 quant, we know it works, write down the recipe"). Full recipe + diagnostic ladder in Current state / in-flight above. Artifacts on ana-ml2 /tank/aimodels/heretic2-nvfp4-work/ + scripts committed in eshpfi services/heretic2-nvfp4-quant/.

  • [2026-07-14] char-rp-reasoning seat: Deckard-PKD → NEO-CODE = Heretic2-Thinking (Qwen3.6-27B) — R36 gate PASSED (tools 0.967, #355 runaway ELIMINATED). #355 was MODEL-level (Deckard emitted qwen3_coder XML malformed → mangled args → retry-runaway), NOT the reasoning-budget bug; NEO-CODE emits it clean. Custom llama.cpp KEPT (qwen3_coder parse — stock b8840 predates it — + PR#25544). Committed f960a73; full record auto-memory charrp-custom-llamacpp-pr25544.

  • [2026-07-14] soong-lab webhook auto-deploy real root cause = gitea webhook.ALLOWED_HOST_LIST (was external, 10.100.0.0/16 = NH3-only; blocked corviduo-dev's Anaheim 10.250.x → gitea refused to deliver, never opened the connection). Fixed to fleet-wide 10.0.0.0/8 (app.ini [webhook]) + gitea restart; listener now logs every delivery. The ufw 10/8 open (also this session) was a real-but-secondary gap. Committed 462d528.

  • [2026-07-13] #355-residual ROOT CAUSE (supersedes the "LiteLLM gateway holds while seat idles" entry below — that was DISPROVEN). char-rp-reasoning enters a non-terminating REASONING loop (tool-call-retry planning) and runs to max_tokens=32768 (~22 min @ 24.7 tok/s, ~13% of requests); the seat GENERATES all 32768 tokens (not idle), and --reasoning-budget 400 is NOT enforced. 3-source-confirmed (spend_logs completion_tokens=32768 ×4; seat eval-time log; pcap 100%-reasoning_content deltas). Server-side fix wanted (operator: no max_tokens ceiling) → routed to brokkr (accepted, pulled dvalin). Lesson (again): confirm before concluding — the seat-idle claim came from reading only the ≤73s requests + missing the concurrent 32768-token slots. See ACTIVE 1.

  • [2026-07-13] Deploy-speed real bottleneck ≠ uv sync (memory's assumption was wrong). Buildx step log: chown -R /app = 251s (copy-up of the root-owned venv into a fresh layer), uv sync only 35.6s, registry layer cache already wired. Fix = drop /app from the chown (validated safe: zero /app runtime writes on both live instances) + uv cache-mount. Shipped as PR #359 (branch off origin/main@b60), worldtree-dev green-lit. Expected ~5min off (~11→~6min). Runner-side BuildKit cache task (b) was already done → moot.

  • [2026-07-13] WT #355 residual 300s hang localized to OUR LiteLLM gateway (holds 2 char-rp-reasoning requests ~21 min while the seat idles), NOT the seat — Deckard seat EXONERATED (completes ≤72s; --reasoning-budget 400 forecloses a mid-thinking hang). Corrects worldtree-dev's "seat wedging" diagnosis. Decisive next = the FIN-check (pcap on corviuo). See in-flight ACTIVE 1. [SUPERSEDED 2026-07-13 — see the ROOT CAUSE entry above; the gateway-hold/slot-leak theory was disproven, the seat was generating 32768 tokens.]

  • [2026-07-13] WT #355 turn-lifecycle fix VALIDATED on worldtree b60 — wedged turns self-terminate cancelled/stalled at the 300s stall-watchdog (turns 2064/2065 vs pre-b60 2061's 16-min no-terminal). worldtree-dev filed follow-ons #356 (rehydrate Tier-3 ctx on resume — the recreate-durability gap), #357 (reclaim orphaned active-turn locks), #358 (LLM-provider read-timeout audit); surfacing to Vuong to prioritize.

  • [2026-07-13] Worldtree deploy bottleneck = the image build (~11 min of a ~12 min deploy), root cause the Dockerfile uv sync ... --no-cache + no BuildKit cache-mount (re-downloads all deps cold every build). Fix split: worldtree-dev Dockerfile cache-mount diff + infra-ops runner-side persistent BuildKit cache. Config-only changes skip the build entirely (pinned recreate).

  • [2026-07-13] Ledger tier-3 consumer ledger:miranda provisioned on personal :8081 (key b38932f5, GPG-delivered+shredded, allowlist 10.100.10.50:8770 live); assistant+thoughtful-assistant capability roles added (gen/gen-reasoning) on personal+demo, canonical d8bd497. Chosen instance = personal (the tier-3-consumer instance, ratatoskr+soong-lab colocated).

  • [2026-07-10] Heimdall grant: ratatoskr affect.full on PERSONAL Worldtree (operator-approved, worldtree-dev R34-v1 request). Added allow-rule ratatoskr-affect-full-allow to /opt/worldtree-personal/config/policies.yaml (principal.user_ids:["ratatoskr"], action affect.full, resource *, effect allow), mirroring the #347 session-history-write-ratatoskr rule exactly + placed right after it. WHY user_ids-based (not tier): ratatoskr's personal key is the minimal readonly-admin observability tier, which is NOT in the tier-based affect-render-baseline-allow (anonymous/user/free/pro/admin) → needs an explicit user_id grant, same as #347. R34-v1 (b46, committed UNPUSHED) gates affect.emit dominant_emotion egress by exposure ceiling (affect.full|safe → present; neither → null); this grant keeps ratatoskr's view alive across the b46 deploy. Surgical exact-string insert (preserves comments), backup policies.yaml.bak-pre-affectfull-20260710, validated via the CONTAINER's own yaml parser (35 rules, +1, payload confirmed). NOT restarted — deliberate: rule is on the bind-mount (shadows baked), INERT until b46 gating ships, so the b46 CI/CD deploy restart activates it (no live-session blip now). Demo untouched (personal-only per key scope). Replied to wt-dev (thread 01KX6DB3…) offering an immediate restart if they want it live for pre-b46 testing. See reference_corviduo_dev_emergency_ops.

  • [2026-07-10] ComfyUI v0.27.1 SUCCESS on irv-ml1 (operator-confirmed execute-now) — landed on torch 2.12.1, SageAttention preserved, crash-loop AVOIDED. The prior attempt (entry below) crash-looped because a torch 2.12.1→2.13 bump broke SageAttention's ABI. This time I checked git diff v0.24.1 v0.27.1 -- requirements.txt FIRST and found core v0.27.1 leaves torch UNPINNED → the version bump does NOT require torch 2.13 (that came only from the mmartial boot-upgrade). So: git checkout v0.27.1 (clean tree) → pip install -r requirements.txt as uid 1000 with a torch-pin constraint file (torch/vision/audio pinned to current +cu129) to block any transitive bump → torch stayed 2.12.1, SageAttention 2.2.0 untouched. Added decord 0.6.0 (fixed SAM3Segment). docker restart comfyui → healthy, /system_stats comfyui_version=0.27.1, "Using sage attention", HTTP 200, DB migrated 0003→0004. Reported the divergence to comfy-dev (thread 01KX6D3C…, reply pending) + asked whether LTXV 2.3 needs a separate torch-2.13 follow-up (their domain; Krea-2's ≥0.25 need is met by 0.27.1). LESSON: before a mmartial ComfyUI version bump, git diff <old> <new> -- requirements.txt — if torch is unpinned, bump the CODE without touching torch (constraint-pin it) and compiled exts (SageAttention) survive. docker exec lands as uid 1025(comfytoo), not 1000 — use -u 1000 + the venv python /comfy/mnt/venv/bin/python. See reference_irv_ml1_comfyui_mmartial.

  • [2026-07-10] ComfyUI 0.25.x bump on irv-ml1 ATTEMPTED → FAILED → ROLLED BACK (snapshot saved it). comfy-dev requested (Vuong-authorized) bumping the irv-ml1 comfyui stack (mmartial image, /opt/docker/compose/comfyui/, 0.24.1) to 0.25.x for Krea-2 + LTXV 2.3. TWO FINDINGS: (1) DISABLE_UPGRADES=false/USE_PIPUPGRADE bumps the VENV (torch 2.12.1→2.13.0 + deps) but does NOT advance the ComfyUI CODE checkout (/comfy/mnt/ComfyUI = /worktank/comfyui/run/ComfyUI stayed 0.24.1 — pinned/detached git, comfy-dev's domain). (2) the torch bump broke SageAttention (2.2.0 _fused.so undefined-symbol c10::impl::cow::materialize_cow_storage vs torch 2.13.0) → --use-sage-attention (REQUIRED launch flag in COMFY_CMDLINE_EXTRA) crash-looped ComfyUI. Net: broke the working state, zero 0.25.x payoff. ROLLBACK WORKED: pre-bump 16G venv snapshot /worktank/comfyui/venv-snapshot-comfyui-0.24.1-20260710.tar restored (torch 2.12.1 + working SageAttention), re-pinned DISABLE_UPGRADES=true, recreated → healthy on 0.24.1, serving :8188. Broken venv parked at /worktank/comfyui/run/venv.broken-torch213-20260710. CORRECTED PATH (sent comfy-dev, thread 01KX655V): comfy-dev git-advances the ComfyUI checkout to 0.25.x + reqs → I handle the torch bump + SageAttention rebuild-against-2.13.0 + re-pin (snapshot stays as the net). LESSON: mmartial DISABLE_UPGRADES gates ONLY the venv pip-upgrades, NOT the ComfyUI git checkout; a torch bump breaks compiled exts (SageAttention) → rebuild-after is mandatory. Bump BLOCKED pending comfy-dev's git-advance. Stack: A6000 (NVIDIA_VISIBLE_DEVICES=1), lkraven-owned compose+venv (uid 1000, no sudo needed), COMFY_CMDLINE_EXTRA OOM flags preserved.

  • [2026-07-10] Biweekly open-weight-releases scan cron set up for brokkr-smithy (Vuong-authorized). Durable systemd --user timer on nh3-dev (brokkr-landscape-scan.timer, OnCalendar *-*-01,15 09:00:00 America/Los_Angeles, Persistent=true; linger on) → .service → wrapper ~/.local/bin/brokkr-landscape-scan.sh runs headless claude -p "$(cat ~/.config/brokkr-landscape-scan/prompt.txt)" --dangerously-skip-permissions in ~/development/brokkr-smithy (ALTHING_HANDLE=brokkr-smithy-dev; explicit PATH — the --user minimal-PATH footgun; per-run logs ~/.local/state/brokkr-landscape-scan/). Prompt = brokkr's payload verbatim (LLM/image/TTS new-release sweep → ranked synthesis → commit+push+notify). VALIDATED: git-push non-interactive (BatchMode ls-remote to gitea, passphraseless key — no agent), headless claude auth (READY smoke). VALIDATED END-TO-END 2026-07-10 (manual run, exit 0): web-sweep→synthesis→commit 2ed2f29→PUSH of scan #2 (open-weight-releases-2026-07-24.md); triaged dwarf input + caught baseline errors, quality strong. HANDLE-COLLISION caught+FIXED — the headless scan shared handle brokkr-smithy-dev with the LIVE session + raced its inbox (eitri's dwarf-reply got stolen by the live monitor); registered a dedicated brokkr-scan-dev handle (add-handle, driver=none) + repointed the wrapper + rewired step-5 notify → althing-cli post --to brokkr-smithy-dev (NO vuong althing handle exists — confirmed). model=default

    • --max-turns 80. First run under the new handle = 7/15. Off-cycle 07-24 doc is a validation artifact (scheduled 1st/15th runs date to their own run-date, no collision) — operator naming-convention call pending. NEXT AUTO-RUN 2026-07-15 09:00 PDT. Manual validation/first run = systemctl --user start brokkr-landscape-scan.service. Open w/ brokkr (thread 01KX63G6): confirm notify-Vuong handle/mechanism + session handle + model/turn-cap. NEXT brokkr task (operator-sequenced after this): TTS audition env — Higgs-TTS-3 + ZONOS2 + Chatterbox baseline, TTFA/RTF + blind-A/B web-listen (thread 01KX6371; needs GPU-placement + HF-token feasibility pass first; brokkr delivers the prompt set after the env's up; protocol doc in brokkr-smithy repo).

  • [2026-07-09] Two parked items closed: phantom qwen3.6-35b-a3b alias VERIFIED already-gone; ana-docker docker log-cap SOLVED no-bounce. (1) Phantom: absent from /v1/models + /model/info (config+DB registry), zero litellm log refs — the parked "400s in /v1/models" note was STALE (already cleaned in the 2026-07-08 gen repoint to -heretic); bare token survives only in 2 config COMMENTS (lines 76/80). Nothing to remove. (2) Log-cap: running containers were UNCAPPED (182M json-logs, top offender 59M) because daemon.json's max-size 10m/max-file 3 only applies to containers CREATED AFTER a daemon restart — it never reaches already-running ones. No-bounce fix = /etc/logrotate.d/docker-containers (copytruncate — dockerd opens json-logs O_APPEND so truncate-in-place resets cleanly, no sparse-file corruption; size 10M, rotate 3, compress, su root root), auto-picked-up by the daily logrotate.timer. Force-ran + gzipped the frozen .1 archives → 182M → ~55M (44M active + 11M gz), every container kept its multi-week uptime (zero bounce, verified). LATENT FOOTGUN FLAGGED (not yet fixed, operator's call): daemon.json declares live-restore:true but the RUNNING daemon has it FALSE (daemon.json was edited after the last daemon start, never reloaded) → the NEXT systemctl restart docker / crash / pkg-upgrade bounces ALL ana-docker containers once. Fix WITHOUT a bounce = systemctl reload docker (SIGHUP loads live-restore into the running daemon; log-opts are NOT SIGHUP-reloadable, which is why logrotate — not the daemon cap — is the enforcer for running containers).

  • [2026-07-09] granite→gen memory_extractor bind host-synced on demo+personal Worldtree (Vuong-directed, #335 Slice-4). Changed model_roles.yaml memory_extractor binds.catalog_id summarizer→gen (overrides intact: thinking:false/temp0/8192) on BOTH /opt/worldtree{,-personal}/config; memory_distiller left on summarizer (range-scoped sed /memory_extractor:/,/memory_distiller:/ — the naive global replace would've hit both); backups *.bak-preqgen-20260709; validated via each container's OWN yaml parser. DEMO LIVE on gen (b43 d501e516732d auto-deployed mid-edit + its restart RACED my edit by ~2min → I restarted worldtree-worldtree-api-1 to activate; healthy, live process resolves memory_extractor=gen). PERSONAL NOW LIVE on gen too — Vuong authorized the restart (via wt-dev); restarted worldtree-personal-worldtree-api-1 after a PRE-FLIGHT that ran the app's OWN load_model_roles() (core/llm/roles.py:121) against the synced config INSIDE the running c9986cd container: gen in catalog, all 9 roles resolve, no DanglingBindingError → proven safe on the OLDER image BEFORE touching it (model_roles-delta-alone clean; no full-config-set sync needed). StartedAt 20:50:55Z, healthy, resolves gen. BOTH instances live on gen. LESSON: the bind-mount /opt/worldtree*/config SHADOWS the baked /app/config-defaults/ → the deploy alone never updates the ACTIVE config; the host edit is required AND a restart activates it (role registry cached at boot) — pre-stage BEFORE the deploy's restart or you race it. FOLLOW-UPS (non-blocking): (a) memory.extractor.user_pass parity block → self-serve from the b43 baked defaults.yaml (pydantic-default no-op); (b) stale gen provider description (Mistral-Small-4 → qwen3.6-35b-a3b-heretic) → wt-dev owns the REPO-side providers.yaml fix (operator's call — NOT purely cosmetic: the gen/dialogue + classifier entries carry Mistral-Small-4 SAMPLING defaults that drive mask/lofn/forseti/mimir dialogue, so wt-dev re-validates for qwen), host cosmetic sync pairs when it deploys. Gave wt-dev the VERIFIED canonical 4-alias set (backends+samplers read from the LIVE gateway config, not the doc); corrected docs/pfi/model-sampler-defaults.md seat 4 (had lagged QwQ-RpR-v4 → Deckard-PKD; live gateway was always Deckard). Operator SCOPED IN the character-RP re-point (2026-07-09): character→char-rp / thoughtful-character→char-rp-reasoning (character-rp per wt-dev's role semantics), moving character RP off the GENERAL qwen onto the dedicated Magidonia/Deckard seats. Relayed to wt-dev w/ the mapping principle + a SAMPLER-OVERRIDE warning (DROP character-rp's old temp0.75/top_p0.85 overrides — carried onto the dedicated seats they'd clobber the canonical RP tuning DOWNWARD) + ratatoskr-reach note (role call is transparent but Magidonia/Deckard quality/latency differs from gen). DONE 2026-07-09: wt-dev committed 5d4fa4a (v1.0.0b44, UNPUSHED — operator drives push); operator directed host-ahead-of-push, so I sourced BOTH config files directly from that unpushed commit (local ~/development/Worldtree checkout — capital W; git show 5d4fa4a:config/…) + mirrored VERBATIM to /opt/worldtree{,-personal}/config on demo+personal, paired-pre-flighted via the app's load_model_roles() (no DanglingBinding), restarted both → LIVE: character→char-rp, thoughtful-character + character-rp→char-rp-reasoning, memory_extractor→gen preserved. Backups *.bak-prerp-20260709. context_window VERIFIED (llama.cpp /props + char-rp-gguf .env): char-rp 131072, char-rp-reasoning 262144 (gave wt-dev to patch the repo from its interim 32768). HOST AHEAD of repo-remote until the operator pushes 5d4fa4a (baked config converges with the bind-mount on push+deploy). GOTCHA: demo≠personal — PERSONAL was already partly re-pointed (2026-07-06 AEON-era character→char-rp) so its delta was mostly stale-AEON-descriptions→Magidonia/Deckard

    • character-rp + stripping personal's char-rp default_params temp0.7/top_p0.8 that CLOBBERED the gateway RP tuning downward; DEMO had no char-rp catalog entries at all (b44 adds them). Diffed each instance vs b44 before applying (both deltas = expected changeset only, nothing instance-specific clobbered). wt-dev PATCHED the context_window in b45/3384a37 (char-rp 131072, char-rp-reasoning 262144). BUMPED HOST-AHEAD on both instances (operator-directed 2026-07-09): mirrored b45's providers.yaml → /opt/worldtree{,-personal}/config, restarted, verified LIVE (char-rp ctx 131072, char-rp-reasoning 262144, bindings intact); backups .bak-b44interim-20260709. HOST now = b45 canonical (providers.yaml) + b44 (model_roles unchanged b44→b45). STILL PENDING: (1) operator's batched push of b44+b45 (5d4fa4a+3384a37) to converge the repo-remote — host is ahead, no fork; (2) user_pass parity block (defaults.yaml — NOT in either, separate). Threads 01KX3SGH/01KX48QP (worldtree-dev), monitor armed. See reference_corviduo_dev_emergency_ops.

  • [2026-07-09] mOrpheus TTS off-the-shelf voice pipeline SHIPPED end-to-end (irv-ml1) + wired into gateway-chat. Full arc (commits): gen served-name honesty rename aeon→qwen3.6-35b-a3b-heretic (99a4a17, vLLM served-name + litellm refs, so /v1/models + spend-logs name the real model); permanent 2-container stack (01eedd8); gateway-chat auto-voice quoted dialogue (c948013); streaming decode TTFA 4.5s→0.8s (da76829); max_tokens 1200→2400→3500 with a context-clamp (f363fe6, 0655a37 — long lines were clipping at 14.6s, and repetition_penalty 1.1 is LOAD-BEARING: at 1.0 the model never stops); AudioContext resume-on-gesture no-sound fix (033f368); pre-chunk by QUOTED SECTION not sentence for prosody (a1f3023→f295cc1); staged clone voices baddy/beatrice/whisper (0655a37 + runtime .wav/.txt in the voices dir); agent voicing prompt (a573514). Load-bearing config, all encoded in stacks/mOrpheus/: bf16 not FP8, image v0.23.0 not latest, GPU=3090 not A6000, rep_penalty 1.1. Serving-viability confirmed: vLLM concurrency near-linear to 8× (707 tok/s).

  • [2026-07-09] granite→gen memory_extractor bind GREEN-lit for worldtree-dev (Worldtree #335 Slice 4). Answered their VRAM/concurrency headroom check: gen (qwen 35B-A3B heretic) has ample headroom for ~2 bursty idle-triggered extractor calls (fixed 0.37 util; KV pool runs 0-2%; --max-num-seqs 16, near-linear batching). Corrected their stale "gen = Mistral Small 4 / 119B-6B" belief (gen IS the qwen 35B-A3B heretic since 2026-07-08). This bind is INDEPENDENT of the full granite RETIRE (reclaim ~32GB on ana-ml2 GPU1) — that stays the operator's call, pending brokkr R33 portfolio + production-concurrency due-diligence. Tracked: althing thread 01KX3SGH… (worldtree-dev) + brokkr's gen-absorbs-granite consult (thread 01KX2V32…) + reference_litellm_gateway.

  • [2026-07-08] RP-SEAT CAMPAIGN CLOSED — char-rp = Magidonia-24B-v4.3 (128K), char-rp-reasoning = Deckard-PKD Qwen3.5-27B (256K); both GGUF/llama.cpp on ana-ml2 GPU0 alongside gen (35B-A3B, util 0.37), ~4G GPU0 margin. Arc: (1) replaced broken Angel NVFP4 with Magidonia prose + QwQ-RpR-v4 reasoning (b268f93); (2) max-context via q8_0 KV (f570604); (3) canonical samplers for all 4 gateway seats, dvalin-derived + char-rp A/B-tuned (aac4bcf); (4) rebalanced gen 0.40→0.37 to fund char-rp 128K (f49c4e4); (5) RE-A/B'd the reasoning seat (operator wanted a DRY-tolerant model): Deckard WON on brokkr's frozen scorer (composite 2.176, 0/30 loops, 0/30 refusals) over RpR-v4 (3.716, 1/30 loop), Pantheon-Reasoning (1.383 but 7/30 refusals), Snowdrop+Gembrain (llama.cpp template-incompat) — deployed (5f79b40); (6) Deckard→256K (41305bf); (7) dvalin CONFIRMED Deckard samplers = the live A/B set is canonical (4954ca0). GATE LESSON: a llama.cpp reasoning seat needs a STOCK template that natively opens <think>/enable_thinking (Qwen3.x/QwQ pass; ChatML + Gemma-4 fail) — no monkeypatching. INFRA: llama-swap b8840 can't load Qwen3.6/Gemma-4 archs → ghcr.io/ggml-org/llama.cpp:server-cuda (pulled on ana-ml2).

  • [2026-07-08] worldtree Mimir deploy-blocker resolved (mid-session): synced persona.envelopes.assistant + envelope_grants:[] verbatim from the baked canonical into BOTH corviduo-dev instances (demo+personal), YAML-validated via each container's own parser; worldtree-dev cleared to push the Mimir-bound image. (Was my parked R32 1C envelope-mirror come due — see reference_corviduo_dev_emergency_ops config-sync recipe.)

  • [2026-07-08] OFF-THE-SHELF INFERENCE PIVOT executed — serve curated abliterated models, stop home-training. Final topology: gen = llmfan46/Qwen3.6-35B-A3B-uncensored-heretic-NVFP4-Experts-Only (LIVE, modelopt, vision, util 0.40), char-rp = an RP unicorn to be found on fresh context (see Current state). Intermediate steps ABANDONED: Pantheon-Reasoning-27B (served briefly as gen — refuses dark fiction via DeepSeek-distilled refusal-reasoning, see Tried); Pantheon-27B-with-MTP for RP (bf16 MTP won't load on the compressed-tensors path); Angel MS3.2-24B (my NVFP4 quant = garbage). Prefer EXISTING community NVFP4/GGUF quants over self-quanting ("don't quant unless you have to" — operator). GGUF serving is now on the table for RP (NEVER Ollama). Gateway sampling-defaults wiring still PENDING.

  • [2026-07-08] DPO was silently running 3 epochs (harness gap) → KILLED at epoch 1.2, retargeted to 0.3 epochs (operator call). Root cause: DpoConfig had NO epochs field + _dpo_config_kwargs didn't pass num_train_epochs → DPO fell through to trl DPOConfig's default 3.0 (SFT correctly pins 1 via SftConfig.epochs

    • _sft_config_kwargs). Objective SATURATED by epoch 0.27 (loss0, grad0, acc 1.0, margins27 flat — the off-policy qwopus rejected pairs are trivially separable), so epochs ~0.3→3 were pure over-optimization + a ~6.5h outage. No mid-run checkpoint (save_steps 500 > total steps; save only at end) → killing lost the run. FIX (3 edits to deployed harness, mtf-dev to canonicalize): DpoConfig.epochs: float = 1 (mirrors SftConfig, float for fractions); _dpo_config_kwargs now passes "num_train_epochs": cfg.epochs; recipe dpo.epochs: 0.3. GPU-free verified (dpo.epochs=0.3 → num_train_epochs=0.3). Relaunched at 0.3 epoch (~30min precompute + ~12min train = ~45min). DONE + SANITY-CHECKED (exit 0, ~70min wall: ~30min fixed precompute + 45 steps @ ~51s/step; train_loss 1.4e-5 @ epoch 0.301). Fresh data/spike/dpo_adapter/adapter_model.safetensors (123MB) + checkpoint-45 banked. 3-way greedy sanity (base vs SFT vs DPO, via peft load + disable_adapter/set_adapter on GPU0): ALL THREE DISTINCT (base≠sft≠dpo) → full SFT→DPO pipeline applies end-to-end at inference. DPO 0.3ep is COHERENT, fluent, NOT degenerate (early-stop avoided over-optimization) but the quality delta on a neutral literary prompt is SUBTLE (DPO shares SFT's structure — it continues from it — with minor stylistic drift, arguably slightly MORE genre-clichéd). Verdict: mechanics proven, quality gain modest as predicted for 0.3ep on off-policy pairs; the real unlock remains on-policy rejected regen + on-domain (explicit E-RP) eval + the LitBench/holdout run. gen+rp RESTORED healthy. Next: serve fork (SGLang-finish vs merge) on the DPO adapter — same rekey_lora_for_vllm.py (zero-z) applies unchanged (mtf-dev confirmed).

  • [2026-07-08] T1 DPO leg is RUNNING (unblocked) — 2 fixes applied to deployed backend.py. Blocker resolved: (1) mtf-dev's v0.0.42 stub _stub_missing_optional_integrations (last-resort sys.meta_path finder → missing mergekit/llm_blender/weave resolve to MagicMock, never called → zero numerics risk; applied VERBATIM to deployed src/model_training_forge/train/backend.py after _unsloth_available() + call-site before from trl import DPOTrainer); (2) my cosmetic warnings_issued shim (trl-0.24 DPOTrainer.init:405 does model.warnings_issued["estimate_tokens"]=True for warning-suppression; custom Qwen3_5 class under transformers 5.5.0 lacks the attr → if not hasattr(model,"warnings_issued"): model.warnings_issued={} before the DPOTrainer(...).train() at backend.py:305 — cosmetic, zero training impact). Both edits are on the DEPLOYED un-git'd copy only → mtf-dev must canonicalize the warnings_issued shim into their repo (told them). DPO confirmed training: model loaded (851 shards), full 1196 pairs processed, in precompute_ref_log_probs (GPU0 93% util, 54.8GB). Completion watcher armed (bg task) → restore gen+rp + verify dpo_adapter + ping mtf-dev on exit. gen+rp STOPPED for the run (authorized window). Output → data/spike/dpo_adapter.

  • [2026-07-08] T1 DPO leg launch — prior BLOCK (now resolved above), kept for the launch recipe. Operator authorized the full DPO stage (via mtf-dev) + went AFK 2h. PROVEN LAUNCH RECIPE (replicates the SFT container aeon-t1-sft exactly, only --stage sft→dpo): sudo docker run -d --name aeon-t1-dpo --entrypoint python3 --gpus all -e CUDA_VISIBLE_DEVICES=0 -e MTF_FORCE_TRL=1 -e PYTHONPATH=/mtf/src -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True -v /home/lkraven/model-training-forge:/mtf -v /tank:/tank -w /mtf aeon-trainer:latest -u scripts/train.py --recipe recipes/training/qwen-3.5-122b-erp-lora/train.aeon-27b.yaml --stage dpo. CRITICAL: --entrypoint python3 is REQUIRED — aeon-trainer's default entrypoint is ["vllm","serve"] (FROM vllm/vllm-openai) → without the override it runs vllm + hits a torch-ABI crash. Dataset verified (pairs_dataset=train.flat.json=1196 pairs). THE BLOCK: from trl import DPOTrainer (backend.py:256) eagerly pulls TRL 0.24.0's WHOLE optional-integration set — cascade: mergekit(missing)→ immutables→mergekit-0.1.4↔pydantic-2.13 HARD incompat (needs pydantic==2.10.6)→llm_blender→dataclasses_json→ llm_blender-0.0.2↔transformers-5.5.0 HARD incompat (TRANSFORMERS_CACHE removed, needs source patch)→weave→ (more). NONE used by our pair-based DPO. pip install mergekit w/deps is UNSAFE (downgrades accelerate 1.14→1.6). Safe partial recipe derived (core libs held: torch2.10/tf5.5.0/trl0.24.0/peft0.19.1/accel1.14.0) but non-convergent → TRULY BLOCKING per operator's carve-out. Did NOT force-hack the proven training image. Handed full diagnosis + recommended fix (lazy-import TRL patch, opt b) to mtf-dev (thread 01KWZG8GJX, expects-reply, monitor armed). gen+rp RESTORED healthy. Relaunch = 1 min once mtf-dev delivers a working image.

142 older entries archived to archival-memory.md.

Tried and abandoned

  • [2026-07-15] docker.service After=remote-fs.target does NOT wait for nofail NFS mounts — nofail drops a mount out of remote-fs.target's blocking set, so the drop-in ordering is silently defeated (paperless still Exited(255) on reboot). Real fix = DIRECT mount->docker ordering via the fstab x-systemd.before=docker.service option (verify systemctl show docker -p After lists the mnt-*.mount units). esh-docker-vm.

  • [2026-07-15] The esh-docker-vm D-state/phantom-container wedge is only cleared by a host REBOOT — reconfirmed: docker stop/rm -f, ctr -n moby task delete, AND systemctl restart docker all fail to clear it; docker exec into a wedged container ALSO fails (setns ... exit status 1), so the in-place restart escape hatch is out. Worse, a daemon restart can HALF-KILL other healthy containers (knocked paperless's granian down + left it wedged). Process dead but dockerd won't reap -> phantom. NFS mounts are _netdev,nofail so the reboot is boot-safe.

  • [2026-07-15] vLLM max-model-len does NOT free GPU VRAM — the KV cache POOL is sized by gpu-memory-utilization, not max-model-len. Lowering max-model-len only caps per-request context + drops max concurrency; the pool still fills the util budget. To actually free VRAM, lower gpu-memory-utilization. (Bit the char-rp-reasoning "drop KV to 150K" ask: the 150K applied but freed 0 VRAM until util dropped 0.39->0.38.)

  • [2026-07-15] Claude Code statusline .cost.total_cost_usd is per-SESSION (Claude Code's own cache/model-aware session accounting), not a lifetime aggregate — the large value just reflects a long, multiple-times-summarized session. And the old statusline hardcoded Sonnet pricing ($3/$15) on an Opus session -> ~5x cost understatement.

  • [2026-07-14] MTP-on-modelopt: NO checkpoint config skips the spec-decode drafter's quant (vLLM 0.24 bug) — 4 config attempts failed before the runtime workaround. All crashed the same way (qwen3_5_mtp.py:256 param_data.shape == loaded_weight.shape AssertionError — bf16 mtp head loaded into a quantized drafter param): (1) mtp excludes in config.json (WRONG file — vLLM modelopt reads hf_quant_config.json); (2) specific-unfused mtp names in hf_quant_config; (3) wildcards mtp*/mtp.layers.0* (is_layer_skipped is EXACT-membership, NOT glob — wildcards match nothing); (4) exact fused+unfused names in both mtp./model. prefixes. Instrumenting is_layer_skipped proved the drafter's exclude list holds ONLY the main model's linear_attn entries — the mtp excludes never reach the draft-model quant config. ONLY fix = a mounted sitecustomize force-skipping mtp.*. LESSON: don't chase checkpoint-config fixes for the mtp-drafter crash; go straight to the runtime patch. Also nvidia-modelopt[hf]==0.43 (AEON's producer version) is a trap — it pins transformers back to 4.57 which can't load qwen3_5 at all; use 0.45 + the FusedMoE guard in quant_modelopt.py.

  • [2026-07-14] AEON's "working NVFP4+MTP RP seat" was pantheon on compressed-tensors (0% MTP accept), not a modelopt MTP proof. vllm-aeon-rp's .env → AEON_RP_MODEL=pantheon-27b-mtp-nvfp4, AEON_RP_QUANT=compressed-tensors — it LOADED (mtp silently skipped, exited 0) but never accelerated. Same vLLM image (:latest = sha256:4091d55 = 0.24.0) as the failed Heretic2 test, so the "AEON ran on an older vLLM" theory was wrong. Don't treat a seat that "ran" as MTP-validated without checking its SpecDecoding acceptance.

  • [2026-07-14] NVFP4 (llm-compressor / compressed-tensors) gives NO batch-1 speedup over GGUF for the Qwen3.5 GDN-hybrid, and its MTP is 0%-accept. Measured base NVFP4 no-MTP ≈53 tok/s decode vs the GGUF NEO-CODE seat ~59.5 (llama.cpp wins single-stream; NVFP4's edge is concurrency, and this hybrid is bandwidth-bound at batch-1 with the BF16 linear_attn/GDN layers dominating). MTP spec-decode = 0% acceptance (vLLM's Qwen3_5MTP drafter won't load the bf16 mtp weights off a compressed-tensors main model → Parameter … not found in params_dict, Avg Draft acceptance rate: 0.0%). Pantheon is identical — its "working NVFP4+MTP" was working structure, never real acceleration. Working native MTP needs the modelopt main-model format (AEON, ~3.3/3 accept). LESSON: don't expect a faster single-stream seat from an llm-compressor NVFP4 quant of this arch; the MTP multiplier is the whole point and it requires modelopt.

  • [2026-07-14] NVFP4 spike: built the full MTP serve scaffolding BEFORE validating a plain NVFP4 serve was coherent. Chased 6 sequential serve-config fixes (entrypoint doubled serve, arch ForCausalLM→ConditionalGeneration, --language-model-only, mamba-cache/max-num-seqs) across a 2.5hr GPU window (quoted 30-60 min) — only to find the served model gibbers (!!!!). LESSON: smoke a PLAIN /v1/completions coherence check on the SIMPLEST config (native arch, no MTP, no splice) FIRST — validate the tracer bullet before building spec-decode scaffolding. Also cost an unnecessary re-quant (the re:mtp.* ignore fix that turned out moot). Diagnostic ladder in Current state.

  • [2026-07-14] MTP graft via top-level mtp.* tensor names does NOT survive AutoModelForCausalLM.from_pretrained — the Qwen3_5ForCausalLM class doesn't expose an mtp module, so the mtp keys are DROPPED at load (quant output = 0 mtp). Fix = SPLICE the BF16 mtp tensors into the quant output post-hoc (how pantheon was built); don't rely on the graft surviving the model round-trip.

  • [2026-07-14] gitea "test-delivery 204" is NOT proof a webhook works (204 = gitea queuing, not the listener receiving) — and a proxy test signing with the listener's OWN secret proves the listener, not gitea's real delivery. Both red herrings cost a round of the soong-lab webhook diagnosis. Diagnose from BOTH ends: sender (docker logs gitea | grep webhook → the deny '<ip>' line) AND an instrumented receiver.

  • [2026-07-13] Relaying a peer's diagnosis as fact without confirming it against raw data. worldtree-dev diagnosed the WT #355 residual as "our llama.cpp seat wedging," which I echoed in a wrap-up; the operator challenged it and the seat logs DISPROVED it (seat completes ≤72s, idle at the wedge onset — the hang is the LiteLLM gateway). Lesson: CONFIRM peer diagnoses (esp. cross-domain ones) before acting/relaying — same discipline that caught the earlier char-rp-reasoning red-herring via a live registry.resolve reproduction.

  • [2026-07-13] althing-cli reply <THREAD_id> (thread id, not a MESSAGE id) → "unknown message_id"; and reply to your OWN message self-addresses to your handle ("replying to your own message"). Reply to a PEER's message id, or use post --to <peer>. Bit me several times this session.

  • [2026-07-09] FP8 breaks mOrpheus audio-token generation. --quantization fp8 on the 3B → 0 valid SNAC frames even at GREEDY (degenerate audio+text mix, no start-of-speech); bf16 is clean (28/28 frames). Quant-breaks- TTS, same class as the Angel-NVFP4 lesson below. bf16 is REQUIRED (so the operator's "util 0.1" wish is moot — the bf16 weights alone are 6.6GB). NB the raw-token benchmark RTF 0.50 was fp8+graphs = never real.

  • [2026-07-09] vllm/vllm-openai:latest crashes on Ampere IMPORT — Blackwell-only kernels (oink/aiter, has_device_capability(100)) die during import on the 3090/A6000. Pin v0.23.0 on irv-ml1's Ampere GPUs. (vllm/vllm-omni:v0.18.0 has a different entrypoint — don't use it either.)

  • [2026-07-09] Per-frame CPU SNAC decode is too slow for streaming — per-call overhead × ~60 frames serialized → RTF 2.2 (WORSE than whole-clip's 1.0). Fix = windowed chunk decode (every 6 frames decode a [2 ctx | 6 | 2 ctx] window, emit the middle 6 → seamless, O(1)/frame, RTF ~0.97, TTFA ~0.8s).

  • [2026-07-09] Sentence-chunking TTS loses prosody — generating each sentence cold flattens the intonation that spans a line. Chunk by QUOTED SECTION (whole quote = one gen call). Also: repetition_penalty >1.1 BREAKS cloning (penalizes the ~1100 in-context reference audio tokens; keep ≤1.1 on the clone path).

  • [2026-07-09] HF whisper datasets aren't actually whispered. Claris-Whispered-English measures voiced 0.8 (not a whisper) + IPA transcripts; datasets audio decode needs torchcodec (wants CUDA-13, incompatible w/ the cu124 venv). LPC DSP-whisperize went unstable (NaN). kokoro af_nicole IS a genuine whisper (voiced 0.24) — that (operator's pointer) is the working whisper reference source, not TTS-voice screening or dataset-hunting.

  • [2026-07-08] Angel (allura-org/MS3.2-24b-Angel) self-quanted to NVFP4 = GARBAGE. llm-compressor W4A4 NVFP4 (compressed-tensors, MLP-quantized, attn/vision bf16) of the Mistral3 dense 24B produces gibberish EVEN AT GREEDY (temp 0) → the quant itself is broken, not the tokenizer or sampler. Same recipe worked on the qwen models. Mistral3 + W4A4 NVFP4 via llm-compressor is bad. → for the RP seat, going GGUF (llama.cpp) to sidestep the whole NVFP4-quant surface.

  • [2026-07-08] Mistral3 + vLLM tokenizer/vision traps (serve MS3.2-24b, vLLM 0.24). (a) HF tokenizer.json for Mistral = GARBAGE output — the card's "use the official Mistral tokenizer" warning is REAL; must use the tekken.json/mistral tokenizer. (b) BUT --tokenizer-mode mistral + vision CRASHES (Failed to apply PixtralProcessor on {'text': '[IMG]'}; and with tekken.json present in auto mode, CachedMistralCommonBackend has no attribute is_fast). So it's mistral-tokenizer OR vision, not both on this vLLM. Text-only + mistral tokenizer serves clean (--limit-mm-per-prompt '{"image": 0}'). GGUF/llama.cpp avoids all of this (native mistral tokenizer + vision).

  • [2026-07-08] Pantheon-Reasoning-27B refuses dark fiction DESPITE an abliterated base. The base (llmfan46 heretic) writes freely (thinking-off), but Gryphe distilled the reasoning traces from DeepSeek 3.2 (safety-aligned) onto every turn (preserve_thinking:true) → the model reasons ITSELF into refusals in the <think> phase (collapses to empty output). Fix: thinking-off OR an uncensor system prompt (both verified). Lesson: a reasoning finetune of an abliterated base can re-censor via its reasoning-trace TEACHER; the raw abliterated base is cleaner — this is WHY the pivot went to the llmfan46 heretic base for gen.

  • [2026-07-08] Pantheon-27B MTP on vLLM compressed-tensors = 0% acceptance. MTP is a separate bf16 head (mtp.*, in model-auxiliary.safetensors, 15 tensors); AEON preserved it by INJECTING the bf16 head into the quant output (NOT re-quantizing — confirmed AEON's nvfp4 mtp is bf16). Built pantheon-27b-mtp = compressed-tensors main + injected bf16 mtp + text_config.mtp_num_hidden_layers=1 → vLLM detected the MTP but SKIPPED the bf16 self_attn weights → 0/192 draft tokens accepted. The bf16 MTP head only loads on the MODELOPT main-model format (like AEON), not compressed-tensors. (Moot — operator dropped MTP for gen; not needed for the non-reasoning RP.)

  • [2026-07-07] vLLM 0.24.0 qwen3_5 LoRA application = silent no-op (#47639). Adapter loads HTTP 200 but zero deltas at inference. NOT quant (NVFP4 AND FP8 both inert). NOT adapter format (separate zc adapter — correct per vLLM's check_unexpected_modules allowlist — loads clean but inert; the fused-key rekey is rejected). The #47640 None-group guard-patch overlay did NOT fix it (failure is UPSTREAM of expand_packed_lora — the separate→fused mapping never happens). Fix PR #47640 is OPEN (unmerged) so no version-bump helps. Merge bakes deltas in (bypasses this) but is static.

  • [2026-07-07] SGLang generic image can't LOAD our NVFP4 AEON — ModelOptModelLoader weight-shape/ packing mismatch ([1024,5120] vs [1024,2560], 2-fp4/byte). NVFP4-on-SGLang needs the dedicated qwen36-27b-nvfp4 dev image or a requant to SGLang's format. bf16 loads fine (arch supported; crash was quant-loader-specific).

  • [2026-07-07] SGLang --lora-target-modules CLI enum REJECTS the GDN names its own resolver asks for (invalid choice: 'in_proj_qkv'); 'all' resolves to the FUSED set (qkv_proj/in_proj_qkvz). SGLang wants its OWN packed layout (base r16 + get_stacked_multiply=3, NOT a pre-fused rank-48 qkv → the [48]-vs-[144] shape assert). A THIRD adapter format; version-exact source needed (:latest=0.5.13, NOT main).

  • [2026-07-07] Engine invocation footguns cost several wasted serve-bounces this session — docker run --rm ate crash logs; duplicated serve (vLLM image entrypoint is already ["vllm","serve"]); --max-lora-rank 48 invalid (choices 1/8/16/32/64… → use 64); parens in echo inside ssh host -c "…" break the remote shell. LESSON: verify engine launch flags (--help, GPU-free) + never --rm a container whose crash logs you need, BEFORE bouncing a production serve.

  • [2026-07-04] LiteLLM (this gateway version) mutates the SHARED deployment config in-place on per-request sampler-param merge → my deliberately-invalid top_k=-5 forwarding-probe bled into a param-less character-rp request (vLLM 400, ONE-OFF, self-cleared by a later valid probe). NOT caching (none configured), NOT a config change. Never fire invalid/distinctive sampler values at a SHARED gateway alias with live consumers — use a throwaway alias, or a docker restart litellm flushes residual carryover. feedback_litellm_shared_param_mutation.

  • [2026-07-04] A systemd --user daemon that shells out to ~/.cargo/bin/~/.local/bin tools needs an explicit Environment=PATH — the minimal --user default silently drops them. The althing herald lost zellij → silent pane-miss for ALL config-backed TUI/pane agents; CC + FIFO routes were unaffected, so it was invisible from a CC session. reference_nh3_dev_althing_herald.

  • [2026-07-04] On-prem T1 train that keeps ANY ana-ml2 serving up = ~6-8 DAYS (1-GPU + NVMe ZeRO-Infinity offload; MoE ~10B-active cuts FLOPs but NOT the 244G base's param I/O). The only fast on-prem path is a FULL ana-ml2 shutdown (both GPUs + the ~421G vLLM RAM freed → base fits in the 566G CPU RAM) → CPU offload → ~1-day full-fleet outage. Cloud (no offload) = hours. reference_t1_cloud_train_plan.

  • [2026-07-01] A personal-Worldtree CI deploy that fails ~85s in with "not found / unauthorized" is usually the pull-only-vs-build RACE, not registry-auth. deploy-personal.yml is PULL-ONLY but fires on the staging/vX tag simultaneously with deploy.yml's build → pulls before the push finishes. FIX: re-run once built, or gate on workflow_run: completed.

  • [2026-07-01] MTP/spec-decode on a SHARED serving model helps single-stream but HURTS moderate-concurrency aggregate + silently ignores min_p/logit_bias (qwopus gen: N=1 +12%, N=4 −20%). Reserve for dedicated/interactive deployments.

  • [2026-07-02] irv-ml1 /worktank ROOT is root-owned — lkraven can't write there (irv-ml1 sudo needs a password) → stage model pulls to /home. PIN THE A6000 BY UUID for training (native-CUDA ordering differs vs docker; the 3090 index 0 is usually near-full → OOM). CUDA_VISIBLE_DEVICES=GPU-<uuid>.

101 older entries archived to archival-memory.md.

Reference in New Issue View Git Blame Copy Permalink