From bc084edbeefebf47983318e54024667ef810b2e8 Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Mon, 3 Aug 2026 00:12:59 -0700 Subject: [PATCH] =?UTF-8?q?memory:=20snapshot=20for=20/clear=20=E2=80=94?= =?UTF-8?q?=20worldtree=20b168/#384/#385=20arc=20complete?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Collapsed the completed-arc bloat out of Current state / in-flight (line 125: 6487->1073 chars; runbooks moved to persistent-memory.d/2026-08-03-worldtree-b168-384-385-arc.md). Added the 2026-08-03 arc Recent-decisions entry. Auto-archived 20 oldest Recent-decisions entries (2026-07-08..07-14) to archival-memory.md (index 345->308 lines). Refreshed the b169-deploy watch, unpushed-count, and handoff. --- archival-memory.md | 244 ++++++++++++++++++ ...s-silently-running-3-epochs-harness-gap.md | 19 -- ...-inference-pivot-executed-serve-curated.md | 8 - ...t-campaign-closed-char-rp-magidonia-24b.md | 11 - ...t1-dpo-leg-is-running-unblocked-2-fixes.md | 12 - ...dpo-leg-launch-prior-block-now-resolved.md | 17 -- ...y-open-weight-releases-scan-cron-set-up.md | 20 -- ...26-07-10-comfyui-0-25-x-bump-on-irv-ml1.md | 16 -- ...7-10-comfyui-v0-27-1-success-on-irv-ml1.md | 14 - ...toskr-affect-full-on-personal-worldtree.md | 13 - ...ot-cause-supersedes-the-litellm-gateway.md | 1 - ...-speed-real-bottleneck-uv-sync-memory-s.md | 1 - ...-consumer-ledger-miranda-provisioned-on.md | 1 - ...eploy-bottleneck-the-image-build-11-min.md | 1 - ...355-residual-300s-hang-localized-to-our.md | 1 - ...rn-lifecycle-fix-validated-on-worldtree.md | 1 - ...-rp-reasoning-seat-deckard-pkd-neo-code.md | 1 - ...-mtp-fast-char-rp-reasoning-seat-landed.md | 1 - ...-resolved-gibberish-pivoted-to-modelopt.md | 1 - ...ue-the-nvfp4-mtp-fast-char-rp-reasoning.md | 1 - ...lab-webhook-auto-deploy-real-root-cause.md | 1 - .../2026-08-03-worldtree-b168-384-385-arc.md | 17 ++ persistent-memory.md | 51 +--- 23 files changed, 268 insertions(+), 185 deletions(-) delete mode 100644 persistent-memory.d/2026-07-08-dpo-was-silently-running-3-epochs-harness-gap.md delete mode 100644 persistent-memory.d/2026-07-08-off-the-shelf-inference-pivot-executed-serve-curated.md delete mode 100644 persistent-memory.d/2026-07-08-rp-seat-campaign-closed-char-rp-magidonia-24b.md delete mode 100644 persistent-memory.d/2026-07-08-t1-dpo-leg-is-running-unblocked-2-fixes.md delete mode 100644 persistent-memory.d/2026-07-08-t1-dpo-leg-launch-prior-block-now-resolved.md delete mode 100644 persistent-memory.d/2026-07-10-biweekly-open-weight-releases-scan-cron-set-up.md delete mode 100644 persistent-memory.d/2026-07-10-comfyui-0-25-x-bump-on-irv-ml1.md delete mode 100644 persistent-memory.d/2026-07-10-comfyui-v0-27-1-success-on-irv-ml1.md delete mode 100644 persistent-memory.d/2026-07-10-heimdall-grant-ratatoskr-affect-full-on-personal-worldtree.md delete mode 100644 persistent-memory.d/2026-07-13-355-residual-root-cause-supersedes-the-litellm-gateway.md delete mode 100644 persistent-memory.d/2026-07-13-deploy-speed-real-bottleneck-uv-sync-memory-s.md delete mode 100644 persistent-memory.d/2026-07-13-ledger-tier-3-consumer-ledger-miranda-provisioned-on.md delete mode 100644 persistent-memory.d/2026-07-13-worldtree-deploy-bottleneck-the-image-build-11-min.md delete mode 100644 persistent-memory.d/2026-07-13-wt-355-residual-300s-hang-localized-to-our.md delete mode 100644 persistent-memory.d/2026-07-13-wt-355-turn-lifecycle-fix-validated-on-worldtree.md delete mode 100644 persistent-memory.d/2026-07-14-char-rp-reasoning-seat-deckard-pkd-neo-code.md delete mode 100644 persistent-memory.d/2026-07-14-nvfp4-mtp-fast-char-rp-reasoning-seat-landed.md delete mode 100644 persistent-memory.d/2026-07-14-nvfp4-quant-chase-resolved-gibberish-pivoted-to-modelopt.md delete mode 100644 persistent-memory.d/2026-07-14-pursue-the-nvfp4-mtp-fast-char-rp-reasoning.md delete mode 100644 persistent-memory.d/2026-07-14-soong-lab-webhook-auto-deploy-real-root-cause.md create mode 100644 persistent-memory.d/2026-08-03-worldtree-b168-384-385-arc.md diff --git a/archival-memory.md b/archival-memory.md index 9846954..fad9881 100644 --- a/archival-memory.md +++ b/archival-memory.md @@ -1482,3 +1482,247 @@ _Archived 2026-08-02._ _Archived 2026-08-02._ + +## Recent decisions (archived 2026-08-03 batch) + +### 2026-07-14-nvfp4-mtp-fast-char-rp-reasoning-seat-landed + +- `[2026-07-14]` **NVFP4+MTP fast char-rp-reasoning seat LANDED + LIVE + gateway-repointed + VRAM-tuned.** Modelopt-format re-quant made MTP work. The load-crash root cause = **vLLM 0.24 does NOT propagate modelopt `exclude_modules` to the spec-decode DRAFT model** → the bf16 mtp head gets quantized → shape crash; NO checkpoint config fixes it (`is_layer_skipped` is exact-membership, and the drafter never sees the mtp excludes) → **workaround = a mounted `sitecustomize` that force-skips `mtp.*` in `is_layer_skipped`** (upstream vLLM bug to file). Productionized as compose stack `heretic2-charrp-reasoning` (:8018, workaround baked in). Gateway `char-rp-reasoning` alias fixed: repointed off the stale GGUF served-name `deckard-pkd-27b`, added `enable_thinking:true`, **dropped `min_p`** (MTP-incompatible), canonical samplers temp1.0/top_p0.95/top_k20. Rebalanced GPU0 (gen 0.37→0.30/16-seq/256K + reasoning 0.39/16-seq/192K+MTP + char-rp 128K, 2.7GB free). All 4 gateway roles verified; vLLM reasoning-parser confirmed **leak-free** (unlike the GGUF budget-forcing). Full record + the 4 quant landmines in `docs/runbooks/heretic2-nvfp4-mtp-seat.md`; committed `982c319`. Open (non-blocking): brokkr P00 (seat is live ahead of it), retire the stopped GGUF reasoning seat, file the vLLM bug. + +_Archived 2026-08-03._ + +### 2026-07-14-nvfp4-quant-chase-resolved-gibberish-pivoted-to-modelopt + +- `[2026-07-14]` **NVFP4 quant chase RESOLVED (gibberish) + PIVOTED to modelopt for MTP.** One ~40-min GPU0 window. Root-caused the `!!!!` to the quant NAMESPACE (text-only `AutoModelForCausalLM`→`model.layers.*` keys; vLLM serves only `Qwen3_5ForConditionalGeneration`, which needs `model.language_model.*`) — found from config diffs + vLLM source with ZERO GPU time; fixed by loading as `AutoModelForImageTextToText`. NVFP4 now serves COHERENT (validated greedy). BUT base NVFP4 ≈53 tok/s ≈ GGUF's 59.5 at batch-1 (no single-stream win) AND MTP = 0% acceptance on compressed-tensors (bf16 mtp head only loads on the modelopt format). Operator chose to **pursue a modelopt-format re-quant** (the only path to the 2-4× MTP goal; AEON-proven on this exact Qwen3.6-27B arch). Scoped + de-risked: AEON `/tank/aimodels/qwen36-27b-aeon-nvfp4` = the modelopt reference (quant_method modelopt, 1967 tensors, 15 bf16 mtp keys identical to graft); nvidia-modelopt 0.45.0 installs + `mtq.quantize`/`NVFP4_DEFAULT_CFG`/`export_hf_checkpoint` API confirmed; pipeline unchanged except swap llm-compressor→modelopt. Seats restored; char-rp-reasoning stays GGUF. Full plan in Current state ★ section. + +_Archived 2026-08-03._ + +### 2026-07-14-pursue-the-nvfp4-mtp-fast-char-rp-reasoning + +- `[2026-07-14]` **Pursue the NVFP4+MTP fast char-rp-reasoning seat to completion** (Vuong-directed via /snapshot: "chase the nvfp4 quant, we know it works, write down the recipe"). Full recipe + diagnostic ladder in Current state / in-flight above. Artifacts on ana-ml2 `/tank/aimodels/heretic2-nvfp4-work/` + scripts committed in eshpfi `services/heretic2-nvfp4-quant/`. + +_Archived 2026-08-03._ + +### 2026-07-14-char-rp-reasoning-seat-deckard-pkd-neo-code + +- `[2026-07-14]` **char-rp-reasoning seat: Deckard-PKD → NEO-CODE = Heretic2-Thinking (Qwen3.6-27B)** — R36 gate PASSED (tools 0.967, #355 runaway ELIMINATED). #355 was MODEL-level (Deckard emitted qwen3_coder XML malformed → mangled args → retry-runaway), NOT the reasoning-budget bug; NEO-CODE emits it clean. Custom llama.cpp KEPT (qwen3_coder parse — stock b8840 predates it — + PR#25544). Committed f960a73; full record auto-memory [[charrp-custom-llamacpp-pr25544]]. + +_Archived 2026-08-03._ + +### 2026-07-14-soong-lab-webhook-auto-deploy-real-root-cause + +- `[2026-07-14]` **soong-lab webhook auto-deploy real root cause = gitea `webhook.ALLOWED_HOST_LIST`** (was `external, 10.100.0.0/16` = NH3-only; blocked corviduo-dev's Anaheim `10.250.x` → gitea refused to deliver, never opened the connection). Fixed to fleet-wide `10.0.0.0/8` (app.ini `[webhook]`) + gitea restart; listener now logs every delivery. The ufw `10/8` open (also this session) was a real-but-secondary gap. Committed 462d528. + +_Archived 2026-08-03._ + +### 2026-07-13-355-residual-root-cause-supersedes-the-litellm-gateway + +- `[2026-07-13]` **#355-residual ROOT CAUSE (supersedes the "LiteLLM gateway holds while seat idles" entry below — that was DISPROVEN).** char-rp-reasoning enters a non-terminating REASONING loop (tool-call-retry planning) and runs to `max_tokens=32768` (~22 min @ 24.7 tok/s, ~13% of requests); the seat GENERATES all 32768 tokens (not idle), and `--reasoning-budget 400` is NOT enforced. 3-source-confirmed (spend_logs completion_tokens=32768 ×4; seat eval-time log; pcap 100%-`reasoning_content` deltas). Server-side fix wanted (operator: no max_tokens ceiling) → routed to brokkr (accepted, pulled dvalin). Lesson (again): confirm before concluding — the seat-idle claim came from reading only the ≤73s requests + missing the concurrent 32768-token slots. See ACTIVE 1. + +_Archived 2026-08-03._ + +### 2026-07-13-deploy-speed-real-bottleneck-uv-sync-memory-s + +- `[2026-07-13]` **Deploy-speed real bottleneck ≠ uv sync (memory's assumption was wrong).** Buildx step log: `chown -R /app` = 251s (copy-up of the root-owned venv into a fresh layer), uv sync only 35.6s, registry layer cache already wired. Fix = drop `/app` from the chown (validated safe: zero /app runtime writes on both live instances) + uv cache-mount. Shipped as PR #359 (branch off origin/main@b60), worldtree-dev green-lit. Expected ~5min off (~11→~6min). Runner-side BuildKit cache task (b) was already done → moot. + +_Archived 2026-08-03._ + +### 2026-07-13-wt-355-residual-300s-hang-localized-to-our + +- `[2026-07-13]` WT #355 residual 300s hang localized to OUR LiteLLM gateway (holds 2 char-rp-reasoning requests ~21 min while the seat idles), NOT the seat — Deckard seat EXONERATED (completes ≤72s; `--reasoning-budget 400` forecloses a mid-thinking hang). Corrects worldtree-dev's "seat wedging" diagnosis. Decisive next = the FIN-check (pcap on corviuo). See in-flight ACTIVE 1. **[SUPERSEDED 2026-07-13 — see the ROOT CAUSE entry above; the gateway-hold/slot-leak theory was disproven, the seat was generating 32768 tokens.]** + +_Archived 2026-08-03._ + +### 2026-07-13-wt-355-turn-lifecycle-fix-validated-on-worldtree + +- `[2026-07-13]` WT #355 turn-lifecycle fix VALIDATED on worldtree b60 — wedged turns self-terminate cancelled/stalled at the 300s stall-watchdog (turns 2064/2065 vs pre-b60 2061's 16-min no-terminal). worldtree-dev filed follow-ons #356 (rehydrate Tier-3 ctx on resume — the recreate-durability gap), #357 (reclaim orphaned active-turn locks), #358 (LLM-provider read-timeout audit); surfacing to Vuong to prioritize. + +_Archived 2026-08-03._ + +### 2026-07-13-worldtree-deploy-bottleneck-the-image-build-11-min + +- `[2026-07-13]` Worldtree deploy bottleneck = the image build (~11 min of a ~12 min deploy), root cause the Dockerfile `uv sync ... --no-cache` + no BuildKit cache-mount (re-downloads all deps cold every build). Fix split: worldtree-dev Dockerfile cache-mount diff + infra-ops runner-side persistent BuildKit cache. Config-only changes skip the build entirely (pinned recreate). + +_Archived 2026-08-03._ + +### 2026-07-13-ledger-tier-3-consumer-ledger-miranda-provisioned-on + +- `[2026-07-13]` Ledger tier-3 consumer `ledger:miranda` provisioned on personal :8081 (key b38932f5, GPG-delivered+shredded, allowlist 10.100.10.50:8770 live); `assistant`+`thoughtful-assistant` capability roles added (gen/gen-reasoning) on personal+demo, canonical d8bd497. Chosen instance = personal (the tier-3-consumer instance, ratatoskr+soong-lab colocated). + +_Archived 2026-08-03._ + +### 2026-07-10-heimdall-grant-ratatoskr-affect-full-on-personal-worldtree + +- `[2026-07-10]` **Heimdall grant: ratatoskr `affect.full` on PERSONAL Worldtree (operator-approved, worldtree-dev + R34-v1 request).** Added allow-rule `ratatoskr-affect-full-allow` to `/opt/worldtree-personal/config/policies.yaml` + (`principal.user_ids:["ratatoskr"]`, action `affect.full`, resource `*`, effect allow), mirroring the #347 + `session-history-write-ratatoskr` rule exactly + placed right after it. **WHY user_ids-based (not tier):** ratatoskr's + personal key is the minimal **readonly-admin** observability tier, which is NOT in the tier-based + `affect-render-baseline-allow` (anonymous/user/free/pro/admin) → needs an explicit user_id grant, same as #347. + R34-v1 (b46, committed UNPUSHED) gates `affect.emit` `dominant_emotion` egress by exposure ceiling (affect.full|safe + → present; neither → null); this grant keeps ratatoskr's view alive across the b46 deploy. Surgical exact-string + insert (preserves comments), backup `policies.yaml.bak-pre-affectfull-20260710`, validated via the CONTAINER's own + yaml parser (35 rules, +1, payload confirmed). **NOT restarted — deliberate:** rule is on the bind-mount (shadows + baked), INERT until b46 gating ships, so the b46 CI/CD deploy restart activates it (no live-session blip now). Demo + untouched (personal-only per key scope). Replied to wt-dev (thread 01KX6DB3…) offering an immediate restart if they + want it live for pre-b46 testing. See [[reference_corviduo_dev_emergency_ops]]. + +_Archived 2026-08-03._ + +### 2026-07-10-comfyui-v0-27-1-success-on-irv-ml1 + +- `[2026-07-10]` **ComfyUI v0.27.1 SUCCESS on irv-ml1 (operator-confirmed execute-now) — landed on torch 2.12.1, + SageAttention preserved, crash-loop AVOIDED.** The prior attempt (entry below) crash-looped because a torch + 2.12.1→2.13 bump broke SageAttention's ABI. This time I checked `git diff v0.24.1 v0.27.1 -- requirements.txt` + FIRST and found **core v0.27.1 leaves `torch` UNPINNED** → the version bump does NOT require torch 2.13 (that came + only from the mmartial boot-upgrade). So: `git checkout v0.27.1` (clean tree) → `pip install -r requirements.txt` + as **uid 1000** with a **torch-pin constraint file** (torch/vision/audio pinned to current +cu129) to block any + transitive bump → torch stayed 2.12.1, SageAttention 2.2.0 untouched. Added decord 0.6.0 (fixed SAM3Segment). + `docker restart comfyui` → healthy, `/system_stats` comfyui_version=0.27.1, "Using sage attention", HTTP 200, DB + migrated 0003→0004. Reported the divergence to comfy-dev (thread 01KX6D3C…, reply pending) + asked whether LTXV 2.3 + needs a separate torch-2.13 follow-up (their domain; Krea-2's ≥0.25 need is met by 0.27.1). **LESSON: before a + mmartial ComfyUI version bump, `git diff -- requirements.txt` — if torch is unpinned, bump the CODE + without touching torch (constraint-pin it) and compiled exts (SageAttention) survive. `docker exec` lands as uid + 1025(comfytoo), not 1000 — use `-u 1000` + the venv python `/comfy/mnt/venv/bin/python`.** See + [[reference_irv_ml1_comfyui_mmartial]]. + +_Archived 2026-08-03._ + +### 2026-07-10-comfyui-0-25-x-bump-on-irv-ml1 + +- `[2026-07-10]` **ComfyUI 0.25.x bump on irv-ml1 ATTEMPTED → FAILED → ROLLED BACK (snapshot saved it).** comfy-dev + requested (Vuong-authorized) bumping the irv-ml1 `comfyui` stack (mmartial image, `/opt/docker/compose/comfyui/`, + 0.24.1) to 0.25.x for Krea-2 + LTXV 2.3. **TWO FINDINGS: (1) `DISABLE_UPGRADES=false`/USE_PIPUPGRADE bumps the + VENV (torch 2.12.1→2.13.0 + deps) but does NOT advance the ComfyUI CODE checkout** (`/comfy/mnt/ComfyUI` = + `/worktank/comfyui/run/ComfyUI` stayed 0.24.1 — pinned/detached git, comfy-dev's domain). **(2) the torch bump + broke SageAttention** (2.2.0 `_fused.so` undefined-symbol `c10::impl::cow::materialize_cow_storage` vs torch + 2.13.0) → `--use-sage-attention` (REQUIRED launch flag in COMFY_CMDLINE_EXTRA) crash-looped ComfyUI. Net: broke + the working state, zero 0.25.x payoff. **ROLLBACK WORKED**: pre-bump 16G venv snapshot + `/worktank/comfyui/venv-snapshot-comfyui-0.24.1-20260710.tar` restored (torch 2.12.1 + working SageAttention), + re-pinned DISABLE_UPGRADES=true, recreated → healthy on 0.24.1, serving :8188. Broken venv parked at + `/worktank/comfyui/run/venv.broken-torch213-20260710`. **CORRECTED PATH (sent comfy-dev, thread 01KX655V):** + comfy-dev git-advances the ComfyUI checkout to 0.25.x + reqs → I handle the torch bump + SageAttention + rebuild-against-2.13.0 + re-pin (snapshot stays as the net). **LESSON: mmartial `DISABLE_UPGRADES` gates ONLY + the venv pip-upgrades, NOT the ComfyUI git checkout; a torch bump breaks compiled exts (SageAttention) → + rebuild-after is mandatory.** Bump BLOCKED pending comfy-dev's git-advance. Stack: A6000 (NVIDIA_VISIBLE_DEVICES=1), + lkraven-owned compose+venv (uid 1000, no sudo needed), COMFY_CMDLINE_EXTRA OOM flags preserved. + +_Archived 2026-08-03._ + +### 2026-07-10-biweekly-open-weight-releases-scan-cron-set-up + +- `[2026-07-10]` **Biweekly open-weight-releases scan cron set up for brokkr-smithy (Vuong-authorized).** Durable + systemd **--user** timer on nh3-dev (`brokkr-landscape-scan.timer`, OnCalendar `*-*-01,15 09:00:00` + America/Los_Angeles, Persistent=true; linger on) → `.service` → wrapper `~/.local/bin/brokkr-landscape-scan.sh` + runs headless `claude -p "$(cat ~/.config/brokkr-landscape-scan/prompt.txt)" --dangerously-skip-permissions` in + `~/development/brokkr-smithy` (ALTHING_HANDLE=brokkr-smithy-dev; **explicit PATH** — the --user minimal-PATH + footgun; per-run logs `~/.local/state/brokkr-landscape-scan/`). Prompt = brokkr's payload verbatim (LLM/image/TTS + new-release sweep → ranked synthesis → commit+push+notify). VALIDATED: git-push non-interactive (BatchMode + ls-remote to gitea, passphraseless key — no agent), headless claude auth (READY smoke). VALIDATED END-TO-END 2026-07-10 (manual run, exit 0): + web-sweep→synthesis→commit `2ed2f29`→PUSH of scan #2 (open-weight-releases-2026-07-24.md); triaged dwarf input + + caught baseline errors, quality strong. **HANDLE-COLLISION caught+FIXED** — the headless scan shared handle + brokkr-smithy-dev with the LIVE session + raced its inbox (eitri's dwarf-reply got stolen by the live monitor); + registered a dedicated **brokkr-scan-dev** handle (`add-handle`, driver=none) + repointed the wrapper + rewired + step-5 notify → `althing-cli post --to brokkr-smithy-dev` (NO vuong althing handle exists — confirmed). model=default + + `--max-turns 80`. First run under the new handle = 7/15. Off-cycle 07-24 doc is a validation artifact (scheduled + 1st/15th runs date to their own run-date, no collision) — operator naming-convention call pending. + **NEXT AUTO-RUN 2026-07-15 09:00 PDT.** Manual validation/first run = `systemctl --user start + brokkr-landscape-scan.service`. Open w/ brokkr (thread 01KX63G6): confirm notify-Vuong handle/mechanism + session + handle + model/turn-cap. **NEXT brokkr task (operator-sequenced after this): TTS audition env** — Higgs-TTS-3 + + ZONOS2 + Chatterbox baseline, TTFA/RTF + blind-A/B web-listen (thread 01KX6371; needs GPU-placement + HF-token + feasibility pass first; brokkr delivers the prompt set after the env's up; protocol doc in brokkr-smithy repo). + +_Archived 2026-08-03._ + +### 2026-07-08-rp-seat-campaign-closed-char-rp-magidonia-24b + +- `[2026-07-08]` **RP-SEAT CAMPAIGN CLOSED — char-rp = Magidonia-24B-v4.3 (128K), char-rp-reasoning = Deckard-PKD + Qwen3.5-27B (256K); both GGUF/llama.cpp on ana-ml2 GPU0 alongside gen (35B-A3B, util 0.37), ~4G GPU0 margin.** + Arc: (1) replaced broken Angel NVFP4 with Magidonia prose + QwQ-RpR-v4 reasoning (b268f93); (2) max-context via q8_0 + KV (f570604); (3) canonical samplers for all 4 gateway seats, dvalin-derived + char-rp A/B-tuned (aac4bcf); + (4) rebalanced gen 0.40→0.37 to fund char-rp 128K (f49c4e4); (5) RE-A/B'd the reasoning seat (operator wanted a + DRY-tolerant model): **Deckard WON** on brokkr's frozen scorer (composite 2.176, 0/30 loops, 0/30 refusals) over + RpR-v4 (3.716, 1/30 loop), Pantheon-Reasoning (1.383 but 7/30 refusals), Snowdrop+Gembrain (llama.cpp + template-incompat) — deployed (5f79b40); (6) Deckard→256K (41305bf); (7) dvalin CONFIRMED Deckard samplers = the + live A/B set is canonical (4954ca0). **GATE LESSON: a llama.cpp reasoning seat needs a STOCK template that natively + opens ``/`enable_thinking` (Qwen3.x/QwQ pass; ChatML + Gemma-4 fail) — no monkeypatching. INFRA: llama-swap + b8840 can't load Qwen3.6/Gemma-4 archs → `ghcr.io/ggml-org/llama.cpp:server-cuda` (pulled on ana-ml2).** + +_Archived 2026-08-03._ + +### 2026-07-08-off-the-shelf-inference-pivot-executed-serve-curated + +- `[2026-07-08]` **OFF-THE-SHELF INFERENCE PIVOT executed — serve curated abliterated models, stop home-training.** + Final topology: **gen = `llmfan46/Qwen3.6-35B-A3B-uncensored-heretic-NVFP4-Experts-Only`** (LIVE, modelopt, vision, + util 0.40), **char-rp = an RP unicorn to be found on fresh context** (see Current state). Intermediate steps + ABANDONED: Pantheon-Reasoning-27B (served briefly as gen — refuses dark fiction via DeepSeek-distilled + refusal-reasoning, see Tried); Pantheon-27B-with-MTP for RP (bf16 MTP won't load on the compressed-tensors path); + Angel MS3.2-24B (my NVFP4 quant = garbage). Prefer EXISTING community NVFP4/GGUF quants over self-quanting + ("don't quant unless you have to" — operator). GGUF serving is now on the table for RP (NEVER Ollama). Gateway + sampling-defaults wiring still PENDING. + +_Archived 2026-08-03._ + +### 2026-07-08-dpo-was-silently-running-3-epochs-harness-gap + +- `[2026-07-08]` **DPO was silently running 3 epochs (harness gap) → KILLED at epoch 1.2, retargeted to 0.3 + epochs (operator call).** Root cause: `DpoConfig` had NO `epochs` field + `_dpo_config_kwargs` didn't pass + `num_train_epochs` → DPO fell through to trl DPOConfig's default 3.0 (SFT correctly pins 1 via SftConfig.epochs + + _sft_config_kwargs). Objective SATURATED by ~epoch 0.27 (loss~0, grad~0, acc 1.0, margins~27 flat — the + off-policy qwopus rejected pairs are trivially separable), so epochs ~0.3→3 were pure over-optimization + a + ~6.5h outage. No mid-run checkpoint (save_steps 500 > total steps; save only at end) → killing lost the run. + FIX (3 edits to deployed harness, mtf-dev to canonicalize): `DpoConfig.epochs: float = 1` (mirrors SftConfig, + float for fractions); `_dpo_config_kwargs` now passes `"num_train_epochs": cfg.epochs`; recipe `dpo.epochs: 0.3`. + GPU-free verified (dpo.epochs=0.3 → num_train_epochs=0.3). Relaunched at 0.3 epoch (~30min precompute + ~12min + train = ~45min). **DONE + SANITY-CHECKED (exit 0, ~70min wall: ~30min fixed precompute + 45 steps @ ~51s/step; + train_loss 1.4e-5 @ epoch 0.301).** Fresh `data/spike/dpo_adapter/adapter_model.safetensors` (123MB) + checkpoint-45 + banked. **3-way greedy sanity (base vs SFT vs DPO, via peft load + disable_adapter/set_adapter on GPU0): + ALL THREE DISTINCT** (base≠sft≠dpo) → full SFT→DPO pipeline applies end-to-end at inference. **DPO 0.3ep is + COHERENT, fluent, NOT degenerate** (early-stop avoided over-optimization) but the quality delta on a neutral + literary prompt is SUBTLE (DPO shares SFT's structure — it continues from it — with minor stylistic drift, + arguably slightly MORE genre-clichéd). Verdict: mechanics proven, quality gain modest as predicted for 0.3ep + on off-policy pairs; the real unlock remains on-policy rejected regen + on-domain (explicit E-RP) eval + + the LitBench/holdout run. gen+rp RESTORED healthy. Next: serve fork (SGLang-finish vs merge) on the DPO + adapter — same rekey_lora_for_vllm.py (zero-z) applies unchanged (mtf-dev confirmed). + +_Archived 2026-08-03._ + +### 2026-07-08-t1-dpo-leg-is-running-unblocked-2-fixes + +- `[2026-07-08]` **T1 DPO leg is RUNNING (unblocked) — 2 fixes applied to deployed backend.py.** + Blocker resolved: (1) **mtf-dev's v0.0.42 stub** `_stub_missing_optional_integrations` (last-resort sys.meta_path + finder → missing mergekit/llm_blender/weave resolve to MagicMock, never called → zero numerics risk; applied + VERBATIM to deployed `src/model_training_forge/train/backend.py` after `_unsloth_available()` + call-site before + `from trl import DPOTrainer`); (2) **my cosmetic `warnings_issued` shim** (trl-0.24 DPOTrainer.__init__:405 does + `model.warnings_issued["estimate_tokens"]=True` for warning-suppression; custom Qwen3_5 class under transformers + 5.5.0 lacks the attr → `if not hasattr(model,"warnings_issued"): model.warnings_issued={}` before the + DPOTrainer(...).train() at backend.py:305 — cosmetic, zero training impact). Both edits are on the DEPLOYED + un-git'd copy only → **mtf-dev must canonicalize the warnings_issued shim into their repo** (told them). DPO + confirmed training: model loaded (851 shards), full 1196 pairs processed, in precompute_ref_log_probs (GPU0 93% + util, 54.8GB). Completion watcher armed (bg task) → restore gen+rp + verify dpo_adapter + ping mtf-dev on exit. + gen+rp STOPPED for the run (authorized window). Output → data/spike/dpo_adapter. + +_Archived 2026-08-03._ + +### 2026-07-08-t1-dpo-leg-launch-prior-block-now-resolved + +- `[2026-07-08]` **T1 DPO leg launch — prior BLOCK (now resolved above), kept for the launch recipe.** + Operator authorized the full DPO stage (via mtf-dev) + went AFK 2h. **PROVEN LAUNCH RECIPE** (replicates the + SFT container `aeon-t1-sft` exactly, only `--stage sft`→`dpo`): `sudo docker run -d --name aeon-t1-dpo + --entrypoint python3 --gpus all -e CUDA_VISIBLE_DEVICES=0 -e MTF_FORCE_TRL=1 -e PYTHONPATH=/mtf/src + -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True -v /home/lkraven/model-training-forge:/mtf -v /tank:/tank + -w /mtf aeon-trainer:latest -u scripts/train.py --recipe recipes/training/qwen-3.5-122b-erp-lora/train.aeon-27b.yaml + --stage dpo`. **CRITICAL: `--entrypoint python3` is REQUIRED** — aeon-trainer's default entrypoint is + `["vllm","serve"]` (FROM vllm/vllm-openai) → without the override it runs vllm + hits a torch-ABI crash. + Dataset verified (pairs_dataset=train.flat.json=1196 pairs). **THE BLOCK:** `from trl import DPOTrainer` + (backend.py:256) eagerly pulls TRL 0.24.0's WHOLE optional-integration set — cascade: mergekit(missing)→ + immutables→**mergekit-0.1.4↔pydantic-2.13 HARD incompat** (needs pydantic==2.10.6)→llm_blender→dataclasses_json→ + **llm_blender-0.0.2↔transformers-5.5.0 HARD incompat** (TRANSFORMERS_CACHE removed, needs source patch)→weave→ + (more). NONE used by our pair-based DPO. `pip install mergekit` w/deps is UNSAFE (downgrades accelerate + 1.14→1.6). Safe partial recipe derived (core libs held: torch2.10/tf5.5.0/trl0.24.0/peft0.19.1/accel1.14.0) + but non-convergent → TRULY BLOCKING per operator's carve-out. Did NOT force-hack the proven training image. + Handed full diagnosis + recommended fix (lazy-import TRL patch, opt b) to mtf-dev (thread 01KWZG8GJX, + expects-reply, monitor armed). gen+rp RESTORED healthy. Relaunch = 1 min once mtf-dev delivers a working image. + +_Archived 2026-08-03._ + diff --git a/persistent-memory.d/2026-07-08-dpo-was-silently-running-3-epochs-harness-gap.md b/persistent-memory.d/2026-07-08-dpo-was-silently-running-3-epochs-harness-gap.md deleted file mode 100644 index a3bd8ef..0000000 --- a/persistent-memory.d/2026-07-08-dpo-was-silently-running-3-epochs-harness-gap.md +++ /dev/null @@ -1,19 +0,0 @@ -- `[2026-07-08]` **DPO was silently running 3 epochs (harness gap) → KILLED at epoch 1.2, retargeted to 0.3 - epochs (operator call).** Root cause: `DpoConfig` had NO `epochs` field + `_dpo_config_kwargs` didn't pass - `num_train_epochs` → DPO fell through to trl DPOConfig's default 3.0 (SFT correctly pins 1 via SftConfig.epochs - + _sft_config_kwargs). Objective SATURATED by ~epoch 0.27 (loss~0, grad~0, acc 1.0, margins~27 flat — the - off-policy qwopus rejected pairs are trivially separable), so epochs ~0.3→3 were pure over-optimization + a - ~6.5h outage. No mid-run checkpoint (save_steps 500 > total steps; save only at end) → killing lost the run. - FIX (3 edits to deployed harness, mtf-dev to canonicalize): `DpoConfig.epochs: float = 1` (mirrors SftConfig, - float for fractions); `_dpo_config_kwargs` now passes `"num_train_epochs": cfg.epochs`; recipe `dpo.epochs: 0.3`. - GPU-free verified (dpo.epochs=0.3 → num_train_epochs=0.3). Relaunched at 0.3 epoch (~30min precompute + ~12min - train = ~45min). **DONE + SANITY-CHECKED (exit 0, ~70min wall: ~30min fixed precompute + 45 steps @ ~51s/step; - train_loss 1.4e-5 @ epoch 0.301).** Fresh `data/spike/dpo_adapter/adapter_model.safetensors` (123MB) + checkpoint-45 - banked. **3-way greedy sanity (base vs SFT vs DPO, via peft load + disable_adapter/set_adapter on GPU0): - ALL THREE DISTINCT** (base≠sft≠dpo) → full SFT→DPO pipeline applies end-to-end at inference. **DPO 0.3ep is - COHERENT, fluent, NOT degenerate** (early-stop avoided over-optimization) but the quality delta on a neutral - literary prompt is SUBTLE (DPO shares SFT's structure — it continues from it — with minor stylistic drift, - arguably slightly MORE genre-clichéd). Verdict: mechanics proven, quality gain modest as predicted for 0.3ep - on off-policy pairs; the real unlock remains on-policy rejected regen + on-domain (explicit E-RP) eval + - the LitBench/holdout run. gen+rp RESTORED healthy. Next: serve fork (SGLang-finish vs merge) on the DPO - adapter — same rekey_lora_for_vllm.py (zero-z) applies unchanged (mtf-dev confirmed). diff --git a/persistent-memory.d/2026-07-08-off-the-shelf-inference-pivot-executed-serve-curated.md b/persistent-memory.d/2026-07-08-off-the-shelf-inference-pivot-executed-serve-curated.md deleted file mode 100644 index 5b77505..0000000 --- a/persistent-memory.d/2026-07-08-off-the-shelf-inference-pivot-executed-serve-curated.md +++ /dev/null @@ -1,8 +0,0 @@ -- `[2026-07-08]` **OFF-THE-SHELF INFERENCE PIVOT executed — serve curated abliterated models, stop home-training.** - Final topology: **gen = `llmfan46/Qwen3.6-35B-A3B-uncensored-heretic-NVFP4-Experts-Only`** (LIVE, modelopt, vision, - util 0.40), **char-rp = an RP unicorn to be found on fresh context** (see Current state). Intermediate steps - ABANDONED: Pantheon-Reasoning-27B (served briefly as gen — refuses dark fiction via DeepSeek-distilled - refusal-reasoning, see Tried); Pantheon-27B-with-MTP for RP (bf16 MTP won't load on the compressed-tensors path); - Angel MS3.2-24B (my NVFP4 quant = garbage). Prefer EXISTING community NVFP4/GGUF quants over self-quanting - ("don't quant unless you have to" — operator). GGUF serving is now on the table for RP (NEVER Ollama). Gateway - sampling-defaults wiring still PENDING. diff --git a/persistent-memory.d/2026-07-08-rp-seat-campaign-closed-char-rp-magidonia-24b.md b/persistent-memory.d/2026-07-08-rp-seat-campaign-closed-char-rp-magidonia-24b.md deleted file mode 100644 index 58d4f02..0000000 --- a/persistent-memory.d/2026-07-08-rp-seat-campaign-closed-char-rp-magidonia-24b.md +++ /dev/null @@ -1,11 +0,0 @@ -- `[2026-07-08]` **RP-SEAT CAMPAIGN CLOSED — char-rp = Magidonia-24B-v4.3 (128K), char-rp-reasoning = Deckard-PKD - Qwen3.5-27B (256K); both GGUF/llama.cpp on ana-ml2 GPU0 alongside gen (35B-A3B, util 0.37), ~4G GPU0 margin.** - Arc: (1) replaced broken Angel NVFP4 with Magidonia prose + QwQ-RpR-v4 reasoning (b268f93); (2) max-context via q8_0 - KV (f570604); (3) canonical samplers for all 4 gateway seats, dvalin-derived + char-rp A/B-tuned (aac4bcf); - (4) rebalanced gen 0.40→0.37 to fund char-rp 128K (f49c4e4); (5) RE-A/B'd the reasoning seat (operator wanted a - DRY-tolerant model): **Deckard WON** on brokkr's frozen scorer (composite 2.176, 0/30 loops, 0/30 refusals) over - RpR-v4 (3.716, 1/30 loop), Pantheon-Reasoning (1.383 but 7/30 refusals), Snowdrop+Gembrain (llama.cpp - template-incompat) — deployed (5f79b40); (6) Deckard→256K (41305bf); (7) dvalin CONFIRMED Deckard samplers = the - live A/B set is canonical (4954ca0). **GATE LESSON: a llama.cpp reasoning seat needs a STOCK template that natively - opens ``/`enable_thinking` (Qwen3.x/QwQ pass; ChatML + Gemma-4 fail) — no monkeypatching. INFRA: llama-swap - b8840 can't load Qwen3.6/Gemma-4 archs → `ghcr.io/ggml-org/llama.cpp:server-cuda` (pulled on ana-ml2).** diff --git a/persistent-memory.d/2026-07-08-t1-dpo-leg-is-running-unblocked-2-fixes.md b/persistent-memory.d/2026-07-08-t1-dpo-leg-is-running-unblocked-2-fixes.md deleted file mode 100644 index 510dec4..0000000 --- a/persistent-memory.d/2026-07-08-t1-dpo-leg-is-running-unblocked-2-fixes.md +++ /dev/null @@ -1,12 +0,0 @@ -- `[2026-07-08]` **T1 DPO leg is RUNNING (unblocked) — 2 fixes applied to deployed backend.py.** - Blocker resolved: (1) **mtf-dev's v0.0.42 stub** `_stub_missing_optional_integrations` (last-resort sys.meta_path - finder → missing mergekit/llm_blender/weave resolve to MagicMock, never called → zero numerics risk; applied - VERBATIM to deployed `src/model_training_forge/train/backend.py` after `_unsloth_available()` + call-site before - `from trl import DPOTrainer`); (2) **my cosmetic `warnings_issued` shim** (trl-0.24 DPOTrainer.__init__:405 does - `model.warnings_issued["estimate_tokens"]=True` for warning-suppression; custom Qwen3_5 class under transformers - 5.5.0 lacks the attr → `if not hasattr(model,"warnings_issued"): model.warnings_issued={}` before the - DPOTrainer(...).train() at backend.py:305 — cosmetic, zero training impact). Both edits are on the DEPLOYED - un-git'd copy only → **mtf-dev must canonicalize the warnings_issued shim into their repo** (told them). DPO - confirmed training: model loaded (851 shards), full 1196 pairs processed, in precompute_ref_log_probs (GPU0 93% - util, 54.8GB). Completion watcher armed (bg task) → restore gen+rp + verify dpo_adapter + ping mtf-dev on exit. - gen+rp STOPPED for the run (authorized window). Output → data/spike/dpo_adapter. diff --git a/persistent-memory.d/2026-07-08-t1-dpo-leg-launch-prior-block-now-resolved.md b/persistent-memory.d/2026-07-08-t1-dpo-leg-launch-prior-block-now-resolved.md deleted file mode 100644 index 3025923..0000000 --- a/persistent-memory.d/2026-07-08-t1-dpo-leg-launch-prior-block-now-resolved.md +++ /dev/null @@ -1,17 +0,0 @@ -- `[2026-07-08]` **T1 DPO leg launch — prior BLOCK (now resolved above), kept for the launch recipe.** - Operator authorized the full DPO stage (via mtf-dev) + went AFK 2h. **PROVEN LAUNCH RECIPE** (replicates the - SFT container `aeon-t1-sft` exactly, only `--stage sft`→`dpo`): `sudo docker run -d --name aeon-t1-dpo - --entrypoint python3 --gpus all -e CUDA_VISIBLE_DEVICES=0 -e MTF_FORCE_TRL=1 -e PYTHONPATH=/mtf/src - -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True -v /home/lkraven/model-training-forge:/mtf -v /tank:/tank - -w /mtf aeon-trainer:latest -u scripts/train.py --recipe recipes/training/qwen-3.5-122b-erp-lora/train.aeon-27b.yaml - --stage dpo`. **CRITICAL: `--entrypoint python3` is REQUIRED** — aeon-trainer's default entrypoint is - `["vllm","serve"]` (FROM vllm/vllm-openai) → without the override it runs vllm + hits a torch-ABI crash. - Dataset verified (pairs_dataset=train.flat.json=1196 pairs). **THE BLOCK:** `from trl import DPOTrainer` - (backend.py:256) eagerly pulls TRL 0.24.0's WHOLE optional-integration set — cascade: mergekit(missing)→ - immutables→**mergekit-0.1.4↔pydantic-2.13 HARD incompat** (needs pydantic==2.10.6)→llm_blender→dataclasses_json→ - **llm_blender-0.0.2↔transformers-5.5.0 HARD incompat** (TRANSFORMERS_CACHE removed, needs source patch)→weave→ - (more). NONE used by our pair-based DPO. `pip install mergekit` w/deps is UNSAFE (downgrades accelerate - 1.14→1.6). Safe partial recipe derived (core libs held: torch2.10/tf5.5.0/trl0.24.0/peft0.19.1/accel1.14.0) - but non-convergent → TRULY BLOCKING per operator's carve-out. Did NOT force-hack the proven training image. - Handed full diagnosis + recommended fix (lazy-import TRL patch, opt b) to mtf-dev (thread 01KWZG8GJX, - expects-reply, monitor armed). gen+rp RESTORED healthy. Relaunch = 1 min once mtf-dev delivers a working image. diff --git a/persistent-memory.d/2026-07-10-biweekly-open-weight-releases-scan-cron-set-up.md b/persistent-memory.d/2026-07-10-biweekly-open-weight-releases-scan-cron-set-up.md deleted file mode 100644 index 024ebbc..0000000 --- a/persistent-memory.d/2026-07-10-biweekly-open-weight-releases-scan-cron-set-up.md +++ /dev/null @@ -1,20 +0,0 @@ -- `[2026-07-10]` **Biweekly open-weight-releases scan cron set up for brokkr-smithy (Vuong-authorized).** Durable - systemd **--user** timer on nh3-dev (`brokkr-landscape-scan.timer`, OnCalendar `*-*-01,15 09:00:00` - America/Los_Angeles, Persistent=true; linger on) → `.service` → wrapper `~/.local/bin/brokkr-landscape-scan.sh` - runs headless `claude -p "$(cat ~/.config/brokkr-landscape-scan/prompt.txt)" --dangerously-skip-permissions` in - `~/development/brokkr-smithy` (ALTHING_HANDLE=brokkr-smithy-dev; **explicit PATH** — the --user minimal-PATH - footgun; per-run logs `~/.local/state/brokkr-landscape-scan/`). Prompt = brokkr's payload verbatim (LLM/image/TTS - new-release sweep → ranked synthesis → commit+push+notify). VALIDATED: git-push non-interactive (BatchMode - ls-remote to gitea, passphraseless key — no agent), headless claude auth (READY smoke). VALIDATED END-TO-END 2026-07-10 (manual run, exit 0): - web-sweep→synthesis→commit `2ed2f29`→PUSH of scan #2 (open-weight-releases-2026-07-24.md); triaged dwarf input + - caught baseline errors, quality strong. **HANDLE-COLLISION caught+FIXED** — the headless scan shared handle - brokkr-smithy-dev with the LIVE session + raced its inbox (eitri's dwarf-reply got stolen by the live monitor); - registered a dedicated **brokkr-scan-dev** handle (`add-handle`, driver=none) + repointed the wrapper + rewired - step-5 notify → `althing-cli post --to brokkr-smithy-dev` (NO vuong althing handle exists — confirmed). model=default - + `--max-turns 80`. First run under the new handle = 7/15. Off-cycle 07-24 doc is a validation artifact (scheduled - 1st/15th runs date to their own run-date, no collision) — operator naming-convention call pending. - **NEXT AUTO-RUN 2026-07-15 09:00 PDT.** Manual validation/first run = `systemctl --user start - brokkr-landscape-scan.service`. Open w/ brokkr (thread 01KX63G6): confirm notify-Vuong handle/mechanism + session - handle + model/turn-cap. **NEXT brokkr task (operator-sequenced after this): TTS audition env** — Higgs-TTS-3 + - ZONOS2 + Chatterbox baseline, TTFA/RTF + blind-A/B web-listen (thread 01KX6371; needs GPU-placement + HF-token - feasibility pass first; brokkr delivers the prompt set after the env's up; protocol doc in brokkr-smithy repo). diff --git a/persistent-memory.d/2026-07-10-comfyui-0-25-x-bump-on-irv-ml1.md b/persistent-memory.d/2026-07-10-comfyui-0-25-x-bump-on-irv-ml1.md deleted file mode 100644 index 9e903ea..0000000 --- a/persistent-memory.d/2026-07-10-comfyui-0-25-x-bump-on-irv-ml1.md +++ /dev/null @@ -1,16 +0,0 @@ -- `[2026-07-10]` **ComfyUI 0.25.x bump on irv-ml1 ATTEMPTED → FAILED → ROLLED BACK (snapshot saved it).** comfy-dev - requested (Vuong-authorized) bumping the irv-ml1 `comfyui` stack (mmartial image, `/opt/docker/compose/comfyui/`, - 0.24.1) to 0.25.x for Krea-2 + LTXV 2.3. **TWO FINDINGS: (1) `DISABLE_UPGRADES=false`/USE_PIPUPGRADE bumps the - VENV (torch 2.12.1→2.13.0 + deps) but does NOT advance the ComfyUI CODE checkout** (`/comfy/mnt/ComfyUI` = - `/worktank/comfyui/run/ComfyUI` stayed 0.24.1 — pinned/detached git, comfy-dev's domain). **(2) the torch bump - broke SageAttention** (2.2.0 `_fused.so` undefined-symbol `c10::impl::cow::materialize_cow_storage` vs torch - 2.13.0) → `--use-sage-attention` (REQUIRED launch flag in COMFY_CMDLINE_EXTRA) crash-looped ComfyUI. Net: broke - the working state, zero 0.25.x payoff. **ROLLBACK WORKED**: pre-bump 16G venv snapshot - `/worktank/comfyui/venv-snapshot-comfyui-0.24.1-20260710.tar` restored (torch 2.12.1 + working SageAttention), - re-pinned DISABLE_UPGRADES=true, recreated → healthy on 0.24.1, serving :8188. Broken venv parked at - `/worktank/comfyui/run/venv.broken-torch213-20260710`. **CORRECTED PATH (sent comfy-dev, thread 01KX655V):** - comfy-dev git-advances the ComfyUI checkout to 0.25.x + reqs → I handle the torch bump + SageAttention - rebuild-against-2.13.0 + re-pin (snapshot stays as the net). **LESSON: mmartial `DISABLE_UPGRADES` gates ONLY - the venv pip-upgrades, NOT the ComfyUI git checkout; a torch bump breaks compiled exts (SageAttention) → - rebuild-after is mandatory.** Bump BLOCKED pending comfy-dev's git-advance. Stack: A6000 (NVIDIA_VISIBLE_DEVICES=1), - lkraven-owned compose+venv (uid 1000, no sudo needed), COMFY_CMDLINE_EXTRA OOM flags preserved. diff --git a/persistent-memory.d/2026-07-10-comfyui-v0-27-1-success-on-irv-ml1.md b/persistent-memory.d/2026-07-10-comfyui-v0-27-1-success-on-irv-ml1.md deleted file mode 100644 index 4a77776..0000000 --- a/persistent-memory.d/2026-07-10-comfyui-v0-27-1-success-on-irv-ml1.md +++ /dev/null @@ -1,14 +0,0 @@ -- `[2026-07-10]` **ComfyUI v0.27.1 SUCCESS on irv-ml1 (operator-confirmed execute-now) — landed on torch 2.12.1, - SageAttention preserved, crash-loop AVOIDED.** The prior attempt (entry below) crash-looped because a torch - 2.12.1→2.13 bump broke SageAttention's ABI. This time I checked `git diff v0.24.1 v0.27.1 -- requirements.txt` - FIRST and found **core v0.27.1 leaves `torch` UNPINNED** → the version bump does NOT require torch 2.13 (that came - only from the mmartial boot-upgrade). So: `git checkout v0.27.1` (clean tree) → `pip install -r requirements.txt` - as **uid 1000** with a **torch-pin constraint file** (torch/vision/audio pinned to current +cu129) to block any - transitive bump → torch stayed 2.12.1, SageAttention 2.2.0 untouched. Added decord 0.6.0 (fixed SAM3Segment). - `docker restart comfyui` → healthy, `/system_stats` comfyui_version=0.27.1, "Using sage attention", HTTP 200, DB - migrated 0003→0004. Reported the divergence to comfy-dev (thread 01KX6D3C…, reply pending) + asked whether LTXV 2.3 - needs a separate torch-2.13 follow-up (their domain; Krea-2's ≥0.25 need is met by 0.27.1). **LESSON: before a - mmartial ComfyUI version bump, `git diff -- requirements.txt` — if torch is unpinned, bump the CODE - without touching torch (constraint-pin it) and compiled exts (SageAttention) survive. `docker exec` lands as uid - 1025(comfytoo), not 1000 — use `-u 1000` + the venv python `/comfy/mnt/venv/bin/python`.** See - [[reference_irv_ml1_comfyui_mmartial]]. diff --git a/persistent-memory.d/2026-07-10-heimdall-grant-ratatoskr-affect-full-on-personal-worldtree.md b/persistent-memory.d/2026-07-10-heimdall-grant-ratatoskr-affect-full-on-personal-worldtree.md deleted file mode 100644 index 9f0b553..0000000 --- a/persistent-memory.d/2026-07-10-heimdall-grant-ratatoskr-affect-full-on-personal-worldtree.md +++ /dev/null @@ -1,13 +0,0 @@ -- `[2026-07-10]` **Heimdall grant: ratatoskr `affect.full` on PERSONAL Worldtree (operator-approved, worldtree-dev - R34-v1 request).** Added allow-rule `ratatoskr-affect-full-allow` to `/opt/worldtree-personal/config/policies.yaml` - (`principal.user_ids:["ratatoskr"]`, action `affect.full`, resource `*`, effect allow), mirroring the #347 - `session-history-write-ratatoskr` rule exactly + placed right after it. **WHY user_ids-based (not tier):** ratatoskr's - personal key is the minimal **readonly-admin** observability tier, which is NOT in the tier-based - `affect-render-baseline-allow` (anonymous/user/free/pro/admin) → needs an explicit user_id grant, same as #347. - R34-v1 (b46, committed UNPUSHED) gates `affect.emit` `dominant_emotion` egress by exposure ceiling (affect.full|safe - → present; neither → null); this grant keeps ratatoskr's view alive across the b46 deploy. Surgical exact-string - insert (preserves comments), backup `policies.yaml.bak-pre-affectfull-20260710`, validated via the CONTAINER's own - yaml parser (35 rules, +1, payload confirmed). **NOT restarted — deliberate:** rule is on the bind-mount (shadows - baked), INERT until b46 gating ships, so the b46 CI/CD deploy restart activates it (no live-session blip now). Demo - untouched (personal-only per key scope). Replied to wt-dev (thread 01KX6DB3…) offering an immediate restart if they - want it live for pre-b46 testing. See [[reference_corviduo_dev_emergency_ops]]. diff --git a/persistent-memory.d/2026-07-13-355-residual-root-cause-supersedes-the-litellm-gateway.md b/persistent-memory.d/2026-07-13-355-residual-root-cause-supersedes-the-litellm-gateway.md deleted file mode 100644 index c51ff11..0000000 --- a/persistent-memory.d/2026-07-13-355-residual-root-cause-supersedes-the-litellm-gateway.md +++ /dev/null @@ -1 +0,0 @@ -- `[2026-07-13]` **#355-residual ROOT CAUSE (supersedes the "LiteLLM gateway holds while seat idles" entry below — that was DISPROVEN).** char-rp-reasoning enters a non-terminating REASONING loop (tool-call-retry planning) and runs to `max_tokens=32768` (~22 min @ 24.7 tok/s, ~13% of requests); the seat GENERATES all 32768 tokens (not idle), and `--reasoning-budget 400` is NOT enforced. 3-source-confirmed (spend_logs completion_tokens=32768 ×4; seat eval-time log; pcap 100%-`reasoning_content` deltas). Server-side fix wanted (operator: no max_tokens ceiling) → routed to brokkr (accepted, pulled dvalin). Lesson (again): confirm before concluding — the seat-idle claim came from reading only the ≤73s requests + missing the concurrent 32768-token slots. See ACTIVE 1. diff --git a/persistent-memory.d/2026-07-13-deploy-speed-real-bottleneck-uv-sync-memory-s.md b/persistent-memory.d/2026-07-13-deploy-speed-real-bottleneck-uv-sync-memory-s.md deleted file mode 100644 index c32ffcc..0000000 --- a/persistent-memory.d/2026-07-13-deploy-speed-real-bottleneck-uv-sync-memory-s.md +++ /dev/null @@ -1 +0,0 @@ -- `[2026-07-13]` **Deploy-speed real bottleneck ≠ uv sync (memory's assumption was wrong).** Buildx step log: `chown -R /app` = 251s (copy-up of the root-owned venv into a fresh layer), uv sync only 35.6s, registry layer cache already wired. Fix = drop `/app` from the chown (validated safe: zero /app runtime writes on both live instances) + uv cache-mount. Shipped as PR #359 (branch off origin/main@b60), worldtree-dev green-lit. Expected ~5min off (~11→~6min). Runner-side BuildKit cache task (b) was already done → moot. diff --git a/persistent-memory.d/2026-07-13-ledger-tier-3-consumer-ledger-miranda-provisioned-on.md b/persistent-memory.d/2026-07-13-ledger-tier-3-consumer-ledger-miranda-provisioned-on.md deleted file mode 100644 index 406ffe6..0000000 --- a/persistent-memory.d/2026-07-13-ledger-tier-3-consumer-ledger-miranda-provisioned-on.md +++ /dev/null @@ -1 +0,0 @@ -- `[2026-07-13]` Ledger tier-3 consumer `ledger:miranda` provisioned on personal :8081 (key b38932f5, GPG-delivered+shredded, allowlist 10.100.10.50:8770 live); `assistant`+`thoughtful-assistant` capability roles added (gen/gen-reasoning) on personal+demo, canonical d8bd497. Chosen instance = personal (the tier-3-consumer instance, ratatoskr+soong-lab colocated). diff --git a/persistent-memory.d/2026-07-13-worldtree-deploy-bottleneck-the-image-build-11-min.md b/persistent-memory.d/2026-07-13-worldtree-deploy-bottleneck-the-image-build-11-min.md deleted file mode 100644 index 3a1efd5..0000000 --- a/persistent-memory.d/2026-07-13-worldtree-deploy-bottleneck-the-image-build-11-min.md +++ /dev/null @@ -1 +0,0 @@ -- `[2026-07-13]` Worldtree deploy bottleneck = the image build (~11 min of a ~12 min deploy), root cause the Dockerfile `uv sync ... --no-cache` + no BuildKit cache-mount (re-downloads all deps cold every build). Fix split: worldtree-dev Dockerfile cache-mount diff + infra-ops runner-side persistent BuildKit cache. Config-only changes skip the build entirely (pinned recreate). diff --git a/persistent-memory.d/2026-07-13-wt-355-residual-300s-hang-localized-to-our.md b/persistent-memory.d/2026-07-13-wt-355-residual-300s-hang-localized-to-our.md deleted file mode 100644 index 6a75d9f..0000000 --- a/persistent-memory.d/2026-07-13-wt-355-residual-300s-hang-localized-to-our.md +++ /dev/null @@ -1 +0,0 @@ -- `[2026-07-13]` WT #355 residual 300s hang localized to OUR LiteLLM gateway (holds 2 char-rp-reasoning requests ~21 min while the seat idles), NOT the seat — Deckard seat EXONERATED (completes ≤72s; `--reasoning-budget 400` forecloses a mid-thinking hang). Corrects worldtree-dev's "seat wedging" diagnosis. Decisive next = the FIN-check (pcap on corviuo). See in-flight ACTIVE 1. **[SUPERSEDED 2026-07-13 — see the ROOT CAUSE entry above; the gateway-hold/slot-leak theory was disproven, the seat was generating 32768 tokens.]** diff --git a/persistent-memory.d/2026-07-13-wt-355-turn-lifecycle-fix-validated-on-worldtree.md b/persistent-memory.d/2026-07-13-wt-355-turn-lifecycle-fix-validated-on-worldtree.md deleted file mode 100644 index dd7d71a..0000000 --- a/persistent-memory.d/2026-07-13-wt-355-turn-lifecycle-fix-validated-on-worldtree.md +++ /dev/null @@ -1 +0,0 @@ -- `[2026-07-13]` WT #355 turn-lifecycle fix VALIDATED on worldtree b60 — wedged turns self-terminate cancelled/stalled at the 300s stall-watchdog (turns 2064/2065 vs pre-b60 2061's 16-min no-terminal). worldtree-dev filed follow-ons #356 (rehydrate Tier-3 ctx on resume — the recreate-durability gap), #357 (reclaim orphaned active-turn locks), #358 (LLM-provider read-timeout audit); surfacing to Vuong to prioritize. diff --git a/persistent-memory.d/2026-07-14-char-rp-reasoning-seat-deckard-pkd-neo-code.md b/persistent-memory.d/2026-07-14-char-rp-reasoning-seat-deckard-pkd-neo-code.md deleted file mode 100644 index 54d02d3..0000000 --- a/persistent-memory.d/2026-07-14-char-rp-reasoning-seat-deckard-pkd-neo-code.md +++ /dev/null @@ -1 +0,0 @@ -- `[2026-07-14]` **char-rp-reasoning seat: Deckard-PKD → NEO-CODE = Heretic2-Thinking (Qwen3.6-27B)** — R36 gate PASSED (tools 0.967, #355 runaway ELIMINATED). #355 was MODEL-level (Deckard emitted qwen3_coder XML malformed → mangled args → retry-runaway), NOT the reasoning-budget bug; NEO-CODE emits it clean. Custom llama.cpp KEPT (qwen3_coder parse — stock b8840 predates it — + PR#25544). Committed f960a73; full record auto-memory [[charrp-custom-llamacpp-pr25544]]. diff --git a/persistent-memory.d/2026-07-14-nvfp4-mtp-fast-char-rp-reasoning-seat-landed.md b/persistent-memory.d/2026-07-14-nvfp4-mtp-fast-char-rp-reasoning-seat-landed.md deleted file mode 100644 index 6f3ca43..0000000 --- a/persistent-memory.d/2026-07-14-nvfp4-mtp-fast-char-rp-reasoning-seat-landed.md +++ /dev/null @@ -1 +0,0 @@ -- `[2026-07-14]` **NVFP4+MTP fast char-rp-reasoning seat LANDED + LIVE + gateway-repointed + VRAM-tuned.** Modelopt-format re-quant made MTP work. The load-crash root cause = **vLLM 0.24 does NOT propagate modelopt `exclude_modules` to the spec-decode DRAFT model** → the bf16 mtp head gets quantized → shape crash; NO checkpoint config fixes it (`is_layer_skipped` is exact-membership, and the drafter never sees the mtp excludes) → **workaround = a mounted `sitecustomize` that force-skips `mtp.*` in `is_layer_skipped`** (upstream vLLM bug to file). Productionized as compose stack `heretic2-charrp-reasoning` (:8018, workaround baked in). Gateway `char-rp-reasoning` alias fixed: repointed off the stale GGUF served-name `deckard-pkd-27b`, added `enable_thinking:true`, **dropped `min_p`** (MTP-incompatible), canonical samplers temp1.0/top_p0.95/top_k20. Rebalanced GPU0 (gen 0.37→0.30/16-seq/256K + reasoning 0.39/16-seq/192K+MTP + char-rp 128K, 2.7GB free). All 4 gateway roles verified; vLLM reasoning-parser confirmed **leak-free** (unlike the GGUF budget-forcing). Full record + the 4 quant landmines in `docs/runbooks/heretic2-nvfp4-mtp-seat.md`; committed `982c319`. Open (non-blocking): brokkr P00 (seat is live ahead of it), retire the stopped GGUF reasoning seat, file the vLLM bug. diff --git a/persistent-memory.d/2026-07-14-nvfp4-quant-chase-resolved-gibberish-pivoted-to-modelopt.md b/persistent-memory.d/2026-07-14-nvfp4-quant-chase-resolved-gibberish-pivoted-to-modelopt.md deleted file mode 100644 index 5ac3ee2..0000000 --- a/persistent-memory.d/2026-07-14-nvfp4-quant-chase-resolved-gibberish-pivoted-to-modelopt.md +++ /dev/null @@ -1 +0,0 @@ -- `[2026-07-14]` **NVFP4 quant chase RESOLVED (gibberish) + PIVOTED to modelopt for MTP.** One ~40-min GPU0 window. Root-caused the `!!!!` to the quant NAMESPACE (text-only `AutoModelForCausalLM`→`model.layers.*` keys; vLLM serves only `Qwen3_5ForConditionalGeneration`, which needs `model.language_model.*`) — found from config diffs + vLLM source with ZERO GPU time; fixed by loading as `AutoModelForImageTextToText`. NVFP4 now serves COHERENT (validated greedy). BUT base NVFP4 ≈53 tok/s ≈ GGUF's 59.5 at batch-1 (no single-stream win) AND MTP = 0% acceptance on compressed-tensors (bf16 mtp head only loads on the modelopt format). Operator chose to **pursue a modelopt-format re-quant** (the only path to the 2-4× MTP goal; AEON-proven on this exact Qwen3.6-27B arch). Scoped + de-risked: AEON `/tank/aimodels/qwen36-27b-aeon-nvfp4` = the modelopt reference (quant_method modelopt, 1967 tensors, 15 bf16 mtp keys identical to graft); nvidia-modelopt 0.45.0 installs + `mtq.quantize`/`NVFP4_DEFAULT_CFG`/`export_hf_checkpoint` API confirmed; pipeline unchanged except swap llm-compressor→modelopt. Seats restored; char-rp-reasoning stays GGUF. Full plan in Current state ★ section. diff --git a/persistent-memory.d/2026-07-14-pursue-the-nvfp4-mtp-fast-char-rp-reasoning.md b/persistent-memory.d/2026-07-14-pursue-the-nvfp4-mtp-fast-char-rp-reasoning.md deleted file mode 100644 index ecd85e1..0000000 --- a/persistent-memory.d/2026-07-14-pursue-the-nvfp4-mtp-fast-char-rp-reasoning.md +++ /dev/null @@ -1 +0,0 @@ -- `[2026-07-14]` **Pursue the NVFP4+MTP fast char-rp-reasoning seat to completion** (Vuong-directed via /snapshot: "chase the nvfp4 quant, we know it works, write down the recipe"). Full recipe + diagnostic ladder in Current state / in-flight above. Artifacts on ana-ml2 `/tank/aimodels/heretic2-nvfp4-work/` + scripts committed in eshpfi `services/heretic2-nvfp4-quant/`. diff --git a/persistent-memory.d/2026-07-14-soong-lab-webhook-auto-deploy-real-root-cause.md b/persistent-memory.d/2026-07-14-soong-lab-webhook-auto-deploy-real-root-cause.md deleted file mode 100644 index 2113020..0000000 --- a/persistent-memory.d/2026-07-14-soong-lab-webhook-auto-deploy-real-root-cause.md +++ /dev/null @@ -1 +0,0 @@ -- `[2026-07-14]` **soong-lab webhook auto-deploy real root cause = gitea `webhook.ALLOWED_HOST_LIST`** (was `external, 10.100.0.0/16` = NH3-only; blocked corviduo-dev's Anaheim `10.250.x` → gitea refused to deliver, never opened the connection). Fixed to fleet-wide `10.0.0.0/8` (app.ini `[webhook]`) + gitea restart; listener now logs every delivery. The ufw `10/8` open (also this session) was a real-but-secondary gap. Committed 462d528. diff --git a/persistent-memory.d/2026-08-03-worldtree-b168-384-385-arc.md b/persistent-memory.d/2026-08-03-worldtree-b168-384-385-arc.md new file mode 100644 index 0000000..cfdcf55 --- /dev/null +++ b/persistent-memory.d/2026-08-03-worldtree-b168-384-385-arc.md @@ -0,0 +1,17 @@ +**Worldtree b168/#384/#385 arc — COMPLETE 2026-08-03.** A long peer-driven arc across worldtree-dev / muninn-dev / mimir-dev / ratatoskr-dev, all on corviduo-dev's demo+personal instances. Sequence: providers.yaml boot-gate pre-sync → b168 deploy → DCC #384 reindex → round-2 full re-ingest → #381 restart → operator-approved production dedup sweep. Landed clean; three of MY foot-guns along the way, each caught + hardened into a fleet runbook rule (see Tried-and-abandoned: `mv -t`, `docker exec -u 1000`, shared-containerd race). + +## providers.yaml pre-sync (boot-gating config) +b168 (commit `293f8f3`) added a `summarization` capability block that in-image `agents/muninn/config.yaml` references → boot-blocking if the host bind-mounted providers.yaml lacks it. Synced both hunks (summarization block + deep-reasoning desc) into demo+personal via `deploy-wt-config`; instance-configs commit `53349f8`. +- **deploy-wt-config runbook:** `~/development/worldtree-instance-configs/scripts/deploy-wt-config {diff|deploy|capture} --file providers.yaml` (per-instance dirs demo/personal/pinned; `deploy` = host write + api/matrix restart + 90s health-gate + auto-rollback; `diff`/`capture` safe). demo+personal providers.yaml are byte-identical. +- **GOTCHAS:** (1) an UNPUSHED source commit → `git show ` 404s and a gitea `raw?ref=` silently falls back to the default branch; verify the commit exists (`/git/commits/`) before trusting a fetch, else ask the peer to paste hunks. (2) a peer's hunk paste may be mis-indented (8-space vs the block's 4-space) → invalid YAML; always YAML-validate after a paste-sourced edit. +- **Config-delta pre-sync rule (verified via `docker inspect`):** worldtree containers bind-mount ONLY `config/` host-side (`/opt/worldtree-*/config/` → providers/model_roles/matrix/policies/defaults/env.public = the pre-syncable set); `agents/` (schemas.yaml, prompts) + all code ship IN-IMAGE. So only a `config/*.yaml` change is boot-blocking-pre-syncable; an `agents/`-or-code delta needs NO host pre-sync (CI carries it). b169's schemas.yaml (#387) was correctly no-pre-sync. + +## #384 reindex + #381 restart + verify +DCC job `mimir-6351554e8e8f`. Reindex: `sudo docker exec -u 1000 worldtree-personal-worldtree-muninn-1 python -m core.muninn --reindex ` (⚠️ MUST `-u 1000` — default-root writes contaminate the uid-1000 KB tree; see Tried-and-abandoned). Then **#381 restart** (stale-Chroma-client fix): `sudo docker restart worldtree-personal-worldtree-api-1` (plain bounce, NO compose up / no image repoint) → healthz/readyz 200 ~25s. +- **Chroma-verify runbook:** `sudo docker exec -i python -` (MUST pass `-i` or stdin never reaches `python -`) → `chromadb.PersistentClient('/data/kb/.chroma').get_collection('fiction').get(where={'job_id':}, include=['metadatas'])`. Chroma persists at container `/data/kb/.chroma` = host volume `worldtree-personal_worldtree-kb`. +- **Retrieval-visibility check (NOT grounding — that's ratatoskr's):** a Mimir session — admin token `~/.config/worldtree/personal-admin-token` (wildcard scope) → POST `/sessions` (agent_id=`mimir`, `record_tool_intermediates=true`) → POST `/sessions/{id}/messages` (STREAMS SSE, not JSON) → parse SSE `tool_result` for `search_library` wing hits → DELETE session. + +## Production dedup sweep (operator-approved) +Deleted the 785 April-era DCC orphan rows (`job_id=b59c147c5ce0`, no wing/source_identity metadata → predate identity tracking) from the `main` collection. Supervised protocol: read-only verify count == 785, back up all rows (ids+docs+embeddings) to `corviduo-dev:/tmp/main-sweep-backup-b59c147c5ce0.json` (reversible), `main.delete(where={job_id})` (assert target==785 first), verify `main` 4009→3224, then **bounce the api** (a separate-process delete leaves the api's in-memory HNSW index holding the vectors until reload — the #381 pattern generalizes to deletes), confirm search now fiction-only. Backup left for /tmp natural cleanup (fiction wing is canonical; `~/archives` has the historical record). + +Result: fiction wing 166 → 1,372 concepts; three consumer verify rounds 0/5 → 5/5 → saturated; #385 budget fix validated (705 vs April's 785 control, extraction AND indexing, zero truncations). worldtree-dev filed #388 for a deploy concurrency-lock (the shared-containerd race fix). See [[2026-08-02-mimir-inbox-arc]]. diff --git a/persistent-memory.md b/persistent-memory.md index 26699dd..130b38d 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -1,6 +1,6 @@ # Persistent memory — eshpfi-management -_Last updated: 2026-08-02_ +_Last updated: 2026-08-03_ > **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its > `Written:` stamp is under an hour old, read it (it carries the in-flight @@ -109,7 +109,7 @@ no longer deployed sidecars here. See Recent decisions.) ## Current state / in-flight -_As of 2026-08-02 — **session at a natural close; nothing infra-ops-side blocked.** A very long run 2026-08-01→02: the mimir-inbox / #377-read-path arc end to end (deploy + 4 bugs found/fixed/verified + a cloned voice). Full arc → `persistent-memory.d/2026-08-02-mimir-inbox-arc.md`; each sub-item also has a Recent-decisions entry. Headlines:_ +_As of 2026-08-03 — **session at a natural close; nothing infra-ops-side blocked.** Two long arcs landed: (1) the mimir-inbox / #377-read-path arc (2026-08-01→02); (2) the **worldtree b168/#384/#385 arc COMPLETE** (2026-08-03 — providers.yaml pre-sync → deploy → DCC+P&P re-ingest 705+667 concepts → #381 restart → operator-approved production dedup sweep; full detail in the 2026-08-03 Recent-decisions entry + `persistent-memory.d/2026-08-02-mimir-inbox-arc.md`). Only live watch = worldtree-dev re-running the personal **b169** deploy (all-clear given). Headlines:_ - **mimir-inbox DEPLOYED + verified on corviduo-dev `10.250.50.152:8091`** (#377 browser-facing half; write+read proven end to end). Live commit `8ece117` (3 redeploys); co-located per the operator's reversed-to-CO-LOCATE ruling. auto-memory `reference_mimir_inbox_deploy`. - **#377 read path fully working** — P&P ingested + queryable via Mimir on personal :8081. Chased through worldtree-dev bugs **#380** (wing-blind index → concepts in wrong collection; fixed b164 + one-shot `--reindex `), **#381** (stale Chroma client → **restart `worldtree-personal-worldtree-api-1` after any ingest/re-index** until their fix), **#382** (intermittent Mimir grounding / silent training-substitution; fixed b166 = `_index.md` per wing job-dir + a prompt rule; **verified 3/3** by ratatoskr-dev). DCC re-file SETTLED = **no** (prompt rule grounds it; `_index.md` rides next re-ingest). - **muninn-gate → muninn-dispatch 0.1.5** (rebuilt off `vh/muninn-gate` `bc04c4c`, image 0.0.14; serves `concept_schema`/`concept_schema_source`). BuildKit gitea secret; recreate w/ `compose up -d` not bare restart. @@ -122,11 +122,11 @@ _As of 2026-08-02 — **session at a natural close; nothing infra-ops-side block **Zonos voice stack (LIVE):** **9 voices** in `zonos-gateway` (`:8890` irv-ml1) — defaults AmericanFemale/Male/BritishFemale/Cora + 4 clones Emmie/Penny/Natalie/Miranda + **donut** (2026-08-02, from the 65-frost bundle, **expanded same day with onyx-58 clips → 52.0s multi-clip ref**); add a voice = drop `.wav` (44.1kHz mono s16 PCM) in `/opt/docker/compose/zonos-gateway/voices/` + `docker compose restart` (host-managed bind-mount, NO rebuild; registry scans at startup). Also mirror into the build-source tree `~/zonos-gateway/voices/` for rebuild-durability. Clone pipeline: `/mnt/smithy/voice_clones/.zip` → `assemble_voice.py` → drop. Dial-in studio http://10.100.10.50:8898/ (nohup on nh3-dev, relaunch `nohup python3 ~/development/zonos-tools/dial-in-studio.py >/tmp/zonos-studio.log 2>&1 &`). -**althing monitor** ARMED (handle `infra-ops`; herald up; wake-listener task-id rotates every re-arm). ⚠️ Re-arm ONLY after an actual FIRE (` completed rc0`), never after a plain operator turn (bounces rc3). Spawn `althing-wake-listener` as its OWN `run_in_background` task — NEVER chain with `&`/`&&` (orphans it → untracked → mail unwatched; I fumbled this ~4× this session, each time = stop-monitor + re-arm clean). **Open watches:** worldtree-dev may ping when their `/embed` code fix reaches PERSONAL (still 503 on b146 until a staging promotion — operator's call); #363 research-wing ingest PARKED; **#381 restart DONE (2026-08-02):** DCC job `mimir-6351554e8e8f` completed (145 sections/97 concepts, wing=fiction); restarted `worldtree-personal-worldtree-api-1` (plain restart, healthz/readyz 200 ~28s); confirmed retrieval-visibility via a Mimir session — `search_library` on `wings:["fiction"]` returned 10 DCC/Princess-Donut hits (fresh Chroma client sees the new writes). Scoped as RETRIEVAL, not grounding (thread `01KZ229N5Q2Y`, replied); grounding verify handed to ratatoskr-dev. **Runbook confirmed:** admin token (`~/.config/worldtree/personal-admin-token`, wildcard scope) → Mimir session (agent_id=`mimir`, `record_tool_intermediates=true`) → post message → read `/sessions/{id}/tool-events` + parse SSE `tool_result` → DELETE session. Message endpoint STREAMS SSE (not JSON). **providers.yaml pre-sync DONE (2026-08-02):** #384/#385 b168 (commit 293f8f3) needed a boot-gating `summarization` capability block (vastblueai-gateway/gen, non-thinking, temp 0.5, 8192, after `classification`) + a deep-reasoning description refresh. Synced BOTH hunks into demo + personal `providers.yaml` via `deploy-wt-config` (health-gated, both api healthy ~18-21s); instance-configs repo commit `53349f8`; replied GO (`01KZ2NMER3EP78X95CMZGP76BK`) → they push main (demo auto-deploy) + cut personal staging tag. **GOTCHAS for next time:** (1) the source commit was UNPUSHED at sync time — `git show ` 404s and a gitea `raw?ref=` silently falls back to the default branch; verify the commit exists before trusting a fetch, else ask the peer to paste hunks. (2) worldtree-dev's hunk paste indented the deep-reasoning `description:` at 8 spaces but the block keys are 4-space → invalid YAML; corrected to 4-space (always YAML-validate after a paste-sourced edit). **deploy-wt-config runbook:** `diff`/`deploy --file providers.yaml` (repo `~/development/worldtree-instance-configs`, per-instance dirs demo/personal/pinned; deploy = host write + api/matrix restart + 90s health-gate + auto-rollback); demo+personal providers.yaml are byte-identical. See [[reference_worldtree_instance_configs_repo]]. **Config-delta pre-sync rule (verified via `docker inspect` 2026-08-03):** the worldtree containers bind-mount ONLY `config/` host-side (`/opt/worldtree-*/config/` → holds providers/model_roles/matrix/policies/defaults/env.public — the pre-syncable set); `agents/` (schemas.yaml, prompts) + all code ship IN-IMAGE. So a config-delta touching `agents/` or code needs NO host pre-sync (CI deploy carries it); only a `config/*.yaml` change is the boot-blocking-pre-sync case (as b168's summarization block was; b169's schemas.yaml was NOT). b168 PUSHED to gitea (main 8920adf..293f8f3 + tags v1.0.0b168 / staging/v1.0.0b168); demo auto-deploys via main, personal via staging tag; worldtree-dev verified my host 4-space == commit byte-for-byte. **#384 recovery on personal DONE (2026-08-02):** b168 (`293f8f3c6077`) confirmed live on all personal containers (preflight — checked `.Config.Image` before executing) → reindex `sudo docker exec worldtree-personal-worldtree-muninn-1 python -m core.muninn --reindex mimir-6351554e8e8f` (exit 0, `indexed=97 skipped=0`) → #381 restart (healthz/readyz 200) → metadata verify 5/5 fiction rows carry non-empty note_id+path (Tier-3 backfill confirmed). Reported `01KZ2PEGZ0DVQSJN4PWMGRD198`; ratatoskr-dev runs first-light re-verify. **⚠️ OPEN (worldtree-dev's call):** reindex logged `Wing git commit failed … BrokenPipeError (reconcile-visible)` / `kb_ingest_degraded phase=indexing` — Chroma rows + on-disk concept/note files landed, only the wing git-versioning step degraded. DISPOSITION (worldtree-dev 2026-08-02): WATCH-ITEM, NO reconcile — the DCC round-2 re-run (via mimir-inbox, post-verify) re-commits the same wing dir and self-heals the missed commit; escalate ONLY if round-2's wing commit also fails (two-in-a-row). Retrieval/provenance unaffected. → **CLOSED (worldtree-dev 2026-08-03):** round-2's wing commits landed clean (verified `exec -u 1000`); wing HEAD = "Muninn ingest: mimir-6351554e8e8f" with P&P beneath; my loose-object chown almost certainly enabled the commit. Self-healed as predicted — watch-item retired. **Chroma-verify runbook:** `sudo docker exec -i python -` (MUST pass `-i` or stdin never reaches `python -`) → `chromadb.PersistentClient('/data/kb/.chroma').get_collection('fiction').get(where={'job_id':}, include=['metadatas'])`; fiction count 166 (69 P&P + 97 DCC). **PENDING production delete-sweep (heads-up 2026-08-02):** worldtree-dev will request (as a request to me / a supervised script, sequenced AFTER ratatoskr's verify) an explicit delete-by-job_id of the **785 April-era DCC orphan rows** in the **'main'** collection (`job_id=b59c147c5ce0`, no source_identity metadata → predate identity tracking, so supersession can't reach them; the #384 stray-row sweep removed `source_identity=76cb5829035553bf`, NOT these). This is a PRODUCTION-COLLECTION MASS DELETE → handle supervised: confirm exact job_id + expected count, snapshot/back up the collection first, and loop the operator BEFORE executing. **DONE 2026-08-03 (operator-approved "go"):** read-only verified 785 (main 4009→3224, target confirmed wing=None/source_identity=None April orphans), backed up all 785 rows (ids+docs+1024-dim embeddings, 18.6MB, reload-verified) → `corviduo-dev:/tmp/main-sweep-backup-b59c147c5ce0.json` (reversible), then `main.delete(where={job_id:b59c147c5ce0})` (asserted target==785 first), verified main==3224 + target cleared to 0, bounced the api (in-memory HNSW reload — a separate-process delete leaves the api's index holding the vectors until restart, the #381 pattern), confirmed search now fiction-only ({fiction:10}, was {main:5,fiction:5}). **#384/#385 CLOSED, worldtree arc COMPLETE.** Backup stays on corviduo-dev:/tmp until worldtree-dev clears it. +**althing monitor** ARMED (handle `infra-ops`; herald up; wake-listener task-id rotates every re-arm). ⚠️ Re-arm ONLY after an actual FIRE (` completed rc0`), never after a plain operator turn (bounces rc3). Spawn `althing-wake-listener` as its OWN `run_in_background` task — NEVER chain with `&`/`&&` (orphans it → untracked → mail unwatched). **ACTIVE WATCH — b169 personal deploy:** worldtree-dev re-running the staging/v1.0.0b169 personal deploy after a pull-fail I diagnosed as a transient shared-containerd concurrent-pull race (NOT disk); all-clear given = just re-run, **do NOT prune** (`6e34a87` is in-use by the running demo instance — pruning would down demo; see Tried-and-abandoned). worldtree-dev verifies health + filed #388 for a deploy concurrency-lock. **PARKED watches:** #363 research-wing ingest (no deadline); worldtree-dev's `/embed` fix reaching PERSONAL (503 on b146 until a staging promotion — operator's call). The **worldtree b168/#384/#385 arc is COMPLETE** — runbooks + full detail in the 2026-08-03 Recent-decisions entry. **Two small pending items (operator's call, non-urgent):** (1) bless/reshape the `env.public` non-secret-env-overlay mechanism in the config repo; (2) the pre-existing herald pane-route errors on `worldtree-codex` + `eitri-smithy-dev` ("route-error: list index out of range" — likely `render_command messages[0]` on empty list; NOT infra-ops's, rec = flag to althing-dev). -**eshpfi has UNPUSHED local commits** — `main` is ~8 ahead of `origin/main` (tonight's mimir-inbox + muninn-gate doc-syncs + Audio8/#383 notes + this snapshot, and the 7-31 muninn-gate/snapshot). **Push is the operator's call.** `stacks/heretic2-charrp-reasoning/` UNTRACKED (pre-existing, operator's); `graphify-out/GRAPH_REPORT.md` = graphify-hook artifact (churns on every commit, ignore). +**eshpfi has UNPUSHED local commits** — `main` is **~21 ahead of `origin/main`** (mostly 2026-08-02→03 memory commits from the worldtree arc + this snapshot; plus the earlier mimir-inbox/muninn-gate/Audio8/#383 work). **Push is the operator's call.** `stacks/heretic2-charrp-reasoning/` UNTRACKED (pre-existing, operator's); `graphify-out/GRAPH_REPORT.md` = graphify-hook artifact (churns on every commit, ignore, don't stage). **PARKED (grok-code/Codex):** operator asked about fronting grok-code / Codex behind the LiteLLM gateway. Rec (given): raw models behind the gateway → **API keys** (native `xai/` + `openai/` providers, the GLM-passthrough pattern); fleet *consults* → the **Heid/Eitri peer-CLI** pattern (Codex already wired). Do NOT reverse-proxy the subscription CLIs (grok CLI / Codex CLI, OAuth-auth) into the gateway — ToS + account-ban risk + brittle. Untracked by operator choice; no decision made. @@ -134,6 +134,8 @@ _As of 2026-08-02 — **session at a natural close; nothing infra-ops-side block ## Recent decisions +- `[2026-08-03]` **worldtree b168/#384/#385 arc COMPLETE** — providers.yaml boot-gate pre-sync → b168 deploy → DCC+P&P re-ingest (705+667 concepts, 0 truncations, #385 budget fix validated vs April's 785 control) → #381 restart → operator-approved production dedup sweep (785 April orphans deleted from `main`, 4009→3224). Fiction wing 166→1,372 concepts; consumer verify 0/5→5/5→saturated. Three of MY foot-guns hardened into fleet runbook rules (`mv -t`, `docker exec -u 1000`, shared-containerd pull-race — see Tried-and-abandoned). Full runbooks (deploy-wt-config, Chroma-verify, config-delta pre-sync rule, #381, sweep) → `persistent-memory.d/2026-08-03-worldtree-b168-384-385-arc.md` + - `[2026-08-02]` **mimir-inbox / #377-read-path arc — deployed + 4 bugs found/fixed/verified + a cloned voice.** mimir-inbox live on corviduo-dev:8091 (#377 write+read proven, live `8ece117`); worldtree-dev #380 (wing-blind index) + #381 (stale-client restart) + #382 (intermittent Mimir grounding) chased and **verified 3/3** by ratatoskr-dev; muninn-gate → dispatch 0.1.5; **donut** voice cloned from the 65-frost Booth bundle into the Zonos gateway; Zonos streaming confirmed already-working. Full arc, procedures, and lessons → `persistent-memory.d/2026-08-02-mimir-inbox-arc.md` - `[2026-07-31]` **muninn-gate (#377 ingestion front door) BUILT + DEPLOYED + healthy on corviduo-dev:8090.** First-boot acceptance passed (watcher:running:true proves ingestion_root byte-identity); submit path deferred to the mimir-inbox era. Full wiring (uid-1000, state-volume mount, staging path-agreement, BuildKit-secret build, deferred repoint + operational guards) → `persistent-memory.d/2026-07-31-muninn-gate-deploy.md` @@ -218,47 +220,8 @@ _As of 2026-08-02 — **session at a natural close; nothing infra-ops-side block - `[2026-07-15]` **Statusline overhauled** (`~/.claude/statusline-command.sh`) — git state / 🔔🔕 monitor-armed / project tag / abs tokens / per-session cost (`.cost.total_cost_usd`) / threshold-colored ctx+rate (green<60 / yellow60-90 / red>90). -- `[2026-07-14]` NVFP4+MTP fast char-rp-reasoning seat LANDED + LIVE + gateway-repointed + VRAM-tuned → `persistent-memory.d/2026-07-14-nvfp4-mtp-fast-char-rp-reasoning-seat-landed.md` -- `[2026-07-14]` NVFP4 quant chase RESOLVED (gibberish) + PIVOTED to modelopt for MTP → `persistent-memory.d/2026-07-14-nvfp4-quant-chase-resolved-gibberish-pivoted-to-modelopt.md` - -- `[2026-07-14]` Pursue the NVFP4+MTP fast char-rp-reasoning seat to completion → `persistent-memory.d/2026-07-14-pursue-the-nvfp4-mtp-fast-char-rp-reasoning.md` - -- `[2026-07-14]` char-rp-reasoning seat: Deckard-PKD → NEO-CODE = Heretic2-Thinking (Qwen3.6-27B) → `persistent-memory.d/2026-07-14-char-rp-reasoning-seat-deckard-pkd-neo-code.md` - -- `[2026-07-14]` soong-lab webhook auto-deploy real root cause = gitea `webhook.ALLOWED_HOST_LIST` → `persistent-memory.d/2026-07-14-soong-lab-webhook-auto-deploy-real-root-cause.md` - -- `[2026-07-13]` #355-residual ROOT CAUSE (supersedes the "LiteLLM gateway holds while seat idles" entry below — that was DISPROVEN) → `persistent-memory.d/2026-07-13-355-residual-root-cause-supersedes-the-litellm-gateway.md` - -- `[2026-07-13]` Deploy-speed real bottleneck ≠ uv sync (memory's assumption was wrong) → `persistent-memory.d/2026-07-13-deploy-speed-real-bottleneck-uv-sync-memory-s.md` - -- `[2026-07-13]` WT #355 residual 300s hang localized to OUR LiteLLM gateway (holds 2 char-rp-reasoning requests ~21 min while the seat idles), NOT the seat — Deckard seat EXONE → `persistent-memory.d/2026-07-13-wt-355-residual-300s-hang-localized-to-our.md` - -- `[2026-07-13]` WT #355 turn-lifecycle fix VALIDATED on worldtree b60 — wedged turns self-terminate cancelled/stalled at the 300s stall-watchdog (turns 2064/2065 vs pre-b60 206 → `persistent-memory.d/2026-07-13-wt-355-turn-lifecycle-fix-validated-on-worldtree.md` - -- `[2026-07-13]` Worldtree deploy bottleneck = the image build (~11 min of a ~12 min deploy), root cause the Dockerfile `uv sync → `persistent-memory.d/2026-07-13-worldtree-deploy-bottleneck-the-image-build-11-min.md` - -- `[2026-07-13]` Ledger tier-3 consumer `ledger:miranda` provisioned on personal :8081 (key b38932f5, GPG-delivered+shredded, allowlist 10.100.10.50:8770 live); → `persistent-memory.d/2026-07-13-ledger-tier-3-consumer-ledger-miranda-provisioned-on.md` - -- `[2026-07-10]` Heimdall grant: ratatoskr `affect.full` on PERSONAL Worldtree (operator-approved, worldtree-dev R34-v1 request) → `persistent-memory.d/2026-07-10-heimdall-grant-ratatoskr-affect-full-on-personal-worldtree.md` - -- `[2026-07-10]` ComfyUI v0.27.1 SUCCESS on irv-ml1 (operator-confirmed execute-now) — landed on torch 2.12.1, SageAttention preserved, crash-loop AVOIDED → `persistent-memory.d/2026-07-10-comfyui-v0-27-1-success-on-irv-ml1.md` - -- `[2026-07-10]` ComfyUI 0.25.x bump on irv-ml1 ATTEMPTED → FAILED → ROLLED BACK (snapshot saved it) → `persistent-memory.d/2026-07-10-comfyui-0-25-x-bump-on-irv-ml1.md` - -- `[2026-07-10]` Biweekly open-weight-releases scan cron set up for brokkr-smithy (Vuong-authorized) → `persistent-memory.d/2026-07-10-biweekly-open-weight-releases-scan-cron-set-up.md` - -- `[2026-07-08]` RP-SEAT CAMPAIGN CLOSED — char-rp = Magidonia-24B-v4.3 (128K), char-rp-reasoning = Deckard-PKD Qwen3.5-27B (256K); both GGUF/llama.cpp on ana-ml2 GPU0 alongside gen (3… → `persistent-memory.d/2026-07-08-rp-seat-campaign-closed-char-rp-magidonia-24b.md` - -- `[2026-07-08]` OFF-THE-SHELF INFERENCE PIVOT executed — serve curated abliterated models, stop home-training → `persistent-memory.d/2026-07-08-off-the-shelf-inference-pivot-executed-serve-curated.md` - -- `[2026-07-08]` DPO was silently running 3 epochs (harness gap) → KILLED at epoch 1.2, retargeted to 0.3 epochs (operator call) → `persistent-memory.d/2026-07-08-dpo-was-silently-running-3-epochs-harness-gap.md` - -- `[2026-07-08]` T1 DPO leg is RUNNING (unblocked) — 2 fixes applied to deployed backend.py → `persistent-memory.d/2026-07-08-t1-dpo-leg-is-running-unblocked-2-fixes.md` - -- `[2026-07-08]` T1 DPO leg launch — prior BLOCK (now resolved above), kept for the launch recipe → `persistent-memory.d/2026-07-08-t1-dpo-leg-launch-prior-block-now-resolved.md` - -_147 older entries archived to archival-memory.md._ +_167 older entries archived to archival-memory.md._ ## Tried and abandoned