memory: snapshot for /clear — worldtree b168/#384/#385 arc complete

Collapsed the completed-arc bloat out of Current state / in-flight (line 125:
6487->1073 chars; runbooks moved to persistent-memory.d/2026-08-03-worldtree-b168-384-385-arc.md).
Added the 2026-08-03 arc Recent-decisions entry. Auto-archived 20 oldest
Recent-decisions entries (2026-07-08..07-14) to archival-memory.md (index 345->308
lines). Refreshed the b169-deploy watch, unpushed-count, and handoff.

archival-memory.md

@@ -1482,3 +1482,247 @@ _Archived 2026-08-02._
 
 _Archived 2026-08-02._
 
+
+## Recent decisions (archived 2026-08-03 batch)
+
+### 2026-07-14-nvfp4-mtp-fast-char-rp-reasoning-seat-landed
+
+- `[2026-07-14]` **NVFP4+MTP fast char-rp-reasoning seat LANDED + LIVE + gateway-repointed + VRAM-tuned.** Modelopt-format re-quant made MTP work. The load-crash root cause = **vLLM 0.24 does NOT propagate modelopt `exclude_modules` to the spec-decode DRAFT model** → the bf16 mtp head gets quantized → shape crash; NO checkpoint config fixes it (`is_layer_skipped` is exact-membership, and the drafter never sees the mtp excludes) → **workaround = a mounted `sitecustomize` that force-skips `mtp.*` in `is_layer_skipped`** (upstream vLLM bug to file). Productionized as compose stack `heretic2-charrp-reasoning` (:8018, workaround baked in). Gateway `char-rp-reasoning` alias fixed: repointed off the stale GGUF served-name `deckard-pkd-27b`, added `enable_thinking:true`, **dropped `min_p`** (MTP-incompatible), canonical samplers temp1.0/top_p0.95/top_k20. Rebalanced GPU0 (gen 0.37→0.30/16-seq/256K + reasoning 0.39/16-seq/192K+MTP + char-rp 128K, 2.7GB free). All 4 gateway roles verified; vLLM reasoning-parser confirmed **leak-free** (unlike the GGUF budget-forcing). Full record + the 4 quant landmines in `docs/runbooks/heretic2-nvfp4-mtp-seat.md`; committed `982c319`. Open (non-blocking): brokkr P00 (seat is live ahead of it), retire the stopped GGUF reasoning seat, file the vLLM bug.
+
+_Archived 2026-08-03._
+
+### 2026-07-14-nvfp4-quant-chase-resolved-gibberish-pivoted-to-modelopt
+
+- `[2026-07-14]` **NVFP4 quant chase RESOLVED (gibberish) + PIVOTED to modelopt for MTP.** One ~40-min GPU0 window. Root-caused the `!!!!` to the quant NAMESPACE (text-only `AutoModelForCausalLM`→`model.layers.*` keys; vLLM serves only `Qwen3_5ForConditionalGeneration`, which needs `model.language_model.*`) — found from config diffs + vLLM source with ZERO GPU time; fixed by loading as `AutoModelForImageTextToText`. NVFP4 now serves COHERENT (validated greedy). BUT base NVFP4 ≈53 tok/s ≈ GGUF's 59.5 at batch-1 (no single-stream win) AND MTP = 0% acceptance on compressed-tensors (bf16 mtp head only loads on the modelopt format). Operator chose to **pursue a modelopt-format re-quant** (the only path to the 2-4× MTP goal; AEON-proven on this exact Qwen3.6-27B arch). Scoped + de-risked: AEON `/tank/aimodels/qwen36-27b-aeon-nvfp4` = the modelopt reference (quant_method modelopt, 1967 tensors, 15 bf16 mtp keys identical to graft); nvidia-modelopt 0.45.0 installs + `mtq.quantize`/`NVFP4_DEFAULT_CFG`/`export_hf_checkpoint` API confirmed; pipeline unchanged except swap llm-compressor→modelopt. Seats restored; char-rp-reasoning stays GGUF. Full plan in Current state ★ section.
+
+_Archived 2026-08-03._
+
+### 2026-07-14-pursue-the-nvfp4-mtp-fast-char-rp-reasoning
+
+- `[2026-07-14]` **Pursue the NVFP4+MTP fast char-rp-reasoning seat to completion** (Vuong-directed via /snapshot: "chase the nvfp4 quant, we know it works, write down the recipe"). Full recipe + diagnostic ladder in Current state / in-flight above. Artifacts on ana-ml2 `/tank/aimodels/heretic2-nvfp4-work/` + scripts committed in eshpfi `services/heretic2-nvfp4-quant/`.
+
+_Archived 2026-08-03._
+
+### 2026-07-14-char-rp-reasoning-seat-deckard-pkd-neo-code
+
+- `[2026-07-14]` **char-rp-reasoning seat: Deckard-PKD → NEO-CODE = Heretic2-Thinking (Qwen3.6-27B)** — R36 gate PASSED (tools 0.967, #355 runaway ELIMINATED). #355 was MODEL-level (Deckard emitted qwen3_coder XML malformed → mangled args → retry-runaway), NOT the reasoning-budget bug; NEO-CODE emits it clean. Custom llama.cpp KEPT (qwen3_coder parse — stock b8840 predates it — + PR#25544). Committed f960a73; full record auto-memory [[charrp-custom-llamacpp-pr25544]].
+
+_Archived 2026-08-03._
+
+### 2026-07-14-soong-lab-webhook-auto-deploy-real-root-cause
+
+- `[2026-07-14]` **soong-lab webhook auto-deploy real root cause = gitea `webhook.ALLOWED_HOST_LIST`** (was `external, 10.100.0.0/16` = NH3-only; blocked corviduo-dev's Anaheim `10.250.x` → gitea refused to deliver, never opened the connection). Fixed to fleet-wide `10.0.0.0/8` (app.ini `[webhook]`) + gitea restart; listener now logs every delivery. The ufw `10/8` open (also this session) was a real-but-secondary gap. Committed 462d528.
+
+_Archived 2026-08-03._
+
+### 2026-07-13-355-residual-root-cause-supersedes-the-litellm-gateway
+
+- `[2026-07-13]` **#355-residual ROOT CAUSE (supersedes the "LiteLLM gateway holds while seat idles" entry below — that was DISPROVEN).** char-rp-reasoning enters a non-terminating REASONING loop (tool-call-retry planning) and runs to `max_tokens=32768` (~22 min @ 24.7 tok/s, ~13% of requests); the seat GENERATES all 32768 tokens (not idle), and `--reasoning-budget 400` is NOT enforced. 3-source-confirmed (spend_logs completion_tokens=32768 ×4; seat eval-time log; pcap 100%-`reasoning_content` deltas). Server-side fix wanted (operator: no max_tokens ceiling) → routed to brokkr (accepted, pulled dvalin). Lesson (again): confirm before concluding — the seat-idle claim came from reading only the ≤73s requests + missing the concurrent 32768-token slots. See ACTIVE 1.
+
+_Archived 2026-08-03._
+
+### 2026-07-13-deploy-speed-real-bottleneck-uv-sync-memory-s
+
+- `[2026-07-13]` **Deploy-speed real bottleneck ≠ uv sync (memory's assumption was wrong).** Buildx step log: `chown -R /app` = 251s (copy-up of the root-owned venv into a fresh layer), uv sync only 35.6s, registry layer cache already wired. Fix = drop `/app` from the chown (validated safe: zero /app runtime writes on both live instances) + uv cache-mount. Shipped as PR #359 (branch off origin/main@b60), worldtree-dev green-lit. Expected ~5min off (~11→~6min). Runner-side BuildKit cache task (b) was already done → moot.
+
+_Archived 2026-08-03._
+
+### 2026-07-13-wt-355-residual-300s-hang-localized-to-our
+
+- `[2026-07-13]` WT #355 residual 300s hang localized to OUR LiteLLM gateway (holds 2 char-rp-reasoning requests ~21 min while the seat idles), NOT the seat — Deckard seat EXONERATED (completes ≤72s; `--reasoning-budget 400` forecloses a mid-thinking hang). Corrects worldtree-dev's "seat wedging" diagnosis. Decisive next = the FIN-check (pcap on corviuo). See in-flight ACTIVE 1. **[SUPERSEDED 2026-07-13 — see the ROOT CAUSE entry above; the gateway-hold/slot-leak theory was disproven, the seat was generating 32768 tokens.]**
+
+_Archived 2026-08-03._
+
+### 2026-07-13-wt-355-turn-lifecycle-fix-validated-on-worldtree
+
+- `[2026-07-13]` WT #355 turn-lifecycle fix VALIDATED on worldtree b60 — wedged turns self-terminate cancelled/stalled at the 300s stall-watchdog (turns 2064/2065 vs pre-b60 2061's 16-min no-terminal). worldtree-dev filed follow-ons #356 (rehydrate Tier-3 ctx on resume — the recreate-durability gap), #357 (reclaim orphaned active-turn locks), #358 (LLM-provider read-timeout audit); surfacing to Vuong to prioritize.
+
+_Archived 2026-08-03._
+
+### 2026-07-13-worldtree-deploy-bottleneck-the-image-build-11-min
+
+- `[2026-07-13]` Worldtree deploy bottleneck = the image build (~11 min of a ~12 min deploy), root cause the Dockerfile `uv sync ... --no-cache` + no BuildKit cache-mount (re-downloads all deps cold every build). Fix split: worldtree-dev Dockerfile cache-mount diff + infra-ops runner-side persistent BuildKit cache. Config-only changes skip the build entirely (pinned recreate).
+
+_Archived 2026-08-03._
+
+### 2026-07-13-ledger-tier-3-consumer-ledger-miranda-provisioned-on
+
+- `[2026-07-13]` Ledger tier-3 consumer `ledger:miranda` provisioned on personal :8081 (key b38932f5, GPG-delivered+shredded, allowlist 10.100.10.50:8770 live); `assistant`+`thoughtful-assistant` capability roles added (gen/gen-reasoning) on personal+demo, canonical d8bd497. Chosen instance = personal (the tier-3-consumer instance, ratatoskr+soong-lab colocated).
+
+_Archived 2026-08-03._
+
+### 2026-07-10-heimdall-grant-ratatoskr-affect-full-on-personal-worldtree
+
+- `[2026-07-10]` **Heimdall grant: ratatoskr `affect.full` on PERSONAL Worldtree (operator-approved, worldtree-dev
+  R34-v1 request).** Added allow-rule `ratatoskr-affect-full-allow` to `/opt/worldtree-personal/config/policies.yaml`
+  (`principal.user_ids:["ratatoskr"]`, action `affect.full`, resource `*`, effect allow), mirroring the #347
+  `session-history-write-ratatoskr` rule exactly + placed right after it. **WHY user_ids-based (not tier):** ratatoskr's
+  personal key is the minimal **readonly-admin** observability tier, which is NOT in the tier-based
+  `affect-render-baseline-allow` (anonymous/user/free/pro/admin) → needs an explicit user_id grant, same as #347.
+  R34-v1 (b46, committed UNPUSHED) gates `affect.emit` `dominant_emotion` egress by exposure ceiling (affect.full|safe
+  → present; neither → null); this grant keeps ratatoskr's view alive across the b46 deploy. Surgical exact-string
+  insert (preserves comments), backup `policies.yaml.bak-pre-affectfull-20260710`, validated via the CONTAINER's own
+  yaml parser (35 rules, +1, payload confirmed). **NOT restarted — deliberate:** rule is on the bind-mount (shadows
+  baked), INERT until b46 gating ships, so the b46 CI/CD deploy restart activates it (no live-session blip now). Demo
+  untouched (personal-only per key scope). Replied to wt-dev (thread 01KX6DB3…) offering an immediate restart if they
+  want it live for pre-b46 testing. See [[reference_corviduo_dev_emergency_ops]].
+
+_Archived 2026-08-03._
+
+### 2026-07-10-comfyui-v0-27-1-success-on-irv-ml1
+
+- `[2026-07-10]` **ComfyUI v0.27.1 SUCCESS on irv-ml1 (operator-confirmed execute-now) — landed on torch 2.12.1,
+  SageAttention preserved, crash-loop AVOIDED.** The prior attempt (entry below) crash-looped because a torch
+  2.12.1→2.13 bump broke SageAttention's ABI. This time I checked `git diff v0.24.1 v0.27.1 -- requirements.txt`
+  FIRST and found **core v0.27.1 leaves `torch` UNPINNED** → the version bump does NOT require torch 2.13 (that came
+  only from the mmartial boot-upgrade). So: `git checkout v0.27.1` (clean tree) → `pip install -r requirements.txt`
+  as **uid 1000** with a **torch-pin constraint file** (torch/vision/audio pinned to current +cu129) to block any
+  transitive bump → torch stayed 2.12.1, SageAttention 2.2.0 untouched. Added decord 0.6.0 (fixed SAM3Segment).
+  `docker restart comfyui` → healthy, `/system_stats` comfyui_version=0.27.1, "Using sage attention", HTTP 200, DB
+  migrated 0003→0004. Reported the divergence to comfy-dev (thread 01KX6D3C…, reply pending) + asked whether LTXV 2.3
+  needs a separate torch-2.13 follow-up (their domain; Krea-2's ≥0.25 need is met by 0.27.1). **LESSON: before a
+  mmartial ComfyUI version bump, `git diff <old> <new> -- requirements.txt` — if torch is unpinned, bump the CODE
+  without touching torch (constraint-pin it) and compiled exts (SageAttention) survive. `docker exec` lands as uid
+  1025(comfytoo), not 1000 — use `-u 1000` + the venv python `/comfy/mnt/venv/bin/python`.** See
+  [[reference_irv_ml1_comfyui_mmartial]].
+
+_Archived 2026-08-03._
+
+### 2026-07-10-comfyui-0-25-x-bump-on-irv-ml1
+
+- `[2026-07-10]` **ComfyUI 0.25.x bump on irv-ml1 ATTEMPTED → FAILED → ROLLED BACK (snapshot saved it).** comfy-dev
+  requested (Vuong-authorized) bumping the irv-ml1 `comfyui` stack (mmartial image, `/opt/docker/compose/comfyui/`,
+  0.24.1) to 0.25.x for Krea-2 + LTXV 2.3. **TWO FINDINGS: (1) `DISABLE_UPGRADES=false`/USE_PIPUPGRADE bumps the
+  VENV (torch 2.12.1→2.13.0 + deps) but does NOT advance the ComfyUI CODE checkout** (`/comfy/mnt/ComfyUI` =
+  `/worktank/comfyui/run/ComfyUI` stayed 0.24.1 — pinned/detached git, comfy-dev's domain). **(2) the torch bump
+  broke SageAttention** (2.2.0 `_fused.so` undefined-symbol `c10::impl::cow::materialize_cow_storage` vs torch
+  2.13.0) → `--use-sage-attention` (REQUIRED launch flag in COMFY_CMDLINE_EXTRA) crash-looped ComfyUI. Net: broke
+  the working state, zero 0.25.x payoff. **ROLLBACK WORKED**: pre-bump 16G venv snapshot
+  `/worktank/comfyui/venv-snapshot-comfyui-0.24.1-20260710.tar` restored (torch 2.12.1 + working SageAttention),
+  re-pinned DISABLE_UPGRADES=true, recreated → healthy on 0.24.1, serving :8188. Broken venv parked at
+  `/worktank/comfyui/run/venv.broken-torch213-20260710`. **CORRECTED PATH (sent comfy-dev, thread 01KX655V):**
+  comfy-dev git-advances the ComfyUI checkout to 0.25.x + reqs → I handle the torch bump + SageAttention
+  rebuild-against-2.13.0 + re-pin (snapshot stays as the net). **LESSON: mmartial `DISABLE_UPGRADES` gates ONLY
+  the venv pip-upgrades, NOT the ComfyUI git checkout; a torch bump breaks compiled exts (SageAttention) →
+  rebuild-after is mandatory.** Bump BLOCKED pending comfy-dev's git-advance. Stack: A6000 (NVIDIA_VISIBLE_DEVICES=1),
+  lkraven-owned compose+venv (uid 1000, no sudo needed), COMFY_CMDLINE_EXTRA OOM flags preserved.
+
+_Archived 2026-08-03._
+
+### 2026-07-10-biweekly-open-weight-releases-scan-cron-set-up
+
+- `[2026-07-10]` **Biweekly open-weight-releases scan cron set up for brokkr-smithy (Vuong-authorized).** Durable
+  systemd **--user** timer on nh3-dev (`brokkr-landscape-scan.timer`, OnCalendar `*-*-01,15 09:00:00`
+  America/Los_Angeles, Persistent=true; linger on) → `.service` → wrapper `~/.local/bin/brokkr-landscape-scan.sh`
+  runs headless `claude -p "$(cat ~/.config/brokkr-landscape-scan/prompt.txt)" --dangerously-skip-permissions` in
+  `~/development/brokkr-smithy` (ALTHING_HANDLE=brokkr-smithy-dev; **explicit PATH** — the --user minimal-PATH
+  footgun; per-run logs `~/.local/state/brokkr-landscape-scan/`). Prompt = brokkr's payload verbatim (LLM/image/TTS
+  new-release sweep → ranked synthesis → commit+push+notify). VALIDATED: git-push non-interactive (BatchMode
+  ls-remote to gitea, passphraseless key — no agent), headless claude auth (READY smoke). VALIDATED END-TO-END 2026-07-10 (manual run, exit 0):
+  web-sweep→synthesis→commit `2ed2f29`→PUSH of scan #2 (open-weight-releases-2026-07-24.md); triaged dwarf input +
+  caught baseline errors, quality strong. **HANDLE-COLLISION caught+FIXED** — the headless scan shared handle
+  brokkr-smithy-dev with the LIVE session + raced its inbox (eitri's dwarf-reply got stolen by the live monitor);
+  registered a dedicated **brokkr-scan-dev** handle (`add-handle`, driver=none) + repointed the wrapper + rewired
+  step-5 notify → `althing-cli post --to brokkr-smithy-dev` (NO vuong althing handle exists — confirmed). model=default
+  + `--max-turns 80`. First run under the new handle = 7/15. Off-cycle 07-24 doc is a validation artifact (scheduled
+  1st/15th runs date to their own run-date, no collision) — operator naming-convention call pending.
+  **NEXT AUTO-RUN 2026-07-15 09:00 PDT.** Manual validation/first run = `systemctl --user start
+  brokkr-landscape-scan.service`. Open w/ brokkr (thread 01KX63G6): confirm notify-Vuong handle/mechanism + session
+  handle + model/turn-cap. **NEXT brokkr task (operator-sequenced after this): TTS audition env** — Higgs-TTS-3 +
+  ZONOS2 + Chatterbox baseline, TTFA/RTF + blind-A/B web-listen (thread 01KX6371; needs GPU-placement + HF-token
+  feasibility pass first; brokkr delivers the prompt set after the env's up; protocol doc in brokkr-smithy repo).
+
+_Archived 2026-08-03._
+
+### 2026-07-08-rp-seat-campaign-closed-char-rp-magidonia-24b
+
+- `[2026-07-08]` **RP-SEAT CAMPAIGN CLOSED — char-rp = Magidonia-24B-v4.3 (128K), char-rp-reasoning = Deckard-PKD
+  Qwen3.5-27B (256K); both GGUF/llama.cpp on ana-ml2 GPU0 alongside gen (35B-A3B, util 0.37), ~4G GPU0 margin.**
+  Arc: (1) replaced broken Angel NVFP4 with Magidonia prose + QwQ-RpR-v4 reasoning (b268f93); (2) max-context via q8_0
+  KV (f570604); (3) canonical samplers for all 4 gateway seats, dvalin-derived + char-rp A/B-tuned (aac4bcf);
+  (4) rebalanced gen 0.40→0.37 to fund char-rp 128K (f49c4e4); (5) RE-A/B'd the reasoning seat (operator wanted a
+  DRY-tolerant model): **Deckard WON** on brokkr's frozen scorer (composite 2.176, 0/30 loops, 0/30 refusals) over
+  RpR-v4 (3.716, 1/30 loop), Pantheon-Reasoning (1.383 but 7/30 refusals), Snowdrop+Gembrain (llama.cpp
+  template-incompat) — deployed (5f79b40); (6) Deckard→256K (41305bf); (7) dvalin CONFIRMED Deckard samplers = the
+  live A/B set is canonical (4954ca0). **GATE LESSON: a llama.cpp reasoning seat needs a STOCK template that natively
+  opens `<think>`/`enable_thinking` (Qwen3.x/QwQ pass; ChatML + Gemma-4 fail) — no monkeypatching. INFRA: llama-swap
+  b8840 can't load Qwen3.6/Gemma-4 archs → `ghcr.io/ggml-org/llama.cpp:server-cuda` (pulled on ana-ml2).**
+
+_Archived 2026-08-03._
+
+### 2026-07-08-off-the-shelf-inference-pivot-executed-serve-curated
+
+- `[2026-07-08]` **OFF-THE-SHELF INFERENCE PIVOT executed — serve curated abliterated models, stop home-training.**
+  Final topology: **gen = `llmfan46/Qwen3.6-35B-A3B-uncensored-heretic-NVFP4-Experts-Only`** (LIVE, modelopt, vision,
+  util 0.40), **char-rp = an RP unicorn to be found on fresh context** (see Current state). Intermediate steps
+  ABANDONED: Pantheon-Reasoning-27B (served briefly as gen — refuses dark fiction via DeepSeek-distilled
+  refusal-reasoning, see Tried); Pantheon-27B-with-MTP for RP (bf16 MTP won't load on the compressed-tensors path);
+  Angel MS3.2-24B (my NVFP4 quant = garbage). Prefer EXISTING community NVFP4/GGUF quants over self-quanting
+  ("don't quant unless you have to" — operator). GGUF serving is now on the table for RP (NEVER Ollama). Gateway
+  sampling-defaults wiring still PENDING.
+
+_Archived 2026-08-03._
+
+### 2026-07-08-dpo-was-silently-running-3-epochs-harness-gap
+
+- `[2026-07-08]` **DPO was silently running 3 epochs (harness gap) → KILLED at epoch 1.2, retargeted to 0.3
+  epochs (operator call).** Root cause: `DpoConfig` had NO `epochs` field + `_dpo_config_kwargs` didn't pass
+  `num_train_epochs` → DPO fell through to trl DPOConfig's default 3.0 (SFT correctly pins 1 via SftConfig.epochs
+  + _sft_config_kwargs). Objective SATURATED by ~epoch 0.27 (loss~0, grad~0, acc 1.0, margins~27 flat — the
+  off-policy qwopus rejected pairs are trivially separable), so epochs ~0.3→3 were pure over-optimization + a
+  ~6.5h outage. No mid-run checkpoint (save_steps 500 > total steps; save only at end) → killing lost the run.
+  FIX (3 edits to deployed harness, mtf-dev to canonicalize): `DpoConfig.epochs: float = 1` (mirrors SftConfig,
+  float for fractions); `_dpo_config_kwargs` now passes `"num_train_epochs": cfg.epochs`; recipe `dpo.epochs: 0.3`.
+  GPU-free verified (dpo.epochs=0.3 → num_train_epochs=0.3). Relaunched at 0.3 epoch (~30min precompute + ~12min
+  train = ~45min). **DONE + SANITY-CHECKED (exit 0, ~70min wall: ~30min fixed precompute + 45 steps @ ~51s/step;
+  train_loss 1.4e-5 @ epoch 0.301).** Fresh `data/spike/dpo_adapter/adapter_model.safetensors` (123MB) + checkpoint-45
+  banked. **3-way greedy sanity (base vs SFT vs DPO, via peft load + disable_adapter/set_adapter on GPU0):
+  ALL THREE DISTINCT** (base≠sft≠dpo) → full SFT→DPO pipeline applies end-to-end at inference. **DPO 0.3ep is
+  COHERENT, fluent, NOT degenerate** (early-stop avoided over-optimization) but the quality delta on a neutral
+  literary prompt is SUBTLE (DPO shares SFT's structure — it continues from it — with minor stylistic drift,
+  arguably slightly MORE genre-clichéd). Verdict: mechanics proven, quality gain modest as predicted for 0.3ep
+  on off-policy pairs; the real unlock remains on-policy rejected regen + on-domain (explicit E-RP) eval +
+  the LitBench/holdout run. gen+rp RESTORED healthy. Next: serve fork (SGLang-finish vs merge) on the DPO
+  adapter — same rekey_lora_for_vllm.py (zero-z) applies unchanged (mtf-dev confirmed).
+
+_Archived 2026-08-03._
+
+### 2026-07-08-t1-dpo-leg-is-running-unblocked-2-fixes
+
+- `[2026-07-08]` **T1 DPO leg is RUNNING (unblocked) — 2 fixes applied to deployed backend.py.**
+  Blocker resolved: (1) **mtf-dev's v0.0.42 stub** `_stub_missing_optional_integrations` (last-resort sys.meta_path
+  finder → missing mergekit/llm_blender/weave resolve to MagicMock, never called → zero numerics risk; applied
+  VERBATIM to deployed `src/model_training_forge/train/backend.py` after `_unsloth_available()` + call-site before
+  `from trl import DPOTrainer`); (2) **my cosmetic `warnings_issued` shim** (trl-0.24 DPOTrainer.__init__:405 does
+  `model.warnings_issued["estimate_tokens"]=True` for warning-suppression; custom Qwen3_5 class under transformers
+  5.5.0 lacks the attr → `if not hasattr(model,"warnings_issued"): model.warnings_issued={}` before the
+  DPOTrainer(...).train() at backend.py:305 — cosmetic, zero training impact). Both edits are on the DEPLOYED
+  un-git'd copy only → **mtf-dev must canonicalize the warnings_issued shim into their repo** (told them). DPO
+  confirmed training: model loaded (851 shards), full 1196 pairs processed, in precompute_ref_log_probs (GPU0 93%
+  util, 54.8GB). Completion watcher armed (bg task) → restore gen+rp + verify dpo_adapter + ping mtf-dev on exit.
+  gen+rp STOPPED for the run (authorized window). Output → data/spike/dpo_adapter.
+
+_Archived 2026-08-03._
+
+### 2026-07-08-t1-dpo-leg-launch-prior-block-now-resolved
+
+- `[2026-07-08]` **T1 DPO leg launch — prior BLOCK (now resolved above), kept for the launch recipe.**
+  Operator authorized the full DPO stage (via mtf-dev) + went AFK 2h. **PROVEN LAUNCH RECIPE** (replicates the
+  SFT container `aeon-t1-sft` exactly, only `--stage sft`→`dpo`): `sudo docker run -d --name aeon-t1-dpo
+  --entrypoint python3 --gpus all -e CUDA_VISIBLE_DEVICES=0 -e MTF_FORCE_TRL=1 -e PYTHONPATH=/mtf/src
+  -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True -v /home/lkraven/model-training-forge:/mtf -v /tank:/tank
+  -w /mtf aeon-trainer:latest -u scripts/train.py --recipe recipes/training/qwen-3.5-122b-erp-lora/train.aeon-27b.yaml
+  --stage dpo`. **CRITICAL: `--entrypoint python3` is REQUIRED** — aeon-trainer's default entrypoint is
+  `["vllm","serve"]` (FROM vllm/vllm-openai) → without the override it runs vllm + hits a torch-ABI crash.
+  Dataset verified (pairs_dataset=train.flat.json=1196 pairs). **THE BLOCK:** `from trl import DPOTrainer`
+  (backend.py:256) eagerly pulls TRL 0.24.0's WHOLE optional-integration set — cascade: mergekit(missing)→
+  immutables→**mergekit-0.1.4↔pydantic-2.13 HARD incompat** (needs pydantic==2.10.6)→llm_blender→dataclasses_json→
+  **llm_blender-0.0.2↔transformers-5.5.0 HARD incompat** (TRANSFORMERS_CACHE removed, needs source patch)→weave→
+  (more). NONE used by our pair-based DPO. `pip install mergekit` w/deps is UNSAFE (downgrades accelerate
+  1.14→1.6). Safe partial recipe derived (core libs held: torch2.10/tf5.5.0/trl0.24.0/peft0.19.1/accel1.14.0)
+  but non-convergent → TRULY BLOCKING per operator's carve-out. Did NOT force-hack the proven training image.
+  Handed full diagnosis + recommended fix (lazy-import TRL patch, opt b) to mtf-dev (thread 01KWZG8GJX,
+  expects-reply, monitor armed). gen+rp RESTORED healthy. Relaunch = 1 min once mtf-dev delivers a working image.
+
+_Archived 2026-08-03._
+

persistent-memory.d/2026-07-08-dpo-was-silently-running-3-epochs-harness-gap.md

@@ -1,19 +0,0 @@
-- `[2026-07-08]` **DPO was silently running 3 epochs (harness gap) → KILLED at epoch 1.2, retargeted to 0.3
-  epochs (operator call).** Root cause: `DpoConfig` had NO `epochs` field + `_dpo_config_kwargs` didn't pass
-  `num_train_epochs` → DPO fell through to trl DPOConfig's default 3.0 (SFT correctly pins 1 via SftConfig.epochs
-  + _sft_config_kwargs). Objective SATURATED by ~epoch 0.27 (loss~0, grad~0, acc 1.0, margins~27 flat — the
-  off-policy qwopus rejected pairs are trivially separable), so epochs ~0.3→3 were pure over-optimization + a
-  ~6.5h outage. No mid-run checkpoint (save_steps 500 > total steps; save only at end) → killing lost the run.
-  FIX (3 edits to deployed harness, mtf-dev to canonicalize): `DpoConfig.epochs: float = 1` (mirrors SftConfig,
-  float for fractions); `_dpo_config_kwargs` now passes `"num_train_epochs": cfg.epochs`; recipe `dpo.epochs: 0.3`.
-  GPU-free verified (dpo.epochs=0.3 → num_train_epochs=0.3). Relaunched at 0.3 epoch (~30min precompute + ~12min
-  train = ~45min). **DONE + SANITY-CHECKED (exit 0, ~70min wall: ~30min fixed precompute + 45 steps @ ~51s/step;
-  train_loss 1.4e-5 @ epoch 0.301).** Fresh `data/spike/dpo_adapter/adapter_model.safetensors` (123MB) + checkpoint-45
-  banked. **3-way greedy sanity (base vs SFT vs DPO, via peft load + disable_adapter/set_adapter on GPU0):
-  ALL THREE DISTINCT** (base≠sft≠dpo) → full SFT→DPO pipeline applies end-to-end at inference. **DPO 0.3ep is
-  COHERENT, fluent, NOT degenerate** (early-stop avoided over-optimization) but the quality delta on a neutral
-  literary prompt is SUBTLE (DPO shares SFT's structure — it continues from it — with minor stylistic drift,
-  arguably slightly MORE genre-clichéd). Verdict: mechanics proven, quality gain modest as predicted for 0.3ep
-  on off-policy pairs; the real unlock remains on-policy rejected regen + on-domain (explicit E-RP) eval +
-  the LitBench/holdout run. gen+rp RESTORED healthy. Next: serve fork (SGLang-finish vs merge) on the DPO
-  adapter — same rekey_lora_for_vllm.py (zero-z) applies unchanged (mtf-dev confirmed).

persistent-memory.d/2026-07-08-off-the-shelf-inference-pivot-executed-serve-curated.md

@@ -1,8 +0,0 @@
-- `[2026-07-08]` **OFF-THE-SHELF INFERENCE PIVOT executed — serve curated abliterated models, stop home-training.**
-  Final topology: **gen = `llmfan46/Qwen3.6-35B-A3B-uncensored-heretic-NVFP4-Experts-Only`** (LIVE, modelopt, vision,
-  util 0.40), **char-rp = an RP unicorn to be found on fresh context** (see Current state). Intermediate steps
-  ABANDONED: Pantheon-Reasoning-27B (served briefly as gen — refuses dark fiction via DeepSeek-distilled
-  refusal-reasoning, see Tried); Pantheon-27B-with-MTP for RP (bf16 MTP won't load on the compressed-tensors path);
-  Angel MS3.2-24B (my NVFP4 quant = garbage). Prefer EXISTING community NVFP4/GGUF quants over self-quanting
-  ("don't quant unless you have to" — operator). GGUF serving is now on the table for RP (NEVER Ollama). Gateway
-  sampling-defaults wiring still PENDING.

persistent-memory.d/2026-07-08-rp-seat-campaign-closed-char-rp-magidonia-24b.md

@@ -1,11 +0,0 @@
-- `[2026-07-08]` **RP-SEAT CAMPAIGN CLOSED — char-rp = Magidonia-24B-v4.3 (128K), char-rp-reasoning = Deckard-PKD
-  Qwen3.5-27B (256K); both GGUF/llama.cpp on ana-ml2 GPU0 alongside gen (35B-A3B, util 0.37), ~4G GPU0 margin.**
-  Arc: (1) replaced broken Angel NVFP4 with Magidonia prose + QwQ-RpR-v4 reasoning (b268f93); (2) max-context via q8_0
-  KV (f570604); (3) canonical samplers for all 4 gateway seats, dvalin-derived + char-rp A/B-tuned (aac4bcf);
-  (4) rebalanced gen 0.40→0.37 to fund char-rp 128K (f49c4e4); (5) RE-A/B'd the reasoning seat (operator wanted a
-  DRY-tolerant model): **Deckard WON** on brokkr's frozen scorer (composite 2.176, 0/30 loops, 0/30 refusals) over
-  RpR-v4 (3.716, 1/30 loop), Pantheon-Reasoning (1.383 but 7/30 refusals), Snowdrop+Gembrain (llama.cpp
-  template-incompat) — deployed (5f79b40); (6) Deckard→256K (41305bf); (7) dvalin CONFIRMED Deckard samplers = the
-  live A/B set is canonical (4954ca0). **GATE LESSON: a llama.cpp reasoning seat needs a STOCK template that natively
-  opens `<think>`/`enable_thinking` (Qwen3.x/QwQ pass; ChatML + Gemma-4 fail) — no monkeypatching. INFRA: llama-swap
-  b8840 can't load Qwen3.6/Gemma-4 archs → `ghcr.io/ggml-org/llama.cpp:server-cuda` (pulled on ana-ml2).**

persistent-memory.d/2026-07-08-t1-dpo-leg-is-running-unblocked-2-fixes.md

@@ -1,12 +0,0 @@
-- `[2026-07-08]` **T1 DPO leg is RUNNING (unblocked) — 2 fixes applied to deployed backend.py.**
-  Blocker resolved: (1) **mtf-dev's v0.0.42 stub** `_stub_missing_optional_integrations` (last-resort sys.meta_path
-  finder → missing mergekit/llm_blender/weave resolve to MagicMock, never called → zero numerics risk; applied
-  VERBATIM to deployed `src/model_training_forge/train/backend.py` after `_unsloth_available()` + call-site before
-  `from trl import DPOTrainer`); (2) **my cosmetic `warnings_issued` shim** (trl-0.24 DPOTrainer.__init__:405 does
-  `model.warnings_issued["estimate_tokens"]=True` for warning-suppression; custom Qwen3_5 class under transformers
-  5.5.0 lacks the attr → `if not hasattr(model,"warnings_issued"): model.warnings_issued={}` before the
-  DPOTrainer(...).train() at backend.py:305 — cosmetic, zero training impact). Both edits are on the DEPLOYED
-  un-git'd copy only → **mtf-dev must canonicalize the warnings_issued shim into their repo** (told them). DPO
-  confirmed training: model loaded (851 shards), full 1196 pairs processed, in precompute_ref_log_probs (GPU0 93%
-  util, 54.8GB). Completion watcher armed (bg task) → restore gen+rp + verify dpo_adapter + ping mtf-dev on exit.
-  gen+rp STOPPED for the run (authorized window). Output → data/spike/dpo_adapter.

persistent-memory.d/2026-07-08-t1-dpo-leg-launch-prior-block-now-resolved.md

@@ -1,17 +0,0 @@
-- `[2026-07-08]` **T1 DPO leg launch — prior BLOCK (now resolved above), kept for the launch recipe.**
-  Operator authorized the full DPO stage (via mtf-dev) + went AFK 2h. **PROVEN LAUNCH RECIPE** (replicates the
-  SFT container `aeon-t1-sft` exactly, only `--stage sft`→`dpo`): `sudo docker run -d --name aeon-t1-dpo
-  --entrypoint python3 --gpus all -e CUDA_VISIBLE_DEVICES=0 -e MTF_FORCE_TRL=1 -e PYTHONPATH=/mtf/src
-  -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True -v /home/lkraven/model-training-forge:/mtf -v /tank:/tank
-  -w /mtf aeon-trainer:latest -u scripts/train.py --recipe recipes/training/qwen-3.5-122b-erp-lora/train.aeon-27b.yaml
-  --stage dpo`. **CRITICAL: `--entrypoint python3` is REQUIRED** — aeon-trainer's default entrypoint is
-  `["vllm","serve"]` (FROM vllm/vllm-openai) → without the override it runs vllm + hits a torch-ABI crash.
-  Dataset verified (pairs_dataset=train.flat.json=1196 pairs). **THE BLOCK:** `from trl import DPOTrainer`
-  (backend.py:256) eagerly pulls TRL 0.24.0's WHOLE optional-integration set — cascade: mergekit(missing)→
-  immutables→**mergekit-0.1.4↔pydantic-2.13 HARD incompat** (needs pydantic==2.10.6)→llm_blender→dataclasses_json→
-  **llm_blender-0.0.2↔transformers-5.5.0 HARD incompat** (TRANSFORMERS_CACHE removed, needs source patch)→weave→
-  (more). NONE used by our pair-based DPO. `pip install mergekit` w/deps is UNSAFE (downgrades accelerate
-  1.14→1.6). Safe partial recipe derived (core libs held: torch2.10/tf5.5.0/trl0.24.0/peft0.19.1/accel1.14.0)
-  but non-convergent → TRULY BLOCKING per operator's carve-out. Did NOT force-hack the proven training image.
-  Handed full diagnosis + recommended fix (lazy-import TRL patch, opt b) to mtf-dev (thread 01KWZG8GJX,
-  expects-reply, monitor armed). gen+rp RESTORED healthy. Relaunch = 1 min once mtf-dev delivers a working image.

persistent-memory.d/2026-07-10-biweekly-open-weight-releases-scan-cron-set-up.md

@@ -1,20 +0,0 @@
-- `[2026-07-10]` **Biweekly open-weight-releases scan cron set up for brokkr-smithy (Vuong-authorized).** Durable
-  systemd **--user** timer on nh3-dev (`brokkr-landscape-scan.timer`, OnCalendar `*-*-01,15 09:00:00`
-  America/Los_Angeles, Persistent=true; linger on) → `.service` → wrapper `~/.local/bin/brokkr-landscape-scan.sh`
-  runs headless `claude -p "$(cat ~/.config/brokkr-landscape-scan/prompt.txt)" --dangerously-skip-permissions` in
-  `~/development/brokkr-smithy` (ALTHING_HANDLE=brokkr-smithy-dev; **explicit PATH** — the --user minimal-PATH
-  footgun; per-run logs `~/.local/state/brokkr-landscape-scan/`). Prompt = brokkr's payload verbatim (LLM/image/TTS
-  new-release sweep → ranked synthesis → commit+push+notify). VALIDATED: git-push non-interactive (BatchMode
-  ls-remote to gitea, passphraseless key — no agent), headless claude auth (READY smoke). VALIDATED END-TO-END 2026-07-10 (manual run, exit 0):
-  web-sweep→synthesis→commit `2ed2f29`→PUSH of scan #2 (open-weight-releases-2026-07-24.md); triaged dwarf input +
-  caught baseline errors, quality strong. **HANDLE-COLLISION caught+FIXED** — the headless scan shared handle
-  brokkr-smithy-dev with the LIVE session + raced its inbox (eitri's dwarf-reply got stolen by the live monitor);
-  registered a dedicated **brokkr-scan-dev** handle (`add-handle`, driver=none) + repointed the wrapper + rewired
-  step-5 notify → `althing-cli post --to brokkr-smithy-dev` (NO vuong althing handle exists — confirmed). model=default
-  + `--max-turns 80`. First run under the new handle = 7/15. Off-cycle 07-24 doc is a validation artifact (scheduled
-  1st/15th runs date to their own run-date, no collision) — operator naming-convention call pending.
-  **NEXT AUTO-RUN 2026-07-15 09:00 PDT.** Manual validation/first run = `systemctl --user start
-  brokkr-landscape-scan.service`. Open w/ brokkr (thread 01KX63G6): confirm notify-Vuong handle/mechanism + session
-  handle + model/turn-cap. **NEXT brokkr task (operator-sequenced after this): TTS audition env** — Higgs-TTS-3 +
-  ZONOS2 + Chatterbox baseline, TTFA/RTF + blind-A/B web-listen (thread 01KX6371; needs GPU-placement + HF-token
-  feasibility pass first; brokkr delivers the prompt set after the env's up; protocol doc in brokkr-smithy repo).

persistent-memory.d/2026-07-10-comfyui-0-25-x-bump-on-irv-ml1.md

@@ -1,16 +0,0 @@
-- `[2026-07-10]` **ComfyUI 0.25.x bump on irv-ml1 ATTEMPTED → FAILED → ROLLED BACK (snapshot saved it).** comfy-dev
-  requested (Vuong-authorized) bumping the irv-ml1 `comfyui` stack (mmartial image, `/opt/docker/compose/comfyui/`,
-  0.24.1) to 0.25.x for Krea-2 + LTXV 2.3. **TWO FINDINGS: (1) `DISABLE_UPGRADES=false`/USE_PIPUPGRADE bumps the
-  VENV (torch 2.12.1→2.13.0 + deps) but does NOT advance the ComfyUI CODE checkout** (`/comfy/mnt/ComfyUI` =
-  `/worktank/comfyui/run/ComfyUI` stayed 0.24.1 — pinned/detached git, comfy-dev's domain). **(2) the torch bump
-  broke SageAttention** (2.2.0 `_fused.so` undefined-symbol `c10::impl::cow::materialize_cow_storage` vs torch
-  2.13.0) → `--use-sage-attention` (REQUIRED launch flag in COMFY_CMDLINE_EXTRA) crash-looped ComfyUI. Net: broke
-  the working state, zero 0.25.x payoff. **ROLLBACK WORKED**: pre-bump 16G venv snapshot
-  `/worktank/comfyui/venv-snapshot-comfyui-0.24.1-20260710.tar` restored (torch 2.12.1 + working SageAttention),
-  re-pinned DISABLE_UPGRADES=true, recreated → healthy on 0.24.1, serving :8188. Broken venv parked at
-  `/worktank/comfyui/run/venv.broken-torch213-20260710`. **CORRECTED PATH (sent comfy-dev, thread 01KX655V):**
-  comfy-dev git-advances the ComfyUI checkout to 0.25.x + reqs → I handle the torch bump + SageAttention
-  rebuild-against-2.13.0 + re-pin (snapshot stays as the net). **LESSON: mmartial `DISABLE_UPGRADES` gates ONLY
-  the venv pip-upgrades, NOT the ComfyUI git checkout; a torch bump breaks compiled exts (SageAttention) →
-  rebuild-after is mandatory.** Bump BLOCKED pending comfy-dev's git-advance. Stack: A6000 (NVIDIA_VISIBLE_DEVICES=1),
-  lkraven-owned compose+venv (uid 1000, no sudo needed), COMFY_CMDLINE_EXTRA OOM flags preserved.

persistent-memory.d/2026-07-10-comfyui-v0-27-1-success-on-irv-ml1.md

@@ -1,14 +0,0 @@
-- `[2026-07-10]` **ComfyUI v0.27.1 SUCCESS on irv-ml1 (operator-confirmed execute-now) — landed on torch 2.12.1,
-  SageAttention preserved, crash-loop AVOIDED.** The prior attempt (entry below) crash-looped because a torch
-  2.12.1→2.13 bump broke SageAttention's ABI. This time I checked `git diff v0.24.1 v0.27.1 -- requirements.txt`
-  FIRST and found **core v0.27.1 leaves `torch` UNPINNED** → the version bump does NOT require torch 2.13 (that came
-  only from the mmartial boot-upgrade). So: `git checkout v0.27.1` (clean tree) → `pip install -r requirements.txt`
-  as **uid 1000** with a **torch-pin constraint file** (torch/vision/audio pinned to current +cu129) to block any
-  transitive bump → torch stayed 2.12.1, SageAttention 2.2.0 untouched. Added decord 0.6.0 (fixed SAM3Segment).
-  `docker restart comfyui` → healthy, `/system_stats` comfyui_version=0.27.1, "Using sage attention", HTTP 200, DB
-  migrated 0003→0004. Reported the divergence to comfy-dev (thread 01KX6D3C…, reply pending) + asked whether LTXV 2.3
-  needs a separate torch-2.13 follow-up (their domain; Krea-2's ≥0.25 need is met by 0.27.1). **LESSON: before a
-  mmartial ComfyUI version bump, `git diff <old> <new> -- requirements.txt` — if torch is unpinned, bump the CODE
-  without touching torch (constraint-pin it) and compiled exts (SageAttention) survive. `docker exec` lands as uid
-  1025(comfytoo), not 1000 — use `-u 1000` + the venv python `/comfy/mnt/venv/bin/python`.** See
-  [[reference_irv_ml1_comfyui_mmartial]].

persistent-memory.d/2026-07-10-heimdall-grant-ratatoskr-affect-full-on-personal-worldtree.md

@@ -1,13 +0,0 @@
-- `[2026-07-10]` **Heimdall grant: ratatoskr `affect.full` on PERSONAL Worldtree (operator-approved, worldtree-dev
-  R34-v1 request).** Added allow-rule `ratatoskr-affect-full-allow` to `/opt/worldtree-personal/config/policies.yaml`
-  (`principal.user_ids:["ratatoskr"]`, action `affect.full`, resource `*`, effect allow), mirroring the #347
-  `session-history-write-ratatoskr` rule exactly + placed right after it. **WHY user_ids-based (not tier):** ratatoskr's
-  personal key is the minimal **readonly-admin** observability tier, which is NOT in the tier-based
-  `affect-render-baseline-allow` (anonymous/user/free/pro/admin) → needs an explicit user_id grant, same as #347.
-  R34-v1 (b46, committed UNPUSHED) gates `affect.emit` `dominant_emotion` egress by exposure ceiling (affect.full|safe
-  → present; neither → null); this grant keeps ratatoskr's view alive across the b46 deploy. Surgical exact-string
-  insert (preserves comments), backup `policies.yaml.bak-pre-affectfull-20260710`, validated via the CONTAINER's own
-  yaml parser (35 rules, +1, payload confirmed). **NOT restarted — deliberate:** rule is on the bind-mount (shadows
-  baked), INERT until b46 gating ships, so the b46 CI/CD deploy restart activates it (no live-session blip now). Demo
-  untouched (personal-only per key scope). Replied to wt-dev (thread 01KX6DB3…) offering an immediate restart if they
-  want it live for pre-b46 testing. See [[reference_corviduo_dev_emergency_ops]].

persistent-memory.d/2026-07-13-355-residual-root-cause-supersedes-the-litellm-gateway.md

@@ -1 +0,0 @@
-- `[2026-07-13]` **#355-residual ROOT CAUSE (supersedes the "LiteLLM gateway holds while seat idles" entry below — that was DISPROVEN).** char-rp-reasoning enters a non-terminating REASONING loop (tool-call-retry planning) and runs to `max_tokens=32768` (~22 min @ 24.7 tok/s, ~13% of requests); the seat GENERATES all 32768 tokens (not idle), and `--reasoning-budget 400` is NOT enforced. 3-source-confirmed (spend_logs completion_tokens=32768 ×4; seat eval-time log; pcap 100%-`reasoning_content` deltas). Server-side fix wanted (operator: no max_tokens ceiling) → routed to brokkr (accepted, pulled dvalin). Lesson (again): confirm before concluding — the seat-idle claim came from reading only the ≤73s requests + missing the concurrent 32768-token slots. See ACTIVE 1.

persistent-memory.d/2026-07-13-deploy-speed-real-bottleneck-uv-sync-memory-s.md

@@ -1 +0,0 @@
-- `[2026-07-13]` **Deploy-speed real bottleneck ≠ uv sync (memory's assumption was wrong).** Buildx step log: `chown -R /app` = 251s (copy-up of the root-owned venv into a fresh layer), uv sync only 35.6s, registry layer cache already wired. Fix = drop `/app` from the chown (validated safe: zero /app runtime writes on both live instances) + uv cache-mount. Shipped as PR #359 (branch off origin/main@b60), worldtree-dev green-lit. Expected ~5min off (~11→~6min). Runner-side BuildKit cache task (b) was already done → moot.

persistent-memory.d/2026-07-13-ledger-tier-3-consumer-ledger-miranda-provisioned-on.md

@@ -1 +0,0 @@
-- `[2026-07-13]` Ledger tier-3 consumer `ledger:miranda` provisioned on personal :8081 (key b38932f5, GPG-delivered+shredded, allowlist 10.100.10.50:8770 live); `assistant`+`thoughtful-assistant` capability roles added (gen/gen-reasoning) on personal+demo, canonical d8bd497. Chosen instance = personal (the tier-3-consumer instance, ratatoskr+soong-lab colocated).

persistent-memory.d/2026-07-13-worldtree-deploy-bottleneck-the-image-build-11-min.md

@@ -1 +0,0 @@
-- `[2026-07-13]` Worldtree deploy bottleneck = the image build (~11 min of a ~12 min deploy), root cause the Dockerfile `uv sync ... --no-cache` + no BuildKit cache-mount (re-downloads all deps cold every build). Fix split: worldtree-dev Dockerfile cache-mount diff + infra-ops runner-side persistent BuildKit cache. Config-only changes skip the build entirely (pinned recreate).

persistent-memory.d/2026-07-13-wt-355-residual-300s-hang-localized-to-our.md

@@ -1 +0,0 @@
-- `[2026-07-13]` WT #355 residual 300s hang localized to OUR LiteLLM gateway (holds 2 char-rp-reasoning requests ~21 min while the seat idles), NOT the seat — Deckard seat EXONERATED (completes ≤72s; `--reasoning-budget 400` forecloses a mid-thinking hang). Corrects worldtree-dev's "seat wedging" diagnosis. Decisive next = the FIN-check (pcap on corviuo). See in-flight ACTIVE 1. **[SUPERSEDED 2026-07-13 — see the ROOT CAUSE entry above; the gateway-hold/slot-leak theory was disproven, the seat was generating 32768 tokens.]**

persistent-memory.d/2026-07-13-wt-355-turn-lifecycle-fix-validated-on-worldtree.md

@@ -1 +0,0 @@
-- `[2026-07-13]` WT #355 turn-lifecycle fix VALIDATED on worldtree b60 — wedged turns self-terminate cancelled/stalled at the 300s stall-watchdog (turns 2064/2065 vs pre-b60 2061's 16-min no-terminal). worldtree-dev filed follow-ons #356 (rehydrate Tier-3 ctx on resume — the recreate-durability gap), #357 (reclaim orphaned active-turn locks), #358 (LLM-provider read-timeout audit); surfacing to Vuong to prioritize.

persistent-memory.d/2026-07-14-char-rp-reasoning-seat-deckard-pkd-neo-code.md

@@ -1 +0,0 @@
-- `[2026-07-14]` **char-rp-reasoning seat: Deckard-PKD → NEO-CODE = Heretic2-Thinking (Qwen3.6-27B)** — R36 gate PASSED (tools 0.967, #355 runaway ELIMINATED). #355 was MODEL-level (Deckard emitted qwen3_coder XML malformed → mangled args → retry-runaway), NOT the reasoning-budget bug; NEO-CODE emits it clean. Custom llama.cpp KEPT (qwen3_coder parse — stock b8840 predates it — + PR#25544). Committed f960a73; full record auto-memory [[charrp-custom-llamacpp-pr25544]].

persistent-memory.d/2026-07-14-nvfp4-mtp-fast-char-rp-reasoning-seat-landed.md

@@ -1 +0,0 @@
-- `[2026-07-14]` **NVFP4+MTP fast char-rp-reasoning seat LANDED + LIVE + gateway-repointed + VRAM-tuned.** Modelopt-format re-quant made MTP work. The load-crash root cause = **vLLM 0.24 does NOT propagate modelopt `exclude_modules` to the spec-decode DRAFT model** → the bf16 mtp head gets quantized → shape crash; NO checkpoint config fixes it (`is_layer_skipped` is exact-membership, and the drafter never sees the mtp excludes) → **workaround = a mounted `sitecustomize` that force-skips `mtp.*` in `is_layer_skipped`** (upstream vLLM bug to file). Productionized as compose stack `heretic2-charrp-reasoning` (:8018, workaround baked in). Gateway `char-rp-reasoning` alias fixed: repointed off the stale GGUF served-name `deckard-pkd-27b`, added `enable_thinking:true`, **dropped `min_p`** (MTP-incompatible), canonical samplers temp1.0/top_p0.95/top_k20. Rebalanced GPU0 (gen 0.37→0.30/16-seq/256K + reasoning 0.39/16-seq/192K+MTP + char-rp 128K, 2.7GB free). All 4 gateway roles verified; vLLM reasoning-parser confirmed **leak-free** (unlike the GGUF budget-forcing). Full record + the 4 quant landmines in `docs/runbooks/heretic2-nvfp4-mtp-seat.md`; committed `982c319`. Open (non-blocking): brokkr P00 (seat is live ahead of it), retire the stopped GGUF reasoning seat, file the vLLM bug.

persistent-memory.d/2026-07-14-nvfp4-quant-chase-resolved-gibberish-pivoted-to-modelopt.md

@@ -1 +0,0 @@
-- `[2026-07-14]` **NVFP4 quant chase RESOLVED (gibberish) + PIVOTED to modelopt for MTP.** One ~40-min GPU0 window. Root-caused the `!!!!` to the quant NAMESPACE (text-only `AutoModelForCausalLM`→`model.layers.*` keys; vLLM serves only `Qwen3_5ForConditionalGeneration`, which needs `model.language_model.*`) — found from config diffs + vLLM source with ZERO GPU time; fixed by loading as `AutoModelForImageTextToText`. NVFP4 now serves COHERENT (validated greedy). BUT base NVFP4 ≈53 tok/s ≈ GGUF's 59.5 at batch-1 (no single-stream win) AND MTP = 0% acceptance on compressed-tensors (bf16 mtp head only loads on the modelopt format). Operator chose to **pursue a modelopt-format re-quant** (the only path to the 2-4× MTP goal; AEON-proven on this exact Qwen3.6-27B arch). Scoped + de-risked: AEON `/tank/aimodels/qwen36-27b-aeon-nvfp4` = the modelopt reference (quant_method modelopt, 1967 tensors, 15 bf16 mtp keys identical to graft); nvidia-modelopt 0.45.0 installs + `mtq.quantize`/`NVFP4_DEFAULT_CFG`/`export_hf_checkpoint` API confirmed; pipeline unchanged except swap llm-compressor→modelopt. Seats restored; char-rp-reasoning stays GGUF. Full plan in Current state ★ section.

persistent-memory.d/2026-07-14-pursue-the-nvfp4-mtp-fast-char-rp-reasoning.md

@@ -1 +0,0 @@
-- `[2026-07-14]` **Pursue the NVFP4+MTP fast char-rp-reasoning seat to completion** (Vuong-directed via /snapshot: "chase the nvfp4 quant, we know it works, write down the recipe"). Full recipe + diagnostic ladder in Current state / in-flight above. Artifacts on ana-ml2 `/tank/aimodels/heretic2-nvfp4-work/` + scripts committed in eshpfi `services/heretic2-nvfp4-quant/`.

persistent-memory.d/2026-07-14-soong-lab-webhook-auto-deploy-real-root-cause.md

@@ -1 +0,0 @@
-- `[2026-07-14]` **soong-lab webhook auto-deploy real root cause = gitea `webhook.ALLOWED_HOST_LIST`** (was `external, 10.100.0.0/16` = NH3-only; blocked corviduo-dev's Anaheim `10.250.x` → gitea refused to deliver, never opened the connection). Fixed to fleet-wide `10.0.0.0/8` (app.ini `[webhook]`) + gitea restart; listener now logs every delivery. The ufw `10/8` open (also this session) was a real-but-secondary gap. Committed 462d528.

persistent-memory.d/2026-08-03-worldtree-b168-384-385-arc.md

@@ -0,0 +1,17 @@
+**Worldtree b168/#384/#385 arc — COMPLETE 2026-08-03.** A long peer-driven arc across worldtree-dev / muninn-dev / mimir-dev / ratatoskr-dev, all on corviduo-dev's demo+personal instances. Sequence: providers.yaml boot-gate pre-sync → b168 deploy → DCC #384 reindex → round-2 full re-ingest → #381 restart → operator-approved production dedup sweep. Landed clean; three of MY foot-guns along the way, each caught + hardened into a fleet runbook rule (see Tried-and-abandoned: `mv -t`, `docker exec -u 1000`, shared-containerd race).
+
+## providers.yaml pre-sync (boot-gating config)
+b168 (commit `293f8f3`) added a `summarization` capability block that in-image `agents/muninn/config.yaml` references → boot-blocking if the host bind-mounted providers.yaml lacks it. Synced both hunks (summarization block + deep-reasoning desc) into demo+personal via `deploy-wt-config`; instance-configs commit `53349f8`.
+- **deploy-wt-config runbook:** `~/development/worldtree-instance-configs/scripts/deploy-wt-config {diff|deploy|capture} <inst> --file providers.yaml` (per-instance dirs demo/personal/pinned; `deploy` = host write + api/matrix restart + 90s health-gate + auto-rollback; `diff`/`capture` safe). demo+personal providers.yaml are byte-identical.
+- **GOTCHAS:** (1) an UNPUSHED source commit → `git show <sha>` 404s and a gitea `raw?ref=<sha>` silently falls back to the default branch; verify the commit exists (`/git/commits/<sha>`) before trusting a fetch, else ask the peer to paste hunks. (2) a peer's hunk paste may be mis-indented (8-space vs the block's 4-space) → invalid YAML; always YAML-validate after a paste-sourced edit.
+- **Config-delta pre-sync rule (verified via `docker inspect`):** worldtree containers bind-mount ONLY `config/` host-side (`/opt/worldtree-*/config/` → providers/model_roles/matrix/policies/defaults/env.public = the pre-syncable set); `agents/` (schemas.yaml, prompts) + all code ship IN-IMAGE. So only a `config/*.yaml` change is boot-blocking-pre-syncable; an `agents/`-or-code delta needs NO host pre-sync (CI carries it). b169's schemas.yaml (#387) was correctly no-pre-sync.
+
+## #384 reindex + #381 restart + verify
+DCC job `mimir-6351554e8e8f`. Reindex: `sudo docker exec -u 1000 worldtree-personal-worldtree-muninn-1 python -m core.muninn --reindex <job>` (⚠️ MUST `-u 1000` — default-root writes contaminate the uid-1000 KB tree; see Tried-and-abandoned). Then **#381 restart** (stale-Chroma-client fix): `sudo docker restart worldtree-personal-worldtree-api-1` (plain bounce, NO compose up / no image repoint) → healthz/readyz 200 ~25s.
+- **Chroma-verify runbook:** `sudo docker exec -i <muninn> python -` (MUST pass `-i` or stdin never reaches `python -`) → `chromadb.PersistentClient('/data/kb/.chroma').get_collection('fiction').get(where={'job_id':<job>}, include=['metadatas'])`. Chroma persists at container `/data/kb/.chroma` = host volume `worldtree-personal_worldtree-kb`.
+- **Retrieval-visibility check (NOT grounding — that's ratatoskr's):** a Mimir session — admin token `~/.config/worldtree/personal-admin-token` (wildcard scope) → POST `/sessions` (agent_id=`mimir`, `record_tool_intermediates=true`) → POST `/sessions/{id}/messages` (STREAMS SSE, not JSON) → parse SSE `tool_result` for `search_library` wing hits → DELETE session.
+
+## Production dedup sweep (operator-approved)
+Deleted the 785 April-era DCC orphan rows (`job_id=b59c147c5ce0`, no wing/source_identity metadata → predate identity tracking) from the `main` collection. Supervised protocol: read-only verify count == 785, back up all rows (ids+docs+embeddings) to `corviduo-dev:/tmp/main-sweep-backup-b59c147c5ce0.json` (reversible), `main.delete(where={job_id})` (assert target==785 first), verify `main` 4009→3224, then **bounce the api** (a separate-process delete leaves the api's in-memory HNSW index holding the vectors until reload — the #381 pattern generalizes to deletes), confirm search now fiction-only. Backup left for /tmp natural cleanup (fiction wing is canonical; `~/archives` has the historical record).
+
+Result: fiction wing 166 → 1,372 concepts; three consumer verify rounds 0/5 → 5/5 → saturated; #385 budget fix validated (705 vs April's 785 control, extraction AND indexing, zero truncations). worldtree-dev filed #388 for a deploy concurrency-lock (the shared-containerd race fix). See [[2026-08-02-mimir-inbox-arc]].