esh-pfi-infrastructure/persistent-memory.md

# Persistent memory — eshpfi-management

_Last updated: 2026-07-14_

## Repo purpose

Reference workspace for PFI infrastructure: server inventory, canonical
Docker Compose stacks, ops playbooks, and conventions. Authoritative
copies of compose files live on the servers under
`/opt/docker/compose/<stack>/`; this repo mirrors them for version
control, editing, planning, and CI-driven deploys. **It was originally
spun up to handle the fleet backups** — keep that lens when triaging
backup/storage issues.

## Tools and conventions

Sister repos (separate gitea repos, deployed by playbooks here):

| Repo | Role | CI status |
|---|---|---|
| `vh/task-board` | MCP + web dashboard for assistant task state (port 7878) | push-to-main → CI deploys (2026-04-29) |
| `vh/vor` | Inquisitor UI sidecar (port 7879) | push-to-main → CI deploys (2026-04-29) |
| `vh/nevermore` | Twice-daily LLM-curated briefing (port 8181, replaces news-digest) | push-to-main → CI deploys (2026-04-30) |
| `vh/asset-engine` | Internal control plane over inference services (port 8200, LAN-direct) | push-to-main → CI deploys (2026-05-12) |
| `vh/althing` | Lean trusted inter-agent message bus — **v2 "email model" (v2.0.0b2, 2026-07)**: per-box local-SQLite bus + courier/receiver for P2P over the 10.x net; pillars = open-loops / per-box herald + wake-listener / roaming owner API `/owner/*` / `althing-mcp` stdio surface. The v0.15 lean-bus cut RIPPED moderation / chamber / forseti-daemon / agent-runner / redis-valkey. | per-box `uv tool install` (NOT CI-deploy); **nh3-dev = the DEV box** (editable install of `~/development/althing`, gets new versions first); **nh3-extdev** a mesh peer (model B: althing-svc + shared `/srv/althing`) |
| `vh/mead-hall` | Bifrost tool-provider sidecar (port 5173 on dev VM 10.100.10.50) | push-to-main → CI deploys (2026-05-16) |
| `vh/skaldsong` | Wizard + reader surface (port 8300, ana-docker, registry-pull pattern) | push-to-main → CI deploys (2026-05-19) |
| `vh/Worldtree` | Conversation API (corviduo-dev demo :8080 / personal :8081 / pinned :8082) — Heimdall auth, Bifrost integration. **gitea-runner builds on ana-docker**; claude-bot ADMIN collaborator (2026-06-20). Now v1.0.0b19. | push-to-main → CI build-and-deploy (runner on ana-docker) |
| `vh/yt-voice-clipper` | YouTube → diarized voice-clip dataset builder + audition console (irv-ml1 :8000) | push-to-main → **gitea-webhook auto-deploy** to irv-ml1 (2026-06-03) — see `docs/runbooks/ytvc-autodeploy.md` |
| `vh/arbo` | Catalog-driven ComfyUI engine (irv-ml1 :8201, comfy-dev owns engine/catalog/image) | push-to-main → gitea Actions CI (deploy-engine.sh, build-local, health-gated) now LIVE; catalog via :9009 webhook |
| `model-training-forge` (mtf-dev) | Fine-tuning recipe forge; **T1 = E-RP writing LoRA, retargeted qwopus-122B→AEON-27B (2026-07-06)** (SFT→DPO, LitBench-RM reward) | training runs, not a deployed sidecar |

(`vh/volva` + Heid were re-architected from systemd daemons to Claude Code
session orchestrators 2026-06-08; their nh3-dev `.service` units were removed —
no longer deployed sidecars here. See Recent decisions.)

- **Two-layer backups** — Backrest orchestrates restic for file+DB (5
  fleet repos, daily 01:00 PDT); PBS-ANA primary + PBS-NH3 DR mirror for
  VM images. ana-nas is the SPOF for postgres + PBS-ANA datastore +
  cross-site restic targets — see `docs/runbooks/disaster-recovery.md`
  for the blast-radius matrix. **⚠️ The restic file+DB layer routes
  through TWO rest-servers** (`rest-server-ana` @ ana-docker:8000 →
  ana-docker/ana-ml2/esh-docker-vm/vm-esh-nas; `rest-server-nh3` @
  nh3-nas:8000 → irv-ml1/nh3-docker). Both depend on their NAS's NFS
  export of `/mnt/backup`. (rest-server-ana recovered 2026-06-20.)

- **`pull-hf-repo.yaml`** is the canonical "get a HuggingFace
  model/dataset onto ana-ml2's shared cache at
  `/tank/aimodels/huggingface/`" playbook. Supports `--var repo_type=model|dataset|space`. Replaces ad-hoc `huggingface_hub.snapshot_download` patterns.

- **Worldtree admin auth — per-instance.** Each Worldtree deployment
  (demo :8080, personal :8081, pinned :8082) has its own Heimdall
  registry and its own bootstrap admin key. Infra-ops's stored
  long-lived admin key (`key_id 61419c92`) at
  `ana-docker:/opt/docker/conf/.secrets/worldtree-infra-ops-admin`
  auths against **demo only**. Personal-instance admin (the
  `~/.config/worldtree/personal-admin-token`, mode 600) POSTs
  `/admin/keys` (mints per-project keys; takes `user_id`+`label`, **no
  scope param** — scopes are tier-derived). **On-instance mint recipe
  (cleaner than DB-manip):** `docker exec worldtree-worldtree-api-1` POST
  `/admin/keys` with the in-container `WORLDTREE_BOOTSTRAP_ADMIN_KEY`; cleartext
  once in `.key`=`wt_live_+16hex`. auto-memory `reference_worldtree_demo_key_mint`.

- **Per-project user keys against personal Worldtree** (issued
  2026-05-19): `skaldsong:79744637`, `skaldsong:7c1dbbbe`,
  `althing:50d85460`, `mead-hall:a360822d`. Mint via `/admin/keys`, drop
  value to `/tmp/wt-personal-<name>.key` mode 600, dev collects +
  shreds (DO NOT cat to chat transcript).

- **Skaldsong CD pattern (registry-pull).** vh/skaldsong's CI builds and
  pushes `gitea.phasefinal.com/vh/skaldsong:<sha>` + `:latest`;
  `playbooks/deploy-skaldsong.yaml` on ana-docker pulls + recreates.
  SHA-pin only. Prereq: host needs `docker login gitea.phasefinal.com` once.

- **gitea internal route for fleet hosts.** gitea is a container on
  **ana-docker** — git-SSH `10.250.50.70:222`, HTTP `:3000`. Fleet/colo
  hosts must use this internal route, NOT public `gitea.phasefinal.com`
  (`38.120.12.44`) — the public path fail2bans the host egress IP. Full
  gotcha in `docs/orientation.md` → Git/gitea.

- **docker-as-root pattern** (for ops with no admin API, or to edit
  deploy-owned/root-owned files without sudo): `docker run --rm -v
  <target-dir>:/wt docker:cli sh -c "..."`. docker-group membership is
  effectively root via bind-mount. **Foot-gun: relative paths in compose.yaml
  resolve against the sandbox CWD but the daemon interprets them against the
  HOST fs — always pass `-e VAR=/abs/path` for any relative-default config dir.**

- **`scripts/elway` sudo handling** — elway prompts for the sudo password
  ONCE via `getpass` before the first `sudo: true` step → can't run
  unattended from a non-TTY tool if any step needs sudo. Sudo-free
  playbooks run fully non-interactive over key SSH.

- **Per-host SSH identity matters for sudo.** infra-ops has NOPASSWD sudo
  on most PFI Linux boxes (corviduo-dev included since 2026-06-15). On
  ana-docker: **default `ssh ana-docker` = `lkraven`** (docker-group, NO
  passwordless sudo); **`ssh infra-ops@ana-docker` HAS NOPASSWD root**. **→
  For any sudo op on ana-docker, use `ssh infra-ops@ana-docker`.** `ssh
  infra-ops@10.100.10.50` (nh3-dev) ALSO NOPASSWD sudo; on **nh3-extdev** infra-ops
  is sudo-LESS by design (`ssh lkraven@10.100.50.42` is the NOPASSWD path). **irv-ml1:
  `ssh irv-ml1` = lkraven, docker-group (plain docker) but sudo needs a PASSWORD
  (no NOPASSWD)** — stage model pulls to `/home`, not root-owned `/worktank`.

## Current state / in-flight

_As of 2026-07-13 — two active tasks (per /snapshot args): the LiteLLM gateway #355-residual investigation and wiring the runner-side persistent BuildKit cache. The big WT #355 validation saga + Ledger provisioning + the assistant-role adds all LANDED this session; the concluded TTS/Zonos/RP-seat/ComfyUI/T1 in-flight was rolled out of this section (it lives in git + auto-memory)._

### ✅ #355-residual — RESOLVED 2026-07-14 via MODEL SWAP (Deckard→NEO-CODE); was ACTIVE 1

**UPDATE 2026-07-14 — FULLY RESOLVED via MODEL SWAP.** The custom llama.cpp (PR #25544 multi-terminator) was necessary but **INSUFFICIENT alone** (Deckard still ran away ~5% of tool-turns). Root cause was **MODEL-LEVEL**: Deckard emitted Qwen's NATIVE qwen3_coder XML tool format **MALFORMED** → llama.cpp leaked the closing tags into the arg value (`{"name":"\n</parameter>\n</function>..."}`) → Bifrost `attach_tool` schema error → retry → reasoning-runaway. **Swapped the model to NEO-CODE = Heretic2-Thinking (Qwen3.6-27B)** (GGUF base_model `DavidAU/Qwen3.6-27B-Heretic2-Uncensored-Finetune-Thinking`; "NEO-CODE" = DavidAU's imatrix quant recipe, not a code finetune) which emits the same native format **CLEANLY** (`{"name":"Sindra"}`) on the same seat/parser. **R36 gate (brokkr 2026-07-14): NEO-CODE ships on ALL axes** — tool-calling **0.967** (attach_tool 1.00, **0 runaways**, beats gen 0.856 + Deckard 0.08/0.80), prose genre-artifact-fine (D2-slop repetition = dialogue-tags/anatomical genre convention, LESS clichéd than gen — not degradation), refusal uncensored-as-spec (creative-dark 38/40) + CSAM-gate-clean (0/100 operational-harm → serve-layer guardrails are the NSFW-RP consumer's job). **LIVE in production** (Q5, 256K ctx, card samplers temp 1.0 — temp 1.0 IMPROVED tools; DRY dropped); Deckard = rollback (`.env`/compose `.bak-pre-neocode-20260714`). **Custom llama.cpp KEPT** (now needed for qwen3_coder parsing — stock b8840 predates it, so stock is NOT a valid NEO-CODE rollback — + #25544 belt-and-suspenders). There was **NEVER a "wrong parser"** — the XML IS Qwen3.5/3.6-native (byte-identical to `Qwen/Qwen3.6-27B`'s template); an earlier "misconfigured OpenHands template" config-override reframe was WRONG and retracted. **OPEN:** seat is ~50 tok/s GGUF → soong's 45s client-timeout + the reasoning-budget-latency squeeze remain the constraint; the DFlash spec-decode recipe (z-lab Qwen3.6-27B drafter, ~4× on vLLM) applied to an NVFP4 Heretic2 vLLM seat is the future faster-seat path (NOT AEON-7-the-model — a non-RP-tuned base abliteration whose 4× speed is DGX-Spark-locked). Full settled record: auto-memory [[charrp-custom-llamacpp-pr25544]]. Historical mid-investigation saga kept below for context.

**BOTTOM LINE (historical):** char-rp-reasoning ran away to `max_tokens=32768` (~22 min, ~13% of turns) because stock llama.cpp **b8840's reasoning-budget forcing can't match Qwen3.5's `<tool_call>` reasoning terminator** (it only knows `</think>`) → on the tool-retry path forcing never fires. **FIXED** by a custom build (master `6eddde0` + unmerged upstream **PR #25544**, multi-terminator) now **LIVE** on the seat (`llamacpp-charrp:custom-latest`). **PARTIALLY VALIDATED**: the fix cut the runaway RATE hugely (soong smoke — most tool-turns now complete ~90 tokens, reasoning ~40 tokens, clean) BUT has a **RESIDUAL GAP** — task 5674 (2026-07-13 ~21:43) still ran away to 32768 (~9min; reasoned >45s before any content → the cap didn't fire on that turn). b61's 300s = client backstop but does NOT propagate to the seat → seat still burns to 32768 (confirmed). **⚠️ DO NOT retire the custom build as "0-gain" — CORRECTED: the seat is meant to be an RPG-GM (tools + prose + NSFW), so the tool-path #355 fix is the TOOL-PATH ENABLER for any reasoning-RP model here → KEEP it.** The REAL problem is the MODEL: **Deckard (RP-abliterated) is a poor tool-caller** (rambles/loops on tool-turns even with the fix — abliteration + heavy RP-tuning damages tool-calling/instruction-following). **MODEL-SWAP underway:** brokkr running a hunt for an NSFW-abliterated + **RELIABLE-tool-calling** + GM-prose **~24-27B** replacement (thread `01KXEQWEN2…`; ★ tool-calling is now a DISQUALIFYING A/B axis — the original A/B that picked Deckard scored prose/loops/refusals but NEVER tested tools). **PLAN: leave soong on char-rp-reasoning; swap the MODEL under it when brokkr delivers (no double-push).** Still **REMOVE-WHEN-#25544-MERGES** for the custom llama.cpp itself (retire custom→stock llama.cpp when the fix ships upstream; independent of the model swap). Records: auto-memory `charrp-custom-llamacpp-pr25544` + eshpfi `stacks/char-rp-gguf/llamacpp-custom/`; rollback = `.env` `LLAMA_REASONING_IMAGE=ghcr.io/mostlygeek/llama-swap:cuda`. Deploy-speed side-quest (#359+#360, ~11m→~4m) + soong-lab CI/CD (green-gated push-to-deploy + red-relay) also landed this session. Full diagnostic saga below (kept for the retire-decision context).

### 🔺 #355-residual ROOT CAUSE (historical saga): seat reasoning-loop to max_tokens
- **DEFINITIVE root cause (3 independent sources, 2026-07-13):** char-rp-reasoning (Deckard-PKD, seat ana-ml2:8018 `llama-charrp-reasoning`) intermittently enters a **non-terminating REASONING loop** and runs to the request's **`max_tokens=32768`** → ~22 min at 24.7 tok/s. ~13% of requests (4 of ~30 in the wedge window). It is NOT the gateway holding, NOT a slot leak, NOT the seat idle — the seat genuinely GENERATES 32768 tokens. My earlier "LiteLLM holds ~21 min while seat idles / slot-leak" read was **DISPROVEN** (the seat wasn't idle; there's no `max_parallel_requests`/`router_settings` in the config at all → no semaphore to leak).
- **Evidence:** (a) spend_logs: 4 requests each `completion_tokens = EXACTLY 32768`, status=success (hit the length cap cleanly); normal requests EOS at 730–2250. (b) seat llama.cpp log: `eval time = 1325064ms / 32768 tokens (24.73 tok/s)`, `truncated=0`. (c) WT↔gateway pcap: runaway stream is **100% `reasoning_content` deltas**, coherent, planning tool-call retries ("The tool call failed due to input schema errors… Let me make these calls one at a time"). NEVER exits `<think>`.
- **MECHANISM — RESOLVED (Step-1 non-disruptive + Dvalin source read, 2026-07-13):** the budget **ARMS but its force-close NEVER FIRES** in the seat's build. Seat log aggregate: 158 `reasoning-budget: activated, budget=400` / 110 `deactivated (natural end)` / **0 forced closes** / 48 never-closed arms (= the runaways). Build = **llama.cpp 8840 (`9e5647aff`)**, image `ghcr.io/mostlygeek/llama-swap:cuda`. Dvalin read commit 9e5647aff: force-inject is **NOT gated on `--reasoning-budget-message`** (server-task.cpp:493-495 `if (!end_tag.empty())`, no message guard; reasoning-budget.cpp:99-108 fires unconditionally at remaining≤0) → so it's genuinely broken in b8840, NOT the missing flag. NOT a re-arm loop (9e5647aff's DONE is a plain break; re-arm landed later on master). Every request carries 8 tools; runaways = the tool-retry-FAILURE subset (model won't self-terminate `</think>`, forcing doesn't save it). `reasoning_control` API is ABSENT in 8840 (`--help` has no flag; binary has no `/control` strings) → brokkr's watchdog needs the bump too.
- **FIX (agreed with brokkr, NO max_tokens ceiling per operator):** (1) IMMEDIATE relief = push worldtree's **b61 270s wall-clock** — bounds the user-facing hang regardless of the seat loop (so the seat bump is NOT urgent). (2) DIAGNOSE-THEN-FIX (Eitri refinement, may SAVE the bump) — first capture the detailed tuple `tokens/start/end/forced` (needs a **verbose / low-budget-matrix seat restart**; 8840's default log has only the simpler activated/deactivated line — no `forced=`/`start=…end=`, confirmed by grep): **`end=0` → the FORCED-injection end_tag isn't reaching the budget path (Deckard template / reasoning-format mismatch) → CHEAP CONFIG fix = explicit budget start/end tags, a restart NOT a bump**; `end>0,forced=0` → end-tag tokenization bug; `end>0 + no forcing across arms` → real state bug → build bump. (NB: 110 `deactivated (natural end)` prove the DETECTION end-tag works, so an `end=0` would be specifically the FORCED-inject tag, a narrow config miss.) Only if the tuple says end>0 → **DURABLE = stage + validate a post-#22740** llama.cpp build OFF the hot path (⚠️ #22717 garbled-forcing regression ~b9018 fixed by #22740 → don't land in the b8840→b9018 window; validate vs `tests/test-reasoning-budget.cpp` + that it loads Deckard Q5_K_M), then surface the seat-image swap to Vuong; `--reasoning-budget-message` additive WITH the bump. Getting the tuple = the disruptive step (bounce the prod RP seat to --verbose/low-budget; a separate test instance won't fit — GPU0 has only ~4G margin). **brokkr FINAL rec (thread wrapped 2026-07-13): go STRAIGHT to the staged bump — the tuple-diagnostic / tag-probe is a LONG-SHOT (the 110 natural closes prove the parser knows the delimiter → more likely a build/state bug than a tag-carry miss). **RESOLVED WITHOUT the bounce (brokkr msg16 deduction): the `deactivated (natural end)` log is the budget SAMPLER's namespace → the sampler matched its `reasoning_budget_end` token → end_tag NON-empty (110×); and per source `forced = tokenize(message+end_tag)` is built in the SAME block → forced also NON-empty. So the tuple is logically end>0/forced>0 → forced tokens EXIST, forcing just never FIRES = STATE-MACHINE bug → the post-#22740 BUILD BUMP. The `end=0` cheap-config branch is RULED OUT. NO diagnostic bounce needed (would only confirm) → no prod-seat-restart go required from Vuong for diagnosis.** XTC HELD. b61 = the immediate relief → NO rush. Deliverable owed to brokkr = the validated bump-target (loads Deckard Q5_K_M + passes tests/test-reasoning-budget.cpp AND — the real acceptance test — a **live low-budget check on the Qwen3.5/Deckard arch showing an ACTUAL forced-close, not just natural ends**, since a forced-close-that-never-fires is exactly what 8840 gets wrong); Dvalin standing by for a 2nd read, Eitri if arch specifics get hairy.** **VERIFICATION (2026-07-13, WebFetch ggml-org/llama.cpp — the build-bump has NO released target; the dwarf "bump to post-#22740" was WRONG, #22740 fixed a DIFFERENT regression #22717):** the real fixes are UNMERGED — #22684 (Qwen3.5/3.6 tool-call-in-`reasoning_content` = our TRIGGER) is **CLOSED-AS-NOT-PLANNED**; #24202 (Qwen tool-parsing) is **OPEN**; **#25544** (reasoning-budget sampler → MULTIPLE terminating sequences) is **OPEN, awaiting review** = OUR forcing fix. MECHANISM NAILED: 8840's budget sampler knows only a SINGLE end tag (`</think>`), but Qwen3.5's tool-path terminates reasoning with `<tool_call>` which the sampler can't match → forcing never fires on the tool path (explains 110 natural-`</think>` closes working + 0 forced + tool-retry specificity). **RE-FRAMED PLAN: no bump target exists yet → b61's 270s wall-clock is the ACTUAL mitigation (not just interim); TRACK PR #25544 and bump only when it merges into a release + validate the forced-close empirically. Seat XTC = optional loop-frequency fallback, not a fix.** **BUILT 2026-07-13 (operator directed the custom build after we verified NO released fix exists): image `llamacpp-charrp:6eddde0-pr25544`/`:custom-latest` on ana-ml2 (reports b10001, merged HEAD `6ce128d0b` = master 6eddde0 + PR #25544, CUDA 12.8.1 sm_120, server target `/app/llama-server`), binary smoke-tested OK. Reproducible recipe + records: eshpfi `stacks/char-rp-gguf/llamacpp-custom/` (build.sh + README) + auto-memory `charrp-custom-llamacpp-pr25544`. **SWAPPED + LIVE 2026-07-13 (operator: swap now)** — reasoning seat recreated on `llamacpp-charrp:custom-latest` (via new `LLAMA_REASONING_IMAGE` compose var; Magidonia char-rp stays stock); Deckard loads clean on Blackwell, serves coherent output, reasoning bounds at 400 (non-tool forcing intact). Per-request `reasoning_budget` override IGNORED (CLI-400 pins it, expected). New build's log DROPPED the `reasoning-budget: activated` line (b10001 format change) → can't watch forcing via seat log; DEFINITIVE tool-path proof = spend_logs `completion_tokens=32768` runaways STOP (baseline 4 today pre-swap). Rollback = set `LLAMA_REASONING_IMAGE=ghcr.io/mostlygeek/llama-swap:cuda` in `/opt/docker/compose/char-rp-gguf/.env` + `docker compose up -d --no-deps llama-charrp-reasoning` (8840 image d6c39f55 preserved). On-seat acceptance = Deckard loads + serves + a FORCED close fires at low budget (not just natural ends). #355 DEPLOY-SPEED task separately CONFIRMED landed: #359 merged→build-and-deploy 11m→~5m (~2×, wt-dev corroborated 5m/4m vs 11m); b61 pushed (v1.0.0b61 747d7a0) = 300s client backstop live on deploy.** (3) Optional interim = mild XTC on the seat (behavior-shaping, not enforcement) — holding unless the loop-frequency reduction is wanted pre-bump. Seat config: canonical `stacks/char-rp-gguf/compose.yaml` (service `llama-charrp-reasoning`), live `/opt/docker/compose/char-rp-gguf/` on ana-ml2. **Cheap `--reasoning-budget-message`-alone test is DEAD (source-confirmed won't fix it) — did NOT waste a restart on it.**
- **Operator direction:** find the SERVER-side fix (params/config), do NOT ceiling max_tokens (legit long outputs must survive). Routed to brokkr-smithy-dev (thread 01KXEEQ0TD6D…) → he accepted, pulled dvalin for the known-bug search, running the dwarf consult. I offered to run the live-seat with/without-tools split-test on his signal (own the seat; mildly disruptive → off-peak).
- **Wasted-GPU side issue + b61 re-measure:** LiteLLM logged these `success` at full 1255–1390s → on b60 it did NOT abort at WT's cancel (whatever WT did, no upstream abort → ~18min wasted GPU/runaway). worldtree-dev's **unpushed b61** adds a **270s provider wall-clock** that runs `_bounded_stream_close`→httpx FIN (force-closes WT's upstream side). So post-b61 the FIN-check flips to "WT closes at ~270s"; the only open question is whether **LiteLLM propagates that client-disconnect to the seat** (my gateway domain). **RE-MEASURE after b61 deploys** — don't assume b60 behavior.
- **Data on hand:** pcaps at `corviduo-dev:~infra-ops/py-spy-355/wedge_*/` are only ~99s windows triggered on the attach_tool failure (too short for the 300s cancel — a limitation, not usable for the FIN-check). LiteLLM config `/opt/docker/conf/litellm/config.yaml` → `/app/config.yaml`; spend_logs = `litellm-db` (`docker exec litellm-db psql -U litellm -d litellm`; camelCase cols double-quoted; request body in `proxy_server_request::jsonb->>'max_tokens'` top-level).

### 🔺 ACTIVE 2 — Worldtree deploy-speed: DELIVERED as PR #359 (green-lit, awaiting merge)
- **Real bottleneck was NOT uv sync (memory's assumption was WRONG).** Pulled the buildx STEP log (gitea `actions_log` on ana-docker, `docker exec gitea` → `/data/gitea/actions_log/vh/Worldtree/*/{task}.log.zst`, zstd). Sub-steps of the ~7min build: `#24 RUN useradd … chown -R /app /data … = **251.2s**` ← the monster; `#25 export image+push = 123.7s`; `#13 uv sync --no-cache = **only 35.6s**`; `#26 cache-export mode=max = 18.4s`. The runner-side registry LAYER cache (`--cache-from/--cache-to type=registry:buildcache,mode=max`) is **already wired in deploy.yml** — my prior "runner cache missing" note was STALE.
- **Root cause:** `chown -R worldtree:worldtree /app` forces BuildKit to copy-up the entire root-owned `.venv` (torch + 125 pkgs) into a fresh layer → ~4min + a duplicate venv baked into the image (also slows export/push).
- **Fix (PR #359, branch `infra/build-cache` off origin/main@b60):** (1) drop `/app` from the chown (keep `/data`); (2) add `--mount=type=cache,target=/root/.cache/uv` + drop `--no-cache`. **VALIDATED against BOTH live containers** (demo+personal): zero files written under `/app` post-boot (PID1-start ref) — all runtime writes → `/data/state`, `/data/kb`, `/home/worldtree`; `/app` is read-only for uid 1000, `/app/config` chowned at first boot by the root entrypoint shim. So the chown was pure waste. Expected **~5min off** (~11→~6min).
- **Status:** worldtree-dev GREEN-LIT (confirmed `--no-cache` not deliberate; b61 is theirs, core/llm-only, zero Dockerfile overlap). PR open + mergeable; they review, merge→main triggers the real build that measures the actual speedup. Deploy trigger unaffected (Dockerfile change → builds). **Done from my side.**

### Landed this session (2026-07-13) — context for the fresh session
- **WT #355 turn-lifecycle fix VALIDATED** (worldtree b60): wedged turns self-terminate cancelled/stalled at the 300s watchdog (turns 2064/2065 = 302s/360s) vs pre-b60 turn 2061's 16-min no-terminal silence. The residual (ACTIVE 1) is a separate LiteLLM matter.
- **Ledger tier-3 provisioned** (`ledger:miranda`) on personal `:8081`: key minted (user `ledger`, key_id `b38932f5`), GPG-delivered + verified + cleartext shredded; scope confirmed (user tier has `bifrost:invoke`+`agents.define`); allowlist `10.100.10.50:8770` activated by the b60 recreate.
- **`assistant`+`thoughtful-assistant` roles** added (bind `gen`/`gen-reasoning`, kind capability, grants `[foundational]`) on personal+demo; canonicalized by worldtree-dev (`d8bd497`).
- **3 zombie soong sessions retired** (2ad34ec6+7d23052f+0775c9b1 via `POST /admin/sessions/<id>/retire`); pre-b60 stale-active backlog HELD (worldtree's #356 rehydrate fix reclaims that class).

### Still-open standing items (carried)
- **T1 SFT LoRA (AEON-27B E-RP)** — adapter banked but BLOCKED on hot-swap serving; DORMANT (operator pivoted to off-the-shelf). Merge fallback primed (~2-3h) if hot-swap abandoned.
- **Backups** — rotate the 5 disclosed rest-server creds (operator, offline); confirm cross-site restic mirror.
- **althing monitor dies on /clear** — re-arm `/althing:monitor` in the fresh session (open await-reply loops: brokkr on the #355-residual, worldtree-dev on the deploy-resilience follow-ons #356/#357/#358).

## Recent decisions

- `[2026-07-13]` **#355-residual ROOT CAUSE (supersedes the "LiteLLM gateway holds while seat idles" entry below — that was DISPROVEN).** char-rp-reasoning enters a non-terminating REASONING loop (tool-call-retry planning) and runs to `max_tokens=32768` (~22 min @ 24.7 tok/s, ~13% of requests); the seat GENERATES all 32768 tokens (not idle), and `--reasoning-budget 400` is NOT enforced. 3-source-confirmed (spend_logs completion_tokens=32768 ×4; seat eval-time log; pcap 100%-`reasoning_content` deltas). Server-side fix wanted (operator: no max_tokens ceiling) → routed to brokkr (accepted, pulled dvalin). Lesson (again): confirm before concluding — the seat-idle claim came from reading only the ≤73s requests + missing the concurrent 32768-token slots. See ACTIVE 1.

- `[2026-07-13]` **Deploy-speed real bottleneck ≠ uv sync (memory's assumption was wrong).** Buildx step log: `chown -R /app` = 251s (copy-up of the root-owned venv into a fresh layer), uv sync only 35.6s, registry layer cache already wired. Fix = drop `/app` from the chown (validated safe: zero /app runtime writes on both live instances) + uv cache-mount. Shipped as PR #359 (branch off origin/main@b60), worldtree-dev green-lit. Expected ~5min off (~11→~6min). Runner-side BuildKit cache task (b) was already done → moot.

- `[2026-07-13]` WT #355 residual 300s hang localized to OUR LiteLLM gateway (holds 2 char-rp-reasoning requests ~21 min while the seat idles), NOT the seat — Deckard seat EXONERATED (completes ≤72s; `--reasoning-budget 400` forecloses a mid-thinking hang). Corrects worldtree-dev's "seat wedging" diagnosis. Decisive next = the FIN-check (pcap on corviuo). See in-flight ACTIVE 1. **[SUPERSEDED 2026-07-13 — see the ROOT CAUSE entry above; the gateway-hold/slot-leak theory was disproven, the seat was generating 32768 tokens.]**

- `[2026-07-13]` WT #355 turn-lifecycle fix VALIDATED on worldtree b60 — wedged turns self-terminate cancelled/stalled at the 300s stall-watchdog (turns 2064/2065 vs pre-b60 2061's 16-min no-terminal). worldtree-dev filed follow-ons #356 (rehydrate Tier-3 ctx on resume — the recreate-durability gap), #357 (reclaim orphaned active-turn locks), #358 (LLM-provider read-timeout audit); surfacing to Vuong to prioritize.

- `[2026-07-13]` Worldtree deploy bottleneck = the image build (~11 min of a ~12 min deploy), root cause the Dockerfile `uv sync ... --no-cache` + no BuildKit cache-mount (re-downloads all deps cold every build). Fix split: worldtree-dev Dockerfile cache-mount diff + infra-ops runner-side persistent BuildKit cache. Config-only changes skip the build entirely (pinned recreate).

- `[2026-07-13]` Ledger tier-3 consumer `ledger:miranda` provisioned on personal :8081 (key b38932f5, GPG-delivered+shredded, allowlist 10.100.10.50:8770 live); `assistant`+`thoughtful-assistant` capability roles added (gen/gen-reasoning) on personal+demo, canonical d8bd497. Chosen instance = personal (the tier-3-consumer instance, ratatoskr+soong-lab colocated).

- `[2026-07-10]` **Heimdall grant: ratatoskr `affect.full` on PERSONAL Worldtree (operator-approved, worldtree-dev
  R34-v1 request).** Added allow-rule `ratatoskr-affect-full-allow` to `/opt/worldtree-personal/config/policies.yaml`
  (`principal.user_ids:["ratatoskr"]`, action `affect.full`, resource `*`, effect allow), mirroring the #347
  `session-history-write-ratatoskr` rule exactly + placed right after it. **WHY user_ids-based (not tier):** ratatoskr's
  personal key is the minimal **readonly-admin** observability tier, which is NOT in the tier-based
  `affect-render-baseline-allow` (anonymous/user/free/pro/admin) → needs an explicit user_id grant, same as #347.
  R34-v1 (b46, committed UNPUSHED) gates `affect.emit` `dominant_emotion` egress by exposure ceiling (affect.full|safe
  → present; neither → null); this grant keeps ratatoskr's view alive across the b46 deploy. Surgical exact-string
  insert (preserves comments), backup `policies.yaml.bak-pre-affectfull-20260710`, validated via the CONTAINER's own
  yaml parser (35 rules, +1, payload confirmed). **NOT restarted — deliberate:** rule is on the bind-mount (shadows
  baked), INERT until b46 gating ships, so the b46 CI/CD deploy restart activates it (no live-session blip now). Demo
  untouched (personal-only per key scope). Replied to wt-dev (thread 01KX6DB3…) offering an immediate restart if they
  want it live for pre-b46 testing. See [[reference_corviduo_dev_emergency_ops]].

- `[2026-07-10]` **ComfyUI v0.27.1 SUCCESS on irv-ml1 (operator-confirmed execute-now) — landed on torch 2.12.1,
  SageAttention preserved, crash-loop AVOIDED.** The prior attempt (entry below) crash-looped because a torch
  2.12.1→2.13 bump broke SageAttention's ABI. This time I checked `git diff v0.24.1 v0.27.1 -- requirements.txt`
  FIRST and found **core v0.27.1 leaves `torch` UNPINNED** → the version bump does NOT require torch 2.13 (that came
  only from the mmartial boot-upgrade). So: `git checkout v0.27.1` (clean tree) → `pip install -r requirements.txt`
  as **uid 1000** with a **torch-pin constraint file** (torch/vision/audio pinned to current +cu129) to block any
  transitive bump → torch stayed 2.12.1, SageAttention 2.2.0 untouched. Added decord 0.6.0 (fixed SAM3Segment).
  `docker restart comfyui` → healthy, `/system_stats` comfyui_version=0.27.1, "Using sage attention", HTTP 200, DB
  migrated 0003→0004. Reported the divergence to comfy-dev (thread 01KX6D3C…, reply pending) + asked whether LTXV 2.3
  needs a separate torch-2.13 follow-up (their domain; Krea-2's ≥0.25 need is met by 0.27.1). **LESSON: before a
  mmartial ComfyUI version bump, `git diff <old> <new> -- requirements.txt` — if torch is unpinned, bump the CODE
  without touching torch (constraint-pin it) and compiled exts (SageAttention) survive. `docker exec` lands as uid
  1025(comfytoo), not 1000 — use `-u 1000` + the venv python `/comfy/mnt/venv/bin/python`.** See
  [[reference_irv_ml1_comfyui_mmartial]].

- `[2026-07-10]` **ComfyUI 0.25.x bump on irv-ml1 ATTEMPTED → FAILED → ROLLED BACK (snapshot saved it).** comfy-dev
  requested (Vuong-authorized) bumping the irv-ml1 `comfyui` stack (mmartial image, `/opt/docker/compose/comfyui/`,
  0.24.1) to 0.25.x for Krea-2 + LTXV 2.3. **TWO FINDINGS: (1) `DISABLE_UPGRADES=false`/USE_PIPUPGRADE bumps the
  VENV (torch 2.12.1→2.13.0 + deps) but does NOT advance the ComfyUI CODE checkout** (`/comfy/mnt/ComfyUI` =
  `/worktank/comfyui/run/ComfyUI` stayed 0.24.1 — pinned/detached git, comfy-dev's domain). **(2) the torch bump
  broke SageAttention** (2.2.0 `_fused.so` undefined-symbol `c10::impl::cow::materialize_cow_storage` vs torch
  2.13.0) → `--use-sage-attention` (REQUIRED launch flag in COMFY_CMDLINE_EXTRA) crash-looped ComfyUI. Net: broke
  the working state, zero 0.25.x payoff. **ROLLBACK WORKED**: pre-bump 16G venv snapshot
  `/worktank/comfyui/venv-snapshot-comfyui-0.24.1-20260710.tar` restored (torch 2.12.1 + working SageAttention),
  re-pinned DISABLE_UPGRADES=true, recreated → healthy on 0.24.1, serving :8188. Broken venv parked at
  `/worktank/comfyui/run/venv.broken-torch213-20260710`. **CORRECTED PATH (sent comfy-dev, thread 01KX655V):**
  comfy-dev git-advances the ComfyUI checkout to 0.25.x + reqs → I handle the torch bump + SageAttention
  rebuild-against-2.13.0 + re-pin (snapshot stays as the net). **LESSON: mmartial `DISABLE_UPGRADES` gates ONLY
  the venv pip-upgrades, NOT the ComfyUI git checkout; a torch bump breaks compiled exts (SageAttention) →
  rebuild-after is mandatory.** Bump BLOCKED pending comfy-dev's git-advance. Stack: A6000 (NVIDIA_VISIBLE_DEVICES=1),
  lkraven-owned compose+venv (uid 1000, no sudo needed), COMFY_CMDLINE_EXTRA OOM flags preserved.

- `[2026-07-10]` **Biweekly open-weight-releases scan cron set up for brokkr-smithy (Vuong-authorized).** Durable
  systemd **--user** timer on nh3-dev (`brokkr-landscape-scan.timer`, OnCalendar `*-*-01,15 09:00:00`
  America/Los_Angeles, Persistent=true; linger on) → `.service` → wrapper `~/.local/bin/brokkr-landscape-scan.sh`
  runs headless `claude -p "$(cat ~/.config/brokkr-landscape-scan/prompt.txt)" --dangerously-skip-permissions` in
  `~/development/brokkr-smithy` (ALTHING_HANDLE=brokkr-smithy-dev; **explicit PATH** — the --user minimal-PATH
  footgun; per-run logs `~/.local/state/brokkr-landscape-scan/`). Prompt = brokkr's payload verbatim (LLM/image/TTS
  new-release sweep → ranked synthesis → commit+push+notify). VALIDATED: git-push non-interactive (BatchMode
  ls-remote to gitea, passphraseless key — no agent), headless claude auth (READY smoke). VALIDATED END-TO-END 2026-07-10 (manual run, exit 0):
  web-sweep→synthesis→commit `2ed2f29`→PUSH of scan #2 (open-weight-releases-2026-07-24.md); triaged dwarf input +
  caught baseline errors, quality strong. **HANDLE-COLLISION caught+FIXED** — the headless scan shared handle
  brokkr-smithy-dev with the LIVE session + raced its inbox (eitri's dwarf-reply got stolen by the live monitor);
  registered a dedicated **brokkr-scan-dev** handle (`add-handle`, driver=none) + repointed the wrapper + rewired
  step-5 notify → `althing-cli post --to brokkr-smithy-dev` (NO vuong althing handle exists — confirmed). model=default
  + `--max-turns 80`. First run under the new handle = 7/15. Off-cycle 07-24 doc is a validation artifact (scheduled
  1st/15th runs date to their own run-date, no collision) — operator naming-convention call pending.
  **NEXT AUTO-RUN 2026-07-15 09:00 PDT.** Manual validation/first run = `systemctl --user start
  brokkr-landscape-scan.service`. Open w/ brokkr (thread 01KX63G6): confirm notify-Vuong handle/mechanism + session
  handle + model/turn-cap. **NEXT brokkr task (operator-sequenced after this): TTS audition env** — Higgs-TTS-3 +
  ZONOS2 + Chatterbox baseline, TTFA/RTF + blind-A/B web-listen (thread 01KX6371; needs GPU-placement + HF-token
  feasibility pass first; brokkr delivers the prompt set after the env's up; protocol doc in brokkr-smithy repo).

- `[2026-07-09]` **Two parked items closed: phantom `qwen3.6-35b-a3b` alias VERIFIED already-gone; ana-docker
  docker log-cap SOLVED no-bounce.** (1) **Phantom**: absent from `/v1/models` + `/model/info` (config+DB
  registry), zero litellm log refs — the parked "400s in /v1/models" note was STALE (already cleaned in the
  2026-07-08 gen repoint to `-heretic`); bare token survives only in 2 config COMMENTS (lines 76/80). Nothing to
  remove. (2) **Log-cap**: running containers were UNCAPPED (182M json-logs, top offender 59M) because
  daemon.json's `max-size 10m/max-file 3` only applies to containers CREATED AFTER a daemon restart — it never
  reaches already-running ones. No-bounce fix = `/etc/logrotate.d/docker-containers` (**copytruncate** — dockerd
  opens json-logs `O_APPEND` so truncate-in-place resets cleanly, no sparse-file corruption; `size 10M`,
  `rotate 3`, `compress`, `su root root`), auto-picked-up by the daily `logrotate.timer`. Force-ran + gzipped the
  frozen `.1` archives → **182M → ~55M** (44M active + 11M gz), every container kept its multi-week uptime
  (zero bounce, verified). **LATENT FOOTGUN FLAGGED (not yet fixed, operator's call): daemon.json declares
  `live-restore:true` but the RUNNING daemon has it FALSE** (daemon.json was edited after the last daemon start,
  never reloaded) → the NEXT `systemctl restart docker` / crash / pkg-upgrade **bounces ALL ana-docker containers
  once**. Fix WITHOUT a bounce = `systemctl reload docker` (SIGHUP loads live-restore into the running daemon;
  log-opts are NOT SIGHUP-reloadable, which is why logrotate — not the daemon cap — is the enforcer for running
  containers).

- `[2026-07-09]` **granite→gen `memory_extractor` bind host-synced on demo+personal Worldtree (Vuong-directed,
  #335 Slice-4).** Changed `model_roles.yaml` memory_extractor `binds.catalog_id` `summarizer`→`gen` (overrides
  intact: thinking:false/temp0/8192) on BOTH `/opt/worldtree{,-personal}/config`; `memory_distiller` left on
  `summarizer` (range-scoped sed `/memory_extractor:/,/memory_distiller:/` — the naive global replace would've hit
  both); backups `*.bak-preqgen-20260709`; validated via each container's OWN yaml parser. **DEMO LIVE on gen**
  (b43 `d501e516732d` auto-deployed mid-edit + its restart RACED my edit by ~2min → I restarted
  `worldtree-worldtree-api-1` to activate; healthy, live process resolves memory_extractor=gen). **PERSONAL
  NOW LIVE on gen too** — Vuong authorized the restart (via wt-dev); restarted `worldtree-personal-worldtree-api-1`
  after a PRE-FLIGHT that ran the app's OWN `load_model_roles()` (`core/llm/roles.py:121`) against the synced config
  INSIDE the running `c9986cd` container: `gen` in catalog, all 9 roles resolve, no `DanglingBindingError` → proven
  safe on the OLDER image BEFORE touching it (model_roles-delta-alone clean; no full-config-set sync needed).
  StartedAt 20:50:55Z, healthy, resolves gen. **BOTH instances live on gen.** **LESSON:
  the bind-mount `/opt/worldtree*/config` SHADOWS the baked `/app/config-defaults/` → the deploy alone never
  updates the ACTIVE config; the host edit is required AND a restart activates it (role registry cached at boot) —
  pre-stage BEFORE the deploy's restart or you race it.** FOLLOW-UPS (non-blocking): (a) `memory.extractor.user_pass`
  parity block → self-serve from the b43 baked `defaults.yaml` (pydantic-default no-op); (b) stale `gen` provider
  description (Mistral-Small-4 → qwen3.6-35b-a3b-heretic) → wt-dev owns the REPO-side `providers.yaml` fix
  (operator's call — NOT purely cosmetic: the gen/dialogue + classifier entries carry Mistral-Small-4 SAMPLING
  defaults that drive mask/lofn/forseti/mimir dialogue, so wt-dev re-validates for qwen), host cosmetic sync pairs
  when it deploys. Gave wt-dev the VERIFIED canonical 4-alias set (backends+samplers read from the LIVE gateway
  config, not the doc); corrected `docs/pfi/model-sampler-defaults.md` seat 4 (had lagged QwQ-RpR-v4 → Deckard-PKD;
  live gateway was always Deckard). Operator SCOPED IN the character-RP re-point (2026-07-09):
  character→char-rp / thoughtful-character→char-rp-reasoning (character-rp per wt-dev's role semantics), moving
  character RP off the GENERAL qwen onto the dedicated Magidonia/Deckard seats. Relayed to wt-dev w/ the mapping
  principle + a SAMPLER-OVERRIDE warning (DROP character-rp's old temp0.75/top_p0.85 overrides — carried onto the
  dedicated seats they'd clobber the canonical RP tuning DOWNWARD) + ratatoskr-reach note (role call is transparent
  but Magidonia/Deckard quality/latency differs from gen). DONE 2026-07-09: wt-dev committed 5d4fa4a (v1.0.0b44,
  UNPUSHED — operator drives push); operator directed host-ahead-of-push, so I sourced BOTH config files directly
  from that unpushed commit (local `~/development/Worldtree` checkout — capital W; `git show 5d4fa4a:config/…`) +
  mirrored VERBATIM to `/opt/worldtree{,-personal}/config` on demo+personal, paired-pre-flighted via the app's
  `load_model_roles()` (no DanglingBinding), restarted both → LIVE: character→char-rp, thoughtful-character +
  character-rp→char-rp-reasoning, memory_extractor→gen preserved. Backups `*.bak-prerp-20260709`. context_window
  VERIFIED (llama.cpp /props + char-rp-gguf `.env`): char-rp **131072**, char-rp-reasoning **262144** (gave wt-dev
  to patch the repo from its interim 32768). **HOST AHEAD of repo-remote until the operator pushes 5d4fa4a** (baked
  config converges with the bind-mount on push+deploy). GOTCHA: demo≠personal — PERSONAL was already partly
  re-pointed (2026-07-06 AEON-era character→char-rp) so its delta was mostly stale-AEON-descriptions→Magidonia/Deckard
  + character-rp + stripping personal's char-rp `default_params` temp0.7/top_p0.8 that CLOBBERED the gateway RP tuning
  downward; DEMO had no char-rp catalog entries at all (b44 adds them). Diffed each instance vs b44 before applying
  (both deltas = expected changeset only, nothing instance-specific clobbered). wt-dev PATCHED the context_window in **b45/3384a37**
  (char-rp 131072, char-rp-reasoning 262144). BUMPED HOST-AHEAD on both instances (operator-directed 2026-07-09):
  mirrored b45's providers.yaml → `/opt/worldtree{,-personal}/config`, restarted, verified LIVE (char-rp ctx
  131072, char-rp-reasoning 262144, bindings intact); backups `.bak-b44interim-20260709`. HOST now = **b45
  canonical** (providers.yaml) + b44 (model_roles unchanged b44→b45). STILL PENDING: (1) operator's batched push
  of **b44+b45** (`5d4fa4a`+`3384a37`) to converge the repo-remote — host is ahead, no fork; (2) user_pass parity
  block (defaults.yaml — NOT in either, separate). Threads `01KX3SGH`/`01KX48QP` (worldtree-dev),
  monitor armed. See [[reference_corviduo_dev_emergency_ops]].

- `[2026-07-09]` **mOrpheus TTS off-the-shelf voice pipeline SHIPPED end-to-end (irv-ml1) + wired into
  gateway-chat.** Full arc (commits): gen served-name honesty rename aeon→qwen3.6-35b-a3b-heretic (99a4a17,
  vLLM served-name + litellm refs, so /v1/models + spend-logs name the real model); permanent 2-container stack
  (01eedd8); gateway-chat auto-voice quoted dialogue (c948013); streaming decode TTFA 4.5s→0.8s (da76829);
  max_tokens 1200→2400→3500 with a context-clamp (f363fe6, 0655a37 — long lines were clipping at 14.6s, and
  `repetition_penalty` 1.1 is LOAD-BEARING: at 1.0 the model never stops); AudioContext resume-on-gesture
  no-sound fix (033f368); pre-chunk by QUOTED SECTION not sentence for prosody (a1f3023→f295cc1); staged clone
  voices baddy/beatrice/whisper (0655a37 + runtime .wav/.txt in the voices dir); agent voicing prompt (a573514).
  **Load-bearing config, all encoded in stacks/mOrpheus/: bf16 not FP8, image v0.23.0 not latest, GPU=3090 not
  A6000, rep_penalty 1.1.** Serving-viability confirmed: vLLM concurrency near-linear to 8× (707 tok/s).

- `[2026-07-09]` **granite→gen memory_extractor bind GREEN-lit for worldtree-dev (Worldtree #335 Slice 4).**
  Answered their VRAM/concurrency headroom check: gen (qwen 35B-A3B heretic) has ample headroom for ~2 bursty
  idle-triggered extractor calls (fixed 0.37 util; KV pool runs 0-2%; --max-num-seqs 16, near-linear batching).
  Corrected their stale "gen = Mistral Small 4 / 119B-6B" belief (gen IS the qwen 35B-A3B heretic since 2026-07-08).
  **This bind is INDEPENDENT of the full granite RETIRE** (reclaim ~32GB on ana-ml2 GPU1) — that stays the
  operator's call, pending brokkr R33 portfolio + production-concurrency due-diligence. Tracked: althing thread
  01KX3SGH… (worldtree-dev) + brokkr's gen-absorbs-granite consult (thread 01KX2V32…) + [[reference_litellm_gateway]].

- `[2026-07-08]` **RP-SEAT CAMPAIGN CLOSED — char-rp = Magidonia-24B-v4.3 (128K), char-rp-reasoning = Deckard-PKD
  Qwen3.5-27B (256K); both GGUF/llama.cpp on ana-ml2 GPU0 alongside gen (35B-A3B, util 0.37), ~4G GPU0 margin.**
  Arc: (1) replaced broken Angel NVFP4 with Magidonia prose + QwQ-RpR-v4 reasoning (b268f93); (2) max-context via q8_0
  KV (f570604); (3) canonical samplers for all 4 gateway seats, dvalin-derived + char-rp A/B-tuned (aac4bcf);
  (4) rebalanced gen 0.40→0.37 to fund char-rp 128K (f49c4e4); (5) RE-A/B'd the reasoning seat (operator wanted a
  DRY-tolerant model): **Deckard WON** on brokkr's frozen scorer (composite 2.176, 0/30 loops, 0/30 refusals) over
  RpR-v4 (3.716, 1/30 loop), Pantheon-Reasoning (1.383 but 7/30 refusals), Snowdrop+Gembrain (llama.cpp
  template-incompat) — deployed (5f79b40); (6) Deckard→256K (41305bf); (7) dvalin CONFIRMED Deckard samplers = the
  live A/B set is canonical (4954ca0). **GATE LESSON: a llama.cpp reasoning seat needs a STOCK template that natively
  opens `<think>`/`enable_thinking` (Qwen3.x/QwQ pass; ChatML + Gemma-4 fail) — no monkeypatching. INFRA: llama-swap
  b8840 can't load Qwen3.6/Gemma-4 archs → `ghcr.io/ggml-org/llama.cpp:server-cuda` (pulled on ana-ml2).**

- `[2026-07-08]` **worldtree Mimir deploy-blocker resolved (mid-session):** synced `persona.envelopes.assistant` +
  `envelope_grants:[]` verbatim from the baked canonical into BOTH corviduo-dev instances (demo+personal),
  YAML-validated via each container's own parser; worldtree-dev cleared to push the Mimir-bound image. (Was my
  parked R32 1C envelope-mirror come due — see [[reference_corviduo_dev_emergency_ops]] config-sync recipe.)

- `[2026-07-08]` **OFF-THE-SHELF INFERENCE PIVOT executed — serve curated abliterated models, stop home-training.**
  Final topology: **gen = `llmfan46/Qwen3.6-35B-A3B-uncensored-heretic-NVFP4-Experts-Only`** (LIVE, modelopt, vision,
  util 0.40), **char-rp = an RP unicorn to be found on fresh context** (see Current state). Intermediate steps
  ABANDONED: Pantheon-Reasoning-27B (served briefly as gen — refuses dark fiction via DeepSeek-distilled
  refusal-reasoning, see Tried); Pantheon-27B-with-MTP for RP (bf16 MTP won't load on the compressed-tensors path);
  Angel MS3.2-24B (my NVFP4 quant = garbage). Prefer EXISTING community NVFP4/GGUF quants over self-quanting
  ("don't quant unless you have to" — operator). GGUF serving is now on the table for RP (NEVER Ollama). Gateway
  sampling-defaults wiring still PENDING.

- `[2026-07-08]` **DPO was silently running 3 epochs (harness gap) → KILLED at epoch 1.2, retargeted to 0.3
  epochs (operator call).** Root cause: `DpoConfig` had NO `epochs` field + `_dpo_config_kwargs` didn't pass
  `num_train_epochs` → DPO fell through to trl DPOConfig's default 3.0 (SFT correctly pins 1 via SftConfig.epochs
  + _sft_config_kwargs). Objective SATURATED by ~epoch 0.27 (loss~0, grad~0, acc 1.0, margins~27 flat — the
  off-policy qwopus rejected pairs are trivially separable), so epochs ~0.3→3 were pure over-optimization + a
  ~6.5h outage. No mid-run checkpoint (save_steps 500 > total steps; save only at end) → killing lost the run.
  FIX (3 edits to deployed harness, mtf-dev to canonicalize): `DpoConfig.epochs: float = 1` (mirrors SftConfig,
  float for fractions); `_dpo_config_kwargs` now passes `"num_train_epochs": cfg.epochs`; recipe `dpo.epochs: 0.3`.
  GPU-free verified (dpo.epochs=0.3 → num_train_epochs=0.3). Relaunched at 0.3 epoch (~30min precompute + ~12min
  train = ~45min). **DONE + SANITY-CHECKED (exit 0, ~70min wall: ~30min fixed precompute + 45 steps @ ~51s/step;
  train_loss 1.4e-5 @ epoch 0.301).** Fresh `data/spike/dpo_adapter/adapter_model.safetensors` (123MB) + checkpoint-45
  banked. **3-way greedy sanity (base vs SFT vs DPO, via peft load + disable_adapter/set_adapter on GPU0):
  ALL THREE DISTINCT** (base≠sft≠dpo) → full SFT→DPO pipeline applies end-to-end at inference. **DPO 0.3ep is
  COHERENT, fluent, NOT degenerate** (early-stop avoided over-optimization) but the quality delta on a neutral
  literary prompt is SUBTLE (DPO shares SFT's structure — it continues from it — with minor stylistic drift,
  arguably slightly MORE genre-clichéd). Verdict: mechanics proven, quality gain modest as predicted for 0.3ep
  on off-policy pairs; the real unlock remains on-policy rejected regen + on-domain (explicit E-RP) eval +
  the LitBench/holdout run. gen+rp RESTORED healthy. Next: serve fork (SGLang-finish vs merge) on the DPO
  adapter — same rekey_lora_for_vllm.py (zero-z) applies unchanged (mtf-dev confirmed).

- `[2026-07-08]` **T1 DPO leg is RUNNING (unblocked) — 2 fixes applied to deployed backend.py.**
  Blocker resolved: (1) **mtf-dev's v0.0.42 stub** `_stub_missing_optional_integrations` (last-resort sys.meta_path
  finder → missing mergekit/llm_blender/weave resolve to MagicMock, never called → zero numerics risk; applied
  VERBATIM to deployed `src/model_training_forge/train/backend.py` after `_unsloth_available()` + call-site before
  `from trl import DPOTrainer`); (2) **my cosmetic `warnings_issued` shim** (trl-0.24 DPOTrainer.__init__:405 does
  `model.warnings_issued["estimate_tokens"]=True` for warning-suppression; custom Qwen3_5 class under transformers
  5.5.0 lacks the attr → `if not hasattr(model,"warnings_issued"): model.warnings_issued={}` before the
  DPOTrainer(...).train() at backend.py:305 — cosmetic, zero training impact). Both edits are on the DEPLOYED
  un-git'd copy only → **mtf-dev must canonicalize the warnings_issued shim into their repo** (told them). DPO
  confirmed training: model loaded (851 shards), full 1196 pairs processed, in precompute_ref_log_probs (GPU0 93%
  util, 54.8GB). Completion watcher armed (bg task) → restore gen+rp + verify dpo_adapter + ping mtf-dev on exit.
  gen+rp STOPPED for the run (authorized window). Output → data/spike/dpo_adapter.

- `[2026-07-08]` **T1 DPO leg launch — prior BLOCK (now resolved above), kept for the launch recipe.**
  Operator authorized the full DPO stage (via mtf-dev) + went AFK 2h. **PROVEN LAUNCH RECIPE** (replicates the
  SFT container `aeon-t1-sft` exactly, only `--stage sft`→`dpo`): `sudo docker run -d --name aeon-t1-dpo
  --entrypoint python3 --gpus all -e CUDA_VISIBLE_DEVICES=0 -e MTF_FORCE_TRL=1 -e PYTHONPATH=/mtf/src
  -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True -v /home/lkraven/model-training-forge:/mtf -v /tank:/tank
  -w /mtf aeon-trainer:latest -u scripts/train.py --recipe recipes/training/qwen-3.5-122b-erp-lora/train.aeon-27b.yaml
  --stage dpo`. **CRITICAL: `--entrypoint python3` is REQUIRED** — aeon-trainer's default entrypoint is
  `["vllm","serve"]` (FROM vllm/vllm-openai) → without the override it runs vllm + hits a torch-ABI crash.
  Dataset verified (pairs_dataset=train.flat.json=1196 pairs). **THE BLOCK:** `from trl import DPOTrainer`
  (backend.py:256) eagerly pulls TRL 0.24.0's WHOLE optional-integration set — cascade: mergekit(missing)→
  immutables→**mergekit-0.1.4↔pydantic-2.13 HARD incompat** (needs pydantic==2.10.6)→llm_blender→dataclasses_json→
  **llm_blender-0.0.2↔transformers-5.5.0 HARD incompat** (TRANSFORMERS_CACHE removed, needs source patch)→weave→
  (more). NONE used by our pair-based DPO. `pip install mergekit` w/deps is UNSAFE (downgrades accelerate
  1.14→1.6). Safe partial recipe derived (core libs held: torch2.10/tf5.5.0/trl0.24.0/peft0.19.1/accel1.14.0)
  but non-convergent → TRULY BLOCKING per operator's carve-out. Did NOT force-hack the proven training image.
  Handed full diagnosis + recommended fix (lazy-import TRL patch, opt b) to mtf-dev (thread 01KWZG8GJX,
  expects-reply, monitor armed). gen+rp RESTORED healthy. Relaunch = 1 min once mtf-dev delivers a working image.

_142 older entries archived to archival-memory.md._

## Tried and abandoned

- `[2026-07-13]` Relaying a peer's diagnosis as fact without confirming it against raw data. worldtree-dev diagnosed the WT #355 residual as "our llama.cpp seat wedging," which I echoed in a wrap-up; the operator challenged it and the seat logs DISPROVED it (seat completes ≤72s, idle at the wedge onset — the hang is the LiteLLM gateway). Lesson: CONFIRM peer diagnoses (esp. cross-domain ones) before acting/relaying — same discipline that caught the earlier char-rp-reasoning red-herring via a live `registry.resolve` reproduction.
- `[2026-07-13]` `althing-cli reply <THREAD_id>` (thread id, not a MESSAGE id) → "unknown message_id"; and `reply` to your OWN message self-addresses to your handle ("replying to your own message"). Reply to a PEER's message id, or use `post --to <peer>`. Bit me several times this session.

- `[2026-07-09]` **FP8 breaks mOrpheus audio-token generation.** `--quantization fp8` on the 3B → 0 valid SNAC
  frames even at GREEDY (degenerate audio+text mix, no start-of-speech); bf16 is clean (28/28 frames). Quant-breaks-
  TTS, same class as the Angel-NVFP4 lesson below. bf16 is REQUIRED (so the operator's "util 0.1" wish is moot — the
  bf16 weights alone are 6.6GB). NB the raw-token benchmark RTF 0.50 was fp8+graphs = never real.
- `[2026-07-09]` **`vllm/vllm-openai:latest` crashes on Ampere IMPORT** — Blackwell-only kernels (oink/aiter,
  `has_device_capability(100)`) die during import on the 3090/A6000. Pin **v0.23.0** on irv-ml1's Ampere GPUs.
  (`vllm/vllm-omni:v0.18.0` has a different entrypoint — don't use it either.)
- `[2026-07-09]` **Per-frame CPU SNAC decode is too slow for streaming** — per-call overhead × ~60 frames serialized
  → RTF 2.2 (WORSE than whole-clip's 1.0). Fix = **windowed chunk decode** (every 6 frames decode a [2 ctx | 6 | 2 ctx]
  window, emit the middle 6 → seamless, O(1)/frame, RTF ~0.97, TTFA ~0.8s).
- `[2026-07-09]` **Sentence-chunking TTS loses prosody** — generating each sentence cold flattens the intonation that
  spans a line. Chunk by QUOTED SECTION (whole quote = one gen call). Also: `repetition_penalty` >1.1 BREAKS cloning
  (penalizes the ~1100 in-context reference audio tokens; keep ≤1.1 on the clone path).
- `[2026-07-09]` **HF whisper datasets aren't actually whispered.** Claris-Whispered-English measures voiced 0.8
  (not a whisper) + IPA transcripts; `datasets` audio decode needs torchcodec (wants CUDA-13, incompatible w/ the
  cu124 venv). LPC DSP-whisperize went unstable (NaN). **kokoro `af_nicole` IS a genuine whisper** (voiced 0.24) —
  that (operator's pointer) is the working whisper reference source, not TTS-voice screening or dataset-hunting.
- `[2026-07-08]` **Angel (allura-org/MS3.2-24b-Angel) self-quanted to NVFP4 = GARBAGE.** llm-compressor W4A4 NVFP4
  (compressed-tensors, MLP-quantized, attn/vision bf16) of the Mistral3 dense 24B produces gibberish EVEN AT GREEDY
  (temp 0) → the quant itself is broken, not the tokenizer or sampler. Same recipe worked on the qwen models.
  Mistral3 + W4A4 NVFP4 via llm-compressor is bad. → for the RP seat, going **GGUF (llama.cpp)** to sidestep the
  whole NVFP4-quant surface.
- `[2026-07-08]` **Mistral3 + vLLM tokenizer/vision traps (serve `MS3.2-24b`, vLLM 0.24).** (a) HF `tokenizer.json`
  for Mistral = **GARBAGE output** — the card's "use the official Mistral tokenizer" warning is REAL; must use the
  `tekken.json`/mistral tokenizer. (b) BUT `--tokenizer-mode mistral` + vision **CRASHES** (`Failed to apply
  PixtralProcessor on {'text': '[IMG]'}`; and with tekken.json present in auto mode, `CachedMistralCommonBackend has
  no attribute is_fast`). So it's **mistral-tokenizer OR vision, not both** on this vLLM. Text-only + mistral
  tokenizer serves clean (`--limit-mm-per-prompt '{"image": 0}'`). **GGUF/llama.cpp avoids all of this** (native
  mistral tokenizer + vision).
- `[2026-07-08]` **Pantheon-Reasoning-27B refuses dark fiction DESPITE an abliterated base.** The base
  (`llmfan46 heretic`) writes freely (thinking-off), but Gryphe distilled the reasoning traces from **DeepSeek 3.2**
  (safety-aligned) onto every turn (`preserve_thinking:true`) → the model reasons ITSELF into refusals in the
  `<think>` phase (collapses to empty output). Fix: thinking-off OR an uncensor system prompt (both verified).
  **Lesson: a reasoning finetune of an abliterated base can re-censor via its reasoning-trace TEACHER; the raw
  abliterated base is cleaner** — this is WHY the pivot went to the llmfan46 heretic base for gen.
- `[2026-07-08]` **Pantheon-27B MTP on vLLM compressed-tensors = 0% acceptance.** MTP is a separate **bf16** head
  (`mtp.*`, in `model-auxiliary.safetensors`, 15 tensors); AEON preserved it by INJECTING the bf16 head into the
  quant output (NOT re-quantizing — confirmed AEON's nvfp4 mtp is bf16). Built pantheon-27b-mtp = compressed-tensors
  main + injected bf16 mtp + `text_config.mtp_num_hidden_layers=1` → vLLM detected the MTP but SKIPPED the bf16
  self_attn weights → 0/192 draft tokens accepted. **The bf16 MTP head only loads on the MODELOPT main-model format
  (like AEON), not compressed-tensors.** (Moot — operator dropped MTP for gen; not needed for the non-reasoning RP.)
- `[2026-07-07]` **vLLM 0.24.0 qwen3_5 LoRA application = silent no-op (#47639).** Adapter loads HTTP 200
  but zero deltas at inference. NOT quant (NVFP4 AND FP8 both inert). NOT adapter format (separate `zc`
  adapter — correct per vLLM's `check_unexpected_modules` allowlist — loads clean but inert; the fused-key
  rekey is rejected). The #47640 None-group guard-patch overlay did NOT fix it (failure is UPSTREAM of
  `expand_packed_lora` — the separate→fused mapping never happens). Fix PR #47640 is OPEN (unmerged) so no
  version-bump helps. Merge bakes deltas in (bypasses this) but is static.
- `[2026-07-07]` **SGLang generic image can't LOAD our NVFP4 AEON** — ModelOptModelLoader weight-shape/
  packing mismatch ([1024,5120] vs [1024,2560], 2-fp4/byte). NVFP4-on-SGLang needs the dedicated
  `qwen36-27b-nvfp4` dev image or a requant to SGLang's format. bf16 loads fine (arch supported; crash was
  quant-loader-specific).
- `[2026-07-07]` **SGLang `--lora-target-modules` CLI enum REJECTS the GDN names its own resolver asks for**
  (invalid choice: 'in_proj_qkv'); `'all'` resolves to the FUSED set (qkv_proj/in_proj_qkvz). SGLang wants
  its OWN packed layout (base r16 + `get_stacked_multiply=3`, NOT a pre-fused rank-48 qkv → the [48]-vs-[144]
  shape assert). A THIRD adapter format; version-exact source needed (`:latest`=0.5.13, NOT `main`).
- `[2026-07-07]` **Engine invocation footguns cost several wasted serve-bounces this session** — `docker run
  --rm` ate crash logs; duplicated `serve` (vLLM image entrypoint is already `["vllm","serve"]`);
  `--max-lora-rank 48` invalid (choices 1/8/16/32/64… → use 64); parens in `echo` inside `ssh host -c "…"`
  break the remote shell. LESSON: verify engine launch flags (`--help`, GPU-free) + never `--rm` a container
  whose crash logs you need, BEFORE bouncing a production serve.
- `[2026-07-04]` **LiteLLM (this gateway version) mutates the SHARED deployment config in-place on
  per-request sampler-param merge** → my deliberately-invalid `top_k=-5` forwarding-probe bled into a
  param-less character-rp request (vLLM 400, ONE-OFF, self-cleared by a later valid probe). NOT
  caching (none configured), NOT a config change. **Never fire invalid/distinctive sampler values at
  a SHARED gateway alias with live consumers** — use a throwaway alias, or a `docker restart litellm`
  flushes residual carryover. `feedback_litellm_shared_param_mutation`.

- `[2026-07-04]` **A systemd `--user` daemon that shells out to `~/.cargo/bin`/`~/.local/bin` tools
  needs an explicit `Environment=PATH`** — the minimal `--user` default silently drops them. The
  althing herald lost `zellij` → silent `pane-miss` for ALL config-backed TUI/pane agents; CC + FIFO
  routes were unaffected, so it was invisible from a CC session. `reference_nh3_dev_althing_herald`.

- `[2026-07-04]` **On-prem T1 train that keeps ANY ana-ml2 serving up = ~6-8 DAYS** (1-GPU + NVMe
  ZeRO-Infinity offload; MoE ~10B-active cuts FLOPs but NOT the 244G base's param I/O). The only fast
  on-prem path is a FULL ana-ml2 shutdown (both GPUs + the ~421G vLLM RAM freed → base fits in the
  566G CPU RAM) → CPU offload → ~1-day full-fleet outage. Cloud (no offload) = hours. `reference_t1_cloud_train_plan`.

- `[2026-07-01]` **A personal-Worldtree CI deploy that fails ~85s in with "not found / unauthorized"
  is usually the pull-only-vs-build RACE, not registry-auth.** `deploy-personal.yml` is PULL-ONLY but
  fires on the `staging/vX` tag simultaneously with `deploy.yml`'s build → pulls before the push
  finishes. FIX: re-run once built, or gate on `workflow_run: completed`.

- `[2026-07-01]` **MTP/spec-decode on a SHARED serving model helps single-stream but HURTS
  moderate-concurrency aggregate + silently ignores `min_p`/`logit_bias`** (qwopus `gen`: N=1 +12%,
  N=4 −20%). Reserve for dedicated/interactive deployments.

- `[2026-07-02]` **irv-ml1 `/worktank` ROOT is root-owned — lkraven can't write there (irv-ml1 sudo
  needs a password) → stage model pulls to `/home`.** PIN THE A6000 BY UUID for training (native-CUDA
  ordering differs vs docker; the 3090 index 0 is usually near-full → OOM). `CUDA_VISIBLE_DEVICES=GPU-<uuid>`.

_101 older entries archived to archival-memory.md._