memory: snapshot — 2026-06-20

WT capability-gateway migration (aliases + swap-transparency rule + wildcard
removal + gen-frontier + v0.37.5 staged/triggered), OmniVoice streaming /tts
(diffusion TTFA floor), arbo v0.14.9, ratatoskr :8392 + admin.events.read
brokered, claude-bot admin on vh/Worldtree. Disk incident root-caused to a
94GB unrotated langfuse-clickhouse log (+ secondary image bloat) → fixed,
image/buildx prune cron added, Langfuse retired. Backup gap: rest-server-ana
is the ACTIVE ana-side restic target down ~months on a failed ana-nas NFS
mount (NOT decommissioned — docs corrected) → recovery deferred to clean
context, along with the docker-daemon log cap. Archived the [2026-06-16]
cluster (13 Recent + 8 Tried) to archival-memory.md.

archival-memory.md

@@ -481,6 +481,74 @@ _Entries moved out of persistent-memory.md to keep the active file scannable. Re
 - `[2026-06-14]` **LiteLLM infra-ops admin key provisioned** (operator) — resolves the LiteLLM half of the credential-migration directive; use it for ALL gateway ops (NOT `sk-corvid`). Value at `~/.config/litellm/infra-ops-key` (mode 600); gateway reachable directly from nh3-dev at 10.250.50.70:4000. (auto-memory `reference_litellm_infra_ops_key`)
   _Archived 2026-06-19._
 
+- `[2026-06-16]` **litellm `strip_empty_tools` pre-call hook shipped** (`d1bea13`) — an empty `tools:[]` 500s vLLM ("tools must not be an empty array"); a global `litellm_settings.callbacks` CustomLogger pops it (+ orphaned `tool_choice`) before forwarding, so it covers EVERY vLLM model, not one. `drop_params` only drops unsupported PARAMS, not empty VALUES. Mounts beside config.yaml (litellm resolves callbacks relative to the config dir). Verified live across granite/mistral/stream. (`stacks/litellm/conf/strip_empty_tools.py`)
+  _Archived 2026-06-20._
+
+- `[2026-06-16]` **single-file `gateway-chat.html` playground shipped** (`984ca3d`, `tools/`) — zero-dep browser chat straight to the gateway (`:4000`, CORS open), system-prompt box, streaming SSE, renders `reasoning_content`, NEVER sends `tools`. Built because the LiteLLM admin-UI playground can't test vLLM-backed models (see Tried-and-abandoned). Serve on-request via `python3 -m http.server -d tools`.
+  _Archived 2026-06-20._
+
+- `[2026-06-16]` **claude-bot granted ADMIN on vh/arbo** (operator) — arbo CI/CD gitea ops now route through the service account, not the operator's personal `vh` (the credential-migration directive). Operator added claude-bot as Administrator collaborator; claude-bot self-serves via its `gitea-password` (basic-auth = FULL perms; its stored token is package-write-only, can't change repo settings). (auto-memory `reference_claude_bot_gitea_creds`)
+  _Archived 2026-06-20._
+
+- `[2026-06-16]` **ratatoskr Tier-3 MEMORY plane wired + live-smoke driven.** The missing server-side step was the per-PORT `BIFROST_CLIENT_ALLOWED_HOSTS` allowlist (`:8391`; consumer key reused, plane-agnostic — my earlier "reachability is the whole wiring" was incomplete). Persist + dispatch proven end-to-end against personal v0.35.3; cross-session recall-INJECTION root-caused to the `score_breakdown` shape seam and handed to worldtree-dev (#295 recall leg open). Driving a real memory-call needs a consumer agent defined WITH a `memory` block (`ctx.memory_config≠None`), not `agent_self_enabled`. (tracking: c99aa49 + auto-memory `reference_bifrost_plane_wiring`) — **[2026-06-16] FLIPPED:** the `score_breakdown`-shape root-cause was disproven; real cause = cold-recall `scope_filter` adds an `agent_self` axis → ratatoskr's conjunctive INV-005 match drops the `{end_user}`-only chunks → 0 hits. Now Worldtree #297. (auto-memory `project_worldtree_295_recall_rootcause`)
+  _Archived 2026-06-20._
+
+- `[2026-06-16]` **infra-ops = the DURABLE admin path on corviduo-dev** (operator granted standing admin; extends the 2026-06-15 bootstrap) — added a `Host corviduo-dev → User infra-ops` `~/.ssh/config` alias + repointed `servers/corviduo-dev/ssh-target` to `infra-ops@`; `ssh corviduo-dev` / `elway corviduo-dev` now land as infra-ops w/ NOPASSWD (vh = explicit-only human path). elway resolves via ssh-config, NOT the ssh-target file. (tracking: c99aa49 + auto-memory `reference_corviduo_dev_emergency_ops`)
+  _Archived 2026-06-20._
+
+- `[2026-06-16]` **demo + personal `character` model flipped qwen3.6-35-a3b → mistral-small-4** (operator) — in `model_roles.yaml` the FIRST `catalog_id` in a role's `binds:` is the default; reordered mistral-first (qwen retained in the switch-allowlist, non-destructive), applied via the PIN-SAFE api recreate (derive `WORLDTREE_IMAGE` from the matrix sibling — never a bare `compose up`, which falls to the crash-blocking stale `:latest`), verified a fresh character agent resolves model=mistral-small-4. (auto-memory `reference_corviduo_dev_emergency_ops`)
+  _Archived 2026-06-20._
+
+- `[2026-06-16]` **NVFP4 ModelOpt MoE now LOADS on vLLM 0.23.0 — SUPERSEDES the
+  2026-06-14 "blocked" finding (#44081 fixed).** Confirmed the cached
+  `nvidia/Qwen3.6-35B-A3B-NVFP4` loads clean + generates on 0.23.0; swapped prod
+  qwen36 FP8→NVFP4 (~13 GB reclaimed on GPU 1, fp16 KV). Validated by brokkr (speed:
+  faster single-stream, ≥parity batched) + comfy-dev (vision anatomy-judge A/B: PASS).
+  One W4A4 tell: minor multi-step chained-reasoning slip — doesn't bite the vision
+  judge. Mistral's NVFP4 is a SEPARATE compressed-tensors path (day-0), distinct from
+  the ModelOpt one. (auto-memory `reference_nvfp4_moe_loads_on_vllm_023`)
+  _Archived 2026-06-20._
+
+- `[2026-06-16]` **Mistral Small 4 = the GPU-0 tenant** (operator) — fills the slot
+  the deferred creative-writing pick held, and supersedes llama-swap (now down).
+  NVFP4 119B MoE, pinned **vLLM v0.22.0** for working vision (0.23.0 breaks it),
+  256K context. Operator's creative-writing model; **abliteration planned** → then it
+  becomes the uncensored-creative model the persona substrate needs. Reasoning via
+  `reasoning_effort` (BINARY none|high — no medium). (tracking: `c77a9aa`/`9a49963`)
+  _Archived 2026-06-20._
+
+- `[2026-06-16]` **Selene judge restored on vLLM FP8, NOT NVFP4** — W4A4 is too
+  aggressive for a precision judge validated at Q6_K; FP8 ≥ Q6_K fidelity, and Selene
+  is text-only Llama-3.1 so dynamic `--quantization fp8` has no vision tower to noise.
+  `selene-1-mini-8b` on GPU 1 :8011. Worldtree's Domari role repointed to it. (`c985ede`)
+  _Archived 2026-06-20._
+
+- `[2026-06-16]` **GPU-1 buffer put to work: selene 0.13→0.17 + qwen36 0.32→0.34**
+  — the two KV-bound generative models that gained Worldtree consumers; granite + the
+  pooling models under-use their util, so growing THEM is wasted. (`e124a2f`)
+  _Archived 2026-06-20._
+
+- `[2026-06-16]` **claude-bot Gitea service account stood up; corviduo-ORG conversion
+  TABLED** (operator). claude-bot is the credential-migration foundation. The proper
+  fix for scoped agent package-write is converting vh (a USER) → a `corviduo` ORG —
+  blast radius assessed (org creation = free; arbo-first migration = cheap, no registry
+  images; full fleet = incremental, ~8 active vh images / 50 repos) — but tabled for
+  later. Registry-push deferred until then. (auto-memory `project_migrate_infra_access_to_claude_credentials`)
+  _Archived 2026-06-20._
+
+- `[2026-06-16]` **arbo engine builds → comfy-dev OWNERSHIP** (operator approved) +
+  Gitea Actions runner stood up on irv-ml1 (host-executor as lkraven). infra-ops keeps
+  box-level escalation (runner, package:write PAT pending corviduo-org, webhook). See
+  the runner entry in Current state. (tracking: this snapshot + arbo thread `01KV70AB2B4B…`)
+  _Archived 2026-06-20._
+
+- `[2026-06-16]` **Worldtree persona substrate migrated qwen→Mistral via capability-
+  named profiles** (v0.35.8→v0.35.10; demo + personal). PRE-SYNC-FIRST is now the
+  standing pattern for any providers.yaml/profile change: sync config to the bind-mount
+  BEFORE the deploy image-swap restart, so new agents + new config land together (the
+  v0.35.9 merge-first attempt race'd → 14 crash-loops). (tracking: worldtree thread `01KV6X0M…`)
+  _Archived 2026-06-20._
+
 ## Tried and abandoned (archived)
 
 - `[2026-04-30]` task-board workflow with
@@ -956,3 +1024,53 @@ _Entries moved out of persistent-memory.md to keep the active file scannable. Re
 
 - `[2026-06-14]` **Fleet/colo hosts must reach gitea over the INTERNAL route, NOT the public IP.** `gitea.phasefinal.com` = public `38.120.12.44` (ana-srv1); gitea is a container on ana-docker, git-SSH `10.250.50.70:222` + HTTP `:3000`. A fleet host egressing to public `:22` gets fail2ban-banned after any retrying git loop → silently wedges webhook auto-deploys (`git fetch` times out under `set -euo pipefail`, aborts before reset). Bit irv-ml1's arbo deploy. `:22` on `10.250.50.70` is ana-docker's HOST sshd (deploy key → Permission denied), NOT gitea. Documented `docs/orientation.md` (`6e58e57`).
   _Archived 2026-06-19._
+
+- `[2026-06-16]` **litellm 500 `Router.acompletion()/aembedding() missing 'messages'/'input'` = a request missing `Content-Type: application/json`, NOT a gateway outage.** curl `-d` defaults to form-encoding → litellm can't parse the JSON body → `data` reaches the router without `messages`/`input` → 500 (should be a 400; litellm #16993). My own diagnostic calls dropped the header → I misread it as a gateway outage and needlessly bounced the gateway ~4× chasing a phantom (image/version/config were fine throughout; a malformed UI-added "Mistral Story Eval" model in the DB was a red herring I deleted). ALWAYS send `-H "Content-Type: application/json"` testing litellm; reproduce with a header'd call before declaring a litellm incident.
+  _Archived 2026-06-20._
+
+- `[2026-06-16]` **LiteLLM admin-UI playground can't test vLLM-backed models** — it auto-sends empty `tools:[]`, vLLM 400s (litellm #6228); the gateway `strip_empty_tools` hook is a PROXY hook and structurally can't reach the UI's in-process `litellm.completion()` call. Off-ramp = `tools/gateway-chat.html`. (Langfuse playground also out: its SSRF guard blocks internal-IP LLM connections, wontfix Langfuse #13097.) (auto-memory `reference_litellm_ui_playground_vllm_deadend`)
+  _Archived 2026-06-20._
+
+- `[2026-06-16]` **A gitea Actions run that "looks like it never fired" (no image, untouched build dir) usually DID fire then skipped or failed-fast.** Bit arbo TWICE: run #3 looked-never-fired but SKIPPED (change-detection `engine=0`); run #4 looked-never-fired but FAILED in ~7s (early-step). Check the run LIST + status via claude-bot (`GET /repos/{o}/{r}/actions/tasks`) BEFORE a runner-connection/dispatch theory. CORRECTION (2026-06-16, proven): action logs ARE readable via API at the per-JOB endpoint — `GET /api/v1/repos/{o}/{r}/actions/runs/{run}/jobs` → take a job `.id` → `GET /api/v1/repos/{o}/{r}/actions/jobs/{job_id}/logs` returns the full plain-text log (claude-bot basic-auth, internal route `10.250.50.70:3000`). Only the RUN-level `/runs/{id}/logs` 404s; the per-job one works, so no UI needed. Gotcha when reading: gitea marks **every** step `conclusion:failure` once any step fails — the per-step conclusions mislead; trust the log body + step *timestamps* (the step with real duration is the culprit) not the conclusions. (Used 2026-06-16 to diagnose arbo run #5/task 1175 = test-gate `uv run pytest` → `Failed to spawn: pytest` = pytest absent from the fresh-checkout venv sync, NOT the SSH-clone comfy-dev suspected.)
+  _Archived 2026-06-20._
+
+- `[2026-06-16]` **vLLM 0.23.0 breaks Mistral MULTIMODAL at startup** (`MistralCommonImageProcessor
+  has no attribute fetch_images`, #44911 regression ~0.22.1+; same class hit Mistral-3.1/
+  Devstral). The dummy-image profiling crashes. Workarounds: `--limit-mm-per-prompt
+  '{"image":0,"video":0}'` loads text-only; for VISION, pin **v0.22.0** (last pre-regression).
+  Tradeoff: v0.22.0 *buffers* reasoning (doesn't stream reasoning_content cleanly); a newer
+  build streams it but breaks vision — vision won (Mistral is the abliteration-baseline).
+  _Archived 2026-06-20._
+
+- `[2026-06-16]` **Mistral `-reasoning` TTFT variance = Triton kernel JIT-compile-during-
+  inference, NOT reasoning time.** `jit_monitor` logs the spike (`_compute_slot_mapping_kernel`,
+  `apply_token_bitmask_inplace_kernel` ← the tool-call/structured-output path). vLLM's startup
+  warmup covers only SOME shapes; a first-hit uncovered shape JITs a kernel (seconds), then
+  CACHES it — warm until container recreate (the Triton cache is IN-container, no volume, so
+  recreate = cold again). Secondary: GPU idle-down to P8/180 MHz between requests. Also:
+  Mistral `reasoning_effort` is BINARY (none|high only — medium/low → HTTP 400).
+  _Archived 2026-06-20._
+
+- `[2026-06-16]` **Worldtree old-baseline instances need the FULL config set synced, not the
+  changeset delta.** Personal (a PRE-mistral baseline) crash-looped (41 restarts) on
+  `DanglingBindingError: role 'echo' binds missing catalog entry 'qwen3.6-35-a3b-heretic'` —
+  its `model_roles.yaml` (2026-06-10, never migrated) still bound the retired heretic that the
+  synced v0.35.10 providers removed. The "model_roles unchanged — skip" was correct only for
+  demo (already at the v0.35.9 baseline). Old-baseline → sync providers + model_roles + defaults.
+  _Archived 2026-06-20._
+
+- `[2026-06-16]` **vh is a Gitea USER, not an org → no scoped package-write for a service
+  account.** Gitea user-namespace packages are owner/admin-only, so claude-bot can't push to
+  `vh/arbo` without being admin. The fix is converting vh → a `corviduo` ORG (tabled). Also:
+  vh's `tea` token lacks `write:admin` — create gitea users via the server-side `gitea admin
+  user create` CLI (`docker exec -u git gitea …`), not the admin API.
+  _Archived 2026-06-20._
+
+- `[2026-06-16]` **`docker compose up -d` on a multi-service vLLM stack recreates ALL its
+  services** (shared `.env` → all configs re-hash) → concurrent memory-profiling race
+  ("No available memory for the cache blocks"). Target one: `docker compose up -d <service>`.
+  (auto-memory `feedback_compose_up_recreates_whole_stack`)
+
+_76 older entries archived to archival-memory.md._
+  _Archived 2026-06-20._
+

persistent-memory.md

@@ -1,6 +1,6 @@
 # Persistent memory — eshpfi-management
 
-_Last updated: 2026-06-19_
+_Last updated: 2026-06-20_
 
 ## Repo purpose
 
@@ -8,7 +8,9 @@ Reference workspace for PFI infrastructure: server inventory, canonical
 Docker Compose stacks, ops playbooks, and conventions. Authoritative
 copies of compose files live on the servers under
 `/opt/docker/compose/<stack>/`; this repo mirrors them for version
-control, editing, planning, and CI-driven deploys.
+control, editing, planning, and CI-driven deploys. **It was originally
+spun up to handle the fleet backups** — keep that lens when triaging
+backup/storage issues.
 
 ## Tools and conventions
 
@@ -23,9 +25,9 @@ Sister repos (separate gitea repos, deployed by playbooks here):
 | `vh/althing` | Inter-agent message bus (chamber UI port 7881, forseti + agent-runner daemons, valkey IPC) | push-to-main → CI deploys (2026-05-14) |
 | `vh/mead-hall` | Bifrost tool-provider sidecar (port 5173 on dev VM 10.100.10.50) | push-to-main → CI deploys (2026-05-16) |
 | `vh/skaldsong` | Wizard + reader surface (port 8300, ana-docker, registry-pull pattern) | push-to-main → CI deploys (2026-05-19) |
-| `vh/worldtree` | Conversation API (corviduo-dev demo :8080 / personal :8081 / pinned :8082) — Heimdall auth, Bifrost integration | push-to-main → CI deploys |
+| `vh/Worldtree` | Conversation API (corviduo-dev demo :8080 / personal :8081 / pinned :8082) — Heimdall auth, Bifrost integration. **gitea-runner builds on ana-docker** (the image-bloat source); claude-bot now ADMIN collaborator (2026-06-20) | push-to-main → CI build-and-deploy (runner on ana-docker) |
 | `vh/yt-voice-clipper` | YouTube → diarized voice-clip dataset builder + audition console (irv-ml1 :8000) | push-to-main → **gitea-webhook auto-deploy** to irv-ml1 (2026-06-03) — see `docs/runbooks/ytvc-autodeploy.md` |
-| `vh/arbo` | Catalog-driven ComfyUI engine (irv-ml1 :8201, comfy-dev owns engine/catalog/image) | push-to-main → **gitea-webhook auto-deploy** to irv-ml1 (catalog-only restart; infra side = `stacks/arbo/`) |
+| `vh/arbo` | Catalog-driven ComfyUI engine (irv-ml1 :8201, comfy-dev owns engine/catalog/image) | push-to-main → gitea Actions CI (deploy-engine.sh, build-local, health-gated) now LIVE; catalog via :9009 webhook |
 
 (`vh/volva` + Heid were re-architected from systemd daemons to Claude Code
 session orchestrators 2026-06-08; their nh3-dev `.service` units were removed —
@@ -35,7 +37,12 @@ no longer deployed sidecars here. See Recent decisions.)
   fleet repos, daily 01:00 PDT); PBS-ANA primary + PBS-NH3 DR mirror for
   VM images. ana-nas is the SPOF for postgres + PBS-ANA datastore +
   cross-site restic targets — see `docs/runbooks/disaster-recovery.md`
-  for the blast-radius matrix.
+  for the blast-radius matrix. **⚠️ The restic file+DB layer routes
+  through TWO rest-servers** (`rest-server-ana` @ ana-docker:8000 →
+  ana-docker/ana-ml2/esh-docker-vm/vm-esh-nas; `rest-server-nh3` @
+  nh3-nas:8000 → irv-ml1/nh3-docker). Both depend on their NAS's NFS
+  export of `/mnt/backup`. **As of 2026-06-20 `rest-server-ana` is DOWN
+  on a failed ana-nas NFS mount — see Current state.**
 
 - **`pull-hf-repo.yaml`** is the canonical "get a HuggingFace
   model/dataset onto ana-ml2's shared cache at
@@ -46,11 +53,11 @@ no longer deployed sidecars here. See Recent decisions.)
   registry and its own bootstrap admin key. Infra-ops's stored
   long-lived admin key (`key_id 61419c92`) at
   `ana-docker:/opt/docker/conf/.secrets/worldtree-infra-ops-admin`
-  auths against **demo only**. For personal-instance admin ops, fetch
+  auths against **demo only**. Personal-instance admin (the
-  the bootstrap admin per-op via
+  `~/.config/worldtree/personal-admin-token`, mode 600) POSTs
-  `docker exec worldtree-personal-worldtree-api-1 printenv WORLDTREE_BOOTSTRAP_ADMIN_KEY`
+  `/admin/keys` (mints per-project keys; takes `user_id`+`label`, **no
-  on corviduo-dev. Used for `POST /admin/keys`, admin diagnostics
+  scope param** — scopes are tier-derived, so admin-tier scopes like
-  (`/admin/sessions/<id>/{bifrost,tools}`, etc.).
+  `admin.events.read` must be granted WT-side by worldtree-dev).
 
 - **Per-project user keys against personal Worldtree** (issued
   2026-05-19): `skaldsong:79744637` (nh3-dev iteration),
@@ -76,288 +83,244 @@ no longer deployed sidecars here. See Recent decisions.)
   IP and wedges webhook deploys. `:22` on `10.250.50.70` is ana-docker's
   HOST sshd, not gitea. Full gotcha in `docs/orientation.md` → Git/gitea.
 
-- **docker-as-root pattern** (for ops that have no admin API, e.g.
+- **docker-as-root pattern** (for ops with no admin API, or to edit
-  `SqliteUserStore.set_bifrost_credentials`): on hosts where the SSH
+  deploy-owned/root-owned files without sudo): `docker run --rm -v
-  user is in the `docker` group but lacks passwordless sudo, run
+  <target-dir>:/wt [-v /var/run/docker.sock:/var/run/docker.sock]
-  `docker run --rm -v <target-dir>:/wt -v /var/run/docker.sock:/var/run/docker.sock docker:cli sh -c "..."` to edit deploy-owned files
+  docker:cli sh -c "..."` (or `alpine` for plain file ops). docker-group
-  without sudo. Documented with security warning in
+  membership is effectively root via bind-mount; treat as sudo-equivalent.
-  `servers/corviduo-dev/README.md`. docker-group membership is
+  **Foot-gun: relative paths in compose.yaml resolve against the sandbox
-  effectively root via bind-mount; treat as a sudo-equivalent grant.
+  CWD but the Docker daemon interprets them against the HOST fs — always
-  **Foot-gun: when running `docker compose` inside this sandbox,
+  pass `-e VAR=/abs/path` for any relative-default config dir.**
-  any relative path in compose.yaml (e.g. `${WORLDTREE_CONFIG_DIR:-./config}`)
-  resolves against the sandbox CWD, but Docker daemon interprets the
-  resulting path against the HOST filesystem. Always pass `-e VAR=/abs/path`
-  to the docker run invocation for any relative-default config dir.**
 
 - **`scripts/elway` sudo handling** — elway prompts for the sudo password
-  ONCE via `getpass` before the first `sudo: true` step. That prompt is
+  ONCE via `getpass` before the first `sudo: true` step → can't run
-  interactive → elway can't run unattended from a non-TTY tool if any step
+  unattended from a non-TTY tool if any step needs sudo. Sudo-free
-  needs sudo. For sudo-free playbooks (no `sudo: true` steps) it runs fully
+  playbooks run fully non-interactive over key SSH.
-  non-interactive over key SSH. To create root-owned dirs WITHOUT host sudo,
+
-  use the docker-daemon-root trick: `docker run --rm -v /worktank:/mnt alpine
+- **Per-host SSH identity matters for sudo.** infra-ops has NOPASSWD sudo
-  sh -c 'mkdir -p /mnt/<x> && chown -R 1000:1000 /mnt/<x>'`.
+  on most PFI Linux boxes (corviduo-dev included since 2026-06-15) — BUT
+  **on ana-docker the session lands as `lkraven` (docker-group, NO
+  passwordless sudo)**. So on ana-docker: docker commands work directly;
+  root-owned-file edits need the docker-as-root trick or infra-ops; but
+  the litellm config + most stack compose/conf are `lkraven`-owned →
+  plain `cp`/edit, no sudo. (Learned 2026-06-20 — see Tried-and-abandoned.)
 
 ## Current state / in-flight
 
-_As of 2026-06-19:_
+_As of 2026-06-20:_
 
-- **gateway-chat web chat is now a PERSISTENT container** (ana-docker `:8091`, `stacks/gateway-chat/`, nginx:alpine, homepage tile) — the model-smoking UI; auto-discovers `/v1/models` (new gateway models just appear) + image-upload for vision. **litellm task-aliases live:** `summarizer`/`classifier` → granite-4.1-8b, `summarizer-large` → gen/qwopus (`gen-nt`/`gen-reasoning-nt` were added then removed — redundant with the `strip_empty_tools` hook). **pi on nh3-dev wired to `gen`** (`~/.pi/agent/models.json` + a `gen` launcher mirroring `glm`; vision-capable, thinking off/on). The litellm `*` wildcard still points at decommissioned llama-swap (see Tried-and-abandoned) — a cleanup that'd make stale model names fail legibly.
+- **Worldtree capability-gateway migration — LANDED + DEPLOY TRIGGERED.**
+  The role→capability indirection (worldtree-dev's #257/ADR-0012 direction;
+  operator: NO `wt-` prefix, reuse the existing capability-alias convention)
+  is live on the LiteLLM gateway: `gen`/`gen-reasoning` (→qwen3.5-122-a10b),
+  `classifier`/`summarizer` (→granite), `chat-judge` (→selene-1-mini-8b),
+  `reranker` (→qwen3-reranker), `scalar-judge` (passthrough → Skywork-Reward-V2
+  @ ana-ml2:8003 — LiteLLM has no reward MODE), `gen-frontier`/`-reasoning`
+  (→GLM 5.2, PAID). **`qwen3-embedding` kept model-named ON PURPOSE** — the
+  swap-transparency rule: embedding vectors aren't swap-transparent, so a
+  generic `embedding` alias is a footgun. The `*` wildcard → dead llama-swap
+  was REMOVED (unknown models now 404 cleanly). WT's full re-point (granite
+  SPLIT into granite-classifier/granite-summarizer + role rebind + edges→gen +
+  selene→chat-judge) landed as worldtree-dev's commit **3fbf338 / v0.37.5**;
+  I staged its `providers.yaml`+`model_roles.yaml` byte-identical into
+  demo+personal bind-mounts and triggered build-and-deploy via claude-bot.
+  Pinned rides CI. (d372103, d0eb09c, 7819f96)
 
-- **Mistral Small 4 heretic is DISPLACED from GPU 0 (staged-down) — GPU 0 is now the `gen` model (Qwopus).** The heretic abliterated NVFP4 (`stacks/mistral-small-4-heretic/`, built in-house, "heretic stays", build tooling `tools/mistral-small4-nvfp4/`, byte-equiv to official) was downed 2026-06-19 to give GPU 0 to the gen swap. **CONSEQUENCE (operator-acknowledged, per the litellm ⚠️ note): the Worldtree demo+personal `character` backend — bound to `mistral-small-4` — is DARK until repointed.** To restore: down Qwopus + `up` the heretic stack (or serve it elsewhere). Prior doc-drift (official-stack README/comments saying "official NVFP4") now moot for GPU 0. (dd3a5c9, f566f61)
+- **🔴 BACKUP GAP — `rest-server-ana` DOWN ~months on a failed ana-nas NFS
+  mount (NOT a decommission — docs corrected the recollection).** The
+  active ana-side restic file+DB target (ana-docker/ana-ml2/esh-docker-vm/
+  vm-esh-nas) is crash-looping: `/etc/fstab` has `10.250.50.50:/mnt/backup`
+  as an NFS mount, but `mnt-backup.mount` is in a `failed` state → /mnt/backup
+  is the empty local ghost dir → `.htpasswd permission denied` (the
+  documented disaster-recovery.md failure). ana-nas is pingable; the mount
+  just isn't attached (bare-`defaults` fstab, no auto-retry). So file+DB
+  restic for 4 hosts has been failing for ~months (PBS VM-image backups are
+  the only cover). **DEFERRED to clean context:** the documented recovery
+  (clear failed mount → rm ghost → `mount /mnt/backup` → restart rest-server)
+  needs root → infra-ops; plus prevention (fstab `_netdev`/retry + a
+  backup-freshness alert — the gap went unnoticed for months).
 
-- **irv-ml1 VRAM consolidated — ComfyUI owns the full 48 GB A6000.** Pinned comfyui `NVIDIA_VISIBLE_DEVICES=1`; the audio/TTS zoo (chatterbox, parakeet live; vibevoice, yt-voice-clipper config-pinned; kokoro already there) moved to the 3090; downed dia2-2b (17-day stale), ace-step, csm-expressiva. comfy-dev torch-pin applied (`DISABLE_UPGRADES=true` @ torch 2.12.1, SageAttention rebuilt + matched). **WATCH:** the 3090 has ~18.7 GB free for the audio zoo — heavy *concurrent* on-demand audio could pressure it; vibevoice deploys from `/worktank/vibevoice/build` (pre-existing repo-vs-deploy drift). (a8550ad; auto-memory `reference_irv_ml1_comfyui_mmartial`)
+- **ana-docker disk incident RESOLVED — root cause was a 94 GB unrotated
+  `langfuse-clickhouse` log, NOT (just) image bloat.** ClickHouse logs
+  verbosely to stdout + there was NO log rotation anywhere (no per-stack
+  `logging:`, no daemon default) → its json.log hit ~94 GB and filled `/`
+  (100%, 28/48 containers unhealthy on `runc /tmp ENOSPC`). Fixed: truncated
+  the log, pruned ~101 GB of images (the gitea-runner accumulated 18
+  worldtree build versions), retired Langfuse. Disk → ~51%. Installed a
+  daily image+buildx prune cron (`lkraven` crontab on ana-docker). **DEFERRED
+  to clean context:** a **docker-daemon default log cap** (`daemon.json`
+  log-opts + docker restart, needs root → infra-ops) — the real systemic fix
+  so no container can ever spew unbounded again. (Note: named buildx-builder
+  state volumes aren't cleared by `docker image/builder prune` — orphaned
+  ones need `docker volume rm`.)
 
-- **ComfyUI acceleration set (9 nodes) installed for comfy-dev on irv-ml1** — TeaCache, WaveSpeed(+FBCache), SageAttention2, Detail-Daemon(+bleh), PAG, dynamic-thresholding, Skimmed_CFG, TensorRT(cu12), SUPIR; comfy-dev wires + benchmarks. The full-A6000 re-tiered Flux-TRT + SUPIR from Ada-dormant to wire-today.
+- **Langfuse RETIRED (2026-06-20, operator).** Redundant with LiteLLM's
+  native logging (`store_prompts_in_spend_logs: true` → full prompts/
+  responses/tokens/cost at `:4000/ui`); nothing used its trace-grouping/eval
+  features; and it's a 6-container crash-prone surface (the 94 GB spew).
+  Dropped the `success_callback`/`failure_callback: [langfuse]` from the
+  litellm config (gateway observability stays native) and `compose down -v`
+  the stack. (f8eda1c)
 
-- **nh3-extdev (10.100.50.42) = sudo-less infra-ops manager box** (successor to nh3-ansible) + `/opt/externs/{gbcnc,surefire,svsconstruction}` client workspaces for **pi-on-GLM-5.2** client agents. **GLM 5.2 wired into litellm** (`glm-5.2` + `-reasoning` via z.ai). **[2026-06-18] pi NOW INSTALLED + all 3 workspaces wired to GLM 5.2 + verified** — `pi` 0.79.7 (earendil-works) on user-level Node v22.23.0 LTS (no root); per-client isolated `PI_CODING_AGENT_DIR` + `models.json`/`settings.json` + `run-pi.sh` launcher (sources scoped `secrets.env` key → litellm gateway, cross-site-reachable; live gbcnc→GLM 5.2 round-trip OK). Idempotent `playbooks/install-pi-nh3-extdev.yaml` (add-client/upgrade). Corrected the scaffold's wrong `.pi/config.example` (it guessed `OPENAI_BASE_URL`/`PI_MODEL` env; real pi reads `models.json`). (527a844) **STILL PENDING (client-coordination-gated, NOT pi-setup):** deploy *pubkeys* onto each client's hosts, populate `repo/servers/` inventory, give workspace `repo/`s a backup git remote; verify the 3 GLM keys are rpm/tpm/budget-capped (functional confirmed, caps not).
+- **OmniVoice gained a streaming `/tts` + language-safe sanitizer** (irv-ml1
+  :8199, `stacks/omnivoice/`), driven by a VENDORED copy of chatterbox-fast's
+  buffer-ratchet scheduler. KEY FINDING: OmniVoice is DIFFUSION → a ~fixed
+  per-call overhead sets a TTFA FLOOR (~0.7 s @16 steps), and measured RTF is
+  ~2.8×/5.6× on the 3090 (NOT the upstream-claimed 40×). Tuned: streaming
+  num_step=16/batch 32, rtf_prior=20. **chatterbox-fast (autoregressive,
+  ~0.5 s TTFA) stays the lower-latency front-end** — OmniVoice is the
+  multilingual/voice-design complement, NOT a replacement. (288d085, cd92b85;
+  auto-memory `reference_omnivoice_streaming_tts`)
 
-- **LitBench-RM stood up → now ON-DEMAND / DOWN.** `SAA-Lab/Llama8B-CreativeWritingVerifier` (R19 creative-quality reward judge for brokkr/Dvalin) served on irv-ml1 A6000 via vLLM `--runner pooling` (`http://10.100.79.3:8202/classify`, raw passage text → scalar). brokkr-validated (does NOT penalize explicit content). Taken DOWN to on-demand (operator) — resident it held ~19.6 GB crowding comfyui's A6000 slot; weights staged on irv-ml1, ~90s respin (command in auto-memory). (auto-memory `reference_litbench_rm_irv_ml1`)
+- **arbo v0.14.9 deployed** (muse prompt-enhance → `summarizer-large`, was
+  granite which mode-collapsed at temp 1.0) via the canonical health-gated
+  `scripts/deploy-engine.sh` on irv-ml1. arbo's gitea Actions CI is now LIVE
+  (runner-label mismatch fixed by comfy-dev) → future src pushes auto-deploy;
+  manual deploy only when CI is down. (auto-memory `reference_arbo_engine_deploy`)
 
-- **litellm gateway healthy; `strip_empty_tools` hook + `gateway-chat.html` shipped** (see Recent decisions). **arbo + brokkr litellm keys are already `all-proxy-models`** — verified, so "grant arbo/brokkr full access" was already done. claude-bot now has **admin on vh/arbo**; **arbo CI/CD Phase 1 is firing** (runner live; comfy-dev owns `deploy.yml`, fixed the change-detection in `9eb743a` — run #4 then failed-fast on their side, reading UI logs).
+- **ratatoskr `:8392` (composite Bifrost provider) added** to the personal
+  WT `BIFROST_CLIENT_ALLOWED_HOSTS` (footgun-safe recreate: inline sha pin +
+  `--pull never`, the `.env` still pins stale `:latest`). And ratatoskr's
+  **admin.events.read key (ratatoskr#11) BROKERED** with worldtree-dev — the
+  scope is tier-derived (not mint-settable), so worldtree-dev defines the
+  isolated admin-tier user/scope, then I mint + secure-deliver. NON-URGENT.
 
-- **ratatoskr Tier-3 MEMORY plane wired + live-smoke driven — persist+dispatch GREEN, recall-injection OPEN (handed off, OUT of loop per operator).** The missing server-side step was the per-PORT url-guard allowlist: added `10.100.10.50:8391` to personal's `BIFROST_CLIENT_ALLOWED_HOSTS` (affect `:8390` was already in; via `playbooks/wire-personal-worldtree-memory-allowlist.yaml`, c99aa49). Consumer key REUSED (plane-agnostic, no mint). Drove define→bind→persist(async `upsert_many` 200)→recall(`search` 200); ratatoskr's `memory.db` durable + 1024-dim indexed. **Promotion gate = a consumer agent defined WITH a `memory` block (`ctx.memory_config≠None`), NOT `agent_self_enabled`.** Cross-session recall-INJECTION root-caused to a SCORE-SHAPE seam: Worldtree's injector reads `score_breakdown.{similarity,salience}`, but ratatoskr's `search()` returns top-level `score` + `chunk.salience` (0.9, ≥0.85) → both gate-reads collapse to 0 → filtered AFTER a clean 200. Handed to worldtree-dev (#295 recall leg OPEN; their ADR-0009 wire-spec call — bifrost returns `score_breakdown`, or Worldtree reads existing fields). (auto-memory `reference_bifrost_plane_wiring`) — **[2026-06-16] FLIPPED →** worldtree-dev's re-triage + my `:8391`-log pull DISPROVED the score_breakdown-shape theory: the wire shape is FINE (Worldtree *synthesizes* `score_breakdown` from the `score`+`chunk.salience` ratatoskr emits). Real cause = the COLD cross-session recall sends `scope_filter` with an extra `agent_self` axis → ratatoskr's conjunctive INV-005 `_scope_matches` drops the `{end_user}`-only chunks → 0 hits. Now Worldtree-side, tracked **Worldtree #297**. (auto-memory `project_worldtree_295_recall_rootcause`)
+- **claude-bot granted ADMIN on vh/Worldtree** (2026-06-20, via operator
+  vh-admin → migration-enabler) so I can self-serve WT deploy re-runs +
+  provision worldtree-dev's own Actions token (pending their gitea username).
 
-- **Worldtree demo + personal `character` model = mistral-small-4** (flipped from qwen3.6-35-a3b, 2026-06-16) — reordered `model_roles.yaml` `character.binds` mistral-first (first bind = default), qwen kept in the switch-allowlist; applied via pin-safe recreate, fresh-agent resolution verified. Backups `model_roles.yaml.bak-pre-mistral-character`.
+- **arbo + brokkr litellm keys = `all-proxy-models`** (verified — full gateway
+  access incl. paid GLM, no key change needed). `all-agents-local` shared key
+  stays fenced to free local models only. claude-bot Gitea service account is
+  the credential-migration foundation (admin on vh/arbo + now vh/Worldtree).
 
-- **ana-ml2 GPU layout (2026-06-19) — both 96GB Blackwells full, ONE model per
+- **Worldtree config-propagation:** demo+personal bind-mount their
-  card.** **GPU 0 = Qwopus3.5-122B-A10B** (the `gen`/`gen-reasoning` model;
+  `model_roles.yaml`/`providers.yaml` (`/opt/worldtree{,-personal}/config`,
-  OpenYourMind Kimi-distilled abliterated NVFP4, VISION-INTACT MoE; `stacks/qwopus3.5-122b/`,
+  infra-ops-deployable); **pinned is IN-IMAGE** (worldtree-dev CI bakes it).
-  :8013, served-name `qwen3.5-122-a10b`). Full **256K** (262144) @ fp8 KV + CUDA
+  worldtree-dev authors the canonical config; infra-ops deploys to the
-  graphs, util 0.95 + `expandable_segments`, **92.7 tok/s** warm, 3.32x concurrency
+  bind-mounted instances byte-identical.
-  @256K, text+image+video, tool-calling `qwen3_coder`. **GPU 1 = qwen-image-bench**
-  (T2I quality JUDGE, NVFP4; replaced qwen3.6-35b-a3b; alias `image-judge`, :8014) +
-  granite-4.1-8b + selene-1-mini-8b + embed + rerank + reward — packed ~90.5/96 GB.
-  **Interconnect = PCIe only (PIX, NO NVLink)** → one-model-per-card is the DELIBERATE
-  optimal layout (zero cross-card traffic); TP=2 rejected this session (see Recent
-  decisions). mistral-small-4 (heretic) + qwen3.6-35b-a3b both displaced.
-  (20e796c, bfae924; auto-memory `reference_nvfp4_moe_loads_on_vllm_023`)
 
-- **arbo engine builds handed to comfy-dev; Gitea Actions runner LIVE on irv-ml1.**
+- **Still open from prior:** Mac Pro migration (`migration-plan.md`,
-  Operator approved comfy-dev owning arbo engine deploys (`deploy-engine.sh`,
+  workstation-only, hardware-gated); R17 v2 corpus push HELD (local-only
-  build-local). Runner = `irv-ml1-arbo` (act_runner v0.2.13, **host-executor as
+  soundgasm-derived, distribution barred); disclosed-keys hygiene rotation
-  lkraven**, repo-scoped to vh/arbo, labels `self-hosted`+`irv-ml1`, systemd unit
+  queue; clean legacy `news-digest` on ana-docker.
-  `act_runner.service`, polls the INTERNAL gitea route). comfy-dev to commit
-  `.gitea/workflows/deploy.yml` — Phase 1 = engine-only auto-deploy (catalog stays
-  on the :9009 webhook), Phase 2 catalog cutover later. infra-ops = box-level
-  escalation only (runner, the `package:write` PAT once corviduo-org exists, webhook,
-  the arbo-deploy.sh HEAD-baseline restart-detection bug still open).
-
-- **claude-bot Gitea service account STOOD UP** (credential-migration foundation) —
-  user id=3, non-admin/scoped, token `arbo-ci`=write:package+read:repository, creds
-  at `~/.config/claude-bot/{gitea-password,gitea-token}` (mode 600). Registry-push
-  DEFERRED: vh is a USER not an org, so Gitea won't grant a scoped service account
-  package-write to `vh/*`. (auto-memory `project_migrate_infra_access_to_claude_credentials`)
-
-- **Worldtree demo + personal MIGRATED to v0.35.10 capability-named profiles** —
-  persona substrate moved qwen→Mistral (profiles `dialogue`/`classification`→
-  mistral-small-4, `deep-reasoning`→mistral-small-4-reasoning), Domari→selene; qwen
-  heretic/heavy retired. Both green. The win: future model swaps (the abliterated
-  Mistral) are now bind-mount-only `providers.yaml` edits — no image/agent/coupling.
-
-- **R18 clip+caption auto-assist models STAGED on ana-ml2** (brokkr/Eitri) — WhisperX + PANNs CNN14_DecisionLevelAtt + Qwen3-Omni-30B-A3B-Captioner at Eitri's `/srv/r18-clip-caption` path contract (symlink → /tank; dedicated `r18clip` user, `workspaces/` 0700). Stub-level offline smoke PASSED (lo-only netns); Eitri closed the scaffold deliverable. AWAITING: Eitri/brokkr's concrete model runners (replace the deterministic stubs) → re-smoke the real path; AND the **hard real-voice gate** — no real victim voice through the shared-box captioner until ephemeral-(b) controls (in-VRAM-only, no shared-disk persistence, access-logged + auditd stood up) are built AND explicitly confirmed to brokkr. (auto-memory `project-r18-clip-caption-staging`)
-
-- **MastMed prototype LIVE + public** — gated static-SPA container on ana-docker:8089 (caddy `basic_auth`, user `mastmed`) fronted by a cloudflared **quick tunnel** (ephemeral `*.trycloudflare.com` — dies on tunnel-container restart → new URL). Swap to a named tunnel (operator's scoped CF token + hostname) for a stable URL anytime; container + auth unchanged.
-
-- **comfyui (irv-ml1) VAE-decode SEGFAULT — RESOLVED 2026-06-15.** Root cause was
-  `comfy_aimdo`'s cudaMalloc hooks conflicting with ComfyUI's load-bearing
-  `--disable-cuda-malloc`: with the optimized backend disabled (cu129<cu130) but
-  the hooks still installed, VAE-decode allocation hit the dead path → native
-  segfault. NOT cu130 (that would re-enable the allocator we deliberately disable).
-  comfy-dev self-fixed: patched `comfy_aimdo/control.py` `init()` → return False
-  (aimdo was non-functional anyway), folded into reprovision-venv.sh (their commit
-  d5a2233). Verified clean across zimage/flux/SDXL.
-
-- **Arbo prod: engine at v0.11.38** (three engine-only fixes 2026-06-16 for granite
-  IR-shape "AI unavailable" degradations: intent_summary default, bare-string→list
-  coercion); now comfy-dev-owned (see the runner entry above). Registry still holds
-  NO arbo image (local-build-only) — that gap closes when the corviduo-org +
-  `package:write` token land (deferred). Auth OFF (WireGuard boundary).
-
-- **R16 splice-pivot yield probe RAN 2026-06-14** (was "awaiting go"). Inline-gen
-  arc stays CLOSED (v1 @ default decode); the splice pivot's cheap de-risk is now
-  executed: 45 standalone NVV ([moan_soft]/[moan_intense]/[groan] × seeds 1-15)
-  from Eleanor.wav on the v4 adapter + objective metrics (Resemblyzer spk-cosine,
-  MFCC-dist, F0, parselmouth HNR) → `/mnt/smithy/scratch/r16_audition_yield`,
-  served on brokkr-audition :8137. 45/45, 0 degenerate; spk-cosine 0.48-0.81.
-  Brokkr notified. NEXT: operator ear-bin (usable/impure) → brokkr's yield% /
-  survivorship / identity analysis. `gen_yield_probe.py` in `irv-ml1:~/r16-vmoan-harness`.
-
-- **R17 v2 corpus characterization still on the local irv-ml1 branch, push
-  HELD** (`r17-v2-characterization`). No-push rule AND ASR-content-exposure
-  call on the intimate-audio batch. Awaiting brokkr-collect or explicit push
-  approval. (LOCAL-ONLY soundgasm-derived corpus — distribution barred,
-  never echo transcripts to the bus.)
-
-- **Mac Pro migration is still the big open project** — `migration-plan.md`
-  (repo root): WORKSTATION-ONLY move of nh3-dev's Hat-1 dev env to an M2
-  Ultra Mac Pro on the 10.100 subnet. Hat-2 fleet sidecars STAY on the Linux
-  VM. **Phase 0** (push pushables + confirm the ~6 local-only repos) is the
-  only time-sensitive step. Cutover = `rsync` working trees, NOT re-clone.
-  Everything else waits on hardware.
-
-- **sglang-vs-vLLM bench stack staged but parked** (`5f049cb`) — the
-  originating SGLang-RadixAttention question resolved (vLLM v1 defaults
-  prefix-caching ON, now pinned explicit on granite+qwen). NVFP4 chase
-  abandoned. MAX_JOBS-on-shared-prod foot-gun caps any from-source build on
-  ana-ml2.
-
-- **pi + GLM 5.1 harness live on nh3-dev** — `glm` launcher runs pi against
-  `glm-5.1` (thinking-off) via the LiteLLM gateway; `glm-5.1-reasoning` for
-  opt-in thinking.
-
-- **Worldtree config-propagation lane is mature + humming** — pre-merge
-  delta-ping → sync-on-merge to demo+personal bind-mounts; pinned stays
-  pre-cutover.
-
-- **Disclosed-keys hygiene queue** (rotate at convenience): HF token, the
-  wt-personal keys, Gitea runner reg token, MINIFLUX_PASSWORD, ZAI_API_KEY.
-  (The shared all-agents LiteLLM key is intentional, not hygiene-debt.)
-
-- **Still open from prior:** clean legacy `news-digest` on ana-docker; the
-  `docker push 60s ceiling` mystery uninstrumented.
 
 ## Recent decisions
 
-- `[2026-06-19]` **litellm task-aliases: `classifier`→granite-4.1-8b, `summarizer-large`→gen/qwopus** (`summarizer`→granite already existed; duplicate-entry aliases per convention). Also added then REMOVED `gen-nt`/`gen-reasoning-nt` (a dummy `noop` tool + `tool_choice:none` baked in so a non-empty tools array always reaches vLLM) — operator reversed; redundant with the global `strip_empty_tools` hook. Live-added the new aliases via `/model/new` (no bounce), then finalized them config-sourced on an operator-authorized bounce (which also dropped the lingering gen-nt). (ef45f6d, 2e5ab72, 75bd4c3)
+- `[2026-06-20]` **Worldtree capability-gateway aliases stood up; NO `wt-`
+  prefix (operator) — reuse the existing capability convention.** gen/
+  gen-reasoning/classifier/summarizer/chat-judge/reranker/scalar-judge
+  (passthrough)/gen-frontier/-reasoning. The **swap-transparency rule**:
+  embedding stays model-named (`qwen3-embedding`) because vectors aren't
+  swap-transparent; everything else can swap behind a stable alias. (d372103,
+  7819f96)
 
-- `[2026-06-19]` **gateway-chat: model-smoking web chat enhanced + stood up as a PERSISTENT container.** Added `/v1/models` auto-discovery + image-upload (vision smoke) to `tools/gateway-chat.html`; deployed as a static-serve stack (`stacks/gateway-chat/`, nginx:alpine, `http://10.250.50.70:8091` on ana-docker, homepage tile). `conf/index.html` mirrors `tools/gateway-chat.html` (re-sync one-liner in the stack README). (4c40b9f, 740bcae)
+- `[2026-06-20]` **Removed the litellm `*` wildcard → llama-swap** (operator;
+  llama-swap decommissioned, :9292 confirmed down). Unknown/typo'd models now
+  404 cleanly instead of a misleading "Connection error" — closes the
+  long-standing footgun. (d0eb09c)
 
-- `[2026-06-19]` **pi on nh3-dev wired to `gen`** — added `gen`/`gen-reasoning` (vision-capable, thinking off/on, 262K ctx) to `~/.pi/agent/models.json` (litellm provider) + a `gen` launcher (`~/.local/bin/gen`) mirroring `glm`. Verified end-to-end (`gen -p` → `PI_GEN_OK`). Local nh3-dev box config, NOT a repo artifact (like the existing `glm` launcher). (untracked by operator choice — local box wiring)
+- `[2026-06-20]` **Langfuse retired** (operator) — redundant with LiteLLM
+  native spend_logs + a 6-container crash-prone surface (the 94 GB clickhouse
+  spew). Callbacks dropped, stack torn down. (f8eda1c)
 
-- `[2026-06-19]` **`gen` model → Qwopus3.5-122B-A10B (vision-intact NVFP4), full 256K @ fp8.** OpenYourMind/Qwopus3.5-122B-A10B-Kimi-K2.6-destilled-abliterated-NVFP4 on ana-ml2 GPU 0, served-name `qwen3.5-122-a10b` (gen/gen-reasoning/qwen-large route unchanged). Replaced the bjk110 text-only qwen3.5-122b (which had replaced mistral-small-4 earlier same-day). **KEY FINDING: the STABLE vLLM image + fp8 KV reaches the full 262144 — nightly+turboquant-4bit-KV was UNNECESSARY** (the hybrid SSM+attn KV pool is small; 11GB fp8 = 870k tokens = 3.32x concurrency @256K). graphs ON → 92.7 tok/s warm; util 0.95 + `expandable_segments` (0.96 OOMs the FusedMoE workspace); text+image+video + tool-calling qwen3_coder all verified. (20e796c, 5b06514)
+- `[2026-06-20]` **OmniVoice streaming `/tts` — vendor chatterbox-fast's
+  scheduler, not its whole package** (operator-approved vendor-copy). Diffusion
+  TTFA floor finding → tuned 16-step streaming + aggressive packing; chatterbox-
+  fast remains the lowest-latency front-end. (288d085, cd92b85)
 
-- `[2026-06-19]` **TP=2 across the two ana-ml2 Blackwells REJECTED** (operator asked; recommended against). `nvidia-smi topo -m` = `PIX` (PCIe single-bridge, **NO NVLink** — datacenter-only). TP all-reduces ~twice/layer over PCIe (~64GB/s vs NVLink ~900GB/s) → all-reduce-bound → SLOWER for models that already fit + would evict the 6 GPU-1 services. **One-model-per-card is the optimal layout for non-NVLinked cards** (zero cross-card traffic). If a single >96GB model is ever wanted, the path is PIPELINE parallelism (PP=2, 1 hop/token) + GPU-1 relocation — NOT TP. (untracked by operator choice; "keep it there")
+- `[2026-06-20]` **claude-bot → ADMIN on vh/Worldtree** (operator-authorized;
+  one-time use of operator vh-admin to enable the migration) — claude-bot
+  self-serves WT deploys/tokens henceforth, reducing personal-cred fallbacks.
 
-- `[2026-06-19]` **qwen-image-bench (T2I quality judge, NVFP4) replaced qwen3.6-35b-a3b on GPU 1** (operator), aliased `image-judge`; comfy-dev/arbo repointed off the killed `qwen3.6-35b-a3b` name. (bfae924, 5dfce04)
+- `[2026-06-20]` **Daily image+buildx prune cron on ana-docker** (the missing
+  automation behind the recurring image bloat) — `lkraven` crontab, prune
+  unused images >48 h + per-builder buildx cache. The real systemic fix
+  (daemon log cap) is deferred to clean context.
 
-- `[2026-06-18]` **heretic abliterated Mistral Small 4 NVFP4 built + LIVE as `mistral-small-4`** (executes the 06-16 "abliteration planned"). `darkc0de/Mistral-Small-4-119B-2603-heretic` → in-house NVFP4 (vision bf16, `device_map=cpu`) → native format (HF Mistral4 is unserveable on vLLM) → drop-in stack `stacks/mistral-small-4-heretic/` under the same `--served-model-name mistral-small-4` (zero litellm change). A/B'd vs official (refusal+ability); operator: "heretic stays." Empirically byte-equivalent to the official NVFP4 (70.80 GB tensors, identical quant scope). (dd3a5c9, f566f61, `tools/mistral-small4-nvfp4/`)
+- `[2026-06-19]` **litellm task-aliases: `classifier`→granite-4.1-8b, `summarizer-large`→gen/qwopus** (`summarizer`→granite already existed). Also added then REMOVED `gen-nt`/`gen-reasoning-nt` (redundant with the global `strip_empty_tools` hook). (ef45f6d, 2e5ab72, 75bd4c3)
 
-- `[2026-06-18]` **irv-ml1 VRAM consolidation + comfy-dev torch-pin** (operator) — ComfyUI pinned to the A6000 exclusively (48 GB), audio zoo → 3090, downed dia2-2b/ace-step/csm-expressiva. comfy-dev's torch-pin: `DISABLE_UPGRADES=true` @ torch 2.12.1, SageAttention rebuilt against it. (a8550ad)
+- `[2026-06-19]` **gateway-chat: model-smoking web chat → PERSISTENT container.** `/v1/models` auto-discovery + image-upload; `stacks/gateway-chat/` (nginx:alpine, ana-docker:8091, homepage tile). (4c40b9f, 740bcae)
 
-- `[2026-06-18]` **ComfyUI acceleration set (9 nodes) installed for comfy-dev** on irv-ml1's `comfyui` (arbo's box) — work order delivered; comfy-dev wires + benchmarks (TeaCache→Wan first).
+- `[2026-06-19]` **pi on nh3-dev wired to `gen`** — `~/.pi/agent/models.json` + a `gen` launcher mirroring `glm`. Local box config, not a repo artifact. (untracked by operator choice)
 
-- `[2026-06-17]` **ana-ml2 `vm.overcommit_memory=1` made durable** (sysctl drop-in, `playbooks/ana-ml2-overcommit-memory.yaml`) — overcommit=0 + zero swap caps CommitLimit at ~RAM/2; the resident vLLM services ate the headroom so a large model-file mmap ENOMEM'd despite ~393 GB free. (fc88eff)
+- `[2026-06-19]` **`gen` model → Qwopus3.5-122B-A10B (vision-intact NVFP4), full 256K @ fp8.** ana-ml2 GPU 0, served-name `qwen3.5-122-a10b`. STABLE vLLM + fp8 KV reaches 262144 — nightly+turboquant-4bit-KV was UNNECESSARY. 92.7 tok/s warm. (20e796c, 5b06514)
 
-- `[2026-06-17]` **GLM 5.2 wired into litellm** (`glm-5.2` + `glm-5.2-reasoning`, z.ai passthrough, `extra_body.thinking.type` toggle — mirrors the GLM-5.1 split). (fe77a35)
+- `[2026-06-19]` **TP=2 across the two ana-ml2 Blackwells REJECTED** — `PIX` (PCIe, NO NVLink) → all-reduce-bound → SLOWER. One-model-per-card is optimal; PP=2 only if a >96 GB model is ever wanted. (untracked by operator choice)
 
-- `[2026-06-17]` **nh3-extdev stood up as a sudo-LESS infra-ops manager box** (operator) — key-only, password-locked, no NOPASSWD/docker (deliberately tighter than the fleet infra-ops identity); successor to nh3-ansible. Hosts `/opt/externs` client workspaces for pi-on-GLM-5.2 client agents. (a841eab; auto-memory `reference_infra_ops_sudo_identity`)
+- `[2026-06-19]` **qwen-image-bench (T2I judge, NVFP4) replaced qwen3.6-35b-a3b on GPU 1**, aliased `image-judge`. (bfae924, 5dfce04)
 
-- `[2026-06-16]` **litellm `strip_empty_tools` pre-call hook shipped** (`d1bea13`) — an empty `tools:[]` 500s vLLM ("tools must not be an empty array"); a global `litellm_settings.callbacks` CustomLogger pops it (+ orphaned `tool_choice`) before forwarding, so it covers EVERY vLLM model, not one. `drop_params` only drops unsupported PARAMS, not empty VALUES. Mounts beside config.yaml (litellm resolves callbacks relative to the config dir). Verified live across granite/mistral/stream. (`stacks/litellm/conf/strip_empty_tools.py`)
+- `[2026-06-18]` **heretic abliterated Mistral Small 4 NVFP4 built + LIVE as `mistral-small-4`** (in-house NVFP4, drop-in stack, A/B'd vs official, "heretic stays"). DISPLACED from GPU 0 on 2026-06-19 for the gen/Qwopus swap. (dd3a5c9, f566f61, `tools/mistral-small4-nvfp4/`)
 
-- `[2026-06-16]` **single-file `gateway-chat.html` playground shipped** (`984ca3d`, `tools/`) — zero-dep browser chat straight to the gateway (`:4000`, CORS open), system-prompt box, streaming SSE, renders `reasoning_content`, NEVER sends `tools`. Built because the LiteLLM admin-UI playground can't test vLLM-backed models (see Tried-and-abandoned). Serve on-request via `python3 -m http.server -d tools`.
+- `[2026-06-18]` **irv-ml1 VRAM consolidation + comfy-dev torch-pin** — ComfyUI pinned to the A6000 (48 GB), audio zoo → 3090; `DISABLE_UPGRADES=true` @ torch 2.12.1. (a8550ad)
 
-- `[2026-06-16]` **claude-bot granted ADMIN on vh/arbo** (operator) — arbo CI/CD gitea ops now route through the service account, not the operator's personal `vh` (the credential-migration directive). Operator added claude-bot as Administrator collaborator; claude-bot self-serves via its `gitea-password` (basic-auth = FULL perms; its stored token is package-write-only, can't change repo settings). (auto-memory `reference_claude_bot_gitea_creds`)
+- `[2026-06-18]` **ComfyUI acceleration set (9 nodes) installed for comfy-dev** on irv-ml1.
 
-- `[2026-06-16]` **ratatoskr Tier-3 MEMORY plane wired + live-smoke driven.** The missing server-side step was the per-PORT `BIFROST_CLIENT_ALLOWED_HOSTS` allowlist (`:8391`; consumer key reused, plane-agnostic — my earlier "reachability is the whole wiring" was incomplete). Persist + dispatch proven end-to-end against personal v0.35.3; cross-session recall-INJECTION root-caused to the `score_breakdown` shape seam and handed to worldtree-dev (#295 recall leg open). Driving a real memory-call needs a consumer agent defined WITH a `memory` block (`ctx.memory_config≠None`), not `agent_self_enabled`. (tracking: c99aa49 + auto-memory `reference_bifrost_plane_wiring`) — **[2026-06-16] FLIPPED:** the `score_breakdown`-shape root-cause was disproven; real cause = cold-recall `scope_filter` adds an `agent_self` axis → ratatoskr's conjunctive INV-005 match drops the `{end_user}`-only chunks → 0 hits. Now Worldtree #297. (auto-memory `project_worldtree_295_recall_rootcause`)
+- `[2026-06-17]` **ana-ml2 `vm.overcommit_memory=1` made durable** (sysctl drop-in) — overcommit=0 + zero swap caps CommitLimit; large model-file mmap ENOMEM'd despite ~393 GB free. (fc88eff)
 
-- `[2026-06-16]` **infra-ops = the DURABLE admin path on corviduo-dev** (operator granted standing admin; extends the 2026-06-15 bootstrap) — added a `Host corviduo-dev → User infra-ops` `~/.ssh/config` alias + repointed `servers/corviduo-dev/ssh-target` to `infra-ops@`; `ssh corviduo-dev` / `elway corviduo-dev` now land as infra-ops w/ NOPASSWD (vh = explicit-only human path). elway resolves via ssh-config, NOT the ssh-target file. (tracking: c99aa49 + auto-memory `reference_corviduo_dev_emergency_ops`)
+- `[2026-06-17]` **GLM 5.2 wired into litellm** (`glm-5.2` + `-reasoning`, z.ai passthrough, `extra_body.thinking.type` toggle). (fe77a35)
 
-- `[2026-06-16]` **demo + personal `character` model flipped qwen3.6-35-a3b → mistral-small-4** (operator) — in `model_roles.yaml` the FIRST `catalog_id` in a role's `binds:` is the default; reordered mistral-first (qwen retained in the switch-allowlist, non-destructive), applied via the PIN-SAFE api recreate (derive `WORLDTREE_IMAGE` from the matrix sibling — never a bare `compose up`, which falls to the crash-blocking stale `:latest`), verified a fresh character agent resolves model=mistral-small-4. (auto-memory `reference_corviduo_dev_emergency_ops`)
+- `[2026-06-17]` **nh3-extdev stood up as a sudo-LESS infra-ops manager box** (successor to nh3-ansible); hosts `/opt/externs` pi-on-GLM-5.2 client workspaces. (a841eab)
 
-- `[2026-06-16]` **NVFP4 ModelOpt MoE now LOADS on vLLM 0.23.0 — SUPERSEDES the
+- `[2026-06-14]` **STANDING DIRECTIVE: migrate ALL infra access to Claude-specific credentials.** Stand up service accounts (claude-bot Gitea user + scoped tokens, distinct litellm admin key); re-mint consumer creds under them; flag personal-cred fallbacks until done. (auto-memory `project_migrate_infra_access_to_claude_credentials`)
-  2026-06-14 "blocked" finding (#44081 fixed).** Confirmed the cached
-  `nvidia/Qwen3.6-35B-A3B-NVFP4` loads clean + generates on 0.23.0; swapped prod
-  qwen36 FP8→NVFP4 (~13 GB reclaimed on GPU 1, fp16 KV). Validated by brokkr (speed:
-  faster single-stream, ≥parity batched) + comfy-dev (vision anatomy-judge A/B: PASS).
-  One W4A4 tell: minor multi-step chained-reasoning slip — doesn't bite the vision
-  judge. Mistral's NVFP4 is a SEPARATE compressed-tensors path (day-0), distinct from
-  the ModelOpt one. (auto-memory `reference_nvfp4_moe_loads_on_vllm_023`)
 
-- `[2026-06-16]` **Mistral Small 4 = the GPU-0 tenant** (operator) — fills the slot
+_101 older entries archived to archival-memory.md._
-  the deferred creative-writing pick held, and supersedes llama-swap (now down).
-  NVFP4 119B MoE, pinned **vLLM v0.22.0** for working vision (0.23.0 breaks it),
-  256K context. Operator's creative-writing model; **abliteration planned** → then it
-  becomes the uncensored-creative model the persona substrate needs. Reasoning via
-  `reasoning_effort` (BINARY none|high — no medium). (tracking: `c77a9aa`/`9a49963`)
-
-- `[2026-06-16]` **Selene judge restored on vLLM FP8, NOT NVFP4** — W4A4 is too
-  aggressive for a precision judge validated at Q6_K; FP8 ≥ Q6_K fidelity, and Selene
-  is text-only Llama-3.1 so dynamic `--quantization fp8` has no vision tower to noise.
-  `selene-1-mini-8b` on GPU 1 :8011. Worldtree's Domari role repointed to it. (`c985ede`)
-
-- `[2026-06-16]` **GPU-1 buffer put to work: selene 0.13→0.17 + qwen36 0.32→0.34**
-  — the two KV-bound generative models that gained Worldtree consumers; granite + the
-  pooling models under-use their util, so growing THEM is wasted. (`e124a2f`)
-
-- `[2026-06-16]` **claude-bot Gitea service account stood up; corviduo-ORG conversion
-  TABLED** (operator). claude-bot is the credential-migration foundation. The proper
-  fix for scoped agent package-write is converting vh (a USER) → a `corviduo` ORG —
-  blast radius assessed (org creation = free; arbo-first migration = cheap, no registry
-  images; full fleet = incremental, ~8 active vh images / 50 repos) — but tabled for
-  later. Registry-push deferred until then. (auto-memory `project_migrate_infra_access_to_claude_credentials`)
-
-- `[2026-06-16]` **arbo engine builds → comfy-dev OWNERSHIP** (operator approved) +
-  Gitea Actions runner stood up on irv-ml1 (host-executor as lkraven). infra-ops keeps
-  box-level escalation (runner, package:write PAT pending corviduo-org, webhook). See
-  the runner entry in Current state. (tracking: this snapshot + arbo thread `01KV70AB2B4B…`)
-
-- `[2026-06-16]` **Worldtree persona substrate migrated qwen→Mistral via capability-
-  named profiles** (v0.35.8→v0.35.10; demo + personal). PRE-SYNC-FIRST is now the
-  standing pattern for any providers.yaml/profile change: sync config to the bind-mount
-  BEFORE the deploy image-swap restart, so new agents + new config land together (the
-  v0.35.9 merge-first attempt race'd → 14 crash-loops). (tracking: worldtree thread `01KV6X0M…`)
-
-- `[2026-06-14]` **STANDING DIRECTIVE: migrate ALL infra access to Claude-specific credentials.** Agents currently reuse the operator's PERSONAL creds for infra ops — vh Gitea **admin** via `tea` (used this session to mint a `read:package` token for ratatoskr, id 13, **under vh**), `sk-corvid` litellm master for vkey admin. Stand up service accounts (a `claude-bot` Gitea user + scoped tokens, a distinct litellm admin key); re-mint consumer creds under them; flag personal-cred fallbacks until done. (auto-memory `project_migrate_infra_access_to_claude_credentials`)
-
-_88 older entries archived to archival-memory.md._
 
 ## Tried and abandoned
 
-- `[2026-06-19]` **litellm CONFIG-loaded models can't be hot-removed — `/model/delete` is DB-only.** Deleting a config.yaml model by id returns `Model with id=… not found in db`; only a process restart drops it. So removing a config alias can't go live WITHOUT a bounce — it ages out on the next restart (the cleaned config just won't reload it). The INVERSE works: `/model/new` live-ADDS to the router (DB) with no bounce — but then a config copy + DB copy coexist after a restart (harmless dup; reconcile by `/model/delete`-ing the DB copy). Clean no-bounce-then-bounce add: live-add via API, delete the DB copy before the eventual bounce so config stays single-source.
+- `[2026-06-20]` **A 94 GB unrotated `langfuse-clickhouse` json.log was the
+  REAL disk-filler (not image bloat) — there was NO log rotation anywhere.**
+  No per-stack `logging:` limit and no docker-daemon default → ClickHouse's
+  verbose stdout grew unbounded to ~94 GB and filled ana-docker's `/`
+  (healthchecks then failed fleet-wide on `runc /tmp: no space left`). Lesson:
+  on a full disk, the symptom (all-unhealthy) is the FORK-can't-write
+  healthcheck, not the containers; find the runaway via per-container-dir `du`,
+  not `docker system df`. The systemic fix is a daemon-default `log-opts`
+  (`max-size`/`max-file`) — NOT just per-service patching.
 
-- `[2026-06-19]` **the litellm `*` wildcard still routes to DECOMMISSIONED llama-swap (`http://10.250.50.54:9292`) → a stale/typo'd model name returns a misleading `Connection error`, not `model not found`.** Bit a consumer (brokkr) calling the renamed-away `qwen-image-judge` (now `image-judge`): the orphaned name fell through `*` → dead llama-swap → connection error, reading like the judge was down — it was healthy throughout. Until the wildcard is removed or repointed, unknown/typo'd names fail illegibly. (incident 2026-06-19)
+- `[2026-06-20]` **OmniVoice's upstream "RTF 0.025 / 40×" does NOT hold —
+  diffusion has a ~fixed per-call overhead → TTFA floor + multi-chunk
+  streaming is a net loss.** Measured ~2.8× (32-step) / 5.6× (16-step) on the
+  3090; every chunk costs ~the same regardless of length, so naive
+  per-sentence streaming triples total gen time + starves. Use few/aggressive
+  chunks; chatterbox-fast (autoregressive) stays the low-TTFA front-end.
 
-- `[2026-06-19]` **A MoE card can't be filled to 0 bytes free — the FusedMoE transient workspace is the floor.** vLLM's FusedMoE kernel allocates a ~3.09 GB transient workspace OUTSIDE its `gpu-memory-utilization` budget, into free VRAM, during graph capture + inference. util 0.96 OOM'd by 0.1 GB on it (`tried to allocate 3.09 GiB, 2.99 free`), worsened by 4.2 GB of PyTorch reserved-but-unallocated fragmentation. FIX: `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True` (reclaims the fragmentation) + leave ~3.2 GB free (util ≤ ~0.95 on a tight-fit MoE). The workspace size is fixed; "fill to 0" is physically impossible for MoE.
+- `[2026-06-20]` **rest-server `.htpasswd: permission denied` = the ana-nas
+  NFS mount FAILED (ghost file on the local mount point), NOT a decommission.**
+  `mnt-backup.mount` stuck `failed` (fstab bare `defaults`, no retry) → the
+  rest-server serves an empty local dir with a root:root 0-byte `.htpasswd`.
+  Documented recovery in disaster-recovery.md. Don't tear down what looks like
+  a crash-looping legacy container until you've checked fstab + the docs — it
+  was the live ana-side restic target.
 
-- `[2026-06-19]` **vLLM decode tok/s: ALWAYS discard the first generation (cold warmup).** Cold single read = 24.8 tok/s; warm steady-state = 92.7 (3 runs identical). A cold read undersells decode ~3–4× — the first gen pays graph-replay/JIT warmup. Measure run 2+ over a ≥256-token output.
+- `[2026-06-20]` **Named buildx builders accumulate cache the standard prunes
+  MISS.** `docker image prune -a` + `docker builder prune` don't touch
+  `buildx_buildkit_<name>-builder0_state` volumes (per-builder buildkit
+  state). Prune them with `docker buildx prune --builder <name> -af`, or
+  `docker volume rm` orphaned ones (builder already removed).
 
-- `[2026-06-19]` **For full native 256K on one 96GB card, nightly+turboquant-4bit-KV was unnecessary for the Qwopus MoE.** Stable fp8 KV already fits 262144 — the hybrid SSM+attention model caches KV only on its attention layers, so the pool is small (11 GB fp8 = 870k tokens). 4-bit turboquant KV (nightly-only) only buys MORE concurrency, at a long-context-recall risk + FA2 fallback (incompatible with FA3). Reach for fp8 first; 4-bit only if you need heavy concurrency at long context.
+- `[2026-06-20]` **On ana-docker I am `lkraven` (no NOPASSWD sudo), not
+  infra-ops.** A `sudo cp` of the litellm config silently failed (password
+  prompt) and I restarted the gateway on an unchanged config (one wasted
+  bounce). The config + most stack files are `lkraven`-owned → plain `cp`,
+  no sudo. Verify the SSH user/ownership before reaching for sudo.
 
-- `[2026-06-18]` **mmartial `comfyui-nvidia-docker` image: root pip installs CRASH-LOOP the container.** `docker exec -u 0 pip install` (the documented node-install pattern) leaves root-owned files in the uid-1000 venv; the image's boot script re-manages that venv AS uid 1000 (its torch-upgrade step) → `Permission denied` on `setuptools/__pycache__` → `Torch installation failed` → crash loop (looks like a torch bug, is ownership). FIX: `chown -R 1000:1000 /comfy/mnt/venv` after any root install (host: `/worktank/comfyui/run/venv`; if crash-looping too fast to exec, `docker stop` → chown host path → `start`). The image ALSO auto-upgrades torch every boot (`USE_PIPUPGRADE`) → compiled exts drift; pin with `DISABLE_UPGRADES=true`. (auto-memory `reference_irv_ml1_comfyui_mmartial`)
+- `[2026-06-19]` **litellm CONFIG-loaded models can't be hot-removed — `/model/delete` is DB-only.** Only a process restart drops a config model. The INVERSE works: `/model/new` live-ADDS (no bounce) but dups with config on restart. Clean no-bounce-add: live-add via API, delete the DB copy before the eventual bounce.
 
-- `[2026-06-17]` **Mistral HF→NVFP4 quant: the model-placement knob is the whole game.** `device_map="auto"` fills GPU0 → OOM during MoE un-fusing; constraining with `max_memory` offloads experts to the *meta* device → `Cannot copy out of meta tensor`. The working config is `device_map="cpu"` (CPU-resident model, sequential pipeline onloads each layer to GPU0). Plus: read shards with plain `read()` + `safetensors.torch.load(bytes)`, NOT `safe_open` (mmaps the whole shard → ENOMEM on `/tank` ZFS for the 50 GB shard, regardless of free RAM/overcommit). And llm-compressor's NVFP4 output KEEPS the `model.` prefix (not prefix-shifted).
+- `[2026-06-19]` **the litellm `*` wildcard routed to DECOMMISSIONED llama-swap → a stale/typo'd model name returned a misleading `Connection error`, not `model not found`.** Bit a brokkr call to the renamed `qwen-image-judge`. **→ RESOLVED 2026-06-20: wildcard removed (d0eb09c) — unknown models now 404 cleanly.**
 
-- `[2026-06-17]` **HF-format Mistral Small 4 is UNSERVEABLE on vLLM** — there is no HF `Mistral4` backbone in any vLLM version; it serves ONLY via the native loader (`--config-format/--load-format/--tokenizer-mode mistral` against params.json + consolidated*.safetensors + tekken.json). So a HF-format quant MUST be converted to native before it can serve.
+- `[2026-06-19]` **A MoE card can't be filled to 0 bytes free — the FusedMoE transient workspace (~3.09 GB) is the floor**, allocated OUTSIDE `gpu-memory-utilization`. FIX: `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True` + leave ~3.2 GB free (util ≤ ~0.95 on a tight MoE).
 
-- `[2026-06-18]` **ComfyUI custom nodes break on version-assumption drift** — the box runs 0.24.1 (not the 0.19.3 a work order assumed); 0.24.1 refactored `precompute_freqs_cis` into class methods, and TeaCache imports it at MODULE level → kills the WHOLE node (guard the LTX-only import). Also `pip install tensorrt` defaults to **cu13** libs against a cu12.9 stack → use `tensorrt-cu12`.
+- `[2026-06-19]` **vLLM decode tok/s: ALWAYS discard the first generation (cold warmup).** Cold ≈ 24.8 tok/s vs warm 92.7 — a cold read undersells decode ~3–4×. Measure run 2+ over a ≥256-token output.
 
-- `[2026-06-16]` **litellm 500 `Router.acompletion()/aembedding() missing 'messages'/'input'` = a request missing `Content-Type: application/json`, NOT a gateway outage.** curl `-d` defaults to form-encoding → litellm can't parse the JSON body → `data` reaches the router without `messages`/`input` → 500 (should be a 400; litellm #16993). My own diagnostic calls dropped the header → I misread it as a gateway outage and needlessly bounced the gateway ~4× chasing a phantom (image/version/config were fine throughout; a malformed UI-added "Mistral Story Eval" model in the DB was a red herring I deleted). ALWAYS send `-H "Content-Type: application/json"` testing litellm; reproduce with a header'd call before declaring a litellm incident.
+- `[2026-06-19]` **For full native 256K on one 96 GB card, nightly+turboquant-4bit-KV was unnecessary for the Qwopus MoE.** Stable fp8 KV already fits 262144 (hybrid SSM+attn caches KV only on attn layers → small pool). Reach for fp8 first.
 
-- `[2026-06-16]` **LiteLLM admin-UI playground can't test vLLM-backed models** — it auto-sends empty `tools:[]`, vLLM 400s (litellm #6228); the gateway `strip_empty_tools` hook is a PROXY hook and structurally can't reach the UI's in-process `litellm.completion()` call. Off-ramp = `tools/gateway-chat.html`. (Langfuse playground also out: its SSRF guard blocks internal-IP LLM connections, wontfix Langfuse #13097.) (auto-memory `reference_litellm_ui_playground_vllm_deadend`)
+- `[2026-06-18]` **mmartial `comfyui-nvidia-docker`: root pip installs CRASH-LOOP the container.** `docker exec -u 0 pip install` leaves root-owned files in the uid-1000 venv → the boot script (uid 1000) fails its torch step → crash loop. FIX: `chown -R 1000:1000 /comfy/mnt/venv`; pin with `DISABLE_UPGRADES=true`. (auto-memory `reference_irv_ml1_comfyui_mmartial`)
 
-- `[2026-06-16]` **A gitea Actions run that "looks like it never fired" (no image, untouched build dir) usually DID fire then skipped or failed-fast.** Bit arbo TWICE: run #3 looked-never-fired but SKIPPED (change-detection `engine=0`); run #4 looked-never-fired but FAILED in ~7s (early-step). Check the run LIST + status via claude-bot (`GET /repos/{o}/{r}/actions/tasks`) BEFORE a runner-connection/dispatch theory. CORRECTION (2026-06-16, proven): action logs ARE readable via API at the per-JOB endpoint — `GET /api/v1/repos/{o}/{r}/actions/runs/{run}/jobs` → take a job `.id` → `GET /api/v1/repos/{o}/{r}/actions/jobs/{job_id}/logs` returns the full plain-text log (claude-bot basic-auth, internal route `10.250.50.70:3000`). Only the RUN-level `/runs/{id}/logs` 404s; the per-job one works, so no UI needed. Gotcha when reading: gitea marks **every** step `conclusion:failure` once any step fails — the per-step conclusions mislead; trust the log body + step *timestamps* (the step with real duration is the culprit) not the conclusions. (Used 2026-06-16 to diagnose arbo run #5/task 1175 = test-gate `uv run pytest` → `Failed to spawn: pytest` = pytest absent from the fresh-checkout venv sync, NOT the SSH-clone comfy-dev suspected.)
+- `[2026-06-18]` **ComfyUI custom nodes break on version-assumption drift** — box runs 0.24.1; TeaCache imports `precompute_freqs_cis` at MODULE level (guard the LTX-only import). `pip install tensorrt` defaults to cu13 → use `tensorrt-cu12`.
 
-- `[2026-06-16]` **vLLM 0.23.0 breaks Mistral MULTIMODAL at startup** (`MistralCommonImageProcessor
+- `[2026-06-17]` **Mistral HF→NVFP4 quant: the placement knob is the whole game.** `device_map="cpu"` (CPU-resident, sequential onload) works; `"auto"` OOMs, `max_memory` offloads to meta. Read shards with plain `read()` + `safetensors.torch.load(bytes)`, NOT `safe_open` (mmaps the 50 GB shard → ENOMEM on `/tank` ZFS).
-  has no attribute fetch_images`, #44911 regression ~0.22.1+; same class hit Mistral-3.1/
-  Devstral). The dummy-image profiling crashes. Workarounds: `--limit-mm-per-prompt
-  '{"image":0,"video":0}'` loads text-only; for VISION, pin **v0.22.0** (last pre-regression).
-  Tradeoff: v0.22.0 *buffers* reasoning (doesn't stream reasoning_content cleanly); a newer
-  build streams it but breaks vision — vision won (Mistral is the abliteration-baseline).
 
-- `[2026-06-16]` **Mistral `-reasoning` TTFT variance = Triton kernel JIT-compile-during-
+- `[2026-06-17]` **HF-format Mistral Small 4 is UNSERVEABLE on vLLM** — no HF `Mistral4` backbone; serves ONLY via the native loader. A HF-format quant MUST be converted to native first.
-  inference, NOT reasoning time.** `jit_monitor` logs the spike (`_compute_slot_mapping_kernel`,
-  `apply_token_bitmask_inplace_kernel` ← the tool-call/structured-output path). vLLM's startup
-  warmup covers only SOME shapes; a first-hit uncovered shape JITs a kernel (seconds), then
-  CACHES it — warm until container recreate (the Triton cache is IN-container, no volume, so
-  recreate = cold again). Secondary: GPU idle-down to P8/180 MHz between requests. Also:
-  Mistral `reasoning_effort` is BINARY (none|high only — medium/low → HTTP 400).
 
-- `[2026-06-16]` **Worldtree old-baseline instances need the FULL config set synced, not the
+_84 older entries archived to archival-memory.md._
-  changeset delta.** Personal (a PRE-mistral baseline) crash-looped (41 restarts) on
-  `DanglingBindingError: role 'echo' binds missing catalog entry 'qwen3.6-35-a3b-heretic'` —
-  its `model_roles.yaml` (2026-06-10, never migrated) still bound the retired heretic that the
-  synced v0.35.10 providers removed. The "model_roles unchanged — skip" was correct only for
-  demo (already at the v0.35.9 baseline). Old-baseline → sync providers + model_roles + defaults.
-
-- `[2026-06-16]` **vh is a Gitea USER, not an org → no scoped package-write for a service
-  account.** Gitea user-namespace packages are owner/admin-only, so claude-bot can't push to
-  `vh/arbo` without being admin. The fix is converting vh → a `corviduo` ORG (tabled). Also:
-  vh's `tea` token lacks `write:admin` — create gitea users via the server-side `gitea admin
-  user create` CLI (`docker exec -u git gitea …`), not the admin API.
-
-- `[2026-06-16]` **`docker compose up -d` on a multi-service vLLM stack recreates ALL its
-  services** (shared `.env` → all configs re-hash) → concurrent memory-profiling race
-  ("No available memory for the cache blocks"). Target one: `docker compose up -d <service>`.
-  (auto-memory `feedback_compose_up_recreates_whole_stack`)
-
-_76 older entries archived to archival-memory.md._