memory: snapshot — 2026-06-20

WT capability-gateway migration (aliases + swap-transparency rule + wildcard
removal + gen-frontier + v0.37.5 staged/triggered), OmniVoice streaming /tts
(diffusion TTFA floor), arbo v0.14.9, ratatoskr :8392 + admin.events.read
brokered, claude-bot admin on vh/Worldtree. Disk incident root-caused to a
94GB unrotated langfuse-clickhouse log (+ secondary image bloat) → fixed,
image/buildx prune cron added, Langfuse retired. Backup gap: rest-server-ana
is the ACTIVE ana-side restic target down ~months on a failed ana-nas NFS
mount (NOT decommissioned — docs corrected) → recovery deferred to clean
context, along with the docker-daemon log cap. Archived the [2026-06-16]
cluster (13 Recent + 8 Tried) to archival-memory.md.

archival-memory.md

@@ -481,6 +481,74 @@ _Entries moved out of persistent-memory.md to keep the active file scannable. Re
 - `[2026-06-14]` **LiteLLM infra-ops admin key provisioned** (operator) — resolves the LiteLLM half of the credential-migration directive; use it for ALL gateway ops (NOT `sk-corvid`). Value at `~/.config/litellm/infra-ops-key` (mode 600); gateway reachable directly from nh3-dev at 10.250.50.70:4000. (auto-memory `reference_litellm_infra_ops_key`)
   _Archived 2026-06-19._
 
+- `[2026-06-16]` **litellm `strip_empty_tools` pre-call hook shipped** (`d1bea13`) — an empty `tools:[]` 500s vLLM ("tools must not be an empty array"); a global `litellm_settings.callbacks` CustomLogger pops it (+ orphaned `tool_choice`) before forwarding, so it covers EVERY vLLM model, not one. `drop_params` only drops unsupported PARAMS, not empty VALUES. Mounts beside config.yaml (litellm resolves callbacks relative to the config dir). Verified live across granite/mistral/stream. (`stacks/litellm/conf/strip_empty_tools.py`)
+  _Archived 2026-06-20._
+
+- `[2026-06-16]` **single-file `gateway-chat.html` playground shipped** (`984ca3d`, `tools/`) — zero-dep browser chat straight to the gateway (`:4000`, CORS open), system-prompt box, streaming SSE, renders `reasoning_content`, NEVER sends `tools`. Built because the LiteLLM admin-UI playground can't test vLLM-backed models (see Tried-and-abandoned). Serve on-request via `python3 -m http.server -d tools`.
+  _Archived 2026-06-20._
+
+- `[2026-06-16]` **claude-bot granted ADMIN on vh/arbo** (operator) — arbo CI/CD gitea ops now route through the service account, not the operator's personal `vh` (the credential-migration directive). Operator added claude-bot as Administrator collaborator; claude-bot self-serves via its `gitea-password` (basic-auth = FULL perms; its stored token is package-write-only, can't change repo settings). (auto-memory `reference_claude_bot_gitea_creds`)
+  _Archived 2026-06-20._
+
+- `[2026-06-16]` **ratatoskr Tier-3 MEMORY plane wired + live-smoke driven.** The missing server-side step was the per-PORT `BIFROST_CLIENT_ALLOWED_HOSTS` allowlist (`:8391`; consumer key reused, plane-agnostic — my earlier "reachability is the whole wiring" was incomplete). Persist + dispatch proven end-to-end against personal v0.35.3; cross-session recall-INJECTION root-caused to the `score_breakdown` shape seam and handed to worldtree-dev (#295 recall leg open). Driving a real memory-call needs a consumer agent defined WITH a `memory` block (`ctx.memory_config≠None`), not `agent_self_enabled`. (tracking: c99aa49 + auto-memory `reference_bifrost_plane_wiring`) — **[2026-06-16] FLIPPED:** the `score_breakdown`-shape root-cause was disproven; real cause = cold-recall `scope_filter` adds an `agent_self` axis → ratatoskr's conjunctive INV-005 match drops the `{end_user}`-only chunks → 0 hits. Now Worldtree #297. (auto-memory `project_worldtree_295_recall_rootcause`)
+  _Archived 2026-06-20._
+
+- `[2026-06-16]` **infra-ops = the DURABLE admin path on corviduo-dev** (operator granted standing admin; extends the 2026-06-15 bootstrap) — added a `Host corviduo-dev → User infra-ops` `~/.ssh/config` alias + repointed `servers/corviduo-dev/ssh-target` to `infra-ops@`; `ssh corviduo-dev` / `elway corviduo-dev` now land as infra-ops w/ NOPASSWD (vh = explicit-only human path). elway resolves via ssh-config, NOT the ssh-target file. (tracking: c99aa49 + auto-memory `reference_corviduo_dev_emergency_ops`)
+  _Archived 2026-06-20._
+
+- `[2026-06-16]` **demo + personal `character` model flipped qwen3.6-35-a3b → mistral-small-4** (operator) — in `model_roles.yaml` the FIRST `catalog_id` in a role's `binds:` is the default; reordered mistral-first (qwen retained in the switch-allowlist, non-destructive), applied via the PIN-SAFE api recreate (derive `WORLDTREE_IMAGE` from the matrix sibling — never a bare `compose up`, which falls to the crash-blocking stale `:latest`), verified a fresh character agent resolves model=mistral-small-4. (auto-memory `reference_corviduo_dev_emergency_ops`)
+  _Archived 2026-06-20._
+
+- `[2026-06-16]` **NVFP4 ModelOpt MoE now LOADS on vLLM 0.23.0 — SUPERSEDES the
+  2026-06-14 "blocked" finding (#44081 fixed).** Confirmed the cached
+  `nvidia/Qwen3.6-35B-A3B-NVFP4` loads clean + generates on 0.23.0; swapped prod
+  qwen36 FP8→NVFP4 (~13 GB reclaimed on GPU 1, fp16 KV). Validated by brokkr (speed:
+  faster single-stream, ≥parity batched) + comfy-dev (vision anatomy-judge A/B: PASS).
+  One W4A4 tell: minor multi-step chained-reasoning slip — doesn't bite the vision
+  judge. Mistral's NVFP4 is a SEPARATE compressed-tensors path (day-0), distinct from
+  the ModelOpt one. (auto-memory `reference_nvfp4_moe_loads_on_vllm_023`)
+  _Archived 2026-06-20._
+
+- `[2026-06-16]` **Mistral Small 4 = the GPU-0 tenant** (operator) — fills the slot
+  the deferred creative-writing pick held, and supersedes llama-swap (now down).
+  NVFP4 119B MoE, pinned **vLLM v0.22.0** for working vision (0.23.0 breaks it),
+  256K context. Operator's creative-writing model; **abliteration planned** → then it
+  becomes the uncensored-creative model the persona substrate needs. Reasoning via
+  `reasoning_effort` (BINARY none|high — no medium). (tracking: `c77a9aa`/`9a49963`)
+  _Archived 2026-06-20._
+
+- `[2026-06-16]` **Selene judge restored on vLLM FP8, NOT NVFP4** — W4A4 is too
+  aggressive for a precision judge validated at Q6_K; FP8 ≥ Q6_K fidelity, and Selene
+  is text-only Llama-3.1 so dynamic `--quantization fp8` has no vision tower to noise.
+  `selene-1-mini-8b` on GPU 1 :8011. Worldtree's Domari role repointed to it. (`c985ede`)
+  _Archived 2026-06-20._
+
+- `[2026-06-16]` **GPU-1 buffer put to work: selene 0.13→0.17 + qwen36 0.32→0.34**
+  — the two KV-bound generative models that gained Worldtree consumers; granite + the
+  pooling models under-use their util, so growing THEM is wasted. (`e124a2f`)
+  _Archived 2026-06-20._
+
+- `[2026-06-16]` **claude-bot Gitea service account stood up; corviduo-ORG conversion
+  TABLED** (operator). claude-bot is the credential-migration foundation. The proper
+  fix for scoped agent package-write is converting vh (a USER) → a `corviduo` ORG —
+  blast radius assessed (org creation = free; arbo-first migration = cheap, no registry
+  images; full fleet = incremental, ~8 active vh images / 50 repos) — but tabled for
+  later. Registry-push deferred until then. (auto-memory `project_migrate_infra_access_to_claude_credentials`)
+  _Archived 2026-06-20._
+
+- `[2026-06-16]` **arbo engine builds → comfy-dev OWNERSHIP** (operator approved) +
+  Gitea Actions runner stood up on irv-ml1 (host-executor as lkraven). infra-ops keeps
+  box-level escalation (runner, package:write PAT pending corviduo-org, webhook). See
+  the runner entry in Current state. (tracking: this snapshot + arbo thread `01KV70AB2B4B…`)
+  _Archived 2026-06-20._
+
+- `[2026-06-16]` **Worldtree persona substrate migrated qwen→Mistral via capability-
+  named profiles** (v0.35.8→v0.35.10; demo + personal). PRE-SYNC-FIRST is now the
+  standing pattern for any providers.yaml/profile change: sync config to the bind-mount
+  BEFORE the deploy image-swap restart, so new agents + new config land together (the
+  v0.35.9 merge-first attempt race'd → 14 crash-loops). (tracking: worldtree thread `01KV6X0M…`)
+  _Archived 2026-06-20._
+
 ## Tried and abandoned (archived)
 
 - `[2026-04-30]` task-board workflow with
@@ -956,3 +1024,53 @@ _Entries moved out of persistent-memory.md to keep the active file scannable. Re
 
 - `[2026-06-14]` **Fleet/colo hosts must reach gitea over the INTERNAL route, NOT the public IP.** `gitea.phasefinal.com` = public `38.120.12.44` (ana-srv1); gitea is a container on ana-docker, git-SSH `10.250.50.70:222` + HTTP `:3000`. A fleet host egressing to public `:22` gets fail2ban-banned after any retrying git loop → silently wedges webhook auto-deploys (`git fetch` times out under `set -euo pipefail`, aborts before reset). Bit irv-ml1's arbo deploy. `:22` on `10.250.50.70` is ana-docker's HOST sshd (deploy key → Permission denied), NOT gitea. Documented `docs/orientation.md` (`6e58e57`).
   _Archived 2026-06-19._
+
+- `[2026-06-16]` **litellm 500 `Router.acompletion()/aembedding() missing 'messages'/'input'` = a request missing `Content-Type: application/json`, NOT a gateway outage.** curl `-d` defaults to form-encoding → litellm can't parse the JSON body → `data` reaches the router without `messages`/`input` → 500 (should be a 400; litellm #16993). My own diagnostic calls dropped the header → I misread it as a gateway outage and needlessly bounced the gateway ~4× chasing a phantom (image/version/config were fine throughout; a malformed UI-added "Mistral Story Eval" model in the DB was a red herring I deleted). ALWAYS send `-H "Content-Type: application/json"` testing litellm; reproduce with a header'd call before declaring a litellm incident.
+  _Archived 2026-06-20._
+
+- `[2026-06-16]` **LiteLLM admin-UI playground can't test vLLM-backed models** — it auto-sends empty `tools:[]`, vLLM 400s (litellm #6228); the gateway `strip_empty_tools` hook is a PROXY hook and structurally can't reach the UI's in-process `litellm.completion()` call. Off-ramp = `tools/gateway-chat.html`. (Langfuse playground also out: its SSRF guard blocks internal-IP LLM connections, wontfix Langfuse #13097.) (auto-memory `reference_litellm_ui_playground_vllm_deadend`)
+  _Archived 2026-06-20._
+
+- `[2026-06-16]` **A gitea Actions run that "looks like it never fired" (no image, untouched build dir) usually DID fire then skipped or failed-fast.** Bit arbo TWICE: run #3 looked-never-fired but SKIPPED (change-detection `engine=0`); run #4 looked-never-fired but FAILED in ~7s (early-step). Check the run LIST + status via claude-bot (`GET /repos/{o}/{r}/actions/tasks`) BEFORE a runner-connection/dispatch theory. CORRECTION (2026-06-16, proven): action logs ARE readable via API at the per-JOB endpoint — `GET /api/v1/repos/{o}/{r}/actions/runs/{run}/jobs` → take a job `.id` → `GET /api/v1/repos/{o}/{r}/actions/jobs/{job_id}/logs` returns the full plain-text log (claude-bot basic-auth, internal route `10.250.50.70:3000`). Only the RUN-level `/runs/{id}/logs` 404s; the per-job one works, so no UI needed. Gotcha when reading: gitea marks **every** step `conclusion:failure` once any step fails — the per-step conclusions mislead; trust the log body + step *timestamps* (the step with real duration is the culprit) not the conclusions. (Used 2026-06-16 to diagnose arbo run #5/task 1175 = test-gate `uv run pytest` → `Failed to spawn: pytest` = pytest absent from the fresh-checkout venv sync, NOT the SSH-clone comfy-dev suspected.)
+  _Archived 2026-06-20._
+
+- `[2026-06-16]` **vLLM 0.23.0 breaks Mistral MULTIMODAL at startup** (`MistralCommonImageProcessor
+  has no attribute fetch_images`, #44911 regression ~0.22.1+; same class hit Mistral-3.1/
+  Devstral). The dummy-image profiling crashes. Workarounds: `--limit-mm-per-prompt
+  '{"image":0,"video":0}'` loads text-only; for VISION, pin **v0.22.0** (last pre-regression).
+  Tradeoff: v0.22.0 *buffers* reasoning (doesn't stream reasoning_content cleanly); a newer
+  build streams it but breaks vision — vision won (Mistral is the abliteration-baseline).
+  _Archived 2026-06-20._
+
+- `[2026-06-16]` **Mistral `-reasoning` TTFT variance = Triton kernel JIT-compile-during-
+  inference, NOT reasoning time.** `jit_monitor` logs the spike (`_compute_slot_mapping_kernel`,
+  `apply_token_bitmask_inplace_kernel` ← the tool-call/structured-output path). vLLM's startup
+  warmup covers only SOME shapes; a first-hit uncovered shape JITs a kernel (seconds), then
+  CACHES it — warm until container recreate (the Triton cache is IN-container, no volume, so
+  recreate = cold again). Secondary: GPU idle-down to P8/180 MHz between requests. Also:
+  Mistral `reasoning_effort` is BINARY (none|high only — medium/low → HTTP 400).
+  _Archived 2026-06-20._
+
+- `[2026-06-16]` **Worldtree old-baseline instances need the FULL config set synced, not the
+  changeset delta.** Personal (a PRE-mistral baseline) crash-looped (41 restarts) on
+  `DanglingBindingError: role 'echo' binds missing catalog entry 'qwen3.6-35-a3b-heretic'` —
+  its `model_roles.yaml` (2026-06-10, never migrated) still bound the retired heretic that the
+  synced v0.35.10 providers removed. The "model_roles unchanged — skip" was correct only for
+  demo (already at the v0.35.9 baseline). Old-baseline → sync providers + model_roles + defaults.
+  _Archived 2026-06-20._
+
+- `[2026-06-16]` **vh is a Gitea USER, not an org → no scoped package-write for a service
+  account.** Gitea user-namespace packages are owner/admin-only, so claude-bot can't push to
+  `vh/arbo` without being admin. The fix is converting vh → a `corviduo` ORG (tabled). Also:
+  vh's `tea` token lacks `write:admin` — create gitea users via the server-side `gitea admin
+  user create` CLI (`docker exec -u git gitea …`), not the admin API.
+  _Archived 2026-06-20._
+
+- `[2026-06-16]` **`docker compose up -d` on a multi-service vLLM stack recreates ALL its
+  services** (shared `.env` → all configs re-hash) → concurrent memory-profiling race
+  ("No available memory for the cache blocks"). Target one: `docker compose up -d <service>`.
+  (auto-memory `feedback_compose_up_recreates_whole_stack`)
+
+_76 older entries archived to archival-memory.md._
+  _Archived 2026-06-20._
+