esh-pfi-infrastructure/persistent-memory.md

# Persistent memory — eshpfi-management

_Last updated: 2026-06-16_

## Repo purpose

Reference workspace for PFI infrastructure: server inventory, canonical
Docker Compose stacks, ops playbooks, and conventions. Authoritative
copies of compose files live on the servers under
`/opt/docker/compose/<stack>/`; this repo mirrors them for version
control, editing, planning, and CI-driven deploys.

## Tools and conventions

Sister repos (separate gitea repos, deployed by playbooks here):

| Repo | Role | CI status |
|---|---|---|
| `vh/task-board` | MCP + web dashboard for assistant task state (port 7878) | push-to-main → CI deploys (2026-04-29) |
| `vh/vor` | Inquisitor UI sidecar (port 7879) | push-to-main → CI deploys (2026-04-29) |
| `vh/nevermore` | Twice-daily LLM-curated briefing (port 8181, replaces news-digest) | push-to-main → CI deploys (2026-04-30) |
| `vh/asset-engine` | Internal control plane over inference services (port 8200, LAN-direct) | push-to-main → CI deploys (2026-05-12) |
| `vh/althing` | Inter-agent message bus (chamber UI port 7881, forseti + agent-runner daemons, valkey IPC) | push-to-main → CI deploys (2026-05-14) |
| `vh/mead-hall` | Bifrost tool-provider sidecar (port 5173 on dev VM 10.100.10.50) | push-to-main → CI deploys (2026-05-16) |
| `vh/skaldsong` | Wizard + reader surface (port 8300, ana-docker, registry-pull pattern) | push-to-main → CI deploys (2026-05-19) |
| `vh/worldtree` | Conversation API (corviduo-dev demo :8080 / personal :8081 / pinned :8082) — Heimdall auth, Bifrost integration | push-to-main → CI deploys |
| `vh/yt-voice-clipper` | YouTube → diarized voice-clip dataset builder + audition console (irv-ml1 :8000) | push-to-main → **gitea-webhook auto-deploy** to irv-ml1 (2026-06-03) — see `docs/runbooks/ytvc-autodeploy.md` |
| `vh/arbo` | Catalog-driven ComfyUI engine (irv-ml1 :8201, comfy-dev owns engine/catalog/image) | push-to-main → **gitea-webhook auto-deploy** to irv-ml1 (catalog-only restart; infra side = `stacks/arbo/`) |

(`vh/volva` + Heid were re-architected from systemd daemons to Claude Code
session orchestrators 2026-06-08; their nh3-dev `.service` units were removed —
no longer deployed sidecars here. See Recent decisions.)

- **Two-layer backups** — Backrest orchestrates restic for file+DB (5
  fleet repos, daily 01:00 PDT); PBS-ANA primary + PBS-NH3 DR mirror for
  VM images. ana-nas is the SPOF for postgres + PBS-ANA datastore +
  cross-site restic targets — see `docs/runbooks/disaster-recovery.md`
  for the blast-radius matrix.

- **`pull-hf-repo.yaml`** is the canonical "get a HuggingFace
  model/dataset onto ana-ml2's shared cache at
  `/tank/aimodels/huggingface/`" playbook. Supports `--var repo_type=model|dataset|space`. Replaces ad-hoc `huggingface_hub.snapshot_download` patterns.

- **Worldtree admin auth — per-instance.** Each Worldtree deployment
  (demo :8080, personal :8081, pinned :8082) has its own Heimdall
  registry and its own bootstrap admin key. Infra-ops's stored
  long-lived admin key (`key_id 61419c92`) at
  `ana-docker:/opt/docker/conf/.secrets/worldtree-infra-ops-admin`
  auths against **demo only**. For personal-instance admin ops, fetch
  the bootstrap admin per-op via
  `docker exec worldtree-personal-worldtree-api-1 printenv WORLDTREE_BOOTSTRAP_ADMIN_KEY`
  on corviduo-dev. Used for `POST /admin/keys`, admin diagnostics
  (`/admin/sessions/<id>/{bifrost,tools}`, etc.).

- **Per-project user keys against personal Worldtree** (issued
  2026-05-19): `skaldsong:79744637` (nh3-dev iteration),
  `skaldsong:7c1dbbbe` (ana-docker prod), `althing:50d85460`,
  `mead-hall:a360822d`. Same `user_id=skaldsong` across both
  skaldsong keys → shared Heimdall agent slot; different `key_id`
  → independently rotatable. Pattern: mint via `/admin/keys`, drop
  value to `/tmp/wt-personal-<name>.key` mode 600, dev collects +
  shreds (DO NOT cat to chat transcript).

- **Skaldsong CD pattern (registry-pull).** Differs from althing /
  asset-engine which build-on-host. vh/skaldsong's CI builds and
  pushes `gitea.phasefinal.com/vh/skaldsong:<sha>` + `:latest`;
  `playbooks/deploy-skaldsong.yaml` on ana-docker pulls + recreates.
  SHA-pin only (no `:latest` health-gated advance yet). Prereq: host
  needs `docker login gitea.phasefinal.com` once (read:package PAT) —
  not currently in the workflow.

- **gitea internal route for fleet hosts.** gitea is a container on
  **ana-docker** — git-SSH `10.250.50.70:222`, HTTP `:3000`. Fleet/colo
  hosts must use this internal route, NOT public `gitea.phasefinal.com`
  (`38.120.12.44`, ana-srv1) — the public path fail2bans the host egress
  IP and wedges webhook deploys. `:22` on `10.250.50.70` is ana-docker's
  HOST sshd, not gitea. Full gotcha in `docs/orientation.md` → Git/gitea.

- **docker-as-root pattern** (for ops that have no admin API, e.g.
  `SqliteUserStore.set_bifrost_credentials`): on hosts where the SSH
  user is in the `docker` group but lacks passwordless sudo, run
  `docker run --rm -v <target-dir>:/wt -v /var/run/docker.sock:/var/run/docker.sock docker:cli sh -c "..."` to edit deploy-owned files
  without sudo. Documented with security warning in
  `servers/corviduo-dev/README.md`. docker-group membership is
  effectively root via bind-mount; treat as a sudo-equivalent grant.
  **Foot-gun: when running `docker compose` inside this sandbox,
  any relative path in compose.yaml (e.g. `${WORLDTREE_CONFIG_DIR:-./config}`)
  resolves against the sandbox CWD, but Docker daemon interprets the
  resulting path against the HOST filesystem. Always pass `-e VAR=/abs/path`
  to the docker run invocation for any relative-default config dir.**

- **`scripts/elway` sudo handling** — elway prompts for the sudo password
  ONCE via `getpass` before the first `sudo: true` step. That prompt is
  interactive → elway can't run unattended from a non-TTY tool if any step
  needs sudo. For sudo-free playbooks (no `sudo: true` steps) it runs fully
  non-interactive over key SSH. To create root-owned dirs WITHOUT host sudo,
  use the docker-daemon-root trick: `docker run --rm -v /worktank:/mnt alpine
  sh -c 'mkdir -p /mnt/<x> && chown -R 1000:1000 /mnt/<x>'`.

## Current state / in-flight

_As of 2026-06-16:_

- **LitBench-RM stood up → now ON-DEMAND / DOWN.** `SAA-Lab/Llama8B-CreativeWritingVerifier` (R19 creative-quality reward judge for brokkr/Dvalin) served on irv-ml1 A6000 via vLLM `--runner pooling` (`http://10.100.79.3:8202/classify`, raw passage text → scalar). brokkr-validated (does NOT penalize explicit content). Taken DOWN to on-demand (operator) — resident it held ~19.6 GB crowding comfyui's A6000 slot; weights staged on irv-ml1, ~90s respin (command in auto-memory). (auto-memory `reference_litbench_rm_irv_ml1`)

- **litellm gateway healthy; `strip_empty_tools` hook + `gateway-chat.html` shipped** (see Recent decisions). **arbo + brokkr litellm keys are already `all-proxy-models`** — verified, so "grant arbo/brokkr full access" was already done. claude-bot now has **admin on vh/arbo**; **arbo CI/CD Phase 1 is firing** (runner live; comfy-dev owns `deploy.yml`, fixed the change-detection in `9eb743a` — run #4 then failed-fast on their side, reading UI logs).

- **ratatoskr Tier-3 MEMORY plane wired + live-smoke driven — persist+dispatch GREEN, recall-injection OPEN (handed off, OUT of loop per operator).** The missing server-side step was the per-PORT url-guard allowlist: added `10.100.10.50:8391` to personal's `BIFROST_CLIENT_ALLOWED_HOSTS` (affect `:8390` was already in; via `playbooks/wire-personal-worldtree-memory-allowlist.yaml`, c99aa49). Consumer key REUSED (plane-agnostic, no mint). Drove define→bind→persist(async `upsert_many` 200)→recall(`search` 200); ratatoskr's `memory.db` durable + 1024-dim indexed. **Promotion gate = a consumer agent defined WITH a `memory` block (`ctx.memory_config≠None`), NOT `agent_self_enabled`.** Cross-session recall-INJECTION root-caused to a SCORE-SHAPE seam: Worldtree's injector reads `score_breakdown.{similarity,salience}`, but ratatoskr's `search()` returns top-level `score` + `chunk.salience` (0.9, ≥0.85) → both gate-reads collapse to 0 → filtered AFTER a clean 200. Handed to worldtree-dev (#295 recall leg OPEN; their ADR-0009 wire-spec call — bifrost returns `score_breakdown`, or Worldtree reads existing fields). (auto-memory `reference_bifrost_plane_wiring`) — **[2026-06-16] FLIPPED →** worldtree-dev's re-triage + my `:8391`-log pull DISPROVED the score_breakdown-shape theory: the wire shape is FINE (Worldtree *synthesizes* `score_breakdown` from the `score`+`chunk.salience` ratatoskr emits). Real cause = the COLD cross-session recall sends `scope_filter` with an extra `agent_self` axis → ratatoskr's conjunctive INV-005 `_scope_matches` drops the `{end_user}`-only chunks → 0 hits. Now Worldtree-side, tracked **Worldtree #297**. (auto-memory `project_worldtree_295_recall_rootcause`)

- **Worldtree demo + personal `character` model = mistral-small-4** (flipped from qwen3.6-35-a3b, 2026-06-16) — reordered `model_roles.yaml` `character.binds` mistral-first (first bind = default), qwen kept in the switch-allowlist; applied via pin-safe recreate, fresh-agent resolution verified. Backups `model_roles.yaml.bak-pre-mistral-character`.

- **ana-ml2 GPU layout RESHAPED again (2026-06-15/16) — both cards now full
  with NVFP4 tenants.** **GPU 0 = Mistral Small 4** (`mistral-small-4` stack,
  `mistralai/Mistral-Small-4-119B-2603-NVFP4`, 119B/6.5B-active MoE, :8010,
  gateway `mistral-small-4` + `mistral-small-4-reasoning`@effort=high). Pinned
  **vLLM v0.22.0** — the LAST release with working Mistral *vision* (#44911
  `fetch_images` regression breaks it on 0.22.1+/0.23.0). Serves the full native
  **256K context** (max-model-len 262144, max-num-seqs 32 — fits the tight card,
  ~5 GB free). Text + vision both work; reasoning via `reasoning_effort` (BINARY:
  none|high). Dedicated single-tenant; the operator's creative-writing model
  (abliteration planned → it succeeds llama-swap). **GPU 1 = qwen36 swapped
  FP8→NVFP4** (`nvidia/Qwen3.6-35B-A3B-NVFP4`, fp16 KV, util 0.34, :8007, gateway
  name `qwen3.6-35b-a3b` UNCHANGED + `-thinking` variant) — the ModelOpt NVFP4
  MoE LOADS on 0.23.0 now (the 2026-06-14 "blocked" finding is RESOLVED). +
  **granite restored** (0.34/131072) + **Selene FP8 judge added** (`selene-1-mini-8b`,
  AtlaAI Selene-1-Mini-Llama-3.1-8B dynamic fp8, util 0.17, :8011, ctx 32768) +
  embed/rerank/reward. ~5.6 GB free. Prefix-caching ON on all 4 generative.
  **llama-swap is DOWN** (decommissioned from GPU 0 for Mistral; its qwen GGUF
  consumers migrated to the gateway). (auto-memory `reference_nvfp4_moe_loads_on_vllm_023`)

- **arbo engine builds handed to comfy-dev; Gitea Actions runner LIVE on irv-ml1.**
  Operator approved comfy-dev owning arbo engine deploys (`deploy-engine.sh`,
  build-local). Runner = `irv-ml1-arbo` (act_runner v0.2.13, **host-executor as
  lkraven**, repo-scoped to vh/arbo, labels `self-hosted`+`irv-ml1`, systemd unit
  `act_runner.service`, polls the INTERNAL gitea route). comfy-dev to commit
  `.gitea/workflows/deploy.yml` — Phase 1 = engine-only auto-deploy (catalog stays
  on the :9009 webhook), Phase 2 catalog cutover later. infra-ops = box-level
  escalation only (runner, the `package:write` PAT once corviduo-org exists, webhook,
  the arbo-deploy.sh HEAD-baseline restart-detection bug still open).

- **claude-bot Gitea service account STOOD UP** (credential-migration foundation) —
  user id=3, non-admin/scoped, token `arbo-ci`=write:package+read:repository, creds
  at `~/.config/claude-bot/{gitea-password,gitea-token}` (mode 600). Registry-push
  DEFERRED: vh is a USER not an org, so Gitea won't grant a scoped service account
  package-write to `vh/*`. (auto-memory `project_migrate_infra_access_to_claude_credentials`)

- **Worldtree demo + personal MIGRATED to v0.35.10 capability-named profiles** —
  persona substrate moved qwen→Mistral (profiles `dialogue`/`classification`→
  mistral-small-4, `deep-reasoning`→mistral-small-4-reasoning), Domari→selene; qwen
  heretic/heavy retired. Both green. The win: future model swaps (the abliterated
  Mistral) are now bind-mount-only `providers.yaml` edits — no image/agent/coupling.

- **R18 clip+caption auto-assist models STAGED on ana-ml2** (brokkr/Eitri) — WhisperX + PANNs CNN14_DecisionLevelAtt + Qwen3-Omni-30B-A3B-Captioner at Eitri's `/srv/r18-clip-caption` path contract (symlink → /tank; dedicated `r18clip` user, `workspaces/` 0700). Stub-level offline smoke PASSED (lo-only netns); Eitri closed the scaffold deliverable. AWAITING: Eitri/brokkr's concrete model runners (replace the deterministic stubs) → re-smoke the real path; AND the **hard real-voice gate** — no real victim voice through the shared-box captioner until ephemeral-(b) controls (in-VRAM-only, no shared-disk persistence, access-logged + auditd stood up) are built AND explicitly confirmed to brokkr. (auto-memory `project-r18-clip-caption-staging`)

- **MastMed prototype LIVE + public** — gated static-SPA container on ana-docker:8089 (caddy `basic_auth`, user `mastmed`) fronted by a cloudflared **quick tunnel** (ephemeral `*.trycloudflare.com` — dies on tunnel-container restart → new URL). Swap to a named tunnel (operator's scoped CF token + hostname) for a stable URL anytime; container + auth unchanged.

- **comfyui (irv-ml1) VAE-decode SEGFAULT — RESOLVED 2026-06-15.** Root cause was
  `comfy_aimdo`'s cudaMalloc hooks conflicting with ComfyUI's load-bearing
  `--disable-cuda-malloc`: with the optimized backend disabled (cu129<cu130) but
  the hooks still installed, VAE-decode allocation hit the dead path → native
  segfault. NOT cu130 (that would re-enable the allocator we deliberately disable).
  comfy-dev self-fixed: patched `comfy_aimdo/control.py` `init()` → return False
  (aimdo was non-functional anyway), folded into reprovision-venv.sh (their commit
  d5a2233). Verified clean across zimage/flux/SDXL.

- **Arbo prod: engine at v0.11.38** (three engine-only fixes 2026-06-16 for granite
  IR-shape "AI unavailable" degradations: intent_summary default, bare-string→list
  coercion); now comfy-dev-owned (see the runner entry above). Registry still holds
  NO arbo image (local-build-only) — that gap closes when the corviduo-org +
  `package:write` token land (deferred). Auth OFF (WireGuard boundary).

- **R16 splice-pivot yield probe RAN 2026-06-14** (was "awaiting go"). Inline-gen
  arc stays CLOSED (v1 @ default decode); the splice pivot's cheap de-risk is now
  executed: 45 standalone NVV ([moan_soft]/[moan_intense]/[groan] × seeds 1-15)
  from Eleanor.wav on the v4 adapter + objective metrics (Resemblyzer spk-cosine,
  MFCC-dist, F0, parselmouth HNR) → `/mnt/smithy/scratch/r16_audition_yield`,
  served on brokkr-audition :8137. 45/45, 0 degenerate; spk-cosine 0.48-0.81.
  Brokkr notified. NEXT: operator ear-bin (usable/impure) → brokkr's yield% /
  survivorship / identity analysis. `gen_yield_probe.py` in `irv-ml1:~/r16-vmoan-harness`.

- **R17 v2 corpus characterization still on the local irv-ml1 branch, push
  HELD** (`r17-v2-characterization`). No-push rule AND ASR-content-exposure
  call on the intimate-audio batch. Awaiting brokkr-collect or explicit push
  approval. (LOCAL-ONLY soundgasm-derived corpus — distribution barred,
  never echo transcripts to the bus.)

- **Mac Pro migration is still the big open project** — `migration-plan.md`
  (repo root): WORKSTATION-ONLY move of nh3-dev's Hat-1 dev env to an M2
  Ultra Mac Pro on the 10.100 subnet. Hat-2 fleet sidecars STAY on the Linux
  VM. **Phase 0** (push pushables + confirm the ~6 local-only repos) is the
  only time-sensitive step. Cutover = `rsync` working trees, NOT re-clone.
  Everything else waits on hardware.

- **sglang-vs-vLLM bench stack staged but parked** (`5f049cb`) — the
  originating SGLang-RadixAttention question resolved (vLLM v1 defaults
  prefix-caching ON, now pinned explicit on granite+qwen). NVFP4 chase
  abandoned. MAX_JOBS-on-shared-prod foot-gun caps any from-source build on
  ana-ml2.

- **pi + GLM 5.1 harness live on nh3-dev** — `glm` launcher runs pi against
  `glm-5.1` (thinking-off) via the LiteLLM gateway; `glm-5.1-reasoning` for
  opt-in thinking.

- **Worldtree config-propagation lane is mature + humming** — pre-merge
  delta-ping → sync-on-merge to demo+personal bind-mounts; pinned stays
  pre-cutover.

- **Disclosed-keys hygiene queue** (rotate at convenience): HF token, the
  wt-personal keys, Gitea runner reg token, MINIFLUX_PASSWORD, ZAI_API_KEY.
  (The shared all-agents LiteLLM key is intentional, not hygiene-debt.)

- **Still open from prior:** clean legacy `news-digest` on ana-docker; the
  `docker push 60s ceiling` mystery uninstrumented.

## Recent decisions

- `[2026-06-16]` **litellm `strip_empty_tools` pre-call hook shipped** (`d1bea13`) — an empty `tools:[]` 500s vLLM ("tools must not be an empty array"); a global `litellm_settings.callbacks` CustomLogger pops it (+ orphaned `tool_choice`) before forwarding, so it covers EVERY vLLM model, not one. `drop_params` only drops unsupported PARAMS, not empty VALUES. Mounts beside config.yaml (litellm resolves callbacks relative to the config dir). Verified live across granite/mistral/stream. (`stacks/litellm/conf/strip_empty_tools.py`)

- `[2026-06-16]` **single-file `gateway-chat.html` playground shipped** (`984ca3d`, `tools/`) — zero-dep browser chat straight to the gateway (`:4000`, CORS open), system-prompt box, streaming SSE, renders `reasoning_content`, NEVER sends `tools`. Built because the LiteLLM admin-UI playground can't test vLLM-backed models (see Tried-and-abandoned). Serve on-request via `python3 -m http.server -d tools`.

- `[2026-06-16]` **claude-bot granted ADMIN on vh/arbo** (operator) — arbo CI/CD gitea ops now route through the service account, not the operator's personal `vh` (the credential-migration directive). Operator added claude-bot as Administrator collaborator; claude-bot self-serves via its `gitea-password` (basic-auth = FULL perms; its stored token is package-write-only, can't change repo settings). (auto-memory `reference_claude_bot_gitea_creds`)

- `[2026-06-16]` **ratatoskr Tier-3 MEMORY plane wired + live-smoke driven.** The missing server-side step was the per-PORT `BIFROST_CLIENT_ALLOWED_HOSTS` allowlist (`:8391`; consumer key reused, plane-agnostic — my earlier "reachability is the whole wiring" was incomplete). Persist + dispatch proven end-to-end against personal v0.35.3; cross-session recall-INJECTION root-caused to the `score_breakdown` shape seam and handed to worldtree-dev (#295 recall leg open). Driving a real memory-call needs a consumer agent defined WITH a `memory` block (`ctx.memory_config≠None`), not `agent_self_enabled`. (tracking: c99aa49 + auto-memory `reference_bifrost_plane_wiring`) — **[2026-06-16] FLIPPED:** the `score_breakdown`-shape root-cause was disproven; real cause = cold-recall `scope_filter` adds an `agent_self` axis → ratatoskr's conjunctive INV-005 match drops the `{end_user}`-only chunks → 0 hits. Now Worldtree #297. (auto-memory `project_worldtree_295_recall_rootcause`)

- `[2026-06-16]` **infra-ops = the DURABLE admin path on corviduo-dev** (operator granted standing admin; extends the 2026-06-15 bootstrap) — added a `Host corviduo-dev → User infra-ops` `~/.ssh/config` alias + repointed `servers/corviduo-dev/ssh-target` to `infra-ops@`; `ssh corviduo-dev` / `elway corviduo-dev` now land as infra-ops w/ NOPASSWD (vh = explicit-only human path). elway resolves via ssh-config, NOT the ssh-target file. (tracking: c99aa49 + auto-memory `reference_corviduo_dev_emergency_ops`)

- `[2026-06-16]` **demo + personal `character` model flipped qwen3.6-35-a3b → mistral-small-4** (operator) — in `model_roles.yaml` the FIRST `catalog_id` in a role's `binds:` is the default; reordered mistral-first (qwen retained in the switch-allowlist, non-destructive), applied via the PIN-SAFE api recreate (derive `WORLDTREE_IMAGE` from the matrix sibling — never a bare `compose up`, which falls to the crash-blocking stale `:latest`), verified a fresh character agent resolves model=mistral-small-4. (auto-memory `reference_corviduo_dev_emergency_ops`)

- `[2026-06-16]` **NVFP4 ModelOpt MoE now LOADS on vLLM 0.23.0 — SUPERSEDES the
  2026-06-14 "blocked" finding (#44081 fixed).** Confirmed the cached
  `nvidia/Qwen3.6-35B-A3B-NVFP4` loads clean + generates on 0.23.0; swapped prod
  qwen36 FP8→NVFP4 (~13 GB reclaimed on GPU 1, fp16 KV). Validated by brokkr (speed:
  faster single-stream, ≥parity batched) + comfy-dev (vision anatomy-judge A/B: PASS).
  One W4A4 tell: minor multi-step chained-reasoning slip — doesn't bite the vision
  judge. Mistral's NVFP4 is a SEPARATE compressed-tensors path (day-0), distinct from
  the ModelOpt one. (auto-memory `reference_nvfp4_moe_loads_on_vllm_023`)

- `[2026-06-16]` **Mistral Small 4 = the GPU-0 tenant** (operator) — fills the slot
  the deferred creative-writing pick held, and supersedes llama-swap (now down).
  NVFP4 119B MoE, pinned **vLLM v0.22.0** for working vision (0.23.0 breaks it),
  256K context. Operator's creative-writing model; **abliteration planned** → then it
  becomes the uncensored-creative model the persona substrate needs. Reasoning via
  `reasoning_effort` (BINARY none|high — no medium). (tracking: `c77a9aa`/`9a49963`)

- `[2026-06-16]` **Selene judge restored on vLLM FP8, NOT NVFP4** — W4A4 is too
  aggressive for a precision judge validated at Q6_K; FP8 ≥ Q6_K fidelity, and Selene
  is text-only Llama-3.1 so dynamic `--quantization fp8` has no vision tower to noise.
  `selene-1-mini-8b` on GPU 1 :8011. Worldtree's Domari role repointed to it. (`c985ede`)

- `[2026-06-16]` **GPU-1 buffer put to work: selene 0.13→0.17 + qwen36 0.32→0.34**
  — the two KV-bound generative models that gained Worldtree consumers; granite + the
  pooling models under-use their util, so growing THEM is wasted. (`e124a2f`)

- `[2026-06-16]` **claude-bot Gitea service account stood up; corviduo-ORG conversion
  TABLED** (operator). claude-bot is the credential-migration foundation. The proper
  fix for scoped agent package-write is converting vh (a USER) → a `corviduo` ORG —
  blast radius assessed (org creation = free; arbo-first migration = cheap, no registry
  images; full fleet = incremental, ~8 active vh images / 50 repos) — but tabled for
  later. Registry-push deferred until then. (auto-memory `project_migrate_infra_access_to_claude_credentials`)

- `[2026-06-16]` **arbo engine builds → comfy-dev OWNERSHIP** (operator approved) +
  Gitea Actions runner stood up on irv-ml1 (host-executor as lkraven). infra-ops keeps
  box-level escalation (runner, package:write PAT pending corviduo-org, webhook). See
  the runner entry in Current state. (tracking: this snapshot + arbo thread `01KV70AB2B4B…`)

- `[2026-06-16]` **Worldtree persona substrate migrated qwen→Mistral via capability-
  named profiles** (v0.35.8→v0.35.10; demo + personal). PRE-SYNC-FIRST is now the
  standing pattern for any providers.yaml/profile change: sync config to the bind-mount
  BEFORE the deploy image-swap restart, so new agents + new config land together (the
  v0.35.9 merge-first attempt race'd → 14 crash-loops). (tracking: worldtree thread `01KV6X0M…`)

- `[2026-06-15]` **qwen36 thinking-split at the gateway** (operator) — base
  `qwen3.6-35b-a3b` = thinking-OFF default (the vision judge needs it), `…-a3b-thinking`
  = opt-in reasoning. One hybrid checkpoint + `--reasoning-parser qwen3`; mirrors the
  glm-5.1 split. The judge's enable_thinking=false is what makes the NVFP4 swap
  transparent. (`6de0844`)

- `[2026-06-15]` **arbo engine rebuilt v0.11.6 → v0.11.22 from main; catalog re-applied (v0.11.23) — curated `/workflows` footer live.** src/ schema changes (Model/Lora.summary, Workflow.blurb/keywords + the /workflows endpoint) need a MANUAL image rebuild (the webhook warns-only on `src/`/`Dockerfile`). Built from `/worktank/arbo/repo`, bumped `ARBO_IMAGE` in the host .env, recreated + verified healthy (auto-rollback armed, unneeded). comfy-dev's v0.11.20 catalog push had first crash-looped prod (old baked schema `extra=forbid` vs new git-mounted catalog fields → ValidationError); they reverted the catalog, so rebuild-from-main loaded clean, then they re-applied (v0.11.23). comfyui is a SEPARATE compose project — the rebuild didn't touch it.

- `[2026-06-15]` **althing-core refreshed to v0.14.1 box-wide on nh3-dev** (forseti + heid both routed it; central-sequenced per DevOps-delegation since `uv tool install --force` swaps the shared binary all handles use). Fixes the monitor non-truncating-lock bug so `stop-monitor`/`--stop` reliably recovers orphans. The editable install (`uv tool install --force --editable .`) covers every nh3-dev handle; `--version` trued up 0.14.0→0.14.1; verified `--stop` works against a real orphan.

- `[2026-06-15]` **ratatoskr Tier-3 affect live smoke GREEN** (consumer-confirmed). Brokered/minted consumer_id="ratatoskr" Heimdall HS256 key on personal Worldtree (the key IS BOTH the create-session Bearer AND ratatoskr's JWT verifier secret — plain utf-8 string, NOT base64/raw); injected ENV-ONLY to `~/.config/ratatoskr/provider.env` (mode 600) on nh3-dev; added `10.100.10.50:8390` to personal's `BIFROST_CLIENT_ALLOWED_HOSTS`; drove define→session (handshake 200)→turn→affect.emit (200, snapshot persisted). Worldtree signs Bifrost with the caller's raw Bearer verbatim. Drive commands + claims contract in althing thread `01KV4ZYSE3…`.

- `[2026-06-15]` **Dense Qwen3-VL-32B-NVFP4 A/B'd as arbo's anatomy hero-judge vs the qwen3.6-35b-a3b MoE → dense LOST, pilot torn down, GPU 0 reclaimed.** Dense NVFP4 loads clean on vLLM where the MoE NVFP4 doesn't (no expert-scale keys); ~44 tok/s on Blackwell GPU 0; vision tower stays high-precision. But comfy-dev's ground-truth A/B: MoE wins (faster + dual-use as general LLM + slight fine-detail edge), and the subtle-proportion tier is below ALL VLs. **Don't re-propose a dense-VL judge swap.** (auto-memory `project-qwen3vl-dense-pilot`)

- `[2026-06-15]` **MastMed prototype containerized + exposed publicly via cloudflared.** Self-contained static SPA (bun→caddy multi-stage build), basic-auth gated (operator's gating call), ana-docker:8089, cloudflared quick tunnel. See in-flight for the ephemeral-URL caveat + named-tunnel upgrade path.

- `[2026-06-15]` **infra-ops bootstrapped onto corviduo-dev** (operator-directed) — infra-ops now has SSH + NOPASSWD sudo there, OVERRIDING the fleet bootstrap script's deliberate partner-box exclusion (explicit host arg). corviduo config edits no longer need `vh` + a sudo password. (auto-memory `reference_corviduo_dev_emergency_ops`)

- `[2026-06-15]` **R18 clip+caption placement = ana-ml2** (operator: "ana-docker… ana-ml2 unless irv-ml1 has room"; irv-ml1 lacked free VRAM for the 30B-A3B captioner). Models staged + stub deployed; see in-flight + the hard real-voice gate.

- `[2026-06-14]` **LiteLLM infra-ops admin key provisioned** (operator) — resolves the LiteLLM half of the credential-migration directive; use it for ALL gateway ops (NOT `sk-corvid`). Value at `~/.config/litellm/infra-ops-key` (mode 600); gateway reachable directly from nh3-dev at 10.250.50.70:4000. (auto-memory `reference_litellm_infra_ops_key`)

- `[2026-06-14]` **ana-ml2 GPU-1 vision upgraded: Qwen3.5-9B → Qwen3.6-35B-A3B (official FP8), served under its TRUE name only.** `qwen36-vl` replaces `qwen35-vl` on :8007 (`a0fed13`). The stale `qwen3.5-9b-fp8` name is KILLED at vLLM AND the litellm gateway (404/400) — a model is NEVER aliased under a prior model's name (silent substitution = downstream footgun; operator directive). Consumer comfy-dev/arbo migrated; arbo vkeys → all-proxy-models; shared `all-agents-local` key repointed qwen3.5-9b-fp8 → qwen3.6-35b-a3b. GPU-1 rebalanced for the ~34 GB FP8 weights (granite 0.35→0.24/64K; embed/rerank 0.05→0.03, reclaimed ~4 GB util-waste). Validated: vision correct, 20-concurrent = no OOM. (auto-memory `feedback_no_false_model_aliases`)

- `[2026-06-14]` **NVFP4 was the lighter fit (~21 GB) but is BLOCKED on vLLM — FP8 is the working vision path.** `nvidia/Qwen3.6-35B-A3B-NVFP4` won't load: the ModelOpt-NVFP4-MoE loader errors on expert/lm_head scale keys across 0.19.1 (`w2_input_scale`) AND 0.22.0 (`lm_head.input_scale`, vllm #44081) — a pattern across modelopt NVFP4 MoEs. Revisit NVFP4 (frees ~13 GB on GPU 1) once fixed; the 21 GB checkpoint stays cached on ana-ml2. **(SUPERSEDED 2026-06-16 — it loads on vLLM 0.23.0; qwen36 swapped to NVFP4. See the top of this section.)**

- `[2026-06-14]` **llama-swap qwen3.5-9b GPU-0 pin DROPPED; GPU 0 reserved for a creative-writing model (pick DEFERRED by operator).** Deep-research (this session) on big-fast-uncensored creative for a 96 GB Blackwell: **GLM-Steam-106B-A12B** (already in the llama-swap config — balanced default) vs **TheDrummer/Behemoth-X-123B-v2** (prose-tier, tops UGI writing+willingness) vs XORTRON-123B (max willingness, weak prose); GGUF-on-llama-swap is the serving path. Tracking: this session + llama-swap config (GLM-Steam present, `untracked by operator choice`).

- `[2026-06-14]` **R16 splice-pivot yield probe executed** (infra-ops ran the irv-ml1 inference for brokkr; brokkr owns design + analysis). See Current state. Tracking: althing thread `01KV010WGS…`, `gen_yield_probe.py` in `irv-ml1:~/r16-vmoan-harness`.

- `[2026-06-14]` **STANDING DIRECTIVE: migrate ALL infra access to Claude-specific credentials.** Agents currently reuse the operator's PERSONAL creds for infra ops — vh Gitea **admin** via `tea` (used this session to mint a `read:package` token for ratatoskr, id 13, **under vh**), `sk-corvid` litellm master for vkey admin. Stand up service accounts (a `claude-bot` Gitea user + scoped tokens, a distinct litellm admin key); re-mint consumer creds under them; flag personal-cred fallbacks until done. (auto-memory `project_migrate_infra_access_to_claude_credentials`)

- `[2026-06-14]` **R16 vmoan inline-generation arc CLOSED — v1 at default decode (rep1.2/temp0.8) is the final Chatterbox-tag inline artifact.** Operator's ear rejected every alternative: v2/v3 windowing (omission vs coherence-loss), v4 multi-tag (cohesion held but lost to capacity-competition), emergent inline-token modulation (degenerates, not modulates), and the gen-time decode-polish sweep (soft tamers cut the NVV itself — same omission family as v2; p0 baseline beat p1). All adapters v1–v4 + `tokenizer.json.v3bak` preserved on `irv-ml1:~/r16-vmoan-harness`. Likely-next direction (deferred, NOT formalized): generate→bin→splice + one-shot-clone NVV pipeline routing around the inline-coherence wall. Tracking: brokkr R16 journal + althing thread `01KV010WGSSMPWRNCPAGSPK15Y`.

- `[2026-06-14]` **Arbo deploy pipeline fixed, hardened, and version-controlled.** Prod rebuilt v0.11.1 → **v0.11.6** backend; the webhook machinery (`arbo-deploy.sh` + `arbo-webhook.py`, :9009 HMAC listener) is now repo-tracked at `stacks/arbo/` (was host-only = recoverability foot-gun). Deploy reaches gitea via the INTERNAL route (`10.250.50.70:222`) and restarts the engine ONLY on `catalog/` changes (graphs/frontend per-request; warn on `src/`|`Dockerfile` only — pyproject/uv.lock churn every commit). Operator kept arbo stack ownership in **eshpfi** (not migrated to comfy-dev's repo). Secret + `.env` stay host-only. Tracking: `6d66bc2`, `6e58e57`, `stacks/arbo/README` Q5.

_73 older entries archived to archival-memory.md._

## Tried and abandoned

- `[2026-06-16]` **litellm 500 `Router.acompletion()/aembedding() missing 'messages'/'input'` = a request missing `Content-Type: application/json`, NOT a gateway outage.** curl `-d` defaults to form-encoding → litellm can't parse the JSON body → `data` reaches the router without `messages`/`input` → 500 (should be a 400; litellm #16993). My own diagnostic calls dropped the header → I misread it as a gateway outage and needlessly bounced the gateway ~4× chasing a phantom (image/version/config were fine throughout; a malformed UI-added "Mistral Story Eval" model in the DB was a red herring I deleted). ALWAYS send `-H "Content-Type: application/json"` testing litellm; reproduce with a header'd call before declaring a litellm incident.

- `[2026-06-16]` **LiteLLM admin-UI playground can't test vLLM-backed models** — it auto-sends empty `tools:[]`, vLLM 400s (litellm #6228); the gateway `strip_empty_tools` hook is a PROXY hook and structurally can't reach the UI's in-process `litellm.completion()` call. Off-ramp = `tools/gateway-chat.html`. (Langfuse playground also out: its SSRF guard blocks internal-IP LLM connections, wontfix Langfuse #13097.) (auto-memory `reference_litellm_ui_playground_vllm_deadend`)

- `[2026-06-16]` **A gitea Actions run that "looks like it never fired" (no image, untouched build dir) usually DID fire then skipped or failed-fast.** Bit arbo TWICE: run #3 looked-never-fired but SKIPPED (change-detection `engine=0`); run #4 looked-never-fired but FAILED in ~7s (early-step). Check the run LIST + status via claude-bot (`GET /repos/{o}/{r}/actions/tasks`) BEFORE a runner-connection/dispatch theory. CORRECTION (2026-06-16, proven): action logs ARE readable via API at the per-JOB endpoint — `GET /api/v1/repos/{o}/{r}/actions/runs/{run}/jobs` → take a job `.id` → `GET /api/v1/repos/{o}/{r}/actions/jobs/{job_id}/logs` returns the full plain-text log (claude-bot basic-auth, internal route `10.250.50.70:3000`). Only the RUN-level `/runs/{id}/logs` 404s; the per-job one works, so no UI needed. Gotcha when reading: gitea marks **every** step `conclusion:failure` once any step fails — the per-step conclusions mislead; trust the log body + step *timestamps* (the step with real duration is the culprit) not the conclusions. (Used 2026-06-16 to diagnose arbo run #5/task 1175 = test-gate `uv run pytest` → `Failed to spawn: pytest` = pytest absent from the fresh-checkout venv sync, NOT the SSH-clone comfy-dev suspected.)

- `[2026-06-16]` **vLLM 0.23.0 breaks Mistral MULTIMODAL at startup** (`MistralCommonImageProcessor
  has no attribute fetch_images`, #44911 regression ~0.22.1+; same class hit Mistral-3.1/
  Devstral). The dummy-image profiling crashes. Workarounds: `--limit-mm-per-prompt
  '{"image":0,"video":0}'` loads text-only; for VISION, pin **v0.22.0** (last pre-regression).
  Tradeoff: v0.22.0 *buffers* reasoning (doesn't stream reasoning_content cleanly); a newer
  build streams it but breaks vision — vision won (Mistral is the abliteration-baseline).

- `[2026-06-16]` **Mistral `-reasoning` TTFT variance = Triton kernel JIT-compile-during-
  inference, NOT reasoning time.** `jit_monitor` logs the spike (`_compute_slot_mapping_kernel`,
  `apply_token_bitmask_inplace_kernel` ← the tool-call/structured-output path). vLLM's startup
  warmup covers only SOME shapes; a first-hit uncovered shape JITs a kernel (seconds), then
  CACHES it — warm until container recreate (the Triton cache is IN-container, no volume, so
  recreate = cold again). Secondary: GPU idle-down to P8/180 MHz between requests. Also:
  Mistral `reasoning_effort` is BINARY (none|high only — medium/low → HTTP 400).

- `[2026-06-16]` **Worldtree old-baseline instances need the FULL config set synced, not the
  changeset delta.** Personal (a PRE-mistral baseline) crash-looped (41 restarts) on
  `DanglingBindingError: role 'echo' binds missing catalog entry 'qwen3.6-35-a3b-heretic'` —
  its `model_roles.yaml` (2026-06-10, never migrated) still bound the retired heretic that the
  synced v0.35.10 providers removed. The "model_roles unchanged — skip" was correct only for
  demo (already at the v0.35.9 baseline). Old-baseline → sync providers + model_roles + defaults.

- `[2026-06-16]` **vh is a Gitea USER, not an org → no scoped package-write for a service
  account.** Gitea user-namespace packages are owner/admin-only, so claude-bot can't push to
  `vh/arbo` without being admin. The fix is converting vh → a `corviduo` ORG (tabled). Also:
  vh's `tea` token lacks `write:admin` — create gitea users via the server-side `gitea admin
  user create` CLI (`docker exec -u git gitea …`), not the admin API.

- `[2026-06-16]` **`docker compose up -d` on a multi-service vLLM stack recreates ALL its
  services** (shared `.env` → all configs re-hash) → concurrent memory-profiling race
  ("No available memory for the cache blocks"). Target one: `docker compose up -d <service>`.
  (auto-memory `feedback_compose_up_recreates_whole_stack`)

- `[2026-06-15]` **A comfyui restart-loop with `exit 0` + `OOMKilled=false` is NOT OOM — suspect a native SEGFAULT.** `docker inspect` shows only the LAST exit code; a GPU OOM is a catchable `RuntimeError: CUDA out of memory`, NEVER `Fatal Python error: Segmentation fault`. Diagnostic: `RestartCount>0` = same container restarted in-place → the docker json-log ACCUMULATES across all cycles, so grep the full `docker logs` for the crash signature + the lines right before each restart banner. comfyui's was at VAE decode (sampling completes, then segfault loading `AutoencodingEngine`) — gen-triggered but native, not VRAM. Suspects logged: torch/cu130 mismatch, an injected `aimdo` cuda-funchooks lib, the VAE.

- `[2026-06-15]` **NEVER blanket-kill `althing-light-monitor` by PID** — they're per-handle and MANY agents run their own on the shared nh3-dev box (heid / ratatoskr-dev / worldtree-dev / bifrost-dev each have one). A `kill $(pgrep -f light-monitor)` downs four peers' monitors (near-miss caught by checking `/proc/<pid>/environ` `ALTHING_HANDLE` + cwd first). Recover your OWN via `althing-cli stop-monitor` (v0.14.1+); a raw `&`-backgrounded monitor is untracked (no harness fire-notification) — always arm via `run_in_background`. (extends the pkill-self-match lesson below)

- `[2026-06-15]` **Manual `docker compose up` on corviduo Worldtree DOWNGRADES to a stale local `:latest` → crash-loop (config/image mismatch).** Compose `image:` is `…/worldtree:latest`, but local `:latest` lagged ~2wk (`21c90d5eeb2d`); CI/CD deploys NEWER SHA images, and a real pull 401s (daemon has no gitea-registry creds — that's CI/CD's). `--pull never` then used the stale latest → personal crash-looped on `ConfigurationError: echo.default_model missing` → **~2-3 min personal outage**. RECOVER: retag `:latest` to the image a HEALTHY SIBLING runs (`docker inspect worldtree-worldtree-api-1 --format '{{.Image}}'` → `docker tag …`), then `up -d --pull never`; verify `Application startup complete`. Don't manually recreate Worldtree on corviduo without pinning the correct image — image+config are CI/CD-coupled. (auto-memory `reference_corviduo_dev_emergency_ops`)

- `[2026-06-15]` **`.claude.json` (Claude Code config) truncates on ENOSPC → invalid JSON → CC resets to defaults on next start.** nh3-dev hit 100% disk (a peer's R19 dataset pull staged ~147G to local /home instead of the 7TB NAS); the in-flight `.claude.json` write was cut mid-key. Repair = strip the dangling token + close the open braces, validate `json.load`. CC keeps its OWN rolling backups at `~/.claude/backups/` + writes a `.claude.json.corrupted.*` snapshot when it self-detects. After ANY ENOSPC on a box running CC/agents: integrity-check `.claude.json` (json.load) AND the althing bus DB (`PRAGMA integrity_check`).

- `[2026-06-15]` **`pkill -f althing-light-monitor` SELF-MATCHES the killing shell** (the pattern is in the command's own argv) → kills itself mid-run (exit 144/truncated output). Stop the light-monitor via `althing-cli stop-monitor` or a captured PID — never `pkill -f <string-present-in-this-command>`. The singleton lock can also RACE to 2 live monitors during re-arm churn; keep exactly one tracked (run_in_background) monitor, and a raw `&` monitor is untracked (no harness fire-notification — don't use it).

- `[2026-06-14]` **vLLM ModelOpt-NVFP4-MoE loader is broken for current multimodal MoEs.** `nvidia/Qwen3.6-35B-A3B-NVFP4` fails weight-load: `KeyError: layers.0.mlp.experts.w2_input_scale` on 0.19.1, `lm_head.input_scale not registered` on 0.22.0 (vllm #44081); same class hits Gemma-4 MoE / Qwen3-30B-A3B NVFP4. The arch + quant ARE recognized (gets past arch resolution + vision-processor load) — it's the per-expert/lm_head scale-key mapping. Don't chase nightlies; use official FP8 until fixed.

- `[2026-06-14]` **vLLM sampler-warmup OOMs on a shared GPU even when weights fit** — it warms the sampler with `max_num_seqs` (default **1024**) dummy requests, and a big vocab (Qwen3.6 = 248K) makes that a huge transient logits tensor. A vision endpoint doesn't need 1024-way concurrency: set `--max-num-seqs 32`. Separately, post-load `ValueError: No available memory for the cache blocks` means util is too thin (weights+activation+graph ate it) — for 34 GB FP8 weights, util ≥ ~0.45 to leave KV room.

- `[2026-06-14]` **Recreating multiple vLLM services concurrently races the memory-profiling assertion** — `AssertionError: Error in memory profiling. Initial free memory X / current Y … other processes … release GPU memory while vLLM is profiling`. Recreate co-tenant vLLM services ONE AT A TIME (force-recreate one, wait healthy, next).

- `[2026-06-14]` **embed/rerank (0.6B) at util 0.05 reserve ~5.5 GB each — mostly util-reservation WASTE, not need.** A 0.6B model needs ~1.2 GB weights + ~2.5 GB CUDA/torch context; util 0.03 (~3.6 GB) fits with room, reclaiming ~4 GB (vLLM reserves the util fraction regardless of actual KV; embedding models barely use KV). Real-need floor ~3 GB — don't go to 0.02.

- `[2026-06-14]` **Fleet/colo hosts must reach gitea over the INTERNAL route, NOT the public IP.** `gitea.phasefinal.com` = public `38.120.12.44` (ana-srv1); gitea is a container on ana-docker, git-SSH `10.250.50.70:222` + HTTP `:3000`. A fleet host egressing to public `:22` gets fail2ban-banned after any retrying git loop → silently wedges webhook auto-deploys (`git fetch` times out under `set -euo pipefail`, aborts before reset). Bit irv-ml1's arbo deploy. `:22` on `10.250.50.70` is ana-docker's HOST sshd (deploy key → Permission denied), NOT gitea. Documented `docs/orientation.md` (`6e58e57`).

- `[2026-06-14]` **Chatterbox-Turbo decode-knob foot-guns** (R16 v1-polish + emergent probes): the turbo length cap is `max_gen_len` (default 1000) on `t3.inference_turbo`, NOT `max_new_tokens` — and `tts_turbo.generate` does NOT forward it (wrap inference_turbo to cap). `rep_pen 2.0 / temp 0.5` BACKFIRES (degenerate 24 s run-on). Soft decode tamers cut the NVV ITSELF, not just the run-on tail (operator: "p1 trims the moaning too") — gen-time polish can't beat v1's defaults. Inline base-NVV tokens DEGENERATE (moan-cascade + gibberish), they don't modulate the surrounding words.

_65 older entries archived to archival-memory.md._