# Persistent memory — eshpfi-management

_Last updated: 2026-06-20_

## Repo purpose

Reference workspace for PFI infrastructure: server inventory, canonical
Docker Compose stacks, ops playbooks, and conventions. Authoritative
copies of compose files live on the servers under
`/opt/docker/compose/<stack>/`; this repo mirrors them for version
control, editing, planning, and CI-driven deploys. **It was originally
spun up to handle the fleet backups** — keep that lens when triaging
backup/storage issues.

## Tools and conventions

Sister repos (separate gitea repos, deployed by playbooks here):

| Repo | Role | CI status |
|---|---|---|
| `vh/task-board` | MCP + web dashboard for assistant task state (port 7878) | push-to-main → CI deploys (2026-04-29) |
| `vh/vor` | Inquisitor UI sidecar (port 7879) | push-to-main → CI deploys (2026-04-29) |
| `vh/nevermore` | Twice-daily LLM-curated briefing (port 8181, replaces news-digest) | push-to-main → CI deploys (2026-04-30) |
| `vh/asset-engine` | Internal control plane over inference services (port 8200, LAN-direct) | push-to-main → CI deploys (2026-05-12) |
| `vh/althing` | Inter-agent message bus (chamber UI port 7881, forseti + agent-runner daemons, valkey IPC) | push-to-main → CI deploys (2026-05-14) |
| `vh/mead-hall` | Bifrost tool-provider sidecar (port 5173 on dev VM 10.100.10.50) | push-to-main → CI deploys (2026-05-16) |
| `vh/skaldsong` | Wizard + reader surface (port 8300, ana-docker, registry-pull pattern) | push-to-main → CI deploys (2026-05-19) |
| `vh/Worldtree` | Conversation API (corviduo-dev demo :8080 / personal :8081 / pinned :8082) — Heimdall auth, Bifrost integration. **gitea-runner builds on ana-docker** (the image-bloat source); claude-bot now ADMIN collaborator (2026-06-20) | push-to-main → CI build-and-deploy (runner on ana-docker) |
| `vh/yt-voice-clipper` | YouTube → diarized voice-clip dataset builder + audition console (irv-ml1 :8000) | push-to-main → **gitea-webhook auto-deploy** to irv-ml1 (2026-06-03) — see `docs/runbooks/ytvc-autodeploy.md` |
| `vh/arbo` | Catalog-driven ComfyUI engine (irv-ml1 :8201, comfy-dev owns engine/catalog/image) | push-to-main → gitea Actions CI (deploy-engine.sh, build-local, health-gated) now LIVE; catalog via :9009 webhook |

(`vh/volva` + Heid were re-architected from systemd daemons to Claude Code
session orchestrators 2026-06-08; their nh3-dev `.service` units were removed —
no longer deployed sidecars here. See Recent decisions.)

- **Two-layer backups** — Backrest orchestrates restic for file+DB (5
  fleet repos, daily 01:00 PDT); PBS-ANA primary + PBS-NH3 DR mirror for
  VM images. ana-nas is the SPOF for postgres + PBS-ANA datastore +
  cross-site restic targets — see `docs/runbooks/disaster-recovery.md`
  for the blast-radius matrix. **⚠️ The restic file+DB layer routes
  through TWO rest-servers** (`rest-server-ana` @ ana-docker:8000 →
  ana-docker/ana-ml2/esh-docker-vm/vm-esh-nas; `rest-server-nh3` @
  nh3-nas:8000 → irv-ml1/nh3-docker). Both depend on their NAS's NFS
  export of `/mnt/backup`. **As of 2026-06-20 `rest-server-ana` is DOWN
  on a failed ana-nas NFS mount — see Current state.**

- **`pull-hf-repo.yaml`** is the canonical "get a HuggingFace
  model/dataset onto ana-ml2's shared cache at
  `/tank/aimodels/huggingface/`" playbook. Supports `--var repo_type=model|dataset|space`. Replaces ad-hoc `huggingface_hub.snapshot_download` patterns.

- **Worldtree admin auth — per-instance.** Each Worldtree deployment
  (demo :8080, personal :8081, pinned :8082) has its own Heimdall
  registry and its own bootstrap admin key. Infra-ops's stored
  long-lived admin key (`key_id 61419c92`) at
  `ana-docker:/opt/docker/conf/.secrets/worldtree-infra-ops-admin`
  auths against **demo only**. Personal-instance admin (the
  `~/.config/worldtree/personal-admin-token`, mode 600) POSTs
  `/admin/keys` (mints per-project keys; takes `user_id`+`label`, **no
  scope param** — scopes are tier-derived, so admin-tier scopes like
  `admin.events.read` must be granted WT-side by worldtree-dev).

- **Per-project user keys against personal Worldtree** (issued
  2026-05-19): `skaldsong:79744637` (nh3-dev iteration),
  `skaldsong:7c1dbbbe` (ana-docker prod), `althing:50d85460`,
  `mead-hall:a360822d`. Same `user_id=skaldsong` across both
  skaldsong keys → shared Heimdall agent slot; different `key_id`
  → independently rotatable. Pattern: mint via `/admin/keys`, drop
  value to `/tmp/wt-personal-<name>.key` mode 600, dev collects +
  shreds (DO NOT cat to chat transcript).

- **Skaldsong CD pattern (registry-pull).** Differs from althing /
  asset-engine which build-on-host. vh/skaldsong's CI builds and
  pushes `gitea.phasefinal.com/vh/skaldsong:<sha>` + `:latest`;
  `playbooks/deploy-skaldsong.yaml` on ana-docker pulls + recreates.
  SHA-pin only (no `:latest` health-gated advance yet). Prereq: host
  needs `docker login gitea.phasefinal.com` once (read:package PAT) —
  not currently in the workflow.

- **gitea internal route for fleet hosts.** gitea is a container on
  **ana-docker** — git-SSH `10.250.50.70:222`, HTTP `:3000`. Fleet/colo
  hosts must use this internal route, NOT public `gitea.phasefinal.com`
  (`38.120.12.44`, ana-srv1) — the public path fail2bans the host egress
  IP and wedges webhook deploys. `:22` on `10.250.50.70` is ana-docker's
  HOST sshd, not gitea. Full gotcha in `docs/orientation.md` → Git/gitea.

- **docker-as-root pattern** (for ops with no admin API, or to edit
  deploy-owned/root-owned files without sudo): `docker run --rm -v
  <target-dir>:/wt [-v /var/run/docker.sock:/var/run/docker.sock]
  docker:cli sh -c "..."` (or `alpine` for plain file ops). docker-group
  membership is effectively root via bind-mount; treat as sudo-equivalent.
  **Foot-gun: relative paths in compose.yaml resolve against the sandbox
  CWD but the Docker daemon interprets them against the HOST fs — always
  pass `-e VAR=/abs/path` for any relative-default config dir.**

- **`scripts/elway` sudo handling** — elway prompts for the sudo password
  ONCE via `getpass` before the first `sudo: true` step → can't run
  unattended from a non-TTY tool if any step needs sudo. Sudo-free
  playbooks run fully non-interactive over key SSH.

- **Per-host SSH identity matters for sudo.** infra-ops has NOPASSWD sudo
  on most PFI Linux boxes (corviduo-dev included since 2026-06-15). On
  ana-docker there are TWO identities: the **default `ssh ana-docker` =
  `lkraven`** (docker-group, NO passwordless sudo — docker works, root-file
  edits don't); **but `ssh infra-ops@ana-docker` HAS NOPASSWD root** (verified
  2026-06-20). **→ For any sudo op on ana-docker (mount, root-owned files,
  service control), use `ssh infra-ops@ana-docker`, NOT the default session.**
  lkraven-owned files (litellm config, most stack compose/conf) still take
  plain `cp`/edit under either identity.

## Current state / in-flight

_As of 2026-06-20:_

- **Worldtree capability-gateway migration — LANDED + DEPLOY TRIGGERED.**
  The role→capability indirection (worldtree-dev's #257/ADR-0012 direction;
  operator: NO `wt-` prefix, reuse the existing capability-alias convention)
  is live on the LiteLLM gateway: `gen`/`gen-reasoning` (→qwen3.5-122-a10b),
  `classifier`/`summarizer` (→granite), `chat-judge` (→selene-1-mini-8b),
  `reranker` (→qwen3-reranker), `scalar-judge` (passthrough → Skywork-Reward-V2
  @ ana-ml2:8003 — LiteLLM has no reward MODE), `gen-frontier`/`-reasoning`
  (→GLM 5.2, PAID). **`qwen3-embedding` kept model-named ON PURPOSE** — the
  swap-transparency rule: embedding vectors aren't swap-transparent, so a
  generic `embedding` alias is a footgun. The `*` wildcard → dead llama-swap
  was REMOVED (unknown models now 404 cleanly). WT's full re-point (granite
  SPLIT into granite-classifier/granite-summarizer + role rebind + edges→gen +
  selene→chat-judge) landed as worldtree-dev's commit **3fbf338 / v0.37.5**;
  I staged its `providers.yaml`+`model_roles.yaml` byte-identical into
  demo+personal bind-mounts and triggered build-and-deploy via claude-bot.
  Pinned rides CI. (d372103, d0eb09c, 7819f96)

- **Worldtree demo → v0.37.7 GREEN (2026-06-20) — two-half fix.** worldtree-dev's
  v0.37.5 config (3fbf338) carried a dangling `granite-4.1-8b` catalog ref →
  demo crash-blocked at startup validation. Fix needed BOTH halves: (a) gitea
  was silently not creating Actions runs so no deploy could fire (the earlier
  disk-full wedged its notifier — `docker restart gitea` cleared it; see
  Tried-and-abandoned); (b) the v0.37.7 IMAGE alone wouldn't fix demo because
  demo+personal read BIND-MOUNTED config — I re-staged `config/providers.yaml`
  + `config/model_roles.yaml` from gitea at **439bebf** byte-identical into BOTH
  `/opt/worldtree{,-personal}/config` (sha256-verified across source + both
  instances; `granite-4.1-8b`→`summarizer`, `granite-classifier/summarizer`→
  `classifier/summarizer`; old configs saved `.bak-pre-439bebf`; NO container
  recreate — avoided the stale-`:latest` footgun). worldtree-dev self-served the
  CI re-run via the claude-bot Actions token → **run 1290 GREEN, demo on
  v0.37.7**, capability routing live. claude-bot token migration DONE for
  worldtree-dev (off vh's personal PAT). **RATATOSKR → READONLY (2026-06-20, final):**
  operator first said admin (worldtree-dev minted interim `ratatoskr-admin` 90db1fbd),
  then flipped to least-privilege → that key REVOKED (DELETE 200). I re-staged 439bebf's
  `config/policies.yaml` to `/opt/worldtree-personal/config` (byte-identical, sha256
  f3ca3e6…, `.bak-pre-439bebf`) and reloaded Heimdall via `docker restart
  worldtree-personal-worldtree-api-1` (same d2e9f05 SHA — the SAFE reload, no recreate/
  no `:latest` flip); personal came back healthy + the `readonly-admin` tier (7 read
  scopes incl. admin.events.read) is now in the loaded policy. **FOLLOW-UP cb2a79a
  (v0.37.9):** the 439bebf tier-only stage was NOT sufficient — WT's PDP is RULE-based,
  so a tier with scopes but NO allow RULES is default-denied; cb2a79a adds the 4
  readonly-admin allow rules. Re-staged cb2a79a `config/policies.yaml` to BOTH demo +
  personal (sha256 d853e51…, `.bak-pre-cb2a79a`) + restarted BOTH API containers
  (`worldtree-worldtree-api-1` + `…-personal-…`); both healthy + serving. (Still on
  d2e9f05 image — CI cb2a79a deploy pending, but the bind-mount policy is the
  load-bearing fix.) worldtree-dev mints the ratatoskr readonly key via their admin key.
  **Lessons: (1) reload bind-mounted WT config via `docker restart <container>`, NEVER
  `compose up` (the `:latest` footgun); (2) a WT tier needs allow RULES, not just
  scopes — the PDP is rule-based, scopes alone default-deny.**

- **🟠 BACKUP DIAGNOSIS (2026-06-20, full probe) — REVISED from "all ana
  backups down": PBS + nh3-restic are HEALTHY & CURRENT; only the ANA-side
  restic file+DB layer is dead, and the one host with NO other cover is
  ana-ml2.** Three layers:
  - ✅ **PBS-ANA (VM images)** — healthy, snapshots TODAY. All PFI guests
    backed up under PBS namespaces `ana-pve` / `esh-pve` / `nh3-pve` (+ tenant
    `sfsrv-pve`); PBS-ANA's own NFS datastore mounts ana-nas fine (1.2T/20T).
    Gaps to note: `ana-pve/vm/100` has no snapshots; **there is NO `esh-pve-nas`
    PBS namespace** → vm-esh-nas + esh-vm-db may be PBS-UNCOVERED (VERIFY).
  - ✅ **restic file+DB nh3-side** (`rest-server-nh3` @ nh3-nas:8000, UP→401) —
    irv-ml1 + nh3-docker, last snapshot TODAY 01:00. Repos at
    `nh3-nas:/volume1/Backup/restic/{irv-ml1,nh3-docker}`.
  - 🔴 **restic file+DB ANA-side** (`rest-server-ana` @ ana-docker:8000) — DOWN,
    crash-looping. Last successful snapshot **2026-05-06 (~6.5 weeks)** across
    ALL 5 clients (ana-docker, ana-ml2, esh-docker-vm, esh-vm-db, vm-esh-nas).
  - **Scheduler:** NOT Backrest (it has **0 plans** — it's just a repo viewer).
    Backups run via **`resticprofile` systemd timers** (`resticprofile-backup@
    profile-default.timer`, daily 01:00; `…-check@…` Sun 05:00). The ana timer
    fires daily and **FAILS SILENTLY** (no alerting → why the gap went unnoticed
    6.5 wks).
  **Root cause:** ana-nas NFS fstab bare-`defaults` (no `_netdev`/retry) →
  `mnt-backup.mount` stuck `failed` → rest-server-ana's bind source
  `/mnt/backup/restic/repo/ana` resolves to an empty LOCAL ghost dir (no
  `.htpasswd`) → crash-loop (`cannot load /data/.htpasswd: permission denied`).
  The REAL repos + `.htpasswd` are intact on ana-nas at
  `/mnt/backup/restic/repo/ana/{<client>/,.htpasswd}`. ana-nas itself is FINE.
  **Real exposure, ranked:** (1) 🔴 **ana-ml2** — BARE METAL, no PBS coverage;
  ana-side restic is its ONLY layer → ~6.5 wks fully unprotected (bespoke
  at-risk: in-house NVFP4 quants / configs / scripts; bulk `/tank` models are
  re-downloadable). (2) 🟠 **esh-vm-db** (a DB VM: postgres+mongo) + vm-esh-nas
  IF PBS-uncovered (verify the esh-pve-nas gap). (3) 🟡 **ana-docker /
  esh-docker-vm** — VMs, PBS image-CURRENT today, only the granular restic
  layer is stale (full-VM restore available; no recent file/DB-granular point).
  **RECOVERY (needs root on ana-docker; the session there = `lkraven`, NO
  NOPASSWD sudo → operator `!`-sudo or bootstrap infra-ops there):** `sudo
  mount -a` → `sudo docker restart rest-server` (recreate if the bind doesn't
  re-resolve onto the NFS; if the ghost persists: stop container → umount → rm
  the ghost `repo/ana` → remount → start) → verify → next 01:00 succeeds.
  **PREVENTION (the systemic fixes):** fstab `_netdev`+retry so it self-heals
  on reboot; a **backup-freshness alert** (the real miss — 6.5 wks unnoticed).
  Recovery steps mirror `docs/runbooks/disaster-recovery.md` Tier-0.
  **⚠️ Disclosed creds:** diagnosing dumped the Backrest config, exposing the 5
  per-repo rest-server basic-auth passwords into the session transcript →
  rotate after recovery (add to the disclosed-keys rotation queue).
  **✅ RECOVERED (2026-06-20):** via `ssh infra-ops@ana-docker` (NOT the default
  lkraven session) — `sudo mount -a` + `reset-failed mnt-backup.mount` re-attached
  the NFS, then `docker compose up -d --force-recreate` rebound rest-server onto
  the real `/data` (now `Loaded htpasswd file`, 401, healthy). Verified end-to-end:
  triggered backups on the two no-PBS hosts' priority — **ana-ml2 snapshot landed
  22:53Z** (total 19→20); ana-docker's larger catch-up was still running at hand-off
  (will land). esh-docker-vm/esh-vm-db/vm-esh-nas auto-catch-up at their next 01:00
  timers (or trigger manually). **Full topology + 2-min freshness-check now documented
  in `docs/runbooks/backups.md`.** **PREVENTION DONE (2026-06-20):** (1) freshness
  alert — `scripts/check-backup-freshness.sh` + daily systemd user timer on nh3-dev
  (08:00) → althing alert to infra-ops on any stale/down layer (proven end-to-end);
  (2) fstab hardened — ana-docker `/mnt/backup` → `noauto,x-systemd.automount,…`
  (autofs self-heals, active next boot; `.bak-pre-harden` saved); (3) esh-pve-nas
  VERIFIED **not** PBS-covered → esh-vm-db (a DB!) + vm-esh-nas are restic-only.
  `/mnt/compose` ALSO hardened + automount activated live (2026-06-20, binds no
  container — proved the autofs pattern works on ana-docker). STILL OPEN: rotate the
  5 rest-server creds (operator, offline); the 3 esh + ana-docker restic clients
  clear at tonight's 01:00.

- **ana-docker disk incident RESOLVED — root cause was a 94 GB unrotated
  `langfuse-clickhouse` log, NOT (just) image bloat.** ClickHouse logs
  verbosely to stdout + there was NO log rotation anywhere (no per-stack
  `logging:`, no daemon default) → its json.log hit ~94 GB and filled `/`
  (100%, 28/48 containers unhealthy on `runc /tmp ENOSPC`). Fixed: truncated
  the log, pruned ~101 GB of images (the gitea-runner accumulated 18
  worldtree build versions), retired Langfuse. Disk → ~51%. Installed a
  daily image+buildx prune cron (`lkraven` crontab on ana-docker). **DEFERRED
  to clean context:** a **docker-daemon default log cap** (`daemon.json`
  log-opts + docker restart, needs root → infra-ops) — the real systemic fix
  so no container can ever spew unbounded again. (Note: named buildx-builder
  state volumes aren't cleared by `docker image/builder prune` — orphaned
  ones need `docker volume rm`.)

- **Langfuse RETIRED (2026-06-20, operator).** Redundant with LiteLLM's
  native logging (`store_prompts_in_spend_logs: true` → full prompts/
  responses/tokens/cost at `:4000/ui`); nothing used its trace-grouping/eval
  features; and it's a 6-container crash-prone surface (the 94 GB spew).
  Dropped the `success_callback`/`failure_callback: [langfuse]` from the
  litellm config (gateway observability stays native) and `compose down -v`
  the stack. (f8eda1c)

- **OmniVoice gained a streaming `/tts` + language-safe sanitizer** (irv-ml1
  :8199, `stacks/omnivoice/`), driven by a VENDORED copy of chatterbox-fast's
  buffer-ratchet scheduler. KEY FINDING: OmniVoice is DIFFUSION → a ~fixed
  per-call overhead sets a TTFA FLOOR (~0.7 s @16 steps), and measured RTF is
  ~2.8×/5.6× on the 3090 (NOT the upstream-claimed 40×). Tuned: streaming
  num_step=16/batch 32, rtf_prior=20. **chatterbox-fast (autoregressive,
  ~0.5 s TTFA) stays the lower-latency front-end** — OmniVoice is the
  multilingual/voice-design complement, NOT a replacement. (288d085, cd92b85;
  auto-memory `reference_omnivoice_streaming_tts`)

- **arbo v0.14.9 deployed** (muse prompt-enhance → `summarizer-large`, was
  granite which mode-collapsed at temp 1.0) via the canonical health-gated
  `scripts/deploy-engine.sh` on irv-ml1. arbo's gitea Actions CI is now LIVE
  (runner-label mismatch fixed by comfy-dev) → future src pushes auto-deploy;
  manual deploy only when CI is down. (auto-memory `reference_arbo_engine_deploy`)

- **ratatoskr `:8392` (composite Bifrost provider) added** to the personal
  WT `BIFROST_CLIENT_ALLOWED_HOSTS` (footgun-safe recreate: inline sha pin +
  `--pull never`, the `.env` still pins stale `:latest`). And ratatoskr's
  **admin.events.read key (ratatoskr#11) BROKERED** with worldtree-dev — the
  scope is tier-derived (not mint-settable), so worldtree-dev defines the
  isolated admin-tier user/scope, then I mint + secure-deliver. NON-URGENT.

- **claude-bot granted ADMIN on vh/Worldtree** (2026-06-20, via operator
  vh-admin → migration-enabler) so I can self-serve WT deploy re-runs +
  provision worldtree-dev's own Actions token (pending their gitea username).

- **arbo + brokkr litellm keys = `all-proxy-models`** (verified — full gateway
  access incl. paid GLM, no key change needed). `all-agents-local` shared key
  stays fenced to free local models only. claude-bot Gitea service account is
  the credential-migration foundation (admin on vh/arbo + now vh/Worldtree).

- **Worldtree config-propagation:** demo+personal bind-mount their
  `model_roles.yaml`/`providers.yaml` (`/opt/worldtree{,-personal}/config`,
  infra-ops-deployable); **pinned is IN-IMAGE** (worldtree-dev CI bakes it).
  worldtree-dev authors the canonical config; infra-ops deploys to the
  bind-mounted instances byte-identical.

- **Still open from prior:** Mac Pro migration (`migration-plan.md`,
  workstation-only, hardware-gated); R17 v2 corpus push HELD (local-only
  soundgasm-derived, distribution barred); disclosed-keys hygiene rotation
  queue; clean legacy `news-digest` on ana-docker.

## Recent decisions

- `[2026-06-20]` **claude-bot issue-scope token minted for worldtree-dev self-serve**
  (closes their last `tea`-as-vh fallback — issues; they already self-serve Actions +
  deploys). Gitea PAT scopes are IMMUTABLE → minted a NEW claude-bot PAT
  `worldtree-dev-actions-issues-20260621` (id 16) with `write:repository`+`write:issue`
  via basic-auth (`claude-bot:gitea-password`, POST `/users/claude-bot/tokens`); dropped
  mode-600 `~/.claude-bot-token-issues` → worldtree-dev swaps. **Gitea COLLAPSES
  `read:issue` into `write:issue`** (write implies read) so the granted set shows
  write-only but read IS satisfied (GET issues → 200). Old token (id 15,
  worldtree-dev-actions-20260620) REVOKED (DELETE 204) after worldtree-dev verified
  read+write end-to-end via `core.issues` (show/list/comment on #311) → claude-bot now
  carries ONE consolidated worldtree-dev token (id 16) + arbo-ci (id 14), no stale dups.
  Advances the credential-migration directive (worldtree-dev fully self-serve: Actions +
  deploys + issues on one token). ALSO this session: re-staged cb2a79a `policies.yaml` (readonly-admin allow
  rules) to demo+personal (see Current-state ratatoskr entry).

- `[2026-06-20]` **rest-server-ana recovered + backup prevention shipped + worldtree-dev
  admin keys provisioned** (operator-directed). rest-server-ana fixed (mount-a +
  force-recreate via `infra-ops@ana-docker`); freshness alert + fstab hardening landed
  (see Current-state BACKUP entry + `docs/runbooks/backups.md`). Minted worldtree-dev
  **admin-tier Heimdall keys** on demo (key_id d113207c) + personal (f4f75adb) via
  `POST /admin/keys {tier:admin}` + `POST /admin/users/worldtree-dev/tier`, verified
  read+write, dropped mode-600 to `~/.wt-admin-{demo,personal}` on nh3-dev → worldtree-dev
  now self-serves key minting (cuts infra-ops out of the per-key loop, like the
  claude-bot gitea token). Cred rotation (5 rest-server pw) BELAYED per operator.

- `[2026-06-20]` **Worldtree capability-gateway aliases stood up; NO `wt-`
  prefix (operator) — reuse the existing capability convention.** gen/
  gen-reasoning/classifier/summarizer/chat-judge/reranker/scalar-judge
  (passthrough)/gen-frontier/-reasoning. The **swap-transparency rule**:
  embedding stays model-named (`qwen3-embedding`) because vectors aren't
  swap-transparent; everything else can swap behind a stable alias. (d372103,
  7819f96)

- `[2026-06-20]` **Removed the litellm `*` wildcard → llama-swap** (operator;
  llama-swap decommissioned, :9292 confirmed down). Unknown/typo'd models now
  404 cleanly instead of a misleading "Connection error" — closes the
  long-standing footgun. (d0eb09c)

- `[2026-06-20]` **Langfuse retired** (operator) — redundant with LiteLLM
  native spend_logs + a 6-container crash-prone surface (the 94 GB clickhouse
  spew). Callbacks dropped, stack torn down. (f8eda1c)

- `[2026-06-20]` **OmniVoice streaming `/tts` — vendor chatterbox-fast's
  scheduler, not its whole package** (operator-approved vendor-copy). Diffusion
  TTFA floor finding → tuned 16-step streaming + aggressive packing; chatterbox-
  fast remains the lowest-latency front-end. (288d085, cd92b85)

- `[2026-06-20]` **claude-bot → ADMIN on vh/Worldtree** (operator-authorized;
  one-time use of operator vh-admin to enable the migration) — claude-bot
  self-serves WT deploys/tokens henceforth, reducing personal-cred fallbacks.

- `[2026-06-20]` **Daily image+buildx prune cron on ana-docker** (the missing
  automation behind the recurring image bloat) — `lkraven` crontab, prune
  unused images >48 h + per-builder buildx cache. The real systemic fix
  (daemon log cap) is deferred to clean context.

- `[2026-06-19]` **litellm task-aliases: `classifier`→granite-4.1-8b, `summarizer-large`→gen/qwopus** (`summarizer`→granite already existed). Also added then REMOVED `gen-nt`/`gen-reasoning-nt` (redundant with the global `strip_empty_tools` hook). (ef45f6d, 2e5ab72, 75bd4c3)

- `[2026-06-19]` **gateway-chat: model-smoking web chat → PERSISTENT container.** `/v1/models` auto-discovery + image-upload; `stacks/gateway-chat/` (nginx:alpine, ana-docker:8091, homepage tile). (4c40b9f, 740bcae)

- `[2026-06-19]` **pi on nh3-dev wired to `gen`** — `~/.pi/agent/models.json` + a `gen` launcher mirroring `glm`. Local box config, not a repo artifact. (untracked by operator choice)

- `[2026-06-19]` **`gen` model → Qwopus3.5-122B-A10B (vision-intact NVFP4), full 256K @ fp8.** ana-ml2 GPU 0, served-name `qwen3.5-122-a10b`. STABLE vLLM + fp8 KV reaches 262144 — nightly+turboquant-4bit-KV was UNNECESSARY. 92.7 tok/s warm. (20e796c, 5b06514)

- `[2026-06-19]` **TP=2 across the two ana-ml2 Blackwells REJECTED** — `PIX` (PCIe, NO NVLink) → all-reduce-bound → SLOWER. One-model-per-card is optimal; PP=2 only if a >96 GB model is ever wanted. (untracked by operator choice)

- `[2026-06-19]` **qwen-image-bench (T2I judge, NVFP4) replaced qwen3.6-35b-a3b on GPU 1**, aliased `image-judge`. (bfae924, 5dfce04)

- `[2026-06-18]` **heretic abliterated Mistral Small 4 NVFP4 built + LIVE as `mistral-small-4`** (in-house NVFP4, drop-in stack, A/B'd vs official, "heretic stays"). DISPLACED from GPU 0 on 2026-06-19 for the gen/Qwopus swap. (dd3a5c9, f566f61, `tools/mistral-small4-nvfp4/`)

- `[2026-06-18]` **irv-ml1 VRAM consolidation + comfy-dev torch-pin** — ComfyUI pinned to the A6000 (48 GB), audio zoo → 3090; `DISABLE_UPGRADES=true` @ torch 2.12.1. (a8550ad)

- `[2026-06-18]` **ComfyUI acceleration set (9 nodes) installed for comfy-dev** on irv-ml1.

- `[2026-06-17]` **ana-ml2 `vm.overcommit_memory=1` made durable** (sysctl drop-in) — overcommit=0 + zero swap caps CommitLimit; large model-file mmap ENOMEM'd despite ~393 GB free. (fc88eff)

- `[2026-06-17]` **GLM 5.2 wired into litellm** (`glm-5.2` + `-reasoning`, z.ai passthrough, `extra_body.thinking.type` toggle). (fe77a35)

- `[2026-06-17]` **nh3-extdev stood up as a sudo-LESS infra-ops manager box** (successor to nh3-ansible); hosts `/opt/externs` pi-on-GLM-5.2 client workspaces. (a841eab)

- `[2026-06-14]` **STANDING DIRECTIVE: migrate ALL infra access to Claude-specific credentials.** Stand up service accounts (claude-bot Gitea user + scoped tokens, distinct litellm admin key); re-mint consumer creds under them; flag personal-cred fallbacks until done. (auto-memory `project_migrate_infra_access_to_claude_credentials`)

_101 older entries archived to archival-memory.md._

## Tried and abandoned

- `[2026-06-20]` **A full disk wedges gitea's Actions notifier SILENTLY — and it
  STAYS wedged after disk recovers; only a `docker restart gitea` clears it.** After
  the 94 GB clickhouse disk-full event, gitea stopped CREATING workflow runs on push
  (push succeeds, ref lands, but NO run spawns — newest run stale, queue empty), while
  the act_runner stayed registered. NOT run-pickup, NOT disk-now (52% free), NO gitea
  errors/panics — a silent notifier stall. Hit BOTH vh/arbo (comfy-dev) and vh/Worldtree
  (worldtree-dev's deploys) → it's global, not repo-specific. Fix: `docker restart gitea`
  on ana-docker (it had been up 2 weeks, never bounced). A restart does NOT retroactively
  fire pushes made during the outage — re-push/re-trigger after. Reinforces the deferred
  docker-daemon-default log-cap fix (the disk-full that started it can't be allowed to recur).

- `[2026-06-20]` **Mistral-Medium-3.5-128B NVFP4 (RecViking) on ana-ml2 GPU 0 —
  ABANDONED, too slow (operator confirmed).** Stood up as a temporary speed-check
  displacing mistral-small-4; measured base **~19.2 tok/s** decode on one Blackwell
  (vs `gen`/Qwopus3.5-122B ~92.7 tok/s warm) → confirmed the operator's "too slow"
  suspicion. KEEP these serving findings: (1) vLLM **nightly** loads the HF-format
  NVFP4 weights DIRECTLY — no Mistral native-convert (unlike Small 4, same `mistral3`
  arch) — via the FlashInfer Cutlass NVFP4 kernel + `turboquant_4bit_nc` KV; TP=1
  fits the ~70 GB NVFP4 on one 96 GB Blackwell. (2) **EAGLE was a dead end**: the
  official native FP8 head (`mistralai/Mistral-Medium-3.5-128B-EAGLE`) crashes in
  CUDA-graph replay with graphs on, and nets NO speedup with `--enforce-eager`
  (~18.5 vs ~19.2 base) — likely low draft acceptance against the NVFP4
  (non-official-FP8) base. Untracked workspace files (`stacks/mistral-medium-3.5/` +
  `playbooks/serve-mistral-medium-3.5.yaml`) removed. ⚠️ Weights still staged on
  ana-ml2 `/tank/aimodels/mistral-medium-3.5-{nvfp4,eagle}` (~70 GB+) — pending
  operator OK to free.

- `[2026-06-20]` **A 94 GB unrotated `langfuse-clickhouse` json.log was the
  REAL disk-filler (not image bloat) — there was NO log rotation anywhere.**
  No per-stack `logging:` limit and no docker-daemon default → ClickHouse's
  verbose stdout grew unbounded to ~94 GB and filled ana-docker's `/`
  (healthchecks then failed fleet-wide on `runc /tmp: no space left`). Lesson:
  on a full disk, the symptom (all-unhealthy) is the FORK-can't-write
  healthcheck, not the containers; find the runaway via per-container-dir `du`,
  not `docker system df`. The systemic fix is a daemon-default `log-opts`
  (`max-size`/`max-file`) — NOT just per-service patching.

- `[2026-06-20]` **OmniVoice's upstream "RTF 0.025 / 40×" does NOT hold —
  diffusion has a ~fixed per-call overhead → TTFA floor + multi-chunk
  streaming is a net loss.** Measured ~2.8× (32-step) / 5.6× (16-step) on the
  3090; every chunk costs ~the same regardless of length, so naive
  per-sentence streaming triples total gen time + starves. Use few/aggressive
  chunks; chatterbox-fast (autoregressive) stays the low-TTFA front-end.

- `[2026-06-20]` **rest-server `.htpasswd: permission denied` = the ana-nas
  NFS mount FAILED (ghost file on the local mount point), NOT a decommission.**
  `mnt-backup.mount` stuck `failed` (fstab bare `defaults`, no retry) → the
  rest-server serves an empty local dir with a root:root 0-byte `.htpasswd`.
  Documented recovery in disaster-recovery.md. Don't tear down what looks like
  a crash-looping legacy container until you've checked fstab + the docs — it
  was the live ana-side restic target.

- `[2026-06-20]` **Named buildx builders accumulate cache the standard prunes
  MISS.** `docker image prune -a` + `docker builder prune` don't touch
  `buildx_buildkit_<name>-builder0_state` volumes (per-builder buildkit
  state). Prune them with `docker buildx prune --builder <name> -af`, or
  `docker volume rm` orphaned ones (builder already removed).

- `[2026-06-20]` **The DEFAULT `ssh ana-docker` is `lkraven` (no NOPASSWD) —
  but `ssh infra-ops@ana-docker` HAS NOPASSWD root** (corrected later same day).
  Early on a `sudo cp` as lkraven silently failed (password prompt) → one
  wasted gateway bounce; I then wrongly concluded "no sudo on ana-docker" and
  nearly punted the rest-server-ana recovery to the operator. The real rule:
  reach for `infra-ops@ana-docker` for sudo ops; lkraven-owned files (litellm
  config, most stack compose/conf) take plain `cp` under either identity.

- `[2026-06-19]` **litellm CONFIG-loaded models can't be hot-removed — `/model/delete` is DB-only.** Only a process restart drops a config model. The INVERSE works: `/model/new` live-ADDS (no bounce) but dups with config on restart. Clean no-bounce-add: live-add via API, delete the DB copy before the eventual bounce.

- `[2026-06-19]` **the litellm `*` wildcard routed to DECOMMISSIONED llama-swap → a stale/typo'd model name returned a misleading `Connection error`, not `model not found`.** Bit a brokkr call to the renamed `qwen-image-judge`. **→ RESOLVED 2026-06-20: wildcard removed (d0eb09c) — unknown models now 404 cleanly.**

- `[2026-06-19]` **A MoE card can't be filled to 0 bytes free — the FusedMoE transient workspace (~3.09 GB) is the floor**, allocated OUTSIDE `gpu-memory-utilization`. FIX: `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True` + leave ~3.2 GB free (util ≤ ~0.95 on a tight MoE).

- `[2026-06-19]` **vLLM decode tok/s: ALWAYS discard the first generation (cold warmup).** Cold ≈ 24.8 tok/s vs warm 92.7 — a cold read undersells decode ~3–4×. Measure run 2+ over a ≥256-token output.

- `[2026-06-19]` **For full native 256K on one 96 GB card, nightly+turboquant-4bit-KV was unnecessary for the Qwopus MoE.** Stable fp8 KV already fits 262144 (hybrid SSM+attn caches KV only on attn layers → small pool). Reach for fp8 first.

- `[2026-06-18]` **mmartial `comfyui-nvidia-docker`: root pip installs CRASH-LOOP the container.** `docker exec -u 0 pip install` leaves root-owned files in the uid-1000 venv → the boot script (uid 1000) fails its torch step → crash loop. FIX: `chown -R 1000:1000 /comfy/mnt/venv`; pin with `DISABLE_UPGRADES=true`. (auto-memory `reference_irv_ml1_comfyui_mmartial`)

- `[2026-06-18]` **ComfyUI custom nodes break on version-assumption drift** — box runs 0.24.1; TeaCache imports `precompute_freqs_cis` at MODULE level (guard the LTX-only import). `pip install tensorrt` defaults to cu13 → use `tensorrt-cu12`.

- `[2026-06-17]` **Mistral HF→NVFP4 quant: the placement knob is the whole game.** `device_map="cpu"` (CPU-resident, sequential onload) works; `"auto"` OOMs, `max_memory` offloads to meta. Read shards with plain `read()` + `safetensors.torch.load(bytes)`, NOT `safe_open` (mmaps the 50 GB shard → ENOMEM on `/tank` ZFS).

- `[2026-06-17]` **HF-format Mistral Small 4 is UNSERVEABLE on vLLM** — no HF `Mistral4` backbone; serves ONLY via the native loader. A HF-format quant MUST be converted to native first.

_84 older entries archived to archival-memory.md._