esh-pfi-infrastructure/persistent-memory.md

Persistent memory — eshpfi-management

Last updated: 2026-06-20

Repo purpose

Reference workspace for PFI infrastructure: server inventory, canonical Docker Compose stacks, ops playbooks, and conventions. Authoritative copies of compose files live on the servers under /opt/docker/compose/<stack>/; this repo mirrors them for version control, editing, planning, and CI-driven deploys. It was originally spun up to handle the fleet backups — keep that lens when triaging backup/storage issues.

Tools and conventions

Sister repos (separate gitea repos, deployed by playbooks here):

Repo	Role	CI status
vh/task-board	MCP + web dashboard for assistant task state (port 7878)	push-to-main → CI deploys (2026-04-29)
vh/vor	Inquisitor UI sidecar (port 7879)	push-to-main → CI deploys (2026-04-29)
vh/nevermore	Twice-daily LLM-curated briefing (port 8181, replaces news-digest)	push-to-main → CI deploys (2026-04-30)
vh/asset-engine	Internal control plane over inference services (port 8200, LAN-direct)	push-to-main → CI deploys (2026-05-12)
vh/althing	Inter-agent message bus (chamber UI port 7881, forseti + agent-runner daemons, valkey IPC)	push-to-main → CI deploys (2026-05-14)
vh/mead-hall	Bifrost tool-provider sidecar (port 5173 on dev VM 10.100.10.50)	push-to-main → CI deploys (2026-05-16)
vh/skaldsong	Wizard + reader surface (port 8300, ana-docker, registry-pull pattern)	push-to-main → CI deploys (2026-05-19)
vh/Worldtree	Conversation API (corviduo-dev demo :8080 / personal :8081 / pinned :8082) — Heimdall auth, Bifrost integration. gitea-runner builds on ana-docker (the image-bloat source); claude-bot now ADMIN collaborator (2026-06-20)	push-to-main → CI build-and-deploy (runner on ana-docker)
vh/yt-voice-clipper	YouTube → diarized voice-clip dataset builder + audition console (irv-ml1 :8000)	push-to-main → gitea-webhook auto-deploy to irv-ml1 (2026-06-03) — see docs/runbooks/ytvc-autodeploy.md
vh/arbo	Catalog-driven ComfyUI engine (irv-ml1 :8201, comfy-dev owns engine/catalog/image)	push-to-main → gitea Actions CI (deploy-engine.sh, build-local, health-gated) now LIVE; catalog via :9009 webhook

(vh/volva + Heid were re-architected from systemd daemons to Claude Code session orchestrators 2026-06-08; their nh3-dev .service units were removed — no longer deployed sidecars here. See Recent decisions.)

• Two-layer backups — Backrest orchestrates restic for file+DB (5 fleet repos, daily 01:00 PDT); PBS-ANA primary + PBS-NH3 DR mirror for VM images. ana-nas is the SPOF for postgres + PBS-ANA datastore + cross-site restic targets — see docs/runbooks/disaster-recovery.md for the blast-radius matrix. ⚠️ The restic file+DB layer routes through TWO rest-servers (rest-server-ana @ ana-docker:8000 → ana-docker/ana-ml2/esh-docker-vm/vm-esh-nas; rest-server-nh3 @ nh3-nas:8000 → irv-ml1/nh3-docker). Both depend on their NAS's NFS export of /mnt/backup. As of 2026-06-20 rest-server-ana is DOWN on a failed ana-nas NFS mount — see Current state.

• pull-hf-repo.yaml is the canonical "get a HuggingFace model/dataset onto ana-ml2's shared cache at /tank/aimodels/huggingface/" playbook. Supports --var repo_type=model|dataset|space. Replaces ad-hoc huggingface_hub.snapshot_download patterns.

• Worldtree admin auth — per-instance. Each Worldtree deployment (demo :8080, personal :8081, pinned :8082) has its own Heimdall registry and its own bootstrap admin key. Infra-ops's stored long-lived admin key (key_id 61419c92) at ana-docker:/opt/docker/conf/.secrets/worldtree-infra-ops-admin auths against demo only. Personal-instance admin (the ~/.config/worldtree/personal-admin-token, mode 600) POSTs /admin/keys (mints per-project keys; takes user_id+label, no scope param — scopes are tier-derived, so admin-tier scopes like admin.events.read must be granted WT-side by worldtree-dev).

• Per-project user keys against personal Worldtree (issued 2026-05-19): skaldsong:79744637 (nh3-dev iteration), skaldsong:7c1dbbbe (ana-docker prod), althing:50d85460, mead-hall:a360822d. Same user_id=skaldsong across both skaldsong keys → shared Heimdall agent slot; different key_id → independently rotatable. Pattern: mint via /admin/keys, drop value to /tmp/wt-personal-<name>.key mode 600, dev collects + shreds (DO NOT cat to chat transcript).

• Skaldsong CD pattern (registry-pull). Differs from althing / asset-engine which build-on-host. vh/skaldsong's CI builds and pushes gitea.phasefinal.com/vh/skaldsong:<sha> + :latest; playbooks/deploy-skaldsong.yaml on ana-docker pulls + recreates. SHA-pin only (no :latest health-gated advance yet). Prereq: host needs docker login gitea.phasefinal.com once (read:package PAT) — not currently in the workflow.

• gitea internal route for fleet hosts. gitea is a container on ana-docker — git-SSH 10.250.50.70:222, HTTP :3000. Fleet/colo hosts must use this internal route, NOT public gitea.phasefinal.com (38.120.12.44, ana-srv1) — the public path fail2bans the host egress IP and wedges webhook deploys. :22 on 10.250.50.70 is ana-docker's HOST sshd, not gitea. Full gotcha in docs/orientation.md → Git/gitea.

• docker-as-root pattern (for ops with no admin API, or to edit deploy-owned/root-owned files without sudo): docker run --rm -v <target-dir>:/wt [-v /var/run/docker.sock:/var/run/docker.sock] docker:cli sh -c "..." (or alpine for plain file ops). docker-group membership is effectively root via bind-mount; treat as sudo-equivalent. Foot-gun: relative paths in compose.yaml resolve against the sandbox CWD but the Docker daemon interprets them against the HOST fs — always pass -e VAR=/abs/path for any relative-default config dir.

• scripts/elway sudo handling — elway prompts for the sudo password ONCE via getpass before the first sudo: true step → can't run unattended from a non-TTY tool if any step needs sudo. Sudo-free playbooks run fully non-interactive over key SSH.

• Per-host SSH identity matters for sudo. infra-ops has NOPASSWD sudo on most PFI Linux boxes (corviduo-dev included since 2026-06-15) — BUT on ana-docker the session lands as lkraven (docker-group, NO passwordless sudo). So on ana-docker: docker commands work directly; root-owned-file edits need the docker-as-root trick or infra-ops; but the litellm config + most stack compose/conf are lkraven-owned → plain cp/edit, no sudo. (Learned 2026-06-20 — see Tried-and-abandoned.)

Current state / in-flight

As of 2026-06-20:

• Worldtree capability-gateway migration — LANDED + DEPLOY TRIGGERED. The role→capability indirection (worldtree-dev's #257/ADR-0012 direction; operator: NO wt- prefix, reuse the existing capability-alias convention) is live on the LiteLLM gateway: gen/gen-reasoning (→qwen3.5-122-a10b), classifier/summarizer (→granite), chat-judge (→selene-1-mini-8b), reranker (→qwen3-reranker), scalar-judge (passthrough → Skywork-Reward-V2 @ ana-ml2:8003 — LiteLLM has no reward MODE), gen-frontier/-reasoning (→GLM 5.2, PAID). qwen3-embedding kept model-named ON PURPOSE — the swap-transparency rule: embedding vectors aren't swap-transparent, so a generic embedding alias is a footgun. The * wildcard → dead llama-swap was REMOVED (unknown models now 404 cleanly). WT's full re-point (granite SPLIT into granite-classifier/granite-summarizer + role rebind + edges→gen + selene→chat-judge) landed as worldtree-dev's commit 3fbf338 / v0.37.5; I staged its providers.yaml+model_roles.yaml byte-identical into demo+personal bind-mounts and triggered build-and-deploy via claude-bot. Pinned rides CI. (d372103, d0eb09c, 7819f96)

• 🔴 BACKUP GAP — rest-server-ana DOWN ~months on a failed ana-nas NFS mount (NOT a decommission — docs corrected the recollection). The active ana-side restic file+DB target (ana-docker/ana-ml2/esh-docker-vm/ vm-esh-nas) is crash-looping: /etc/fstab has 10.250.50.50:/mnt/backup as an NFS mount, but mnt-backup.mount is in a failed state → /mnt/backup is the empty local ghost dir → .htpasswd permission denied (the documented disaster-recovery.md failure). ana-nas is pingable; the mount just isn't attached (bare-defaults fstab, no auto-retry). So file+DB restic for 4 hosts has been failing for ~months (PBS VM-image backups are the only cover). DEFERRED to clean context: the documented recovery (clear failed mount → rm ghost → mount /mnt/backup → restart rest-server) needs root → infra-ops; plus prevention (fstab _netdev/retry + a backup-freshness alert — the gap went unnoticed for months).

• ana-docker disk incident RESOLVED — root cause was a 94 GB unrotated langfuse-clickhouse log, NOT (just) image bloat. ClickHouse logs verbosely to stdout + there was NO log rotation anywhere (no per-stack logging:, no daemon default) → its json.log hit ~94 GB and filled / (100%, 28/48 containers unhealthy on runc /tmp ENOSPC). Fixed: truncated the log, pruned ~101 GB of images (the gitea-runner accumulated 18 worldtree build versions), retired Langfuse. Disk → ~51%. Installed a daily image+buildx prune cron (lkraven crontab on ana-docker). DEFERRED to clean context: a docker-daemon default log cap (daemon.json log-opts + docker restart, needs root → infra-ops) — the real systemic fix so no container can ever spew unbounded again. (Note: named buildx-builder state volumes aren't cleared by docker image/builder prune — orphaned ones need docker volume rm.)

• Langfuse RETIRED (2026-06-20, operator). Redundant with LiteLLM's native logging (store_prompts_in_spend_logs: true → full prompts/ responses/tokens/cost at :4000/ui); nothing used its trace-grouping/eval features; and it's a 6-container crash-prone surface (the 94 GB spew). Dropped the success_callback/failure_callback: [langfuse] from the litellm config (gateway observability stays native) and compose down -v the stack. (f8eda1c)

• OmniVoice gained a streaming /tts + language-safe sanitizer (irv-ml1 :8199, stacks/omnivoice/), driven by a VENDORED copy of chatterbox-fast's buffer-ratchet scheduler. KEY FINDING: OmniVoice is DIFFUSION → a ~fixed per-call overhead sets a TTFA FLOOR (~0.7 s @16 steps), and measured RTF is ~2.8×/5.6× on the 3090 (NOT the upstream-claimed 40×). Tuned: streaming num_step=16/batch 32, rtf_prior=20. chatterbox-fast (autoregressive, ~0.5 s TTFA) stays the lower-latency front-end — OmniVoice is the multilingual/voice-design complement, NOT a replacement. (288d085, cd92b85; auto-memory reference_omnivoice_streaming_tts)

• arbo v0.14.9 deployed (muse prompt-enhance → summarizer-large, was granite which mode-collapsed at temp 1.0) via the canonical health-gated scripts/deploy-engine.sh on irv-ml1. arbo's gitea Actions CI is now LIVE (runner-label mismatch fixed by comfy-dev) → future src pushes auto-deploy; manual deploy only when CI is down. (auto-memory reference_arbo_engine_deploy)

• ratatoskr :8392 (composite Bifrost provider) added to the personal WT BIFROST_CLIENT_ALLOWED_HOSTS (footgun-safe recreate: inline sha pin + --pull never, the .env still pins stale :latest). And ratatoskr's admin.events.read key (ratatoskr#11) BROKERED with worldtree-dev — the scope is tier-derived (not mint-settable), so worldtree-dev defines the isolated admin-tier user/scope, then I mint + secure-deliver. NON-URGENT.

• claude-bot granted ADMIN on vh/Worldtree (2026-06-20, via operator vh-admin → migration-enabler) so I can self-serve WT deploy re-runs + provision worldtree-dev's own Actions token (pending their gitea username).

• arbo + brokkr litellm keys = all-proxy-models (verified — full gateway access incl. paid GLM, no key change needed). all-agents-local shared key stays fenced to free local models only. claude-bot Gitea service account is the credential-migration foundation (admin on vh/arbo + now vh/Worldtree).

• Worldtree config-propagation: demo+personal bind-mount their model_roles.yaml/providers.yaml (/opt/worldtree{,-personal}/config, infra-ops-deployable); pinned is IN-IMAGE (worldtree-dev CI bakes it). worldtree-dev authors the canonical config; infra-ops deploys to the bind-mounted instances byte-identical.

• Still open from prior: Mac Pro migration (migration-plan.md, workstation-only, hardware-gated); R17 v2 corpus push HELD (local-only soundgasm-derived, distribution barred); disclosed-keys hygiene rotation queue; clean legacy news-digest on ana-docker.

Recent decisions

• [2026-06-20] Worldtree capability-gateway aliases stood up; NO wt- prefix (operator) — reuse the existing capability convention. gen/ gen-reasoning/classifier/summarizer/chat-judge/reranker/scalar-judge (passthrough)/gen-frontier/-reasoning. The swap-transparency rule: embedding stays model-named (qwen3-embedding) because vectors aren't swap-transparent; everything else can swap behind a stable alias. (d372103, 7819f96)

• [2026-06-20] Removed the litellm * wildcard → llama-swap (operator; llama-swap decommissioned, :9292 confirmed down). Unknown/typo'd models now 404 cleanly instead of a misleading "Connection error" — closes the long-standing footgun. (d0eb09c)

• [2026-06-20] Langfuse retired (operator) — redundant with LiteLLM native spend_logs + a 6-container crash-prone surface (the 94 GB clickhouse spew). Callbacks dropped, stack torn down. (f8eda1c)

• [2026-06-20] OmniVoice streaming /tts — vendor chatterbox-fast's scheduler, not its whole package (operator-approved vendor-copy). Diffusion TTFA floor finding → tuned 16-step streaming + aggressive packing; chatterbox- fast remains the lowest-latency front-end. (288d085, cd92b85)

• [2026-06-20] claude-bot → ADMIN on vh/Worldtree (operator-authorized; one-time use of operator vh-admin to enable the migration) — claude-bot self-serves WT deploys/tokens henceforth, reducing personal-cred fallbacks.

• [2026-06-20] Daily image+buildx prune cron on ana-docker (the missing automation behind the recurring image bloat) — lkraven crontab, prune unused images >48 h + per-builder buildx cache. The real systemic fix (daemon log cap) is deferred to clean context.

• [2026-06-19] litellm task-aliases: classifier→granite-4.1-8b, summarizer-large→gen/qwopus (summarizer→granite already existed). Also added then REMOVED gen-nt/gen-reasoning-nt (redundant with the global strip_empty_tools hook). (ef45f6d, 2e5ab72, 75bd4c3)

• [2026-06-19] gateway-chat: model-smoking web chat → PERSISTENT container. /v1/models auto-discovery + image-upload; stacks/gateway-chat/ (nginx:alpine, ana-docker:8091, homepage tile). (4c40b9f, 740bcae)

• [2026-06-19] pi on nh3-dev wired to gen — ~/.pi/agent/models.json + a gen launcher mirroring glm. Local box config, not a repo artifact. (untracked by operator choice)

• [2026-06-19] gen model → Qwopus3.5-122B-A10B (vision-intact NVFP4), full 256K @ fp8. ana-ml2 GPU 0, served-name qwen3.5-122-a10b. STABLE vLLM + fp8 KV reaches 262144 — nightly+turboquant-4bit-KV was UNNECESSARY. 92.7 tok/s warm. (20e796c, 5b06514)

• [2026-06-19] TP=2 across the two ana-ml2 Blackwells REJECTED — PIX (PCIe, NO NVLink) → all-reduce-bound → SLOWER. One-model-per-card is optimal; PP=2 only if a >96 GB model is ever wanted. (untracked by operator choice)

• [2026-06-19] qwen-image-bench (T2I judge, NVFP4) replaced qwen3.6-35b-a3b on GPU 1, aliased image-judge. (bfae924, 5dfce04)

• [2026-06-18] heretic abliterated Mistral Small 4 NVFP4 built + LIVE as mistral-small-4 (in-house NVFP4, drop-in stack, A/B'd vs official, "heretic stays"). DISPLACED from GPU 0 on 2026-06-19 for the gen/Qwopus swap. (dd3a5c9, f566f61, tools/mistral-small4-nvfp4/)

• [2026-06-18] irv-ml1 VRAM consolidation + comfy-dev torch-pin — ComfyUI pinned to the A6000 (48 GB), audio zoo → 3090; DISABLE_UPGRADES=true @ torch 2.12.1. (a8550ad)

• [2026-06-18] ComfyUI acceleration set (9 nodes) installed for comfy-dev on irv-ml1.

• [2026-06-17] ana-ml2 vm.overcommit_memory=1 made durable (sysctl drop-in) — overcommit=0 + zero swap caps CommitLimit; large model-file mmap ENOMEM'd despite ~393 GB free. (fc88eff)

• [2026-06-17] GLM 5.2 wired into litellm (glm-5.2 + -reasoning, z.ai passthrough, extra_body.thinking.type toggle). (fe77a35)

• [2026-06-17] nh3-extdev stood up as a sudo-LESS infra-ops manager box (successor to nh3-ansible); hosts /opt/externs pi-on-GLM-5.2 client workspaces. (a841eab)

• [2026-06-14] STANDING DIRECTIVE: migrate ALL infra access to Claude-specific credentials. Stand up service accounts (claude-bot Gitea user + scoped tokens, distinct litellm admin key); re-mint consumer creds under them; flag personal-cred fallbacks until done. (auto-memory project_migrate_infra_access_to_claude_credentials)

101 older entries archived to archival-memory.md.

Tried and abandoned

• [2026-06-20] A 94 GB unrotated langfuse-clickhouse json.log was the REAL disk-filler (not image bloat) — there was NO log rotation anywhere. No per-stack logging: limit and no docker-daemon default → ClickHouse's verbose stdout grew unbounded to ~94 GB and filled ana-docker's / (healthchecks then failed fleet-wide on runc /tmp: no space left). Lesson: on a full disk, the symptom (all-unhealthy) is the FORK-can't-write healthcheck, not the containers; find the runaway via per-container-dir du, not docker system df. The systemic fix is a daemon-default log-opts (max-size/max-file) — NOT just per-service patching.

• [2026-06-20] OmniVoice's upstream "RTF 0.025 / 40×" does NOT hold — diffusion has a ~fixed per-call overhead → TTFA floor + multi-chunk streaming is a net loss. Measured ~2.8× (32-step) / 5.6× (16-step) on the 3090; every chunk costs ~the same regardless of length, so naive per-sentence streaming triples total gen time + starves. Use few/aggressive chunks; chatterbox-fast (autoregressive) stays the low-TTFA front-end.

• [2026-06-20] rest-server .htpasswd: permission denied = the ana-nas NFS mount FAILED (ghost file on the local mount point), NOT a decommission. mnt-backup.mount stuck failed (fstab bare defaults, no retry) → the rest-server serves an empty local dir with a root:root 0-byte .htpasswd. Documented recovery in disaster-recovery.md. Don't tear down what looks like a crash-looping legacy container until you've checked fstab + the docs — it was the live ana-side restic target.

• [2026-06-20] Named buildx builders accumulate cache the standard prunes MISS. docker image prune -a + docker builder prune don't touch buildx_buildkit_<name>-builder0_state volumes (per-builder buildkit state). Prune them with docker buildx prune --builder <name> -af, or docker volume rm orphaned ones (builder already removed).

• [2026-06-20] On ana-docker I am lkraven (no NOPASSWD sudo), not infra-ops. A sudo cp of the litellm config silently failed (password prompt) and I restarted the gateway on an unchanged config (one wasted bounce). The config + most stack files are lkraven-owned → plain cp, no sudo. Verify the SSH user/ownership before reaching for sudo.

• [2026-06-19] litellm CONFIG-loaded models can't be hot-removed — /model/delete is DB-only. Only a process restart drops a config model. The INVERSE works: /model/new live-ADDS (no bounce) but dups with config on restart. Clean no-bounce-add: live-add via API, delete the DB copy before the eventual bounce.

• [2026-06-19] the litellm * wildcard routed to DECOMMISSIONED llama-swap → a stale/typo'd model name returned a misleading Connection error, not model not found. Bit a brokkr call to the renamed qwen-image-judge. → RESOLVED 2026-06-20: wildcard removed (d0eb09c) — unknown models now 404 cleanly.

• [2026-06-19] A MoE card can't be filled to 0 bytes free — the FusedMoE transient workspace (~3.09 GB) is the floor, allocated OUTSIDE gpu-memory-utilization. FIX: PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True + leave ~3.2 GB free (util ≤ ~0.95 on a tight MoE).

• [2026-06-19] vLLM decode tok/s: ALWAYS discard the first generation (cold warmup). Cold ≈ 24.8 tok/s vs warm 92.7 — a cold read undersells decode ~3–4×. Measure run 2+ over a ≥256-token output.

• [2026-06-19] For full native 256K on one 96 GB card, nightly+turboquant-4bit-KV was unnecessary for the Qwopus MoE. Stable fp8 KV already fits 262144 (hybrid SSM+attn caches KV only on attn layers → small pool). Reach for fp8 first.

• [2026-06-18] mmartial comfyui-nvidia-docker: root pip installs CRASH-LOOP the container. docker exec -u 0 pip install leaves root-owned files in the uid-1000 venv → the boot script (uid 1000) fails its torch step → crash loop. FIX: chown -R 1000:1000 /comfy/mnt/venv; pin with DISABLE_UPGRADES=true. (auto-memory reference_irv_ml1_comfyui_mmartial)

• [2026-06-18] ComfyUI custom nodes break on version-assumption drift — box runs 0.24.1; TeaCache imports precompute_freqs_cis at MODULE level (guard the LTX-only import). pip install tensorrt defaults to cu13 → use tensorrt-cu12.

• [2026-06-17] Mistral HF→NVFP4 quant: the placement knob is the whole game. device_map="cpu" (CPU-resident, sequential onload) works; "auto" OOMs, max_memory offloads to meta. Read shards with plain read() + safetensors.torch.load(bytes), NOT safe_open (mmaps the 50 GB shard → ENOMEM on /tank ZFS).

• [2026-06-17] HF-format Mistral Small 4 is UNSERVEABLE on vLLM — no HF Mistral4 backbone; serves ONLY via the native loader. A HF-format quant MUST be converted to native first.

84 older entries archived to archival-memory.md.