Persistent memory — eshpfi-management

Last updated: 2026-06-20

Repo purpose

Reference workspace for PFI infrastructure: server inventory, canonical Docker Compose stacks, ops playbooks, and conventions. Authoritative copies of compose files live on the servers under /opt/docker/compose/<stack>/; this repo mirrors them for version control, editing, planning, and CI-driven deploys. It was originally spun up to handle the fleet backups — keep that lens when triaging backup/storage issues.

Tools and conventions

Sister repos (separate gitea repos, deployed by playbooks here):

Repo	Role	CI status
`vh/task-board`	MCP + web dashboard for assistant task state (port 7878)	push-to-main → CI deploys (2026-04-29)
`vh/vor`	Inquisitor UI sidecar (port 7879)	push-to-main → CI deploys (2026-04-29)
`vh/nevermore`	Twice-daily LLM-curated briefing (port 8181, replaces news-digest)	push-to-main → CI deploys (2026-04-30)
`vh/asset-engine`	Internal control plane over inference services (port 8200, LAN-direct)	push-to-main → CI deploys (2026-05-12)
`vh/althing`	Inter-agent message bus (chamber UI port 7881, forseti + agent-runner daemons, valkey IPC)	push-to-main → CI deploys (2026-05-14)
`vh/mead-hall`	Bifrost tool-provider sidecar (port 5173 on dev VM 10.100.10.50)	push-to-main → CI deploys (2026-05-16)
`vh/skaldsong`	Wizard + reader surface (port 8300, ana-docker, registry-pull pattern)	push-to-main → CI deploys (2026-05-19)
`vh/Worldtree`	Conversation API (corviduo-dev demo :8080 / personal :8081 / pinned :8082) — Heimdall auth, Bifrost integration. gitea-runner builds on ana-docker (the image-bloat source); claude-bot now ADMIN collaborator (2026-06-20)	push-to-main → CI build-and-deploy (runner on ana-docker)
`vh/yt-voice-clipper`	YouTube → diarized voice-clip dataset builder + audition console (irv-ml1 :8000)	push-to-main → gitea-webhook auto-deploy to irv-ml1 (2026-06-03) — see `docs/runbooks/ytvc-autodeploy.md`
`vh/arbo`	Catalog-driven ComfyUI engine (irv-ml1 :8201, comfy-dev owns engine/catalog/image)	push-to-main → gitea Actions CI (deploy-engine.sh, build-local, health-gated) now LIVE; catalog via :9009 webhook

(vh/volva + Heid were re-architected from systemd daemons to Claude Code session orchestrators 2026-06-08; their nh3-dev .service units were removed — no longer deployed sidecars here. See Recent decisions.)

Two-layer backups — Backrest orchestrates restic for file+DB (5 fleet repos, daily 01:00 PDT); PBS-ANA primary + PBS-NH3 DR mirror for VM images. ana-nas is the SPOF for postgres + PBS-ANA datastore + cross-site restic targets — see docs/runbooks/disaster-recovery.md for the blast-radius matrix. ⚠️ The restic file+DB layer routes through TWO rest-servers (rest-server-ana @ ana-docker:8000 → ana-docker/ana-ml2/esh-docker-vm/vm-esh-nas; rest-server-nh3 @ nh3-nas:8000 → irv-ml1/nh3-docker). Both depend on their NAS's NFS export of /mnt/backup. As of 2026-06-20 rest-server-ana is DOWN on a failed ana-nas NFS mount — see Current state.
pull-hf-repo.yaml is the canonical "get a HuggingFace model/dataset onto ana-ml2's shared cache at /tank/aimodels/huggingface/" playbook. Supports --var repo_type=model|dataset|space. Replaces ad-hoc huggingface_hub.snapshot_download patterns.
Worldtree admin auth — per-instance. Each Worldtree deployment (demo :8080, personal :8081, pinned :8082) has its own Heimdall registry and its own bootstrap admin key. Infra-ops's stored long-lived admin key (key_id 61419c92) at ana-docker:/opt/docker/conf/.secrets/worldtree-infra-ops-admin auths against demo only. Personal-instance admin (the ~/.config/worldtree/personal-admin-token, mode 600) POSTs /admin/keys (mints per-project keys; takes user_id+label, no scope param — scopes are tier-derived, so admin-tier scopes like admin.events.read must be granted WT-side by worldtree-dev).
Per-project user keys against personal Worldtree (issued 2026-05-19): skaldsong:79744637 (nh3-dev iteration), skaldsong:7c1dbbbe (ana-docker prod), althing:50d85460, mead-hall:a360822d. Same user_id=skaldsong across both skaldsong keys → shared Heimdall agent slot; different key_id → independently rotatable. Pattern: mint via /admin/keys, drop value to /tmp/wt-personal-<name>.key mode 600, dev collects + shreds (DO NOT cat to chat transcript).
Skaldsong CD pattern (registry-pull). Differs from althing / asset-engine which build-on-host. vh/skaldsong's CI builds and pushes gitea.phasefinal.com/vh/skaldsong:<sha> + :latest; playbooks/deploy-skaldsong.yaml on ana-docker pulls + recreates. SHA-pin only (no :latest health-gated advance yet). Prereq: host needs docker login gitea.phasefinal.com once (read:package PAT) — not currently in the workflow.
gitea internal route for fleet hosts. gitea is a container on ana-docker — git-SSH 10.250.50.70:222, HTTP :3000. Fleet/colo hosts must use this internal route, NOT public gitea.phasefinal.com (38.120.12.44, ana-srv1) — the public path fail2bans the host egress IP and wedges webhook deploys. :22 on 10.250.50.70 is ana-docker's HOST sshd, not gitea. Full gotcha in docs/orientation.md → Git/gitea.
docker-as-root pattern (for ops with no admin API, or to edit deploy-owned/root-owned files without sudo): docker run --rm -v <target-dir>:/wt [-v /var/run/docker.sock:/var/run/docker.sock] docker:cli sh -c "..." (or alpine for plain file ops). docker-group membership is effectively root via bind-mount; treat as sudo-equivalent. Foot-gun: relative paths in compose.yaml resolve against the sandbox CWD but the Docker daemon interprets them against the HOST fs — always pass -e VAR=/abs/path for any relative-default config dir.
scripts/elway sudo handling — elway prompts for the sudo password ONCE via getpass before the first sudo: true step → can't run unattended from a non-TTY tool if any step needs sudo. Sudo-free playbooks run fully non-interactive over key SSH.
Per-host SSH identity matters for sudo. infra-ops has NOPASSWD sudo on most PFI Linux boxes (corviduo-dev included since 2026-06-15). On ana-docker there are TWO identities: the default ssh ana-docker = lkraven (docker-group, NO passwordless sudo — docker works, root-file edits don't); but ssh infra-ops@ana-docker HAS NOPASSWD root (verified 2026-06-20). → For any sudo op on ana-docker (mount, root-owned files, service control), use ssh infra-ops@ana-docker, NOT the default session. lkraven-owned files (litellm config, most stack compose/conf) still take plain cp/edit under either identity.

Current state / in-flight

As of 2026-06-20:

Worldtree capability-gateway migration — LANDED + DEPLOY TRIGGERED. The role→capability indirection (worldtree-dev's #257/ADR-0012 direction; operator: NO wt- prefix, reuse the existing capability-alias convention) is live on the LiteLLM gateway: gen/gen-reasoning (→qwen3.5-122-a10b), classifier/summarizer (→granite), chat-judge (→selene-1-mini-8b), reranker (→qwen3-reranker), scalar-judge (passthrough → Skywork-Reward-V2 @ ana-ml2:8003 — LiteLLM has no reward MODE), gen-frontier/-reasoning (→GLM 5.2, PAID). qwen3-embedding kept model-named ON PURPOSE — the swap-transparency rule: embedding vectors aren't swap-transparent, so a generic embedding alias is a footgun. The * wildcard → dead llama-swap was REMOVED (unknown models now 404 cleanly). WT's full re-point (granite SPLIT into granite-classifier/granite-summarizer + role rebind + edges→gen + selene→chat-judge) landed as worldtree-dev's commit 3fbf338 / v0.37.5; I staged its providers.yaml+model_roles.yaml byte-identical into demo+personal bind-mounts and triggered build-and-deploy via claude-bot. Pinned rides CI. (d372103, d0eb09c, 7819f96)
Worldtree demo → v0.37.7 GREEN (2026-06-20) — two-half fix. worldtree-dev's v0.37.5 config (3fbf338) carried a dangling granite-4.1-8b catalog ref → demo crash-blocked at startup validation. Fix needed BOTH halves: (a) gitea was silently not creating Actions runs so no deploy could fire (the earlier disk-full wedged its notifier — docker restart gitea cleared it; see Tried-and-abandoned); (b) the v0.37.7 IMAGE alone wouldn't fix demo because demo+personal read BIND-MOUNTED config — I re-staged config/providers.yaml
- config/model_roles.yaml from gitea at 439bebf byte-identical into BOTH /opt/worldtree{,-personal}/config (sha256-verified across source + both instances; granite-4.1-8b→summarizer, granite-classifier/summarizer→ classifier/summarizer; old configs saved .bak-pre-439bebf; NO container recreate — avoided the stale-:latest footgun). worldtree-dev self-served the CI re-run via the claude-bot Actions token → run 1290 GREEN, demo on v0.37.7, capability routing live. claude-bot token migration DONE for worldtree-dev (off vh's personal PAT). RATATOSKR → READONLY (2026-06-20, final): operator first said admin (worldtree-dev minted interim ratatoskr-admin 90db1fbd), then flipped to least-privilege → that key REVOKED (DELETE 200). I re-staged 439bebf's config/policies.yaml to /opt/worldtree-personal/config (byte-identical, sha256 f3ca3e6…, .bak-pre-439bebf) and reloaded Heimdall via docker restart worldtree-personal-worldtree-api-1 (same d2e9f05 SHA — the SAFE reload, no recreate/ no :latest flip); personal came back healthy + the readonly-admin tier (7 read scopes incl. admin.events.read) is now in the loaded policy. FOLLOW-UP cb2a79a (v0.37.9): the 439bebf tier-only stage was NOT sufficient — WT's PDP is RULE-based, so a tier with scopes but NO allow RULES is default-denied; cb2a79a adds the 4 readonly-admin allow rules. Re-staged cb2a79a config/policies.yaml to BOTH demo + personal (sha256 d853e51…, .bak-pre-cb2a79a) + restarted BOTH API containers (worldtree-worldtree-api-1 + …-personal-…); both healthy + serving. (Still on d2e9f05 image — CI cb2a79a deploy pending, but the bind-mount policy is the load-bearing fix.) worldtree-dev mints the ratatoskr readonly key via their admin key. Lessons: (1) reload bind-mounted WT config via docker restart <container>, NEVER compose up (the :latest footgun); (2) a WT tier needs allow RULES, not just scopes — the PDP is rule-based, scopes alone default-deny.
🟠 BACKUP DIAGNOSIS (2026-06-20, full probe) — REVISED from "all ana backups down": PBS + nh3-restic are HEALTHY & CURRENT; only the ANA-side restic file+DB layer is dead, and the one host with NO other cover is ana-ml2. Three layers:
- ✅ PBS-ANA (VM images) — healthy, snapshots TODAY. All PFI guests backed up under PBS namespaces ana-pve / esh-pve / nh3-pve (+ tenant sfsrv-pve); PBS-ANA's own NFS datastore mounts ana-nas fine (1.2T/20T). Gaps to note: ana-pve/vm/100 has no snapshots; there is NO esh-pve-nas PBS namespace → vm-esh-nas + esh-vm-db may be PBS-UNCOVERED (VERIFY).
- ✅ restic file+DB nh3-side (rest-server-nh3 @ nh3-nas:8000, UP→401) — irv-ml1 + nh3-docker, last snapshot TODAY 01:00. Repos at nh3-nas:/volume1/Backup/restic/{irv-ml1,nh3-docker}.
- 🔴 restic file+DB ANA-side (rest-server-ana @ ana-docker:8000) — DOWN, crash-looping. Last successful snapshot 2026-05-06 (~6.5 weeks) across ALL 5 clients (ana-docker, ana-ml2, esh-docker-vm, esh-vm-db, vm-esh-nas).
- Scheduler: NOT Backrest (it has 0 plans — it's just a repo viewer). Backups run via resticprofile systemd timers (resticprofile-backup@ profile-default.timer, daily 01:00; …-check@… Sun 05:00). The ana timer fires daily and FAILS SILENTLY (no alerting → why the gap went unnoticed 6.5 wks). Root cause: ana-nas NFS fstab bare-defaults (no _netdev/retry) → mnt-backup.mount stuck failed → rest-server-ana's bind source /mnt/backup/restic/repo/ana resolves to an empty LOCAL ghost dir (no .htpasswd) → crash-loop (cannot load /data/.htpasswd: permission denied). The REAL repos + .htpasswd are intact on ana-nas at /mnt/backup/restic/repo/ana/{<client>/,.htpasswd}. ana-nas itself is FINE. Real exposure, ranked: (1) 🔴 ana-ml2 — BARE METAL, no PBS coverage; ana-side restic is its ONLY layer → ~6.5 wks fully unprotected (bespoke at-risk: in-house NVFP4 quants / configs / scripts; bulk /tank models are re-downloadable). (2) 🟠 esh-vm-db (a DB VM: postgres+mongo) + vm-esh-nas IF PBS-uncovered (verify the esh-pve-nas gap). (3) 🟡 ana-docker / esh-docker-vm — VMs, PBS image-CURRENT today, only the granular restic layer is stale (full-VM restore available; no recent file/DB-granular point). RECOVERY (needs root on ana-docker; the session there = lkraven, NO NOPASSWD sudo → operator !-sudo or bootstrap infra-ops there): sudo mount -a → sudo docker restart rest-server (recreate if the bind doesn't re-resolve onto the NFS; if the ghost persists: stop container → umount → rm the ghost repo/ana → remount → start) → verify → next 01:00 succeeds. PREVENTION (the systemic fixes): fstab _netdev+retry so it self-heals on reboot; a backup-freshness alert (the real miss — 6.5 wks unnoticed). Recovery steps mirror docs/runbooks/disaster-recovery.md Tier-0. ⚠️ Disclosed creds: diagnosing dumped the Backrest config, exposing the 5 per-repo rest-server basic-auth passwords into the session transcript → rotate after recovery (add to the disclosed-keys rotation queue). ✅ RECOVERED (2026-06-20): via ssh infra-ops@ana-docker (NOT the default lkraven session) — sudo mount -a + reset-failed mnt-backup.mount re-attached the NFS, then docker compose up -d --force-recreate rebound rest-server onto the real /data (now Loaded htpasswd file, 401, healthy). Verified end-to-end: triggered backups on the two no-PBS hosts' priority — ana-ml2 snapshot landed 22:53Z (total 19→20); ana-docker's larger catch-up was still running at hand-off (will land). esh-docker-vm/esh-vm-db/vm-esh-nas auto-catch-up at their next 01:00 timers (or trigger manually). Full topology + 2-min freshness-check now documented in docs/runbooks/backups.md. PREVENTION DONE (2026-06-20): (1) freshness alert — scripts/check-backup-freshness.sh + daily systemd user timer on nh3-dev (08:00) → althing alert to infra-ops on any stale/down layer (proven end-to-end); (2) fstab hardened — ana-docker /mnt/backup → noauto,x-systemd.automount,… (autofs self-heals, active next boot; .bak-pre-harden saved); (3) esh-pve-nas VERIFIED not PBS-covered → esh-vm-db (a DB!) + vm-esh-nas are restic-only. /mnt/compose ALSO hardened + automount activated live (2026-06-20, binds no container — proved the autofs pattern works on ana-docker). STILL OPEN: rotate the 5 rest-server creds (operator, offline); the 3 esh + ana-docker restic clients clear at tonight's 01:00.
ana-docker disk incident RESOLVED — root cause was a 94 GB unrotated langfuse-clickhouse log, NOT (just) image bloat. ClickHouse logs verbosely to stdout + there was NO log rotation anywhere (no per-stack logging:, no daemon default) → its json.log hit ~94 GB and filled / (100%, 28/48 containers unhealthy on runc /tmp ENOSPC). Fixed: truncated the log, pruned ~101 GB of images (the gitea-runner accumulated 18 worldtree build versions), retired Langfuse. Disk → ~51%. Installed a daily image+buildx prune cron (lkraven crontab on ana-docker). DEFERRED to clean context: a docker-daemon default log cap (daemon.json log-opts + docker restart, needs root → infra-ops) — the real systemic fix so no container can ever spew unbounded again. (Note: named buildx-builder state volumes aren't cleared by docker image/builder prune — orphaned ones need docker volume rm.)
Langfuse RETIRED (2026-06-20, operator). Redundant with LiteLLM's native logging (store_prompts_in_spend_logs: true → full prompts/ responses/tokens/cost at :4000/ui); nothing used its trace-grouping/eval features; and it's a 6-container crash-prone surface (the 94 GB spew). Dropped the success_callback/failure_callback: [langfuse] from the litellm config (gateway observability stays native) and compose down -v the stack. (f8eda1c)
OmniVoice gained a streaming /tts + language-safe sanitizer (irv-ml1 :8199, stacks/omnivoice/), driven by a VENDORED copy of chatterbox-fast's buffer-ratchet scheduler. KEY FINDING: OmniVoice is DIFFUSION → a ~fixed per-call overhead sets a TTFA FLOOR (~0.7 s @16 steps), and measured RTF is ~2.8×/5.6× on the 3090 (NOT the upstream-claimed 40×). Tuned: streaming num_step=16/batch 32, rtf_prior=20. chatterbox-fast (autoregressive, ~0.5 s TTFA) stays the lower-latency front-end — OmniVoice is the multilingual/voice-design complement, NOT a replacement. (288d085, cd92b85; auto-memory reference_omnivoice_streaming_tts)
arbo v0.14.9 deployed (muse prompt-enhance → summarizer-large, was granite which mode-collapsed at temp 1.0) via the canonical health-gated scripts/deploy-engine.sh on irv-ml1. arbo's gitea Actions CI is now LIVE (runner-label mismatch fixed by comfy-dev) → future src pushes auto-deploy; manual deploy only when CI is down. (auto-memory reference_arbo_engine_deploy)
ratatoskr :8392 (composite Bifrost provider) added to the personal WT BIFROST_CLIENT_ALLOWED_HOSTS (footgun-safe recreate: inline sha pin + --pull never, the .env still pins stale :latest). And ratatoskr's admin.events.read key (ratatoskr#11) BROKERED with worldtree-dev — the scope is tier-derived (not mint-settable), so worldtree-dev defines the isolated admin-tier user/scope, then I mint + secure-deliver. NON-URGENT.
claude-bot granted ADMIN on vh/Worldtree (2026-06-20, via operator vh-admin → migration-enabler) so I can self-serve WT deploy re-runs + provision worldtree-dev's own Actions token (pending their gitea username).
arbo + brokkr litellm keys = all-proxy-models (verified — full gateway access incl. paid GLM, no key change needed). all-agents-local shared key stays fenced to free local models only. claude-bot Gitea service account is the credential-migration foundation (admin on vh/arbo + now vh/Worldtree).
Worldtree config-propagation: demo+personal bind-mount their model_roles.yaml/providers.yaml (/opt/worldtree{,-personal}/config, infra-ops-deployable); pinned is IN-IMAGE (worldtree-dev CI bakes it). worldtree-dev authors the canonical config; infra-ops deploys to the bind-mounted instances byte-identical.
Still open from prior: Mac Pro migration (migration-plan.md, workstation-only, hardware-gated); R17 v2 corpus push HELD (local-only soundgasm-derived, distribution barred); disclosed-keys hygiene rotation queue; clean legacy news-digest on ana-docker.

Recent decisions

[2026-06-20] claude-bot issue-scope token minted for worldtree-dev self-serve (closes their last tea-as-vh fallback — issues; they already self-serve Actions + deploys). Gitea PAT scopes are IMMUTABLE → minted a NEW claude-bot PAT worldtree-dev-actions-issues-20260621 (id 16) with write:repository+write:issue via basic-auth (claude-bot:gitea-password, POST /users/claude-bot/tokens); dropped mode-600 ~/.claude-bot-token-issues → worldtree-dev swaps. Gitea COLLAPSES read:issue into write:issue (write implies read) so the granted set shows write-only but read IS satisfied (GET issues → 200). Revoke the old write:repository-only token once they confirm swap. Advances the credential-migration directive. ALSO this session: re-staged cb2a79a policies.yaml (readonly-admin allow rules) to demo+personal (see Current-state ratatoskr entry).
[2026-06-20] rest-server-ana recovered + backup prevention shipped + worldtree-dev admin keys provisioned (operator-directed). rest-server-ana fixed (mount-a + force-recreate via infra-ops@ana-docker); freshness alert + fstab hardening landed (see Current-state BACKUP entry + docs/runbooks/backups.md). Minted worldtree-dev admin-tier Heimdall keys on demo (key_id d113207c) + personal (f4f75adb) via POST /admin/keys {tier:admin} + POST /admin/users/worldtree-dev/tier, verified read+write, dropped mode-600 to ~/.wt-admin-{demo,personal} on nh3-dev → worldtree-dev now self-serves key minting (cuts infra-ops out of the per-key loop, like the claude-bot gitea token). Cred rotation (5 rest-server pw) BELAYED per operator.
[2026-06-20] Worldtree capability-gateway aliases stood up; NO wt- prefix (operator) — reuse the existing capability convention. gen/ gen-reasoning/classifier/summarizer/chat-judge/reranker/scalar-judge (passthrough)/gen-frontier/-reasoning. The swap-transparency rule: embedding stays model-named (qwen3-embedding) because vectors aren't swap-transparent; everything else can swap behind a stable alias. (d372103, 7819f96)
[2026-06-20] Removed the litellm * wildcard → llama-swap (operator; llama-swap decommissioned, :9292 confirmed down). Unknown/typo'd models now 404 cleanly instead of a misleading "Connection error" — closes the long-standing footgun. (d0eb09c)
[2026-06-20] Langfuse retired (operator) — redundant with LiteLLM native spend_logs + a 6-container crash-prone surface (the 94 GB clickhouse spew). Callbacks dropped, stack torn down. (f8eda1c)
[2026-06-20] OmniVoice streaming /tts — vendor chatterbox-fast's scheduler, not its whole package (operator-approved vendor-copy). Diffusion TTFA floor finding → tuned 16-step streaming + aggressive packing; chatterbox- fast remains the lowest-latency front-end. (288d085, cd92b85)
[2026-06-20] claude-bot → ADMIN on vh/Worldtree (operator-authorized; one-time use of operator vh-admin to enable the migration) — claude-bot self-serves WT deploys/tokens henceforth, reducing personal-cred fallbacks.
[2026-06-20] Daily image+buildx prune cron on ana-docker (the missing automation behind the recurring image bloat) — lkraven crontab, prune unused images >48 h + per-builder buildx cache. The real systemic fix (daemon log cap) is deferred to clean context.
[2026-06-19] litellm task-aliases: classifier→granite-4.1-8b, summarizer-large→gen/qwopus (summarizer→granite already existed). Also added then REMOVED gen-nt/gen-reasoning-nt (redundant with the global strip_empty_tools hook). (ef45f6d, 2e5ab72, 75bd4c3)
[2026-06-19] gateway-chat: model-smoking web chat → PERSISTENT container. /v1/models auto-discovery + image-upload; stacks/gateway-chat/ (nginx:alpine, ana-docker:8091, homepage tile). (4c40b9f, 740bcae)
[2026-06-19] pi on nh3-dev wired to gen — ~/.pi/agent/models.json + a gen launcher mirroring glm. Local box config, not a repo artifact. (untracked by operator choice)
[2026-06-19] gen model → Qwopus3.5-122B-A10B (vision-intact NVFP4), full 256K @ fp8. ana-ml2 GPU 0, served-name qwen3.5-122-a10b. STABLE vLLM + fp8 KV reaches 262144 — nightly+turboquant-4bit-KV was UNNECESSARY. 92.7 tok/s warm. (20e796c, 5b06514)
[2026-06-19] TP=2 across the two ana-ml2 Blackwells REJECTED — PIX (PCIe, NO NVLink) → all-reduce-bound → SLOWER. One-model-per-card is optimal; PP=2 only if a >96 GB model is ever wanted. (untracked by operator choice)
[2026-06-19] qwen-image-bench (T2I judge, NVFP4) replaced qwen3.6-35b-a3b on GPU 1, aliased image-judge. (bfae924, 5dfce04)
[2026-06-18] heretic abliterated Mistral Small 4 NVFP4 built + LIVE as mistral-small-4 (in-house NVFP4, drop-in stack, A/B'd vs official, "heretic stays"). DISPLACED from GPU 0 on 2026-06-19 for the gen/Qwopus swap. (dd3a5c9, f566f61, tools/mistral-small4-nvfp4/)
[2026-06-18] irv-ml1 VRAM consolidation + comfy-dev torch-pin — ComfyUI pinned to the A6000 (48 GB), audio zoo → 3090; DISABLE_UPGRADES=true @ torch 2.12.1. (a8550ad)
[2026-06-18] ComfyUI acceleration set (9 nodes) installed for comfy-dev on irv-ml1.
[2026-06-17] ana-ml2 vm.overcommit_memory=1 made durable (sysctl drop-in) — overcommit=0 + zero swap caps CommitLimit; large model-file mmap ENOMEM'd despite ~393 GB free. (fc88eff)
[2026-06-17] GLM 5.2 wired into litellm (glm-5.2 + -reasoning, z.ai passthrough, extra_body.thinking.type toggle). (fe77a35)
[2026-06-17] nh3-extdev stood up as a sudo-LESS infra-ops manager box (successor to nh3-ansible); hosts /opt/externs pi-on-GLM-5.2 client workspaces. (a841eab)
[2026-06-14] STANDING DIRECTIVE: migrate ALL infra access to Claude-specific credentials. Stand up service accounts (claude-bot Gitea user + scoped tokens, distinct litellm admin key); re-mint consumer creds under them; flag personal-cred fallbacks until done. (auto-memory project_migrate_infra_access_to_claude_credentials)

101 older entries archived to archival-memory.md.

Tried and abandoned

[2026-06-20] A full disk wedges gitea's Actions notifier SILENTLY — and it STAYS wedged after disk recovers; only a docker restart gitea clears it. After the 94 GB clickhouse disk-full event, gitea stopped CREATING workflow runs on push (push succeeds, ref lands, but NO run spawns — newest run stale, queue empty), while the act_runner stayed registered. NOT run-pickup, NOT disk-now (52% free), NO gitea errors/panics — a silent notifier stall. Hit BOTH vh/arbo (comfy-dev) and vh/Worldtree (worldtree-dev's deploys) → it's global, not repo-specific. Fix: docker restart gitea on ana-docker (it had been up 2 weeks, never bounced). A restart does NOT retroactively fire pushes made during the outage — re-push/re-trigger after. Reinforces the deferred docker-daemon-default log-cap fix (the disk-full that started it can't be allowed to recur).
[2026-06-20] Mistral-Medium-3.5-128B NVFP4 (RecViking) on ana-ml2 GPU 0 — ABANDONED, too slow (operator confirmed). Stood up as a temporary speed-check displacing mistral-small-4; measured base ~19.2 tok/s decode on one Blackwell (vs gen/Qwopus3.5-122B ~92.7 tok/s warm) → confirmed the operator's "too slow" suspicion. KEEP these serving findings: (1) vLLM nightly loads the HF-format NVFP4 weights DIRECTLY — no Mistral native-convert (unlike Small 4, same mistral3 arch) — via the FlashInfer Cutlass NVFP4 kernel + turboquant_4bit_nc KV; TP=1 fits the ~70 GB NVFP4 on one 96 GB Blackwell. (2) EAGLE was a dead end: the official native FP8 head (mistralai/Mistral-Medium-3.5-128B-EAGLE) crashes in CUDA-graph replay with graphs on, and nets NO speedup with --enforce-eager (~18.5 vs ~19.2 base) — likely low draft acceptance against the NVFP4 (non-official-FP8) base. Untracked workspace files (stacks/mistral-medium-3.5/ + playbooks/serve-mistral-medium-3.5.yaml) removed. ⚠️ Weights still staged on ana-ml2 /tank/aimodels/mistral-medium-3.5-{nvfp4,eagle} (~70 GB+) — pending operator OK to free.
[2026-06-20] A 94 GB unrotated langfuse-clickhouse json.log was the REAL disk-filler (not image bloat) — there was NO log rotation anywhere. No per-stack logging: limit and no docker-daemon default → ClickHouse's verbose stdout grew unbounded to ~94 GB and filled ana-docker's / (healthchecks then failed fleet-wide on runc /tmp: no space left). Lesson: on a full disk, the symptom (all-unhealthy) is the FORK-can't-write healthcheck, not the containers; find the runaway via per-container-dir du, not docker system df. The systemic fix is a daemon-default log-opts (max-size/max-file) — NOT just per-service patching.
[2026-06-20] OmniVoice's upstream "RTF 0.025 / 40×" does NOT hold — diffusion has a ~fixed per-call overhead → TTFA floor + multi-chunk streaming is a net loss. Measured ~2.8× (32-step) / 5.6× (16-step) on the 3090; every chunk costs ~the same regardless of length, so naive per-sentence streaming triples total gen time + starves. Use few/aggressive chunks; chatterbox-fast (autoregressive) stays the low-TTFA front-end.
[2026-06-20] rest-server .htpasswd: permission denied = the ana-nas NFS mount FAILED (ghost file on the local mount point), NOT a decommission. mnt-backup.mount stuck failed (fstab bare defaults, no retry) → the rest-server serves an empty local dir with a root:root 0-byte .htpasswd. Documented recovery in disaster-recovery.md. Don't tear down what looks like a crash-looping legacy container until you've checked fstab + the docs — it was the live ana-side restic target.
[2026-06-20] Named buildx builders accumulate cache the standard prunes MISS. docker image prune -a + docker builder prune don't touch buildx_buildkit_<name>-builder0_state volumes (per-builder buildkit state). Prune them with docker buildx prune --builder <name> -af, or docker volume rm orphaned ones (builder already removed).
[2026-06-20] The DEFAULT ssh ana-docker is lkraven (no NOPASSWD) — but ssh infra-ops@ana-docker HAS NOPASSWD root (corrected later same day). Early on a sudo cp as lkraven silently failed (password prompt) → one wasted gateway bounce; I then wrongly concluded "no sudo on ana-docker" and nearly punted the rest-server-ana recovery to the operator. The real rule: reach for infra-ops@ana-docker for sudo ops; lkraven-owned files (litellm config, most stack compose/conf) take plain cp under either identity.
[2026-06-19] litellm CONFIG-loaded models can't be hot-removed — /model/delete is DB-only. Only a process restart drops a config model. The INVERSE works: /model/new live-ADDS (no bounce) but dups with config on restart. Clean no-bounce-add: live-add via API, delete the DB copy before the eventual bounce.
[2026-06-19] the litellm * wildcard routed to DECOMMISSIONED llama-swap → a stale/typo'd model name returned a misleading Connection error, not model not found. Bit a brokkr call to the renamed qwen-image-judge. → RESOLVED 2026-06-20: wildcard removed (d0eb09c) — unknown models now 404 cleanly.
[2026-06-19] A MoE card can't be filled to 0 bytes free — the FusedMoE transient workspace (~3.09 GB) is the floor, allocated OUTSIDE gpu-memory-utilization. FIX: PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True + leave ~3.2 GB free (util ≤ ~0.95 on a tight MoE).
[2026-06-19] vLLM decode tok/s: ALWAYS discard the first generation (cold warmup). Cold ≈ 24.8 tok/s vs warm 92.7 — a cold read undersells decode ~3–4×. Measure run 2+ over a ≥256-token output.
[2026-06-19] For full native 256K on one 96 GB card, nightly+turboquant-4bit-KV was unnecessary for the Qwopus MoE. Stable fp8 KV already fits 262144 (hybrid SSM+attn caches KV only on attn layers → small pool). Reach for fp8 first.
[2026-06-18] mmartial comfyui-nvidia-docker: root pip installs CRASH-LOOP the container. docker exec -u 0 pip install leaves root-owned files in the uid-1000 venv → the boot script (uid 1000) fails its torch step → crash loop. FIX: chown -R 1000:1000 /comfy/mnt/venv; pin with DISABLE_UPGRADES=true. (auto-memory reference_irv_ml1_comfyui_mmartial)
[2026-06-18] ComfyUI custom nodes break on version-assumption drift — box runs 0.24.1; TeaCache imports precompute_freqs_cis at MODULE level (guard the LTX-only import). pip install tensorrt defaults to cu13 → use tensorrt-cu12.
[2026-06-17] Mistral HF→NVFP4 quant: the placement knob is the whole game. device_map="cpu" (CPU-resident, sequential onload) works; "auto" OOMs, max_memory offloads to meta. Read shards with plain read() + safetensors.torch.load(bytes), NOT safe_open (mmaps the 50 GB shard → ENOMEM on /tank ZFS).
[2026-06-17] HF-format Mistral Small 4 is UNSERVEABLE on vLLM — no HF Mistral4 backbone; serves ONLY via the native loader. A HF-format quant MUST be converted to native first.

84 older entries archived to archival-memory.md.

32 KiB Raw Blame History Unescape Escape