vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity
Files
13bfa4a621d5004af977345cb48be86822428cd5
esh-pfi-infrastructure/persistent-memory.md
T

20 KiB
Raw Blame History

Persistent memory — eshpfi-management

Last updated: 2026-06-21

Repo purpose

Reference workspace for PFI infrastructure: server inventory, canonical Docker Compose stacks, ops playbooks, and conventions. Authoritative copies of compose files live on the servers under /opt/docker/compose/<stack>/; this repo mirrors them for version control, editing, planning, and CI-driven deploys. It was originally spun up to handle the fleet backups — keep that lens when triaging backup/storage issues.

Tools and conventions

Sister repos (separate gitea repos, deployed by playbooks here):

Repo Role CI status
vh/task-board MCP + web dashboard for assistant task state (port 7878) push-to-main → CI deploys (2026-04-29)
vh/vor Inquisitor UI sidecar (port 7879) push-to-main → CI deploys (2026-04-29)
vh/nevermore Twice-daily LLM-curated briefing (port 8181, replaces news-digest) push-to-main → CI deploys (2026-04-30)
vh/asset-engine Internal control plane over inference services (port 8200, LAN-direct) push-to-main → CI deploys (2026-05-12)
vh/althing Inter-agent message bus (chamber UI port 7881, forseti + agent-runner daemons, valkey IPC) push-to-main → CI deploys (2026-05-14)
vh/mead-hall Bifrost tool-provider sidecar (port 5173 on dev VM 10.100.10.50) push-to-main → CI deploys (2026-05-16)
vh/skaldsong Wizard + reader surface (port 8300, ana-docker, registry-pull pattern) push-to-main → CI deploys (2026-05-19)
vh/Worldtree Conversation API (corviduo-dev demo :8080 / personal :8081 / pinned :8082) — Heimdall auth, Bifrost integration. gitea-runner builds on ana-docker (the image-bloat source); claude-bot now ADMIN collaborator (2026-06-20) push-to-main → CI build-and-deploy (runner on ana-docker)
vh/yt-voice-clipper YouTube → diarized voice-clip dataset builder + audition console (irv-ml1 :8000) push-to-main → gitea-webhook auto-deploy to irv-ml1 (2026-06-03) — see docs/runbooks/ytvc-autodeploy.md
vh/arbo Catalog-driven ComfyUI engine (irv-ml1 :8201, comfy-dev owns engine/catalog/image) push-to-main → gitea Actions CI (deploy-engine.sh, build-local, health-gated) now LIVE; catalog via :9009 webhook

(vh/volva + Heid were re-architected from systemd daemons to Claude Code session orchestrators 2026-06-08; their nh3-dev .service units were removed — no longer deployed sidecars here. See Recent decisions.)

  • Two-layer backups — Backrest orchestrates restic for file+DB (5 fleet repos, daily 01:00 PDT); PBS-ANA primary + PBS-NH3 DR mirror for VM images. ana-nas is the SPOF for postgres + PBS-ANA datastore + cross-site restic targets — see docs/runbooks/disaster-recovery.md for the blast-radius matrix. ⚠️ The restic file+DB layer routes through TWO rest-servers (rest-server-ana @ ana-docker:8000 → ana-docker/ana-ml2/esh-docker-vm/vm-esh-nas; rest-server-nh3 @ nh3-nas:8000 → irv-ml1/nh3-docker). Both depend on their NAS's NFS export of /mnt/backup. As of 2026-06-20 rest-server-ana is DOWN on a failed ana-nas NFS mount — see Current state.

  • pull-hf-repo.yaml is the canonical "get a HuggingFace model/dataset onto ana-ml2's shared cache at /tank/aimodels/huggingface/" playbook. Supports --var repo_type=model|dataset|space. Replaces ad-hoc huggingface_hub.snapshot_download patterns.

  • Worldtree admin auth — per-instance. Each Worldtree deployment (demo :8080, personal :8081, pinned :8082) has its own Heimdall registry and its own bootstrap admin key. Infra-ops's stored long-lived admin key (key_id 61419c92) at ana-docker:/opt/docker/conf/.secrets/worldtree-infra-ops-admin auths against demo only. Personal-instance admin (the ~/.config/worldtree/personal-admin-token, mode 600) POSTs /admin/keys (mints per-project keys; takes user_id+label, no scope param — scopes are tier-derived, so admin-tier scopes like admin.events.read must be granted WT-side by worldtree-dev).

  • Per-project user keys against personal Worldtree (issued 2026-05-19): skaldsong:79744637 (nh3-dev iteration), skaldsong:7c1dbbbe (ana-docker prod), althing:50d85460, mead-hall:a360822d. Same user_id=skaldsong across both skaldsong keys → shared Heimdall agent slot; different key_id → independently rotatable. Pattern: mint via /admin/keys, drop value to /tmp/wt-personal-<name>.key mode 600, dev collects + shreds (DO NOT cat to chat transcript).

  • Skaldsong CD pattern (registry-pull). Differs from althing / asset-engine which build-on-host. vh/skaldsong's CI builds and pushes gitea.phasefinal.com/vh/skaldsong:<sha> + :latest; playbooks/deploy-skaldsong.yaml on ana-docker pulls + recreates. SHA-pin only (no :latest health-gated advance yet). Prereq: host needs docker login gitea.phasefinal.com once (read:package PAT) — not currently in the workflow.

  • gitea internal route for fleet hosts. gitea is a container on ana-docker — git-SSH 10.250.50.70:222, HTTP :3000. Fleet/colo hosts must use this internal route, NOT public gitea.phasefinal.com (38.120.12.44, ana-srv1) — the public path fail2bans the host egress IP and wedges webhook deploys. :22 on 10.250.50.70 is ana-docker's HOST sshd, not gitea. Full gotcha in docs/orientation.md → Git/gitea.

  • docker-as-root pattern (for ops with no admin API, or to edit deploy-owned/root-owned files without sudo): docker run --rm -v <target-dir>:/wt [-v /var/run/docker.sock:/var/run/docker.sock] docker:cli sh -c "..." (or alpine for plain file ops). docker-group membership is effectively root via bind-mount; treat as sudo-equivalent. Foot-gun: relative paths in compose.yaml resolve against the sandbox CWD but the Docker daemon interprets them against the HOST fs — always pass -e VAR=/abs/path for any relative-default config dir.

  • scripts/elway sudo handling — elway prompts for the sudo password ONCE via getpass before the first sudo: true step → can't run unattended from a non-TTY tool if any step needs sudo. Sudo-free playbooks run fully non-interactive over key SSH.

  • Per-host SSH identity matters for sudo. infra-ops has NOPASSWD sudo on most PFI Linux boxes (corviduo-dev included since 2026-06-15). On ana-docker there are TWO identities: the default ssh ana-docker = lkraven (docker-group, NO passwordless sudo — docker works, root-file edits don't); but ssh infra-ops@ana-docker HAS NOPASSWD root (verified 2026-06-20). → For any sudo op on ana-docker (mount, root-owned files, service control), use ssh infra-ops@ana-docker, NOT the default session. lkraven-owned files (litellm config, most stack compose/conf) still take plain cp/edit under either identity.

Current state / in-flight

As of 2026-06-21:

  • Backups — RECOVERED + hardened (2026-06-20). rest-server-ana is back (the ~6.5-wk silent ana-side restic outage is closed); ana-ml2 + ana-docker re-snapshotted; daily freshness alert live (nh3-dev systemd timer → scripts/check-backup-freshness.sh → althing on any stale/down layer); both ana-docker NFS mounts moved to self-healing automount. Topology + 2-min check: docs/runbooks/backups.md. STILL OPEN: rotate the 5 disclosed rest-server creds (operator, offline); confirm esh-vm-db's resticprofile includes DB dumps (esh-pve-nas has NO PBS namespace → it is restic-only). Detail in Recent decisions.

  • Worldtree — demo + personal GREEN (v0.37.7/.9). Capability-gateway migration live; ratatoskr readonly-admin tier staged (cb2a79a). worldtree-dev fully self-serve: admin keys (demo d113207c / personal f4f75adb) + one consolidated claude-bot Gitea token (Actions+deploys+issues). Nothing pending. Detail in Recent decisions.

  • R22 (brokkr/dwarves) — CONCLUDED, gateway-only. Phase B (internal-model hooks) CANCELLED (Worldtree model-agnostic → no deploy path). R22 whole footprint = gateway calls to free qwen3.5-122-a10b via a PERSISTENT full-access key at /home/lkraven/.r22-gateway-key (mode 600, carries paid GLM, do NOT delete). Detail in Recent decisions.

  • Open follow-ups (low-priority): clean phantom qwen3.6-35b-a3b off the gateway (lists in /v1/models but 400s — backend displaced); docker-daemon default log-cap on ana-docker (root → infra-ops; systemic fix behind the 94 GB disk incident).

  • Standing / parked (from prior): Mac Pro migration (migration-plan.md, hardware-gated); R17 v2 corpus push HELD (distribution barred); disclosed-keys rotation queue; clean legacy news-digest on ana-docker.

  • Worldtree config-propagation (reference): demo+personal bind-mount providers.yaml/model_roles.yaml/policies.yaml from /opt/worldtree{,-personal}/config (infra-ops-deployable, byte-identical from worldtree-dev canonical sha); pinned is IN-IMAGE. Reload via docker restart <container>, NEVER compose up (stale-:latest footgun); WT PDP is rule-based (a tier needs allow RULES, not just scopes).

Recent decisions

  • [2026-06-20] R22 (brokkr/eitri/dwarves) stood down to gateway; full-access R22 key minted; Phase B parked. Operator caught the raw-weights/GPU ask as premature — Phase A (P00 + NL-rendering) is PROMPT-LEVEL on the LiteLLM gateway, not raw weights (no GPU, no Blackwell displacement). Real blocker was gateway ACCESS: the shared all-agents-local key only serves granite chat, and granite-8b is too weak to be the P00 model-under-test (capability-confound). Operator said "full access" → minted r22-brokkr-phaseA LiteLLM key (scope all-proxy-models, incl PAID GLM), dropped ~/.r22-gateway-key (mode 600) for eitri — re-minted PERSISTENT after eitri's stateless tool-env couldn't hold it in env + shredded the one-shot drop (old key revoked); the live full-access key now lives at /home/lkraven/.r22-gateway-key, read per-command, don't delete (cost surface — carries paid GLM; operator reconfirmed full-open + declined to narrow, 2026-06-20 — settled, don't re-surface). Lesson: stateless consumers need a persistent mode-600 file read per-call, NOT shred-after-load. P00 MUT = qwen3.5-122-a10b (alias gen) — 122B, capable, FREE-LOCAL (no per-token cost); told them keep frozen calibration on it, NOT the paid GLM the key can now reach. PHASE B (raw-weights hooks: Dvalin activation-steering on hidden states + Regin ECM logit-injection on a 7-8B Qwen/Llama via transformers LogitsProcessor) CANCELLED 2026-06-20 (not deferred) — operator cut the internals arms: Worldtree is model-agnostic (regard/ affect renders via context manipulation = prompt-level only), so the internal-model approach has no deploy path (the pragmatic-outcome gate firing). R22's ENTIRE compute is now the gateway (free qwen3.5-122-a10b MUT); NO raw-weights/GPU ever; the A6000 slice reservation is released (was never provisioned). The full-scope R22 key stays (pinned to the free qwen-122, GLM untouched). FOLLOW-UP: clean phantom qwen3.6-35b-a3b off the gateway (shows in /v1/models but 400s "invalid model" — backend displaced 2026-06-19; the no-false-alias footgun). OPERATOR STEER (2026-06-20): R22 is research for a PRAGMATIC/deployable outcome, NOT advancing-the-art ("another day's work"). Before the dwarves invest in Phase B, establish a deploy path (distilled steering-vector / fine-tune applied to a model we already serve, e.g. Worldtree's) — don't research an avenue we won't implement. Phase A/P00 (the cheap prompt-level gate) runs regardless (it just tells us if the affect signal exists). Operator is taking the dwarf discussion directly.

  • [2026-06-20] claude-bot issue-scope token minted for worldtree-dev self-serve (closes their last tea-as-vh fallback — issues; they already self-serve Actions + deploys). Gitea PAT scopes are IMMUTABLE → minted a NEW claude-bot PAT worldtree-dev-actions-issues-20260621 (id 16) with write:repository+write:issue via basic-auth (claude-bot:gitea-password, POST /users/claude-bot/tokens); dropped mode-600 ~/.claude-bot-token-issues → worldtree-dev swaps. Gitea COLLAPSES read:issue into write:issue (write implies read) so the granted set shows write-only but read IS satisfied (GET issues → 200). Old token (id 15, worldtree-dev-actions-20260620) REVOKED (DELETE 204) after worldtree-dev verified read+write end-to-end via core.issues (show/list/comment on #311) → claude-bot now carries ONE consolidated worldtree-dev token (id 16) + arbo-ci (id 14), no stale dups. Advances the credential-migration directive (worldtree-dev fully self-serve: Actions + deploys + issues on one token). ALSO this session: re-staged cb2a79a policies.yaml (readonly-admin allow rules) to demo+personal (see Current-state ratatoskr entry).

  • [2026-06-20] rest-server-ana recovered + backup prevention shipped + worldtree-dev admin keys provisioned (operator-directed). rest-server-ana fixed (mount-a + force-recreate via infra-ops@ana-docker); freshness alert + fstab hardening landed (see Current-state BACKUP entry + docs/runbooks/backups.md). Minted worldtree-dev admin-tier Heimdall keys on demo (key_id d113207c) + personal (f4f75adb) via POST /admin/keys {tier:admin} + POST /admin/users/worldtree-dev/tier, verified read+write, dropped mode-600 to ~/.wt-admin-{demo,personal} on nh3-dev → worldtree-dev now self-serves key minting (cuts infra-ops out of the per-key loop, like the claude-bot gitea token). Cred rotation (5 rest-server pw) BELAYED per operator.

  • [2026-06-20] Worldtree capability-gateway aliases stood up; NO wt- prefix (operator) — reuse the existing capability convention. gen/ gen-reasoning/classifier/summarizer/chat-judge/reranker/scalar-judge (passthrough)/gen-frontier/-reasoning. The swap-transparency rule: embedding stays model-named (qwen3-embedding) because vectors aren't swap-transparent; everything else can swap behind a stable alias. (d372103, 7819f96)

  • [2026-06-20] Removed the litellm * wildcard → llama-swap (operator; llama-swap decommissioned, :9292 confirmed down). Unknown/typo'd models now 404 cleanly instead of a misleading "Connection error" — closes the long-standing footgun. (d0eb09c)

  • [2026-06-20] Langfuse retired (operator) — redundant with LiteLLM native spend_logs + a 6-container crash-prone surface (the 94 GB clickhouse spew). Callbacks dropped, stack torn down. (f8eda1c)

  • [2026-06-20] OmniVoice streaming /tts — vendor chatterbox-fast's scheduler, not its whole package (operator-approved vendor-copy). Diffusion TTFA floor finding → tuned 16-step streaming + aggressive packing; chatterbox- fast remains the lowest-latency front-end. (288d085, cd92b85)

  • [2026-06-20] claude-bot → ADMIN on vh/Worldtree (operator-authorized; one-time use of operator vh-admin to enable the migration) — claude-bot self-serves WT deploys/tokens henceforth, reducing personal-cred fallbacks.

  • [2026-06-20] Daily image+buildx prune cron on ana-docker (the missing automation behind the recurring image bloat) — lkraven crontab, prune unused images >48 h + per-builder buildx cache. The real systemic fix (daemon log cap) is deferred to clean context.

  • [2026-06-14] STANDING DIRECTIVE: migrate ALL infra access to Claude-specific credentials. Stand up service accounts (claude-bot Gitea user + scoped tokens, distinct litellm admin key); re-mint consumer creds under them; flag personal-cred fallbacks until done. (auto-memory project_migrate_infra_access_to_claude_credentials)

113 older entries archived to archival-memory.md.

Tried and abandoned

  • [2026-06-20] A full disk wedges gitea's Actions notifier SILENTLY — and it STAYS wedged after disk recovers; only a docker restart gitea clears it. After the 94 GB clickhouse disk-full event, gitea stopped CREATING workflow runs on push (push succeeds, ref lands, but NO run spawns — newest run stale, queue empty), while the act_runner stayed registered. NOT run-pickup, NOT disk-now (52% free), NO gitea errors/panics — a silent notifier stall. Hit BOTH vh/arbo (comfy-dev) and vh/Worldtree (worldtree-dev's deploys) → it's global, not repo-specific. Fix: docker restart gitea on ana-docker (it had been up 2 weeks, never bounced). A restart does NOT retroactively fire pushes made during the outage — re-push/re-trigger after. Reinforces the deferred docker-daemon-default log-cap fix (the disk-full that started it can't be allowed to recur).

  • [2026-06-20] Mistral-Medium-3.5-128B NVFP4 (RecViking) on ana-ml2 GPU 0 — ABANDONED, too slow (operator confirmed). Stood up as a temporary speed-check displacing mistral-small-4; measured base ~19.2 tok/s decode on one Blackwell (vs gen/Qwopus3.5-122B ~92.7 tok/s warm) → confirmed the operator's "too slow" suspicion. KEEP these serving findings: (1) vLLM nightly loads the HF-format NVFP4 weights DIRECTLY — no Mistral native-convert (unlike Small 4, same mistral3 arch) — via the FlashInfer Cutlass NVFP4 kernel + turboquant_4bit_nc KV; TP=1 fits the ~70 GB NVFP4 on one 96 GB Blackwell. (2) EAGLE was a dead end: the official native FP8 head (mistralai/Mistral-Medium-3.5-128B-EAGLE) crashes in CUDA-graph replay with graphs on, and nets NO speedup with --enforce-eager (~18.5 vs ~19.2 base) — likely low draft acceptance against the NVFP4 (non-official-FP8) base. Untracked workspace files (stacks/mistral-medium-3.5/ + playbooks/serve-mistral-medium-3.5.yaml) removed. ⚠️ Weights still staged on ana-ml2 /tank/aimodels/mistral-medium-3.5-{nvfp4,eagle} (~70 GB+) — pending operator OK to free.

  • [2026-06-20] A 94 GB unrotated langfuse-clickhouse json.log was the REAL disk-filler (not image bloat) — there was NO log rotation anywhere. No per-stack logging: limit and no docker-daemon default → ClickHouse's verbose stdout grew unbounded to ~94 GB and filled ana-docker's / (healthchecks then failed fleet-wide on runc /tmp: no space left). Lesson: on a full disk, the symptom (all-unhealthy) is the FORK-can't-write healthcheck, not the containers; find the runaway via per-container-dir du, not docker system df. The systemic fix is a daemon-default log-opts (max-size/max-file) — NOT just per-service patching.

  • [2026-06-20] OmniVoice's upstream "RTF 0.025 / 40×" does NOT hold — diffusion has a ~fixed per-call overhead → TTFA floor + multi-chunk streaming is a net loss. Measured ~2.8× (32-step) / 5.6× (16-step) on the 3090; every chunk costs ~the same regardless of length, so naive per-sentence streaming triples total gen time + starves. Use few/aggressive chunks; chatterbox-fast (autoregressive) stays the low-TTFA front-end.

  • [2026-06-20] rest-server .htpasswd: permission denied = the ana-nas NFS mount FAILED (ghost file on the local mount point), NOT a decommission. mnt-backup.mount stuck failed (fstab bare defaults, no retry) → the rest-server serves an empty local dir with a root:root 0-byte .htpasswd. Documented recovery in disaster-recovery.md. Don't tear down what looks like a crash-looping legacy container until you've checked fstab + the docs — it was the live ana-side restic target.

  • [2026-06-20] Named buildx builders accumulate cache the standard prunes MISS. docker image prune -a + docker builder prune don't touch buildx_buildkit_<name>-builder0_state volumes (per-builder buildkit state). Prune them with docker buildx prune --builder <name> -af, or docker volume rm orphaned ones (builder already removed).

  • [2026-06-20] The DEFAULT ssh ana-docker is lkraven (no NOPASSWD) — but ssh infra-ops@ana-docker HAS NOPASSWD root (corrected later same day). Early on a sudo cp as lkraven silently failed (password prompt) → one wasted gateway bounce; I then wrongly concluded "no sudo on ana-docker" and nearly punted the rest-server-ana recovery to the operator. The real rule: reach for infra-ops@ana-docker for sudo ops; lkraven-owned files (litellm config, most stack compose/conf) take plain cp under either identity.

93 older entries archived to archival-memory.md.

Reference in New Issue View Git Blame Copy Permalink