# Persistent memory — eshpfi-management _Last updated: 2026-05-17_ ## Repo purpose Reference workspace for PFI infrastructure: server inventory, canonical Docker Compose stacks, ops playbooks, and conventions. Authoritative copies of compose files live on the servers under `/opt/docker/compose//`; this repo mirrors them for version control, editing, planning, and CI-driven deploys. ## Tools and conventions Sister repos (separate gitea repos, deployed by playbooks here): | Repo | Role | CI status | |---|---|---| | `vh/task-board` | MCP + web dashboard for assistant task state (port 7878) | push-to-main → CI deploys (2026-04-29) | | `vh/vor` | Inquisitor UI sidecar (port 7879) | push-to-main → CI deploys (2026-04-29) | | `vh/nevermore` | Twice-daily LLM-curated briefing (port 8181, replaces news-digest) | push-to-main → CI deploys (2026-04-30) | | `vh/asset-engine` | Internal control plane over inference services (port 8200, LAN-direct) | push-to-main → CI deploys (2026-05-12) | | `vh/althing` | Inter-agent message bus (chamber UI port 7881, forseti + agent-runner daemons, valkey IPC) | push-to-main → CI deploys (2026-05-14) | | `vh/mead-hall` | Bifrost tool-provider sidecar (port 5173 on dev VM 10.100.10.50) | push-to-main → CI deploys (2026-05-16) | | `vh/skaldsong` | TBD — first CI workflow shipped 2026-05-17 | push-to-main → CI deploys (2026-05-17) | | `vh/worldtree` | Conversation API (corviduo-dev, port 8080) — Heimdall auth, Bifrost integration | push-to-main → CI deploys | - **Two-layer backups** — Backrest orchestrates restic for file+DB (5 fleet repos, daily 01:00 PDT); PBS-ANA primary + PBS-NH3 DR mirror for VM images. ana-nas is the SPOF for postgres + PBS-ANA datastore + cross-site restic targets — see `docs/runbooks/disaster-recovery.md` for the blast-radius matrix. - **`pull-hf-repo.yaml`** is the canonical "get a HuggingFace model/dataset onto ana-ml2's shared cache at `/tank/aimodels/huggingface/`" playbook. Supports `--var repo_type=model|dataset|space`. Replaces ad-hoc `huggingface_hub.snapshot_download` patterns. - **Worldtree admin auth** — infra-ops holds a permanent admin-tier Heimdall key issued 2026-05-17 (`key_id 61419c92`), stored at `ana-docker:/opt/docker/conf/.secrets/worldtree-infra-ops-admin` mode 600. Used for `POST /admin/keys`, admin diagnostic reads (`/admin/sessions//{bifrost,tools}`, etc.), and any other Worldtree admin work. Replaces the docker-as-root + bootstrap-admin pattern that was in use 2026-05-12 through 2026-05-16. - **docker-as-root pattern** (for ops that have no admin API, e.g. `SqliteUserStore.set_bifrost_credentials`): on hosts where the SSH user is in the `docker` group but lacks passwordless sudo, run `docker run --rm -v :/wt -v /var/run/docker.sock:/var/run/docker.sock docker:cli sh -c "..."` to edit deploy-owned files without sudo. Documented with security warning in `servers/corviduo-dev/README.md`. docker-group membership is effectively root via bind-mount; treat as a sudo-equivalent grant. ## Current state / in-flight _As of 2026-05-17:_ - **mead-hall Bifrost integration stalled mid-Worldtree.** Bifrost consumer registration done from infra side (HS256, `set_bifrost_credentials` via docker-as-root + admin key, secret-drop pattern), `BIFROST_CLIENT_ALLOWED_HOSTS` env wired through to corviduo-dev's container. But the actual tracer dispatch shows `session_not_bifrost_bound` per `/admin/sessions//bifrost` — the handshake doesn't register bfclient into `_bifrost_clients[]`. worldtree-dev investigating; not infra-side work. - **althing-chamber stack at 4 services.** chamber (port 7881) + forseti (moderator daemon) + agent-runner (Phase 2 worldtree-driver dispatcher) + valkey (Phase 3.1 pub/sub bridge for cross-process streaming). Phase 3.1 verified end-to-end via event histogram on 2026-05-17. Forseti is the canonical dev for this stack (galdrabok is on a different project). - **Worldtree admin workflow shift** (2026-05-17, per vh): infra-ops uses its own permanent admin-tier key for admin operations rather than reaching into bootstrap admin via docker-as-root. Admin ops with HTTP endpoints go over HTTPS; ops without (like `set_bifrost_credentials`) still use docker-as-root but with my own auth, not borrowed. - **Open hygiene debt** — these keys/tokens were disclosed across the althing bus and/or in CC transcripts during this session and warrant rotation at convenience: - Gitea runner registration token (`a1135753...`) — rotate via Gitea admin UI's runner-token reset. - mead-hall's Worldtree bearer (`wt_live_80e1570620ef2aba998dc63954cce3a6`) — tracer issuance 2026-05-16. - Worldtree provider keys `Z_AI_API_KEY` + `ZAI_API_KEY` — from the 2026-05-12 corviduo-dev outage. - chamber config.yaml's `forseti.api_key` + `agent_runner.api_key` (Worldtree user-tier keys) — exposed when grepping config to add `streaming_enabled` 2026-05-17. - **Still open from prior session** (2026-05-08, unchanged): rotate `MINIFLUX_PASSWORD` (leaked twice); clean up legacy `news-digest` detritus on ana-docker (`/opt/docker/compose/news-digest/`, `/opt/docker/data/news-digest/`, image `local/news-digest:v5`); watch nh3-nas `/volume1` (was 65%; recheck retention or expand before ~80%); the `docker push 60s client-side ceiling` mystery remains uninstrumented. ## Recent decisions - `[2026-05-17]` Phase 3.1 cross-process streaming uses Valkey 8 alpine as a sibling compose service in `stacks/althing-chamber/`, redis-protocol pub/sub for high-volume `msg_delta` / `msg_thinking` / `msg_start` / `msg_complete` event kinds. DB bridge keeps `msg_curated` + `floor_grant` (structured / canonical). Two-channel architecture, no overlap. chamber + agent-runner `depends_on: valkey: service_healthy`. - `[2026-05-17]` Worldtree admin workflow shift (per vh): infra-ops gets its own permanent admin-tier key (`61419c92`, stored at `ana-docker:/opt/docker/conf/.secrets/worldtree-infra-ops-admin`). Future admin ops route through this key, not the bootstrap admin via docker-as-root. - `[2026-05-17]` Worldtree env-var addition checklist: anytime introducing `os.environ.get("FOO")` in worldtree code, update BOTH `.env.example` AND `compose.yaml`'s `&worldtree-env` anchor in the same PR. Same Z_AI_API_KEY-shape footgun bit `BIFROST_CLIENT_ALLOWED_HOSTS` (#170) until worldtree-dev added the passthrough line in `08f02b2`. - `[2026-05-16]` althing-chamber Phase 2: added `althing-agent-runner` as third compose service (worldtree-driver agent dispatcher). All three althing services use the same image; `command:` selects entrypoint. Safe to enable preemptively (sleeps when no driver=worldtree handles declared). - `[2026-05-14]` althing-chamber stack scaffolded: chamber + forseti. Internal LAN-only at port 7881 (chamber default 7878 collides with task-board). Two-service compose, shared SQLite bind-mount, build-on-host pattern via vh/althing's gitea-workflow. Forseti is the canonical dev for this stack (galdrabok is on a different project). - `[2026-05-13]` `vllm-qwen3` → `vllm` stack rename. Added `vllm-reward` service (Skywork-Reward-V2-Llama-3.1-8B-AWQ classifier). Three vLLM services share GPU 1 (embed 0.20, rerank 0.20, reward 0.30 utilization; 30% headroom). All use `--runner pooling`; classification drives via model's `architectures: [LlamaForSequenceClassification]` in config.json, NOT `--task classify` (deprecated in vLLM 0.19.1). - `[2026-05-13]` `pull-hf-repo.yaml` is the canonical HF-fetch playbook on ana-ml2. Supports `--var repo_type=model|dataset|space`. Replaces ad-hoc `huggingface_hub.snapshot_download` calls. - `[2026-05-13]` Selene-1-Mini-Llama-3.1-8B added to llama-swap as judge model. mradermacher `i1-Q6_K` imatrix quant (~6.5GB). AtlaAI reward/eval model — temp 0.01, ctx 32K, q8_0 KV cache. New JUDGE / EVAL MODELS section in `stacks/llama-swap/conf/config.yaml`. - `[2026-05-13]` /tend-docs first pass deletions: `stacks/infinity/` removed (retired by vllm). Archived `docs/asset-engine/design-brief.md` → `docs/archive/asset-engine/` with archival header. Fixed `pfi-pve` VM list to full `qm list` enumeration. Dropped stale weak-password section from `pfi-postgres` (rotation done 2026-04-23). - `[2026-05-12]` corviduo-dev (Worldtree-team dev VM, 10.250.50.152, CT 106 on pfi-pve) added to `servers/` inventory. Treat like SF client hosts: PFI hosts + provides emergency-ops backstop; Worldtree team owns OS config + deploys + backup decisions. - `[2026-05-12]` Worldtree `:latest` tag drift bug — fixed by health-gated `:latest` advance in vh/worldtree's deploy workflow (architect commit `8ef3801`): only tag `:latest` AFTER the new container's `/health` probe passes. Build-on-host stacks here don't have this problem because the playbook always builds the SHA-tagged image from a `git reset --hard ` checkout. - `[2026-05-12]` asset-engine stack scaffolded LAN-direct at `http://10.250.50.70:8200`. Initially included Traefik labels for public hostname; user pulled them out (internal tool, no public TLS surface needed). Pattern: internal tools default LAN-direct; Traefik wiring only when external/TLS required. - `[2026-05-12]` asset-engine catalog gains `lifecycle: { stack, vram_gb, gpu_device_id }` per irv-ml1 service for the orchestrator feature. SSH keypair scaffolded at `ana-docker:/opt/docker/conf/asset-engine/ssh/` for asset-engine container → irv-ml1 orchestration via dedicated ed25519 key. ## Tried and abandoned - `[2026-05-17]` `--task classify` for Skywork in vLLM 0.19.1 — flag was deprecated. Use `--runner pooling`; the model's `architectures: [LlamaForSequenceClassification]` in config.json drives the classification head. Surfaced as `vllm: error: unrecognized arguments: --task classify` in container logs. - `[2026-05-17]` Trusting that `.env` edit alone propagates a new env var into a worldtree container — `compose.yaml`'s `&worldtree-env` anchor must explicitly declare the passthrough or the value silently doesn't land. Same footgun bit `Z_AI_API_KEY` (2026-05-12) AND `BIFROST_CLIENT_ALLOWED_HOSTS` (2026-05-17). Cost ~10 min of "why is env empty?" diagnosis each time. Worldtree-side fix in `vh/worldtree@08f02b2`. - `[2026-05-17]` `--force-recreate --pull never` from the docker:cli sandbox without explicit `-e WORLDTREE_IMAGE=` re-pins the container to `:latest`, even when a newer SHA-tagged image is on disk. Symptom: container "recreated" but actually reverted to a stale image. Pass `-e WORLDTREE_IMAGE=...:` to the docker run invocation. Worldtree-dev's `8ef3801` health-gated `:latest` advance is the long-term fix. - `[2026-05-13]` Initial Voxtral default voice `alloy` (OpenAI-compat naming) — vLLM-Omni serving Voxtral does NOT translate aliases. Native presets are `_` shape (`neutral_female`, `casual_male`, etc.). Always live-probe `/v1/audio/voices` for the exact wrapper-deployed preset names before setting a catalog default. Same caveat for Qwen3-TTS (wrapper exposes 15 voices: 9 Qwen presets + 6 OpenAI aliases) and Kyutai-TTS (NillPointer wrapper has NO voice-listing endpoint at all; voices are filesystem paths under the `kyutai/tts-voices` HF repo). - `[2026-05-12]` Defaulting asset-engine to Traefik-routed (`asset-engine.phasefinal.com` with `anaprod` cert resolver) on first scaffold — user pulled it back to LAN-direct. Internal tools default LAN-direct; only add Traefik when an external/TLS surface is actually needed. - `[2026-05-12]` Routing althing thread replies through `galdrabok` when the actual dev handle is `forseti` — bus rejected `to=forseti` initially because thread participants list was `[galdrabok, infra]`. Solved by starting a new thread with `forseti` as the direct recipient. Lesson: when the bus auto-resolves a sender handle that doesn't match the actual dev role, start a fresh thread rather than fighting the participant list. - `[2026-05-08]` Filtering Traefik's UTC access log by Gitea-local-PDT timestamp substrings (`grep "2026/05/08 15:1[2-7]"`) returned zero matches and led to a wrong "no /v2/ traffic in 12 days" conclusion. **Gitea logs in PDT, Traefik logs in UTC** — same host, different timezones. Always normalize timezones (UTC) when correlating logs across services on the same box. Cost: ~30 min in the wrong direction. - `[2026-05-08]` Validating a user-proposed Traefik/Gitea timeout bump for "Traefik is dropping connections on big docker pushes" without first verifying which component was actually in the failure path. Traefik turned out to be innocent (`499 / 60012ms` = client closed, Traefik never timed out), Gitea's `PER_WRITE_TIMEOUT` governs response writes (wrong direction), and the real ceiling was a ~60s client-side timer no compose change can reach. **Lesson: validate the diagnostic premise — which component is actually in the failure path? — before refining the proposed fix.** - `[2026-05-08]` Bumping Gitea `PER_WRITE_TIMEOUT` / `PER_WRITE_PER_KB_TIMEOUT` to address `unexpected EOF` on `/v2/.../blobs/uploads/` PATCH — wrong direction. Both govern **response writes**, not request body reads. `unexpected EOF` from Go's HTTP server means the client closed mid-body-upload; not a knob Gitea exposes server-side. - `[2026-04-30]` task-board workflow with `container: image: debian:bookworm-slim` — fails: `actions/checkout@v4` needs `node` at runtime, slim image lacks it. Switched to `node:20-bookworm-slim` (has node + apt) or runner-label default. (Pattern revisited 2026-05-17 for skaldsong-dev: container override needs `nodejs` apt-installed unless it IS the default.) - `[2026-04-30]` Dropping the `container:` directive before runner re-registration with docker-schema labels — runner silently falls back to **host mode** (jobs run inside the alpine `act_runner` container itself, no apt). The `:host` suffix in startup logs (`labels updated to: [pfi-fleet:host ana-docker:host]`) is the giveaway. Fix: register with `pfi-fleet:docker://` schema labels. - `[2026-04-30]` Updating runner labels by editing `.env` and bouncing — doesn't take. The `.runner` registration cache pins labels at first registration; env-var updates are read each start but the stored token + UUID are tied to the original label set on the gitea side. Fix: stop runner, delete `.runner`, generate new admin registration token, redeploy. - `[2026-04-30]` `git reset --hard origin/` in `deploy-task-board.yaml` (and the in-repo nevermore playbook before fix) — invalid syntax: `origin/` prefix only works for branch refs. SHAs need `git reset --hard ` directly. Resolved with `git rev-parse --verify --quiet "origin/{{ ref }}^{commit}"` first, then bare `"{{ ref }}^{commit}"` fallback. - `[2026-04-30]` Assuming `DEPLOY_SSH_KEY` was at user scope after task-board wiring — it was actually only repo-scope on `vh/task-board`. vor's first CI run failed with empty SSH key (`printf '%s\n' "" > ~/.ssh/id_ed25519`). Fix: copy secret to user scope at `gitea.phasefinal.com/user/settings/actions/secrets`. - `[2026-04-30]` `grep -vE "^(#|$)"` to inspect `.env` for sanity — leaked the full `MINIFLUX_PASSWORD` line into the transcript. Then a follow-up redaction attempt with `sed -E "s/=(.{4}).*$/=\1/"` still leaked the first 4 chars. Lesson: when probing secret-bearing files, use field-by-field SELECTIVE inspection (`grep -E "^(KEY1|KEY2)="`) rather than negative filters; for any password line, `grep -c` (existence) or `test -n "$(...)"` (non-empty), never `cat` or value-printing.