esh-pfi-infrastructure/persistent-memory.md

Persistent memory — eshpfi-management

Last updated: 2026-05-31

Repo purpose

Reference workspace for PFI infrastructure: server inventory, canonical Docker Compose stacks, ops playbooks, and conventions. Authoritative copies of compose files live on the servers under /opt/docker/compose/<stack>/; this repo mirrors them for version control, editing, planning, and CI-driven deploys.

Tools and conventions

Sister repos (separate gitea repos, deployed by playbooks here):

Repo	Role	CI status
vh/task-board	MCP + web dashboard for assistant task state (port 7878)	push-to-main → CI deploys (2026-04-29)
vh/vor	Inquisitor UI sidecar (port 7879)	push-to-main → CI deploys (2026-04-29)
vh/nevermore	Twice-daily LLM-curated briefing (port 8181, replaces news-digest)	push-to-main → CI deploys (2026-04-30)
vh/asset-engine	Internal control plane over inference services (port 8200, LAN-direct)	push-to-main → CI deploys (2026-05-12)
vh/althing	Inter-agent message bus (chamber UI port 7881, forseti + agent-runner daemons, valkey IPC)	push-to-main → CI deploys (2026-05-14)
vh/mead-hall	Bifrost tool-provider sidecar (port 5173 on dev VM 10.100.10.50)	push-to-main → CI deploys (2026-05-16)
vh/skaldsong	Wizard + reader surface (port 8300, ana-docker, registry-pull pattern)	push-to-main → CI deploys (2026-05-19)
vh/worldtree	Conversation API (corviduo-dev demo :8080 / personal :8081 / pinned :8082) — Heimdall auth, Bifrost integration	push-to-main → CI deploys
vh/volva	Codex peer agent on althing bus (single-turn oracle, systemd daemon on nh3-dev)	manual install via deploy/volva.service (2026-05-18)

• Two-layer backups — Backrest orchestrates restic for file+DB (5 fleet repos, daily 01:00 PDT); PBS-ANA primary + PBS-NH3 DR mirror for VM images. ana-nas is the SPOF for postgres + PBS-ANA datastore + cross-site restic targets — see docs/runbooks/disaster-recovery.md for the blast-radius matrix.

• pull-hf-repo.yaml is the canonical "get a HuggingFace model/dataset onto ana-ml2's shared cache at /tank/aimodels/huggingface/" playbook. Supports --var repo_type=model|dataset|space. Replaces ad-hoc huggingface_hub.snapshot_download patterns.

• Worldtree admin auth — per-instance. Each Worldtree deployment (demo :8080, personal :8081, pinned :8082) has its own Heimdall registry and its own bootstrap admin key. Infra-ops's stored long-lived admin key (key_id 61419c92) at ana-docker:/opt/docker/conf/.secrets/worldtree-infra-ops-admin auths against demo only. For personal-instance admin ops, fetch the bootstrap admin per-op via docker exec worldtree-personal-worldtree-api-1 printenv WORLDTREE_BOOTSTRAP_ADMIN_KEY on corviduo-dev. Used for POST /admin/keys, admin diagnostics (/admin/sessions/<id>/{bifrost,tools}, etc.).

• Per-project user keys against personal Worldtree (issued 2026-05-19): skaldsong:79744637 (nh3-dev iteration), skaldsong:7c1dbbbe (ana-docker prod), althing:50d85460, mead-hall:a360822d. Same user_id=skaldsong across both skaldsong keys → shared Heimdall agent slot; different key_id → independently rotatable. Pattern: mint via /admin/keys, drop value to /tmp/wt-personal-<name>.key mode 600, dev collects + shreds (DO NOT cat to chat transcript).

• Skaldsong CD pattern (registry-pull). Differs from althing / asset-engine which build-on-host. vh/skaldsong's CI builds and pushes gitea.phasefinal.com/vh/skaldsong:<sha> + :latest; playbooks/deploy-skaldsong.yaml on ana-docker pulls + recreates. SHA-pin only (no :latest health-gated advance yet). Prereq: host needs docker login gitea.phasefinal.com once (read:package PAT) — not currently in the workflow.

• docker-as-root pattern (for ops that have no admin API, e.g. SqliteUserStore.set_bifrost_credentials): on hosts where the SSH user is in the docker group but lacks passwordless sudo, run docker run --rm -v <target-dir>:/wt -v /var/run/docker.sock:/var/run/docker.sock docker:cli sh -c "..." to edit deploy-owned files without sudo. Documented with security warning in servers/corviduo-dev/README.md. docker-group membership is effectively root via bind-mount; treat as a sudo-equivalent grant. Foot-gun: when running docker compose inside this sandbox, any relative path in compose.yaml (e.g. ${WORLDTREE_CONFIG_DIR:-./config}) resolves against the sandbox CWD, but Docker daemon interprets the resulting path against the HOST filesystem. Always pass -e VAR=/abs/path to the docker run invocation for any relative-default config dir.

Current state / in-flight

As of 2026-05-31:

• Dia2 LIVE as two instances on irv-ml1; zonos + csm still down. dia2-2b (:8200, Dia2-2B) + dia2-1b (:8202, Dia2-1B) running + healthy on GPU 0 from local/dia:v2; catalog status: ready (both exercised). Legacy Dia 1.6B retired. zonos: REST adapter built (stacks/zonos/adapter/, local/zonos-api) but the stack is DOWN — brought down to free 3090 VRAM for Dia2; catalog entry status: down, adapter port 8203. csm: still PARKED (status: down, gated — needs CSM_HF_TOKEN before first start; non-reproducible → experimental when up). ComfyUI :8188 still a catalog-deferred UI link (open hand-off to asset-engine-dev, no form-field API).

• Dia2 voices staged — re-stage if host rebuilt. The 43 curated voices baked at /app/voices aren't reachable from /tts's clone path, so they're copied into /worktank/dia/reference_audio (restic-included). This is a one-time manual host step (docker exec dia2-2b sh -c 'cp -n /app/voices/* /app/reference_audio/'), documented in stacks/dia/README.md, NOT auto-run on recreate. Catalog voice default is voice_mode: clone + clone_reference_filename: Abigail.wav. Open offer (not done): make staging auto-run via container entrypoint.

• asset-engine catalog fully synced + pushed. docs/asset-engine/ services.yaml (canonical) and vh/asset-engine's vendored data/services.yaml (re-synced via sync_catalog.py, commit 58dadfe) both on origin; catalog v1, 18 services; schema regenerated. asset-engine deploy-on-push triggered. asset-engine-dev notified on althing thread 01KSZM5MNQYP2DZ25Y2F2DXDBY of the breaking dia→dia2 swap (drop dia tile; add dia2-2b/dia2-1b/zonos), the /tts field shapes, and a UI-hardening ask (don't serialize empty optional selects as the string "undefined"; enforce clone_reference_filename required-when-clone).

• ttyd fleet driver seat operational on nh3-dev (browser/iPad/Vision Pro into the zellij Claude session): RO :7681 / drive :7682 via a Caddy front, network-gated (no auth). Lives in ~/.config on nh3-dev (out of this repo). Full wiring + gotchas → auto-memory reference_ttyd_fleet_seat.

• Worldtree all instances healthy on v0.29.13 (last-known, not re-checked this session). Demo (:8080), personal (:8081), pinned (:8082). v0.29.x = saga→echo refactor + selene-judgment validator + canonical *.yaml.example configs.

• Skaldsong on v0.32.2 (ana-docker:8300). Kokoro TTS streaming via SSE working end-to-end; default voice af_heart. Last-known.

• artemis-31b-v1i live on llama-swap + worldtree personal. BeaverAI Gemma 4 31B Q6_K (~28.6GB, 128K ctx), reasoning via --reasoning-format deepseek.

• All commits pushed (operator-authorized 2026-05-31). The prior pending-push backlog + this session's TTS/catalog work are all on origin (eshpfi-management through 5c47843; asset-engine through 58dadfe). No local-only backlog. (Standing rule still holds: never push without an explicit greenlight — this push was granted.)

• VibeVoice catalogued for skaldsong #73 contract (TTS alongside Kokoro). Endpoint http://10.100.79.3:8194, OpenAI-compat. NO per-word timings (diffusion); read-along is a v2 forced-alignment upgrade (Parakeet irv-ml1:8765 or Whisper).

• Disclosed-keys hygiene queue — rotate at convenience: /tmp/wt-personal-skaldsong-prod.key on nh3-dev (shred after verify); mead-hall's prior Worldtree bearer (superseded by a360822d); Worldtree Z_AI_API_KEY/ZAI_API_KEY (2026-05-12 outage); chamber forseti/agent_runner api_keys (superseded by 50d85460); Gitea runner registration token (a1135753...).

• Still open from prior sessions: rotate MINIFLUX_PASSWORD (leaked twice); clean up legacy news-digest detritus on ana-docker; watch nh3-nas /volume1 (was 65%; recheck before ~80%); the docker push 60s client-side ceiling mystery remains uninstrumented.

Recent decisions

• [2026-05-31] Dia2 deployed as two fixed-model instances (dia2-2b :8200, dia2-1b :8202) from local/dia:v2, retiring legacy Dia 1.6B; catalog dia entry removed → dia2-2b + dia2-1b (breaking for asset-engine). Rationale: the devnen wrapper is single-model and IGNORES the OpenAI model field (verified on its live OpenAPI), so the only way to offer both Dia2 models as real per-request asset-engine choices is one fixed endpoint per model. 3139e81 (deploy), db15638 (catalog swap).

• [2026-05-31] Both dia2 catalog entries route to the wrapper's richer /tts endpoint (not /v1/audio/speech) to expose the full control surface (cfg_scale/temperature/top_p/cfg_filter_top_k/voice_mode/clone); all defaults sourced from the wrapper's CustomTTSRequest Pydantic blessed values. Voice default is voice_mode: clone + clone_reference_filename: Abigail.wav so a stable (non-random-gender) voice is the out-of-box behavior. 55602b7, 5c47843.

• [2026-05-31] Zonos REST adapter (stacks/zonos/adapter/, local/zonos-api) — thin OpenAI-ish /v1/audio/speech FastAPI in front of the Gradio-only Zonos SDK; JSON-envelope {audio, audio_format, seed} (Zonos is the fleet's first seedable TTS). Port 8203 (moved off 8201 — collided with csm). Built; NOT deployed (stack down for VRAM). Also fixed the upstream image's missing CMD (71df6f7). 81efa8d.

• [2026-05-31] Catalog schema regenerated: added CatalogLifecycle + reproducibility.seed_field (b7b2130). Resolves the stale-schema hand-off; catalog now validates clean. (asset_engine catalog.py Pydantic already supported both — schema file was just behind.)

• [2026-05-31] TTS bench expanded with stacks/{dia,zonos,csm} (666f7f3 dia+zonos, a4b8c2a csm): dia = Nari dialogue TTS via devnen/Dia-TTS-Server (OpenAI-compat, :8200), zonos = Zyphra Gradio audition surface (:8199, no API), csm = Sesame conversational (gated model, :8201). dia + csm catalogued in docs/asset-engine/services.yaml status: down + repro audits (8eac39b); all three deployed PARKED to irv-ml1 (no GPU spin). Zonos + ComfyUI = catalog-deferred (no form-field API) → asset-engine UI links, handed to asset-engine-dev. The bench already had Fish S2-Pro / Chatterbox-Turbo / IndexTTS-2 / CosyVoice3 / Kokoro / VibeVoice / Qwen3-TTS / Kyutai, so the research-agent's top-2 picks were already deployed.

• [2026-05-31] Remote browser/iPad/Vision-Pro driver seat for the agent-fleet zellij session Claude stood up on nh3-dev (ttyd behind Caddy, network-gated). Brokkr-smithy-dev request; no Tailscale (WG at firewall). Out of this repo — full architecture + the HTTP2/OSC52/Safari-auth gotchas in auto-memory reference_ttyd_fleet_seat.

• [2026-05-30] esh-docker-vm NFS boot-ordering fix: playbooks/fix-esh-nfs-boot-ordering.yaml (c0458d9, + 53157b1 drop-in filename-collision fix) adds _netdev,nofail to the four 10.0.50.50 NFS mounts + a docker After=remote-fs.target drop-in — resolves paperless Exited(255) on every reboot. traefik also gained restart: unless-stopped (was Docker-default no → never auto-started after a reboot; "homepage broken" was collateral via the traefik route). Full incident → auto-memory incident_esh_docker_nfs_boot_race.

• [2026-05-27] Worldtree v0.29.x landed full saga→echo refactor + config-validator hardening. Net image:

  • v0.29.10 — create_provider checks model_catalog family: before regex auto-detect (artemis routing fix)
  • v0.29.11 — collect-then-raise echo startup validators (UX fix for cascading one-at-a-time crash cycle that bit demo + personal during the v0.29.0 → v0.29.9 deploy)
  • v0.29.12 — ships providers.yaml.example + defaults.yaml.example canonical configs (REQUIRED/OPTIONAL annotated, guaranteed-pass- validator). Replaces reverse-engineer-from-validator-errors as the operator's path.
  • v0.29.13 — reasoning_content extraction in base provider + catalog family: lookup fix (artemis reasoning route through worldtree GemmaProvider). Operator-asked, worldtree-dev-shipped, all auto-deployed via CI.

• [2026-05-27] artemis-31b-v1i added to llama-swap + worldtree personal providers.yaml. BeaverAI Gemma 4 31B Q6_K (~28.6GB) pulled via pull-hf-repo.yaml, llama-swap config mirrors gemma4-31b-dense settings at 128K ctx, --reasoning-format deepseek (gemma format not supported in deployed llama.cpp). Worldtree catalog entry has family: gemma so GemmaProvider routes reasoning tokens correctly. Pinned locally — push pending operator greenlight.

• [2026-05-27] Skaldsong streaming TTS v0.32.0 → v0.32.1 → v0.32.2. v0.32.0: chunked-batch via SSE on /api/stories/{id}/audio?voice=X, one Kokoro POST per paragraph. v0.32.1: defensive catch-all in event_stream after silent JSONDecodeError (made the real exception surface in an SSE error frame). v0.32.2: NDJSON parsing — Kokoro /dev/captioned_speech returns multi-line JSON for large inputs (one object per chunk); was using resp.json() which only parsed the first object. Verified end-to-end clean.

22 older entries archived to archival-memory.md.

Tried and abandoned

• [2026-05-31] Building the dia2-capable image surfaced THREE upstream packaging quirks: (1) pip install -e nari-labs/dia2 fails — its pyproject build backend lacks the PEP 660 build_editable hook; (2) plain pip install builds an empty UNKNOWN-0.0.0 wheel because the base image's setuptools (59.6) is older than dia2's required >=70, so it ignores the [project] metadata; (3) --no-deps (which the wrapper's own auto-installer uses) leaves transformers/sphn/whisper-timestamped missing. Fix that worked (local/dia:v2, stacks/dia/dia2-image/): copy the pure-python dia2/ package straight into site-packages + install ONLY those 3 deps; torch 2.12 / numpy 2.2 in the base already satisfy Dia2, so no risky CUDA/torch churn.

• [2026-05-31] Dia2 predefined voices (43, baked at /app/voices) are NOT reachable from the /tts clone path — it resolves clone_reference_filename against the reference_audio dir ONLY (engine.py). The OpenAI /v1/audio/speech voice param DOES auto-resolve them by name (separate code path), which masked the gap in early testing. Fix: stage /app/voices/* into /worktank/dia/reference_audio so /tts clone can use them. (So a "verify on the endpoint the catalog actually targets" lesson — testing the OpenAI endpoint nearly shipped a catalog that 404s on /tts.)

• [2026-05-31] voice_mode=clone with an empty clone_reference_filename → asset-engine serializes it as the literal string "undefined" → /tts 404 Reference audio file not found: undefined. Reproduced directly. Worked around in the catalog (default the field to a real voice); proper fix is asset-engine-side (don't send "undefined" for empty optionals; enforce conditional-required) — flagged to asset-engine-dev.

• [2026-05-31] asset-engine docs/asset-engine/services.schema.json is a DERIVED artifact (regen from the Pydantic model via dump_schema.py) and had DRIFTED: it rejected the lifecycle field that 12/14 live services already use (13 pre-existing validation errors against the committed catalog). [2026-05-31] RESOLVED: schema regenerated with CatalogLifecycle

  • reproducibility.seed_field (b7b2130); catalog now validates clean. Standing lesson: the schema is generated from asset_engine's catalog.py — hand-editing services.yaml shape without regenerating re-introduces drift.

• [2026-05-30] esh-docker-vm: hard NFS mounts from 10.0.50.50 froze a container worker in UNKILLABLE D-state when the NAS stalled — docker restart fails with "tried to kill container, but did not receive an exit event"; only a host reboot clears it. Separately, fstab defaults (no _netdev) made NFS-bind containers Exited(255) on reboot. Diagnosis + recovery steps → auto-memory incident_esh_docker_nfs_boot_race.

• [2026-05-31] ttyd-over-TLS forces HTTP/2 (no flag disables it) which kills ttyd's terminal WebSocket (blank screen); Safari/WebKit never sends HTTP basic-auth on WS upgrades (→ ttyd "User code denied"). Both solved for the fleet seat (Caddy forces HTTP/1.1; auth dropped to network-gating) — detail in auto-memory reference_ttyd_fleet_seat.

• [2026-05-27] docker compose up -d from within the docker:cli sandbox: ${VAR:-./config} defaults resolve ./config to the sandbox CWD (e.g. /wt/config). Docker daemon interprets the resulting path against the HOST filesystem, not the sandbox's. Auto-creates an empty dir on the host → container saw an empty bind target → entrypoint shim seeded image-baked defaults (lost all the host-side providers.yaml patches I'd applied earlier in the day). Root cause for ratatoskr-dev's Tier 3 sessions 500. Fix: pass -e WORLDTREE_CONFIG_DIR=/opt/worldtree-personal/config to the docker run invocation. Captured into the docker-as-root pattern note in Tools and conventions.

• [2026-05-27] :latest-pinned compose + private gitea registry + sandboxed pull = recreate on ancient cached :latest. Local :latest tag never advanced because deploy.yml pulls by SHA (not by tag); sandbox can't docker pull (no creds for private registry); compose up -d --pull never used the 2-week-old cached :latest. Symptom: personal worldtree's matrix container crash-looped because the ancient image didn't understand the new config schema. Fix: retag SHA → :latest on host first (docker tag <registry>/<repo>:<sha> <registry>/<repo>:latest), then recreate with --pull never. Better long-term: pin SHA explicitly in .env, advance in CI.

• [2026-05-27] Container recreate during in-flight skaldsong gen kills the runner. Pattern: my llama-swap restart at 23:57Z + a v0.31.2 compose up -d at 02:47Z both took out active dramatist passes. Confirmed by skaldsong-dev INV-006 ("partial state only flips on host restart"). With deploys landing every ~10min during active dev and stories taking >5min, runner-killed-by-recreate is structural, not incidental. Roadmap (skaldsong-dev): pre-shutdown signal handler (cleanup row state to failed/aborted), per-scene resume-from-checkpoint, /api/admin/quiesce endpoint. None shipped yet; operator's call which path.

• [2026-05-27] --reasoning-format gemma on artemis-31b-v1i — not supported in the deployed llama.cpp build. Accepts only none | deepseek | deepseek-legacy. deepseek populates the reasoning_content SSE delta field which Worldtree GemmaProvider's fallback path checks. Was load-bearing for getting reasoning tokens through to ratatoskr's wire.

• [2026-05-27] head -c N piped after a streaming curl SIGPIPEs the curl, killing the request early. Bit me twice while probing SSE-shape samples. Use file-write + separate read.

24 older entries archived to archival-memory.md.