esh-pfi-infrastructure/persistent-memory.md

# Persistent memory — eshpfi-management

_Last updated: 2026-05-31_

## Repo purpose

Reference workspace for PFI infrastructure: server inventory, canonical
Docker Compose stacks, ops playbooks, and conventions. Authoritative
copies of compose files live on the servers under
`/opt/docker/compose/<stack>/`; this repo mirrors them for version
control, editing, planning, and CI-driven deploys.

## Tools and conventions

Sister repos (separate gitea repos, deployed by playbooks here):

| Repo | Role | CI status |
|---|---|---|
| `vh/task-board` | MCP + web dashboard for assistant task state (port 7878) | push-to-main → CI deploys (2026-04-29) |
| `vh/vor` | Inquisitor UI sidecar (port 7879) | push-to-main → CI deploys (2026-04-29) |
| `vh/nevermore` | Twice-daily LLM-curated briefing (port 8181, replaces news-digest) | push-to-main → CI deploys (2026-04-30) |
| `vh/asset-engine` | Internal control plane over inference services (port 8200, LAN-direct) | push-to-main → CI deploys (2026-05-12) |
| `vh/althing` | Inter-agent message bus (chamber UI port 7881, forseti + agent-runner daemons, valkey IPC) | push-to-main → CI deploys (2026-05-14) |
| `vh/mead-hall` | Bifrost tool-provider sidecar (port 5173 on dev VM 10.100.10.50) | push-to-main → CI deploys (2026-05-16) |
| `vh/skaldsong` | Wizard + reader surface (port 8300, ana-docker, registry-pull pattern) | push-to-main → CI deploys (2026-05-19) |
| `vh/worldtree` | Conversation API (corviduo-dev demo :8080 / personal :8081 / pinned :8082) — Heimdall auth, Bifrost integration | push-to-main → CI deploys |
| `vh/volva` | Codex peer agent on althing bus (single-turn oracle, systemd daemon on nh3-dev) | manual install via `deploy/volva.service` (2026-05-18) |

- **Two-layer backups** — Backrest orchestrates restic for file+DB (5
  fleet repos, daily 01:00 PDT); PBS-ANA primary + PBS-NH3 DR mirror for
  VM images. ana-nas is the SPOF for postgres + PBS-ANA datastore +
  cross-site restic targets — see `docs/runbooks/disaster-recovery.md`
  for the blast-radius matrix.

- **`pull-hf-repo.yaml`** is the canonical "get a HuggingFace
  model/dataset onto ana-ml2's shared cache at
  `/tank/aimodels/huggingface/`" playbook. Supports `--var repo_type=model|dataset|space`. Replaces ad-hoc `huggingface_hub.snapshot_download` patterns.

- **Worldtree admin auth — per-instance.** Each Worldtree deployment
  (demo :8080, personal :8081, pinned :8082) has its own Heimdall
  registry and its own bootstrap admin key. Infra-ops's stored
  long-lived admin key (`key_id 61419c92`) at
  `ana-docker:/opt/docker/conf/.secrets/worldtree-infra-ops-admin`
  auths against **demo only**. For personal-instance admin ops, fetch
  the bootstrap admin per-op via
  `docker exec worldtree-personal-worldtree-api-1 printenv WORLDTREE_BOOTSTRAP_ADMIN_KEY`
  on corviduo-dev. Used for `POST /admin/keys`, admin diagnostics
  (`/admin/sessions/<id>/{bifrost,tools}`, etc.).

- **Per-project user keys against personal Worldtree** (issued
  2026-05-19): `skaldsong:79744637` (nh3-dev iteration),
  `skaldsong:7c1dbbbe` (ana-docker prod), `althing:50d85460`,
  `mead-hall:a360822d`. Same `user_id=skaldsong` across both
  skaldsong keys → shared Heimdall agent slot; different `key_id`
  → independently rotatable. Pattern: mint via `/admin/keys`, drop
  value to `/tmp/wt-personal-<name>.key` mode 600, dev collects +
  shreds (DO NOT cat to chat transcript).

- **Skaldsong CD pattern (registry-pull).** Differs from althing /
  asset-engine which build-on-host. vh/skaldsong's CI builds and
  pushes `gitea.phasefinal.com/vh/skaldsong:<sha>` + `:latest`;
  `playbooks/deploy-skaldsong.yaml` on ana-docker pulls + recreates.
  SHA-pin only (no `:latest` health-gated advance yet). Prereq: host
  needs `docker login gitea.phasefinal.com` once (read:package PAT) —
  not currently in the workflow.

- **docker-as-root pattern** (for ops that have no admin API, e.g.
  `SqliteUserStore.set_bifrost_credentials`): on hosts where the SSH
  user is in the `docker` group but lacks passwordless sudo, run
  `docker run --rm -v <target-dir>:/wt -v /var/run/docker.sock:/var/run/docker.sock docker:cli sh -c "..."` to edit deploy-owned files
  without sudo. Documented with security warning in
  `servers/corviduo-dev/README.md`. docker-group membership is
  effectively root via bind-mount; treat as a sudo-equivalent grant.
  **Foot-gun: when running `docker compose` inside this sandbox,
  any relative path in compose.yaml (e.g. `${WORLDTREE_CONFIG_DIR:-./config}`)
  resolves against the sandbox CWD, but Docker daemon interprets the
  resulting path against the HOST filesystem. Always pass `-e VAR=/abs/path`
  to the docker run invocation for any relative-default config dir.**

## Current state / in-flight

_As of 2026-05-31:_

- **Dia2 LIVE as two instances on irv-ml1; zonos + csm still down.**
  `dia2-2b` (:8200, Dia2-2B) + `dia2-1b` (:8202, Dia2-1B) running +
  healthy on GPU 0 from `local/dia:v2`; catalog `status: ready` (both
  exercised). Legacy Dia 1.6B retired. **zonos**: REST adapter built
  (`stacks/zonos/adapter/`, `local/zonos-api`) but the stack is DOWN —
  brought down to free 3090 VRAM for Dia2; catalog entry `status: down`,
  adapter port 8203. **csm**: still PARKED (`status: down`, gated — needs
  `CSM_HF_TOKEN` before first start; non-reproducible → `experimental`
  when up). ComfyUI `:8188` still a catalog-deferred UI **link** (open
  hand-off to asset-engine-dev, no form-field API).

- **Dia2 voices staged — re-stage if host rebuilt.** The 43 curated
  voices baked at `/app/voices` aren't reachable from `/tts`'s clone path,
  so they're copied into `/worktank/dia/reference_audio` (restic-included).
  This is a **one-time manual host step** (`docker exec dia2-2b sh -c 'cp
  -n /app/voices/* /app/reference_audio/'`), documented in
  `stacks/dia/README.md`, NOT auto-run on recreate. Catalog voice default
  is `voice_mode: clone` + `clone_reference_filename: Abigail.wav`. Open
  offer (not done): make staging auto-run via container entrypoint.

- **asset-engine catalog fully synced + pushed.** `docs/asset-engine/
  services.yaml` (canonical) and `vh/asset-engine`'s vendored
  `data/services.yaml` (re-synced via `sync_catalog.py`, commit
  `58dadfe`) both on origin; catalog v1, 18 services; schema regenerated.
  asset-engine deploy-on-push triggered. asset-engine-dev notified on
  althing thread `01KSZM5MNQYP2DZ25Y2F2DXDBY` of the breaking dia→dia2
  swap (drop dia tile; add dia2-2b/dia2-1b/zonos), the `/tts` field
  shapes, and a UI-hardening ask (don't serialize empty optional selects
  as the string `"undefined"`; enforce clone_reference_filename
  required-when-clone).

- **ttyd fleet driver seat operational on nh3-dev** (browser/iPad/Vision
  Pro into the zellij `Claude` session): RO `:7681` / drive `:7682` via
  a Caddy front, network-gated (no auth). Lives in `~/.config` on
  nh3-dev (out of this repo). Full wiring + gotchas → auto-memory
  `reference_ttyd_fleet_seat`.

- **Worldtree all instances healthy on v0.29.13** (last-known, not
  re-checked this session). Demo (:8080), personal (:8081), pinned
  (:8082). v0.29.x = saga→echo refactor + selene-judgment validator +
  canonical `*.yaml.example` configs.

- **Skaldsong on v0.32.2** (ana-docker:8300). Kokoro TTS streaming via
  SSE working end-to-end; default voice `af_heart`. Last-known.

- **artemis-31b-v1i live on llama-swap + worldtree personal.** BeaverAI
  Gemma 4 31B Q6_K (~28.6GB, 128K ctx), reasoning via
  `--reasoning-format deepseek`.

- **All commits pushed (operator-authorized 2026-05-31).** The prior
  pending-push backlog + this session's TTS/catalog work are all on
  origin (eshpfi-management through `5c47843`; asset-engine through
  `58dadfe`). No local-only backlog. (Standing rule still holds: never
  push without an explicit greenlight — this push was granted.)

- **VibeVoice catalogued for skaldsong #73 contract** (TTS alongside
  Kokoro). Endpoint `http://10.100.79.3:8194`, OpenAI-compat. NO
  per-word timings (diffusion); read-along is a v2 forced-alignment
  upgrade (Parakeet irv-ml1:8765 or Whisper).

- **Disclosed-keys hygiene queue** — rotate at convenience:
  `/tmp/wt-personal-skaldsong-prod.key` on nh3-dev (shred after verify);
  mead-hall's prior Worldtree bearer (superseded by `a360822d`);
  Worldtree `Z_AI_API_KEY`/`ZAI_API_KEY` (2026-05-12 outage); chamber
  `forseti`/`agent_runner` api_keys (superseded by `50d85460`); Gitea
  runner registration token (`a1135753...`).

- **Still open from prior sessions:** rotate `MINIFLUX_PASSWORD`
  (leaked twice); clean up legacy `news-digest` detritus on ana-docker;
  watch nh3-nas `/volume1` (was 65%; recheck before ~80%); the
  `docker push 60s client-side ceiling` mystery remains uninstrumented.

## Recent decisions

- `[2026-05-31]` Dia2 deployed as **two fixed-model instances**
  (`dia2-2b` :8200, `dia2-1b` :8202) from `local/dia:v2`, retiring
  legacy Dia 1.6B; catalog `dia` entry **removed → dia2-2b + dia2-1b**
  (breaking for asset-engine). Rationale: the devnen wrapper is
  single-model and IGNORES the OpenAI `model` field (verified on its live
  OpenAPI), so the only way to offer both Dia2 models as real per-request
  asset-engine choices is one fixed endpoint per model. `3139e81`
  (deploy), `db15638` (catalog swap).

- `[2026-05-31]` Both dia2 catalog entries route to the wrapper's richer
  **`/tts`** endpoint (not `/v1/audio/speech`) to expose the full control
  surface (cfg_scale/temperature/top_p/cfg_filter_top_k/voice_mode/clone);
  all defaults sourced from the wrapper's `CustomTTSRequest` Pydantic
  blessed values. Voice default is `voice_mode: clone` +
  `clone_reference_filename: Abigail.wav` so a stable (non-random-gender)
  voice is the out-of-box behavior. `55602b7`, `5c47843`.

- `[2026-05-31]` Zonos REST adapter (`stacks/zonos/adapter/`,
  `local/zonos-api`) — thin OpenAI-ish `/v1/audio/speech` FastAPI in front
  of the Gradio-only Zonos SDK; JSON-envelope `{audio, audio_format,
  seed}` (Zonos is the fleet's first seedable TTS). Port 8203 (moved off
  8201 — collided with csm). Built; NOT deployed (stack down for VRAM).
  Also fixed the upstream image's missing CMD (`71df6f7`). `81efa8d`.

- `[2026-05-31]` Catalog schema regenerated: added `CatalogLifecycle` +
  `reproducibility.seed_field` (`b7b2130`). Resolves the stale-schema
  hand-off; catalog now validates clean. (asset_engine `catalog.py`
  Pydantic already supported both — schema file was just behind.)

- `[2026-05-31]` TTS bench expanded with `stacks/{dia,zonos,csm}`
  (`666f7f3` dia+zonos, `a4b8c2a` csm): dia = Nari dialogue TTS via
  devnen/Dia-TTS-Server (OpenAI-compat, :8200), zonos = Zyphra Gradio
  audition surface (:8199, no API), csm = Sesame conversational
  (gated model, :8201). dia + csm catalogued in
  `docs/asset-engine/services.yaml` `status: down` + repro audits
  (`8eac39b`); all three deployed PARKED to irv-ml1 (no GPU spin).
  Zonos + ComfyUI = catalog-deferred (no form-field API) → asset-engine
  UI **links**, handed to asset-engine-dev. The bench already had Fish
  S2-Pro / Chatterbox-Turbo / IndexTTS-2 / CosyVoice3 / Kokoro /
  VibeVoice / Qwen3-TTS / Kyutai, so the research-agent's top-2 picks
  were already deployed.

- `[2026-05-31]` Remote browser/iPad/Vision-Pro driver seat for the
  agent-fleet zellij session `Claude` stood up on nh3-dev (ttyd behind
  Caddy, network-gated). Brokkr-smithy-dev request; no Tailscale (WG at
  firewall). Out of this repo — full architecture + the
  HTTP2/OSC52/Safari-auth gotchas in auto-memory
  `reference_ttyd_fleet_seat`.

- `[2026-05-30]` esh-docker-vm NFS boot-ordering fix:
  `playbooks/fix-esh-nfs-boot-ordering.yaml` (`c0458d9`, + `53157b1`
  drop-in filename-collision fix) adds `_netdev,nofail` to the four
  10.0.50.50 NFS mounts + a docker `After=remote-fs.target` drop-in —
  resolves paperless `Exited(255)` on every reboot. traefik also gained
  `restart: unless-stopped` (was Docker-default `no` → never
  auto-started after a reboot; "homepage broken" was collateral via the
  traefik route). Full incident → auto-memory
  `incident_esh_docker_nfs_boot_race`.

- `[2026-05-27]` Worldtree v0.29.x landed full saga→echo refactor +
  config-validator hardening. Net image:
  - v0.29.10 — `create_provider` checks model_catalog `family:`
    before regex auto-detect (artemis routing fix)
  - v0.29.11 — collect-then-raise echo startup validators (UX fix
    for cascading one-at-a-time crash cycle that bit demo + personal
    during the v0.29.0 → v0.29.9 deploy)
  - v0.29.12 — ships `providers.yaml.example` + `defaults.yaml.example`
    canonical configs (REQUIRED/OPTIONAL annotated, guaranteed-pass-
    validator). Replaces reverse-engineer-from-validator-errors as
    the operator's path.
  - v0.29.13 — `reasoning_content` extraction in base provider +
    catalog `family:` lookup fix (artemis reasoning route through
    worldtree GemmaProvider).
  Operator-asked, worldtree-dev-shipped, all auto-deployed via CI.

- `[2026-05-27]` artemis-31b-v1i added to llama-swap + worldtree
  personal providers.yaml. BeaverAI Gemma 4 31B Q6_K (~28.6GB)
  pulled via `pull-hf-repo.yaml`, llama-swap config mirrors
  gemma4-31b-dense settings at 128K ctx, `--reasoning-format deepseek`
  (gemma format not supported in deployed llama.cpp). Worldtree
  catalog entry has `family: gemma` so GemmaProvider routes reasoning
  tokens correctly. Pinned locally — push pending operator greenlight.

- `[2026-05-27]` Skaldsong streaming TTS v0.32.0 → v0.32.1 → v0.32.2.
  v0.32.0: chunked-batch via SSE on `/api/stories/{id}/audio?voice=X`,
  one Kokoro POST per paragraph. v0.32.1: defensive catch-all in
  event_stream after silent JSONDecodeError (made the real
  exception surface in an SSE error frame). v0.32.2: NDJSON parsing
  — Kokoro `/dev/captioned_speech` returns multi-line JSON for
  large inputs (one object per chunk); was using `resp.json()` which
  only parsed the first object. Verified end-to-end clean.

_22 older entries archived to archival-memory.md._

## Tried and abandoned

- `[2026-05-31]` Building the dia2-capable image surfaced THREE upstream
  packaging quirks: (1) `pip install -e nari-labs/dia2` fails — its
  pyproject build backend lacks the PEP 660 `build_editable` hook; (2)
  plain `pip install` builds an empty `UNKNOWN-0.0.0` wheel because the
  base image's setuptools (59.6) is older than dia2's required `>=70`, so
  it ignores the `[project]` metadata; (3) `--no-deps` (which the wrapper's
  own auto-installer uses) leaves `transformers`/`sphn`/`whisper-timestamped`
  missing. Fix that worked (`local/dia:v2`, `stacks/dia/dia2-image/`): copy
  the pure-python `dia2/` package straight into site-packages + install
  ONLY those 3 deps; torch 2.12 / numpy 2.2 in the base already satisfy
  Dia2, so no risky CUDA/torch churn.

- `[2026-05-31]` Dia2 predefined voices (43, baked at `/app/voices`) are
  NOT reachable from the `/tts` clone path — it resolves
  `clone_reference_filename` against the reference_audio dir ONLY
  (`engine.py`). The OpenAI `/v1/audio/speech` `voice` param DOES
  auto-resolve them by name (separate code path), which masked the gap in
  early testing. Fix: stage `/app/voices/*` into
  `/worktank/dia/reference_audio` so `/tts` clone can use them. (So a
  "verify on the endpoint the catalog actually targets" lesson — testing
  the OpenAI endpoint nearly shipped a catalog that 404s on `/tts`.)

- `[2026-05-31]` `voice_mode=clone` with an empty `clone_reference_filename`
  → asset-engine serializes it as the literal string `"undefined"` →
  `/tts` 404 `Reference audio file not found: undefined`. Reproduced
  directly. Worked around in the catalog (default the field to a real
  voice); proper fix is asset-engine-side (don't send `"undefined"` for
  empty optionals; enforce conditional-required) — flagged to
  asset-engine-dev.

- `[2026-05-31]` asset-engine `docs/asset-engine/services.schema.json`
  is a DERIVED artifact (regen from the Pydantic model via
  `dump_schema.py`) and had DRIFTED: it rejected the `lifecycle` field
  that 12/14 live services already use (13 pre-existing validation
  errors against the committed catalog).
  **[2026-05-31] RESOLVED:** schema regenerated with `CatalogLifecycle`
  + `reproducibility.seed_field` (`b7b2130`); catalog now validates clean.
  Standing lesson: the schema is generated from asset_engine's
  `catalog.py` — hand-editing `services.yaml` shape without regenerating
  re-introduces drift.

- `[2026-05-30]` esh-docker-vm: `hard` NFS mounts from 10.0.50.50 froze
  a container worker in UNKILLABLE D-state when the NAS stalled —
  `docker restart` fails with "tried to kill container, but did not
  receive an exit event"; only a host reboot clears it. Separately,
  `fstab defaults` (no `_netdev`) made NFS-bind containers `Exited(255)`
  on reboot. Diagnosis + recovery steps → auto-memory
  `incident_esh_docker_nfs_boot_race`.

- `[2026-05-31]` ttyd-over-TLS forces HTTP/2 (no flag disables it) which
  kills ttyd's terminal WebSocket (blank screen); Safari/WebKit never
  sends HTTP basic-auth on WS upgrades (→ ttyd "User code denied"). Both
  solved for the fleet seat (Caddy forces HTTP/1.1; auth dropped to
  network-gating) — detail in auto-memory `reference_ttyd_fleet_seat`.

- `[2026-05-27]` `docker compose up -d` from within the `docker:cli`
  sandbox: `${VAR:-./config}` defaults resolve `./config` to the
  sandbox CWD (e.g. `/wt/config`). Docker daemon interprets the
  resulting path against the HOST filesystem, not the sandbox's.
  Auto-creates an empty dir on the host → container saw an empty
  bind target → entrypoint shim seeded image-baked defaults
  (lost all the host-side providers.yaml patches I'd applied earlier
  in the day). Root cause for ratatoskr-dev's Tier 3 sessions 500.
  Fix: pass `-e WORLDTREE_CONFIG_DIR=/opt/worldtree-personal/config`
  to the docker run invocation. Captured into the docker-as-root
  pattern note in Tools and conventions.

- `[2026-05-27]` `:latest`-pinned compose + private gitea registry +
  sandboxed pull = recreate on ancient cached `:latest`. Local
  `:latest` tag never advanced because deploy.yml pulls by SHA
  (not by tag); sandbox can't `docker pull` (no creds for private
  registry); `compose up -d --pull never` used the 2-week-old
  cached `:latest`. Symptom: personal worldtree's matrix container
  crash-looped because the ancient image didn't understand the new
  config schema. Fix: retag SHA → `:latest` on host first
  (`docker tag <registry>/<repo>:<sha> <registry>/<repo>:latest`),
  then recreate with `--pull never`. Better long-term: pin SHA
  explicitly in `.env`, advance in CI.

- `[2026-05-27]` Container recreate during in-flight skaldsong gen
  kills the runner. Pattern: my llama-swap restart at 23:57Z + a
  v0.31.2 `compose up -d` at 02:47Z both took out active dramatist
  passes. Confirmed by skaldsong-dev INV-006 ("partial state only
  flips on host restart"). With deploys landing every ~10min during
  active dev and stories taking >5min, runner-killed-by-recreate is
  structural, not incidental. Roadmap (skaldsong-dev): pre-shutdown
  signal handler (cleanup row state to failed/aborted), per-scene
  resume-from-checkpoint, `/api/admin/quiesce` endpoint. None shipped
  yet; operator's call which path.

- `[2026-05-27]` `--reasoning-format gemma` on artemis-31b-v1i —
  not supported in the deployed llama.cpp build. Accepts only
  `none | deepseek | deepseek-legacy`. `deepseek` populates the
  `reasoning_content` SSE delta field which Worldtree GemmaProvider's
  fallback path checks. Was load-bearing for getting reasoning
  tokens through to ratatoskr's wire.

- `[2026-05-27]` `head -c N` piped after a streaming curl SIGPIPEs
  the curl, killing the request early. Bit me twice while probing
  SSE-shape samples. Use file-write + separate read.

_24 older entries archived to archival-memory.md._