snapshot: roll persistent-memory.md from 2026-05-17 → 2026-05-20

Captures the May 18-20 session arc:

- Skaldsong CD shipped on ana-docker:8300 via registry-pull pattern
  (differs from althing/asset-engine which build-on-host); three
  first-deploy footguns surfaced + canonical-patched.
- Worldtree CD disk-hygiene PR #184 (vh/Worldtree) — watermark gate,
  eager post-deploy SHA prune, set -e hardening in SSH blocks.
  Approved by worldtree-dev, pending merge.
- mead-hall Bifrost v0.3 wire-layer + data-flow end-to-end smoke
  green (task #32 closed) — resolves the "stalled mid-Worldtree"
  state in the prior snapshot.
- Volva systemd install — three-stage debug (PATH, ReadWritePaths,
  handle pinning); impersonation footgun caught via PID/journal
  cross-reference.
- Per-project user keys minted against personal Worldtree (:8081)
  for skaldsong, althing, mead-hall + skaldsong-prod. Personal
  becomes multi-consumer dev iteration instance; demo stays for
  isolation.

File now at 434 lines (over 300 soft cap); next /snapshot run will
surface archival candidates.

persistent-memory.md

@@ -1,6 +1,6 @@
 # Persistent memory — eshpfi-management
 
-_Last updated: 2026-05-17_
+_Last updated: 2026-05-20_
 
 ## Repo purpose
 
@@ -22,8 +22,9 @@ Sister repos (separate gitea repos, deployed by playbooks here):
 | `vh/asset-engine` | Internal control plane over inference services (port 8200, LAN-direct) | push-to-main → CI deploys (2026-05-12) |
 | `vh/althing` | Inter-agent message bus (chamber UI port 7881, forseti + agent-runner daemons, valkey IPC) | push-to-main → CI deploys (2026-05-14) |
 | `vh/mead-hall` | Bifrost tool-provider sidecar (port 5173 on dev VM 10.100.10.50) | push-to-main → CI deploys (2026-05-16) |
-| `vh/skaldsong` | TBD — first CI workflow shipped 2026-05-17 | push-to-main → CI deploys (2026-05-17) |
-| `vh/worldtree` | Conversation API (corviduo-dev, port 8080) — Heimdall auth, Bifrost integration | push-to-main → CI deploys |
+| `vh/skaldsong` | Wizard + reader surface (port 8300, ana-docker, registry-pull pattern) | push-to-main → CI deploys (2026-05-19) |
+| `vh/worldtree` | Conversation API (corviduo-dev demo :8080 / personal :8081 / pinned :8082) — Heimdall auth, Bifrost integration | push-to-main → CI deploys |
+| `vh/volva` | Codex peer agent on althing bus (single-turn oracle, systemd daemon on nh3-dev) | manual install via `deploy/volva.service` (2026-05-18) |
 
 - **Two-layer backups** — Backrest orchestrates restic for file+DB (5
   fleet repos, daily 01:00 PDT); PBS-ANA primary + PBS-NH3 DR mirror for
@@ -35,13 +36,33 @@ Sister repos (separate gitea repos, deployed by playbooks here):
   model/dataset onto ana-ml2's shared cache at
   `/tank/aimodels/huggingface/`" playbook. Supports `--var repo_type=model|dataset|space`. Replaces ad-hoc `huggingface_hub.snapshot_download` patterns.
 
-- **Worldtree admin auth** — infra-ops holds a permanent admin-tier
-  Heimdall key issued 2026-05-17 (`key_id 61419c92`), stored at
+- **Worldtree admin auth — per-instance.** Each Worldtree deployment
+  (demo :8080, personal :8081, pinned :8082) has its own Heimdall
+  registry and its own bootstrap admin key. Infra-ops's stored
+  long-lived admin key (`key_id 61419c92`) at
   `ana-docker:/opt/docker/conf/.secrets/worldtree-infra-ops-admin`
-  mode 600. Used for `POST /admin/keys`, admin diagnostic reads
-  (`/admin/sessions/<id>/{bifrost,tools}`, etc.), and any other
-  Worldtree admin work. Replaces the docker-as-root + bootstrap-admin
-  pattern that was in use 2026-05-12 through 2026-05-16.
+  auths against **demo only**. For personal-instance admin ops, fetch
+  the bootstrap admin per-op via
+  `docker exec worldtree-personal-worldtree-api-1 printenv WORLDTREE_BOOTSTRAP_ADMIN_KEY`
+  on corviduo-dev. Used for `POST /admin/keys`, admin diagnostics
+  (`/admin/sessions/<id>/{bifrost,tools}`, etc.).
+
+- **Per-project user keys against personal Worldtree** (issued
+  2026-05-19): `skaldsong:79744637` (nh3-dev iteration),
+  `skaldsong:7c1dbbbe` (ana-docker prod), `althing:50d85460`,
+  `mead-hall:a360822d`. Same `user_id=skaldsong` across both
+  skaldsong keys → shared Heimdall agent slot; different `key_id`
+  → independently rotatable. Pattern: mint via `/admin/keys`, drop
+  value to `/tmp/wt-personal-<name>.key` mode 600, dev collects +
+  shreds (DO NOT cat to chat transcript).
+
+- **Skaldsong CD pattern (registry-pull).** Differs from althing /
+  asset-engine which build-on-host. vh/skaldsong's CI builds and
+  pushes `gitea.phasefinal.com/vh/skaldsong:<sha>` + `:latest`;
+  `playbooks/deploy-skaldsong.yaml` on ana-docker pulls + recreates.
+  SHA-pin only (no `:latest` health-gated advance yet). Prereq: host
+  needs `docker login gitea.phasefinal.com` once (read:package PAT) —
+  not currently in the workflow.
 
 - **docker-as-root pattern** (for ops that have no admin API, e.g.
   `SqliteUserStore.set_bifrost_credentials`): on hosts where the SSH
@@ -53,50 +74,94 @@ Sister repos (separate gitea repos, deployed by playbooks here):
 
 ## Current state / in-flight
 
-_As of 2026-05-17:_
+_As of 2026-05-20:_
 
-- **mead-hall Bifrost integration stalled mid-Worldtree.** Bifrost
-  consumer registration done from infra side (HS256, `set_bifrost_credentials` via docker-as-root + admin key, secret-drop pattern), `BIFROST_CLIENT_ALLOWED_HOSTS` env wired through to corviduo-dev's container. But the actual tracer dispatch shows
-  `session_not_bifrost_bound` per `/admin/sessions/<id>/bifrost` — the
-  handshake doesn't register bfclient into `_bifrost_clients[]`.
-  worldtree-dev investigating; not infra-side work.
+- **Worldtree CD disk-hygiene PR #184 approved, ready for merge.**
+  Three commits on `infra/disk-watermark-gate-cd` in vh/Worldtree:
+  watermark gate (pre-pull recovery), eager post-deploy SHA prune
+  (post-`:latest`-advance), `set -e` hardening in SSH blocks. Pending
+  worldtree-dev's merge button. Follow-up extension to
+  `deploy-personal.yml` + `deploy-pinned.yml` + Beszel disk-usage
+  metric all queued, not bundled.
 
-- **althing-chamber stack at 4 services.** chamber (port 7881) +
-  forseti (moderator daemon) + agent-runner (Phase 2 worldtree-driver
-  dispatcher) + valkey (Phase 3.1 pub/sub bridge for cross-process
-  streaming). Phase 3.1 verified end-to-end via event histogram on
-  2026-05-17. Forseti is the canonical dev for this stack (galdrabok
-  is on a different project).
+- **Skaldsong live on ana-docker:8300 (registry-pull pattern), pointed
+  at personal Worldtree (:8081).** Three first-deploy footguns
+  surfaced + canonical-patched: SPA path mismatch (`/app/web/dist` →
+  `/app/spa`), CORS env shape (pydantic-settings expects JSON array),
+  verify-step race vs healthcheck `start_period` (grep `^Up` not
+  `healthy`). nh3-dev hand-launched skaldsong on :8080 also still
+  running for dev iteration — same user_id=skaldsong, different
+  key_id, both authorized.
 
-- **Worldtree admin workflow shift** (2026-05-17, per vh): infra-ops
-  uses its own permanent admin-tier key for admin operations rather
-  than reaching into bootstrap admin via docker-as-root. Admin ops
-  with HTTP endpoints go over HTTPS; ops without (like
-  `set_bifrost_credentials`) still use docker-as-root but with my own
-  auth, not borrowed.
+- **Worldtree personal (:8081) is the iteration-target instance.**
+  skaldsong + mead-hall + althing-chamber all rotated to personal
+  via the per-project user keys minted 2026-05-19. Demo (:8080)
+  stays as the canonical prod surface; pinned (:8082) intentionally
+  lags. Personal's `BIFROST_CLIENT_ALLOWED_HOSTS` now mirrors demo's:
+  `10.100.10.50:5173,10.100.10.50:8080,10.250.50.70:8300`.
 
-- **Open hygiene debt** — these keys/tokens were disclosed across the
-  althing bus and/or in CC transcripts during this session and warrant
-  rotation at convenience:
-  - Gitea runner registration token (`a1135753...`) — rotate via Gitea
-    admin UI's runner-token reset.
-  - mead-hall's Worldtree bearer (`wt_live_80e1570620ef2aba998dc63954cce3a6`) — tracer issuance 2026-05-16.
-  - Worldtree provider keys `Z_AI_API_KEY` + `ZAI_API_KEY` — from the
-    2026-05-12 corviduo-dev outage.
+- **Disclosed-keys hygiene queue** — these warrant rotation at
+  convenience (mostly in-bus disclosure during this session):
+  - `/tmp/wt-personal-skaldsong-prod.key` on nh3-dev — operator
+    should shred after verify (file mode 600, lkraven-owned)
+  - mead-hall's prior Worldtree bearer
+    (`wt_live_80e1570620ef2aba998dc63954cce3a6`) — now superseded
+    by personal-instance key `a360822d` per the 2026-05-19 rotation
+  - Worldtree provider keys `Z_AI_API_KEY` + `ZAI_API_KEY` — from
+    the 2026-05-12 corviduo-dev outage
   - chamber config.yaml's `forseti.api_key` + `agent_runner.api_key`
-    (Worldtree user-tier keys) — exposed when grepping config to add
-    `streaming_enabled` 2026-05-17.
+    — superseded by personal-instance key `50d85460` 2026-05-19
+  - Gitea runner registration token (`a1135753...`) — rotate via
+    Gitea admin UI's runner-token reset
 
-- **Still open from prior session** (2026-05-08, unchanged): rotate
-  `MINIFLUX_PASSWORD` (leaked twice); clean up legacy `news-digest`
-  detritus on ana-docker (`/opt/docker/compose/news-digest/`,
-  `/opt/docker/data/news-digest/`, image `local/news-digest:v5`);
-  watch nh3-nas `/volume1` (was 65%; recheck retention or expand
-  before ~80%); the `docker push 60s client-side ceiling` mystery
-  remains uninstrumented.
+- **Still open from prior sessions:** rotate `MINIFLUX_PASSWORD`
+  (leaked twice); clean up legacy `news-digest` detritus on ana-docker
+  (`/opt/docker/compose/news-digest/`, `/opt/docker/data/news-digest/`,
+  image `local/news-digest:v5`); watch nh3-nas `/volume1` (was 65%;
+  recheck retention or expand before ~80%); the `docker push 60s
+  client-side ceiling` mystery remains uninstrumented.
 
 ## Recent decisions
 
+- `[2026-05-19]` Worldtree CD disk-hygiene strategy: watermark gate
+  (env-tunable threshold + window, fail-loud on still-low post-prune)
+  + eager post-deploy prune (only after `:latest` advance succeeds,
+  uses `docker image prune -a --filter "until=24h"` which respects
+  in-use semantic — protects pinned + personal images automatically).
+  Combined: demo VM holds ~24h of deploy history instead of unbounded
+  accumulation. Shipped in vh/Worldtree PR #184 (`306cd61` +
+  `613dac2` + `bd91df5`).
+
+- `[2026-05-19]` Skaldsong CD shape: shape (1) of three operator
+  options — container + Gitea registry + pull-restart, matching
+  Worldtree's pattern. Target host ana-docker (NOT nh3-dev where
+  skaldsong-dev runs for iteration). SHA-pin only for now; health-gated
+  `:latest` advance is a follow-up once `/health` exercises Worldtree
+  + Kokoro reachability.
+
+- `[2026-05-19]` Skaldsong prod (ana-docker) switched from demo
+  Worldtree (:8080) to personal (:8081). Same `user_id=skaldsong`
+  as the nh3-dev hand-launch key — shared Heimdall agent slot
+  (`skaldsong:wizard-v2`), different `key_id`s for independent
+  rotation. Demo Worldtree stays for isolation; personal becomes
+  the multi-consumer dev iteration instance.
+
+- `[2026-05-19]` mead-hall Bifrost v0.3 end-to-end smoke green.
+  Closed task #32 (althing thread `01KRV1M2KW6N6HBEXGTH72QXCA`).
+  Wire layer (handshake + binding + dispatch) + data-flow
+  (per-dispatch JWT claims → `ctx.session_id` populated → real
+  session-scoped data) + agent-loop (LLM reads + quotes back) all
+  proven. Resolves the "stalled mid-Worldtree" state from the
+  2026-05-17 snapshot.
+
+- `[2026-05-18]` Volva systemd install complete after three-stage
+  debug. Final unit at `/etc/systemd/system/volva.service` runs as
+  `User=lkraven` with `ProtectHome=read-only` +
+  `ReadWritePaths=/home/lkraven/.althing /home/lkraven/.codex`
+  carve-outs for state writes. `VOLVA_ALTHING_CLI=/home/lkraven/
+  .local/bin/althing-cli` + `ALTHING_HANDLE=volva` both pinned in
+  env.sh.
+
 - `[2026-05-17]` Phase 3.1 cross-process streaming uses Valkey 8
   alpine as a sibling compose service in `stacks/althing-chamber/`,
   redis-protocol pub/sub for high-volume `msg_delta` / `msg_thinking` /
@@ -179,6 +244,81 @@ _As of 2026-05-17:_
 
 ## Tried and abandoned
 
+- `[2026-05-19]` Naive `docker rmi worldtree:<old-sha> --force`
+  for CD SHA cleanup — would untag pinned/personal worldtree images
+  since all three deployments share corviduo-dev. Use
+  `docker image prune -a --filter "until=Xh"` instead — respects
+  in-use semantic (Docker won't remove an image referenced by any
+  container on the host), so pinned/personal protected automatically.
+
+- `[2026-05-19]` Skaldsong CD first attempt: `docker pull` step
+  failed with 401 unauthorized. ana-docker had no `docker login`
+  for `gitea.phasefinal.com`. My playbook prereq note ("docker login
+  has been done at least once") was an unverified assumption. One-time
+  manual login persists in `~/.docker/config.json`; architectural fix
+  (workflow-side `ssh ana-docker 'docker login ...'` step using
+  `REGISTRY_USER`/`REGISTRY_TOKEN` secrets) flagged as v2.
+
+- `[2026-05-19]` `SKALDSONG_HOST_CORS_ORIGINS=http://10.250.50.70:8300`
+  as a bare URL — pydantic-settings parses complex env vars via
+  `json.loads()`; first-boot crashloop with `SettingsError: error
+  parsing value for field "cors_origins"`. Must be JSON array literal:
+  `SKALDSONG_HOST_CORS_ORIGINS=["http://..."]`.
+
+- `[2026-05-19]` `SKALDSONG_HOST_STATIC_ASSETS_PATH=/app/web/dist`
+  in compose — mismatched Dockerfile reality. The Dockerfile COPYs
+  SvelteKit build output flat into `/app/spa` (not `/app/spa/dist`).
+  Lifted the path from skaldsong-dev's CD-ask message ("`/app/web/dist`")
+  rather than verifying against the actual Dockerfile they shipped.
+  Lesson: when encoding container-internal paths in compose, verify
+  against the Dockerfile, not the design-doc.
+
+- `[2026-05-19]` Playbook verify step `docker ps | grep healthy`
+  racing the container's `start_period` (30s in compose's
+  healthcheck). Verify ran 0.09s after `compose up -d --force-recreate`
+  — well before docker's healthcheck could flip the status from
+  `(health: starting)` to `(healthy)`. False-negative; container was
+  operationally up (the earlier `/health` poll verify already
+  confirmed). Fix: grep `^Up` not `healthy`. /health-200 IS the
+  liveness check; docker's `(healthy)` is just a delayed echo.
+
+- `[2026-05-19]` Volva daemon impersonating volva-dev (the human)
+  for hours of debugging because cwd-based handle resolution.
+  `session_handles.json` maps `/home/lkraven/development/volva` →
+  `volva-dev`; the daemon's systemd unit sets
+  `WorkingDirectory=/home/lkraven/development/volva` and env.sh
+  didn't pin `ALTHING_HANDLE`, so the daemon polled volva-dev's
+  inbox and replied AS volva-dev. Catch: cross-reference journal
+  PIDs against `message_received` events to verify which voice
+  generated which reply. Fix: pin `ALTHING_HANDLE=volva` in env.sh
+  (worldtree-dev's `e39be87` made this the default in
+  env.sh.template). Real follow-on risk: hallucinated structured
+  feedback from impersonating-daemon can bootstrap real (and
+  correct) downstream commits — the technical artifact survives
+  but the conversational attribution rots.
+
+- `[2026-05-18]` Volva env.sh.template `$HOME` in commented examples
+  — systemd's `EnvironmentFile=` parser doesn't expand `$HOME`;
+  uncommenting lands the literal `$HOME/...` string. Volva-dev's
+  `f4dda73` swapped to `/home/<svc-user>/...` placeholders.
+
+- `[2026-05-18]` Initial Volva systemd unit's `ProtectHome=read-only`
+  without `ReadWritePaths=` — althing-cli's SQLite (`~/.althing/
+  althing.db`) and codex's session state (`~/.codex/`) both need to
+  write. Container started but every poll failed with "db path not
+  writable". Surgical fix: `ReadWritePaths=/home/lkraven/.althing
+  /home/lkraven/.codex` (preserves the hardening intent, only carves
+  out the specific dirs).
+
+- `[2026-05-18]` Trusting that env.sh's `export VOLVA_ROOT="$(cd
+  "$(dirname "${BASH_SOURCE[0]}")" && pwd)"` template line works
+  under systemd — `EnvironmentFile=` parser aborts on the first
+  unparseable line (command substitution), and `VOLVA_ALTHING_CLI`
+  declared below silently never lands. Symptom: `Environment=`
+  property empty, daemon error "althing-cli not found at
+  'althing-cli'". Fix: replace command-substitution with literal
+  path. Volva-dev's `d436c3c` dropped VOLVA_ROOT entirely upstream.
+
 - `[2026-05-17]` `--task classify` for Skywork in vLLM 0.19.1 —
   flag was deprecated. Use `--runner pooling`; the model's
   `architectures: [LlamaForSequenceClassification]` in config.json