From 1a0059f5895617ed562311a64d31a0947ac082c7 Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Wed, 20 May 2026 21:51:36 -0700 Subject: [PATCH] =?UTF-8?q?snapshot:=20roll=20persistent-memory.md=20from?= =?UTF-8?q?=202026-05-17=20=E2=86=92=202026-05-20?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Captures the May 18-20 session arc: - Skaldsong CD shipped on ana-docker:8300 via registry-pull pattern (differs from althing/asset-engine which build-on-host); three first-deploy footguns surfaced + canonical-patched. - Worldtree CD disk-hygiene PR #184 (vh/Worldtree) — watermark gate, eager post-deploy SHA prune, set -e hardening in SSH blocks. Approved by worldtree-dev, pending merge. - mead-hall Bifrost v0.3 wire-layer + data-flow end-to-end smoke green (task #32 closed) — resolves the "stalled mid-Worldtree" state in the prior snapshot. - Volva systemd install — three-stage debug (PATH, ReadWritePaths, handle pinning); impersonation footgun caught via PID/journal cross-reference. - Per-project user keys minted against personal Worldtree (:8081) for skaldsong, althing, mead-hall + skaldsong-prod. Personal becomes multi-consumer dev iteration instance; demo stays for isolation. File now at 434 lines (over 300 soft cap); next /snapshot run will surface archival candidates. --- persistent-memory.md | 228 ++++++++++++++++++++++++++++++++++--------- 1 file changed, 184 insertions(+), 44 deletions(-) diff --git a/persistent-memory.md b/persistent-memory.md index 4ce904f..382611c 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -1,6 +1,6 @@ # Persistent memory — eshpfi-management -_Last updated: 2026-05-17_ +_Last updated: 2026-05-20_ ## Repo purpose @@ -22,8 +22,9 @@ Sister repos (separate gitea repos, deployed by playbooks here): | `vh/asset-engine` | Internal control plane over inference services (port 8200, LAN-direct) | push-to-main → CI deploys (2026-05-12) | | `vh/althing` | Inter-agent message bus (chamber UI port 7881, forseti + agent-runner daemons, valkey IPC) | push-to-main → CI deploys (2026-05-14) | | `vh/mead-hall` | Bifrost tool-provider sidecar (port 5173 on dev VM 10.100.10.50) | push-to-main → CI deploys (2026-05-16) | -| `vh/skaldsong` | TBD — first CI workflow shipped 2026-05-17 | push-to-main → CI deploys (2026-05-17) | -| `vh/worldtree` | Conversation API (corviduo-dev, port 8080) — Heimdall auth, Bifrost integration | push-to-main → CI deploys | +| `vh/skaldsong` | Wizard + reader surface (port 8300, ana-docker, registry-pull pattern) | push-to-main → CI deploys (2026-05-19) | +| `vh/worldtree` | Conversation API (corviduo-dev demo :8080 / personal :8081 / pinned :8082) — Heimdall auth, Bifrost integration | push-to-main → CI deploys | +| `vh/volva` | Codex peer agent on althing bus (single-turn oracle, systemd daemon on nh3-dev) | manual install via `deploy/volva.service` (2026-05-18) | - **Two-layer backups** — Backrest orchestrates restic for file+DB (5 fleet repos, daily 01:00 PDT); PBS-ANA primary + PBS-NH3 DR mirror for @@ -35,13 +36,33 @@ Sister repos (separate gitea repos, deployed by playbooks here): model/dataset onto ana-ml2's shared cache at `/tank/aimodels/huggingface/`" playbook. Supports `--var repo_type=model|dataset|space`. Replaces ad-hoc `huggingface_hub.snapshot_download` patterns. -- **Worldtree admin auth** — infra-ops holds a permanent admin-tier - Heimdall key issued 2026-05-17 (`key_id 61419c92`), stored at +- **Worldtree admin auth — per-instance.** Each Worldtree deployment + (demo :8080, personal :8081, pinned :8082) has its own Heimdall + registry and its own bootstrap admin key. Infra-ops's stored + long-lived admin key (`key_id 61419c92`) at `ana-docker:/opt/docker/conf/.secrets/worldtree-infra-ops-admin` - mode 600. Used for `POST /admin/keys`, admin diagnostic reads - (`/admin/sessions//{bifrost,tools}`, etc.), and any other - Worldtree admin work. Replaces the docker-as-root + bootstrap-admin - pattern that was in use 2026-05-12 through 2026-05-16. + auths against **demo only**. For personal-instance admin ops, fetch + the bootstrap admin per-op via + `docker exec worldtree-personal-worldtree-api-1 printenv WORLDTREE_BOOTSTRAP_ADMIN_KEY` + on corviduo-dev. Used for `POST /admin/keys`, admin diagnostics + (`/admin/sessions//{bifrost,tools}`, etc.). + +- **Per-project user keys against personal Worldtree** (issued + 2026-05-19): `skaldsong:79744637` (nh3-dev iteration), + `skaldsong:7c1dbbbe` (ana-docker prod), `althing:50d85460`, + `mead-hall:a360822d`. Same `user_id=skaldsong` across both + skaldsong keys → shared Heimdall agent slot; different `key_id` + → independently rotatable. Pattern: mint via `/admin/keys`, drop + value to `/tmp/wt-personal-.key` mode 600, dev collects + + shreds (DO NOT cat to chat transcript). + +- **Skaldsong CD pattern (registry-pull).** Differs from althing / + asset-engine which build-on-host. vh/skaldsong's CI builds and + pushes `gitea.phasefinal.com/vh/skaldsong:` + `:latest`; + `playbooks/deploy-skaldsong.yaml` on ana-docker pulls + recreates. + SHA-pin only (no `:latest` health-gated advance yet). Prereq: host + needs `docker login gitea.phasefinal.com` once (read:package PAT) — + not currently in the workflow. - **docker-as-root pattern** (for ops that have no admin API, e.g. `SqliteUserStore.set_bifrost_credentials`): on hosts where the SSH @@ -53,50 +74,94 @@ Sister repos (separate gitea repos, deployed by playbooks here): ## Current state / in-flight -_As of 2026-05-17:_ +_As of 2026-05-20:_ -- **mead-hall Bifrost integration stalled mid-Worldtree.** Bifrost - consumer registration done from infra side (HS256, `set_bifrost_credentials` via docker-as-root + admin key, secret-drop pattern), `BIFROST_CLIENT_ALLOWED_HOSTS` env wired through to corviduo-dev's container. But the actual tracer dispatch shows - `session_not_bifrost_bound` per `/admin/sessions//bifrost` — the - handshake doesn't register bfclient into `_bifrost_clients[]`. - worldtree-dev investigating; not infra-side work. +- **Worldtree CD disk-hygiene PR #184 approved, ready for merge.** + Three commits on `infra/disk-watermark-gate-cd` in vh/Worldtree: + watermark gate (pre-pull recovery), eager post-deploy SHA prune + (post-`:latest`-advance), `set -e` hardening in SSH blocks. Pending + worldtree-dev's merge button. Follow-up extension to + `deploy-personal.yml` + `deploy-pinned.yml` + Beszel disk-usage + metric all queued, not bundled. -- **althing-chamber stack at 4 services.** chamber (port 7881) + - forseti (moderator daemon) + agent-runner (Phase 2 worldtree-driver - dispatcher) + valkey (Phase 3.1 pub/sub bridge for cross-process - streaming). Phase 3.1 verified end-to-end via event histogram on - 2026-05-17. Forseti is the canonical dev for this stack (galdrabok - is on a different project). +- **Skaldsong live on ana-docker:8300 (registry-pull pattern), pointed + at personal Worldtree (:8081).** Three first-deploy footguns + surfaced + canonical-patched: SPA path mismatch (`/app/web/dist` → + `/app/spa`), CORS env shape (pydantic-settings expects JSON array), + verify-step race vs healthcheck `start_period` (grep `^Up` not + `healthy`). nh3-dev hand-launched skaldsong on :8080 also still + running for dev iteration — same user_id=skaldsong, different + key_id, both authorized. -- **Worldtree admin workflow shift** (2026-05-17, per vh): infra-ops - uses its own permanent admin-tier key for admin operations rather - than reaching into bootstrap admin via docker-as-root. Admin ops - with HTTP endpoints go over HTTPS; ops without (like - `set_bifrost_credentials`) still use docker-as-root but with my own - auth, not borrowed. +- **Worldtree personal (:8081) is the iteration-target instance.** + skaldsong + mead-hall + althing-chamber all rotated to personal + via the per-project user keys minted 2026-05-19. Demo (:8080) + stays as the canonical prod surface; pinned (:8082) intentionally + lags. Personal's `BIFROST_CLIENT_ALLOWED_HOSTS` now mirrors demo's: + `10.100.10.50:5173,10.100.10.50:8080,10.250.50.70:8300`. -- **Open hygiene debt** — these keys/tokens were disclosed across the - althing bus and/or in CC transcripts during this session and warrant - rotation at convenience: - - Gitea runner registration token (`a1135753...`) — rotate via Gitea - admin UI's runner-token reset. - - mead-hall's Worldtree bearer (`wt_live_80e1570620ef2aba998dc63954cce3a6`) — tracer issuance 2026-05-16. - - Worldtree provider keys `Z_AI_API_KEY` + `ZAI_API_KEY` — from the - 2026-05-12 corviduo-dev outage. +- **Disclosed-keys hygiene queue** — these warrant rotation at + convenience (mostly in-bus disclosure during this session): + - `/tmp/wt-personal-skaldsong-prod.key` on nh3-dev — operator + should shred after verify (file mode 600, lkraven-owned) + - mead-hall's prior Worldtree bearer + (`wt_live_80e1570620ef2aba998dc63954cce3a6`) — now superseded + by personal-instance key `a360822d` per the 2026-05-19 rotation + - Worldtree provider keys `Z_AI_API_KEY` + `ZAI_API_KEY` — from + the 2026-05-12 corviduo-dev outage - chamber config.yaml's `forseti.api_key` + `agent_runner.api_key` - (Worldtree user-tier keys) — exposed when grepping config to add - `streaming_enabled` 2026-05-17. + — superseded by personal-instance key `50d85460` 2026-05-19 + - Gitea runner registration token (`a1135753...`) — rotate via + Gitea admin UI's runner-token reset -- **Still open from prior session** (2026-05-08, unchanged): rotate - `MINIFLUX_PASSWORD` (leaked twice); clean up legacy `news-digest` - detritus on ana-docker (`/opt/docker/compose/news-digest/`, - `/opt/docker/data/news-digest/`, image `local/news-digest:v5`); - watch nh3-nas `/volume1` (was 65%; recheck retention or expand - before ~80%); the `docker push 60s client-side ceiling` mystery - remains uninstrumented. +- **Still open from prior sessions:** rotate `MINIFLUX_PASSWORD` + (leaked twice); clean up legacy `news-digest` detritus on ana-docker + (`/opt/docker/compose/news-digest/`, `/opt/docker/data/news-digest/`, + image `local/news-digest:v5`); watch nh3-nas `/volume1` (was 65%; + recheck retention or expand before ~80%); the `docker push 60s + client-side ceiling` mystery remains uninstrumented. ## Recent decisions +- `[2026-05-19]` Worldtree CD disk-hygiene strategy: watermark gate + (env-tunable threshold + window, fail-loud on still-low post-prune) + + eager post-deploy prune (only after `:latest` advance succeeds, + uses `docker image prune -a --filter "until=24h"` which respects + in-use semantic — protects pinned + personal images automatically). + Combined: demo VM holds ~24h of deploy history instead of unbounded + accumulation. Shipped in vh/Worldtree PR #184 (`306cd61` + + `613dac2` + `bd91df5`). + +- `[2026-05-19]` Skaldsong CD shape: shape (1) of three operator + options — container + Gitea registry + pull-restart, matching + Worldtree's pattern. Target host ana-docker (NOT nh3-dev where + skaldsong-dev runs for iteration). SHA-pin only for now; health-gated + `:latest` advance is a follow-up once `/health` exercises Worldtree + + Kokoro reachability. + +- `[2026-05-19]` Skaldsong prod (ana-docker) switched from demo + Worldtree (:8080) to personal (:8081). Same `user_id=skaldsong` + as the nh3-dev hand-launch key — shared Heimdall agent slot + (`skaldsong:wizard-v2`), different `key_id`s for independent + rotation. Demo Worldtree stays for isolation; personal becomes + the multi-consumer dev iteration instance. + +- `[2026-05-19]` mead-hall Bifrost v0.3 end-to-end smoke green. + Closed task #32 (althing thread `01KRV1M2KW6N6HBEXGTH72QXCA`). + Wire layer (handshake + binding + dispatch) + data-flow + (per-dispatch JWT claims → `ctx.session_id` populated → real + session-scoped data) + agent-loop (LLM reads + quotes back) all + proven. Resolves the "stalled mid-Worldtree" state from the + 2026-05-17 snapshot. + +- `[2026-05-18]` Volva systemd install complete after three-stage + debug. Final unit at `/etc/systemd/system/volva.service` runs as + `User=lkraven` with `ProtectHome=read-only` + + `ReadWritePaths=/home/lkraven/.althing /home/lkraven/.codex` + carve-outs for state writes. `VOLVA_ALTHING_CLI=/home/lkraven/ + .local/bin/althing-cli` + `ALTHING_HANDLE=volva` both pinned in + env.sh. + - `[2026-05-17]` Phase 3.1 cross-process streaming uses Valkey 8 alpine as a sibling compose service in `stacks/althing-chamber/`, redis-protocol pub/sub for high-volume `msg_delta` / `msg_thinking` / @@ -179,6 +244,81 @@ _As of 2026-05-17:_ ## Tried and abandoned +- `[2026-05-19]` Naive `docker rmi worldtree: --force` + for CD SHA cleanup — would untag pinned/personal worldtree images + since all three deployments share corviduo-dev. Use + `docker image prune -a --filter "until=Xh"` instead — respects + in-use semantic (Docker won't remove an image referenced by any + container on the host), so pinned/personal protected automatically. + +- `[2026-05-19]` Skaldsong CD first attempt: `docker pull` step + failed with 401 unauthorized. ana-docker had no `docker login` + for `gitea.phasefinal.com`. My playbook prereq note ("docker login + has been done at least once") was an unverified assumption. One-time + manual login persists in `~/.docker/config.json`; architectural fix + (workflow-side `ssh ana-docker 'docker login ...'` step using + `REGISTRY_USER`/`REGISTRY_TOKEN` secrets) flagged as v2. + +- `[2026-05-19]` `SKALDSONG_HOST_CORS_ORIGINS=http://10.250.50.70:8300` + as a bare URL — pydantic-settings parses complex env vars via + `json.loads()`; first-boot crashloop with `SettingsError: error + parsing value for field "cors_origins"`. Must be JSON array literal: + `SKALDSONG_HOST_CORS_ORIGINS=["http://..."]`. + +- `[2026-05-19]` `SKALDSONG_HOST_STATIC_ASSETS_PATH=/app/web/dist` + in compose — mismatched Dockerfile reality. The Dockerfile COPYs + SvelteKit build output flat into `/app/spa` (not `/app/spa/dist`). + Lifted the path from skaldsong-dev's CD-ask message ("`/app/web/dist`") + rather than verifying against the actual Dockerfile they shipped. + Lesson: when encoding container-internal paths in compose, verify + against the Dockerfile, not the design-doc. + +- `[2026-05-19]` Playbook verify step `docker ps | grep healthy` + racing the container's `start_period` (30s in compose's + healthcheck). Verify ran 0.09s after `compose up -d --force-recreate` + — well before docker's healthcheck could flip the status from + `(health: starting)` to `(healthy)`. False-negative; container was + operationally up (the earlier `/health` poll verify already + confirmed). Fix: grep `^Up` not `healthy`. /health-200 IS the + liveness check; docker's `(healthy)` is just a delayed echo. + +- `[2026-05-19]` Volva daemon impersonating volva-dev (the human) + for hours of debugging because cwd-based handle resolution. + `session_handles.json` maps `/home/lkraven/development/volva` → + `volva-dev`; the daemon's systemd unit sets + `WorkingDirectory=/home/lkraven/development/volva` and env.sh + didn't pin `ALTHING_HANDLE`, so the daemon polled volva-dev's + inbox and replied AS volva-dev. Catch: cross-reference journal + PIDs against `message_received` events to verify which voice + generated which reply. Fix: pin `ALTHING_HANDLE=volva` in env.sh + (worldtree-dev's `e39be87` made this the default in + env.sh.template). Real follow-on risk: hallucinated structured + feedback from impersonating-daemon can bootstrap real (and + correct) downstream commits — the technical artifact survives + but the conversational attribution rots. + +- `[2026-05-18]` Volva env.sh.template `$HOME` in commented examples + — systemd's `EnvironmentFile=` parser doesn't expand `$HOME`; + uncommenting lands the literal `$HOME/...` string. Volva-dev's + `f4dda73` swapped to `/home//...` placeholders. + +- `[2026-05-18]` Initial Volva systemd unit's `ProtectHome=read-only` + without `ReadWritePaths=` — althing-cli's SQLite (`~/.althing/ + althing.db`) and codex's session state (`~/.codex/`) both need to + write. Container started but every poll failed with "db path not + writable". Surgical fix: `ReadWritePaths=/home/lkraven/.althing + /home/lkraven/.codex` (preserves the hardening intent, only carves + out the specific dirs). + +- `[2026-05-18]` Trusting that env.sh's `export VOLVA_ROOT="$(cd + "$(dirname "${BASH_SOURCE[0]}")" && pwd)"` template line works + under systemd — `EnvironmentFile=` parser aborts on the first + unparseable line (command substitution), and `VOLVA_ALTHING_CLI` + declared below silently never lands. Symptom: `Environment=` + property empty, daemon error "althing-cli not found at + 'althing-cli'". Fix: replace command-substitution with literal + path. Volva-dev's `d436c3c` dropped VOLVA_ROOT entirely upstream. + - `[2026-05-17]` `--task classify` for Skywork in vLLM 0.19.1 — flag was deprecated. Use `--runner pooling`; the model's `architectures: [LlamaForSequenceClassification]` in config.json