snapshot: roll persistent-memory.md from 2026-05-08 → 2026-05-17

Captures 9 days of multi-stack work:
  - asset-engine catalog (lifecycle blocks for orchestrator,
    defaults audit, voice probe of voxtral/qwen3-tts/kyutai-tts)
  - asset-engine stack scaffold + LAN-direct deploy
  - corviduo-dev inventory (Worldtree-team dev VM)
  - Worldtree outage recovery + :latest tag drift fix
  - vor key injection
  - vllm-qwen3 → vllm rename + vllm-reward (Skywork-Reward-V2 AWQ)
  - pull-hf-repo.yaml (model+dataset+space puller)
  - selene-1-mini-8b judge model in llama-swap
  - /tend-docs first pass (delete infinity, archive design-brief,
    fix pfi-pve VM list, drop stale postgres-rotate section)
  - althing-chamber stack (chamber + forseti + agent-runner + valkey
    across Phase 1/2/3.1)
  - Worldtree admin workflow shift (infra-ops permanent admin key
    replaces docker-as-root + bootstrap admin)
  - mead-hall Bifrost consumer registration
  - skaldsong CI bootstrap (3 captured gotchas)
  - Phase 3 + Phase 3.1 streaming activation

Sections grew: Tools and conventions (+sister-repo rows, admin auth
note, docker-as-root pattern), Recent decisions (+12), Tried and
abandoned (+5). Current state / in-flight rewritten to today's
posture (Worldtree handshake gap, hygiene-debt rotation list, prior
open items still active).

294 lines total; under the 300 soft cap. No archival this run.

persistent-memory.md

@@ -1,6 +1,6 @@
 # Persistent memory — eshpfi-management
 
-_Last updated: 2026-05-08_
+_Last updated: 2026-05-17_
 
 ## Repo purpose
 
@@ -19,6 +19,11 @@ Sister repos (separate gitea repos, deployed by playbooks here):
 | `vh/task-board` | MCP + web dashboard for assistant task state (port 7878) | push-to-main → CI deploys (2026-04-29) |
 | `vh/vor` | Inquisitor UI sidecar (port 7879) | push-to-main → CI deploys (2026-04-29) |
 | `vh/nevermore` | Twice-daily LLM-curated briefing (port 8181, replaces news-digest) | push-to-main → CI deploys (2026-04-30) |
+| `vh/asset-engine` | Internal control plane over inference services (port 8200, LAN-direct) | push-to-main → CI deploys (2026-05-12) |
+| `vh/althing` | Inter-agent message bus (chamber UI port 7881, forseti + agent-runner daemons, valkey IPC) | push-to-main → CI deploys (2026-05-14) |
+| `vh/mead-hall` | Bifrost tool-provider sidecar (port 5173 on dev VM 10.100.10.50) | push-to-main → CI deploys (2026-05-16) |
+| `vh/skaldsong` | TBD — first CI workflow shipped 2026-05-17 | push-to-main → CI deploys (2026-05-17) |
+| `vh/worldtree` | Conversation API (corviduo-dev, port 8080) — Heimdall auth, Bifrost integration | push-to-main → CI deploys |
 
 - **Two-layer backups** — Backrest orchestrates restic for file+DB (5
   fleet repos, daily 01:00 PDT); PBS-ANA primary + PBS-NH3 DR mirror for
@@ -26,74 +31,199 @@ Sister repos (separate gitea repos, deployed by playbooks here):
   cross-site restic targets — see `docs/runbooks/disaster-recovery.md`
   for the blast-radius matrix.
 
+- **`pull-hf-repo.yaml`** is the canonical "get a HuggingFace
+  model/dataset onto ana-ml2's shared cache at
+  `/tank/aimodels/huggingface/`" playbook. Supports `--var repo_type=model|dataset|space`. Replaces ad-hoc `huggingface_hub.snapshot_download` patterns.
+
+- **Worldtree admin auth** — infra-ops holds a permanent admin-tier
+  Heimdall key issued 2026-05-17 (`key_id 61419c92`), stored at
+  `ana-docker:/opt/docker/conf/.secrets/worldtree-infra-ops-admin`
+  mode 600. Used for `POST /admin/keys`, admin diagnostic reads
+  (`/admin/sessions/<id>/{bifrost,tools}`, etc.), and any other
+  Worldtree admin work. Replaces the docker-as-root + bootstrap-admin
+  pattern that was in use 2026-05-12 through 2026-05-16.
+
+- **docker-as-root pattern** (for ops that have no admin API, e.g.
+  `SqliteUserStore.set_bifrost_credentials`): on hosts where the SSH
+  user is in the `docker` group but lacks passwordless sudo, run
+  `docker run --rm -v <target-dir>:/wt -v /var/run/docker.sock:/var/run/docker.sock docker:cli sh -c "..."` to edit deploy-owned files
+  without sudo. Documented with security warning in
+  `servers/corviduo-dev/README.md`. docker-group membership is
+  effectively root via bind-mount; treat as a sudo-equivalent grant.
+
 ## Current state / in-flight
 
-_As of 2026-05-08:_
+_As of 2026-05-17:_
 
-- **CI/CD pipe is live for three repos** (task-board, vor, nevermore).
+- **mead-hall Bifrost integration stalled mid-Worldtree.** Bifrost
-  Pattern: workflow checks out triggering repo + `vh/esh-pfi-infrastructure`
+  consumer registration done from infra side (HS256, `set_bifrost_credentials` via docker-as-root + admin key, secret-drop pattern), `BIFROST_CLIENT_ALLOWED_HOSTS` env wired through to corviduo-dev's container. But the actual tracer dispatch shows
-  (for elway), SSHes to ana-docker, runs an elway playbook pinned to the
+  `session_not_bifrost_bound` per `/admin/sessions/<id>/bifrost` — the
-  triggering commit SHA. User-scope secrets `DEPLOY_SSH_KEY` and
+  handshake doesn't register bfclient into `_bifrost_clients[]`.
-  `MGMT_REPO_TOKEN` cover all repos under `vh/`.
+  worldtree-dev investigating; not infra-side work.
-- **Open**: rotate `MINIFLUX_PASSWORD` — leaked in a prior session's
+
-  transcript while seeding nevermore's `.env` (full leak once, partial
+- **althing-chamber stack at 4 services.** chamber (port 7881) +
-  leak once).
+  forseti (moderator daemon) + agent-runner (Phase 2 worldtree-driver
-- **Open**: clean up old `news-digest` detritus on ana-docker —
+  dispatcher) + valkey (Phase 3.1 pub/sub bridge for cross-process
-  `/opt/docker/compose/news-digest/`, `/opt/docker/data/news-digest/`,
+  streaming). Phase 3.1 verified end-to-end via event histogram on
-  image `local/news-digest:v5`. Migrate the historical `edition-*.html`
+  2026-05-17. Forseti is the canonical dev for this stack (galdrabok
-  archive files into `/opt/docker/data/nevermore/` first.
+  is on a different project).
-- **Watch**: nh3-nas `/volume1` at 65%; plan retention review or
+
-  capacity expansion before ~80%.
+- **Worldtree admin workflow shift** (2026-05-17, per vh): infra-ops
-- **Possible next**: aggregator service + homepage widget for live
+  uses its own permanent admin-tier key for admin operations rather
-  "fleet backup health" dashboard. Discussed but not committed; user
+  than reaching into bootstrap admin via docker-as-root. Admin ops
-  weighing against a `/schedule` weekly digest as the lighter
+  with HTTP endpoints go over HTTPS; ops without (like
-  alternative.
+  `set_bifrost_credentials`) still use docker-as-root but with my own
-- **Open mystery**: `docker push` from outside the LAN to
+  auth, not borrowed.
-  `gitea.phasefinal.com` has a **~60s client-side per-PATCH ceiling**
+
-  for chunked blob uploads. We confirmed the timer is client-side
+- **Open hygiene debt** — these keys/tokens were disclosed across the
-  (Traefik logged `499 / 60012ms` — client closed; Gitea logged the
+  althing bus and/or in CC transcripts during this session and warrant
-  same as `unexpected EOF`). Where exactly the 60s lives in the docker
+  rotation at convenience:
-  daemon / containerd stack — and whether it's tunable — is
+  - Gitea runner registration token (`a1135753...`) — rotate via Gitea
-  unidentified. Worked around for now by shrinking images below the
+    admin UI's runner-token reset.
-  cliff; if it bites again, instrument with `dockerd -D` + strace on a
+  - mead-hall's Worldtree bearer (`wt_live_80e1570620ef2aba998dc63954cce3a6`) — tracer issuance 2026-05-16.
-  fresh push to find the actual timer.
+  - Worldtree provider keys `Z_AI_API_KEY` + `ZAI_API_KEY` — from the
+    2026-05-12 corviduo-dev outage.
+  - chamber config.yaml's `forseti.api_key` + `agent_runner.api_key`
+    (Worldtree user-tier keys) — exposed when grepping config to add
+    `streaming_enabled` 2026-05-17.
+
+- **Still open from prior session** (2026-05-08, unchanged): rotate
+  `MINIFLUX_PASSWORD` (leaked twice); clean up legacy `news-digest`
+  detritus on ana-docker (`/opt/docker/compose/news-digest/`,
+  `/opt/docker/data/news-digest/`, image `local/news-digest:v5`);
+  watch nh3-nas `/volume1` (was 65%; recheck retention or expand
+  before ~80%); the `docker push 60s client-side ceiling` mystery
+  remains uninstrumented.
 
 ## Recent decisions
 
-- `[2026-05-08]` Pin `torch` to CPU-only via
+- `[2026-05-17]` Phase 3.1 cross-process streaming uses Valkey 8
-  `--index-url https://download.pytorch.org/whl/cpu` in
+  alpine as a sibling compose service in `stacks/althing-chamber/`,
-  `Worldtree/Dockerfile` before the rest of `requirements.txt`.
+  redis-protocol pub/sub for high-volume `msg_delta` / `msg_thinking` /
-  `sentence-transformers` transitively pulls torch and grabs the CUDA
+  `msg_start` / `msg_complete` event kinds. DB bridge keeps
-  flavor by default (~2.5GB of nvidia libs). Worldtree runs on
+  `msg_curated` + `floor_grant` (structured / canonical). Two-channel
-  `ana-docker` (no GPU reservation in `compose.yaml`) so CUDA is dead
+  architecture, no overlap. chamber + agent-runner `depends_on:
-  weight. Image: 5.62GB → 1.6GB. The failing push from off-LAN now
+  valkey: service_healthy`.
-  succeeds.
+
-- `[2026-05-08]` Split `Worldtree/requirements.txt` into runtime +
+- `[2026-05-17]` Worldtree admin workflow shift (per vh): infra-ops
-  `requirements-dev.txt` (pytest, pytest-asyncio, ruff, pylint).
+  gets its own permanent admin-tier key (`61419c92`, stored at
-  ~50MB out of the runtime image; smaller attack surface.
+  `ana-docker:/opt/docker/conf/.secrets/worldtree-infra-ops-admin`).
-- `[2026-05-01]` nevermore **replaces** news-digest at port 8181 (not
+  Future admin ops route through this key, not the bootstrap admin
-  coexist on 8182). User chose one stack at a time over soak window.
+  via docker-as-root.
-  ~30s briefing gap accepted as the cost.
+
-- `[2026-04-30]` Bumped nevermore deploy playbook's first-render wait
+- `[2026-05-17]` Worldtree env-var addition checklist: anytime
-  from 60s to 6 min — first deploy with empty `/output` requires a
+  introducing `os.environ.get("FOO")` in worldtree code, update BOTH
-  worker LLM cycle (2-4 min) before `index.html` exists.
+  `.env.example` AND `compose.yaml`'s `&worldtree-env` anchor in the
-- `[2026-04-29]` Workflow drops explicit `container:` directive after
+  same PR. Same Z_AI_API_KEY-shape footgun bit
-  runner re-registration with `:docker://node:20-bookworm-slim` schema
+  `BIFROST_CLIENT_ALLOWED_HOSTS` (#170) until worldtree-dev added the
-  labels — single source of truth for the build environment.
+  passthrough line in `08f02b2`.
-- `[2026-04-29]` Deploy playbooks accept SHA refs in addition to branch
+
-  names (`git rev-parse <ref>` then `origin/<ref>^{commit}` fallback).
+- `[2026-05-16]` althing-chamber Phase 2: added `althing-agent-runner`
-  CI passes `--var ref=${{ github.sha }}`; manual runs pass `main` /
+  as third compose service (worldtree-driver agent dispatcher).
-  `v0.1.0`. Same code path either way.
+  All three althing services use the same image; `command:` selects
-- `[2026-04-29]` Single central Gitea Actions runner on ana-docker
+  entrypoint. Safe to enable preemptively (sleeps when no
-  (`f014d55`). Parameterized `playbooks/deploy-gitea-runner.yaml` so
+  driver=worldtree handles declared).
-  site-local runners (nh3, esh) drop in via `--var` overrides — not
+
-  copy-pasted playbooks.
+- `[2026-05-14]` althing-chamber stack scaffolded: chamber + forseti.
-- `[2026-04-29]` `nevermore` carries forward `hidden.json` and
+  Internal LAN-only at port 7881 (chamber default 7878 collides with
-  historical `MINIFLUX_PASSWORD` from `news-digest` during cutover
+  task-board). Two-service compose, shared SQLite bind-mount, build-on-host pattern via vh/althing's gitea-workflow. Forseti is the
-  (preserves user state across stack rename).
+  canonical dev for this stack (galdrabok is on a different project).
+
+- `[2026-05-13]` `vllm-qwen3` → `vllm` stack rename. Added
+  `vllm-reward` service (Skywork-Reward-V2-Llama-3.1-8B-AWQ classifier).
+  Three vLLM services share GPU 1 (embed 0.20, rerank 0.20, reward 0.30
+  utilization; 30% headroom). All use `--runner pooling`; classification
+  drives via model's `architectures: [LlamaForSequenceClassification]`
+  in config.json, NOT `--task classify` (deprecated in vLLM 0.19.1).
+
+- `[2026-05-13]` `pull-hf-repo.yaml` is the canonical HF-fetch
+  playbook on ana-ml2. Supports `--var repo_type=model|dataset|space`.
+  Replaces ad-hoc `huggingface_hub.snapshot_download` calls.
+
+- `[2026-05-13]` Selene-1-Mini-Llama-3.1-8B added to llama-swap as
+  judge model. mradermacher `i1-Q6_K` imatrix quant (~6.5GB). AtlaAI
+  reward/eval model — temp 0.01, ctx 32K, q8_0 KV cache. New JUDGE /
+  EVAL MODELS section in `stacks/llama-swap/conf/config.yaml`.
+
+- `[2026-05-13]` /tend-docs first pass deletions: `stacks/infinity/`
+  removed (retired by vllm). Archived
+  `docs/asset-engine/design-brief.md` → `docs/archive/asset-engine/`
+  with archival header. Fixed `pfi-pve` VM list to full
+  `qm list` enumeration. Dropped stale weak-password section from
+  `pfi-postgres` (rotation done 2026-04-23).
+
+- `[2026-05-12]` corviduo-dev (Worldtree-team dev VM, 10.250.50.152,
+  CT 106 on pfi-pve) added to `servers/` inventory. Treat like SF
+  client hosts: PFI hosts + provides emergency-ops backstop;
+  Worldtree team owns OS config + deploys + backup decisions.
+
+- `[2026-05-12]` Worldtree `:latest` tag drift bug — fixed by
+  health-gated `:latest` advance in vh/worldtree's deploy workflow
+  (architect commit `8ef3801`): only tag `:latest` AFTER the new
+  container's `/health` probe passes. Build-on-host stacks here
+  don't have this problem because the playbook always builds the
+  SHA-tagged image from a `git reset --hard <ref>` checkout.
+
+- `[2026-05-12]` asset-engine stack scaffolded LAN-direct at
+  `http://10.250.50.70:8200`. Initially included Traefik labels for
+  public hostname; user pulled them out (internal tool, no public
+  TLS surface needed). Pattern: internal tools default LAN-direct;
+  Traefik wiring only when external/TLS required.
+
+- `[2026-05-12]` asset-engine catalog gains `lifecycle: { stack,
+  vram_gb, gpu_device_id }` per irv-ml1 service for the
+  orchestrator feature. SSH keypair scaffolded at
+  `ana-docker:/opt/docker/conf/asset-engine/ssh/` for
+  asset-engine container → irv-ml1 orchestration via dedicated
+  ed25519 key.
 
 ## Tried and abandoned
 
+- `[2026-05-17]` `--task classify` for Skywork in vLLM 0.19.1 —
+  flag was deprecated. Use `--runner pooling`; the model's
+  `architectures: [LlamaForSequenceClassification]` in config.json
+  drives the classification head. Surfaced as `vllm: error:
+  unrecognized arguments: --task classify` in container logs.
+
+- `[2026-05-17]` Trusting that `.env` edit alone propagates a new env
+  var into a worldtree container — `compose.yaml`'s `&worldtree-env`
+  anchor must explicitly declare the passthrough or the value silently
+  doesn't land. Same footgun bit `Z_AI_API_KEY` (2026-05-12) AND
+  `BIFROST_CLIENT_ALLOWED_HOSTS` (2026-05-17). Cost ~10 min of "why is
+  env empty?" diagnosis each time. Worldtree-side fix in
+  `vh/worldtree@08f02b2`.
+
+- `[2026-05-17]` `--force-recreate --pull never` from the docker:cli
+  sandbox without explicit `-e WORLDTREE_IMAGE=<sha>` re-pins the
+  container to `:latest`, even when a newer SHA-tagged image is on
+  disk. Symptom: container "recreated" but actually reverted to a
+  stale image. Pass `-e WORLDTREE_IMAGE=...:<sha>` to the docker run
+  invocation. Worldtree-dev's `8ef3801` health-gated `:latest`
+  advance is the long-term fix.
+
+- `[2026-05-13]` Initial Voxtral default voice `alloy` (OpenAI-compat
+  naming) — vLLM-Omni serving Voxtral does NOT translate aliases.
+  Native presets are `<register>_<gender>` shape (`neutral_female`,
+  `casual_male`, etc.). Always live-probe `/v1/audio/voices` for the
+  exact wrapper-deployed preset names before setting a catalog
+  default. Same caveat for Qwen3-TTS (wrapper exposes 15 voices: 9
+  Qwen presets + 6 OpenAI aliases) and Kyutai-TTS (NillPointer
+  wrapper has NO voice-listing endpoint at all; voices are
+  filesystem paths under the `kyutai/tts-voices` HF repo).
+
+- `[2026-05-12]` Defaulting asset-engine to Traefik-routed
+  (`asset-engine.phasefinal.com` with `anaprod` cert resolver) on
+  first scaffold — user pulled it back to LAN-direct. Internal
+  tools default LAN-direct; only add Traefik when an external/TLS
+  surface is actually needed.
+
+- `[2026-05-12]` Routing althing thread replies through `galdrabok`
+  when the actual dev handle is `forseti` — bus rejected
+  `to=forseti` initially because thread participants list was
+  `[galdrabok, infra]`. Solved by starting a new thread with
+  `forseti` as the direct recipient. Lesson: when the bus auto-resolves a sender handle that doesn't match the actual dev role,
+  start a fresh thread rather than fighting the participant list.
+
 - `[2026-05-08]` Filtering Traefik's UTC access log by Gitea-local-PDT
   timestamp substrings (`grep "2026/05/08 15:1[2-7]"`) returned zero
   matches and led to a wrong "no /v2/ traffic in 12 days" conclusion.
@@ -101,6 +231,7 @@ _As of 2026-05-08:_
   timezones. Always normalize timezones (UTC) when correlating logs
   across services on the same box. Cost: ~30 min in the wrong
   direction.
+
 - `[2026-05-08]` Validating a user-proposed Traefik/Gitea timeout bump
   for "Traefik is dropping connections on big docker pushes" without
   first verifying which component was actually in the failure path.
@@ -110,17 +241,21 @@ _As of 2026-05-08:_
   client-side timer no compose change can reach. **Lesson: validate
   the diagnostic premise — which component is actually in the failure
   path? — before refining the proposed fix.**
+
 - `[2026-05-08]` Bumping Gitea `PER_WRITE_TIMEOUT` /
   `PER_WRITE_PER_KB_TIMEOUT` to address `unexpected EOF` on
   `/v2/.../blobs/uploads/` PATCH — wrong direction. Both govern
   **response writes**, not request body reads. `unexpected EOF` from
   Go's HTTP server means the client closed mid-body-upload; not a
   knob Gitea exposes server-side.
+
 - `[2026-04-30]` task-board workflow with
   `container: image: debian:bookworm-slim` — fails:
   `actions/checkout@v4` needs `node` at runtime, slim image lacks it.
   Switched to `node:20-bookworm-slim` (has node + apt) or runner-label
-  default.
+  default. (Pattern revisited 2026-05-17 for skaldsong-dev: container
+  override needs `nodejs` apt-installed unless it IS the default.)
+
 - `[2026-04-30]` Dropping the `container:` directive before runner
   re-registration with docker-schema labels — runner silently falls
   back to **host mode** (jobs run inside the alpine `act_runner`
@@ -128,23 +263,27 @@ _As of 2026-05-08:_
   (`labels updated to: [pfi-fleet:host ana-docker:host]`) is the
   giveaway. Fix: register with `pfi-fleet:docker://<image>` schema
   labels.
+
 - `[2026-04-30]` Updating runner labels by editing `.env` and bouncing
   — doesn't take. The `.runner` registration cache pins labels at
   first registration; env-var updates are read each start but the
   stored token + UUID are tied to the original label set on the gitea
   side. Fix: stop runner, delete `.runner`, generate new admin
   registration token, redeploy.
+
 - `[2026-04-30]` `git reset --hard origin/<sha>` in
   `deploy-task-board.yaml` (and the in-repo nevermore playbook before
   fix) — invalid syntax: `origin/` prefix only works for branch refs.
   SHAs need `git reset --hard <sha>` directly. Resolved with
   `git rev-parse --verify --quiet "origin/{{ ref }}^{commit}"` first,
   then bare `"{{ ref }}^{commit}"` fallback.
+
 - `[2026-04-30]` Assuming `DEPLOY_SSH_KEY` was at user scope after
   task-board wiring — it was actually only repo-scope on
   `vh/task-board`. vor's first CI run failed with empty SSH key
   (`printf '%s\n' "" > ~/.ssh/id_ed25519`). Fix: copy secret to user
   scope at `gitea.phasefinal.com/user/settings/actions/secrets`.
+
 - `[2026-04-30]` `grep -vE "^(#|$)"` to inspect `.env` for sanity —
   leaked the full `MINIFLUX_PASSWORD` line into the transcript. Then a
   follow-up redaction attempt with `sed -E "s/=(.{4}).*$/=\1<redacted>/"`