esh-pfi-infrastructure/persistent-memory.md

# Persistent memory — eshpfi-management

_Last updated: 2026-06-04_

## Repo purpose

Reference workspace for PFI infrastructure: server inventory, canonical
Docker Compose stacks, ops playbooks, and conventions. Authoritative
copies of compose files live on the servers under
`/opt/docker/compose/<stack>/`; this repo mirrors them for version
control, editing, planning, and CI-driven deploys.

## Tools and conventions

Sister repos (separate gitea repos, deployed by playbooks here):

| Repo | Role | CI status |
|---|---|---|
| `vh/task-board` | MCP + web dashboard for assistant task state (port 7878) | push-to-main → CI deploys (2026-04-29) |
| `vh/vor` | Inquisitor UI sidecar (port 7879) | push-to-main → CI deploys (2026-04-29) |
| `vh/nevermore` | Twice-daily LLM-curated briefing (port 8181, replaces news-digest) | push-to-main → CI deploys (2026-04-30) |
| `vh/asset-engine` | Internal control plane over inference services (port 8200, LAN-direct) | push-to-main → CI deploys (2026-05-12) |
| `vh/althing` | Inter-agent message bus (chamber UI port 7881, forseti + agent-runner daemons, valkey IPC) | push-to-main → CI deploys (2026-05-14) |
| `vh/mead-hall` | Bifrost tool-provider sidecar (port 5173 on dev VM 10.100.10.50) | push-to-main → CI deploys (2026-05-16) |
| `vh/skaldsong` | Wizard + reader surface (port 8300, ana-docker, registry-pull pattern) | push-to-main → CI deploys (2026-05-19) |
| `vh/worldtree` | Conversation API (corviduo-dev demo :8080 / personal :8081 / pinned :8082) — Heimdall auth, Bifrost integration | push-to-main → CI deploys |
| `vh/volva` | Codex peer agent on althing bus (single-turn oracle, systemd daemon on nh3-dev) | manual install via `deploy/volva.service` (2026-05-18) |
| `vh/yt-voice-clipper` | YouTube → diarized voice-clip dataset builder + audition console (irv-ml1 :8000) | push-to-main → **gitea-webhook auto-deploy** to irv-ml1 (2026-06-03) — see `docs/runbooks/ytvc-autodeploy.md` |

- **Two-layer backups** — Backrest orchestrates restic for file+DB (5
  fleet repos, daily 01:00 PDT); PBS-ANA primary + PBS-NH3 DR mirror for
  VM images. ana-nas is the SPOF for postgres + PBS-ANA datastore +
  cross-site restic targets — see `docs/runbooks/disaster-recovery.md`
  for the blast-radius matrix.

- **`pull-hf-repo.yaml`** is the canonical "get a HuggingFace
  model/dataset onto ana-ml2's shared cache at
  `/tank/aimodels/huggingface/`" playbook. Supports `--var repo_type=model|dataset|space`. Replaces ad-hoc `huggingface_hub.snapshot_download` patterns.

- **Worldtree admin auth — per-instance.** Each Worldtree deployment
  (demo :8080, personal :8081, pinned :8082) has its own Heimdall
  registry and its own bootstrap admin key. Infra-ops's stored
  long-lived admin key (`key_id 61419c92`) at
  `ana-docker:/opt/docker/conf/.secrets/worldtree-infra-ops-admin`
  auths against **demo only**. For personal-instance admin ops, fetch
  the bootstrap admin per-op via
  `docker exec worldtree-personal-worldtree-api-1 printenv WORLDTREE_BOOTSTRAP_ADMIN_KEY`
  on corviduo-dev. Used for `POST /admin/keys`, admin diagnostics
  (`/admin/sessions/<id>/{bifrost,tools}`, etc.).

- **Per-project user keys against personal Worldtree** (issued
  2026-05-19): `skaldsong:79744637` (nh3-dev iteration),
  `skaldsong:7c1dbbbe` (ana-docker prod), `althing:50d85460`,
  `mead-hall:a360822d`. Same `user_id=skaldsong` across both
  skaldsong keys → shared Heimdall agent slot; different `key_id`
  → independently rotatable. Pattern: mint via `/admin/keys`, drop
  value to `/tmp/wt-personal-<name>.key` mode 600, dev collects +
  shreds (DO NOT cat to chat transcript).

- **Skaldsong CD pattern (registry-pull).** Differs from althing /
  asset-engine which build-on-host. vh/skaldsong's CI builds and
  pushes `gitea.phasefinal.com/vh/skaldsong:<sha>` + `:latest`;
  `playbooks/deploy-skaldsong.yaml` on ana-docker pulls + recreates.
  SHA-pin only (no `:latest` health-gated advance yet). Prereq: host
  needs `docker login gitea.phasefinal.com` once (read:package PAT) —
  not currently in the workflow.

- **docker-as-root pattern** (for ops that have no admin API, e.g.
  `SqliteUserStore.set_bifrost_credentials`): on hosts where the SSH
  user is in the `docker` group but lacks passwordless sudo, run
  `docker run --rm -v <target-dir>:/wt -v /var/run/docker.sock:/var/run/docker.sock docker:cli sh -c "..."` to edit deploy-owned files
  without sudo. Documented with security warning in
  `servers/corviduo-dev/README.md`. docker-group membership is
  effectively root via bind-mount; treat as a sudo-equivalent grant.
  **Foot-gun: when running `docker compose` inside this sandbox,
  any relative path in compose.yaml (e.g. `${WORLDTREE_CONFIG_DIR:-./config}`)
  resolves against the sandbox CWD, but Docker daemon interprets the
  resulting path against the HOST filesystem. Always pass `-e VAR=/abs/path`
  to the docker run invocation for any relative-default config dir.**

- **`scripts/elway` sudo handling** — elway prompts for the sudo password
  ONCE via `getpass` before the first `sudo: true` step. That prompt is
  interactive → elway can't run unattended from a non-TTY tool if any step
  needs sudo. For sudo-free playbooks (no `sudo: true` steps) it runs fully
  non-interactive over key SSH. To create root-owned dirs WITHOUT host sudo,
  use the docker-daemon-root trick: `docker run --rm -v /worktank:/mnt alpine
  sh -c 'mkdir -p /mnt/<x> && chown -R 1000:1000 /mnt/<x>'`.

## Current state / in-flight

_As of 2026-06-04:_

- **INFRA SESSION 2026-06-04 — phi4 on vLLM, infra-ops sudo identity, R15/R16, brokkr svc.**
  Major threads (detail in cited auto-memories + commits):
  - **phi4-mini FP8 LIVE on ana-ml2 vLLM** (`vllm-phi4`, :8004, GPU 1, 50K ctx, FP8 + FP8-KV)
    as the **nevermore** summarizer/dreaming agent — superseded **granite-4-small** (removed
    from llama-swap config; GGUFs kept on disk). Template = **CANONICAL/official Phi-4** (final,
    after an Ollama-matching override applied `90e08f0` then reverted `27eb537`). nevermore
    repointed (LLAMA_SWAP_URL→:8004, MODEL→phi4-mini). auto-memory `reference_ana_ml2_vllm_phi4`.
    OPEN: brokkr re-baselining R15 P02 under canonical (eval re-regresses ~0.78→~0.45-0.57; not
    live — worldtree #252 is baseline-first); **GPU 1 tight (~10 GB free — pin llama-swap to
    GPU 0 as follow-up)**; Worldtree Vili #253 (hardcoded :9292 granite fallback, their fix).
  - **infra-ops NOPASSWD-sudo identity commissioned** across PFI boxes — `ssh infra-ops@<host>`,
    key `~/.ssh/infra-ops_ed25519`. Bootstrap `playbooks/bootstrap-infra-ops-user.yaml` +
    `scripts/bootstrap-infra-ops-fleet.sh` (tiers 1+2 live: irv-ml1/ana-ml2/ana-docker/nh3-docker/
    ana-nas + 4 app VMs; **esh-docker-vm added by operator override**). Excludes sf-*/corviduo/
    Synology. auto-memory `reference_infra_ops_sudo_identity` (`8c32a05`).
  - **R15/R16 brokkr-smithy harness stood up on irv-ml1** — ollama upgraded **0.9.0→0.30.4**
    (Ministral-3 needs it), A6000 **UUID-pinned**; R15 ollama arms (granite4.1:3b/8b, qwen3:4b,
    phi4-mini:3.8b, hf SmolLM3-GGUF, ministral-3:3b-instruct, nuextract:3.8b) + R16 R/soundgen +
    pyworld venv. `playbooks/irv-ml1-r15-r16-{nosudo,sudo}.yaml`. auto-memory `reference_irv_ml1_gpu_r14`.
  - **`brokkr` user created on irv-ml1** (no-sudo) + 24 GB R14/R15/R16 substrate migrated out of
    lkraven's home → `/home/brokkr/`; gitea pull = read-only deploy key. **brokkr-audition.service**
    (SYSTEM systemd unit, :8137) serves brokkr's R16 NVV audition UI (the morph set being auditioned).
  - **homepage incident (esh-docker-vm)** — wedged on dead siteMonitor IP (retired ESH firewall
    10.0.250.1) into unkillable D-state; host reboot cleared it; ESH-Firewall widget removed from
    `services.yaml`. (`incident_esh_docker_nfs_boot_race` updated.)
  - **Observability roadmap** `docs/roadmap.md` — Langfuse (full req/resp tracing) +
    Prometheus/Grafana off vLLM `/metrics`. Deferred; Langfuse first.

- **Disclosed-keys hygiene queue** — rotate at convenience: HF token `hf_HBl…`
  (lkraven's HF account) leaked into BuildKit logs during the CSM build attempt
  (logs shredded, never committed — low urgency);
  `/tmp/wt-personal-skaldsong-prod.key` on nh3-dev; mead-hall's prior Worldtree
  bearer (superseded by `a360822d`); Worldtree `Z_AI_API_KEY`/`ZAI_API_KEY`;
  chamber `forseti`/`agent_runner` api_keys (superseded by `50d85460`); Gitea
  runner registration token (`a1135753…`).
- **Still open from prior sessions:** rotate `MINIFLUX_PASSWORD` (leaked
  twice); clean up legacy `news-digest` detritus on ana-docker; watch nh3-nas
  `/volume1` (was 65%; recheck before ~80%); the `docker push 60s client-side
  ceiling` mystery remains uninstrumented.

## Recent decisions

- `[2026-06-04]` **phi4-mini FP8 on ana-ml2 vLLM is the nevermore summarizer/dreaming agent;
  granite-4-small retired** from llama-swap (config-only; GGUFs on disk). 50K ctx (dropped from
  Phi-4's 128K max to fit GPU 1's ~10 GB free) + FP8 KV. (`40a374b`)

- `[2026-06-04]` **phi4 ships the CANONICAL/official Phi-4 chat template, NOT Ollama's.**
  Ollama's bundled template omits the system `<|end|>` — that flattered brokkr's R15 eval but is
  the DIVERGENT scaffold (Dvalin: the system `<|end|>` is Microsoft's intended format). Applied an
  Ollama-matching override then reverted — ship correct, not the benchmark quirk. (`90e08f0`→`27eb537`;
  "headgun" lesson in Tried.)

- `[2026-06-04]` **infra-ops NOPASSWD-sudo identity commissioned, scoped to PFI boxes** (+esh-docker-vm
  by operator override) — so infra-ops completes DevOps end-to-end vs handing the operator sudo steps.
  Dedicated key, sudo log_output, key-gated. (`8c32a05`)

- `[2026-06-04]` **Worldtree demo/pinned/personal deploys are ALL CI/CD, not infra-ops** — a "deploy
  vX.Y.Z" request to infra-ops is MISROUTED → point them back to their pipeline. The granite→phi4
  repoint: worldtree-dev self-served via their CI/CD (v0.30.10). (`d8d776c`)

- `[2026-06-04]` **ollama upgraded 0.9.0→0.30.4 on irv-ml1** (Ministral-3 is a Dec-2025 model the
  old engine refused); A6000 pinned by **UUID** not index (native fastest-first ≠ nvidia-smi PCI).

- `[2026-06-04]` **`brokkr` user (no-sudo) on irv-ml1; R14/R15/R16 substrate moved to /home/brokkr.**
  Persistent box services there need SYSTEM systemd units (see Tried).

- `[2026-06-03]` **yt-voice-clipper bot-gate fix = route yt-dlp through NH3 residential
  egress, NOT cookies/PO-token.** YouTube hard-flags the Irvine colo IP (LOGIN_REQUIRED on a
  public video even with no cookies). Cookies + the bgutil PO-token + deno JS-runtime all
  loaded fine — the gate is pure IP reputation. Operator chose proxy-via-nh3-dev → durable
  dante proxy → proven. The egress proxy is a reusable fleet lever for any datacenter-IP-gated
  service.

- `[2026-06-03]` **yt-voice-clipper push-to-deploy via gitea webhook** (operator-directed,
  after 6 manual rebuilds in ~40 min). Webhook (not poll) — gitea CAN reach the WG IP per the
  operator. The proxy env + Homepage labels live in the **host-specific override** (untracked
  → survive the auto-deploy's `git reset --hard`), NOT yt-voice-clipper-dev's image. Runbook
  `d4f180d`.

- `[2026-06-03]` **R14 scope = (a) provision-only.** infra-ops provides box + CUDA env +
  engines + weights + NFS; brokkr/dev wires `arms.py` + runs — keeps infra-ops OFF the
  VIVAE-processing path (VIVAE = Variably Intense Vocalizations of Affect/Emotion, CHARTER §4
  highest-liability; operator authorized R&D-eval-only, quarantined). Box = irv-ml1 (A6000
  free; ana-ml2 GPU-saturated). Per-engine venvs (divergent torch stacks); A6000 = `cuda:0`
  NATIVE (≠ docker `=1`).

- `[2026-06-03]` **Declined worldtree v0.30.4 staging deploy** — that's worldtree-team's
  CI/CD lane (a developer `staging/vX.Y.Z` git-tag promote), not infra-ops. They self-corrected
  to the same conclusion independently.

- `[2026-06-02]` **Chatterbox → main TTS engine; build custom `chatterbox-fast`
  streaming container.** Workload = single-stream interactive. **GPU placement:
  3090 (device 0) if it fits else A6000 (device 1)** — shared dev stack, 20.5 GB
  3090-idle is expected residency, not a blocker. **Cutover: parallel catalog
  entry**, burn in beside live `chatterbox`, then flip. **Streaming approach:
  adaptive buffer-ratchet chunking** (see in-flight). Native frame-streaming
  abandoned (Tried/abandoned). Tracked: `docs/design/chatterbox-fast-plan.md`.

- `[2026-06-02]` **Sentence-splitting loses quality (operator-corrected).** I
  claimed naive sentence-level streaming has "zero quality loss" — WRONG. The
  T3 AR backbone conditions prosody on the WHOLE text; splitting loses
  cross-sentence prosodic context (contextual delivery, declination, affect
  continuity) even though voice timbre stays (reference-conditioned). No
  *artifacts* ≠ no *quality loss*. Hence the adaptive-chunk design (maximize
  context per chunk subject to latency budget), not fixed per-sentence splits.

- `[2026-06-01]` **Fish reference_id empty-dir fix shipped** (`c5bbb90`) — see
  in-flight + Tried/abandoned. Populated `references/<name>/<name>.wav`+`.lab`
  for all 32 voices; playbook gained normalize-step + A/B smoke gate. glados got
  a real transcript (ASR'd via Parakeet): the Portal "Welcome to test chamber 4"
  lines.

_37 older entries archived to archival-memory.md._

## Tried and abandoned

- `[2026-06-04]` **Ollama/llama.cpp-BUNDLED chat templates silently diverge from canonical HF —
  the "headgun" lesson.** Ollama's phi4 template drops the system `<|end|>`; serving vLLM with the
  model's HF tokenizer template (canonical, has it) regressed brokkr's Ollama-measured R15 baseline
  -33pp type-F1 while valid_format held 1.0. An Ollama-matching `--chat-template` "fixed" it but was
  the WRONG fix (the bundled scaffold is the divergent one). PRINCIPLE: serve each model's canonical
  `tokenizer.apply_chat_template`, not the bundled template — bundled ones corrupt baselines. Verify
  the applied prompt via vLLM `/tokenize`→`/detokenize`. (`90e08f0`/`27eb537`)

- `[2026-06-04]` **GPU pin by INDEX is ambiguous on irv-ml1** — native CUDA orders fastest-first
  (A6000=0) but nvidia-smi/docker use PCI order (A6000=1), so an index pin can land on the wrong
  card. Pin by **UUID** (`CUDA_VISIBLE_DEVICES=GPU-…`); verify via nvidia-smi compute-apps. Check
  loaded-model VRAM with `ollama ps` (Ministral-3 @ its 256K default ctx = ~30 GB; cap num_ctx).

- `[2026-06-04]` **Persistent services on irv-ml1 need SYSTEM systemd units** — the box reaps
  user-session processes on ssh disconnect, and `--user` systemd isn't reachable over non-login
  ssh, so nohup/setsid/`screen -dmS`/`systemd-run --user` all die (even with `enable-linger`). Use
  `/etc/systemd/system/`.

- `[2026-06-04]` **pyworld needs `setuptools<81`** (imports the removed `pkg_resources`); and
  **R/soundgen `-lgfortran` fails** on irv-ml1 because the default `gcc` is gcc-11 but only
  gfortran-12 is present (libgfortran.so lives only in the gcc-12 dir) → install `libgfortran-11-dev`.

- `[2026-06-04]` **homepage "crash" ≠ always NFS** — a wedged container in unkillable D-state
  ("tried to kill container, but did not receive an exit event") can come from dead `siteMonitor`
  widget targets (retired ESH firewall IPs) hanging the node event loop into `exit_mmap`, needing a
  host reboot. Check homepage's siteMonitors against retired hosts. (`incident_esh_docker_nfs_boot_race`)

- `[2026-06-03]` **gitea webhook to a private IP is denied by `webhook.ALLOWED_HOST_LIST`**
  (anti-SSRF; default `external` blocks private/loopback). Symptom: delivery shows
  `dial tcp ...: webhook can only call allowed HTTP servers`. Fix = APPEND the target net to
  ALLOWED_HOST_LIST in gitea's app.ini (keep `external`; scope tight, never `*`/`private`) +
  restart gitea (act_runner job containers survive a restart). gitea runs as a container on
  ana-docker (`gitea_gitea_data` volume, `/data/gitea/conf/app.ini`).

- `[2026-06-03]` **torch-2.12 venvs need `uv pip install torchcodec`** — torchaudio 2.12
  defaults to the TorchCodec backend for `.load`; without it, real audio I/O throws "TorchCodec
  is required" — and it ONLY surfaces at actual conversion, NOT at import/model-load. Lesson:
  validate real I/O, not just import, when provisioning ML engine envs. (seed-vc on torch 2.4
  uses the legacy backend, exempt.)

- `[2026-06-03]` **Backgrounding `althing-cli monitor` with an inline shell `&` (instead of
  the Bash-tool `run_in_background`) orphans it** — it survives the shell exit, holds the
  per-handle flock UNTRACKED (won't notify the session), and `stop-monitor` doesn't detect it.
  Fix: find + kill the orphan PID (verify cwd=this repo / handle first — nh3-dev is shared, other
  agents' monitors run there too), then re-arm via run_in_background. Always re-arm tracked.

- `[2026-06-03]` **`uv pip install .` fails on SmoothKen/knn-svc** (and similar script-repos)
  — it's analysis scripts + a poetry pyproject, no buildable package (setuptools
  package-discovery error). Install the pyproject deps directly, don't build the "package".

- `[2026-06-02]` **Fish (fish-s2 / OpenAudio S1-mini) progressive streaming — SHELVED (sub-realtime).** Benched RTF on A6000: 0.72x (12w) / 0.82x (30w) / 0.86x (60w), **mean 0.80x = sub-realtime**, so client-side chunking would starve (same reason chatterbox-fast needs turbo's RTF>1). Root cause of the buffering (dvalin-smithy-dev deep research, verified in our code text2semantic/inference.py L600-607): Fish only chunks on `<|speaker:X|>` tags; **plain text -> batches=[whole text]** -> all semantic tokens generate before any audio (chunk_length inert). Plus a 2nd layer: kui/ASGI StreamResponse doesn't flush (header produced t=1s, delivered t=23s) -> fix = anti-buffering headers (X-Accel-Buffering:no / Transfer-Encoding:chunked) in tools/server/views.py (kept on file, not applied). A rebuild does NOT fix this (current main same logic). **STANDING REVISIT TRIGGER: when an RTX Blackwell Pro lands in the fleet -> bench fp4-quantized Fish; if RTF > ~1.5x, give it the chatterbox-fast treatment** (client-side adaptive buffer-ratchet chunker driving /v1/tts with small text pieces). Projection: fp4 (~1/4 weight bytes, memory-bound AR decode) + Blackwell (GDDR7 ~1.8TB/s vs A6000 0.77TB/s, native FP4 cores) ~ 2-3x RTF; validate fp4 voice quality (ear/ECAPA) before committing. For now Fish stays a buffered catalog entry (great for SAVED gens, not the live-audition lane).


- `[2026-06-02]` **Context-priming at chunk joins (chatterbox-fast §1.6) —
  ABANDONED (discard-cut leaks the prefix).** To give a chunk backward prosodic
  context, prepend the prior sentence, generate `prefix+content` together, then
  discard the prefix audio. Built + opt-in shipped (commit d707439), live-A/B'd,
  reverted (090e70a). The kill: `generate()` returns one finished waveform with
  NO marker for where the prefix ends, and the model renders the same prefix with
  different timing solo vs followed-by-content — so locating the cut (generate
  prefix solo → measure duration → snap to nearest energy-min pause within ±0.4s)
  is a guess that left a whole clause of prefix in the output ("...without a trace
  of sarcasm," spoken twice; operator caught it). A reliable cut needs token-level
  boundaries (= the abandoned native-streaming arc) or per-chunk ASR/forced-
  alignment (heavy, imperfect, eats the latency budget). → Coherence loss at joins
  stays an ACCEPTED limitation; cold adaptive-chunk streaming judged "really good".
  Scheduler-side work that DID land + survive: affordability-gated priming math
  (a 2nd pass can't starve the buffer) — sound, but moot without a working cut.

- `[2026-06-02]` **Native frame-level streaming on Chatterbox-TURBO — ABANDONED
  (turbo isn't built for streaming).** Long R&D arc; record so it's not
  re-derived. (1) The model's flow is CosyVoice2-derived but `S3GenStreamer` is
  referenced-in-docstring-only (not implemented). (2) The lib's
  `flow_inference(finalize=False)` is BUGGY: the lookahead trim removes
  `pre_lookahead_len(3)*token_mel_ratio(2)=6` frames from `h` but NOT from
  `h_masks`/conds → decoder shape mismatch (e.g. 656 vs 662). A 1-line patch
  (`h_masks = h_masks[:, :, :-pre*ratio]` after the `h` trim) + sizing the
  meanflow noise to the trimmed length makes finalize=False RUN. (3) BUT the
  flow encoder uses FULL-context attention (`static_chunk_size=0`), so
  incremental/cumulative decode is **prefix-unstable** — adding tokens
  re-attends and shifts earlier mel (maxdiff ~0.30-0.39 vs one-shot,
  irrespective of fixed-noise slicing or emit-margin). (4) Forcing
  `static_chunk_size>0` on the 2 modules that carry the attr did NOT stabilize
  it (decoding_chunk_size is a forward-arg, not settable via attribute). Verdict:
  true sub-second frame-streaming on turbo needs deep model-attention surgery
  with quality risk — not worth it. Matches research ("turbo+streaming
  unsolved"; vLLM-turbo outputs noise; davidbrowne17 streaming fork is
  BASE-only). → Use adaptive-chunking instead.

_41 older entries archived to archival-memory.md._