esh-pfi-infrastructure/persistent-memory.md

Persistent memory — eshpfi-management

Last updated: 2026-06-04

Repo purpose

Reference workspace for PFI infrastructure: server inventory, canonical Docker Compose stacks, ops playbooks, and conventions. Authoritative copies of compose files live on the servers under /opt/docker/compose/<stack>/; this repo mirrors them for version control, editing, planning, and CI-driven deploys.

Tools and conventions

Sister repos (separate gitea repos, deployed by playbooks here):

Repo	Role	CI status
vh/task-board	MCP + web dashboard for assistant task state (port 7878)	push-to-main → CI deploys (2026-04-29)
vh/vor	Inquisitor UI sidecar (port 7879)	push-to-main → CI deploys (2026-04-29)
vh/nevermore	Twice-daily LLM-curated briefing (port 8181, replaces news-digest)	push-to-main → CI deploys (2026-04-30)
vh/asset-engine	Internal control plane over inference services (port 8200, LAN-direct)	push-to-main → CI deploys (2026-05-12)
vh/althing	Inter-agent message bus (chamber UI port 7881, forseti + agent-runner daemons, valkey IPC)	push-to-main → CI deploys (2026-05-14)
vh/mead-hall	Bifrost tool-provider sidecar (port 5173 on dev VM 10.100.10.50)	push-to-main → CI deploys (2026-05-16)
vh/skaldsong	Wizard + reader surface (port 8300, ana-docker, registry-pull pattern)	push-to-main → CI deploys (2026-05-19)
vh/worldtree	Conversation API (corviduo-dev demo :8080 / personal :8081 / pinned :8082) — Heimdall auth, Bifrost integration	push-to-main → CI deploys
vh/volva	Codex peer agent on althing bus (single-turn oracle, systemd daemon on nh3-dev)	manual install via deploy/volva.service (2026-05-18)
vh/yt-voice-clipper	YouTube → diarized voice-clip dataset builder + audition console (irv-ml1 :8000)	push-to-main → gitea-webhook auto-deploy to irv-ml1 (2026-06-03) — see docs/runbooks/ytvc-autodeploy.md

• Two-layer backups — Backrest orchestrates restic for file+DB (5 fleet repos, daily 01:00 PDT); PBS-ANA primary + PBS-NH3 DR mirror for VM images. ana-nas is the SPOF for postgres + PBS-ANA datastore + cross-site restic targets — see docs/runbooks/disaster-recovery.md for the blast-radius matrix.

• pull-hf-repo.yaml is the canonical "get a HuggingFace model/dataset onto ana-ml2's shared cache at /tank/aimodels/huggingface/" playbook. Supports --var repo_type=model|dataset|space. Replaces ad-hoc huggingface_hub.snapshot_download patterns.

• Worldtree admin auth — per-instance. Each Worldtree deployment (demo :8080, personal :8081, pinned :8082) has its own Heimdall registry and its own bootstrap admin key. Infra-ops's stored long-lived admin key (key_id 61419c92) at ana-docker:/opt/docker/conf/.secrets/worldtree-infra-ops-admin auths against demo only. For personal-instance admin ops, fetch the bootstrap admin per-op via docker exec worldtree-personal-worldtree-api-1 printenv WORLDTREE_BOOTSTRAP_ADMIN_KEY on corviduo-dev. Used for POST /admin/keys, admin diagnostics (/admin/sessions/<id>/{bifrost,tools}, etc.).

• Per-project user keys against personal Worldtree (issued 2026-05-19): skaldsong:79744637 (nh3-dev iteration), skaldsong:7c1dbbbe (ana-docker prod), althing:50d85460, mead-hall:a360822d. Same user_id=skaldsong across both skaldsong keys → shared Heimdall agent slot; different key_id → independently rotatable. Pattern: mint via /admin/keys, drop value to /tmp/wt-personal-<name>.key mode 600, dev collects + shreds (DO NOT cat to chat transcript).

• Skaldsong CD pattern (registry-pull). Differs from althing / asset-engine which build-on-host. vh/skaldsong's CI builds and pushes gitea.phasefinal.com/vh/skaldsong:<sha> + :latest; playbooks/deploy-skaldsong.yaml on ana-docker pulls + recreates. SHA-pin only (no :latest health-gated advance yet). Prereq: host needs docker login gitea.phasefinal.com once (read:package PAT) — not currently in the workflow.

• docker-as-root pattern (for ops that have no admin API, e.g. SqliteUserStore.set_bifrost_credentials): on hosts where the SSH user is in the docker group but lacks passwordless sudo, run docker run --rm -v <target-dir>:/wt -v /var/run/docker.sock:/var/run/docker.sock docker:cli sh -c "..." to edit deploy-owned files without sudo. Documented with security warning in servers/corviduo-dev/README.md. docker-group membership is effectively root via bind-mount; treat as a sudo-equivalent grant. Foot-gun: when running docker compose inside this sandbox, any relative path in compose.yaml (e.g. ${WORLDTREE_CONFIG_DIR:-./config}) resolves against the sandbox CWD, but Docker daemon interprets the resulting path against the HOST filesystem. Always pass -e VAR=/abs/path to the docker run invocation for any relative-default config dir.

• scripts/elway sudo handling — elway prompts for the sudo password ONCE via getpass before the first sudo: true step. That prompt is interactive → elway can't run unattended from a non-TTY tool if any step needs sudo. For sudo-free playbooks (no sudo: true steps) it runs fully non-interactive over key SSH. To create root-owned dirs WITHOUT host sudo, use the docker-daemon-root trick: docker run --rm -v /worktank:/mnt alpine sh -c 'mkdir -p /mnt/<x> && chown -R 1000:1000 /mnt/<x>'.

Current state / in-flight

As of 2026-06-04:

• INFRA SESSION 2026-06-04 — phi4 on vLLM, infra-ops sudo identity, R15/R16, brokkr svc. Major threads (detail in cited auto-memories + commits):

  • phi4-mini FP8 LIVE on ana-ml2 vLLM (vllm-phi4, :8004, GPU 1, 50K ctx, FP8 + FP8-KV) as the nevermore summarizer/dreaming agent — superseded granite-4-small (removed from llama-swap config; GGUFs kept on disk). Template = CANONICAL/official Phi-4 (final, after an Ollama-matching override applied 90e08f0 then reverted 27eb537). nevermore repointed (LLAMA_SWAP_URL→:8004, MODEL→phi4-mini). auto-memory reference_ana_ml2_vllm_phi4. OPEN: brokkr re-baselining R15 P02 under canonical (eval re-regresses ~0.78→~0.45-0.57; not live — worldtree #252 is baseline-first); GPU 1 tight (~10 GB free — pin llama-swap to GPU 0 as follow-up); Worldtree Vili #253 (hardcoded :9292 granite fallback, their fix).
  • infra-ops NOPASSWD-sudo identity commissioned across PFI boxes — ssh infra-ops@<host>, key ~/.ssh/infra-ops_ed25519. Bootstrap playbooks/bootstrap-infra-ops-user.yaml + scripts/bootstrap-infra-ops-fleet.sh (tiers 1+2 live: irv-ml1/ana-ml2/ana-docker/nh3-docker/ ana-nas + 4 app VMs; esh-docker-vm added by operator override). Excludes sf-*/corviduo/ Synology. auto-memory reference_infra_ops_sudo_identity (8c32a05).
  • R15/R16 brokkr-smithy harness stood up on irv-ml1 — ollama upgraded 0.9.0→0.30.4 (Ministral-3 needs it), A6000 UUID-pinned; R15 ollama arms (granite4.1:3b/8b, qwen3:4b, phi4-mini:3.8b, hf SmolLM3-GGUF, ministral-3:3b-instruct, nuextract:3.8b) + R16 R/soundgen + pyworld venv. playbooks/irv-ml1-r15-r16-{nosudo,sudo}.yaml. auto-memory reference_irv_ml1_gpu_r14.
  • brokkr user created on irv-ml1 (no-sudo) + 24 GB R14/R15/R16 substrate migrated out of lkraven's home → /home/brokkr/; gitea pull = read-only deploy key. brokkr-audition.service (SYSTEM systemd unit, :8137) serves brokkr's R16 NVV audition UI (the morph set being auditioned).
  • homepage incident (esh-docker-vm) — wedged on dead siteMonitor IP (retired ESH firewall 10.0.250.1) into unkillable D-state; host reboot cleared it; ESH-Firewall widget removed from services.yaml. (incident_esh_docker_nfs_boot_race updated.)
  • Observability roadmap docs/roadmap.md — Langfuse (full req/resp tracing) + Prometheus/Grafana off vLLM /metrics. Deferred; Langfuse first.

• Disclosed-keys hygiene queue — rotate at convenience: HF token hf_HBl… (lkraven's HF account) leaked into BuildKit logs during the CSM build attempt (logs shredded, never committed — low urgency); /tmp/wt-personal-skaldsong-prod.key on nh3-dev; mead-hall's prior Worldtree bearer (superseded by a360822d); Worldtree Z_AI_API_KEY/ZAI_API_KEY; chamber forseti/agent_runner api_keys (superseded by 50d85460); Gitea runner registration token (a1135753…).

• Still open from prior sessions: rotate MINIFLUX_PASSWORD (leaked twice); clean up legacy news-digest detritus on ana-docker; watch nh3-nas /volume1 (was 65%; recheck before ~80%); the docker push 60s client-side ceiling mystery remains uninstrumented.

Recent decisions

• [2026-06-04] phi4-mini FP8 on ana-ml2 vLLM is the nevermore summarizer/dreaming agent; granite-4-small retired from llama-swap (config-only; GGUFs on disk). 50K ctx (dropped from Phi-4's 128K max to fit GPU 1's ~10 GB free) + FP8 KV. (40a374b)

• [2026-06-04] phi4 ships the CANONICAL/official Phi-4 chat template, NOT Ollama's. Ollama's bundled template omits the system <|end|> — that flattered brokkr's R15 eval but is the DIVERGENT scaffold (Dvalin: the system <|end|> is Microsoft's intended format). Applied an Ollama-matching override then reverted — ship correct, not the benchmark quirk. (90e08f0→27eb537; "headgun" lesson in Tried.)

• [2026-06-04] infra-ops NOPASSWD-sudo identity commissioned, scoped to PFI boxes (+esh-docker-vm by operator override) — so infra-ops completes DevOps end-to-end vs handing the operator sudo steps. Dedicated key, sudo log_output, key-gated. (8c32a05)

• [2026-06-04] Worldtree demo/pinned/personal deploys are ALL CI/CD, not infra-ops — a "deploy vX.Y.Z" request to infra-ops is MISROUTED → point them back to their pipeline. The granite→phi4 repoint: worldtree-dev self-served via their CI/CD (v0.30.10). (d8d776c)

• [2026-06-04] ollama upgraded 0.9.0→0.30.4 on irv-ml1 (Ministral-3 is a Dec-2025 model the old engine refused); A6000 pinned by UUID not index (native fastest-first ≠ nvidia-smi PCI).

• [2026-06-04] brokkr user (no-sudo) on irv-ml1; R14/R15/R16 substrate moved to /home/brokkr. Persistent box services there need SYSTEM systemd units (see Tried).

• [2026-06-03] yt-voice-clipper bot-gate fix = route yt-dlp through NH3 residential egress, NOT cookies/PO-token. YouTube hard-flags the Irvine colo IP (LOGIN_REQUIRED on a public video even with no cookies). Cookies + the bgutil PO-token + deno JS-runtime all loaded fine — the gate is pure IP reputation. Operator chose proxy-via-nh3-dev → durable dante proxy → proven. The egress proxy is a reusable fleet lever for any datacenter-IP-gated service.

• [2026-06-03] yt-voice-clipper push-to-deploy via gitea webhook (operator-directed, after 6 manual rebuilds in ~40 min). Webhook (not poll) — gitea CAN reach the WG IP per the operator. The proxy env + Homepage labels live in the host-specific override (untracked → survive the auto-deploy's git reset --hard), NOT yt-voice-clipper-dev's image. Runbook d4f180d.

• [2026-06-03] R14 scope = (a) provision-only. infra-ops provides box + CUDA env + engines + weights + NFS; brokkr/dev wires arms.py + runs — keeps infra-ops OFF the VIVAE-processing path (VIVAE = Variably Intense Vocalizations of Affect/Emotion, CHARTER §4 highest-liability; operator authorized R&D-eval-only, quarantined). Box = irv-ml1 (A6000 free; ana-ml2 GPU-saturated). Per-engine venvs (divergent torch stacks); A6000 = cuda:0 NATIVE (≠ docker =1).

• [2026-06-03] Declined worldtree v0.30.4 staging deploy — that's worldtree-team's CI/CD lane (a developer staging/vX.Y.Z git-tag promote), not infra-ops. They self-corrected to the same conclusion independently.

• [2026-06-02] Chatterbox → main TTS engine; build custom chatterbox-fast streaming container. Workload = single-stream interactive. GPU placement: 3090 (device 0) if it fits else A6000 (device 1) — shared dev stack, 20.5 GB 3090-idle is expected residency, not a blocker. Cutover: parallel catalog entry, burn in beside live chatterbox, then flip. Streaming approach: adaptive buffer-ratchet chunking (see in-flight). Native frame-streaming abandoned (Tried/abandoned). Tracked: docs/design/chatterbox-fast-plan.md.

• [2026-06-02] Sentence-splitting loses quality (operator-corrected). I claimed naive sentence-level streaming has "zero quality loss" — WRONG. The T3 AR backbone conditions prosody on the WHOLE text; splitting loses cross-sentence prosodic context (contextual delivery, declination, affect continuity) even though voice timbre stays (reference-conditioned). No artifacts ≠ no quality loss. Hence the adaptive-chunk design (maximize context per chunk subject to latency budget), not fixed per-sentence splits.

• [2026-06-01] Fish reference_id empty-dir fix shipped (c5bbb90) — see in-flight + Tried/abandoned. Populated references/<name>/<name>.wav+.lab for all 32 voices; playbook gained normalize-step + A/B smoke gate. glados got a real transcript (ASR'd via Parakeet): the Portal "Welcome to test chamber 4" lines.

37 older entries archived to archival-memory.md.

Tried and abandoned

• [2026-06-04] Ollama/llama.cpp-BUNDLED chat templates silently diverge from canonical HF — the "headgun" lesson. Ollama's phi4 template drops the system <|end|>; serving vLLM with the model's HF tokenizer template (canonical, has it) regressed brokkr's Ollama-measured R15 baseline -33pp type-F1 while valid_format held 1.0. An Ollama-matching --chat-template "fixed" it but was the WRONG fix (the bundled scaffold is the divergent one). PRINCIPLE: serve each model's canonical tokenizer.apply_chat_template, not the bundled template — bundled ones corrupt baselines. Verify the applied prompt via vLLM /tokenize→/detokenize. (90e08f0/27eb537)

• [2026-06-04] GPU pin by INDEX is ambiguous on irv-ml1 — native CUDA orders fastest-first (A6000=0) but nvidia-smi/docker use PCI order (A6000=1), so an index pin can land on the wrong card. Pin by UUID (CUDA_VISIBLE_DEVICES=GPU-…); verify via nvidia-smi compute-apps. Check loaded-model VRAM with ollama ps (Ministral-3 @ its 256K default ctx = ~30 GB; cap num_ctx).

• [2026-06-04] Persistent services on irv-ml1 need SYSTEM systemd units — the box reaps user-session processes on ssh disconnect, and --user systemd isn't reachable over non-login ssh, so nohup/setsid/screen -dmS/systemd-run --user all die (even with enable-linger). Use /etc/systemd/system/.

• [2026-06-04] pyworld needs setuptools<81 (imports the removed pkg_resources); and R/soundgen -lgfortran fails on irv-ml1 because the default gcc is gcc-11 but only gfortran-12 is present (libgfortran.so lives only in the gcc-12 dir) → install libgfortran-11-dev.

• [2026-06-04] homepage "crash" ≠ always NFS — a wedged container in unkillable D-state ("tried to kill container, but did not receive an exit event") can come from dead siteMonitor widget targets (retired ESH firewall IPs) hanging the node event loop into exit_mmap, needing a host reboot. Check homepage's siteMonitors against retired hosts. (incident_esh_docker_nfs_boot_race)

• [2026-06-03] gitea webhook to a private IP is denied by webhook.ALLOWED_HOST_LIST (anti-SSRF; default external blocks private/loopback). Symptom: delivery shows dial tcp ...: webhook can only call allowed HTTP servers. Fix = APPEND the target net to ALLOWED_HOST_LIST in gitea's app.ini (keep external; scope tight, never */private) + restart gitea (act_runner job containers survive a restart). gitea runs as a container on ana-docker (gitea_gitea_data volume, /data/gitea/conf/app.ini).

• [2026-06-03] torch-2.12 venvs need uv pip install torchcodec — torchaudio 2.12 defaults to the TorchCodec backend for .load; without it, real audio I/O throws "TorchCodec is required" — and it ONLY surfaces at actual conversion, NOT at import/model-load. Lesson: validate real I/O, not just import, when provisioning ML engine envs. (seed-vc on torch 2.4 uses the legacy backend, exempt.)

• [2026-06-03] Backgrounding althing-cli monitor with an inline shell & (instead of the Bash-tool run_in_background) orphans it — it survives the shell exit, holds the per-handle flock UNTRACKED (won't notify the session), and stop-monitor doesn't detect it. Fix: find + kill the orphan PID (verify cwd=this repo / handle first — nh3-dev is shared, other agents' monitors run there too), then re-arm via run_in_background. Always re-arm tracked.

• [2026-06-03] uv pip install . fails on SmoothKen/knn-svc (and similar script-repos) — it's analysis scripts + a poetry pyproject, no buildable package (setuptools package-discovery error). Install the pyproject deps directly, don't build the "package".

• [2026-06-02] Fish (fish-s2 / OpenAudio S1-mini) progressive streaming — SHELVED (sub-realtime). Benched RTF on A6000: 0.72x (12w) / 0.82x (30w) / 0.86x (60w), mean 0.80x = sub-realtime, so client-side chunking would starve (same reason chatterbox-fast needs turbo's RTF>1). Root cause of the buffering (dvalin-smithy-dev deep research, verified in our code text2semantic/inference.py L600-607): Fish only chunks on <|speaker:X|> tags; plain text -> batches=[whole text] -> all semantic tokens generate before any audio (chunk_length inert). Plus a 2nd layer: kui/ASGI StreamResponse doesn't flush (header produced t=1s, delivered t=23s) -> fix = anti-buffering headers (X-Accel-Buffering:no / Transfer-Encoding:chunked) in tools/server/views.py (kept on file, not applied). A rebuild does NOT fix this (current main same logic). STANDING REVISIT TRIGGER: when an RTX Blackwell Pro lands in the fleet -> bench fp4-quantized Fish; if RTF > ~1.5x, give it the chatterbox-fast treatment (client-side adaptive buffer-ratchet chunker driving /v1/tts with small text pieces). Projection: fp4 (~1/4 weight bytes, memory-bound AR decode) + Blackwell (GDDR7 ~1.8TB/s vs A6000 0.77TB/s, native FP4 cores) ~ 2-3x RTF; validate fp4 voice quality (ear/ECAPA) before committing. For now Fish stays a buffered catalog entry (great for SAVED gens, not the live-audition lane).

• [2026-06-02] Context-priming at chunk joins (chatterbox-fast §1.6) — ABANDONED (discard-cut leaks the prefix). To give a chunk backward prosodic context, prepend the prior sentence, generate prefix+content together, then discard the prefix audio. Built + opt-in shipped (commit d707439), live-A/B'd, reverted (090e70a). The kill: generate() returns one finished waveform with NO marker for where the prefix ends, and the model renders the same prefix with different timing solo vs followed-by-content — so locating the cut (generate prefix solo → measure duration → snap to nearest energy-min pause within ±0.4s) is a guess that left a whole clause of prefix in the output ("...without a trace of sarcasm," spoken twice; operator caught it). A reliable cut needs token-level boundaries (= the abandoned native-streaming arc) or per-chunk ASR/forced- alignment (heavy, imperfect, eats the latency budget). → Coherence loss at joins stays an ACCEPTED limitation; cold adaptive-chunk streaming judged "really good". Scheduler-side work that DID land + survive: affordability-gated priming math (a 2nd pass can't starve the buffer) — sound, but moot without a working cut.

• [2026-06-02] Native frame-level streaming on Chatterbox-TURBO — ABANDONED (turbo isn't built for streaming). Long R&D arc; record so it's not re-derived. (1) The model's flow is CosyVoice2-derived but S3GenStreamer is referenced-in-docstring-only (not implemented). (2) The lib's flow_inference(finalize=False) is BUGGY: the lookahead trim removes pre_lookahead_len(3)*token_mel_ratio(2)=6 frames from h but NOT from h_masks/conds → decoder shape mismatch (e.g. 656 vs 662). A 1-line patch (h_masks = h_masks[:, :, :-pre*ratio] after the h trim) + sizing the meanflow noise to the trimmed length makes finalize=False RUN. (3) BUT the flow encoder uses FULL-context attention (static_chunk_size=0), so incremental/cumulative decode is prefix-unstable — adding tokens re-attends and shifts earlier mel (maxdiff ~0.30-0.39 vs one-shot, irrespective of fixed-noise slicing or emit-margin). (4) Forcing static_chunk_size>0 on the 2 modules that carry the attr did NOT stabilize it (decoding_chunk_size is a forward-arg, not settable via attribute). Verdict: true sub-second frame-streaming on turbo needs deep model-attention surgery with quality risk — not worth it. Matches research ("turbo+streaming unsolved"; vLLM-turbo outputs noise; davidbrowne17 streaming fork is BASE-only). → Use adaptive-chunking instead.

41 older entries archived to archival-memory.md.