# Persistent memory — eshpfi-management

_Last updated: 2026-06-11_

## Repo purpose

Reference workspace for PFI infrastructure: server inventory, canonical
Docker Compose stacks, ops playbooks, and conventions. Authoritative
copies of compose files live on the servers under
`/opt/docker/compose/<stack>/`; this repo mirrors them for version
control, editing, planning, and CI-driven deploys.

## Tools and conventions

Sister repos (separate gitea repos, deployed by playbooks here):

| Repo | Role | CI status |
|---|---|---|
| `vh/task-board` | MCP + web dashboard for assistant task state (port 7878) | push-to-main → CI deploys (2026-04-29) |
| `vh/vor` | Inquisitor UI sidecar (port 7879) | push-to-main → CI deploys (2026-04-29) |
| `vh/nevermore` | Twice-daily LLM-curated briefing (port 8181, replaces news-digest) | push-to-main → CI deploys (2026-04-30) |
| `vh/asset-engine` | Internal control plane over inference services (port 8200, LAN-direct) | push-to-main → CI deploys (2026-05-12) |
| `vh/althing` | Inter-agent message bus (chamber UI port 7881, forseti + agent-runner daemons, valkey IPC) | push-to-main → CI deploys (2026-05-14) |
| `vh/mead-hall` | Bifrost tool-provider sidecar (port 5173 on dev VM 10.100.10.50) | push-to-main → CI deploys (2026-05-16) |
| `vh/skaldsong` | Wizard + reader surface (port 8300, ana-docker, registry-pull pattern) | push-to-main → CI deploys (2026-05-19) |
| `vh/worldtree` | Conversation API (corviduo-dev demo :8080 / personal :8081 / pinned :8082) — Heimdall auth, Bifrost integration | push-to-main → CI deploys |
| `vh/yt-voice-clipper` | YouTube → diarized voice-clip dataset builder + audition console (irv-ml1 :8000) | push-to-main → **gitea-webhook auto-deploy** to irv-ml1 (2026-06-03) — see `docs/runbooks/ytvc-autodeploy.md` |

(`vh/volva` + Heid were re-architected from systemd daemons to Claude Code
session orchestrators 2026-06-08; their nh3-dev `.service` units were removed —
no longer deployed sidecars here. See Recent decisions.)

- **Two-layer backups** — Backrest orchestrates restic for file+DB (5
  fleet repos, daily 01:00 PDT); PBS-ANA primary + PBS-NH3 DR mirror for
  VM images. ana-nas is the SPOF for postgres + PBS-ANA datastore +
  cross-site restic targets — see `docs/runbooks/disaster-recovery.md`
  for the blast-radius matrix.

- **`pull-hf-repo.yaml`** is the canonical "get a HuggingFace
  model/dataset onto ana-ml2's shared cache at
  `/tank/aimodels/huggingface/`" playbook. Supports `--var repo_type=model|dataset|space`. Replaces ad-hoc `huggingface_hub.snapshot_download` patterns.

- **Worldtree admin auth — per-instance.** Each Worldtree deployment
  (demo :8080, personal :8081, pinned :8082) has its own Heimdall
  registry and its own bootstrap admin key. Infra-ops's stored
  long-lived admin key (`key_id 61419c92`) at
  `ana-docker:/opt/docker/conf/.secrets/worldtree-infra-ops-admin`
  auths against **demo only**. For personal-instance admin ops, fetch
  the bootstrap admin per-op via
  `docker exec worldtree-personal-worldtree-api-1 printenv WORLDTREE_BOOTSTRAP_ADMIN_KEY`
  on corviduo-dev. Used for `POST /admin/keys`, admin diagnostics
  (`/admin/sessions/<id>/{bifrost,tools}`, etc.).

- **Per-project user keys against personal Worldtree** (issued
  2026-05-19): `skaldsong:79744637` (nh3-dev iteration),
  `skaldsong:7c1dbbbe` (ana-docker prod), `althing:50d85460`,
  `mead-hall:a360822d`. Same `user_id=skaldsong` across both
  skaldsong keys → shared Heimdall agent slot; different `key_id`
  → independently rotatable. Pattern: mint via `/admin/keys`, drop
  value to `/tmp/wt-personal-<name>.key` mode 600, dev collects +
  shreds (DO NOT cat to chat transcript).

- **Skaldsong CD pattern (registry-pull).** Differs from althing /
  asset-engine which build-on-host. vh/skaldsong's CI builds and
  pushes `gitea.phasefinal.com/vh/skaldsong:<sha>` + `:latest`;
  `playbooks/deploy-skaldsong.yaml` on ana-docker pulls + recreates.
  SHA-pin only (no `:latest` health-gated advance yet). Prereq: host
  needs `docker login gitea.phasefinal.com` once (read:package PAT) —
  not currently in the workflow.

- **docker-as-root pattern** (for ops that have no admin API, e.g.
  `SqliteUserStore.set_bifrost_credentials`): on hosts where the SSH
  user is in the `docker` group but lacks passwordless sudo, run
  `docker run --rm -v <target-dir>:/wt -v /var/run/docker.sock:/var/run/docker.sock docker:cli sh -c "..."` to edit deploy-owned files
  without sudo. Documented with security warning in
  `servers/corviduo-dev/README.md`. docker-group membership is
  effectively root via bind-mount; treat as a sudo-equivalent grant.
  **Foot-gun: when running `docker compose` inside this sandbox,
  any relative path in compose.yaml (e.g. `${WORLDTREE_CONFIG_DIR:-./config}`)
  resolves against the sandbox CWD, but Docker daemon interprets the
  resulting path against the HOST filesystem. Always pass `-e VAR=/abs/path`
  to the docker run invocation for any relative-default config dir.**

- **`scripts/elway` sudo handling** — elway prompts for the sudo password
  ONCE via `getpass` before the first `sudo: true` step. That prompt is
  interactive → elway can't run unattended from a non-TTY tool if any step
  needs sudo. For sudo-free playbooks (no `sudo: true` steps) it runs fully
  non-interactive over key SSH. To create root-owned dirs WITHOUT host sudo,
  use the docker-daemon-root trick: `docker run --rm -v /worktank:/mnt alpine
  sh -c 'mkdir -p /mnt/<x> && chown -R 1000:1000 /mnt/<x>'`.

## Current state / in-flight

_As of 2026-06-11:_

- **Mac Pro migration is the big open project** — `migration-plan.md` (repo root) frames a
  WORKSTATION-ONLY migration of nh3-dev's dev env to an M2 Ultra Mac Pro Rack, racked in NH3 on
  the 10.100 subnet. nh3-dev wears two hats; only **Hat 1** (the ~40 repos, all of `~/.claude`,
  dotfiles, toolchain) moves. **Hat 2** fleet sidecars (egress SOCKS5 proxy, mead-hall, ttyd
  seats, `althing-forseti`) STAY on the Linux VM. **Phase 0 is the only time-sensitive step**:
  push the pushable repos + confirm which of ~6 no-upstream repos are local-only. Cutover =
  `rsync` the working trees, **NOT re-clone** (re-clone would lose unsaved work in ~15 repos).
  macOS gotcha pre-flagged: `/home/lkraven` → `/Users/lkraven` repath. Everything else waits on
  the hardware landing in the rack.

- **Two commits/branches committed-not-pushed** (operator's push call): `95b2701` (GLM thinking
  config) on main; and the R17 v2 results branch `r17-v2-characterization` on the irv-ml1
  checkout (push HELD — see decisions).

- **R17 v2 corpus characterization DONE on irv-ml1** (brokkr, ~8.5h): 119 clips → **18 MULTI /
  101 ELIGIBLE** (Stage-0 drop list) + 19936 candidates + ASR speech + blend verdicts. Outputs
  on the local branch; **push HELD** — no-push rule AND the `speech.json` are ASR transcripts of
  the intimate-audio batch (content-exposure call). Awaiting brokkr-collect or explicit push approval.

- **pi coding agent + GLM 5.1 harness live on nh3-dev** — `glm` launcher runs pi against `glm-5.1`
  (thinking-off) via the LiteLLM gateway; `glm-5.1-reasoning` for opt-in thinking.

- **Worldtree config-propagation lane is mature + humming** — pre-merge delta-ping → sync-on-merge.
  All of v0.33.x→v0.34.1 reconciled to the demo+personal bind-mounts; demo/personal both on v0.34.x,
  pinned stays pre-cutover (full migration at its next re-image).

- **Disclosed-keys hygiene queue** carries forward (rotate at convenience): HF token, the wt-personal
  keys, Gitea runner reg token, MINIFLUX_PASSWORD. `ZAI_API_KEY` (z.ai MCP web-tools key) now also
  lives in `~/.config/secrets/env.sh` (shared from Worldtree env for the Brokkr panel).

- **Still open from prior:** clean legacy `news-digest` on ana-docker; pin llama-swap to GPU 0; the
  `docker push 60s ceiling` mystery uninstrumented.

## Recent decisions

- `[2026-06-11]` **GLM thinking inverted at the LiteLLM gateway** (operator call): `glm-5.1` defaults
  thinking-OFF; new `glm-5.1-reasoning` alias = same z.ai upstream with thinking ON (opt-in).
  Mechanism: `litellm_params.extra_body:{thinking:{type:disabled}}` — LiteLLM `drop_params` STRIPS a
  top-level `thinking`/`reasoning_effort`, but forwards `extra_body` verbatim to z.ai (the only channel
  that works; verified reasoning_tokens 0 vs >0). Shared-gateway change — affects ALL glm-5.1 callers
  (brokkr's all-models key included). (`95b2701`, auto-memory `reference_litellm_gateway`)

- `[2026-06-11]` **pi coding agent installed on nh3-dev as a GLM 5.1 harness** — `@earendil-works/
  pi-coding-agent` via **bun** (user-level; npm's global prefix is `/usr` → needs sudo, bun avoids it).
  Config `~/.pi/agent/models.json` (litellm provider → gateway), launcher `~/.local/bin/glm` sources
  the gateway key + selects the model. pi is OpenAI-compatible; proxy-safe compat flags for the GLM path.

- `[2026-06-11]` **z.ai web-tools (regin) = z.ai hosted MCP path, NOT the `/paas/v4` Tool API.** Direct
  `api.z.ai/api/paas/v4/web_search` → 429/1113 "insufficient balance" (coding-plan keys bill tools on a
  separate quota path); `open.bigmodel.cn` is the China platform (404, different account). WORKS: MCP
  streamable-HTTP at `https://api.z.ai/api/mcp/{web_search_prime,web_reader}/mcp`, `Authorization: Bearer
  $ZAI_API_KEY` (the **MCP** key — distinct from `Z_AI_API_KEY` the LLM key). Reference impl = Worldtree's
  Leif agent (`tools/web/zai_client.py` + `core/clients/mcp.py`).

- `[2026-06-10]` **Mac Pro migration framed: workstation-only** (M2 Ultra ARM, racked NH3 on-subnet);
  sidecars stay Linux. `migration-plan.md`. (See in-flight.)

- `[2026-06-10]` **Worldtree deployed-config propagation is infra-ops's OWNED lane** (operator ruling).
  worldtree-dev pings the config delta on every config-touching commit (pre-merge); infra-ops syncs
  `config/*.yaml` from MERGED canonical to the `/opt/worldtree*/config` bind-mounts on demo+personal
  (Heimdall hot-reloads policies; defaults are code-defaulted). The **v0.33.8 9-HOUR demo outage** — a
  `model_roles.yaml` hard-startup-dep that shipped in canonical 3 releases earlier but never reached the
  VM — is the failure mode this lane prevents. providers.yaml stays hand-tuned (artemis graft on
  personal). corviduo-dev emergency-ops = `ssh vh@10.250.50.152` (alias doesn't resolve, lkraven denied,
  infra-ops key excluded), docker no-sudo. (auto-memory `reference_worldtree_deploys_cicd`,
  `reference_corviduo_dev_emergency_ops`)

- `[2026-06-09]` **LiteLLM scoped virtual keys issued to consumers** (operator-authorized): `brokkr-
  smithy` (all-proxy-models), `arbo-prompt-enhance` (comfy-dev, granite-only). Pattern: mint via
  `/key/generate` (master `sk-corvid`), scope-restricted + rotatable, value → 600 file never the bus.
  (auto-memory `reference_litellm_gateway`)

- `[2026-06-08]` **volva.service + heid.service removed from nh3-dev** — vestigial systemd daemons;
  Heid/Volva re-architected from Python pollers to Claude Code session orchestrators (heid commit
  `12aa5a9`); binaries+dirs gone, volva.service was crash-looping 203/EXEC. Cleanup at heid's request
  (the bus-content-driven sudo got the harness guardrail; operator green-lit). (`6e2f80e`)

- `[2026-06-05]` **Granite 4.1 8B FP8 replaced phi4-mini as the production summarizer** (supersedes
  the 2026-06-04 phi4 decision below). Beat phi4 on precision in brokkr's R15 P03. **Staying FP8, not
  Q4/AWQ** — primary workload (agent memory + summarization) is high-concurrency, where FP8-on-Ada
  scales ~linearly (profiled 2010 tok/s @ C=32; single-stream 67.5 is batch-1 GEMV physics, not a
  config bug — placement/kernel/contention all ruled out). vLLM `vllm-granite` :8004 GPU 1, official
  IBM compressed-tensors FP8, CUDA graphs. **GPU-1 retune** (trio utils 0.2/0.2/0.3→0.07/0.07/0.18,
  granite 0.36) freed ~10 GB → CUDA graphs + a Granite-text-LoRA hedge. nevermore repointed. (`34a43a0`,
  auto-memory `reference_ana_ml2_vllm_granite`)

- `[2026-06-05]` **Langfuse v3 stood up on ana-docker (:3001) as the gateway trace UI**; LiteLLM
  `success_callback:[langfuse]` live (project `gateway`). Pretty prompt/completion/reasoning traces +
  an `outputTokensPerSecond` tok/s dashboard. NOT a prerequisite — spend_logs already capture
  tokens+latency. (`9171e6a`, auto-memory `reference_litellm_gateway`)

- `[2026-06-05]` **Ollama BANNED fleet-wide** (operator directive) — never stand one up; tear down any
  found; serve via llama-swap or vLLM. Torn down irv-ml1 :11434 (freed 19 GB). (auto-memory
  `feedback_avoid_ollama`)

- `[2026-06-05]` **ComfyUI / FLUX.2 work split to `~/development/comfy-dev`** (dedicated repo + agent).
  FLUX.2-klein (fp8 + q8 GGUF, stock + uncensored encoders) installed on the irv-ml1 Docker ComfyUI;
  eshpfi keeps the `comfyui` stack compose, comfy-dev owns the model/workflow knowledge. (auto-memory
  `reference_irv_ml1_ampere_quant`)

- `[2026-06-05]` **Worldtree summarizer config refresh DEFERRED to Worldtree #254** (granite-4.1-8b is
  the structured-output profile, ON HOLD, no live consumer; the conversation summarizer defaults to
  claude-haiku — the "phi4 erroring" premise was wrong). No instance changes now; worldtree-dev hands
  the exact providers.yaml + consumer config when #254 un-holds, infra-ops applies to the bind mounts.
  **CORRECTION to the 2026-06-04 "deploys ALL CICD" line:** the bind-mount CONFIGS (providers.yaml,
  vh-owned on corviduo `/opt/worldtree*/config`) ARE infra-ops's to apply directly — only the
  app/image DEPLOY is CICD; the `.env` is deploy-owned. (auto-memory `reference_worldtree_deploys_cicd`)

- `[2026-06-04]` **phi4-mini FP8 on ana-ml2 vLLM is the nevermore summarizer/dreaming agent;
  granite-4-small retired** from llama-swap (config-only; GGUFs on disk). 50K ctx (dropped from
  Phi-4's 128K max to fit GPU 1's ~10 GB free) + FP8 KV. (`40a374b`)

- `[2026-06-04]` **phi4 ships the CANONICAL/official Phi-4 chat template, NOT Ollama's.**
  Ollama's bundled template omits the system `<|end|>` — that flattered brokkr's R15 eval but is
  the DIVERGENT scaffold (Dvalin: the system `<|end|>` is Microsoft's intended format). Applied an
  Ollama-matching override then reverted — ship correct, not the benchmark quirk. (`90e08f0`→`27eb537`;
  "headgun" lesson in Tried.)

- `[2026-06-04]` **infra-ops NOPASSWD-sudo identity commissioned, scoped to PFI boxes** (+esh-docker-vm
  by operator override) — so infra-ops completes DevOps end-to-end vs handing the operator sudo steps.
  Dedicated key, sudo log_output, key-gated. (`8c32a05`)

- `[2026-06-04]` **Worldtree demo/pinned/personal deploys are ALL CI/CD, not infra-ops** — a "deploy
  vX.Y.Z" request to infra-ops is MISROUTED → point them back to their pipeline. The granite→phi4
  repoint: worldtree-dev self-served via their CI/CD (v0.30.10). (`d8d776c`)

- `[2026-06-04]` **ollama upgraded 0.9.0→0.30.4 on irv-ml1** (Ministral-3 is a Dec-2025 model the
  old engine refused); A6000 pinned by **UUID** not index (native fastest-first ≠ nvidia-smi PCI).

- `[2026-06-04]` **`brokkr` user (no-sudo) on irv-ml1; R14/R15/R16 substrate moved to /home/brokkr.**
  Persistent box services there need SYSTEM systemd units (see Tried).

- `[2026-06-03]` **yt-voice-clipper bot-gate fix = route yt-dlp through NH3 residential
  egress, NOT cookies/PO-token.** YouTube hard-flags the Irvine colo IP (LOGIN_REQUIRED on a
  public video even with no cookies). Cookies + the bgutil PO-token + deno JS-runtime all
  loaded fine — the gate is pure IP reputation. Operator chose proxy-via-nh3-dev → durable
  dante proxy → proven. The egress proxy is a reusable fleet lever for any datacenter-IP-gated
  service.

- `[2026-06-03]` **yt-voice-clipper push-to-deploy via gitea webhook** (operator-directed,
  after 6 manual rebuilds in ~40 min). Webhook (not poll) — gitea CAN reach the WG IP per the
  operator. The proxy env + Homepage labels live in the **host-specific override** (untracked
  → survive the auto-deploy's `git reset --hard`), NOT yt-voice-clipper-dev's image. Runbook
  `d4f180d`.

- `[2026-06-03]` **R14 scope = (a) provision-only.** infra-ops provides box + CUDA env +
  engines + weights + NFS; brokkr/dev wires `arms.py` + runs — keeps infra-ops OFF the
  VIVAE-processing path (VIVAE = Variably Intense Vocalizations of Affect/Emotion, CHARTER §4
  highest-liability; operator authorized R&D-eval-only, quarantined). Box = irv-ml1 (A6000
  free; ana-ml2 GPU-saturated). Per-engine venvs (divergent torch stacks); A6000 = `cuda:0`
  NATIVE (≠ docker `=1`).

- `[2026-06-03]` **Declined worldtree v0.30.4 staging deploy** — that's worldtree-team's
  CI/CD lane (a developer `staging/vX.Y.Z` git-tag promote), not infra-ops. They self-corrected
  to the same conclusion independently.

_40 older entries archived to archival-memory.md._

## Tried and abandoned

- `[2026-06-11]` **A completion-poll `while pgrep -f <scriptname>` SELF-MATCHES its own remote shell
  argv** — the poll's command line contains the script name, so its own `pgrep -f` always finds itself
  → the loop never exits, the poll never fires. Use a match pattern ABSENT from the poll command (pgrep
  the python stage, or a sentinel file), not the driver's own name. (Caught only because the operator
  asked "check status"; the job had already finished cleanly.)

- `[2026-06-08]` **Demucs `uv pip install demucs` pulls torch 2.12/torchaudio 2.11 → `ta.save()`
  requires torchcodec → dies AFTER separating (0 stems written, rc=1).** Same class as the torch-2.12
  torchcodec foot-gun below. Fix = pin `torch==torchaudio==2.4.1` (pre-torchcodec save backend) +
  `UV_LINK_MODE=copy` for the EPERM-hardlink quirk. Lesson restated: validate the SAVE path, not just
  import + GPU inference, on a bleeding-edge torch.

- `[2026-06-05]` **vLLM 0.19 CUDA-graph-capture OOMs on a SHARED GPU** — it fills the KV cache to the
  `--gpu-memory-utilization` budget WITHOUT reserving graph-capture memory, so `capture_model` OOMs
  AFTER weights+KV load (model/KV log looks healthy, then crash-loops; saw 11 restarts at util 0.36
  with 237 MB free). Fix: free co-tenant room (right-size the other vLLM services) OR `--enforce-eager`
  (no graphs, ~15-25% slower decode). FP8 single-stream is batch-1 GEMV (memory-bound, FP8 tensor cores
  need batch>1) → Q4 wins single-stream by physics; FP8 wins under concurrency. (`reference_ana_ml2_vllm_granite`)

- `[2026-06-05]` **Langfuse has NO public dashboard-creation API** — dashboards/widgets are postgres
  rows (`dashboards`/`dashboard_widgets`); build by cloning a default-dashboard row + swapping the
  measure. tok/s is NOT a per-generation field (null on the observation) — it's the
  `outputTokensPerSecond` MEASURE, computed at metrics-API/dashboard query time; no native per-call
  tok/s display exists (streaming doesn't change that). langfuse-web needs `HOSTNAME=0.0.0.0` (Next.js
  standalone binds one net-IP otherwise, unreachable via the published port once also on tnet). Host
  3000 is gitea's → langfuse on 3001.

- `[2026-06-05]` **`sudo` over non-interactive ssh FAILS SILENTLY where the user lacks NOPASSWD** (esh +
  corviduo are OUTSIDE the infra-ops identity) → empty output misread as "empty file." Read
  world-readable files WITHOUT sudo. corviduo ssh = `vh@10.250.50.152`; bind-mount configs are
  vh-owned (editable), the `.env` is deploy-owned 600 (vh can't edit it, no sudo).

- `[2026-06-05]` **Worldtree summarizer-model is NOT an env var** — no `WORLDTREE_SUMMARIZER_MODEL` on
  the containers; it defaults to claude-haiku in code, opt-in via config not `.env`. Don't trust an
  ".env-flip" recipe — inspect the live container env + the vh-owned config files first. (Inspection
  corrected a wrong "summarizer erroring on phi4" premise → saved churning 3 live instances.)

- `[2026-06-04]` **Ollama/llama.cpp-BUNDLED chat templates silently diverge from canonical HF —
  the "headgun" lesson.** Ollama's phi4 template drops the system `<|end|>`; serving vLLM with the
  model's HF tokenizer template (canonical, has it) regressed brokkr's Ollama-measured R15 baseline
  -33pp type-F1 while valid_format held 1.0. An Ollama-matching `--chat-template` "fixed" it but was
  the WRONG fix (the bundled scaffold is the divergent one). PRINCIPLE: serve each model's canonical
  `tokenizer.apply_chat_template`, not the bundled template — bundled ones corrupt baselines. Verify
  the applied prompt via vLLM `/tokenize`→`/detokenize`. (`90e08f0`/`27eb537`)

- `[2026-06-04]` **GPU pin by INDEX is ambiguous on irv-ml1** — native CUDA orders fastest-first
  (A6000=0) but nvidia-smi/docker use PCI order (A6000=1), so an index pin can land on the wrong
  card. Pin by **UUID** (`CUDA_VISIBLE_DEVICES=GPU-…`); verify via nvidia-smi compute-apps. Check
  loaded-model VRAM with `ollama ps` (Ministral-3 @ its 256K default ctx = ~30 GB; cap num_ctx).

- `[2026-06-04]` **Persistent services on irv-ml1 need SYSTEM systemd units** — the box reaps
  user-session processes on ssh disconnect, and `--user` systemd isn't reachable over non-login
  ssh, so nohup/setsid/`screen -dmS`/`systemd-run --user` all die (even with `enable-linger`). Use
  `/etc/systemd/system/`.

- `[2026-06-04]` **pyworld needs `setuptools<81`** (imports the removed `pkg_resources`); and
  **R/soundgen `-lgfortran` fails** on irv-ml1 because the default `gcc` is gcc-11 but only
  gfortran-12 is present (libgfortran.so lives only in the gcc-12 dir) → install `libgfortran-11-dev`.

- `[2026-06-04]` **homepage "crash" ≠ always NFS** — a wedged container in unkillable D-state
  ("tried to kill container, but did not receive an exit event") can come from dead `siteMonitor`
  widget targets (retired ESH firewall IPs) hanging the node event loop into `exit_mmap`, needing a
  host reboot. Check homepage's siteMonitors against retired hosts. (`incident_esh_docker_nfs_boot_race`)

- `[2026-06-03]` **gitea webhook to a private IP is denied by `webhook.ALLOWED_HOST_LIST`**
  (anti-SSRF; default `external` blocks private/loopback). Symptom: delivery shows
  `dial tcp ...: webhook can only call allowed HTTP servers`. Fix = APPEND the target net to
  ALLOWED_HOST_LIST in gitea's app.ini (keep `external`; scope tight, never `*`/`private`) +
  restart gitea (act_runner job containers survive a restart). gitea runs as a container on
  ana-docker (`gitea_gitea_data` volume, `/data/gitea/conf/app.ini`).

- `[2026-06-03]` **torch-2.12 venvs need `uv pip install torchcodec`** — torchaudio 2.12
  defaults to the TorchCodec backend for `.load`; without it, real audio I/O throws "TorchCodec
  is required" — and it ONLY surfaces at actual conversion, NOT at import/model-load. Lesson:
  validate real I/O, not just import, when provisioning ML engine envs. (seed-vc on torch 2.4
  uses the legacy backend, exempt.)

- `[2026-06-03]` **Backgrounding `althing-cli monitor` with an inline shell `&` (instead of
  the Bash-tool `run_in_background`) orphans it** — it survives the shell exit, holds the
  per-handle flock UNTRACKED (won't notify the session), and `stop-monitor` doesn't detect it.
  Fix: find + kill the orphan PID (verify cwd=this repo / handle first — nh3-dev is shared, other
  agents' monitors run there too), then re-arm via run_in_background. Always re-arm tracked.

- `[2026-06-03]` **`uv pip install .` fails on SmoothKen/knn-svc** (and similar script-repos)
  — it's analysis scripts + a poetry pyproject, no buildable package (setuptools
  package-discovery error). Install the pyproject deps directly, don't build the "package".

_44 older entries archived to archival-memory.md._