esh-pfi-infrastructure/persistent-memory.md

Persistent memory — eshpfi-management

Last updated: 2026-09-25 ~1555 PT (⭐⭐ NH3 visit DONE: SB off, IGFX restored, AMT cabled but NOT on the network. nh3-ml1 LIVE + parity-verified + monitored; gateway routing = Prime's call. lxc-pve fix; kernel -43 btmtk oops noted. gx10 AC-restore VALIDATED.)

Always check for /tmp/infra-ops-handoff.md — if it exists and its Written: stamp is under 8 hours old, read it (it carries the in-flight handoff from the previous session), then delete it. Older than 8 hours: stale — delete it unread.

(Raised from 1 h to 8 h by operator 2026-09-13 — a one-hour window deleted the handoff unread across any overnight gap, which is the exact case it exists for. 8 h also matches the global CLAUDE.md and the /snapshot skill default.)

Repo purpose

• 2026-09-10 Beszel fleet wiring: all seven requested hosts plus existing corviduo-dev report up. /tank and other data filesystems now have real usage metrics; NVIDIA telemetry covers ana-ml2 and irv-ml1. Thirty alerts deliver to infra-ops, explicitly chosen by operator; Miranda routing is deferred. A real low-threshold disk alert reached althing, then the threshold was restored to 85%/5 min. Homepage has one native overview widget (reachability counts, not degraded health). Dedicated superuser approved and stored in Vaultwarden. See persistent-memory.d/2026-09-10-beszel-fleet-wiring.md and stacks/beszel/README.md.

Reference workspace for PFI infrastructure: server inventory, canonical Docker Compose stacks, ops playbooks, and conventions. Authoritative copies of compose files live on the servers under /opt/docker/compose/<stack>/; this repo mirrors them for version control, editing, planning, and CI-driven deploys. It was originally spun up to handle the fleet backups — keep that lens when triaging backup/storage issues.

Tools and conventions

Sister repos (separate gitea repos, deployed by playbooks here):

Repo	Role	CI status
vh/task-board	MCP + web dashboard for assistant task state (port 7878) — [2026-09-24] MOTHBALLED by Prime (superseded by the High Seat + ledger); container removed on ana-docker, data/image/compose kept (e6da607)	push-to-main → CI deploys (2026-04-29)
vh/vor	Inquisitor UI sidecar (port 7879)	push-to-main → CI deploys (2026-04-29)
vh/nevermore	Twice-daily LLM-curated briefing (port 8181, replaces news-digest)	push-to-main → CI deploys (2026-04-30)
vh/asset-engine	Internal control plane over inference services (port 8200, LAN-direct)	push-to-main → CI deploys (2026-05-12)
vh/althing	Lean trusted inter-agent message bus — v3.0.0 "the post office" as of 2026-08-28 (U9b flag day, one-way, no rollback): ONE container on nh3-dev at http://10.100.50.40:8390 is the only stateful component; althing-po-herald one per box; althing-listen one per session; postbox is the client. Every v2 command was DELETED, not deprecated — althing-cli→postbox, althing-wake-listener→althing-listen, althing-light-monitor/althing-receiver gone. Sessions need BOTH ALTHING_POST_OFFICE and ALTHING_HANDLE; there is no default address. ⚠ An unreachable post office is an OUTAGE, never an empty inbox. → persistent-memory.d/2026-08-28-althing-v3-cutover.md	per-box install (NOT CI-deploy); nh3-dev = container host + repo; nh3-extdev = system WHEEL at /opt/uv-tools, needs its own wheel install (playbooks/nh3-extdev-althing-v3.yaml)
vh/mead-hall	Bifrost tool-provider sidecar (port 5173 on dev VM 10.100.10.50)	push-to-main → CI deploys (2026-05-16)
vh/skaldsong	Wizard + reader surface (port 8300, ana-docker, registry-pull pattern)	push-to-main → CI deploys (2026-05-19)
vh/Worldtree	Conversation API (corviduo-dev demo :8080 / personal :8081 / pinned :8082) — Heimdall auth, Bifrost integration. gitea-runner builds on ana-docker; claude-bot ADMIN collaborator (2026-06-20). Now v1.0.0b19.	push-to-main → CI build-and-deploy (runner on ana-docker)
vh/yt-voice-clipper	YouTube → diarized voice-clip dataset builder + audition console (irv-ml1 :8000)	push-to-main → gitea-webhook auto-deploy to irv-ml1 (2026-06-03) — see docs/runbooks/ytvc-autodeploy.md
vh/arbo	Catalog-driven ComfyUI engine (irv-ml1 :8201, comfy-dev owns engine/catalog/image)	push-to-main → gitea Actions CI (deploy-engine.sh, build-local, health-gated) now LIVE; catalog via :9009 webhook
vh/zonos-gateway	OpenAI-compatible TTS gateway over stock ZONOS2 (:8890 irv-ml1); emotion dials-first + voice mapping; reached via LiteLLM ext-tts alias. v0.2.1 (2026-07-18): voice-resolved emotion presets (resolve_preset(name,voice); angry/happy/startled_happy per-voice). 8 voices incl. 4 clones	pushed to gitea (main 8f1885b/v0.2.1); deployed irv-ml1 tree still NON-git (hand-updated build context — CI-wire = open follow-up). Spec docs/EMOTION-DIALS-SPEC.md; host-managed voices bind-mount (./voices:/app/voices, drop wav + restart, no rebuild)
vh/soong-lab	Noonien Soong character-design studio (SPA + /api + WT /bifrost/tool-call); containerized 2026-07-18, LIVE on corviduo-dev :8443 (image vh/soong-lab:latest). soong-dev owns Dockerfile/compose/workflow; infra-ops owns the host	CI = Gitea Actions build+push+DEPLOY on tag/dispatch (fleet recipe: docker:cli + raw buildx, pushes AS vh; auto-redeploy LIVE 2026-07-18 — runner SSHes corviduo-dev as deploy, compose pull && up -d from /opt/soong-lab, health-gated on /api/version). Manual redeploy sudo -u deploy bash -c 'cd /opt/soong-lab && docker compose pull && docker compose up -d'. → archival-memory.md (archived 2026-08-16)
model-training-forge (mtf-dev)	Fine-tuning recipe forge; T1 = E-RP writing LoRA, retargeted qwopus-122B→AEON-27B (2026-07-06) (SFT→DPO, LitBench-RM reward)	training runs, not a deployed sidecar

(vh/volva + Heid were re-architected from systemd daemons to Claude Code session orchestrators 2026-06-08; their nh3-dev .service units were removed — no longer deployed sidecars here. See Recent decisions.)

• Two-layer backups — Backrest orchestrates restic for file+DB (5 fleet repos, daily 01:00 PDT); PBS-ANA primary + PBS-NH3 DR mirror for VM images. ana-nas is the SPOF for postgres + PBS-ANA datastore + cross-site restic targets — see docs/runbooks/disaster-recovery.md for the blast-radius matrix. ⚠️ The restic file+DB layer routes through TWO rest-servers (rest-server-ana @ ana-docker:8000 → ana-docker/ana-ml2/esh-docker-vm/vm-esh-nas; rest-server-nh3 @ nh3-nas:8000 → irv-ml1/nh3-docker). Both depend on their NAS's NFS export of /mnt/backup. (rest-server-ana recovered 2026-06-20.)

• pull-hf-repo.yaml is the canonical "get a HuggingFace model/dataset onto ana-ml2's shared cache at /tank/aimodels/huggingface/" playbook. Supports --var repo_type=model|dataset|space. Replaces ad-hoc huggingface_hub.snapshot_download patterns.

• Worldtree admin auth — per-instance. Each Worldtree deployment (demo :8080, personal :8081, pinned :8082) has its own Heimdall registry and its own bootstrap admin key. Infra-ops's stored long-lived admin key (key_id 61419c92) at ana-docker:/opt/docker/conf/.secrets/worldtree-infra-ops-admin auths against demo only. Personal-instance admin (the ~/.config/worldtree/personal-admin-token, mode 600) POSTs /admin/keys (mints per-project keys; takes user_id+label, no scope param — scopes are tier-derived). On-instance mint recipe (cleaner than DB-manip): docker exec worldtree-worldtree-api-1 POST /admin/keys with the in-container WORLDTREE_BOOTSTRAP_ADMIN_KEY; cleartext once in .key=wt_live_+16hex. auto-memory reference_worldtree_demo_key_mint.

• Per-project user keys against personal Worldtree (issued 2026-05-19): skaldsong:79744637, skaldsong:7c1dbbbe, althing:50d85460, mead-hall:a360822d. Mint via /admin/keys, drop value to /tmp/wt-personal-<name>.key mode 600, dev collects + shreds (DO NOT cat to chat transcript).

• Skaldsong CD pattern (registry-pull). vh/skaldsong's CI builds and pushes gitea.phasefinal.com/vh/skaldsong:<sha> + :latest; playbooks/deploy-skaldsong.yaml on ana-docker pulls + recreates. SHA-pin only. Prereq: host needs docker login gitea.phasefinal.com once.

• gitea internal route for fleet hosts. gitea is a container on ana-docker — git-SSH 10.250.50.70:222, HTTP :3000. Fleet/colo hosts must use this internal route, NOT public gitea.phasefinal.com (38.120.12.44) — the public path fail2bans the host egress IP. Full gotcha in docs/orientation.md → Git/gitea.

• docker-as-root pattern (for ops with no admin API, or to edit deploy-owned/root-owned files without sudo): docker run --rm -v <target-dir>:/wt docker:cli sh -c "...". docker-group membership is effectively root via bind-mount. Foot-gun: relative paths in compose.yaml resolve against the sandbox CWD but the daemon interprets them against the HOST fs — always pass -e VAR=/abs/path for any relative-default config dir.

• scripts/elway sudo handling — elway prompts for the sudo password ONCE via getpass before the first sudo: true step → can't run unattended from a non-TTY tool if any step needs sudo. Sudo-free playbooks run fully non-interactive over key SSH.

• Per-host SSH identity matters for sudo. infra-ops has NOPASSWD sudo on most PFI Linux boxes (corviduo-dev included since 2026-06-15). On ana-docker: default ssh ana-docker = lkraven (docker-group, NO passwordless sudo); ssh infra-ops@ana-docker HAS NOPASSWD root. → For any sudo op on ana-docker, use ssh infra-ops@ana-docker. ssh infra-ops@10.100.10.50 (nh3-dev) ALSO NOPASSWD sudo; on nh3-extdev infra-ops is sudo-LESS by design (ssh lkraven@10.100.50.42 is the NOPASSWD path) — [2026-09-23] measured: infra-ops HAS NOPASSWD sudo on nh3-extdev (fleet ownership audit; also CLAUDE.md 2026-09-05). irv-ml1: ssh irv-ml1 = lkraven, docker-group (plain docker) but sudo needs a PASSWORD (no NOPASSWD) — stage model pulls to /home, not root-owned /worktank.

Current state / in-flight

As of 2026-09-25 ~1555 PT.

⭐⭐ NH3 site visit DONE; nh3-pve back up 1523; nh3-ml1 LIVE (not yet in the gateway)

Prime, after the visit: "secure boot disabled, amt enabled, card plugged." The post-boot checklist is all green: the uplink is on enp2s0f1np1, all 7 onboot guests are up, NFS automounts, the post office is push/reachable, and svos, hermes-gateway, Booth, ttyd, zellij-web and dev-backup are running.

• Secure Boot OFF. IGFX restored (00:02.0 boot_vga=1), so the NanoKVM should have video again (not checked). NVIDIA 580.178.04 DKMS is on the host.
• nh3-ml1 = CT 109 @ 10.100.50.80, TEI embed+rerank, parity-verified vs esh-ml1: the two hosts cannot be told apart and run at the same speed. Monitoring is wired: Beszel 1feeeq61g4mkqre with 5 alerts, Kuma #29/#30, Homepage cards, Dozzle agent (hub 7→8 clients). DNS nh3-ml1.nh3.internal is set. servers/nh3-ml1/README.md exists.
• ⏳ Prime's call: gateway routing. Recommendation: load-share qwen3-embedding + reranker across esh-ml1 and nh3-ml1.
• ✅ AMT on nh3-mgmt, static 10.100.250.61 (2306), nh3-pve-amt.nh3.internal, Homepage NH3-PVE-AMT. Password in the vault as nh3-pve/amt-admin.
  • UDM port 6 is native mgmt with tags blocked.
  • KVM on, redirection listener on, Opt-in None; set through scripts/amt-wsman.py.
  • Root cause earlier tonight: Linux had the port down (the igc PHY was off). Fixed with auto enp88s0, IPv6 off and arp_ignore=8.
  • ⏸ AMT follow-ups PARKED (Prime, 2026-09-25 2326) until the two new MS-03s arrive and esh-pve's MS-01 is also on AMT. Parked items:
    • MeshCommander test.
    • Dummy HDMI plug, then the NanoKVM → gx10 move.
    • An OOB path that does not run through nh3-pve. Today ANA/ESH → NH3 goes via nh3-scale, CT 107 on nh3-pve, so remote AMT dies with the host. Recommended fix: route 10.100.250.0/24 over the FortiGate↔UDM IPsec pfi-ana-nh3.
    • Adding AMT to TRMM's MeshCentral (rmm-mesh, public IP; wants 2FA).
• Found and fixed: lxc-pve 6.0.0-1 broke Docker in the CT (runc 1.5 sysctl reopen denied). Upgraded that one package to 6.0.0-2 (Proxmox fix #7006), and gpu-lxc.yaml now does this itself.
• Found, NOT fixed: the headers install on 09-25 pulled kernel 6.8.12-43, which booted at the visit. It oopses in Bluetooth (btmtk) on every boot, and so does esh-pve on -42. Benign so far. The proposed fix is to blacklist btusb on both PVE hosts, effective at the next boot.
• Not yet proven: nh3-ml1 coming back by itself after an nh3-pve reboot. nvidia-persistenced is ordered before pve-guests, but only the config was checked; nothing has been rebooted. → persistent-memory.d/2026-09-25-nh3-ml1-live.md

⭐ esh-ml1 = the fleet's utility GPU host (2026-09-25): embed/rerank (TEI) + reward (vLLM)

TEI is the fleet embed/rerank engine (Prime) and esh-ml1 is the SOLE gateway backend (nh3-ml1 is live and parity-verified; routing it in is Prime's call). The Skywork reward seat moved there too; the audit found nothing supersedes it. fv-ml1 GPU 1 freed ~15 GB across the day; only vllm-coder is left in stacks/vllm, and parakeet stays on fv-ml1. Monitoring is wired: Beszel with GPU and 5 alerts, Kuma #27/#28, Homepage cards, a Dozzle agent. Open follow-ups:

• worldtree-dev: Domari's Skywork path is broken (dead IP + wrong schema). Told 2026-09-25; waiting on them.
• ✅ nh3-dev Beszel agent no longer binds NAS shares (it died on every NH3 cold start); fixed 2026-09-25, stacks/beszel/hosts/nh3-dev.yaml.
• Dozzle nh3-docker agent has been stopped by hand since ~2026-04. Revive or drop.
• Beszel superuser password was echoed into this session's transcript (local only, not in memory). Rotation offered to Prime. → Recent decisions.

✅ homepage wedge cleared — esh-docker-vm rebooted 2026-09-25 0613 (Prime)

Prime ordered the reboot and ruled do NOT move Homepage (stays on esh-docker-vm). Down 0613:44–0616:58 PT; homepage needed a manual docker start (a failed restart had marked it stopped). Signature + recovery: servers/esh-docker-vm/README.md.

✅ pfi-gx10 power-on after AC loss — VALIDATED 2026-09-25 (Prime)

The UEFI patch works: an AC pull and replug brings the box up by itself. A shutdown is still not a test. Re-check after any BIOS update. Revert details are in servers/pfi-gx10/README.md. → persistent-memory.d/2026-09-24-gx10-ac-restore-patch.md

Still open from 09-22

• Build the NRestarts flap sampler — approved 2026-09-22, not started (design + traps in services/althing-notify-failure/README.md; 163bb97).

Live threads

• Claude sessions on nh3-dev: the ~1430 shutdown killed them again; relaunching is Prime's call.
• Homepage hermes-gateway seat rename resolved (2026-09-25): the operator's 2026-09-24 handle split is real (postbox handles records it; infra-hermes confirmed the seat daemon carries hermes-gateway). The rename was already live and is committed as 422a27c.
• High Seat backup proof DONE 2026-09-25 0111: infra-hermes reported and I verified that restic snapshot cd113bef (nh3-dev, 0100) contains ~/.high-seat/reports/grok.963ddd3f428b7fc7.jsonl. Told him to retire the daily 0110 check unless svos-dev asked for an ongoing one.
• ESH has a single outside route (esh-scale on esh-pve) — noted, untracked.
• origin/main is current through 58390a4. Everything after it is unpushed (2bf418c…7ddd116 plus this snapshot, 2026-09-25). Pushing is Prime's call.

Recent decisions

• [2026-09-26] Two dataset-foundry utility seats LIVE on nh3-ml1 (brokkr; operator approval relayed): LFM2.5-VL-3B on llama.cpp :8030 (gateway lfm25-vl-3b, LiteLLM restarted 36 s at 0039) and VibeVoice-ASR-Streaming-1.5B on audio.cpp :8031, not vibevoice.cpp as specced; the GGUF card names audio.cpp. Controls: VL read a synthetic image exactly, but hallucinates when no image is sent; ASR WER 3/69 on the bundled LibriSpeech clips. libsoxr added (linear resampling misheard a word). First ASR request ~31 s cold. Answered brokkr: the old 27B image seat (10.250.50.54:8015) was retired 09-14; baseline = gateway image-judge (Flash-Next).

• [2026-09-26] Coder seat STAYS on fv-ml1 (Prime). The nh3-ml1 copy gave the same quality (teacher-forced true-code logprob diff +0.008 ± 0.019) but ran ~5× slower (64-tok FIM ~1.0 s vs 0.2 s; 63 vs 338 tok/s), and freeing 6.3 GB on fv-ml1 GPU 1 (20 GB spare) bought little. Copy removed; recipe kept in stacks/coder-seat/. The RTX 2000E suits embed/rerank/classify, not latency-sensitive generation. Only coder was a candidate: parakeet stays (earlier ruling), voices is a generation seat.

• [2026-09-25] nh3-ml1 LIVE after the NH3 visit (SB off, IGFX restored): driver + CT 109 + TEI, parity vs esh-ml1 indistinguishable (cos min 0.999993 = own floor; overlap@10 1.000 vs MRL-256 control 0.684), same speed; monitoring/DNS wired. Autonomous: lxc-pve 6.0.0-2 upgrade on nh3-pve (Docker-in-LXC fix #7006). Gateway routing + AMT follow-up are Prime's. → persistent-memory.d/2026-09-25-nh3-ml1-live.md

• [2026-09-25] nh3-ml1 = CT 109 @ 10.100.50.80, second TEI embed/rerank backend on nh3-pve's RTX 2000E; GPU playbooks made host-generic — BUILD DEFERRED: blocked on nh3-pve Secure Boot until the NH3 visit; SB handling and gateway routing are Prime's calls. Tracked: 7ddd116, Current state. → persistent-memory.d/2026-09-25-nh3-ml1-standup.md

• [2026-09-25] nh3-pve OOB: HOLD blind until the next NH3 visit (Prime via Miranda, 1333). Target: NanoKVM → pfi-gx10, and the MS-01 on its own vPro. The IGFX BIOS fix rides the same visit, since AMT KVM captures only the iGPU. Tracked: servers/nh3-pve/README.md (2b49be4, cdd7605).

• [2026-09-25] MS-01 foot-gun: a GPU in the PCIe slot renames every NIC (the slot's root port takes bus 01, so the X710 goes enp2s0f0np0→enp3s0f0np0). On nh3-pve that means booting with no uplink. Pinned by MAC with .link files before the install, and the AMT port was pulled from vmbr0. → servers/nh3-pve/README.md, playbooks/nh3-pve-pin-nic-names.yaml (2118449)

• [2026-09-25] Direct ESH→esh-ml1 consumer path — PARKED (no ESH-side callers in 7 days; consumers go through the gateway). Tracked at persistent-memory.d/2026-09-24-esh-ml1-embed-rerank.md.

• [2026-09-25] Created empty private repo corviduo/norn (git@gitea.phasefinal.com:corviduo/norn.git) as claude-bot, at brokkr-smithy-dev's relay of the operator's Norn ruling; the norn-dev handle is the operator's to declare.

• [2026-09-25] Reward seat audited (Skywork-Reward-V2-Llama-3.1-8B still #1 of 188 on RewardBench 2; our AWQ ≈ bf16 within noise; double BOS costs ~2.7 pts) and moved fv-ml1 → esh-ml1; esh-ml1 monitoring wired (Beszel+GPU, Kuma, Homepage, Dozzle); Dozzle hub's stale agent IPs fixed; nh3-dev Beszel agent revived. → persistent-memory.d/2026-09-25-reward-move-and-esh-ml1-monitoring.md

• [2026-09-25] TEI adopted as the fleet embed/rerank engine (Prime); esh-ml1 is the sole gateway backend for qwen3-embedding + reranker; fv-ml1's vLLM embed/rerank seats retired (~6.1 GB freed); parakeet stays on fv-ml1. → persistent-memory.d/2026-09-25-tei-fleet-embed-rerank.md

• [2026-09-24] esh-ml1 built — RTX 2000E Ada as CT 110 on esh-pve (host driver 580.178.04 DKMS, loaded live, no reboot), vLLM embed+rerank at measured parity with fv-ml1, wired as LiteLLM order: 2 failover; dead DB alias reranker-a3-bge-v2-m3 (pre-relocation IP) repaired. → persistent-memory.d/2026-09-24-esh-ml1-embed-rerank.md

• [2026-09-24] esh-pve: VM 102 retired, a 14-day hung VFIO process found and cleared, T400 → RTX 2000 Ada; Prime chose LXC + host driver for embed/rerank (implementation deferred to next session, tracked here + servers/esh-pve/README.md). → persistent-memory.d/2026-09-24-esh-pve-vm102-retired-gpu-swap.md

• [2026-09-24] pfi-gx10 AC-restore UEFI patch applied but NOT validated; box OFF until Prime's AC pull 2026-09-25 (tracking: servers/pfi-gx10/README.md, e5197a3). — [2026-09-25] ✅ VALIDATED: Prime pulled and replugged AC, and it came up by itself at 1108:54 (2b49be4). → persistent-memory.d/2026-09-24-gx10-ac-restore-patch.md

• [2026-09-24] NH3 power outage recovered — pbs-nh3 had no onboot (set), NFS boot race fixed with automount (1cbde50), every other Claude session on nh3-dev died. → persistent-memory.d/2026-09-24-nh3-power-outage-recovery.md

• [2026-09-24] Miranda standing order is a repo CLAUDE.md operating parameter (4b29492, aligned to the global send protocol in bcf3342): high-urgency matters go to her, fixed or not; URGENT only when it cannot wait (she phones Prime). Channel verified end to end (thread 01M3A0RP4Q8T0KNGH8TMFSNDA6); it depends on svos + hermes-gateway. Prime's callsign PRiMe / papa romeo mike is a name, not an authenticator (617b759, 62817a2).

• [2026-09-24] task-board mothballed (Prime): container removed on ana-docker, data/image/compose kept, Kuma monitor deleted, task_* instructions removed from CLAUDE.md and the fork template (e6da607). Its hooks had sent no traffic in 30 days.

• [2026-09-24] Military 24-hour Pacific clock times carried into the Codex/Grok shared bootstrap docs/fleettools/AGENT-BOOTSTRAP.md (ad2b4d9); Claude seats get it from the global CLAUDE.md.

• [2026-09-24] Worldtree admin.memory.forget stays OFF on demo/personal until an instance needs it — a destructive erase; enabling it is a per-change operator yes via deploy-wt-config.

• [2026-09-24] A git checkout under root:docker needs safe.directory for its deploy user — the 09-14 normalization (826a63b) silently broke yt-voice-clipper's webhook deploy until v0.3.13; fixed on irv-ml1 and recorded in fleet conventions (eea9eb2). Sweep found no other case.

• [2026-09-23] elway sudo uploads land root:root, validated and staged; fleet ownership audit built; 33 mis-owned root files fixed on 10 hosts. → persistent-memory.d/2026-09-23-elway-ownership-fix-fleet-audit.md

• [2026-09-23] headscale-ddns hardened (fedd4b6): Cloudflare calls retry and validate every body (pick()), no write without both IDs, the run ends on a confirmation, only a global v4 is published, curl -q. Three of four weekly failures were empty zone lookups. heid bug-hunt "Talus" folded (one finding was my own regression).

• [2026-09-23] esh-docker-vm restic was skipped 09-22..23 by my own Kuma move — a dead uptime-kuma lookup aborted pre-backup.sh under set -e (25e41d2). Then, by Prime's decision, the redundant Paperless pg_dump went too: it had failed auth every night since 2026-04-24 and left a 0-byte dump in every snapshot; the DB is covered at source by esh-vm-db's pg_dumpall (6e8da46).

• [2026-09-23] hermes-gateway restart exit-1 is a Hermes race, not a crash — the planned-stop watcher consumes the marker before SIGTERM re-runs the handler; drop-in SuccessExitStatus=1 stops the false OnFailure page at zero coverage cost (c2b0a05). svos had its own stop-timeout (an open board SSE tab), fixed by svos-dev with timeout_graceful_shutdown=5.

• [2026-09-23] Booth link board cleared to 14 durable links (208 removed: drops, release posts, research links, token-bearing URLs; Scriberr reposted at scriberr.fv.internal:8080). Prime's rule: durable debug links only.

• [2026-09-22] Both carried calls approved — build the NRestarts flap sampler (163bb97); the restic content-assertion ruling is ratified and stays (ba60fda).

• [2026-09-22] ⭐ safe-rm installed on nh3-dev and delegated fleet-wide to infra-hermes. ⚠ The package installs INERT and looks fine — Debian's /etc/zsh/zprofile has 0 non-comment lines so the shipped profile.d hook never fires under zsh; verify with command -v rm, never dpkg -l. ⚠ And it does NOT cover the habit that prompted it: measured, it refuses rm -rf /home and deletes an unset-variable path without complaint. Blacklist, not heuristic; set -u is the actual cover. Wiring ruled: /etc/bash.bashrc above the case $- guard, not /etc/environment (an rc self-guards, smaller blast radius, covers bash-over-ssh).

• [2026-09-22] The acceptance probe for a guard must not be able to destroy what it tests (infra-hermes). rm -rf /home to prove safe-rm refuses is a test whose premise IS the thing under test. Guarded form is now standard: [ "$(command -v rm)" = /usr/share/safe-rm/bin/rm ] && rm -rf /home || echo NOT-GUARDED.

• [2026-09-22] ⚠ {"sent": true} is a claim about transmission, never about effect. pane_send structurally cannot deliver a harness command (every relay is prefixed with the card id), so D-0010 promised an unachievable /clear and its receipt reported success. Consumed an operator approval. → persistent-memory.d/2026-09-22-instrument-errors.md

• [2026-09-22] D-0010/D-0011 were misrouted to this seat by pane_find matching a ROLLING PANE TITLE. Genuine and operator-approved, wrong seat; fleet_telemetry held the right mapping and carries the warning, while pane_find is the title search and carries none. Belayed both; svos-dev owns the fix in server/bifrost.py. ⚠ Do not edit that file.

• [2026-09-22] headscale-ddns exited 1 silently and the alarm carried no cause — both failure paths were || exit 1 with no message. Now every exit names its reason and the WAN lookup retries 3×. The script was also untracked; a fix to what every mesh client resolves through lived on one disk. 30517fd.

• [2026-09-22] ⭐⭐ Five instrument errors in one day, and the shape is one thing: a tool that enumerates "things that are fine" has selected against its own subject. --state=running skipped the units most needing hooks; awk '{print $1}' dropped systemd's ●-decorated FAILED rows; grep -ic restic on the wrapper missed the check script; restic ls's header line made an absent path read as "blobs gone"; and a cooldown test that invoked before failing the unit. Every one reported cleanly while looking at the wrong thing. The rule is not "verify" — it is verify, then ask what the verification could not have seen. → persistent-memory.d/2026-09-22-instrument-errors.md

• [2026-09-22] Fleet alert bridge generalized — beszel-althing → althing-alert-bridge, route registry (/beszel + /kuma), each with its own prefix/footer/parser. /beszel FROZEN and proven byte-identical (its three original tests kept unchanged). An alert that lies about its own source is worse than no alert. Commit 6f0a9b9.

• [2026-09-22] Uptime Kuma rebuilt from scratch on 2.5.5, moved esh-docker-vm → ana-docker. ⚠ :latest is a TRAP — it tracks 1.x, so an Aug-2026 pull gave a Dec-2024 build. No REST CRUD API in either major; automation is Socket.IO via first-party scripts/kuma (the community wrapper is abandoned at 1.23.1). Monitors + channel + status page all declared in stacks/uptimekuma/monitors.yaml. Commits 3a85a6b, 94899d6.

• [2026-09-22] Beszel and Uptime Kuma are DISJOINT, not redundant — Beszel's alerts bind to a system with a threshold; there is no URL column, so it is structurally incapable of "this endpoint should return 200". Homepage is DISPLAY ONLY (38 polls, alerts nobody). The empty seam between them is where homepage sat dead for three days. Settled; do not re-litigate.

• [2026-09-22] Failed-START alarms on 23 nh3-dev units (services/althing-notify-failure/). ⚠ Coverage is far narrower than it looks — only the 7 timer oneshots and svos.service can actually reach failed; ten daemons flap forever instead. svos.service's divergent 5-min window is LOAD-BEARING (operator ruling 2026-09-11) — do not harmonise it. Commits 1be39d1, 163bb97.

• [2026-09-22] Backup coverage is a property of the SYSTEM, never one job's scope — establish it by querying the repo for the path in a real snapshot, never by reading a job's SRC=. A peer read dev-backup alone and told the operator, unhedged, that Miranda's conversation had "never been backed up"; all five paths were in that night's restic snapshot. docs/runbooks/backups.md.

• [2026-09-22] restic checks now assert CONTENT and are DISCOVERED not enumerated — conjunctive (recent AND present AND restores non-zero bytes); repos discovered from each NAS, hand list demoted to an EXPECTED set that distinguishes "vanished" from "stale". nh3-dev had never been on the watch list at all. Commits ba60fda, 4979869.

• [2026-09-22] irv-ml1: Irvine is a TENANCY behind a Fortinet PFI does not control — its TLS inspection breaks Tailscale relay/control intermittently (41 cert warnings/week). No fix on our side; do not go looking. ⚠ Diagnose reachability with tailscale ping, NOT status output — headscale said online and status said active, 19.7GB while nothing answered. Commit 38bb20c.

• [2026-09-21] ⭐⭐⭐ lv-mccarthy GATED and NOT SHIPPED — voice passes, memorisation is the cleanest in the line, and the length discipline is gone. 720 generations, 3 arms. Voice +0.152 at 2.9× floor for ckpt450 (ckpt900 +0.172 at only 1.2×, its spread one outlier seed), and ~3/4 of the gain survives stripping every punctuation mark, so it is not the cheap win. ⭐ Memorisation: ckpt450 at 0.12 against the author's own held-out 0.12 — identical, longest match 11 words against the author's coincidental 12, all 96 matches READ and every one stock grammar (he looked at the wolf and he looked at him); the name-shaped hits are the RENAMED inventions. Axis C is the blocker: 20% / 28% of generations overshoot the 90–140 band against base's 1%, worst case a degenerate loop at 279 words. ⚠ My own prereg's axis C transcribed score_beats.py's v1 criteria including "in-band up on base", which the operator RETIRED 2026-09-15 because base maxes it — under the v2 (ran-on only) ckpt450 passes by 0.01 against a 0.200 floor, but that reading was found AFTER the numbers and was not used. Fix the prereg prospectively. → persistent-memory.d/2026-09-21-lv-mccarthy-gate.md

• [2026-09-21] ⭐⭐⭐ The two-epoch recipe is now 0 for 3, and this time the loss curve was CONFIDENTLY wrong. On Brontë and Hemingway the epoch-1/epoch-2 checkpoints were TIED on eval loss, so preferring the earlier one cost nothing. Here the curve RESOLVED epoch 2 as better — ckpt900 +4.9× the 0.00393 median neighbour jitter above ckpt450, nowhere near tied — and epoch 2 lost every axis that resolves: 4.0× wider seed spread (0.148 vs 0.037), 1.8× the author's memorisation rate vs 1.0×, more ran-on (0.28 vs 0.20), worse on-beat. Its only win is a 0.019 voice point estimate, inside the floor, and its spread is ONE outlier seed (0.605 vs 0.457/0.531/0.554) — the third occurrence of that shape in the later checkpoint after lv-bronte's ckpt925 and lv-hemingway's ckpt1750. Durable: on this schedule the eval-loss minimum is not the ship candidate, and the curve's CONFIDENCE about it carries no information. Default this for Faulkner/Morrison/Chandler rather than re-deriving it.

• [2026-09-21] ⭐⭐ The memorisation control that shipped broken is now committed, and it turned a 12× red flag into a clean pass. memorization_check.py used base-unadapted as its negative control; base writes summary while the adapted arms write pastiche, so it cannot collide with a register it does not imitate and its zero is unearned. The lv-hemingway gate established this, computed the correct control (the author's own held-out text) by hand, and left no instrument — so the finding was not reproducible. Now --train-only --heldout-reference, refusing the unsafe combination, validated by reproducing Hemingway's hand-computed row to the digit. ⭐ On McCarthy it is the difference between reporting ckpt450 as memorising (0.12 vs base 0.00 = 12×) and clean (0.12 vs the author's 0.12 = 1.0×). ⭐ And the "the register makes collisions inevitable" story that was FALSE for Hemingway (0.01) is TRUE for McCarthy (0.12) — measured, not assumed; neither number transfers between authors. Also new: show_memorisation_matches.py, because rate and exposure are different questions and the reading was hand-done too. Commits b4ba731 7eadbd6.

• [2026-09-21] ⚠⚠ My confound detector was counting apostrophes as quote marks, and I saw it FIRE before I saw the bug. voice_distance.py's quote class shipped with ' and ’ in it — on the one corpus whose signature is dont/aint/wont. It reported the held-out reference at 121.1 "quote marks" per 10k for a corpus whose builder ASSERTS 0.0, and fired the pre-registered trigger on a base arm whose true density is 19.9. Fixing a detector to measure the quantity the frozen rule names is not moving the rule, but the fix un-fires the trigger, which is indistinguishable from shopping — so the trigger was made MOOT instead of adjudicated: the normalised read is load-bearing unconditionally, both columns reported, zero verdict effect. ⚠ The lesson: I controlled strip_punct (2500→0) and the byte-identity of the default path, and never asked the quote counter for a value whose answer I already knew. Commit 0d80e49.

• [2026-09-21] ⭐⭐⭐ The ops log shipped — and then failed FOUR ways in its first hours, every one recording something unfindable. Claim dropped by a sub-tool, hook behind graphify's eight exit 0s, no handle in the env, ssh-target written as a hostname. The general shape is configured ≠ effective; twelve instruments reported confidently and wrongly across three days, five of them mine. → persistent-memory.d/2026-09-21-ops-log-and-the-instruments-that-lied.md

• [2026-09-21] ⭐⭐ The Booth gained blur + a closed keep round trip, after shipping TWO controls that did nothing — a reveal handler Jinja discarded for sitting after {% endblock %}, and a × a sibling form covered by 30×22 px. Both operator-found, both from reading templates instead of rendering them. scripts/layout-probe.py took four iterations to become trustworthy. ⚠ Blur is COSMETIC and a test asserts the 200 on purpose. → persistent-memory.d/2026-09-21-booth-two-dead-controls.md

• [2026-09-21] ⭐⭐ claude-bot is an Owner in corviduo/pfi/vastblue; cicada + draupnir moved to pfi; vh-token use is now standing-authorized from the vault. ⚠ vh is a USER not an org, so no namespace-scoped admin exists — the only realization of the original ask was site-admin, surfaced rather than executed. ⚠⚠ ~/.config/claude-bot/gitea-token is DEAD and had been misreporting permissions; the working one is gitea-token-repo-create. → persistent-memory.d/2026-09-21-gitea-orgs-and-the-dead-token.md

• [2026-09-21] ⭐⭐ nh3-dev 84%→72%: 27 GB reclaimed, and a 252 MB LoRA adapter rescued from /tmp, which this box sweeps at 3 days. babyyarros existed nowhere else; hash-verified to smithy before any deletion. ⚠⚠ My 7-day session prune then deleted an ACTIVE session's dir — directory mtime does not reflect subdirectory writes. Do not re-run that predicate. → persistent-memory.d/2026-09-21-disk-triage-and-the-rescued-adapter.md

• [2026-09-21] Draupnir geometry engine provisioned on irv-ml1 and acceptance-tested (34c4179, e574b91, 2e08edc). build123d 0.12.0 + OCP, FreeCAD 1.0.0 AppImage headless, OrcaSlicer 2.4.2 containerised — Debian 12's glibc 2.36 cannot run any current Orca build (needs GLIBC_2.38, verified by ldd), and reaching back for an Ubuntu-22.04 build would pin permanently to stale. ⚠ OrcaSlicer writes result.json into CWD on EVERY invocation, --help included. Playbook playbooks/irv-ml1-draupnir-engine.yaml.

• [2026-09-21] Backup alarm verdict split: STALE (exit 1) vs ERRORED-JOBS (exit 3) (7fe4102). It had been printing STALE over 37 FRESH layers and zero stale ones — a false statement of fact, flagged by infra-hermes. STALE is a claim about backup AGE; a job that ran and errored is a different claim with different urgency. Wrapper mirrors the code and sends 🟡 not 🔴.

• [2026-09-21] vh/forgefirm mirrored from github.com/openglow-org/forgefirm, following the house convention read off the existing 17: vh/ namespace, upstream casing, 8h0m0s interval (16 of 18), visibility matching upstream. Verified by HEAD SHA (08b29fee) against upstream, not by the 201. ⚠ vh/NetAlertX interval 0s is deliberate — operator: "no longer interesting to us". Not a broken mirror; do not re-enable.

• [2026-09-20] ravenpen.com REGISTERED by the operator — the 09-18 hold is discharged and hamr-dev is answered. infra-ops deliberately did NOT execute this twice, because it was a non-refundable purchase reaching us as a peer relay with no registrar credential on our side; the standing instruction was to post registrar + expiry back on thread 01M2SERDB3DR3RV7JS1J0GMF0H only once the operator bought it himself. Done. Facts from RDAP (Verisign, authoritative) rather than a dashboard: registrar Cloudflare, Inc. (IANA 1910), registered 2026-09-20T20:12:30Z, expires 2028-09-20 (two-year), clientTransferProhibited. Zone 2df4c5eb4ea4b9410423bdebcb6c5192 active on vh@phasefinal, activated 0.4 s after creation — registered THROUGH Cloudflare Registrar, which is why the zone's original_registrar is null. ⚠ The zone is BARE — zero DNS records, so the name resolves to nothing and mail to it bounces; correct for bought-not-built, but say so before anyone points at it. ⚠ Scope boundary measured, not assumed: the fleet infra-ops Cloudflare token is Zone·DNS·Edit and 403s on the Registrar API — I can build records in the zone, and I can NOT read auto-renew state, renew, or transfer. Auto-renew is therefore UNCONFIRMED; do not let anyone assume it. ⚠ The token is vaulted, not on disk — secret get 'nh3-dev/.config/cloudflare/infra-ops-dns-token'; the memory's "vaulted at nh3-dev/..." names a VAULT KEY, and reading it as a filesystem path wastes a step.

• [2026-09-19] ⭐⭐ FV is a dedicated 20 A circuit carrying fv-ml1 AND the R420 running OPNsense, and nothing else (operator-confirmed). The ceiling is 1920 W, not 2400 — a GPU host running for hours is a continuous load, so NEC's 80% rule governs. Measured: BMC dcmi power reading 390 W instantaneous / 461 W max over 2423 s at idle GPUs, so the non-GPU baseline is ~313 W (EPYC 9254 24C/96T, 5+ drives, 4 PSUs). GPUs are capped 275 W each against a 300 W stock TGP (power.default_limit) — a 100 W saving across four cards, NOT the 200 W you get by measuring against the 325 W firmware ceiling; the operator corrected me on exactly that. Derived worst case ~1625 W capped (85%) vs ~1725 W at stock (90%). ⚠ Keep the caps — 90% leaves nothing for a heavier-than-estimated R420, PSU efficiency, or a warm day. ⚠⚠ The coupling is worse than the trip: OPNsense IS the FV edge and shares the breaker with the thing most likely to trip it, so an overload takes the router with it and removes the remote path needed to recover. Four PSUs do not help — they are all downstream of one breaker. ⚠ NOT measured: fv-ml1 under real 4-GPU load (the 461 W max is idle-ish), whether the BMC reports AC or DC (±10% ≈ 150 W at load), and the R420's actual draw. Recorded in servers/fv-ml1/README.md § Power. ⚠ Also fixed there: the README claimed 2x GPUs; nvidia-smi reports four.

• [2026-09-19] ⭐⭐ althing 3.7.0 rolled out to both infra-ops surfaces — and the rollout broke the claim tooling built that morning. Post office on nh3-docker (built from althing@6db955f, manifest sha256:df0709b3, 10.9 s recreate, volume preserved) and the nh3-extdev system wheel + herald. forseti independently verified both. ⚠ Everything CONTENT-verified, never tag-verified: postbox --version inside the image before the push and inside the running container after; data continuity proven by reading forseti's own message back out of the running 3.7.0 store. ⚠ deploy-stack.sh rsyncs with --delete, so the host-side .bak-<version> compose convention is GONE — it converges rather than accretes; rollback is git history + the retained 3.6.3 registry digest. ⚠ The claim bug: a 45-minute operation claim was refreshed and then released by deploy-stack.sh's exit trap, silently dropping the protection mid-rollout. Fixed 3e7d3a3 — ops-log claim exits 10 when the claim is already the caller's and leaves the holder file UNTOUCHED (a refresh would overwrite the reason and TTL the original claimant chose).

• [2026-09-19] ⭐⭐⭐ Three agents commit as one git author, and closing that gap took three instruments to get right. An unattributable commit (e43e262) appeared in the push set between two of mine — unidentifiable from git (all commits are Vuong Hoang by convention), from the ops log (host changes only) or from the bus. Turned out to be infra-hermes; forseti disclaimed it and the files' own ~/.hermes/hermes-agent imports gave it away. Three fixes, each exposing the next: (1) 0dc8e90 a post-commit hook recording the althing handle; (2) 4e778ae its fallback said lkraven@nh3-dev, which reads like an answer while being the absence of one — now unattributed(login), and handles never contain @ or parens; (3) 9141a41 the hook was appended behind graphify's eight exit 0 paths, so a graphify-out/-only or empty commit could NEVER be recorded. ⚠ Root cause was control flow, not the lock race proposed — see feedback_control_flow_before_concurrency. Attribution must run FIRST, never behind another hook's interestingness filter. ⚠ Also: ALTHING_HANDLE lived only in althing-infra-hermes-seat-run.sh, not in hermes-gateway.service / the pump unit — hermes fixed it with drop-ins. ⚠ systemctl show -p Environment reports the CONFIGURED env, not the running process's — it read correct while /proc/<pid>/environ did not have the variable at all. Check the process, never the config.

• [2026-09-19] ⭐⭐⭐ The backup alarm had no wire for three weeks, and fixing it exposed two more instruments that pass without looking. (1) backup-freshness-alert.sh called althing-cli, DELETED by the 2026-08-28 v3 cutover — the check detected stale backups every morning and told nobody. Swapped to postbox, added the missing ALTHING_POST_OFFICE to the unit and its installer (the binary swap alone would have failed differently), recipient infra-ops→infra-hermes (it was mailing itself), exit 2 now means the ALARM is broken vs exit 1 the backups, and --test-alert is a positive control because nothing had ever exercised the healthy path. (2) The checker is now coverage-aware — a guest is "not backed up by policy" only when NO enabled vzdump job covers it, a UNION across jobs: reading one exclude list would have silently stopped alarming on ana CT 109, which is excluded from the 03:00 job AND has its own 22:00 job. (3) It now reads vzdump TASK STATUS, because snapshot age is structurally blind to a job that runs and errors nightly — that cost 6 days on both CT 107 and VM 102, and it immediately found a third case on esh-nas-pve whose guests all read 0-1h FRESH. ⚠ ops-log earned its keep on day one: it captured all three failed playbook attempts automatically. Commits e979ccb, 5be25be, ba26852.

• [2026-09-19] ⭐⭐ elway evaluated when: / creates: / removes: / changed_when: WITHOUT the step's sudo, and it fails silently in the dangerous direction. A root-requiring when: (pvesh get … exits 255 unprivileged) returns non-zero → elway reports SKIPPED → a playbook that never performed its change reports overall OK, because "skipped" is indistinguishable from working idempotency. Three consecutive runs of the VM-102 exclusion reported success while changing nothing; only the unconditional verify phase caught it. creates: failed the other way — a /root/… path is unreadable to the login user, so test -e said absent and the step re-ran forever. Fixed (754db4b); sudo-less steps unchanged. ⚠ Audit any existing playbook with a root-requiring condition — a green run proves less than you thought. Two neighbouring traps: never grep -A<n> a PVE job in jobs.cfg (vmid sorts LAST, so adding a comment pushed it out of the window and every condition inverted); and pvesh JSON carries both a top-level "enabled":1 and a nested "fleecing":{"enabled":"0"}, so a greedy .*"enabled": sed matches the wrong one.

• [2026-09-19] ESH VM 102 (esh-vm-workstation) excluded from the nightly backup job — operator ruling. It is a Windows 11 Parsec/RDP sandbox (no password, no state to recover), and its vzdump had failed nightly since ~09-06 with timeout waiting on systemd, manufacturing a real job errors every night. infra-hermes argued correctly that a stopped guest still holds state and the job defect is the real fault; the operator overrode on what the guest actually IS — the one input neither agent had. ⚠ Policy, not a fix: the systemd-scope failure on stopped guests is UNDIAGNOSED, merely no longer reached on this node. A stopped guest that DOES matter would hit the same wall. Job uses an explicit vmid INCLUDE list, so exclusion = removal; reason recorded in the job's own comment. e50e57b, playbooks/esh-exclude-vm102-from-backup.yaml.

• [2026-09-19] ⭐⭐⭐ The ops log is BUILT — scripts/ops-log, automatic writers, and a detector for the path they cannot cover. Central on nh3-dev because both agents are the same unix user there (journald was the tempting alternative and would have split the log silently along the infra-ops/lkraven axis). Claim is mkdir-atomic, held across the whole diff→prompt→apply window, auto-breaks stale AND logs the break. deploy-stack.sh refuses a foreign claim (exit 3); ops-log being broken never blocks a deploy. 136-stack baseline laid so the detector starts from today. ⚠ Caught my own instrument saying "clean" for an unreachable host — now INCOMPLETE/exit 5. See in-flight § BUILT for the four settled design questions and the known hole (actors outside nh3-dev).

• [2026-09-19] ⭐⭐⭐ infra-hermes is this session's ASSISTANT, and the division of labour is now standing policy. infra-ops keeps improving infrastructure tooling plus the hard calls; infra-hermes does day-to-day checks, triage and routine operations; either may perform infra ops; infra-ops may task him downward and he escalates upward as needed. Three operator answers, verbatim in intent: (1) build the ops log — see in-flight; (2) he is NOT Miranda, so the global CLAUDE.md's sole-trusted-relay exception does not cover him and a directive he relays is information rather than authorization (reversible relayed work executes, irreversible or fleet-affecting goes to the operator); (3) yes, task him. Structural facts recorded in CLAUDE.md § "infra-hermes IS a real peer" rather than here, because a fresh session must have them without reading this file. ⚠ He is a Hermes bus seat on nh3-dev (althing-pump-infra-hermes.service, enabled, route declared) — round trip proven both directions 2026-09-19 04:46.

• [2026-09-18] ⭐⭐⭐ NH3↔Anaheim had been running over a throttled DERP relay, not a direct path — 78 GB of fleet traffic on someone else's free infrastructure. Four additive objects on ana-gw gave ana-scale a stable inbound UDP 41641 endpoint; tailscale ping 373–522 ms → 6 ms direct, cross-site HTTP 1.2 s → 0.015 s, STT via the ANA gateway 1.4 s → 0.25 s. ⚠ That box runs central-nat, so a policy dstaddr is the REAL internal address, not the VIP. No OOB access — back up with show to a local file and make additive changes ONLY. irv-ml1 still relayed. → persistent-memory.d/2026-09-18-nh3-ana-derp-relay.md

• [2026-09-18] ⭐⭐⭐ .internal DNS was failing ~10% of lookups fleet-wide, from two independent causes. A PUBLIC resolver was the fallback for a PRIVATE zone (Cloudflare answers NXDOMAIN authoritatively, so a transient miss became a hard failure) — now a cross-site ring, each site local-first with a different site as backup. Then the root cause: all three AdGuards shipped ratelimit: 20 shared across an entire /24, silently dropping queries at 5 s each. Set to 0. Hard failures 3/40 → 0/40; burst timeouts 40/60 → 0/60. ⚠ resolv.conf is DHCP-managed — change it at the UDM/FortiGate, not the file. → persistent-memory.d/2026-09-18-fleet-dns-ring-and-ratelimit.md

• [2026-09-18] ⭐⭐ SearXNG had ONE working general web engine and every health check said fine. 7 of 55 were enabled-by-default and six of those are dictionary/translation engines — inactive: false only makes an engine SELECTABLE, disabled: false puts it in the DEFAULT set. Now seven. ⭐ This stack tracks :latest ON PURPOSE (upstream ships engine-handler fixes continuously; a pin freezes breakage). ⭐ The Brave key is committed in plaintext by explicit operator decision — scoped to one low-value credential, NOT a change to the no-secrets rule. → persistent-memory.d/2026-09-18-searxng-one-engine-to-seven.md

• [2026-09-18] ⭐⭐ althing search returned ZERO for every hyphenated query — silently, for every handle, and on this fleet that is most of our hostnames. FTS5 read the hyphen as a column filter; search() caught the error, its probe passed, and it returned []. Routed to forseti (they own the code, I own rollout) → v3.6.3 deployed, 17 s bus outage. The reported symptom, a message that never arrived, was NOT a defect: the reporter's evidence file was truncated at 8 KiB. ⚠ No CI builds the post-office image. → persistent-memory.d/2026-09-18-althing-363-hyphen-search.md

• [2026-09-18] ⭐ FleetTools: one capability index, autoloaded by Claude, Codex and Grok from a single symlinked file. docs/fleettools/ + ~/FLEETTOOLS.md; absolute detail paths because a non-Claude agent cats them. Rule zero is query-live-inventories-never-a-written-list. The global CLAUDE.md tools section went 231 lines → 39, keeping only the three rules that govern behaviour rather than lookup. → persistent-memory.d/2026-09-18-fleettools-agent-index.md

• [2026-09-18] Miranda's Hermes plugin install is now a SYMLINK to the svos repo, not a copy (operator-approved). Verified empirically — is_dir()/exists() are stat-based and follow symlinks — and the copy had ALREADY drifted (README, since 09-15). Pre-restart gate is uv run python scripts/sync_plugin.py --check; hermes plugins list is confirmation-after, never permission-before. Copy kept at ~/backups/svos_miranda-copy-20260918-1444. ⚠ The install now tracks a working tree — an uncommitted edit in svos is what loads at the next restart.

• [2026-09-18] Worldtree's env.sh secrets are vaulted — 10 entries under worldtree/ (gitea, matrix as/hs, openai, uv-index, vastblueai, wt-admin demo+personal, z-ai, zai). ⚠ Z_AI_API_KEY and ZAI_API_KEY are DIFFERENT keys despite the near-identical names (fp 26bde3fa vs a4884152). ANTHROPIC_API_KEY was empty and skipped.

• [2026-09-17] ⭐⭐ The next voice seat was MEASURED, not chosen by taste — and the corpus size ranking INVERTS the voice ranking at the top. Our two largest authors are Stephen King (76 works, 12.1M words) and Agatha Christie (72, 5.5M); neither should get a seat, Christie being the Krakauer failure mode exactly (genius in plot architecture, prose deliberately transparent, invisible to a char-bigram Delta). Picks, in order: Faulkner (~15 pure novels, ~1.6M words — highest voice signal in the catalogue, AND he is McCarthy's stylistic ancestor, so training him next supplies the hard-negative sister the gate has lacked since the Brontë record named it missing); Morrison (11 novels after pruning criticism/anthology, ~818k — 11 val units, beating Hemingway's 10); Chandler (7 novels + a 409k short-story omnibus — fills the first-person hardboiled gap, Hemingway-class corpus size). ⚠ Faulkner's catalogue rows carry a 446k-word Snopes omnibus that duplicates novels also present individually — the Hemingway 90-96% collection-duplication trap, needs the containment pass first. → persistent-memory.d/2026-09-17-next-voice-seats.md

• [2026-09-17] ⭐⭐ Romantasy IS a real register, we already trained its most distinctive member, and the obvious next pick is its worst. Measured on the gate's own instrument (char-bigram Burrows's Delta, ~120k words/author from mid-work), with within-author floors and cross-genre positive controls. Cluster median pair 0.537 = 1.2x the worst floor against controls at 1.4-1.9x — tighter than cross-genre but NOT collapsed. Two findings survive either floor reading: Yarros is the cluster OUTLIER (4 of the 5 largest pair distances involve her), so a second romantasy seat buys measurably less than the first did; and Maas is the centroid (the two smallest distances in the matrix are hers), so the obvious commercial pick is the least distinctive. If the lane gets a seat it is Kenyon — furthest from Yarros at 0.674 and 27 works = 27 val units, the best-powered gate the line could build (Hemingway 10, McCarthy 6, Brontë 4). ⚠ Sensitivity floor stated: one sample per pair, no repeat draws; the rank ordering is indicative, fine gaps are not resolvable. → persistent-memory.d/2026-09-17-romantasy-register-measured.md

• [2026-09-17] ⚠⚠ dragonfireacoustics.com expires 2026-10-30 — six weeks — at eNom with NO transfer lock, and its sibling domain was already lost exactly this way. RDAP: registrar eNom, LLC (IANA 48, Tucows; abuse@enom.com), created 2008-10-30, status active only — no clientTransferProhibited. Registrant redacted (Verisign RDAP is thin, eNom's own endpoint 404s). Meanwhile dragonfirepro.com was re-registered 2025-09-27 by a Hungarian registrar and is now for sale on expireddomains.com — the customer lapsed it and a drop-catcher took it. ⭐ This also makes the cert fix mandatory rather than tidy: two of the three SANs name a domain a THIRD PARTY now owns, so that request can never validate, and our Virtualmin has been retrying it often enough to get the Let's Encrypt account PAUSED. Nobody is minding this domain — 13-month-dead cert, 403 homepage, sibling already gone — so the six-week clock is a real risk to an 18-year-old .com with a live site. Customer-facing; nothing touched. ⚠ If it is ever transferred, DNS does NOT come with the registration — the nameservers are eNom's name-services.com and the zone must be recreated first or mail dies. The whole zone is two facts plus a landmine: * (WILDCARD) → 199.250.192.76 which is dead (no HTTP at all, and it is what the apex/mail/webmail/admin/ftp all answer with), www → 38.120.12.45 (us), and 7 Google Workspace MX records that must not be lost. No DNSSEC (delegationSigned: false), so no transfer complication. ⚠ Also found: no SPF and no DMARC at the apex on a Google Workspace domain — a live deliverability problem independent of everything else. Transfer gate is the TAC/EPP code from the eNom account, not the lock; the missing lock is not authorization. 60-day rule is satisfied (last changed 2025-10-24).

• [2026-09-17] dragonfireacoustics.com IS configured on pfi-ana-webhost, and the whole thing is dead — a forgotten public-facing VM. It is a ServerAlias on the dragonfirepro.com Virtualmin vhost (DocumentRoot /home/dragonfirepro/public_html, suexec 1001), which is the only enabled site on the box. State: www. → 38.120.12.45 → DNAT to 10.250.50.52 (proven, not inferred — identical cert SHA-256 inside and out), Apache answers 403, and the Let's Encrypt cert (CN=dragonfirepro.com, SAN www.dragonfireacoustics.com) expired 2025-08-17, 13 months ago. The apex points to 199.250.192.76, not ours and not answering at all. ⚠ CORRECTED — the customer did not migrate dragonfirepro.com, they LOST it. RDAP: re-registered 2025-09-27 through Domain Science Kft (Hungary, IANA 3882) on dns-redirect.com nameservers, and it now redirects to expireddomains.com listed FOR SALE as an "established .com (6y)". It lapsed, dropped and was sniped. (The earlier "moved to a SaaS platform" read of its 6 foreign A records was wrong — that is parking infrastructure.) /home/dragonfirepro mtime 2025-04-12. ⚠ The VM is Debian 11, whose LTS window closed end of August 2026 — an unsupported OS exposed on public 80/443 for a site serving nothing. Retire / fix / tell-the-customer-to-repoint is an OPERATOR call: it is a customer relationship, not a technical one. Nothing touched.

• [2026-09-17] headscale now split-DNSes nh3.phasefinal.com to the three AdGuards, so mesh clients can resolve the internal-only wildcard (talk, booth — public DNS has no record for them; the fleet AdGuard answers 10.100.10.50). Operator-approved, scoped to nh3 rather than all of phasefinal.com. Config /etc/headscale/config.yaml in CT 106 on nh3-pve, backup config.yaml.bak-2026-09-17-splitdns, restarted, and the new route read back from a node's netmap rather than assumed. ⚠ Two things worth knowing: split DNS works fine here with global: [] — headscale issue #1161's "split ignored without global" does NOT apply to v0.29.3, verified on the live mesh — and override_local_dns: true would REQUIRE global, which is the config that makes a roaming laptop lose ALL DNS when the mesh is down. That is why split, not global. Routing was never the problem: nh3-scale already serves 10.100.0.0/16.

• [2026-09-17] ESH is back on the Cityside static 128.177.138.182/30 and the site is healthy — confirmed on four axes, not one. UDM WAN1 wan_type is static again (switched back from the DHCP set during the 09-17 outage), stat/health names Cityside Fiber with 0 disconnected and Verizon-5G idle at failover priority 2, esh-docker-vm's egress EQUALS the WAN ip so nothing is behind CGNAT, and colo→ESH reads 5.0 ms / 0% loss at 2005/2142 Mbps (Cityside CGNAT was 9 ms, Verizon failover 33–37 ms). ⭐ The FortiGate infra-ops trusthost3 pin un-broke itself and that was VERIFIED: from ESH, ana-gw tcp/22 is open and offers a password prompt, which a trusthost mismatch would never do. ⚠ The two 7-day crowdsec entries are being left to expire 2026-09-23 on purpose — Cityside failed twice in six hours, so they are cheap insurance. → persistent-memory.d/2026-09-17-esh-fiber-outages.md

• [2026-09-17] Operator ruled "leave it" on lv-hemingway's 3 separator-hidden names. So leak_gate.py exits 1 on a SHIPPED tree by design; a future session seeing that red result should read this line, not start fixing. lv-bronte re-ran clean.

• [2026-09-17] ⭐⭐⭐ The leak gate PASSED lv-mccarthy while five protagonist names sat in all six copies, and the blind spot generalises to every corpus in the line. \b(Surface)\b cannot match a name with a character inserted in it, so a mangled occurrence is unrenameable AND unreportable: B ell, C higurh, M oss, T oadvine (a small-caps drop cap kept as its own token) and Toad-vine, Glan-ton (a print line-break hyphen). Every VISIBLE occurrence had been renamed, which is what made it invisible. Same family as lv-bronte's _Antigua_, now generalised: any separator inside a name blinds a word-boundary scan. Fixed in the corpus builder (rules 4+5, counted), and leak_gate.py now runs a separator-tolerant pass with its own controls that FAILS the gate — validated against the pre-fix tree. ⚠ Its fragment filter is load-bearing: a naive scan returns 18 false positives on Hemingway (God damn, I run) against 3 real. Whole D1→D3 chain reproduced byte-identically before and after. Commit c559664. → persistent-memory.d/2026-09-17-mccarthy-split-name-leak.md

• [2026-09-17] ⭐⭐ The SHIPPED lv-bronte adapter emits mid-sentence line breaks at 12.46 per 1k chars, and nothing downstream looks for it. Its corpus is 100% hard-wrapped at ~68 chars (Gutenberg plain text) and the wrap transfers: base control 0.00, ckpt475 (shipped) 12.46, ckpt925 11.79, every Hemingway arm 0.00 on a 0%-wrapped corpus. Both controls fire. score_beats.py passed Brontë's damage axis anyway. McCarthy is the MIXED case — The Road wrapped, the other five works not — which is worse to learn than either pure one, so build_sft_pairs.py --reflow-hard-wraps (DEFECT 4) fixes it at pair time, off by default. ⚠ The obvious fix, joining every interior newline, CORRUPTS 46 two-speaker exchanges whose blank line was lost — and unmarked dialogue is the one thing this adapter exists to learn. The rule splits on sentence-final punctuation and takes the cheaper error deliberately.

• [2026-09-17] ⭐ The mccarthy register names the punctuation ON PURPOSE, and that is a gate-design call made before any McCarthy number existed. eval-*.sh drives the base control arm with the SAME system prompt via --system-from, and voice_distance.py is Burrows's Delta over character bigrams — so a tic left OUT of the register is a cheap win only the adapter can take, on a corpus measuring 0.0 quote marks per 10k against Hemingway's 838. Stating them hands them to the control too. Cost stated up front: the voice axis gets harder, and McCarthy's 276-passage val split (against Brontë's 44) is why that trade is affordable here and was not there.

• [2026-09-17] lv-mccarthy's D1→D3 chain was RECOVERED, not remembered — there was no runbook and the commands went over non-interactive ssh, so no history survived. Rebuilt candidates and matched sha256 against the artifacts on disk: 6 works, the entity map, the final map and all 36 copy files byte-identical. Now pinned in scripts/mccarthy-corpus/RUNBOOK.md with every deviation. ⚠ D1 must run on nh3-dev (the builder reads the kvasir catalogue by absolute path); the prior "on gx10" note is true of D2 onward only. ⚠ No phrase map exists for this corpus, so the gate's phrase audit never ran — Yarros and Brontë both had one.

• [2026-09-17] Measured and DELIBERATELY not changed, three of them. The oversize-passage drop is 13.9% of McCarthy's train words, between Hemingway's 10.0% and the shipped Brontë's 18.3% — in range, no change. BEAT_PROMPT asserts the passage is first-person and McCarthy is third; measured inert (0 narrator-retries against Hemingway's 615 of 7,094), so the prompt was left alone. Blood Meridian's 131 dash-separated chapter-argument paragraphs DID warrant a change and --drop-leading-heading now eats them (0 in every other work of all three corpora).

• [2026-09-17] ⭐⭐ A unit splitter must choose by SIZE, not count — and the val split scales with WORK COUNT, not corpus size. scripts/r49-corpus/split_units.py + a multi-index --holdout-chapter. The inherited most-units rule gave Cities of the Plain 4 units of 22,312w (the book's PARTS); the single-index holdout would have given McCarthy a Brontë-class 18k-word val reference on a 588k corpus. Both fixed, both caught by controls. → persistent-memory.d/2026-09-17-mccarthy-d1-d3.md

• [2026-09-17] ⭐ lv-mccarthy D1–D3 complete on gx10, leak gate PASSED (0 of 75 renameable, 0 of 37 sub-threshold, both controls green). Three McCarthy-specific calls, each forced by a measurement: corpus-scoped rename (the Border Trilogy shares 9 surfaces across books), a new mccarthy name preset (Hemingway's carries it_IT/fr_FR and McCarthy writes neither), and --min-cap 5 to match the entity map's floor — the first gate run failed with 45 survivors purely because rename's floor was 8 and the map's was 5. → persistent-memory.d/2026-09-17-mccarthy-d1-d3.md

• [2026-09-17] ⭐⭐ PARKED lv-krakauer, and the reason is a selection criterion the line was missing: ask whether the author HAS a voice before investigating whether the corpus is clean. Operator: "he's a great writer because of his research, not because he has a strong identifiable voice." I surveyed, built the corpus, measured containment and fixed three stripping defects before anyone asked the question that decided it. henge id 82. → auto-memory feedback_voice_worth_adapting_before_corpus.

• [2026-09-17] The althing route-declaring SessionStart hook is documented but NOT installed on nh3-dev — dev_launch.py has zero occurrences of "route", no hook declares one, and every live route was hand-declared at a different minute. A relaunched pane therefore drops to pull mode and stops receiving mail. Raised with the operator, not acted on, because installing it edits ~/.claude/settings.json and a peer's question is the wrong authorisation for a config change. Tracked at althing thread 01M2R0KPPE85SVJ96KSQ3YKKQP.

• [2026-09-17] Hemingway ships as-is: operator ruled "ship stands" on both measured corpus defects — the 0.96% beat contamination and the 130 non-name entity-map surfaces. audit_pairs_sourcenames.py --filter-out and audit_entity_map.py exist and are the instruments if that is ever revisited; neither was run against the shipped adapter.

• [2026-09-17] ⭐⭐⭐ PARKED lv-krakauer, and the reason is a selection criterion the line was missing: ask whether the author HAS a voice before investigating whether the corpus is clean. Operator: "he's a great writer because of his research, not because he has a strong identifiable voice." A voice adapter is worth its ~6 hours only if the target has a prose signature a reader could pick out blind; Krakauer's excellence is reporting, which an adapter cannot carry. I surveyed, built, measured containment and fixed three stripping defects before anyone asked the question that decided it. For each candidate, say what the voice IS in one sentence and how it shows up in char-bigram space, before the first catalogue query. henge id 82. → auto-memory feedback_voice_worth_adapting_before_corpus.

• [2026-09-17] ⭐⭐ A unit splitter must choose by SIZE, not by count — the inherited rule silently produced 22,000-word "chapters". scripts/r49-corpus/split_units.py: a marker mode qualifies only if its median unit is inside [600, 12000] AND no unit holds half the work; among qualifying modes PRIORITY breaks the tie (contents > chapter-word > roman > bare-numeral > caps-title), and paragraph-block sections are the fallback for works with no divisions. ⭐ Both rules exist because a control caught them: scoring by "median closest to target" chose caps-title (6 units, one holding 97% of the book) over True at First Light's real 20 chapters, because a median cannot see that distribution and a max bound can. Positive control: 8/10 Hemingway works reproduce the shipped mode and count exactly. Negative control: 40,000 words with no blank lines → 1 unit, refuses to fabricate divisions. Commit 705fa3a.

• [2026-09-17] ⭐ lv-mccarthy D1 built — 167 units, 584,756 words — and the whole job was protecting a style that reads as damage. 0.0 quote marks per 10k (Hemingway 838), dont/aint/wont. The builder runs NO typography normalisation and asserts the quote density afterwards. Two truncated catalogue rows dropped for complete mobi siblings; all 15 containment pairs measured (worst 0.10%); back matter in 4 of 6 works carried the author's name 26 times → 0. ⚠ The back-matter strip runs BEFORE the split here — Blood Meridian and The Crossing end with a dumped TOC of bare roman numerals, the exact shape of a chapter marker. Commit f3bf3ca.

• [2026-09-17] ⭐ lv-krakauer D1 built — 126 units, 422,880 words — and its name guard caught three defects nothing else would have reported. Back matter searched only the LAST unit while the apparatus sat in unit 37 of 41; relying on the splitter to drop front matter failed because the ebook TOC sits above the author's note and gave it a Chapter Thirty-Two to start on; and zero was the wrong bar — 2 survivors are Krakauer writing about his own father in Into the Wild's autobiographical chapters, so the allowance is pinned at 2 with every survivor printed. ⚠ Both strips are windowed in the OPPOSITE direction from McCarthy's, because Krakauer's ALSO BY/Copyright/About the Author sit at 0.0–0.6% of the file. Commit 4be0630.

• [2026-09-17] ⚠ triage_disposition = 'accepted' in the Kvasir catalogue does NOT mean the extraction succeeded. Blood Meridian's epub row holds 1,167 words of a 117,000-word book, The Crossing's 222 of 150,000 — real prose, real titles, accepted. Faulkner's The Mansion is 39 words. near_dup_pairs holds ONE row in the entire 1,284-work library and is blind to a fragment beside its full sibling. Word-count every master before trusting a row, and note that word count alone cannot tell a truncated novel from a legitimately short work.

• [2026-09-17] ⭐⭐⭐ lv-hemingway SHIPPED (ckpt850) with the line's strongest voice result — and the memorisation control it passed turned out to be the WRONG control. Voice +0.413 delta_cb at 6.4x the floor, closing 73.8% of the achievable span (lv-bronte closed 48%). ⚠ memorization_check.py uses the base-unadapted arm as its negative control, but base writes 18,035 words of summary against the adapted arms' 27,413 of pastiche — text that does not imitate the register cannot collide with its n-grams, so a 0.00 there means "different register", not "did not memorise". The right reference is the author himself: held-out Hemingway against the train split collides at 0.01 while the adapter does at 0.07, so the comfortable "his plain register makes collisions inevitable" story is FALSE and was refuted rather than assumed. All 19 matched runs were READ: stock dialogue, max 9 words, no proper noun — shorter than the 10-word run unseen Hemingway shares with the train split by coincidence. ⭐ A negative control that differs from the candidate in a way correlated with the metric is not a control. → persistent-memory.d/2026-09-17-lv-hemingway-gate.md

• [2026-09-17] ⭐⭐ The v2 voice floor is now PAIRWISE, and it retroactively passes lv-bronte. lv-bronte's ckpt475 shipped as a voice-axis FAILURE at +0.193 against a 0.251 floor contributed entirely by ckpt925 — a third arm nobody was shipping, on one outlier seed. Scored against the arm it was actually compared to the floor is 0.091 and it clears at 2.1x. The rule was changed prospectively, pre-registered for lv-hemingway before any Hemingway number existed, on an argument independent of the answer: the sampling variability of a difference A−B depends on A and B, never on a third arm C. The previous session found the defect and deliberately declined to exploit it; this follows from fixing it. lv-hemingway passes under both rules, so its verdict does not lean on the change. Caveats amended append-only in the compose, the NFS README and the gate record. Commits 0bb4938 2e9b118.

• [2026-09-17] ⭐ The beat-contamination leak IS present in Hemingway — 70 of 7,094 train beats (0.96%), 0 of 200 val. scripts/r49-corpus/audit_pairs_sourcenames.py closes the blind spot leak_gate.py has by construction (it reads the corpus and the renamed copies, never the generated beats). Cross-validated on real data: the fixed Brontë pairs return 0 of 3,858 and pairs-full.CONTAMINATED returns 15 of 792 = 1.89% with the recorded names. --filter-out yields a verified-clean 7,024-pair set in one command; the retrain is the operator's call. The val split being clean is why the gate could run at all.

• [2026-09-17] ⭐ audit_entity_map.py — the rename can DAMAGE the prose and no gate will ever say so. Mirror of audit_stoplist.py: surfaces wrongly held IN the map rather than out of it. 130 of 946 Hemingway surfaces flagged (1,616 instances, 0.162% of words) — African, Chinese, X-ray, Coca-Cola, Ritz, Prado renamed into invented names — plus 16 bare initials incl. C at 274 occurrences. Signal is a preceding article; controls derived from the corpus, not hand-picked. Every hit reported for READING: the Widow and the Informer are genuine epithet-names that should be renamed. Commit 051b99e.

• [2026-09-17] The two-epoch recipe is now 0 for 2 and should stop being carried forward. Hemingway's eval minimum is step 1750, but step 850 is +0.0040 against a 0.0044 median neighbour jitter — three checkpoints inside one jitter — and 850 won every resolving axis (2.3x tighter seed spread, lower memorisation, less ran-on). Same outcome as Brontë. What IS robust on this schedule is the epoch-3 collapse: +0.0762 = 17.4x jitter.

• [2026-09-17] gitea was reaching the PUBLIC route from every repo on nh3-dev — brokkr-smithy, sleipnir, Galdrabok, kvasir — and brokkr-smithy is pushed several times a week, so the fail2ban trigger was live, not dormant. Measured before acting (no split-horizon rewrite, no ssh alias, ssh -G confirmed port 22 to 38.120.12.44). Fixed by overriding the NAME once in ~/.ssh/config rather than rewriting N remotes, so fresh clones and unaudited repos are covered too. Verified with a real git ls-remote, not by inspection. Commit dcc1abc. Flagged by brokkr-smithy-dev; vh/imogen created for them the same session.

• [2026-09-17] servers/fv-ml1/ssh-target was bare 10.251.50.54, so deploy-stack.sh connected as lkraven and could not write the infra-ops-owned /opt/docker/compose/ — and lkraven's sudo on fv-ml1 needs a password, so DEPLOY_SUDO=1 failed too. Now infra-ops@10.251.50.54; --validate-only still clean, deploy works. ⚠ Other hosts' ssh-target files may carry the same gap — a read-only refresh works as either user, so the fault only surfaces on a deploy.

• [2026-09-17] ⭐⭐ A leak class the corpus gate structurally CANNOT see: the beat-writing model recognises the book and restores the author's real character names. 1.8% of Brontë beats named Rochester/Jane/Brocklehurst while 0 responses did. Worst for public-domain classics; Yarros and Hemingway's clean runs are NOT evidence they are immune. Patched as a sourcename reject + --source-entities. → persistent-memory.d/2026-09-17-beat-contamination-leak.md

• [2026-09-17] ⭐ A stoplist entry is an assertion the leak gate can no longer check — stoplisting removes a surface from the entity map, so a wrongly stoplisted CHARACTER is an undetectable leak. Three were wrong on Brontë (Leaven, Pierrot, Samuel); scripts/r49-corpus/audit_stoplist.py finds them by honorific and now gates the pipeline. Commit 8bb7686.

• [2026-09-17] ESH: Cityside Fiber failed TWICE (19:09 and ~01:06); operator switched WAN1 to DHCP to restore service and has a ticket for the static. crowdsec esh allowlist carries both failover egresses with 7-day expiries — the rotation-fragility is live. → persistent-memory.d/2026-09-17-esh-fiber-outages.md

• [2026-09-17] ⭐⭐ lv-bronte SHIPPED on voices-seat (ckpt475) DESPITE failing the v2 VOICE axis — additive, reversible, safety-axis clean. Both candidates closed 48–52% of the achievable distance to Brontë but +0.193/+0.210 sit under a 0.251 noise floor set by ONE outlier seed in the arm not being shipped; cause is structural (81 val pairs vs Hemingway's 200) and not cheaply fixable. ckpt475 is the pick if it ships. The two-epoch recipe did NOT transfer. → persistent-memory.d/2026-09-17-lv-bronte-gate.md

• [2026-09-16] ⭐⭐ Instruction-pair SFT BEATS raw-text for author voice, and the raw-text incumbent never cleared its own control (+0.141 against a 0.153 floor). lv-yarros shipped; the v1 decision rule was amended by the operator after measurement showed it gated on axes the unadapted carrier already maxes. → persistent-memory.d/2026-09-16-lv-voices-line.md

• [2026-09-16] ⭐ voices-seat live: one carrier, N lv-<author> LoRA adapters, hot-swap measured at 0.24 s. LoRA costs 24.3% of decode against a 0.1% A-vs-A floor and is worth paying; --gpu-memory-utilization is a request against TOTAL VRAM and only a pinned KV makes it predictive. → persistent-memory.d/2026-09-16-voices-seat-lora.md

• [2026-09-16] ⭐ lv-hemingway corpus gated at 994,760 words — and half the catalogue had to be EXCLUDED. 169,759 words of measured 90–96% collection duplication, a Sherwood Anderson parody, and the author's own name 95 times in publisher back matter; the gender resolver needed a corpus base-rate correction to stop reading women as men. → persistent-memory.d/2026-09-16-lv-hemingway-corpus.md

• [2026-09-16] Grok token broker built then SHELVED — operator ruled "keep the jail stop the a/b", so the renewal feature has no consumer. ⛔ Do NOT arm probe-rotation: the risk did not shrink (it reaches BOTH Gróa transports through one shared session) and the payoff went to zero. → persistent-memory.d/2026-09-16-grok-broker-shelved.md

• [2026-09-15] ⚠⚠ DO NOT carry "a client-side timeout is not a cancellation" as a rule — it is FALSE as stated. A clean abandon cancels itself ~6 s later (measured); yet six requests genuinely orphaned on vllm-erp-seat. Some propagate, some do not, boundary unknown — which argues for a detector, not a rule. ⭐⭐ The durable artifact: a serving engine's KV cache CYCLES, an orphaned one only CLIMBS — request count and throughput are ambiguous between loaded and wedged, and I called the seat healthy twice off them (correctly, on the evidence). ⚠ A max_tokens ceiling would NOT have prevented it: the worst offender had 16384 set, hit it, and returned 24,594 chars of whitespace. → persistent-memory.d/2026-09-15-client-abandon-cancellation-boundary.md

• [2026-09-15] ⚠⚠ --gpu-memory-utilization DOES NOT PREDICT RESIDENT VRAM — measure it, never compute it. Wrong in both directions on fv-ml1: vllm-cyberprev util 0.40 (expect ~39,155 MiB) holds 47,124 (+8 GB over); vllm-gen-small util 0.48 (expect ~46,986) holds 36,942 (−10 GB under). Planning a placement off the fractions would have been 8 GB wrong. Read nvidia-smi --query-compute-apps. Full per-seat residency table + the breeze shuffle arithmetic → persistent-memory.d/2026-09-15-breeze-placement-sizing.md

• [2026-09-15] breeze-tts stays on irv-ml1; the TTS-stack move to fv-ml1 is PARKED (park id 75, move-the-tts-stack-breeze-tts-bragi-tts-gateway), triggered on evacuating embed/rerank/reward. ⚠ Trigger as stated says "gpu0" but those three are on GPU 1 (~0.16 util, ~15.7 GB; GPU 1 is the tight card at 0.975 / 4,336 MiB free) — confirm which he meant before executing. All three services move together because only breeze-tts is GPU-resident (~10.3 GiB, growing) while bragi and tts-gateway are CPU proxies, and co-location is what avoids a cross-site hop per TTS call. breeze-tts sizing — original recommendation NOT to move it. ~10.3 GiB measured under load at 53 min uptime, up from 9.2 GiB shortly after warm-up (it grows; n=2, plateau unmeasured) — so GPU 0's 11,982 MiB free is a 1.7 GB margin and shrinking, on the live chat serving path. ⚠ Two measurement traps: it reports nothing at idle on the wrong card (BREEZE_GPU_DEVICES=0 = the 3090, not the A6000), and an early reading understates it. ⭐ The real objection is topology: tts-gateway is on irv-ml1 and reaches it same-box, so moving breeze alone adds a cross-site hop to every TTS call against a 478 ms first-sample budget. GPU 3 would fit it but spends the reserve. → persistent-memory.d/2026-09-15-breeze-placement-sizing.md

• [2026-09-15] Parakeet STT live on fv-ml1 GPU 0, behind LiteLLM ext-stt / whisper-1. ⚠ Placed on GPU 3 first, which was wrong — operator caught it. A ~800 MiB seat should ride the card with the most uncommitted headroom (GPU 0, util 0.88, ~13 GB spare), not put the first fingerprint on the one pristine 96 GB card: vLLM sizes KV cache against TOTAL VRAM, so any tenant on an empty card eats a future full-size seat's profiling margin (flash-next needs 93 of 96 GiB). GPU 3 is now a deliberate reserve at 2 MiB. Retargeted the existing stacks/parakeet/ (sherpa-onnx + our own FastAPI wrapper) from irv-ml1; v3 int8, 25 languages. ⚠ ORT's CUDA EP compiles kernels lazily and the first decode on sm_120 took 45.7 s — every later call ~0.5 s; a startup warmup in app.py now absorbs it, so the first real request is 0.65 s instead of a 45 s hang that no client would wait through. GPU use was verified by a process on GPU 3 (922 MiB), not by the provider=cuda log line, because ORT falls back to CPU silently and still returns correct text. Silence → "" (null control), known sentence → near-exact (positive control). → persistent-memory.d/2026-09-15-parakeet-stt-fv-ml1.md

• [2026-09-15] ⭐⭐⭐ THE FLEET'S CHARACTERISTIC FAILURE, named: a confident answer from a broken instrument. Nine instances in one night, every one of which PASSED A CHECK — provider=cuda while ORT ran on CPU; node --check green on a file whose SERVED script was dead; secret get returning "" with exit 0; find() turning a failed listing into an authoritative "not found"; a 401 rendering as "0 toolsets"; compat ✓ on a typo'd path; doctor exit 0 on ERROR; ss | grep python missing a listener named hermes; SIGTERM freeing a port 35 s before the process died. ⚠ The tell: whenever "broken" and "legitimately empty/absent/off" produce the same output. Remedies: measure the output not the input, positive AND true-negative controls, refuse to emit the ambiguous value, and never declare victory on a plausible fix. → persistent-memory.d/2026-09-15-silent-wrong-answer-pattern.md

• [2026-09-15] secret get returned EMPTY with exit 0 under concurrency (svos-dev found it; 0/4 succeeded here). Root cause is bw unlock racing at session establishment, not item reads — so a lock inside the read wrapper cannot work. Fixed: command-level lock, cmd_get refuses an empty value, and find() no longer coerces empty stdout to []. ⚠ ~/.local/bin/secret was a plain COPY — now a symlink. 0193b31.

• [2026-09-15] ⭐⭐ A check that reads an artifact AS STORED cannot see a transformation between storage and execution — named twice in one night and it generalises. node --check on a source file passes while the SERVED page's inline script is dead (a JS 'didn\'t' inside a Python string arrives as 'didn't' and closes it); provider=cuda in a log echoes configured intent while ORT silently ran on CPU. Both check the INPUT to a transformation and get reported as checks of its OUTPUT. Remedy: gate the wire, not the file — tts-stack tools/gate_served_page.py. ⚠ My first version had a gap tts-dev closed: a worklet inside a template literal is just a string to a parse of the enclosing script, so its syntax error surfaces as a rejected addModule promise and silent degradation. I checked the instance, not the class. → persistent-memory.d/2026-09-15-talk-v10-deploy.md

• [2026-09-15] ⚠⚠ The talk-deploy "permission problem" NEVER EXISTED — and I built a fix for it anyway. /opt/docker/compose on nh3-dev is root:docker 2775, sessions run as lkraven, lkraven is in docker; a mkdir settles it in one second and nobody ran one for nine days. There is no tts-dev OS account at all. It held because a stale memory row supplied a mechanism, the operator's routing instruction ("give it to infra") was misread as corroboration of a capability limit — different claims, only one ever stated — and I repeated it to the operator as fact. Then, told to fix "the harness issue", I inferred an auto-mode classifier refusal and committed a settings.json to tts-dev's repo on that inference; their mkdir disproved it and I reverted. ⭐ "I can't do X" is a hypothesis until someone pastes the error. ⚠ That commit also overclaimed a doc fix that failed — never chain an edit and its commit in one invocation. → persistent-memory.d/2026-09-15-silent-wrong-answer-pattern.md

• [2026-09-15] talk v10 LIVE on nh3-dev :8092 — the fleet speaks and listens on one page. First consumer of the ext-stt Parakeet seat: POST /api/listen, push-to-talk, barge-in. Gated build→throwaway→teardown→cutover, then re-gated against production (a gate that only ran against the throwaway proves the image, not the deployment). ⚠ Deploys route through infra-ops only because tts-dev's identity is not in nh3-dev's docker group — a permissions accident, not a judgement call; group-vs-relay is in front of the operator.

• [2026-09-15] ⭐⭐ Two restart patterns from svos-dev worth stealing: (a) DRY-RUN BOOT against the still-held port — start the new process while the old one holds the socket; it proves every check above the bind and dies on [Errno 98], so a one-way restart becomes a rehearsed one at zero cost. (b) ⚠ SIGTERM freed the port but left the process alive for 35 s — a script waiting on the port would have run two copies. Kill by PID, wait on the PID, never on the port. A freed port is not evidence of a dead process.

• [2026-09-15] ⭐ svos_miranda ENABLED and LIVE in Hermes — but agent.disabled_toolsets is permanently OFF by operator ruling ("i dont want the tools disabled everywhere"). That key is a global end-of-pipeline subtraction, not api_server-scoped: measured 46 tools → 20 on a default session. It is also unnecessary — platform_toolsets.api_server: [svos_miranda] alone resolves an api_server session to exactly the 8 tools, write-klass absent. Gateway restarted 02:10 (PID 3107822→3901622, observed); /v1/toolsets now 29 rows incl. svos_miranda; operator's own surface verified intact at 46. ⚠ SVOS must stop verifying against the GLOBAL roster before it restarts — it will see 29 and refuse, by design now. → persistent-memory.d/2026-09-15-svos-miranda-plugin-validation.md

• [2026-09-15] irv-ml1 parakeet RETIRED; voice-studio STOPPED. Both operator rulings. Parakeet lost tts-dev's bench to the FV seat at both clip lengths and to whisper-large-v3 at 6.24 s; no gateway alias depended on it and every other host reference was a port-register comment. voice-studio existed for the dots mint loop, which Breeze obsoleted 2026-09-06 — retired rather than repaired.

• [2026-09-15] svos_miranda Hermes plugin validated; found its load blocker. Absolute intra-package imports (from hermes_plugin.x) could not resolve at the documented install name — fixed by svos-dev at c964e64. ⚠ hermes plugins validate and doctor can NEVER pass this plugin, by construction: validate's probe stub is config-blind AND returns None from register_tool (which the plugin's guard reads as a collision), and doctor runs under a temp HERMES_HOME with no config. ⚠ doctor exits 0 on ERROR (use --ci); compat reads a nonexistent path as a pass. Roster verified 8/7 by a probe supplying real settings. → persistent-memory.d/2026-09-15-svos-miranda-plugin-validation.md

• [2026-09-15] ⚠ ana-docker resolves NO .internal names — its /etc/resolv.conf is 1.1.1.1/1.0.0.1, not the fleet AdGuard. LiteLLM only reaches irv-ml1.nh3.internal because of a hand-pinned extra_hosts in its compose. New gateway aliases therefore use raw IPs; adding a hosts entry would mean recreating the container and bouncing the gateway for every consumer. Fleet-wide DNS fix is unowned.

• [2026-09-15] ⚠⚠ irv-ml1 still points at the retired wg0 lifeline 10.100.79.3 in 96 places — and one is a LIVE breakage, not a dead link. voice-studio cannot reach studio-gate (both up, separate docker networks, gate URL is the dead IP) and has been failing since the 2026-09-06 cutover with nothing alerting. 8 running containers carry dead homepage.href labels; waterland-studio's siteMonitor too. ✅ tts-gateway/ext-tts verified UNAFFECTED. Not fixed — wants a scheduled pass, not a 02:00 improvisation. ⭐ Third instance of the same shape: a retired address needs a grep by ADDRESS, not by hostname, and labels live in no file until the container is recreated. → persistent-memory.d/2026-09-15-irv-ml1-dead-wg0-address.md

• [2026-09-15] Parakeet bench settled by tts-dev — FV wins at both clip lengths and beats the incumbent Whisper; IRV seat is now retirable. FV 155 ms / 391 ms on 1.84 s / 6.24 s clips vs IRV 354 / 1010 vs whisper-large-v3 457 / 690 — IRV is slower than Whisper at 6.24 s. Length sweep (n=9/cell, first 3 discarded) fits ~58 ms fixed + 56 ms per audio-second, asymptote ~17.8x realtime, which independently reproduces our 17x on a different harness. Gateway hop measured below harness resolution (±30 ms), so ext-stt is the right consumer path. ⚠ tts-dev retracted their own plan's 60-120 ms projection: published RTFx is BATCHED THROUGHPUT, not single-stream latency — the two differ by ~200x. ⚠ Their between-run variance is ±20% because GPU 0 is the live chat path; our 0.50 s median was taken on an idle GPU 3 and is a best case.

• [2026-09-15] Mesh membership retired for fv-ml1 and nh3-dev — six nodes left, each with a job. fv-ml1 gets break-glass rejoin instead of standing membership; nh3-dev's retirement also removed the nh3-scale masquerade exception it had required. Exactly one live reusable pre-auth key remains fleet-wide. → persistent-memory.d/2026-09-15-fv-mesh-watchdog.md

• [2026-09-15] FV cross-site routing fixed — one OPNsense outbound-NAT rule had been scoped to Anaheim only. fv-ml1 now reaches NH3/ESH/IRV/ANA/mesh/internet; four rules, all src=10.251.50.0/24. The diagnostic signature is the valuable part: every layer looks correct and the discriminator is that every other site pair works. → persistent-memory.d/2026-09-15-fv-cross-site-snat.md

• [2026-09-15] Break-glass mesh path on fv-ml1 — inverted from a restore-watchdog on the operator's suggestion: the box is OFF the mesh and the watchdog JOINS it on fleet loss. Exposed a rejoin key expiring in 4 days; replaced with a dedicated 1-year key and the two stale reusable keys retired. → persistent-memory.d/2026-09-15-fv-mesh-watchdog.md

• [2026-09-15] Fleet identity/group/path conventions pinned + docker trees → root:docker 2775 setgid on 5 hosts. svc-* in 800-849, infra-ops 850, docker 851, vh for new hosts with no retro-renames; 0777 cleared; linus deleted; llmuser de-privileged. → persistent-memory.d/2026-09-15-fleet-identity-conventions.md

• [2026-09-15] nh3-dev unreachable from the mesh at its LAN address — Tailscale's ts-input anti-spoof, not DNS. Fixed with a masquerade exception on nh3-scale. ⚠ Do NOT instead advertise the /32 from nh3-dev; that black-holes it from every other site while its own LAN keeps working. → persistent-memory.d/2026-09-15-nh3-dev-ts-input-masquerade.md

• [2026-09-15] ESPHome pinned to 2026.8.2 + kb KB-search tool shipped. Untagged image had drifted a year; config relocated into restic with 539 MB of regenerable cache excluded; remote-build disabled (⚠ two switches, only one closes the port). kb exists because Worldtree's /search searches messages, not notes, and returns a clean empty result for a note that exists. → persistent-memory.d/2026-09-15-esphome-and-kb.md

• [2026-09-15] Hermes bearer rotation hold released — svos-dev split their HS256 signing key off the shared value (svos 7165272). ⚠ When the svos_miranda plugin arrives it will reference the dispatch key, not the bearer (expected), and its tools array is legitimately seven or eight entries; any other number is a real fault. Commit e641931.

• [2026-09-14] fv-ml1 rebalance: cyberprev→sec (mog-sec retired), NEW gen-small A3B seat, all sec/gen/char at native 262K in-band, coder reclaimed, seat catalog + bench shipped. cyberprev = hotdogs cyber-SFT (name-repaired past a tripled-prefix unsloth export bug, house NVFP4 quant); gen-small = llmfan46 Qwen3.6-35B-A3B Heretic (already on disk), MTP 69.6%. Serial depth-tested all seats clean (0 OOM); warm tok/s 62.7-337.3. Commits 1418edb→dfa91a8. → persistent-memory.d/2026-09-14-fv-seat-rebalance-gen-small.md

• [2026-09-14] fv-ml1 all-night seat reorg — MTP k=3 on gen-large (+52%@conc1), gen consolidated onto flash-next (27B dense retired, 38 GB freed), char-rp restored to MeroMero-v2-31B, Sentinel-R3 served + dflash cutover (beat MTP 2.40 vs 2.18). ✅ gen-large RESOLVED 2026-09-14 — orcarouter serving: PLE bf16→FP8 convert + ple_embedding_dtype + layer_types rename; NO source build needed. → persistent-memory.d/2026-09-14-fv-seat-reorg-and-orca-blocker.md

• [2026-09-13] STANDING POLICY (operator): cap GPU power limits at BUILD time, not after discovering the constraint. → persistent-memory.d/2026-09-13-standing-policy-operator-cap-gpu-power.md

• [2026-09-13] **FV SITE DARK — every Fountain Valley address including the BMC went unreachable ~2.5 min into a two-card ** → persistent-memory.d/2026-09-13-fv-site-dark-every-fountain-valley.md

• [2026-09-13] ⭐⭐ Qwen3.8-Flash-Next serving on ONE card with its 51B n-gram table in host RAM — the first seat whose weights do not fit its GPU. stacks/flash-next-seat/, fv-ml1 GPU 2 :8022, plus a gen-large LiteLLM alias. Measured: 74.36 GiB weights resident, 14.00 GiB KV = 560,654 tokens at the full 262,144 context, 67 GiB host RSS, 75.5/212.3/387.8 tok/s at conc 1/4/8 (⚠ n=1). ⭐ The offload is vLLM #54371 (UVA, merged 2026-09-09) which supersedes the paused #53899 — it has no worker process, so #53899's whole bug family (TP=1 deadlock #53960, pidfd_getfd/ptrace gate, stale-output-under-graphs) is designed out; in v0.29.1rc0, not v0.29.0. ⚠ text_config.ple_embedding_dtype is the load-or-fail discriminator for any community build. ⚠⚠ --kv-cache-memory makes vLLM SKIP MEMORY PROFILING and ignore --gpu-memory-utilization — 16 GiB nearly OOM'd on a 155K prefill with no visible failure; 14 GiB is the measured-safe value and vLLM's own "17.46 GiB to fully utilize" is 3.5 GiB too high. ⚠ MTP is off pending measurement here, not written off — the recipe's number is cross-harness and tested k=3 only, while the head is ONE layer run autoregressively, so k=1 is unpublished and may win (services/flash-next-mtp-bench/, one off_A rep banked before the outage). ⚠ A container once ran (healthy) with PORTS=[] — verify docker port, not the healthcheck. → persistent-memory.d/2026-09-13-flash-next-seat-and-fv-outage.md

• [2026-09-13] Finished the ana-ml2→fv-ml1 renumber the cutover missed: 16 live Homepage entries pointed at the dead 10.250.50.54 and zero at the live IP. The sweep allowlist was built from files that mention the HOST and a homepage.href mentions only an IP, so every label-only stack fell outside it by construction; 24 files fixed plus host copies, allowlist extended with how to derive it next time. Two bugs fell out: deploy-stack.sh rejected any stack name containing a dot (so qwen3.5-122b/qwopus3.5-122b/mistral-medium-3.5 could not be deployed at all), and scriberr's CORS allowlist held only the dead IP and the dead scriberr.ana.internal. ⚠ Incomplete: the 10 running containers were never recreated and a plain power-on will NOT apply labels — the staged compose up -d recovery does. Eight stacks deliberately not pushed (real host drift); three of those are untracked host-only stacks. Commits 3132a16, 969a1b6, d79f104.

• [2026-09-13] FV→ANA fixed, Beszel18/18 up: scoped OPNsense hybrid NAT for fv-ml1→ANA; prior NAT/filter rules preserved and rollback-guarded. → persistent-memory.d/2026-09-13-fv-to-ana-nat.md

• [2026-09-12] ⭐⭐⭐ FV CUTOVER EXECUTED — fv-ml1 live at Fountain Valley, renamed, renumbered to 10.251/16, serving inference; BMC online after finding it was tagging 802.1q VLAN 250 into an untagged port; and the box has FOUR RTX PRO 6000 (391 GB VRAM), not the two every doc claimed. Also: OPNsense write APIs need an X-CSRFToken header scraped from a <script> block — session cookies alone 403, which is why reboots appeared to work while the operator was power-cycling by hand. → persistent-memory.d/2026-09-12-fv-cutover-executed.md

• [2026-09-12] esh-vm-db Restic fixed: stale April PG dump caused by TCP auth + masked errors; peer-auth/fail-closed hook and bounded retries installed. Snapshot bc5eeaff and repository check verified. → persistent-memory.d/2026-09-12-esh-vm-db-restic-repair.md

• [2026-09-11] Worldtree memory-split (U6) — PROTOCOL AGREED with worldtree-dev: nobody flips memory.reader.enabled or m → persistent-memory.d/2026-09-11-worldtree-memory-split-u6-protocol-agreed.md

• [2026-09-03] Run 3c is STAGED on pfi-gx10 and deliberately NOT launched — the launch is a 13.3 h commitment and the operator stood this port down once already. Base shards AND the encoded corpus sha256-verified identical to ana-ml2's, so the transformers 5.15.1→5.16.1 / x86-64→aarch64 delta is measured inert. ⚠ The encode-cache FILENAME differs by design (base_model_path is in the key) — input hash, not output. ⚠ Tripped the pkill -f ssh self-match again; the launcher guards on a pidfile because of it. → persistent-memory.d/2026-09-03-gx10-run3c-staged.md

• [2026-08-25] Fused MoE kernel path — DEFERRED, tracked at park fused-moe-kernel-path-for-gemma-4-moe-training (id 47). Operator: "note the fused MoE kernel for round two… if we nail it soon, the math has us wanting to restart the run anyway." Training MFU is 8.6% (27.1 of a benchmarked 313.8 TFLOPS) because transformers runs the Gemma-4 experts in a Python loop — 128 experts × 30 layers, ~11,500 iterations per step under gradient checkpointing. ⚠ The same fused 3-D expert layout that made bitsandbytes skip 88.5% of the model is exactly what a grouped GEMM wants — the format is good for storage and for fused kernels, and hostile only to naive iteration. Two fixes: group_by_length (−29.9% compute, free, but breaks the seeded order manifest and re-opens a batch-composition call brokkr already made) and a grouped-GEMM/compiled MoE forward (the remaining ~10×). Not applied to the live run — restarting mid-flight to change batch ordering was judged a bad trade at step ~50 of 1,312.

• [2026-08-24] nconnect=8 on /mnt/smithy — approved but DEFERRED at operator instruction. brokkr-smithy-dev pre-approved it for "once the FortiGate work settles" and does not need re-asking; the operator declined it in this session's scope. Tracked at althing thread 01M0R46SFYF83099N16WD67KGD.

• [2026-08-19] AI-tab Dormant regrouping BELAYED by the operator — six seats (char-rp Magidonia, char-rp-reasoning Heretic2, Granite summarizer, Qwen-Image-Bench, Skaldsong, Chatterbox Fast) show amber EXITED inside live groups rather than AI - Dormant. Fix is a label change + recreate per stack; needs the operator's read on which are retired vs temporarily down. untracked by operator choice (his words: "belay the ai dormant regrouping for now").

104 older entries archived to archival-memory.md.

Tried and abandoned

• [2026-09-25] The esh-pve NVIDIA DKMS recipe on nh3-pve — it built, then the kernel refused to load it: nh3-pve has SECURE BOOT ON (esh-pve, the same MS-01, has it off). The installer rolled back. The playbook now pre-flights mokutil --sb-state (7ddd116).

• [2026-09-25] elway against an unpinned host-key name — every when: hit ssh rc 255, so all 9 steps showed SKIPPED, not failed. Only the verify phase went red. Pin the name (the IP key matched) or use the IP. Untracked elway defect: rc 255 in when: should fail.

• [2026-09-24] Testing "Restore AC Power Loss" with an OS shutdown — a shutdown stays off BY DESIGN; only pulling and restoring AC tests it. A community README claimed the two are indistinguishable; trusting it cost Prime two trips to the pfi-gx10 power button.

• [2026-09-23] booth link --help — there is no help flag; it posts --help to the operator's link board as a link. Read booth with no args for usage.

• [2026-09-21] Using directory mtime as a liveness test when pruning session scratchpads — find /tmp/claude-1000 -mindepth 2 -maxdepth 2 -type d -mtime +7 deleted an ACTIVE session's working directory. A directory's mtime does not change when files are written into its SUBDIRECTORIES, so a session writing continuously to <id>/tasks/ looks 7+ days idle at <id>/. Four safety assertions passed; none of them asked whether the liveness test was sound. Use the deepest recent file, or cross-reference running claude PIDs.

• [2026-09-18] Routing SearXNG's egress through a SOCKS5 proxy on esh-scale — one day live, reverted. It fixed nothing, and the reason I gave for reverting it was itself wrong: the rollback was the counterfactual and it falsified my own published claim. Kept reverted on its own merits (no measurable gain, added a hard ESH dependency for all fleet search). → persistent-memory.d/2026-09-18-searxng-esh-egress-reverted.md

• [2026-09-18] api_key: !ENV SEARXNG_BRAVE_API_KEY in searxng settings — this build has NO !ENV YAML constructor, so the file was unparseable and the container crash-looped ten times with fleet search down. There is no env-var path into that file at all: the loader reads only SEARXNG_SETTINGS_PATH and the entrypoint substitutes only ultrasecretkey. Literal or nothing.

• [2026-09-15] ⚠⚠ Probing OPNsense API endpoints by POSTing at them — one was /api/core/system/reboot and it took the FV site dark for 3.5 min. Endpoints are ACTIONS; a 200 means it ran. The call I wanted was documented in this repo's own docs/pfi/opnsense-api-reference.md. → persistent-memory.d/2026-09-15-opnsense-api-reboot.md

• [2026-09-15] Advertising 10.100.10.50/32 from nh3-dev to make its LAN address mesh-reachable — black-holed it from ESH/ANA/FV/IRV while its own LAN and the internet kept working, so a one-host check passes cleanly. lookup 52 at rule priority 5270 beats main at 32766. Fix belongs at the router. → persistent-memory.d/2026-09-15-nh3-dev-ts-input-masquerade.md

• [2026-09-15] Remote-site MASQUERADE rules on nh3-scale for the asymmetric-return theory — they fired (counters incremented) but were not the fix. Reverted rather than left to accumulate as NAT achieving nothing.

115 older entries archived to archival-memory.md.