Persistent memory — eshpfi-management

Last updated: 2026-09-26 ~1620 PT (nh3-pve AMT live; nh3-ml1 load-shared + 3 brokkr foundry seats; esh-matter (Matter server) live for HA; augaman repo + agent live; coder stays on fv-ml1; VibeVoice → Q8_0.)

Always check for /tmp/infra-ops-handoff.md — if it exists and its Written: stamp is under 8 hours old, read it (it carries the in-flight handoff from the previous session), then delete it. Older than 8 hours: stale — delete it unread.

(Raised from 1 h to 8 h by operator 2026-09-13 — a one-hour window deleted the handoff unread across any overnight gap, which is the exact case it exists for. 8 h also matches the global CLAUDE.md and the /snapshot skill default.)

Repo purpose

2026-09-10 Beszel fleet wiring: all seven requested hosts plus existing corviduo-dev report up. /tank and other data filesystems now have real usage metrics; NVIDIA telemetry covers ana-ml2 and irv-ml1. Thirty alerts deliver to infra-ops, explicitly chosen by operator; Miranda routing is deferred. A real low-threshold disk alert reached althing, then the threshold was restored to 85%/5 min. Homepage has one native overview widget (reachability counts, not degraded health). Dedicated superuser approved and stored in Vaultwarden. See persistent-memory.d/2026-09-10-beszel-fleet-wiring.md and stacks/beszel/README.md.

Reference workspace for PFI infrastructure: server inventory, canonical Docker Compose stacks, ops playbooks, and conventions. Authoritative copies of compose files live on the servers under /opt/docker/compose/<stack>/; this repo mirrors them for version control, editing, planning, and CI-driven deploys. It was originally spun up to handle the fleet backups — keep that lens when triaging backup/storage issues.

Tools and conventions

Sister repos (separate gitea repos, deployed by playbooks here):

Repo	Role	CI status
`vh/task-board`	MCP + web dashboard for assistant task state (port 7878) — [2026-09-24] MOTHBALLED by Prime (superseded by the High Seat + ledger); container removed on ana-docker, data/image/compose kept (`e6da607`)	push-to-main → CI deploys (2026-04-29)
`vh/vor`	Inquisitor UI sidecar (port 7879)	push-to-main → CI deploys (2026-04-29)
`vh/nevermore`	Twice-daily LLM-curated briefing (port 8181, replaces news-digest)	push-to-main → CI deploys (2026-04-30)
`vh/asset-engine`	Internal control plane over inference services (port 8200, LAN-direct)	push-to-main → CI deploys (2026-05-12)
`vh/althing`	Lean trusted inter-agent message bus — v3.0.0 "the post office" as of 2026-08-28 (U9b flag day, one-way, no rollback): ONE container on nh3-dev at `http://10.100.50.40:8390` is the only stateful component; `althing-po-herald` one per box; `althing-listen` one per session; `postbox` is the client. Every v2 command was DELETED, not deprecated — `althing-cli`→`postbox`, `althing-wake-listener`→`althing-listen`, `althing-light-monitor`/`althing-receiver` gone. Sessions need BOTH `ALTHING_POST_OFFICE` and `ALTHING_HANDLE`; there is no default address. ⚠ An unreachable post office is an OUTAGE, never an empty inbox. → `persistent-memory.d/2026-08-28-althing-v3-cutover.md`	per-box install (NOT CI-deploy); nh3-dev = container host + repo; nh3-extdev = system WHEEL at `/opt/uv-tools`, needs its own wheel install (`playbooks/nh3-extdev-althing-v3.yaml`)
`vh/mead-hall`	Bifrost tool-provider sidecar (port 5173 on dev VM 10.100.10.50)	push-to-main → CI deploys (2026-05-16)
`vh/skaldsong`	Wizard + reader surface (port 8300, ana-docker, registry-pull pattern)	push-to-main → CI deploys (2026-05-19)
`vh/Worldtree`	Conversation API (corviduo-dev demo :8080 / personal :8081 / pinned :8082) — Heimdall auth, Bifrost integration. gitea-runner builds on ana-docker; claude-bot ADMIN collaborator (2026-06-20). Now v1.0.0b19.	push-to-main → CI build-and-deploy (runner on ana-docker)
`vh/yt-voice-clipper`	YouTube → diarized voice-clip dataset builder + audition console (irv-ml1 :8000)	push-to-main → gitea-webhook auto-deploy to irv-ml1 (2026-06-03) — see `docs/runbooks/ytvc-autodeploy.md`
`vh/arbo`	Catalog-driven ComfyUI engine (irv-ml1 :8201, comfy-dev owns engine/catalog/image)	push-to-main → gitea Actions CI (deploy-engine.sh, build-local, health-gated) now LIVE; catalog via :9009 webhook
`vh/zonos-gateway`	OpenAI-compatible TTS gateway over stock ZONOS2 (`:8890` irv-ml1); emotion dials-first + voice mapping; reached via LiteLLM `ext-tts` alias. v0.2.1 (2026-07-18): voice-resolved emotion presets (`resolve_preset(name,voice)`; angry/happy/startled_happy per-voice). 8 voices incl. 4 clones	pushed to gitea (main `8f1885b`/`v0.2.1`); deployed irv-ml1 tree still NON-git (hand-updated build context — CI-wire = open follow-up). Spec `docs/EMOTION-DIALS-SPEC.md`; host-managed voices bind-mount (`./voices:/app/voices`, drop wav + restart, no rebuild)
`vh/soong-lab`	Noonien Soong character-design studio (SPA + /api + WT `/bifrost/tool-call`); containerized 2026-07-18, LIVE on corviduo-dev `:8443` (image `vh/soong-lab:latest`). soong-dev owns Dockerfile/compose/workflow; infra-ops owns the host	CI = Gitea Actions build+push+DEPLOY on tag/dispatch (fleet recipe: docker:cli + raw buildx, pushes AS vh; auto-redeploy LIVE 2026-07-18 — runner SSHes corviduo-dev as `deploy`, `compose pull && up -d` from /opt/soong-lab, health-gated on /api/version). Manual redeploy `sudo -u deploy bash -c 'cd /opt/soong-lab && docker compose pull && docker compose up -d'`. → `archival-memory.md` (archived 2026-08-16)
`model-training-forge` (mtf-dev)	Fine-tuning recipe forge; T1 = E-RP writing LoRA, retargeted qwopus-122B→AEON-27B (2026-07-06) (SFT→DPO, LitBench-RM reward)	training runs, not a deployed sidecar

(vh/volva + Heid were re-architected from systemd daemons to Claude Code session orchestrators 2026-06-08; their nh3-dev .service units were removed — no longer deployed sidecars here. See Recent decisions.)

Two-layer backups — Backrest orchestrates restic for file+DB (5 fleet repos, daily 01:00 PDT); PBS-ANA primary + PBS-NH3 DR mirror for VM images. ana-nas is the SPOF for postgres + PBS-ANA datastore + cross-site restic targets — see docs/runbooks/disaster-recovery.md for the blast-radius matrix. ⚠️ The restic file+DB layer routes through TWO rest-servers (rest-server-ana @ ana-docker:8000 → ana-docker/ana-ml2/esh-docker-vm/vm-esh-nas; rest-server-nh3 @ nh3-nas:8000 → irv-ml1/nh3-docker). Both depend on their NAS's NFS export of /mnt/backup. (rest-server-ana recovered 2026-06-20.)
pull-hf-repo.yaml is the canonical "get a HuggingFace model/dataset onto ana-ml2's shared cache at /tank/aimodels/huggingface/" playbook. Supports --var repo_type=model|dataset|space. Replaces ad-hoc huggingface_hub.snapshot_download patterns.
Worldtree admin auth — per-instance. Each Worldtree deployment (demo :8080, personal :8081, pinned :8082) has its own Heimdall registry and its own bootstrap admin key. Infra-ops's stored long-lived admin key (key_id 61419c92) at ana-docker:/opt/docker/conf/.secrets/worldtree-infra-ops-admin auths against demo only. Personal-instance admin (the ~/.config/worldtree/personal-admin-token, mode 600) POSTs /admin/keys (mints per-project keys; takes user_id+label, no scope param — scopes are tier-derived). On-instance mint recipe (cleaner than DB-manip): docker exec worldtree-worldtree-api-1 POST /admin/keys with the in-container WORLDTREE_BOOTSTRAP_ADMIN_KEY; cleartext once in .key=wt_live_+16hex. auto-memory reference_worldtree_demo_key_mint.
Per-project user keys against personal Worldtree (issued 2026-05-19): skaldsong:79744637, skaldsong:7c1dbbbe, althing:50d85460, mead-hall:a360822d. Mint via /admin/keys, drop value to /tmp/wt-personal-<name>.key mode 600, dev collects + shreds (DO NOT cat to chat transcript).
Skaldsong CD pattern (registry-pull). vh/skaldsong's CI builds and pushes gitea.phasefinal.com/vh/skaldsong:<sha> + :latest; playbooks/deploy-skaldsong.yaml on ana-docker pulls + recreates. SHA-pin only. Prereq: host needs docker login gitea.phasefinal.com once.
gitea internal route for fleet hosts. gitea is a container on ana-docker — git-SSH 10.250.50.70:222, HTTP :3000. Fleet/colo hosts must use this internal route, NOT public gitea.phasefinal.com (38.120.12.44) — the public path fail2bans the host egress IP. Full gotcha in docs/orientation.md → Git/gitea.
docker-as-root pattern (for ops with no admin API, or to edit deploy-owned/root-owned files without sudo): docker run --rm -v <target-dir>:/wt docker:cli sh -c "...". docker-group membership is effectively root via bind-mount. Foot-gun: relative paths in compose.yaml resolve against the sandbox CWD but the daemon interprets them against the HOST fs — always pass -e VAR=/abs/path for any relative-default config dir.
scripts/elway sudo handling — elway prompts for the sudo password ONCE via getpass before the first sudo: true step → can't run unattended from a non-TTY tool if any step needs sudo. Sudo-free playbooks run fully non-interactive over key SSH.
Per-host SSH identity matters for sudo. infra-ops has NOPASSWD sudo on most PFI Linux boxes (corviduo-dev included since 2026-06-15). On ana-docker: default ssh ana-docker = lkraven (docker-group, NO passwordless sudo); ssh infra-ops@ana-docker HAS NOPASSWD root. → For any sudo op on ana-docker, use ssh infra-ops@ana-docker. ssh infra-ops@10.100.10.50 (nh3-dev) ALSO NOPASSWD sudo; on nh3-extdev infra-ops is sudo-LESS by design (ssh lkraven@10.100.50.42 is the NOPASSWD path) — [2026-09-23] measured: infra-ops HAS NOPASSWD sudo on nh3-extdev (fleet ownership audit; also CLAUDE.md 2026-09-05). irv-ml1: ssh irv-ml1 = lkraven, docker-group (plain docker) but sudo needs a PASSWORD (no NOPASSWD) — stage model pulls to /home, not root-owned /worktank.

Current state / in-flight

As of 2026-09-26 ~1620 PT.

nh3-pve + nh3-ml1: post-visit, all live (2026-09-25/26)

nh3-pve: Secure Boot OFF; IGFX restored; NVIDIA 580.178.04 DKMS; kernel 6.8.12-43 (benign btmtk oops every boot). AMT live: static 10.100.250.61 on nh3-mgmt (UDM port 6), KVM on, Opt-in None, password in the vault as nh3-pve/amt-admin. → servers/nh3-pve/README.md
nh3-ml1 (CT 109 @ 10.100.50.80):
- TEI embed/rerank is load-shared with esh-ml1 through the gateway, with router_settings.enable_weighted_failover.
- brokkr's foundry seats:
  - lfm-vl :8030 (gateway lfm25-vl-3b);
  - lfm-vl-uncensored :8032 (direct; passed brokkr's eval);
  - vibevoice-asr :8031 (audio.cpp, Q8_0).
- GPU ~11.3/16 GB. → servers/nh3-ml1/README.md
Not yet proven: nh3-ml1 coming back by itself after an nh3-pve reboot (only the config was checked).

esh-matter: Matter server for Home Assistant (2026-09-26)

CT 111 @ 10.0.90.20, on VLAN 90 (esh-iot) only. matter.js 1.4.0; :5580 is firewalled to HA 10.0.50.46. HA's matter integration is loaded (ha-dev).
Next is Prime's: share the Aqara W200 into HA via Matter multi-admin. If commissioning misbehaves, the UniFi mDNS reflector on esh-iot is the first knob (a Prime/infra-ops change). → servers/esh-matter/README.md

augaman: face recognition for Cicada

pfi/augaman; augaman-dev is live and has written the three contracts (contract review next). infra-ops' part waits for augaman-dev to reach deploy (see Recent decisions 2026-09-26).

esh-ml1

The reward seat has no working consumer: Worldtree's Domari path is broken (dead IP + wrong schema). worldtree-dev was told on 09-25; waiting on them.
The nh3-docker Dozzle agent has been stopped by hand since ~2026-04. Revive or drop.
The Beszel superuser password was echoed into a session transcript (local only, not in memory). Rotation offered to Prime.

Live threads

git: main is ahead of origin by 2 at 2fdbac6, plus this snapshot. Pushing is Prime's call.
ESH has a single outside route (esh-scale on esh-pve). Noted, untracked.

Recent decisions

[2026-09-26] augaman: infra-ops owes the gallery backup + first deploy — DEFERRED until augaman-dev reaches deploy. → persistent-memory.d/2026-09-26-augaman-infra-ops-owes-the-gallery-backup-first-deploy.md
[2026-09-26] zellij-fleet@.service installed, NOT enabled (for svos-dev's seat_up; owns the fleet zellij server in its own cgroup). Enabling @Claude at boot is Prime's call when seat_up ships. Tracked: 0ad7799, services/zellij-fleet/README.md.
[2026-09-26] btusb blacklist on nh3-pve + esh-pve — proposed, awaiting Prime (low priority). Every 6.8.12-4x boot oopses in btmtk (benign). Also noted for the next NH3 visit: nh3-pve runs ~20 °C warmer than esh-pve (workload-confounded; check airflow). Tracked: servers/nh3-pve/README.md.
[2026-09-26] GPU-LXC Temperature alerts watch the GPU, not the host CPU (SENSORS=-coretemp_*,acpitz); hypervisor CPU alerts at >95 °C on nh3-pve/esh-pve. The first nh3-ml1 alert was nh3-pve's CPU during vzdump. 6d901c0.
[2026-09-26] Abliterated LFM2.5-VL-3B parallel seat (:8032) passed brokkr's 6-image eval and stays (direct only; stock seat kept for SFW A/B). 9cc3824, 944bb36.
[2026-09-25] AMT follow-ups PARKED until the two MS-03s arrive and esh-pve's MS-01 is on AMT (Prime 2326). — Parked: MeshCommander test; dummy HDMI plug, then NanoKVM → gx10; an OOB path not through nh3-scale… → persistent-memory.d/2026-09-25-amt-follow-ups-parked-until-the-two-ms-03s-arrive-and-esh.md
[2026-09-25] nh3-pve AMT LIVE: static 10.100.250.61 on nh3-mgmt (UDM port 6), KVM on, Opt-in None — (Prime). → persistent-memory.d/2026-09-25-nh3-pve-amt-live-static-10-100-250-61-on-nh3-mgmt-udm-port.md
[2026-09-25] pfi-gx10 AC-restore VALIDATED by Prime's AC pull; Homepage stays on esh-docker-vm (Prime) after the mmap_lock wedge reboot. servers/pfi-gx10/README.md, servers/esh-docker-vm/README.md.
[2026-09-26] VibeVoice ASR → Q8_0 (Prime). WER on the 4 bundled LibriSpeech clips 3/69 → 2/69 (only the I'm/I am artifact left), ×3 identical; RTF 0.09–0.17; +1.1 GB VRAM (nh3-ml1 ~11.3/16 GB). Q4_K file removed.
[2026-09-26] esh-matter LIVE: a Matter server (matter.js 1.4.0) on CT 111 @ 10.0.90.20, VLAN 90 only — , for ha-dev (operator-approved, relayed). → persistent-memory.d/2026-09-26-esh-matter-live-a-matter-server-matter-js-1-4-0-on-ct-111.md
[2026-09-26] Embed/rerank LOAD-SHARED across esh-ml1 + nh3-ml1 (Prime). — Second deployments were added for qwen3-embedding and reranker (config) and for reranker-a3-bge-v2-m3 (DB… → persistent-memory.d/2026-09-26-embed-rerank-load-shared-across-esh-ml1-nh3-ml1-prime.md
[2026-09-26] Two dataset-foundry utility seats LIVE on nh3-ml1 (brokkr; operator approval relayed): — LFM2.5-VL-3B on llama.cpp :8030 (gateway lfm25-vl-3b, LiteLLM restarted 36 s at 0039) and… → persistent-memory.d/2026-09-26-two-dataset-foundry-utility-seats-live-on-nh3-ml1-brokkr.md
[2026-09-26] Coder seat STAYS on fv-ml1 (Prime). — The nh3-ml1 copy gave the same quality (teacher-forced true-code logprob diff +0.008 ± 0.019) but ran ~5×… → persistent-memory.d/2026-09-26-coder-seat-stays-on-fv-ml1-prime.md
[2026-09-25] nh3-ml1 LIVE after the NH3 visit (SB off, IGFX restored): driver + CT 109 + TEI, parity vs esh-ml1 indistinguishable (cos min 0.999993 = own floor; overlap@10 1.000 vs MRL-256 control 0.684), same speed; monitoring/DNS wired. Autonomous: lxc-pve 6.0.0-2 upgrade on nh3-pve (Docker-in-LXC fix #7006). Gateway routing + AMT follow-up are Prime's. → persistent-memory.d/2026-09-25-nh3-ml1-live.md
[2026-09-25] nh3-ml1 = CT 109 @ 10.100.50.80, second TEI embed/rerank backend on nh3-pve's RTX 2000E; GPU playbooks made host-generic — BUILD DEFERRED: blocked on nh3-pve Secure Boot until the NH3 visit; SB handling and gateway routing are Prime's calls. Tracked: 7ddd116, Current state. → persistent-memory.d/2026-09-25-nh3-ml1-standup.md
[2026-09-25] nh3-pve OOB: HOLD blind until the next NH3 visit (Prime via Miranda, 1333). Target: NanoKVM → pfi-gx10, and the MS-01 on its own vPro. The IGFX BIOS fix rides the same visit, since AMT KVM captures only the iGPU. Tracked: servers/nh3-pve/README.md (2b49be4, cdd7605).
[2026-09-25] MS-01 foot-gun: a GPU in the PCIe slot renames every NIC — (the slot's root port takes bus 01, so the X710 goes enp2s0f0np0→enp3s0f0np0). → persistent-memory.d/2026-09-25-ms-01-foot-gun-a-gpu-in-the-pcie-slot-renames-every-nic.md
[2026-09-25] Direct ESH→esh-ml1 consumer path — PARKED (no ESH-side callers in 7 days; consumers go through the gateway). Tracked at persistent-memory.d/2026-09-24-esh-ml1-embed-rerank.md.
[2026-09-25] Created empty private repo corviduo/norn (git@gitea.phasefinal.com:corviduo/norn.git) as claude-bot, at brokkr-smithy-dev's relay of the operator's Norn ruling; the norn-dev handle is the operator's to declare.
[2026-09-25] Reward seat audited (Skywork-Reward-V2-Llama-3.1-8B still #1 of 188 on RewardBench 2; our AWQ ≈ bf16 within noise; double BOS costs ~2.7 pts) and moved fv-ml1 → esh-ml1; esh-ml1 monitoring wired (Beszel+GPU, Kuma, Homepage, Dozzle); Dozzle hub's stale agent IPs fixed; nh3-dev Beszel agent revived. → persistent-memory.d/2026-09-25-reward-move-and-esh-ml1-monitoring.md
[2026-09-25] TEI adopted as the fleet embed/rerank engine (Prime); esh-ml1 is the sole gateway backend for qwen3-embedding + reranker; fv-ml1's vLLM embed/rerank seats retired (~6.1 GB freed); parakeet stays on fv-ml1. → persistent-memory.d/2026-09-25-tei-fleet-embed-rerank.md
[2026-09-24] esh-ml1 built — RTX 2000E Ada as CT 110 on esh-pve (host driver 580.178.04 DKMS, loaded live, no reboot), vLLM embed+rerank at measured parity with fv-ml1, wired as LiteLLM order: 2 failover; dead DB alias reranker-a3-bge-v2-m3 (pre-relocation IP) repaired. → persistent-memory.d/2026-09-24-esh-ml1-embed-rerank.md
[2026-09-24] esh-pve: VM 102 retired, a 14-day hung VFIO process found and cleared, T400 → RTX 2000 Ada; Prime chose LXC + host driver for embed/rerank (implementation deferred to next session, tracked here + servers/esh-pve/README.md). → persistent-memory.d/2026-09-24-esh-pve-vm102-retired-gpu-swap.md
[2026-09-24] pfi-gx10 AC-restore UEFI patch applied but NOT validated; box OFF until Prime's AC pull 2026-09-25 (tracking: servers/pfi-gx10/README.md, e5197a3). — [2026-09-25] ✅ VALIDATED: Prime pulled and replugged AC, and it came up by itself at 1108:54 (2b49be4). → persistent-memory.d/2026-09-24-gx10-ac-restore-patch.md
[2026-09-24] NH3 power outage recovered — pbs-nh3 had no onboot (set), NFS boot race fixed with automount (1cbde50), every other Claude session on nh3-dev died. → persistent-memory.d/2026-09-24-nh3-power-outage-recovery.md
[2026-09-24] Miranda standing order is a repo CLAUDE.md operating parameter — (4b29492, aligned to the global send protocol in bcf3342): high-urgency matters go to her, fixed or not… → persistent-memory.d/2026-09-24-miranda-standing-order-is-a-repo-claude-md-operating.md
[2026-09-24] task-board mothballed (Prime): container removed on ana-docker, data/image/compose kept, Kuma monitor deleted, task_* instructions removed from CLAUDE.md and the fork template (e6da607). Its hooks had sent no traffic in 30 days.
[2026-09-24] Military 24-hour Pacific clock times carried into the Codex/Grok shared bootstrap docs/fleettools/AGENT-BOOTSTRAP.md (ad2b4d9); Claude seats get it from the global CLAUDE.md.
[2026-09-24] Worldtree admin.memory.forget stays OFF on demo/personal until an instance needs it — a destructive erase; enabling it is a per-change operator yes via deploy-wt-config.
[2026-09-24] A git checkout under root:docker needs safe.directory for its deploy user — the 09-14 normalization (826a63b) silently broke yt-voice-clipper's webhook deploy until v0.3.13; fixed on irv-ml1 and recorded in fleet conventions (eea9eb2). Sweep found no other case.
[2026-09-23] elway sudo uploads land root:root, validated and staged; fleet ownership audit built; 33 mis-owned root files fixed on 10 hosts. → persistent-memory.d/2026-09-23-elway-ownership-fix-fleet-audit.md
[2026-09-23] headscale-ddns hardened — (fedd4b6): Cloudflare calls retry and validate every body (pick()), no write without both IDs, the run… → persistent-memory.d/2026-09-23-headscale-ddns-hardened.md
[2026-09-23] esh-docker-vm restic was skipped 09-22..23 by my own Kuma move — a dead uptime-kuma lookup aborted pre-backup.sh under set -e (25e41d2). → persistent-memory.d/2026-09-23-esh-docker-vm-restic-was-skipped-09-22-23-by-my-own-kuma.md
[2026-09-23] hermes-gateway restart exit-1 is a Hermes race, not a crash — the planned-stop watcher consumes the marker before SIGTERM re-runs the handler; drop-in… → persistent-memory.d/2026-09-23-hermes-gateway-restart-exit-1-is-a-hermes-race-not-a-crash.md
[2026-09-23] Booth link board cleared to 14 durable links (208 removed: drops, release posts, research links, token-bearing URLs; Scriberr reposted at scriberr.fv.internal:8080). Prime's rule: durable debug links only.
[2026-09-22] Both carried calls approved — build the NRestarts flap sampler (163bb97); the restic content-assertion ruling is ratified and stays (ba60fda).
[2026-09-22] safe-rm installed on nh3-dev and delegated fleet-wide to infra-hermes. — ⚠ The package installs INERT and looks fine — Debian's /etc/zsh/zprofile has 0 non-comment lines so the… → persistent-memory.d/2026-09-22-safe-rm-installed-on-nh3-dev-and-delegated-fleet-wide-to.md
[2026-09-22] The acceptance probe for a guard must not be able to destroy what it tests — (infra-hermes). → persistent-memory.d/2026-09-22-the-acceptance-probe-for-a-guard-must-not-be-able-to.md
[2026-09-22] ⚠ {"sent": true} is a claim about transmission, never about effect. pane_send structurally cannot deliver a harness command (every relay is prefixed with the card id), so D-0010 promised an unachievable /clear and its receipt reported success. Consumed an operator approval. → persistent-memory.d/2026-09-22-instrument-errors.md
[2026-09-22] D-0010/D-0011 were misrouted to this seat by pane_find matching a ROLLING PANE TITLE. — Genuine and operator-approved, wrong seat; fleet_telemetry held the right mapping and carries the warning… → persistent-memory.d/2026-09-22-d-0010-d-0011-were-misrouted-to-this-seat-by-panefind.md
[2026-09-22] headscale-ddns exited 1 silently and the alarm carried no cause — both failure paths were || exit 1 with no message. Now every exit names its reason and the WAN lookup retries 3×. The script was also untracked; a fix to what every mesh client resolves through lived on one disk. 30517fd.
[2026-09-22] ⭐⭐ Five instrument errors in one day, and the shape is one thing: a tool that enumerates "things that are fine" has selected against its own subject. --state=running skipped the units most needing hooks; awk '{print $1}' dropped systemd's ●-decorated FAILED rows; grep -ic restic on the wrapper missed the check script; restic ls's header line made an absent path read as "blobs gone"; and a cooldown test that invoked before failing the unit. Every one reported cleanly while looking at the wrong thing. The rule is not "verify" — it is verify, then ask what the verification could not have seen. → persistent-memory.d/2026-09-22-instrument-errors.md
[2026-09-22] Fleet alert bridge generalized — beszel-althing → althing-alert-bridge, route registry (/beszel + /kuma), each with its own… → persistent-memory.d/2026-09-22-fleet-alert-bridge-generalized.md
[2026-09-22] Uptime Kuma rebuilt from scratch on 2.5.5, moved esh-docker-vm → ana-docker. — ⚠ :latest is a TRAP — it tracks 1.x, so an Aug-2026 pull gave a Dec-2024 build. → persistent-memory.d/2026-09-22-uptime-kuma-rebuilt-from-scratch-on-2-5-5-moved-esh-docker.md
[2026-09-22] Beszel and Uptime Kuma are DISJOINT, not redundant — Beszel's alerts bind to a system with a threshold; there is no URL column, so it is structurally incapable… → persistent-memory.d/2026-09-22-beszel-and-uptime-kuma-are-disjoint-not-redundant.md
[2026-09-22] Failed-START alarms on 23 nh3-dev units — (services/althing-notify-failure/). → persistent-memory.d/2026-09-22-failed-start-alarms-on-23-nh3-dev-units.md
[2026-09-22] Backup coverage is a property of the SYSTEM, never one job's scope — establish it by querying the repo for the path in a real snapshot, never by reading a job's SRC=. → persistent-memory.d/2026-09-22-backup-coverage-is-a-property-of-the-system-never-one-job-s.md
[2026-09-22] restic checks now assert CONTENT and are DISCOVERED not enumerated — conjunctive (recent AND present AND restores non-zero bytes); repos discovered from each NAS, hand list… → persistent-memory.d/2026-09-22-restic-checks-now-assert-content-and-are-discovered-not.md
[2026-09-22] irv-ml1: Irvine is a TENANCY behind a Fortinet PFI does not control — its TLS inspection breaks Tailscale relay/control intermittently (41 cert warnings/week). → persistent-memory.d/2026-09-22-irv-ml1-irvine-is-a-tenancy-behind-a-fortinet-pfi-does-not.md
[2026-09-21] ⭐⭐⭐ lv-mccarthy GATED and NOT SHIPPED — voice passes, memorisation is the cleanest in the line, and the length discipline is gone. 720 generations, 3 arms. Voice +0.152 at 2.9× floor for ckpt450 (ckpt900 +0.172 at only 1.2×, its spread one outlier seed), and ~3/4 of the gain survives stripping every punctuation mark, so it is not the cheap win. ⭐ Memorisation: ckpt450 at 0.12 against the author's own held-out 0.12 — identical, longest match 11 words against the author's coincidental 12, all 96 matches READ and every one stock grammar (he looked at the wolf and he looked at him); the name-shaped hits are the RENAMED inventions. Axis C is the blocker: 20% / 28% of generations overshoot the 90–140 band against base's 1%, worst case a degenerate loop at 279 words. ⚠ My own prereg's axis C transcribed score_beats.py's v1 criteria including "in-band up on base", which the operator RETIRED 2026-09-15 because base maxes it — under the v2 (ran-on only) ckpt450 passes by 0.01 against a 0.200 floor, but that reading was found AFTER the numbers and was not used. Fix the prereg prospectively. → persistent-memory.d/2026-09-21-lv-mccarthy-gate.md
[2026-09-21] The two-epoch recipe is now 0 for 3, and this time the loss curve was CONFIDENTLY wrong. — On Brontë and Hemingway the epoch-1/epoch-2 checkpoints were TIED on eval loss, so preferring the earlier one… → persistent-memory.d/2026-09-21-the-two-epoch-recipe-is-now-0-for-3-and-this-time-the-loss.md
[2026-09-21] The memorisation control that shipped broken is now committed, and it turned a 12× red flag into a clean pass. → persistent-memory.d/2026-09-21-the-memorisation-control-that-shipped-broken-is-now.md
[2026-09-21] My confound detector was counting apostrophes as quote marks, and I saw it FIRE before I saw the bug. → persistent-memory.d/2026-09-21-my-confound-detector-was-counting-apostrophes-as-quote.md
[2026-09-21] ⭐⭐⭐ The ops log shipped — and then failed FOUR ways in its first hours, every one recording something unfindable. Claim dropped by a sub-tool, hook behind graphify's eight exit 0s, no handle in the env, ssh-target written as a hostname. The general shape is configured ≠ effective; twelve instruments reported confidently and wrongly across three days, five of them mine. → persistent-memory.d/2026-09-21-ops-log-and-the-instruments-that-lied.md
[2026-09-21] ⭐⭐ The Booth gained blur + a closed keep round trip, after shipping TWO controls that did nothing — a reveal handler Jinja discarded for sitting after {% endblock %}, and a × a sibling form covered by 30×22 px. Both operator-found, both from reading templates instead of rendering them. scripts/layout-probe.py took four iterations to become trustworthy. ⚠ Blur is COSMETIC and a test asserts the 200 on purpose. → persistent-memory.d/2026-09-21-booth-two-dead-controls.md
[2026-09-21] ⭐⭐ claude-bot is an Owner in corviduo/pfi/vastblue; cicada + draupnir moved to pfi; vh-token use is now standing-authorized from the vault. ⚠ vh is a USER not an org, so no namespace-scoped admin exists — the only realization of the original ask was site-admin, surfaced rather than executed. ⚠⚠ ~/.config/claude-bot/gitea-token is DEAD and had been misreporting permissions; the working one is gitea-token-repo-create. → persistent-memory.d/2026-09-21-gitea-orgs-and-the-dead-token.md
[2026-09-21] ⭐⭐ nh3-dev 84%→72%: 27 GB reclaimed, and a 252 MB LoRA adapter rescued from /tmp, which this box sweeps at 3 days. babyyarros existed nowhere else; hash-verified to smithy before any deletion. ⚠⚠ My 7-day session prune then deleted an ACTIVE session's dir — directory mtime does not reflect subdirectory writes. Do not re-run that predicate. → persistent-memory.d/2026-09-21-disk-triage-and-the-rescued-adapter.md
[2026-09-21] Draupnir geometry engine provisioned on irv-ml1 and acceptance-tested — (34c4179, e574b91, 2e08edc). → persistent-memory.d/2026-09-21-draupnir-geometry-engine-provisioned-on-irv-ml1-and.md
[2026-09-21] Backup alarm verdict split: STALE (exit 1) vs ERRORED-JOBS (exit 3) — (7fe4102). → persistent-memory.d/2026-09-21-backup-alarm-verdict-split-stale-exit-1-vs-errored-jobs.md
[2026-09-21] vh/forgefirm mirrored — from github.com/openglow-org/forgefirm, following the house convention read off the existing 17: vh/… → persistent-memory.d/2026-09-21-vh-forgefirm-mirrored.md
[2026-09-20] ravenpen.com REGISTERED by the operator — the 09-18 hold is discharged and hamr-dev is answered. — infra-ops deliberately did NOT execute this twice, because it was a non-refundable purchase reaching us as a… → persistent-memory.d/2026-09-20-ravenpen-com-registered-by-the-operator-the-09-18-hold-is.md
[2026-09-19] FV is a dedicated 20 A circuit carrying fv-ml1 AND the R420 running OPNsense, and nothing else (operator-confirmed). → persistent-memory.d/2026-09-19-fv-is-a-dedicated-20-a-circuit-carrying-fv-ml1-and-the-r420.md
[2026-09-19] althing 3.7.0 rolled out to both infra-ops surfaces — and the rollout broke the claim tooling built that morning. → persistent-memory.d/2026-09-19-althing-3-7-0-rolled-out-to-both-infra-ops-surfaces-and-the.md
[2026-09-19] Three agents commit as one git author, and closing that gap took three instruments to get right. — An unattributable commit (e43e262) appeared in the push set between two of mine — unidentifiable from git… → persistent-memory.d/2026-09-19-three-agents-commit-as-one-git-author-and-closing-that-gap.md
[2026-09-19] The backup alarm had no wire for three weeks, and fixing it exposed two more instruments that pass without looking. → persistent-memory.d/2026-09-19-the-backup-alarm-had-no-wire-for-three-weeks-and-fixing-it.md
[2026-09-19] elway evaluated when: / creates: / removes: / changed_when: WITHOUT the step's sudo, and it fails silently in the dangerous direction. → persistent-memory.d/2026-09-19-elway-evaluated-when-creates-removes-changedwhen-without.md
[2026-09-19] ESH VM 102 (esh-vm-workstation) excluded from the nightly backup job — operator ruling. — It is a Windows 11 Parsec/RDP sandbox (no password, no state to recover), and its vzdump had failed… → persistent-memory.d/2026-09-19-esh-vm-102-esh-vm-workstation-excluded-from-the-nightly.md
[2026-09-19] The ops log is BUILT — scripts/ops-log, automatic writers, and a detector for the path they cannot cover. → persistent-memory.d/2026-09-19-the-ops-log-is-built-scripts-ops-log-automatic-writers-and.md
[2026-09-19] infra-hermes is this session's ASSISTANT, and the division of labour is now standing policy. — infra-ops keeps improving infrastructure tooling plus the hard calls; infra-hermes does **day-to-day… → persistent-memory.d/2026-09-19-infra-hermes-is-this-session-s-assistant-and-the-division.md
[2026-09-18] ⭐⭐⭐ NH3↔Anaheim had been running over a throttled DERP relay, not a direct path — 78 GB of fleet traffic on someone else's free infrastructure. Four additive objects on ana-gw gave ana-scale a stable inbound UDP 41641 endpoint; tailscale ping 373–522 ms → 6 ms direct, cross-site HTTP 1.2 s → 0.015 s, STT via the ANA gateway 1.4 s → 0.25 s. ⚠ That box runs central-nat, so a policy dstaddr is the REAL internal address, not the VIP. No OOB access — back up with show to a local file and make additive changes ONLY. irv-ml1 still relayed. → persistent-memory.d/2026-09-18-nh3-ana-derp-relay.md
[2026-09-18] ⭐⭐⭐ .internal DNS was failing ~10% of lookups fleet-wide, from two independent causes. A PUBLIC resolver was the fallback for a PRIVATE zone (Cloudflare answers NXDOMAIN authoritatively, so a transient miss became a hard failure) — now a cross-site ring, each site local-first with a different site as backup. Then the root cause: all three AdGuards shipped ratelimit: 20 shared across an entire /24, silently dropping queries at 5 s each. Set to 0. Hard failures 3/40 → 0/40; burst timeouts 40/60 → 0/60. ⚠ resolv.conf is DHCP-managed — change it at the UDM/FortiGate, not the file. → persistent-memory.d/2026-09-18-fleet-dns-ring-and-ratelimit.md
[2026-09-18] ⭐⭐ SearXNG had ONE working general web engine and every health check said fine. 7 of 55 were enabled-by-default and six of those are dictionary/translation engines — inactive: false only makes an engine SELECTABLE, disabled: false puts it in the DEFAULT set. Now seven. ⭐ This stack tracks :latest ON PURPOSE (upstream ships engine-handler fixes continuously; a pin freezes breakage). ⭐ The Brave key is committed in plaintext by explicit operator decision — scoped to one low-value credential, NOT a change to the no-secrets rule. → persistent-memory.d/2026-09-18-searxng-one-engine-to-seven.md
[2026-09-18] ⭐⭐ althing search returned ZERO for every hyphenated query — silently, for every handle, and on this fleet that is most of our hostnames. FTS5 read the hyphen as a column filter; search() caught the error, its probe passed, and it returned []. Routed to forseti (they own the code, I own rollout) → v3.6.3 deployed, 17 s bus outage. The reported symptom, a message that never arrived, was NOT a defect: the reporter's evidence file was truncated at 8 KiB. ⚠ No CI builds the post-office image. → persistent-memory.d/2026-09-18-althing-363-hyphen-search.md
[2026-09-18] ⭐ FleetTools: one capability index, autoloaded by Claude, Codex and Grok from a single symlinked file. docs/fleettools/ + ~/FLEETTOOLS.md; absolute detail paths because a non-Claude agent cats them. Rule zero is query-live-inventories-never-a-written-list. The global CLAUDE.md tools section went 231 lines → 39, keeping only the three rules that govern behaviour rather than lookup. → persistent-memory.d/2026-09-18-fleettools-agent-index.md
[2026-09-18] Miranda's Hermes plugin install is now a SYMLINK to the svos repo, not a copy — (operator-approved). → persistent-memory.d/2026-09-18-miranda-s-hermes-plugin-install-is-now-a-symlink-to-the.md
[2026-09-18] Worldtree's env.sh secrets are vaulted — 10 entries under worldtree/ (gitea, matrix as/hs, openai, uv-index, vastblueai, wt-admin demo+personal… → persistent-memory.d/2026-09-18-worldtree-s-env-sh-secrets-are-vaulted.md
[2026-09-17] ⭐⭐ The next voice seat was MEASURED, not chosen by taste — and the corpus size ranking INVERTS the voice ranking at the top. Our two largest authors are Stephen King (76 works, 12.1M words) and Agatha Christie (72, 5.5M); neither should get a seat, Christie being the Krakauer failure mode exactly (genius in plot architecture, prose deliberately transparent, invisible to a char-bigram Delta). Picks, in order: Faulkner (~15 pure novels, ~1.6M words — highest voice signal in the catalogue, AND he is McCarthy's stylistic ancestor, so training him next supplies the hard-negative sister the gate has lacked since the Brontë record named it missing); Morrison (11 novels after pruning criticism/anthology, ~818k — 11 val units, beating Hemingway's 10); Chandler (7 novels + a 409k short-story omnibus — fills the first-person hardboiled gap, Hemingway-class corpus size). ⚠ Faulkner's catalogue rows carry a 446k-word Snopes omnibus that duplicates novels also present individually — the Hemingway 90-96% collection-duplication trap, needs the containment pass first. → persistent-memory.d/2026-09-17-next-voice-seats.md
[2026-09-17] ⭐⭐ Romantasy IS a real register, we already trained its most distinctive member, and the obvious next pick is its worst. Measured on the gate's own instrument (char-bigram Burrows's Delta, ~120k words/author from mid-work), with within-author floors and cross-genre positive controls. Cluster median pair 0.537 = 1.2x the worst floor against controls at 1.4-1.9x — tighter than cross-genre but NOT collapsed. Two findings survive either floor reading: Yarros is the cluster OUTLIER (4 of the 5 largest pair distances involve her), so a second romantasy seat buys measurably less than the first did; and Maas is the centroid (the two smallest distances in the matrix are hers), so the obvious commercial pick is the least distinctive. If the lane gets a seat it is Kenyon — furthest from Yarros at 0.674 and 27 works = 27 val units, the best-powered gate the line could build (Hemingway 10, McCarthy 6, Brontë 4). ⚠ Sensitivity floor stated: one sample per pair, no repeat draws; the rank ordering is indicative, fine gaps are not resolvable. → persistent-memory.d/2026-09-17-romantasy-register-measured.md
[2026-09-17] dragonfireacoustics.com expires 2026-10-30 — six weeks — at eNom with NO transfer lock, and its sibling domain was already lost exactly this way. → persistent-memory.d/2026-09-17-dragonfireacoustics-com-expires-2026-10-30-six-weeks-at.md ⚠ If it is ever transferred, DNS does NOT come with the registration — the nameservers are eNom's name-services.com and the zone must be recreated first or mail dies. The whole zone is two facts plus a landmine: * (WILDCARD) → 199.250.192.76 which is dead (no HTTP at all, and it is what the apex/mail/webmail/admin/ftp all answer with), www → 38.120.12.45 (us), and 7 Google Workspace MX records that must not be lost. No DNSSEC (delegationSigned: false), so no transfer complication. ⚠ Also found: no SPF and no DMARC at the apex on a Google Workspace domain — a live deliverability problem independent of everything else. Transfer gate is the TAC/EPP code from the eNom account, not the lock; the missing lock is not authorization. 60-day rule is satisfied (last changed 2025-10-24).
[2026-09-17] dragonfireacoustics.com IS configured on pfi-ana-webhost, and the whole thing is dead — a forgotten public-facing VM. → persistent-memory.d/2026-09-17-dragonfireacoustics-com-is-configured-on-pfi-ana-webhost.md
[2026-09-17] headscale now split-DNSes nh3.phasefinal.com to the three AdGuards, so mesh clients can resolve the internal-only wildcard → persistent-memory.d/2026-09-17-headscale-now-split-dnses-nh3-phasefinal-com-to-the-three.md
[2026-09-17] ESH is back on the Cityside static 128.177.138.182/30 and the site is healthy — confirmed on four axes, not one. UDM WAN1 wan_type is static again (switched back from the DHCP set during the 09-17 outage), stat/health names Cityside Fiber with 0 disconnected and Verizon-5G idle at failover priority 2, esh-docker-vm's egress EQUALS the WAN ip so nothing is behind CGNAT, and colo→ESH reads 5.0 ms / 0% loss at 2005/2142 Mbps (Cityside CGNAT was 9 ms, Verizon failover 33–37 ms). ⭐ The FortiGate infra-ops trusthost3 pin un-broke itself and that was VERIFIED: from ESH, ana-gw tcp/22 is open and offers a password prompt, which a trusthost mismatch would never do. ⚠ The two 7-day crowdsec entries are being left to expire 2026-09-23 on purpose — Cityside failed twice in six hours, so they are cheap insurance. → persistent-memory.d/2026-09-17-esh-fiber-outages.md
[2026-09-17] Operator ruled "leave it" on lv-hemingway's 3 separator-hidden names. So leak_gate.py exits 1 on a SHIPPED tree by design; a future session seeing that red result should read this line, not start fixing. lv-bronte re-ran clean.
[2026-09-17] ⭐⭐⭐ The leak gate PASSED lv-mccarthy while five protagonist names sat in all six copies, and the blind spot generalises to every corpus in the line. \b(Surface)\b cannot match a name with a character inserted in it, so a mangled occurrence is unrenameable AND unreportable: B ell, C higurh, M oss, T oadvine (a small-caps drop cap kept as its own token) and Toad-vine, Glan-ton (a print line-break hyphen). Every VISIBLE occurrence had been renamed, which is what made it invisible. Same family as lv-bronte's _Antigua_, now generalised: any separator inside a name blinds a word-boundary scan. Fixed in the corpus builder (rules 4+5, counted), and leak_gate.py now runs a separator-tolerant pass with its own controls that FAILS the gate — validated against the pre-fix tree. ⚠ Its fragment filter is load-bearing: a naive scan returns 18 false positives on Hemingway (God damn, I run) against 3 real. Whole D1→D3 chain reproduced byte-identically before and after. Commit c559664. → persistent-memory.d/2026-09-17-mccarthy-split-name-leak.md
[2026-09-17] The SHIPPED lv-bronte adapter emits mid-sentence line breaks at 12.46 per 1k chars, and nothing downstream looks for it. → persistent-memory.d/2026-09-17-the-shipped-lv-bronte-adapter-emits-mid-sentence-line.md
[2026-09-17] The mccarthy register names the punctuation ON PURPOSE, and that is a gate-design call made before any McCarthy number existed. → persistent-memory.d/2026-09-17-the-mccarthy-register-names-the-punctuation-on-purpose-and.md
[2026-09-17] lv-mccarthy's D1→D3 chain was RECOVERED, not remembered — there was no runbook and the commands went over non-interactive ssh, so no history survived. → persistent-memory.d/2026-09-17-lv-mccarthy-s-d1d3-chain-was-recovered-not-remembered-there.md
[2026-09-17] Measured and DELIBERATELY not changed, three of them. — The oversize-passage drop is 13.9% of McCarthy's train words, between Hemingway's 10.0% and the shipped… → persistent-memory.d/2026-09-17-measured-and-deliberately-not-changed-three-of-them.md
[2026-09-17] ⭐⭐ A unit splitter must choose by SIZE, not count — and the val split scales with WORK COUNT, not corpus size. scripts/r49-corpus/split_units.py + a multi-index --holdout-chapter. The inherited most-units rule gave Cities of the Plain 4 units of 22,312w (the book's PARTS); the single-index holdout would have given McCarthy a Brontë-class 18k-word val reference on a 588k corpus. Both fixed, both caught by controls. → persistent-memory.d/2026-09-17-mccarthy-d1-d3.md
[2026-09-17] ⭐ lv-mccarthy D1–D3 complete on gx10, leak gate PASSED (0 of 75 renameable, 0 of 37 sub-threshold, both controls green). Three McCarthy-specific calls, each forced by a measurement: corpus-scoped rename (the Border Trilogy shares 9 surfaces across books), a new mccarthy name preset (Hemingway's carries it_IT/fr_FR and McCarthy writes neither), and --min-cap 5 to match the entity map's floor — the first gate run failed with 45 survivors purely because rename's floor was 8 and the map's was 5. → persistent-memory.d/2026-09-17-mccarthy-d1-d3.md
[2026-09-17] PARKED lv-krakauer, and the reason is a selection criterion the line was missing: ask whether the author HAS a voice before investigating whether the… → persistent-memory.d/2026-09-17-parked-lv-krakauer-and-the-reason-is-a-selection-criterion.md
[2026-09-17] The althing route-declaring SessionStart hook is documented but NOT installed on nh3-dev — dev_launch.py has zero occurrences of "route", no hook declares one, and every live route was hand-declared… → persistent-memory.d/2026-09-17-the-althing-route-declaring-sessionstart-hook-is-documented.md
[2026-09-17] Hemingway ships as-is: operator ruled "ship stands" on both measured corpus defects — the 0.96% beat contamination and the 130 non-name entity-map surfaces. → persistent-memory.d/2026-09-17-hemingway-ships-as-is-operator-ruled-ship-stands-on-both.md
[2026-09-17] PARKED lv-krakauer, and the reason is a selection criterion the line was missing: ask whether the author HAS a voice before investigating whether the… → persistent-memory.d/2026-09-17-parked-lv-krakauer-and-the-reason-is-a-selection-criterion-2.md
[2026-09-17] A unit splitter must choose by SIZE, not by count — the inherited rule silently produced 22,000-word "chapters". → persistent-memory.d/2026-09-17-a-unit-splitter-must-choose-by-size-not-by-count-the.md
[2026-09-17] lv-mccarthy D1 built — 167 units, 584,756 words — and the whole job was protecting a style that reads as damage. → persistent-memory.d/2026-09-17-lv-mccarthy-d1-built-167-units-584-756-words-and-the-whole.md
[2026-09-17] lv-krakauer D1 built — 126 units, 422,880 words — and its name guard caught three defects nothing else would have reported. → persistent-memory.d/2026-09-17-lv-krakauer-d1-built-126-units-422-880-words-and-its-name.md
[2026-09-17] triage_disposition = 'accepted' in the Kvasir catalogue does NOT mean the extraction succeeded. — Blood Meridian's epub row holds 1,167 words of a 117,000-word book, The Crossing's 222 of 150,000 — real… → persistent-memory.d/2026-09-17-triagedisposition-accepted-in-the-kvasir-catalogue-does-not.md
[2026-09-17] ⭐⭐⭐ lv-hemingway SHIPPED (ckpt850) with the line's strongest voice result — and the memorisation control it passed turned out to be the WRONG control. Voice +0.413 delta_cb at 6.4x the floor, closing 73.8% of the achievable span (lv-bronte closed 48%). ⚠ memorization_check.py uses the base-unadapted arm as its negative control, but base writes 18,035 words of summary against the adapted arms' 27,413 of pastiche — text that does not imitate the register cannot collide with its n-grams, so a 0.00 there means "different register", not "did not memorise". The right reference is the author himself: held-out Hemingway against the train split collides at 0.01 while the adapter does at 0.07, so the comfortable "his plain register makes collisions inevitable" story is FALSE and was refuted rather than assumed. All 19 matched runs were READ: stock dialogue, max 9 words, no proper noun — shorter than the 10-word run unseen Hemingway shares with the train split by coincidence. ⭐ A negative control that differs from the candidate in a way correlated with the metric is not a control. → persistent-memory.d/2026-09-17-lv-hemingway-gate.md
[2026-09-17] The v2 voice floor is now PAIRWISE, and it retroactively passes lv-bronte. — lv-bronte's ckpt475 shipped as a voice-axis FAILURE at +0.193 against a 0.251 floor contributed entirely by… → persistent-memory.d/2026-09-17-the-v2-voice-floor-is-now-pairwise-and-it-retroactively.md
[2026-09-17] The beat-contamination leak IS present in Hemingway — 70 of 7,094 train beats (0.96%), 0 of 200 val. → persistent-memory.d/2026-09-17-the-beat-contamination-leak-is-present-in-hemingway-70-of-7.md
[2026-09-17] audit_entity_map.py — the rename can DAMAGE the prose and no gate will ever say so. — Mirror of audit_stoplist.py: surfaces wrongly held IN the map rather than out of it. → persistent-memory.d/2026-09-17-auditentitymap-py-the-rename-can-damage-the-prose-and-no.md
[2026-09-17] The two-epoch recipe is now 0 for 2 and should stop being carried forward. — Hemingway's eval minimum is step 1750, but step 850 is +0.0040 against a 0.0044 median neighbour jitter … → persistent-memory.d/2026-09-17-the-two-epoch-recipe-is-now-0-for-2-and-should-stop-being.md
[2026-09-17] gitea was reaching the PUBLIC route from every repo on nh3-dev — brokkr-smithy, sleipnir, Galdrabok, kvasir — and brokkr-smithy is pushed several times a week, so the… → persistent-memory.d/2026-09-17-gitea-was-reaching-the-public-route-from-every-repo-on-nh3.md
[2026-09-17] servers/fv-ml1/ssh-target was bare 10.251.50.54, so deploy-stack.sh connected as lkraven and could not write the infra-ops-owned… → persistent-memory.d/2026-09-17-servers-fv-ml1-ssh-target-was-bare-10-251-50-54-so-deploy.md
[2026-09-17] ⭐⭐ A leak class the corpus gate structurally CANNOT see: the beat-writing model recognises the book and restores the author's real character names. 1.8% of Brontë beats named Rochester/Jane/Brocklehurst while 0 responses did. Worst for public-domain classics; Yarros and Hemingway's clean runs are NOT evidence they are immune. Patched as a sourcename reject + --source-entities. → persistent-memory.d/2026-09-17-beat-contamination-leak.md
[2026-09-17] A stoplist entry is an assertion the leak gate can no longer check — stoplisting removes a surface from the entity map, so a wrongly stoplisted CHARACTER is an undetectable leak. → persistent-memory.d/2026-09-17-a-stoplist-entry-is-an-assertion-the-leak-gate-can-no.md
[2026-09-17] ESH: Cityside Fiber failed TWICE (19:09 and ~01:06); operator switched WAN1 to DHCP to restore service and has a ticket for the static. crowdsec esh allowlist carries both failover egresses with 7-day expiries — the rotation-fragility is live. → persistent-memory.d/2026-09-17-esh-fiber-outages.md
[2026-09-17] ⭐⭐ lv-bronte SHIPPED on voices-seat (ckpt475) DESPITE failing the v2 VOICE axis — additive, reversible, safety-axis clean. Both candidates closed 48–52% of the achievable distance to Brontë but +0.193/+0.210 sit under a 0.251 noise floor set by ONE outlier seed in the arm not being shipped; cause is structural (81 val pairs vs Hemingway's 200) and not cheaply fixable. ckpt475 is the pick if it ships. The two-epoch recipe did NOT transfer. → persistent-memory.d/2026-09-17-lv-bronte-gate.md
[2026-09-16] ⭐⭐ Instruction-pair SFT BEATS raw-text for author voice, and the raw-text incumbent never cleared its own control (+0.141 against a 0.153 floor). lv-yarros shipped; the v1 decision rule was amended by the operator after measurement showed it gated on axes the unadapted carrier already maxes. → persistent-memory.d/2026-09-16-lv-voices-line.md
[2026-09-16] ⭐ voices-seat live: one carrier, N lv-<author> LoRA adapters, hot-swap measured at 0.24 s. LoRA costs 24.3% of decode against a 0.1% A-vs-A floor and is worth paying; --gpu-memory-utilization is a request against TOTAL VRAM and only a pinned KV makes it predictive. → persistent-memory.d/2026-09-16-voices-seat-lora.md
[2026-09-16] ⭐ lv-hemingway corpus gated at 994,760 words — and half the catalogue had to be EXCLUDED. 169,759 words of measured 90–96% collection duplication, a Sherwood Anderson parody, and the author's own name 95 times in publisher back matter; the gender resolver needed a corpus base-rate correction to stop reading women as men. → persistent-memory.d/2026-09-16-lv-hemingway-corpus.md
[2026-09-16] Grok token broker built then SHELVED — operator ruled "keep the jail stop the a/b", so the renewal feature has no consumer. ⛔ Do NOT arm probe-rotation: the risk did not shrink (it reaches BOTH Gróa transports through one shared session) and the payoff went to zero. → persistent-memory.d/2026-09-16-grok-broker-shelved.md
[2026-09-15] ⚠⚠ DO NOT carry "a client-side timeout is not a cancellation" as a rule — it is FALSE as stated. A clean abandon cancels itself ~6 s later (measured); yet six requests genuinely orphaned on vllm-erp-seat. Some propagate, some do not, boundary unknown — which argues for a detector, not a rule. ⭐⭐ The durable artifact: a serving engine's KV cache CYCLES, an orphaned one only CLIMBS — request count and throughput are ambiguous between loaded and wedged, and I called the seat healthy twice off them (correctly, on the evidence). ⚠ A max_tokens ceiling would NOT have prevented it: the worst offender had 16384 set, hit it, and returned 24,594 chars of whitespace. → persistent-memory.d/2026-09-15-client-abandon-cancellation-boundary.md
[2026-09-15] ⚠⚠ --gpu-memory-utilization DOES NOT PREDICT RESIDENT VRAM — measure it, never compute it. Wrong in both directions on fv-ml1: vllm-cyberprev util 0.40 (expect ~39,155 MiB) holds 47,124 (+8 GB over); vllm-gen-small util 0.48 (expect ~46,986) holds 36,942 (−10 GB under). Planning a placement off the fractions would have been 8 GB wrong. Read nvidia-smi --query-compute-apps. Full per-seat residency table + the breeze shuffle arithmetic → persistent-memory.d/2026-09-15-breeze-placement-sizing.md
[2026-09-15] breeze-tts stays on irv-ml1; the TTS-stack move to fv-ml1 is PARKED (park id 75, move-the-tts-stack-breeze-tts-bragi-tts-gateway), triggered on evacuating embed/rerank/reward. ⚠ Trigger as stated says "gpu0" but those three are on GPU 1 (~0.16 util, ~15.7 GB; GPU 1 is the tight card at 0.975 / 4,336 MiB free) — confirm which he meant before executing. All three services move together because only breeze-tts is GPU-resident (~10.3 GiB, growing) while bragi and tts-gateway are CPU proxies, and co-location is what avoids a cross-site hop per TTS call. breeze-tts sizing — original recommendation NOT to move it. ~10.3 GiB measured under load at 53 min uptime, up from 9.2 GiB shortly after warm-up (it grows; n=2, plateau unmeasured) — so GPU 0's 11,982 MiB free is a 1.7 GB margin and shrinking, on the live chat serving path. ⚠ Two measurement traps: it reports nothing at idle on the wrong card (BREEZE_GPU_DEVICES=0 = the 3090, not the A6000), and an early reading understates it. ⭐ The real objection is topology: tts-gateway is on irv-ml1 and reaches it same-box, so moving breeze alone adds a cross-site hop to every TTS call against a 478 ms first-sample budget. GPU 3 would fit it but spends the reserve. → persistent-memory.d/2026-09-15-breeze-placement-sizing.md
[2026-09-15] Parakeet STT live on fv-ml1 GPU 0, behind LiteLLM ext-stt / whisper-1. ⚠ Placed on GPU 3 first, which was wrong — operator caught it. A ~800 MiB seat should ride the card with the most uncommitted headroom (GPU 0, util 0.88, ~13 GB spare), not put the first fingerprint on the one pristine 96 GB card: vLLM sizes KV cache against TOTAL VRAM, so any tenant on an empty card eats a future full-size seat's profiling margin (flash-next needs 93 of 96 GiB). GPU 3 is now a deliberate reserve at 2 MiB. Retargeted the existing stacks/parakeet/ (sherpa-onnx + our own FastAPI wrapper) from irv-ml1; v3 int8, 25 languages. ⚠ ORT's CUDA EP compiles kernels lazily and the first decode on sm_120 took 45.7 s — every later call ~0.5 s; a startup warmup in app.py now absorbs it, so the first real request is 0.65 s instead of a 45 s hang that no client would wait through. GPU use was verified by a process on GPU 3 (922 MiB), not by the provider=cuda log line, because ORT falls back to CPU silently and still returns correct text. Silence → "" (null control), known sentence → near-exact (positive control). → persistent-memory.d/2026-09-15-parakeet-stt-fv-ml1.md
[2026-09-15] ⭐⭐⭐ THE FLEET'S CHARACTERISTIC FAILURE, named: a confident answer from a broken instrument. Nine instances in one night, every one of which PASSED A CHECK — provider=cuda while ORT ran on CPU; node --check green on a file whose SERVED script was dead; secret get returning "" with exit 0; find() turning a failed listing into an authoritative "not found"; a 401 rendering as "0 toolsets"; compat ✓ on a typo'd path; doctor exit 0 on ERROR; ss | grep python missing a listener named hermes; SIGTERM freeing a port 35 s before the process died. ⚠ The tell: whenever "broken" and "legitimately empty/absent/off" produce the same output. Remedies: measure the output not the input, positive AND true-negative controls, refuse to emit the ambiguous value, and never declare victory on a plausible fix. → persistent-memory.d/2026-09-15-silent-wrong-answer-pattern.md
[2026-09-15] secret get returned EMPTY with exit 0 under concurrency — (svos-dev found it; 0/4 succeeded here). → persistent-memory.d/2026-09-15-secret-get-returned-empty-with-exit-0-under-concurrency.md
[2026-09-15] ⭐⭐ A check that reads an artifact AS STORED cannot see a transformation between storage and execution — named twice in one night and it generalises. node --check on a source file passes while the SERVED page's inline script is dead (a JS 'didn\'t' inside a Python string arrives as 'didn't' and closes it); provider=cuda in a log echoes configured intent while ORT silently ran on CPU. Both check the INPUT to a transformation and get reported as checks of its OUTPUT. Remedy: gate the wire, not the file — tts-stack tools/gate_served_page.py. ⚠ My first version had a gap tts-dev closed: a worklet inside a template literal is just a string to a parse of the enclosing script, so its syntax error surfaces as a rejected addModule promise and silent degradation. I checked the instance, not the class. → persistent-memory.d/2026-09-15-talk-v10-deploy.md
[2026-09-15] ⚠⚠ The talk-deploy "permission problem" NEVER EXISTED — and I built a fix for it anyway. /opt/docker/compose on nh3-dev is root:docker 2775, sessions run as lkraven, lkraven is in docker; a mkdir settles it in one second and nobody ran one for nine days. There is no tts-dev OS account at all. It held because a stale memory row supplied a mechanism, the operator's routing instruction ("give it to infra") was misread as corroboration of a capability limit — different claims, only one ever stated — and I repeated it to the operator as fact. Then, told to fix "the harness issue", I inferred an auto-mode classifier refusal and committed a settings.json to tts-dev's repo on that inference; their mkdir disproved it and I reverted. ⭐ "I can't do X" is a hypothesis until someone pastes the error. ⚠ That commit also overclaimed a doc fix that failed — never chain an edit and its commit in one invocation. → persistent-memory.d/2026-09-15-silent-wrong-answer-pattern.md
[2026-09-15] talk v10 LIVE on nh3-dev :8092 — the fleet speaks and listens on one page. — First consumer of the ext-stt Parakeet seat: POST /api/listen, push-to-talk, barge-in. → persistent-memory.d/2026-09-15-talk-v10-live-on-nh3-dev-8092-the-fleet-speaks-and-listens.md
[2026-09-15] Two restart patterns from svos-dev worth stealing: (a) DRY-RUN BOOT against the still-held port — start the new process while the old one holds the socket; it proves every check above the bind and dies on… → persistent-memory.d/2026-09-15-two-restart-patterns-from-svos-dev-worth-stealing-a-dry-run.md
[2026-09-15] ⭐ svos_miranda ENABLED and LIVE in Hermes — but agent.disabled_toolsets is permanently OFF by operator ruling ("i dont want the tools disabled everywhere"). That key is a global end-of-pipeline subtraction, not api_server-scoped: measured 46 tools → 20 on a default session. It is also unnecessary — platform_toolsets.api_server: [svos_miranda] alone resolves an api_server session to exactly the 8 tools, write-klass absent. Gateway restarted 02:10 (PID 3107822→3901622, observed); /v1/toolsets now 29 rows incl. svos_miranda; operator's own surface verified intact at 46. ⚠ SVOS must stop verifying against the GLOBAL roster before it restarts — it will see 29 and refuse, by design now. → persistent-memory.d/2026-09-15-svos-miranda-plugin-validation.md
[2026-09-15] irv-ml1 parakeet RETIRED; voice-studio STOPPED. — Both operator rulings. → persistent-memory.d/2026-09-15-irv-ml1-parakeet-retired-voice-studio-stopped.md
[2026-09-15] svos_miranda Hermes plugin validated; found its load blocker. Absolute intra-package imports (from hermes_plugin.x) could not resolve at the documented install name — fixed by svos-dev at c964e64. ⚠ hermes plugins validate and doctor can NEVER pass this plugin, by construction: validate's probe stub is config-blind AND returns None from register_tool (which the plugin's guard reads as a collision), and doctor runs under a temp HERMES_HOME with no config. ⚠ doctor exits 0 on ERROR (use --ci); compat reads a nonexistent path as a pass. Roster verified 8/7 by a probe supplying real settings. → persistent-memory.d/2026-09-15-svos-miranda-plugin-validation.md
[2026-09-15] ana-docker resolves NO .internal names — its /etc/resolv.conf is 1.1.1.1/1.0.0.1, not the fleet AdGuard. → persistent-memory.d/2026-09-15-ana-docker-resolves-no-internal-names.md
[2026-09-15] ⚠⚠ irv-ml1 still points at the retired wg0 lifeline 10.100.79.3 in 96 places — and one is a LIVE breakage, not a dead link. voice-studio cannot reach studio-gate (both up, separate docker networks, gate URL is the dead IP) and has been failing since the 2026-09-06 cutover with nothing alerting. 8 running containers carry dead homepage.href labels; waterland-studio's siteMonitor too. ✅ tts-gateway/ext-tts verified UNAFFECTED. Not fixed — wants a scheduled pass, not a 02:00 improvisation. ⭐ Third instance of the same shape: a retired address needs a grep by ADDRESS, not by hostname, and labels live in no file until the container is recreated. → persistent-memory.d/2026-09-15-irv-ml1-dead-wg0-address.md
[2026-09-15] Parakeet bench settled by tts-dev — FV wins at both clip lengths and beats the incumbent Whisper; IRV seat is now retirable. → persistent-memory.d/2026-09-15-parakeet-bench-settled-by-tts-dev-fv-wins-at-both-clip.md
[2026-09-15] Mesh membership retired for fv-ml1 and nh3-dev — six nodes left, each with a job. fv-ml1 gets break-glass rejoin instead of standing membership; nh3-dev's retirement also removed the nh3-scale masquerade exception it had required. Exactly one live reusable pre-auth key remains fleet-wide. → persistent-memory.d/2026-09-15-fv-mesh-watchdog.md
[2026-09-15] FV cross-site routing fixed — one OPNsense outbound-NAT rule had been scoped to Anaheim only. fv-ml1 now reaches NH3/ESH/IRV/ANA/mesh/internet; four rules, all src=10.251.50.0/24. The diagnostic signature is the valuable part: every layer looks correct and the discriminator is that every other site pair works. → persistent-memory.d/2026-09-15-fv-cross-site-snat.md
[2026-09-15] Break-glass mesh path on fv-ml1 — inverted from a restore-watchdog on the operator's suggestion: the box is OFF the mesh and the watchdog JOINS it on fleet loss. Exposed a rejoin key expiring in 4 days; replaced with a dedicated 1-year key and the two stale reusable keys retired. → persistent-memory.d/2026-09-15-fv-mesh-watchdog.md
[2026-09-15] Fleet identity/group/path conventions pinned + docker trees → root:docker 2775 setgid on 5 hosts. svc-* in 800-849, infra-ops 850, docker 851, vh for new hosts with no retro-renames; 0777 cleared; linus deleted; llmuser de-privileged. → persistent-memory.d/2026-09-15-fleet-identity-conventions.md
[2026-09-15] nh3-dev unreachable from the mesh at its LAN address — Tailscale's ts-input anti-spoof, not DNS. Fixed with a masquerade exception on nh3-scale. ⚠ Do NOT instead advertise the /32 from nh3-dev; that black-holes it from every other site while its own LAN keeps working. → persistent-memory.d/2026-09-15-nh3-dev-ts-input-masquerade.md
[2026-09-15] ESPHome pinned to 2026.8.2 + kb KB-search tool shipped. Untagged image had drifted a year; config relocated into restic with 539 MB of regenerable cache excluded; remote-build disabled (⚠ two switches, only one closes the port). kb exists because Worldtree's /search searches messages, not notes, and returns a clean empty result for a note that exists. → persistent-memory.d/2026-09-15-esphome-and-kb.md
[2026-09-15] Hermes bearer rotation hold released — svos-dev split their HS256 signing key off the shared value (svos 7165272) → persistent-memory.d/2026-09-15-hermes-bearer-rotation-hold-released-svos-dev-split-their.md
[2026-09-14] fv-ml1 rebalance: cyberprev→sec (mog-sec retired), NEW gen-small A3B seat, all sec/gen/char at native 262K in-band, coder reclaimed, seat catalog + bench shipped. cyberprev = hotdogs cyber-SFT (name-repaired past a tripled-prefix unsloth export bug, house NVFP4 quant); gen-small = llmfan46 Qwen3.6-35B-A3B Heretic (already on disk), MTP 69.6%. Serial depth-tested all seats clean (0 OOM); warm tok/s 62.7-337.3. Commits 1418edb→dfa91a8. → persistent-memory.d/2026-09-14-fv-seat-rebalance-gen-small.md
[2026-09-14] fv-ml1 all-night seat reorg — MTP k=3 on gen-large (+52%@conc1), gen consolidated onto flash-next (27B dense retired, 38 GB freed), char-rp restored to MeroMero-v2-31B, Sentinel-R3 served + dflash cutover (beat MTP 2.40 vs 2.18). ✅ gen-large RESOLVED 2026-09-14 — orcarouter serving: PLE bf16→FP8 convert + ple_embedding_dtype + layer_types rename; NO source build needed. → persistent-memory.d/2026-09-14-fv-seat-reorg-and-orca-blocker.md
[2026-09-13] STANDING POLICY (operator): cap GPU power limits at BUILD time, not after discovering the constraint. → persistent-memory.d/2026-09-13-standing-policy-operator-cap-gpu-power.md
[2026-09-13] **FV SITE DARK — every Fountain Valley address including the BMC went unreachable ~2.5 min into a two-card ** → persistent-memory.d/2026-09-13-fv-site-dark-every-fountain-valley.md
[2026-09-13] ⭐⭐ Qwen3.8-Flash-Next serving on ONE card with its 51B n-gram table in host RAM — the first seat whose weights do not fit its GPU. stacks/flash-next-seat/, fv-ml1 GPU 2 :8022, plus a gen-large LiteLLM alias. Measured: 74.36 GiB weights resident, 14.00 GiB KV = 560,654 tokens at the full 262,144 context, 67 GiB host RSS, 75.5/212.3/387.8 tok/s at conc 1/4/8 (⚠ n=1). ⭐ The offload is vLLM #54371 (UVA, merged 2026-09-09) which supersedes the paused #53899 — it has no worker process, so #53899's whole bug family (TP=1 deadlock #53960, pidfd_getfd/ptrace gate, stale-output-under-graphs) is designed out; in v0.29.1rc0, not v0.29.0. ⚠ text_config.ple_embedding_dtype is the load-or-fail discriminator for any community build. ⚠⚠ --kv-cache-memory makes vLLM SKIP MEMORY PROFILING and ignore --gpu-memory-utilization — 16 GiB nearly OOM'd on a 155K prefill with no visible failure; 14 GiB is the measured-safe value and vLLM's own "17.46 GiB to fully utilize" is 3.5 GiB too high. ⚠ MTP is off pending measurement here, not written off — the recipe's number is cross-harness and tested k=3 only, while the head is ONE layer run autoregressively, so k=1 is unpublished and may win (services/flash-next-mtp-bench/, one off_A rep banked before the outage). ⚠ A container once ran (healthy) with PORTS=[] — verify docker port, not the healthcheck. → persistent-memory.d/2026-09-13-flash-next-seat-and-fv-outage.md
[2026-09-13] Finished the ana-ml2→fv-ml1 renumber the cutover missed: 16 live Homepage entries pointed at the dead 10.250.50.54 and zero at the live IP. → persistent-memory.d/2026-09-13-finished-the-ana-ml2fv-ml1-renumber-the-cutover-missed-16.md
[2026-09-13] FV→ANA fixed, Beszel18/18 up: scoped OPNsense hybrid NAT for fv-ml1→ANA; prior NAT/filter rules preserved and rollback-guarded. → persistent-memory.d/2026-09-13-fv-to-ana-nat.md
[2026-09-11] Worldtree memory-split (U6) — PROTOCOL AGREED with worldtree-dev: nobody flips memory.reader.enabled or m → persistent-memory.d/2026-09-11-worldtree-memory-split-u6-protocol-agreed.md
[2026-09-03] Run 3c is STAGED on pfi-gx10 and deliberately NOT launched — the launch is a 13.3 h commitment and the operator stood this port down once already. Base shards AND the encoded corpus sha256-verified identical to ana-ml2's, so the transformers 5.15.1→5.16.1 / x86-64→aarch64 delta is measured inert. ⚠ The encode-cache FILENAME differs by design (base_model_path is in the key) — input hash, not output. ⚠ Tripped the pkill -f ssh self-match again; the launcher guards on a pidfile because of it. → persistent-memory.d/2026-09-03-gx10-run3c-staged.md
[2026-08-25] Fused MoE kernel path — DEFERRED, tracked at park fused-moe-kernel-path-for-gemma-4-moe-training (id 47). → persistent-memory.d/2026-08-25-fused-moe-kernel-path-deferred-tracked-at-park-fused-moe.md
[2026-08-24] nconnect=8 on /mnt/smithy — approved but DEFERRED at operator instruction. brokkr-smithy-dev pre-approved it for "once the FortiGate work settles" and does not need re-asking; the operator declined it in this session's scope. Tracked at althing thread 01M0R46SFYF83099N16WD67KGD.
[2026-08-19] AI-tab Dormant regrouping BELAYED by the operator — six seats (char-rp Magidonia, char-rp-reasoning Heretic2, Granite summarizer, Qwen-Image-Bench, Skaldsong… → persistent-memory.d/2026-08-19-ai-tab-dormant-regrouping-belayed-by-the-operator.md

106 older entries archived to archival-memory.md.

Tried and abandoned

[2026-09-26] Greedy exact-match as parity for a small generative seat (coder) — same-host repeats matched only 30–38% (run B hit the prefix cache, so a different numeric path; near-tie… → persistent-memory.d/2026-09-26-greedy-exact-match-as-parity-for-a-small-generative-seat.md
[2026-09-26] zsh traps in ad-hoc test loops: set -- $a does NOT word-split, so alert POSTs went out with empty fields; and local path=… inside a function CLOBBERS $PATH (zsh's tied array), so the failover test ran nothing while TEI sat stopped. Use other names and ${=var}.
[2026-09-25] Concluding "not on the UDM" from a port table read 15 s after link-up — UniFi polls (~60 s); AMT was on port 6 as Prime said. Wait a poll, and prefer a which-port-lists-the-MAC check. auto-memory feedback_polled_stats_lag_the_event.
[2026-09-25] The esh-pve NVIDIA DKMS recipe on nh3-pve — it built, then the kernel refused to load it: nh3-pve has SECURE BOOT ON (esh-pve, the same MS-01, has it off). The installer rolled back. The playbook now pre-flights mokutil --sb-state (7ddd116).
[2026-09-25] elway against an unpinned host-key name — every when: hit ssh rc 255, so all 9 steps showed SKIPPED, not failed. Only the verify phase went red. Pin the name (the IP key matched) or use the IP. Untracked elway defect: rc 255 in when: should fail.
[2026-09-24] Testing "Restore AC Power Loss" with an OS shutdown — a shutdown stays off BY DESIGN; only pulling and restoring AC tests it. A community README claimed the two are indistinguishable; trusting it cost Prime two trips to the pfi-gx10 power button.
[2026-09-23] booth link --help — there is no help flag; it posts --help to the operator's link board as a link. Read booth with no args for usage.
[2026-09-21] Using directory mtime as a liveness test when pruning session scratchpads — find /tmp/claude-1000 -mindepth 2 -maxdepth 2 -type d -mtime +7 deleted an ACTIVE session's working… → persistent-memory.d/2026-09-21-using-directory-mtime-as-a-liveness-test-when-pruning.md
[2026-09-18] Routing SearXNG's egress through a SOCKS5 proxy on esh-scale — one day live, reverted. It fixed nothing, and the reason I gave for reverting it was itself wrong: the rollback was the counterfactual and it falsified my own published claim. Kept reverted on its own merits (no measurable gain, added a hard ESH dependency for all fleet search). → persistent-memory.d/2026-09-18-searxng-esh-egress-reverted.md
[2026-09-18] api_key: !ENV SEARXNG_BRAVE_API_KEY in searxng settings — this build has NO !ENV YAML constructor, so the file was unparseable and the container crash-looped ten… → persistent-memory.d/2026-09-18-apikey-env-searxngbraveapikey-in-searxng-settings.md
[2026-09-15] ⚠⚠ Probing OPNsense API endpoints by POSTing at them — one was /api/core/system/reboot and it took the FV site dark for 3.5 min. Endpoints are ACTIONS; a 200 means it ran. The call I wanted was documented in this repo's own docs/pfi/opnsense-api-reference.md. → persistent-memory.d/2026-09-15-opnsense-api-reboot.md
[2026-09-15] Advertising 10.100.10.50/32 from nh3-dev to make its LAN address mesh-reachable — black-holed it from ESH/ANA/FV/IRV while its own LAN and the internet kept working, so a one-host check passes cleanly. lookup 52 at rule priority 5270 beats main at 32766. Fix belongs at the router. → persistent-memory.d/2026-09-15-nh3-dev-ts-input-masquerade.md
[2026-09-15] Remote-site MASQUERADE rules on nh3-scale for the asymmetric-return theory — they fired (counters incremented) but were not the fix. Reverted rather than left to accumulate as NAT achieving nothing.

115 older entries archived to archival-memory.md.

74 KiB Raw Blame History Unescape Escape