memory: snapshot — 2026-10-08 patch night, nh3-svos a737a06 + post office live, albok 0.1.13/0.1.14, gateway glm-5.3, heid sweep incident; 16 entries archived

archival-memory.md

@@ -14735,3 +14735,275 @@ longer takes a full-card seat. That is the trade; it is the operator's call, and
 gateway should move with it either way.
   _Archived 2026-10-06._
 
+## Recent decisions (archived 2026-10-08 batch)
+
+# `[2026-09-24]` esh-ml1 built: RTX 2000E Ada serving embed + rerank, LiteLLM order-2 failover behind fv-ml1
+
+Executes Prime's 2026-09-24 decision (LXC + host driver, not a VFIO VM). Done
+2204–2240 PT the same night, **with no esh-pve reboot**.
+
+**Host (esh-pve).** NVIDIA **580.178.04** open modules via DKMS from NVIDIA's
+`-no-compat32.run` (sha256 checked against NVIDIA's published sum), applied
+live by `playbooks/pve-nvidia-host.yaml`: nouveau was never loaded and the
+T400's vfio ids matched nothing, so nothing held the card. Our own
+`nvidia-persistenced.service` creates `/dev/nvidia{0,ctl,-uvm,-uvm-tools}`
+**before `pve-guests`** (else `pct start` refuses the `devN:` entries). Retired
+`vfio.conf` (moved to `/root/nvidia/`) and `blacklist nvidia`. Initramfs NOT
+rebuilt on purpose (boot path untouched). 580 because the fleet's vLLM
+`v0.24.0` image is CUDA 13.0 (driver ≥ 580); fv-ml1 runs 580.65.06. `.run`
+not an apt repo: one file on both sides makes the host-module/container-lib
+version match true by construction; apt would let an upgrade move one side.
+
+**CT 110 `esh-ml1`** (`playbooks/gpu-lxc.yaml`): unprivileged Debian 12,
+nesting+keyctl, 6c/16G/80G `local-lvm`, `10.0.50.80` VLAN 50 (static; UDM pool
+is `.150–.250`), `startup order=30` after esh-scale/VMs. NVIDIA userspace from
+the same `.run` with `--no-kernel-modules`; docker-ce; nvidia-container-toolkit
+with `no-cgroups=true`. **Docker-in-unprivileged-LXC with the GPU worked first
+try on lxc-pve 6.0.0-2** (no AppArmor workaround needed). Fleet ids per
+`docs/pfi/fleet-conventions.md` (infra-ops 850, docker 851, vh 1000 — vh has
+no key/password yet). **Not in the vzdump job, on purpose** (rebuildable).
+DNS `esh-ml1.esh.internal` synced to all three resolvers.
+
+**Stack `stacks/embed-rerank`:** `vllm-embed` :8001 + `vllm-rerank-bge` :8013,
+same models/flags/vLLM digest (`251eba5cc7c1`) as fv-ml1, 0.20 mem-util each →
+4,823 MiB used of 16,380.
+
+**Parity (the load-bearing measurement; embeddings are model-specific).** 11
+texts × 2 runs per site. Embed cosine FV-vs-ESH median 0.999908 / min 0.999772,
+inside both self-noise floors (FV-vs-FV 0.999927 / 0.999791; ESH-vs-ESH
+0.999911 / 0.999809). Negative control (different texts) 0.07–0.38. Sensitivity
+floor ~2×10⁻⁴ cosine. Rerank |score| max 0.000145 vs FV-FV floor 0.000181,
+identical ranking.
+
+**Gateway.** `qwen3-embedding` and `reranker` each got a second deployment
+(`order: 2`, esh-ml1); fv-ml1 is `order: 1`. LiteLLM v1.97 order-fallback
+proven with throwaway groups (created + deleted): primary refusing → ESH in
++0.15 s (3/3); primary host-down (no ARP) → ESH but **~18.7 s on every call,
+no cooldown** (3/3); dead-only negative control → 500 (3/3). Post-restart
+production traffic is served by fv-ml1 (header `x-litellm-model-api-base`).
+`/health?model=` returned 503 with empty lists for both groups — not
+investigated. Gateway restart: liveliness back in ~52 s.
+
+**Found + fixed:** DB-only alias `reranker-a3-bge-v2-m3` pointed at ana-ml2's
+pre-relocation IP `10.250.50.54` — dead since 2026-09-12 (500 after ~23 s), no
+callers in 7-day spend logs. PATCHed to `10.251.50.54:8013`. Sweep of
+`/model/info` found no other `10.250.50.54` targets.
+
+**Who actually uses these names (spend logs, 7 days to 2026-09-25 05:35Z):**
+`qwen3-embedding` — worldtree-gateway 481 (nh3-dev + corviduo-dev), nevermore
+20; `reranker` — nevermore 13. **No ESH-side consumer called either in 7 days**
+(Open WebUI's RAG embeds only on document upload). So today esh-ml1's value is
+an FV-outage failover for Worldtree + nevermore, not local service for ESH.
+
+**Open:** a direct ESH→esh-ml1 path (survives a mesh outage) would help only
+ESH consumers, which are idle — recommended parked. esh-ml1 not yet in
+Homepage docker.yaml or Beszel. The ~18.7 s host-down failover penalty is
+untuned (router retries/cooldown).
+
+**Speed A/B vs fv-ml1 (2026-09-25 0628–0640 PT, table in `servers/esh-ml1/README.md`).**
+Single queries: a wash (on-box ESH ~9 ms vs FV ~12 ms; from ana-docker ESH is
+faster because it is network-closer). Bulk: ESH 3–10× slower (bulk embed ~50 vs
+~403 passages/s; rerank-20 ~170 vs ~51 ms p50, ~5.7 vs ~55 req/s at conc 8).
+Noise floor 3–5% on throughput. ⇒ failover role confirmed; load-sharing would
+roughly double rerank latency for half the calls.
+
+**TEI bake-off (2026-09-25, `stacks/tei-bakeoff/README.md`).** Prime asked to
+evaluate a purpose-built engine instead of vLLM → HF Text Embeddings Inference
+1.9.4 (Infinity ruled out: last release 2025-08-22, dropped by us in May for no
+Qwen3). Verdict: TEI **matches** (embed cosine vs vLLM-FV median 0.999925, inside
+noise; TEI queries against the OLD vLLM index overlap@10 0.988 vs 0.986 self-noise
+→ no re-embed needed; rerank decisions identical, tail scores move ≤0.019) but is
+**not faster** on the Ada (novel 40 s vs 31 s; tiny queries ~25% faster). It is
+**much lighter**: ~2.6 GB VRAM for both vs vLLM's 4.8 GB budget, 8 GB image vs 30,
+~4 s restart vs ~24 s. Gateway: embed drop-in via `hosted_vllm/`; rerank needs the
+`huggingface/` provider. Both engines left running side by side pending Prime's call.
+  _Archived 2026-10-08._
+
+# `[2026-09-24]` esh-pve: VM 102 retired, a 14-day hung VFIO process found, T400 → RTX 2000 Ada
+
+**VM 102 `esh-vm-workstation` retired permanently** (Prime: "not the machine I
+want running a virtual windows machine"). Final backup
+`pbs-ana:backup/vm/102/2026-09-10T10:32:20Z` (verification ok) set
+**protected** so prune cannot take it; `qm destroy 102 --purge`.
+
+⚠⚠ **Found: `kvm -id 102` had been hung in D state since the 2026-09-10 0332
+vzdump.** vzdump starts a stopped VM (here with the T400 passed through,
+`x-vga=1`) to read its disks; that process never exited. It held **16 GB of
+locked RAM**, the GPU, and two LVs, while `qm status 102` said `stopped` the
+whole time. Same class as the 2026-08-19 hard freeze: VFIO + this box. Only a
+reboot cleared it.
+
+**GPU swap, Prime on site, 2119–2137.** Guests shut down cleanly; host powered
+off; the T400 was removed and an **NVIDIA RTX 2000 / 2000E Ada** installed
+(`01:00.0`, `10de:28b0`, subsystem `10de:1871`). Host back 2136:49; leftover
+LVs removed; all guests auto-started; esh-docker-vm NFS, traefik (0×403) and
+AdGuard verified. The new card has **no driver bound**; `/etc/modprobe.d` still
+lists the T400's vfio ids (`10de:1ff2,10de:10fa`), now matching nothing.
+Commits `719e3fa`, `7cbd4c3`.
+
+**Decision (Prime, 2026-09-24): the card serves embedding + reranking as an
+LXC with the NVIDIA driver on the host — NOT a VFIO VM.** Rationale: both
+esh-pve hangs this year came from VFIO passthrough; LXC avoids it and pins no
+VM RAM. Accepted cost: a driver on the hypervisor, rebuilt on PVE kernel
+updates, host/container versions must match. **Implementation deferred to the
+next session** (tracking: this entry + `servers/esh-pve/README.md`).
+
+**ESH single-route observation:** esh-scale (CT 108, ESH's only mesh subnet
+router) lives on esh-pve, so while esh-pve was down *all* of ESH looked dark
+from outside, including esh-pve-nas and vm-esh-nas, which were up. Untracked by
+operator choice so far — a second ESH route is an idea, not a decision.
+
+**vPro:** the copper cable is in the AMT-capable port (**I226-LM**, `enp89s0`,
+MAC `58:47:ca:76:99:32`), verified at 2500 Mb/s; the I226-V (`enp88s0`) is
+not AMT-capable. Both links set back admin-down on the host. **MEBx
+provisioning deferred by Prime** ("some other time"): Ctrl+P at boot, enable
+manageability, static IP, KVM on with User Opt-in None, activate network
+access; MeshCentral should live on ana-docker/nh3-docker, never on esh-pve.
+  _Archived 2026-10-08._
+
+# `[2026-09-24]` pfi-gx10 "power on after AC loss" — UEFI patch applied, NOT validated
+
+pfi-gx10 (ASUS Ascent GX10, BIOS `GX10DGX.0105.2026.0505.1153`, no BMC) stayed
+off after the NH3 outage. With Prime's approval (after the permission
+classifier first refused it as externally-sourced firmware code), bytes 3–4
+of the 81-byte UEFI var `Setup-ec87d643-eba4-4bb5-a1e5-3f3e36b20da9` were
+changed `0x1d,0x00 → 0x1c,0x01` via `efivar -w`; readback shows exactly those
+two bytes changed. Recipe: community tool `Fheodoor01/DGX-Spark---Power-on-boot-`
+(diff-derived from a power-on unit vs a stay-off unit; author-verified on
+GX10DGX.0103/.0104). **Byte 3's meaning is unknown**; the resulting pair
+matches factory power-on units.
+
+**Original saved twice:** `/var/lib/efi-setup-backup/Setup-ec87d643-20260924-pre-poweron.bin`
+on the box and `servers/pfi-gx10/efi-Setup-ec87d643-20260924-pre-poweron.bin`
+(sha256 `6d6f6f77…`, raw incl. the 4-byte attr header). Revert procedure in
+`servers/pfi-gx10/README.md`. Commit `e5197a3`.
+
+⚠⚠ **An OS shutdown is NOT a test of this setting.** It means "start when
+power RETURNS after a cut"; a `poweroff` is supposed to stay off by design.
+The tool's README claims the two are indistinguishable — false here. I
+trusted it and ran two poweroff "tests" (1421, and 1435 after the reboot the
+README says the firmware needs); both stayed off, each costing Prime a trip to
+the button, and both prove nothing. Prime called it out directly.
+
+**Status:** the box is OFF (the 1435 shutdown) with the patch in place, and
+Prime left NH3. **He will pull and restore AC on 2026-09-25.** Comes up alone
+→ works; drop "NOT YET VALIDATED" from the README. Stays off → he presses the
+button, revert from the backup, and set it in firmware via a PiKVM (he intends
+to add one with ATX control, which also covers remote power-on regardless).
+
+**Validated 2026-09-25 (Prime, relayed by Miranda 1333):** Prime replugged AC and
+the box came up by itself at 1108:54 PT, with no button press. The patch works.
+  _Archived 2026-10-08._
+
+# `[2026-09-24]` NH3 power outage — recovered; three latent gaps closed, one needs a hand
+
+Power at NH3 dropped and returned; every NH3 box booted at 1233–1234 PT. The
+core recovered unaided (post office, headscale, egress proxy, DNS, WAN IP
+unchanged at `70.230.226.88`). Three things did not, each a latent config gap
+that only a site-wide power event exposes:
+
+1. **pbs-nh3 (VM 105) had no `onboot` flag** and stayed down. Set `onboot=1`
+   and started it; datastore NFS mounted (its line uses `bg`). `qm guest exec
+   105` works (guest agent on) — the root path, since infra-ops is not
+   provisioned there.
+2. **NFS boot race.** nh3-nas is the slowest box to serve NFS, so every plain
+   fstab NFS line failed at boot (nh3-docker `/mnt/compose` `/mnt/backup`,
+   nh3-dev `/mnt/backup`). nh3-dev's `/mnt/smithy` already had
+   `x-systemd.automount` and self-healed on its next access (1234 failed, 1309
+   mounted, no hand) — the existence proof. `playbooks/nh3-nfs-automount.yaml`
+   brought the rest to that shape (`_netdev,nofail,x-systemd.automount,
+   x-systemd.mount-timeout=30`), commit `1cbde50`. ⚠ nh3-dev's installer
+   cdrom fstab line is a pre-existing `findmnt --verify` error; the playbook
+   judges only errors a rewrite ADDS, after its first run aborted on it.
+3. **pfi-gx10 (bare metal, no BMC) did not power on** — ASUS ships "Restore AC
+   Power Loss" = stay off. See `[[2026-09-24-gx10-ac-restore-patch]]`.
+
+Also: **every Claude session on nh3-dev died** with the reboot except
+infra-ops, infra-hermes, jekyll. Relaunch is Prime's call (dev-launch).
+`nh3-laser` (VM 104) is on-demand and **stays off** by Prime's ruling;
+`servers/nh3-pve/README.md` now carries an expected-after-power-loss column per
+guest so the next triage does not flag it.
+
+The headscale-ddns failures 1141–1202 were the WAN being down, correctly
+refused (no public v4); every run since 1247 reads `unchanged`.
+  _Archived 2026-10-08._
+
+# `[2026-09-24]` Miranda standing order is a repo CLAUDE.md operating parameter
+
+**Miranda standing order is a repo CLAUDE.md operating parameter** (`4b29492`, aligned to the global send protocol in `bcf3342`): high-urgency matters go to her, fixed or not; URGENT only when it cannot wait (she phones Prime). Channel verified end to end (thread `01M3A0RP4Q8T0KNGH8TMFSNDA6`); it depends on svos + hermes-gateway. Prime's callsign **PRiMe / papa romeo mike** is a name, not an authenticator (`617b759`, `62817a2`).
+  _Archived 2026-10-08._
+
+- `[2026-09-24]` **task-board mothballed** (Prime): container removed on ana-docker, data/image/compose kept, Kuma monitor deleted, task_* instructions removed from CLAUDE.md and the fork template (`e6da607`). Its hooks had sent no traffic in 30 days.
+  _Archived 2026-10-08._
+
+- `[2026-09-24]` **Military 24-hour Pacific clock times** carried into the Codex/Grok shared bootstrap `docs/fleettools/AGENT-BOOTSTRAP.md` (`ad2b4d9`); Claude seats get it from the global CLAUDE.md.
+  _Archived 2026-10-08._
+
+- `[2026-09-24]` **A git checkout under `root:docker` needs `safe.directory` for its deploy user** — the 09-14 normalization (`826a63b`) silently broke yt-voice-clipper's webhook deploy until v0.3.13; fixed on irv-ml1 and recorded in fleet conventions (`eea9eb2`). Sweep found no other case.
+  _Archived 2026-10-08._
+
+# `[2026-09-23]` elway sudo uploads land root:root; fleet ownership audit; 33 files fixed
+
+elway's sudo upload was scp-as-user then `sudo mv`, and mv keeps the owner, so
+every file it installed as root (systemd units, `/etc` configs, root-run
+scripts) landed owned by infra-ops or lkraven. A sudoers drop-in installed that
+way would break sudo outright. Commit `466f7aa`:
+
+- sudo uploads chown to `root:root` by default; `upload.owner:` / `--owner`
+  override; owner refused on non-sudo uploads.
+- chown+chmod act on the STAGED file, then one `mv` publishes it, so a failed
+  chown cannot leave the live path mis-owned. A trap removes the staged file on
+  every exit. A directory dest is refused before anything moves.
+- `mode` was spliced unquoted into the remote root shell; now octal-validated
+  and quoted. `mode`/`owner` must be quoted YAML strings (bare `0644` → 420).
+- `preflight()` resolves every step before any remote action.
+- 24 unit tests; heid bug-hunt "Puck" (Gróa + seat) folded.
+
+`scripts/fleet-ownership-audit.sh` (read-only; exit 0/4/5): tier A root-parsed
+paths (symlinks judged by target), tier X root-run unit Exec paths incl.
+drop-ins, tier B /opt summary. Unprivileged or partial hosts report INCOMPLETE,
+never clean; completion is nonce-marked. Positive controls on nh3-dev fired for
+each tier. **Six hosts have no infra-ops identity** (ana-wg, nh3-nas, pbs-ana,
+pbs-nh3, pfi-postgres, vm-esh-nas) and audit unprivileged.
+
+**33 files chowned to root on 10 hosts** with Prime's approval (the first
+attempt was blocked by the permission classifier): 31 elway-placed (beszel
+agent trio on 6 hosts, esh-vm-db/ana-docker restic hooks and retry drop-ins,
+irv-ml1 units, nh3-dev alert bridge, fv-ml1 retired file) plus 2 non-elway
+root-exec holes (ana-docker `/usr/local/bin/resticprofile` owned by llmuser;
+esh-pve-nas `/usr/local/lib/libigdgmm.so.11.3.1343` uid 1000). Re-audit: 0 of
+the 33 remain; all affected services still active. Left as low-risk and
+reported: ana-docker `/root/.nvm` (lkraven), irv-ml1 `uv`/`uvx`, fv-ml1
+`btop` and a root uv-cache python.
+  _Archived 2026-10-08._
+
+# `[2026-09-23]` headscale-ddns hardened
+
+**headscale-ddns hardened** (`fedd4b6`): Cloudflare calls retry and validate every body (`pick()`), no write without both IDs, the run ends on a confirmation, only a global v4 is published, `curl -q`. Three of four weekly failures were empty zone lookups. heid bug-hunt "Talus" folded (one finding was my own regression).
+  _Archived 2026-10-08._
+
+# `[2026-09-23]` esh-docker-vm restic was skipped 09-22..23 by my own Kuma move
+
+**esh-docker-vm restic was skipped 09-22..23 by my own Kuma move** — a dead uptime-kuma lookup aborted `pre-backup.sh` under `set -e` (`25e41d2`). Then, by Prime's decision, the redundant Paperless pg_dump went too: it had failed auth every night since 2026-04-24 and left a 0-byte dump in every snapshot; the DB is covered at source by esh-vm-db's pg_dumpall (`6e8da46`).
+  _Archived 2026-10-08._
+
+# `[2026-09-23]` hermes-gateway restart exit-1 is a Hermes race, not a crash
+
+**hermes-gateway restart exit-1 is a Hermes race, not a crash** — the planned-stop watcher consumes the marker before SIGTERM re-runs the handler; drop-in `SuccessExitStatus=1` stops the false OnFailure page at zero coverage cost (`c2b0a05`). svos had its own stop-timeout (an open board SSE tab), fixed by svos-dev with `timeout_graceful_shutdown=5`.
+  _Archived 2026-10-08._
+
+- `[2026-09-23]` **Booth link board cleared to 14 durable links** (208 removed: drops, release posts, research links, token-bearing URLs; Scriberr reposted at `scriberr.fv.internal:8080`). Prime's rule: durable debug links only.
+  _Archived 2026-10-08._
+
+# `[2026-09-22]` safe-rm installed on nh3-dev and delegated fleet-wide to infra-hermes.
+
+⭐ **safe-rm installed on nh3-dev and delegated fleet-wide to infra-hermes.** ⚠ The package installs INERT and looks fine — Debian's `/etc/zsh/zprofile` has 0 non-comment lines so the shipped `profile.d` hook never fires under zsh; verify with `command -v rm`, never `dpkg -l`. ⚠ And it does NOT cover the habit that prompted it: measured, it refuses `rm -rf /home` and deletes an unset-variable path without complaint. Blacklist, not heuristic; `set -u` is the actual cover. Wiring ruled: `/etc/bash.bashrc` above the `case $-` guard, not `/etc/environment` (an rc self-guards, smaller blast radius, covers bash-over-ssh).
+  _Archived 2026-10-08._
+
+## Tried and abandoned (archived 2026-10-08 batch)
+
+- `[2026-09-24]` **Testing "Restore AC Power Loss" with an OS shutdown** — a shutdown stays off BY DESIGN; only pulling and restoring AC tests it. A community README claimed the two are indistinguishable; trusting it cost Prime two trips to the pfi-gx10 power button.
+  _Archived 2026-10-08._
+
+- `[2026-09-23]` **`booth link --help`** — there is no help flag; it posts `--help` to the operator's link board as a link. Read `booth` with no args for usage.
+  _Archived 2026-10-08._

persistent-memory.d/2026-09-22-safe-rm-installed-on-nh3-dev-and-delegated-fleet-wide-to.md

@@ -1,3 +0,0 @@
-# `[2026-09-22]` safe-rm installed on nh3-dev and delegated fleet-wide to infra-hermes.
-
-⭐ **safe-rm installed on nh3-dev and delegated fleet-wide to infra-hermes.** ⚠ The package installs INERT and looks fine — Debian's `/etc/zsh/zprofile` has 0 non-comment lines so the shipped `profile.d` hook never fires under zsh; verify with `command -v rm`, never `dpkg -l`. ⚠ And it does NOT cover the habit that prompted it: measured, it refuses `rm -rf /home` and deletes an unset-variable path without complaint. Blacklist, not heuristic; `set -u` is the actual cover. Wiring ruled: `/etc/bash.bashrc` above the `case $-` guard, not `/etc/environment` (an rc self-guards, smaller blast radius, covers bash-over-ssh).

persistent-memory.d/2026-09-23-elway-ownership-fix-fleet-audit.md

@@ -1,33 +0,0 @@
-# `[2026-09-23]` elway sudo uploads land root:root; fleet ownership audit; 33 files fixed
-
-elway's sudo upload was scp-as-user then `sudo mv`, and mv keeps the owner, so
-every file it installed as root (systemd units, `/etc` configs, root-run
-scripts) landed owned by infra-ops or lkraven. A sudoers drop-in installed that
-way would break sudo outright. Commit `466f7aa`:
-
-- sudo uploads chown to `root:root` by default; `upload.owner:` / `--owner`
-  override; owner refused on non-sudo uploads.
-- chown+chmod act on the STAGED file, then one `mv` publishes it, so a failed
-  chown cannot leave the live path mis-owned. A trap removes the staged file on
-  every exit. A directory dest is refused before anything moves.
-- `mode` was spliced unquoted into the remote root shell; now octal-validated
-  and quoted. `mode`/`owner` must be quoted YAML strings (bare `0644` → 420).
-- `preflight()` resolves every step before any remote action.
-- 24 unit tests; heid bug-hunt "Puck" (Gróa + seat) folded.
-
-`scripts/fleet-ownership-audit.sh` (read-only; exit 0/4/5): tier A root-parsed
-paths (symlinks judged by target), tier X root-run unit Exec paths incl.
-drop-ins, tier B /opt summary. Unprivileged or partial hosts report INCOMPLETE,
-never clean; completion is nonce-marked. Positive controls on nh3-dev fired for
-each tier. **Six hosts have no infra-ops identity** (ana-wg, nh3-nas, pbs-ana,
-pbs-nh3, pfi-postgres, vm-esh-nas) and audit unprivileged.
-
-**33 files chowned to root on 10 hosts** with Prime's approval (the first
-attempt was blocked by the permission classifier): 31 elway-placed (beszel
-agent trio on 6 hosts, esh-vm-db/ana-docker restic hooks and retry drop-ins,
-irv-ml1 units, nh3-dev alert bridge, fv-ml1 retired file) plus 2 non-elway
-root-exec holes (ana-docker `/usr/local/bin/resticprofile` owned by llmuser;
-esh-pve-nas `/usr/local/lib/libigdgmm.so.11.3.1343` uid 1000). Re-audit: 0 of
-the 33 remain; all affected services still active. Left as low-risk and
-reported: ana-docker `/root/.nvm` (lkraven), irv-ml1 `uv`/`uvx`, fv-ml1
-`btop` and a root uv-cache python.

persistent-memory.d/2026-09-23-esh-docker-vm-restic-was-skipped-09-22-23-by-my-own-kuma.md

@@ -1,3 +0,0 @@
-# `[2026-09-23]` esh-docker-vm restic was skipped 09-22..23 by my own Kuma move
-
-**esh-docker-vm restic was skipped 09-22..23 by my own Kuma move** — a dead uptime-kuma lookup aborted `pre-backup.sh` under `set -e` (`25e41d2`). Then, by Prime's decision, the redundant Paperless pg_dump went too: it had failed auth every night since 2026-04-24 and left a 0-byte dump in every snapshot; the DB is covered at source by esh-vm-db's pg_dumpall (`6e8da46`).

persistent-memory.d/2026-09-23-headscale-ddns-hardened.md

@@ -1,3 +0,0 @@
-# `[2026-09-23]` headscale-ddns hardened
-
-**headscale-ddns hardened** (`fedd4b6`): Cloudflare calls retry and validate every body (`pick()`), no write without both IDs, the run ends on a confirmation, only a global v4 is published, `curl -q`. Three of four weekly failures were empty zone lookups. heid bug-hunt "Talus" folded (one finding was my own regression).

persistent-memory.d/2026-09-23-hermes-gateway-restart-exit-1-is-a-hermes-race-not-a-crash.md

@@ -1,3 +0,0 @@
-# `[2026-09-23]` hermes-gateway restart exit-1 is a Hermes race, not a crash
-
-**hermes-gateway restart exit-1 is a Hermes race, not a crash** — the planned-stop watcher consumes the marker before SIGTERM re-runs the handler; drop-in `SuccessExitStatus=1` stops the false OnFailure page at zero coverage cost (`c2b0a05`). svos had its own stop-timeout (an open board SSE tab), fixed by svos-dev with `timeout_graceful_shutdown=5`.

persistent-memory.d/2026-09-24-esh-ml1-embed-rerank.md

@@ -1,80 +0,0 @@
-# `[2026-09-24]` esh-ml1 built: RTX 2000E Ada serving embed + rerank, LiteLLM order-2 failover behind fv-ml1
-
-Executes Prime's 2026-09-24 decision (LXC + host driver, not a VFIO VM). Done
-2204–2240 PT the same night, **with no esh-pve reboot**.
-
-**Host (esh-pve).** NVIDIA **580.178.04** open modules via DKMS from NVIDIA's
-`-no-compat32.run` (sha256 checked against NVIDIA's published sum), applied
-live by `playbooks/pve-nvidia-host.yaml`: nouveau was never loaded and the
-T400's vfio ids matched nothing, so nothing held the card. Our own
-`nvidia-persistenced.service` creates `/dev/nvidia{0,ctl,-uvm,-uvm-tools}`
-**before `pve-guests`** (else `pct start` refuses the `devN:` entries). Retired
-`vfio.conf` (moved to `/root/nvidia/`) and `blacklist nvidia`. Initramfs NOT
-rebuilt on purpose (boot path untouched). 580 because the fleet's vLLM
-`v0.24.0` image is CUDA 13.0 (driver ≥ 580); fv-ml1 runs 580.65.06. `.run`
-not an apt repo: one file on both sides makes the host-module/container-lib
-version match true by construction; apt would let an upgrade move one side.
-
-**CT 110 `esh-ml1`** (`playbooks/gpu-lxc.yaml`): unprivileged Debian 12,
-nesting+keyctl, 6c/16G/80G `local-lvm`, `10.0.50.80` VLAN 50 (static; UDM pool
-is `.150–.250`), `startup order=30` after esh-scale/VMs. NVIDIA userspace from
-the same `.run` with `--no-kernel-modules`; docker-ce; nvidia-container-toolkit
-with `no-cgroups=true`. **Docker-in-unprivileged-LXC with the GPU worked first
-try on lxc-pve 6.0.0-2** (no AppArmor workaround needed). Fleet ids per
-`docs/pfi/fleet-conventions.md` (infra-ops 850, docker 851, vh 1000 — vh has
-no key/password yet). **Not in the vzdump job, on purpose** (rebuildable).
-DNS `esh-ml1.esh.internal` synced to all three resolvers.
-
-**Stack `stacks/embed-rerank`:** `vllm-embed` :8001 + `vllm-rerank-bge` :8013,
-same models/flags/vLLM digest (`251eba5cc7c1`) as fv-ml1, 0.20 mem-util each →
-4,823 MiB used of 16,380.
-
-**Parity (the load-bearing measurement; embeddings are model-specific).** 11
-texts × 2 runs per site. Embed cosine FV-vs-ESH median 0.999908 / min 0.999772,
-inside both self-noise floors (FV-vs-FV 0.999927 / 0.999791; ESH-vs-ESH
-0.999911 / 0.999809). Negative control (different texts) 0.07–0.38. Sensitivity
-floor ~2×10⁻⁴ cosine. Rerank |score| max 0.000145 vs FV-FV floor 0.000181,
-identical ranking.
-
-**Gateway.** `qwen3-embedding` and `reranker` each got a second deployment
-(`order: 2`, esh-ml1); fv-ml1 is `order: 1`. LiteLLM v1.97 order-fallback
-proven with throwaway groups (created + deleted): primary refusing → ESH in
-+0.15 s (3/3); primary host-down (no ARP) → ESH but **~18.7 s on every call,
-no cooldown** (3/3); dead-only negative control → 500 (3/3). Post-restart
-production traffic is served by fv-ml1 (header `x-litellm-model-api-base`).
-`/health?model=` returned 503 with empty lists for both groups — not
-investigated. Gateway restart: liveliness back in ~52 s.
-
-**Found + fixed:** DB-only alias `reranker-a3-bge-v2-m3` pointed at ana-ml2's
-pre-relocation IP `10.250.50.54` — dead since 2026-09-12 (500 after ~23 s), no
-callers in 7-day spend logs. PATCHed to `10.251.50.54:8013`. Sweep of
-`/model/info` found no other `10.250.50.54` targets.
-
-**Who actually uses these names (spend logs, 7 days to 2026-09-25 05:35Z):**
-`qwen3-embedding` — worldtree-gateway 481 (nh3-dev + corviduo-dev), nevermore
-20; `reranker` — nevermore 13. **No ESH-side consumer called either in 7 days**
-(Open WebUI's RAG embeds only on document upload). So today esh-ml1's value is
-an FV-outage failover for Worldtree + nevermore, not local service for ESH.
-
-**Open:** a direct ESH→esh-ml1 path (survives a mesh outage) would help only
-ESH consumers, which are idle — recommended parked. esh-ml1 not yet in
-Homepage docker.yaml or Beszel. The ~18.7 s host-down failover penalty is
-untuned (router retries/cooldown).
-
-**Speed A/B vs fv-ml1 (2026-09-25 0628–0640 PT, table in `servers/esh-ml1/README.md`).**
-Single queries: a wash (on-box ESH ~9 ms vs FV ~12 ms; from ana-docker ESH is
-faster because it is network-closer). Bulk: ESH 3–10× slower (bulk embed ~50 vs
-~403 passages/s; rerank-20 ~170 vs ~51 ms p50, ~5.7 vs ~55 req/s at conc 8).
-Noise floor 3–5% on throughput. ⇒ failover role confirmed; load-sharing would
-roughly double rerank latency for half the calls.
-
-**TEI bake-off (2026-09-25, `stacks/tei-bakeoff/README.md`).** Prime asked to
-evaluate a purpose-built engine instead of vLLM → HF Text Embeddings Inference
-1.9.4 (Infinity ruled out: last release 2025-08-22, dropped by us in May for no
-Qwen3). Verdict: TEI **matches** (embed cosine vs vLLM-FV median 0.999925, inside
-noise; TEI queries against the OLD vLLM index overlap@10 0.988 vs 0.986 self-noise
-→ no re-embed needed; rerank decisions identical, tail scores move ≤0.019) but is
-**not faster** on the Ada (novel 40 s vs 31 s; tiny queries ~25% faster). It is
-**much lighter**: ~2.6 GB VRAM for both vs vLLM's 4.8 GB budget, 8 GB image vs 30,
-~4 s restart vs ~24 s. Gateway: embed drop-in via `hosted_vllm/`; rerank needs the
-`huggingface/` provider. Both engines left running side by side pending Prime's call.

persistent-memory.d/2026-09-24-esh-pve-vm102-retired-gpu-swap.md

@@ -1,40 +0,0 @@
-# `[2026-09-24]` esh-pve: VM 102 retired, a 14-day hung VFIO process found, T400 → RTX 2000 Ada
-
-**VM 102 `esh-vm-workstation` retired permanently** (Prime: "not the machine I
-want running a virtual windows machine"). Final backup
-`pbs-ana:backup/vm/102/2026-09-10T10:32:20Z` (verification ok) set
-**protected** so prune cannot take it; `qm destroy 102 --purge`.
-
-⚠⚠ **Found: `kvm -id 102` had been hung in D state since the 2026-09-10 0332
-vzdump.** vzdump starts a stopped VM (here with the T400 passed through,
-`x-vga=1`) to read its disks; that process never exited. It held **16 GB of
-locked RAM**, the GPU, and two LVs, while `qm status 102` said `stopped` the
-whole time. Same class as the 2026-08-19 hard freeze: VFIO + this box. Only a
-reboot cleared it.
-
-**GPU swap, Prime on site, 2119–2137.** Guests shut down cleanly; host powered
-off; the T400 was removed and an **NVIDIA RTX 2000 / 2000E Ada** installed
-(`01:00.0`, `10de:28b0`, subsystem `10de:1871`). Host back 2136:49; leftover
-LVs removed; all guests auto-started; esh-docker-vm NFS, traefik (0×403) and
-AdGuard verified. The new card has **no driver bound**; `/etc/modprobe.d` still
-lists the T400's vfio ids (`10de:1ff2,10de:10fa`), now matching nothing.
-Commits `719e3fa`, `7cbd4c3`.
-
-**Decision (Prime, 2026-09-24): the card serves embedding + reranking as an
-LXC with the NVIDIA driver on the host — NOT a VFIO VM.** Rationale: both
-esh-pve hangs this year came from VFIO passthrough; LXC avoids it and pins no
-VM RAM. Accepted cost: a driver on the hypervisor, rebuilt on PVE kernel
-updates, host/container versions must match. **Implementation deferred to the
-next session** (tracking: this entry + `servers/esh-pve/README.md`).
-
-**ESH single-route observation:** esh-scale (CT 108, ESH's only mesh subnet
-router) lives on esh-pve, so while esh-pve was down *all* of ESH looked dark
-from outside, including esh-pve-nas and vm-esh-nas, which were up. Untracked by
-operator choice so far — a second ESH route is an idea, not a decision.
-
-**vPro:** the copper cable is in the AMT-capable port (**I226-LM**, `enp89s0`,
-MAC `58:47:ca:76:99:32`), verified at 2500 Mb/s; the I226-V (`enp88s0`) is
-not AMT-capable. Both links set back admin-down on the host. **MEBx
-provisioning deferred by Prime** ("some other time"): Ctrl+P at boot, enable
-manageability, static IP, KVM on with User Opt-in None, activate network
-access; MeshCentral should live on ana-docker/nh3-docker, never on esh-pve.

persistent-memory.d/2026-09-24-gx10-ac-restore-patch.md

@@ -1,32 +0,0 @@
-# `[2026-09-24]` pfi-gx10 "power on after AC loss" — UEFI patch applied, NOT validated
-
-pfi-gx10 (ASUS Ascent GX10, BIOS `GX10DGX.0105.2026.0505.1153`, no BMC) stayed
-off after the NH3 outage. With Prime's approval (after the permission
-classifier first refused it as externally-sourced firmware code), bytes 3–4
-of the 81-byte UEFI var `Setup-ec87d643-eba4-4bb5-a1e5-3f3e36b20da9` were
-changed `0x1d,0x00 → 0x1c,0x01` via `efivar -w`; readback shows exactly those
-two bytes changed. Recipe: community tool `Fheodoor01/DGX-Spark---Power-on-boot-`
-(diff-derived from a power-on unit vs a stay-off unit; author-verified on
-GX10DGX.0103/.0104). **Byte 3's meaning is unknown**; the resulting pair
-matches factory power-on units.
-
-**Original saved twice:** `/var/lib/efi-setup-backup/Setup-ec87d643-20260924-pre-poweron.bin`
-on the box and `servers/pfi-gx10/efi-Setup-ec87d643-20260924-pre-poweron.bin`
-(sha256 `6d6f6f77…`, raw incl. the 4-byte attr header). Revert procedure in
-`servers/pfi-gx10/README.md`. Commit `e5197a3`.
-
-⚠⚠ **An OS shutdown is NOT a test of this setting.** It means "start when
-power RETURNS after a cut"; a `poweroff` is supposed to stay off by design.
-The tool's README claims the two are indistinguishable — false here. I
-trusted it and ran two poweroff "tests" (1421, and 1435 after the reboot the
-README says the firmware needs); both stayed off, each costing Prime a trip to
-the button, and both prove nothing. Prime called it out directly.
-
-**Status:** the box is OFF (the 1435 shutdown) with the patch in place, and
-Prime left NH3. **He will pull and restore AC on 2026-09-25.** Comes up alone
-→ works; drop "NOT YET VALIDATED" from the README. Stays off → he presses the
-button, revert from the backup, and set it in firmware via a PiKVM (he intends
-to add one with ATX control, which also covers remote power-on regardless).
-
-**Validated 2026-09-25 (Prime, relayed by Miranda 1333):** Prime replugged AC and
-the box came up by itself at 1108:54 PT, with no button press. The patch works.

persistent-memory.d/2026-09-24-miranda-standing-order-is-a-repo-claude-md-operating.md

@@ -1,3 +0,0 @@
-# `[2026-09-24]` Miranda standing order is a repo CLAUDE.md operating parameter
-
-**Miranda standing order is a repo CLAUDE.md operating parameter** (`4b29492`, aligned to the global send protocol in `bcf3342`): high-urgency matters go to her, fixed or not; URGENT only when it cannot wait (she phones Prime). Channel verified end to end (thread `01M3A0RP4Q8T0KNGH8TMFSNDA6`); it depends on svos + hermes-gateway. Prime's callsign **PRiMe / papa romeo mike** is a name, not an authenticator (`617b759`, `62817a2`).

persistent-memory.d/2026-09-24-nh3-power-outage-recovery.md

@@ -1,31 +0,0 @@
-# `[2026-09-24]` NH3 power outage — recovered; three latent gaps closed, one needs a hand
-
-Power at NH3 dropped and returned; every NH3 box booted at 1233–1234 PT. The
-core recovered unaided (post office, headscale, egress proxy, DNS, WAN IP
-unchanged at `70.230.226.88`). Three things did not, each a latent config gap
-that only a site-wide power event exposes:
-
-1. **pbs-nh3 (VM 105) had no `onboot` flag** and stayed down. Set `onboot=1`
-   and started it; datastore NFS mounted (its line uses `bg`). `qm guest exec
-   105` works (guest agent on) — the root path, since infra-ops is not
-   provisioned there.
-2. **NFS boot race.** nh3-nas is the slowest box to serve NFS, so every plain
-   fstab NFS line failed at boot (nh3-docker `/mnt/compose` `/mnt/backup`,
-   nh3-dev `/mnt/backup`). nh3-dev's `/mnt/smithy` already had
-   `x-systemd.automount` and self-healed on its next access (1234 failed, 1309
-   mounted, no hand) — the existence proof. `playbooks/nh3-nfs-automount.yaml`
-   brought the rest to that shape (`_netdev,nofail,x-systemd.automount,
-   x-systemd.mount-timeout=30`), commit `1cbde50`. ⚠ nh3-dev's installer
-   cdrom fstab line is a pre-existing `findmnt --verify` error; the playbook
-   judges only errors a rewrite ADDS, after its first run aborted on it.
-3. **pfi-gx10 (bare metal, no BMC) did not power on** — ASUS ships "Restore AC
-   Power Loss" = stay off. See `[[2026-09-24-gx10-ac-restore-patch]]`.
-
-Also: **every Claude session on nh3-dev died** with the reboot except
-infra-ops, infra-hermes, jekyll. Relaunch is Prime's call (dev-launch).
-`nh3-laser` (VM 104) is on-demand and **stays off** by Prime's ruling;
-`servers/nh3-pve/README.md` now carries an expected-after-power-loss column per
-guest so the next triage does not flag it.
-
-The headscale-ddns failures 1141–1202 were the WAN being down, correctly
-refused (no public v4); every run since 1247 reads `unchanged`.

persistent-memory.d/2026-10-07-albok-service-0-1-13-sensitivity-gate.md

@@ -0,0 +1 @@
+- `[2026-10-07]` **albok-service 0.1.13 (unit 4.2, the sensitivity gate) deployed on nh3-albok at 1135 at albok-dev's ask 8413.** Rollback is NOT free: the first-start scan writes `sensitivity` sidecars, which 0.1.12 cannot load, so rolling back needs the strip snippet first. The census found `leaks` 8: six hold the shared all-agents key, two are false positives on a doc placeholder, nothing to rotate. → `stacks/albok-service/README.md`

persistent-memory.d/2026-10-07-albok-service-0-1-14.md

@@ -0,0 +1 @@
+- `[2026-10-07]` **albok-service 0.1.14 (unit 4.3: sidecar listing, byte-less ingest, a `.browse/` view; detector 2) deployed at 1804 at albok-dev's ask 8524.** `leaks` went from 8 to 6 when the placeholder false positives cleared. Reindexes ran one per wing afterwards to render the view. ⚠ **OWED: the `.browse/` exposure test.** Pick a share surface (sshfs `follow_symlinks` is the lead), rename `.browse/` on the server, and check that a fresh open through the client's cached listing FAILS. Nothing is shared until it passes; report to albok-dev. → `stacks/albok-service/README.md`

persistent-memory.d/2026-10-07-gateway-glm-5-3-gen-frontier.md

@@ -0,0 +1 @@
+- `[2026-10-07]` **Gateway: glm-5.3 added; gen-frontier / gen-frontier-reasoning moved 5.2 → 5.3 (Prime); every GLM 5.2/5.3 entry advertises 1,048,576 in / 131,072 out with max_tokens 131,072 as the default.** 5.3 cannot turn reasoning off, so the gen-frontier pair is now effort low / high (measured ~260 vs ~1,490 reasoning tokens). ⚠ z.ai's `thinking: disabled` is light thinking, not zero, on 5.1/5.2/5.3 alike (220–340 reasoning tokens on a hard prompt). Restarted 1725 (61 s), held 2 h for albok-dev's load A/B. The gate script's first two runs could never pass: `grep -c` exits 1 on a zero count. Images: glm-5.x is text only (z.ai rejects image parts); glm-4.6v and glm-5.3-flash read images on our plan, but neither is on the gateway. → `stacks/litellm/conf/config.yaml`

persistent-memory.d/2026-10-07-nh3-svos-refresh-a737a06-apps.md

@@ -0,0 +1 @@
+- `[2026-10-07]` **nh3-svos refreshed to SVOS a737a06 (schema 11) at svos-dev's ask (Prime's 2240 decision).** Live: app-knowledge (Albok token per A111), plus app-terminal, degraded by design (no seat on the box), on its own origin `svos-terminal.phasefinal.com` (no Basic auth, mesh-only). app-postoffice is declared and stopped (⚠ the bridge handle is Prime's to declare); Miranda's packages are only staged, for U13. Acceptance: 17/17 HTTPS checks and the Chromium phone/desk checks passed. ⚠ The interface fails once on thaw (it starts before the kernel listens); the restart recovers it, but the hook's single status check can race it. Reported to svos-dev. → `servers/nh3-svos/README.md`

persistent-memory.d/2026-10-08-heid-runaway-sweep-killed.md

@@ -0,0 +1,16 @@
+# 2026-10-08 — heid's runaway sweep killed (Prime: "kill sweep immediately")
+
+**Symptom.** nh3-dev's root disk flapped on the Beszel 85% line (2300 and 2329 10-07). The uv-cache prune (−6.7 GiB) bought minutes. `du` could not see the growth.
+
+**Cause (found by infra-hermes, confirmed by infra-ops).** `heidnotes/sweep.py` under `heid/dispatch-log/work/01M4CC74W1HK46D1ZP9J26YSZK` was left running by a consult agent of svos-dev's U11 bug hunt, which had been relayed and closed at 1748. It swept the dispatch work dir recursively, and its output `heidnotes/final_sweep.txt` sat INSIDE that dir. So it read its own output and wrote it back out, forever: the tail was lines quoting `final_sweep.txt:66200017 …`. The work dir was then cleaned while the file was still open, so ~76 GB lived in a deleted-but-open file that only `/proc/<pid>/fd` showed.
+
+**Two holders, not one.** infra-hermes saw PID 2332755; a second copy, PID 2368442 (its zsh wrapper 2368332, launched from the `heid` Claude session, pid 35804), held the same inode on fd 3. Killing the first freed nothing. Only when the second died did root go 86% → 66% (52 → 123 GB free).
+
+**Lessons.**
+- A deleted-but-open file IS readable through `/proc/<pid>/fd/<n>`. That is how the self-feeding loop was confirmed before the kill, turning "runaway, kill it" from a guess into a verdict.
+- Before claiming that space will free, enumerate EVERY holder of the inode (walk `/proc/*/fd`), not just the first PID found.
+- Kill by explicit PID. Matching `heidnotes/sweep.py` also caught the wrapper shell, which was harmless here, and `pgrep -f` self-matched the checking command.
+
+**Fix at the source (heid `8bf2fe57`).** The sweep tool skips `heidnotes/` and any file over 20 MB. The consult-agent brief forbids leaving a process running after the report. heid is adding a check for lingering processes before each workdir clean. The same bug had hit a second agent at ~3.4 GB earlier that day, and that agent caught it itself.
+
+Ops-log 2026-10-08T06:37:07Z `nh3-dev kill heid-sweep`.

persistent-memory.d/2026-10-08-patch-night-fv-ml1-kernel-nvidia-zfs-docker-hosts.md

@@ -0,0 +1 @@
+- `[2026-10-08]` **Patch night (Prime's go): fv-ml1 → kernel 6.12.111, ZFS 2.3.9, NVIDIA 580.178.04, toolkit 1.20.1 (one reboot, 13/13 seats healthy in ~7 min, every alias answered); Docker 29.4.1 → 29.8.2 on esh-docker-vm and nh3-docker (post office ~25 s blip), all containers back.** ⚠ fv-ml1 boots ZFSBootMenu, not GRUB; trixie's apt rejects NVIDIA's debian12 repo key (SHA1 binding), so driver debs are gpgv/sha256-verified by hand. Holds re-applied on all three. Rollback snapshot `zroot/ROOT/debian@pre-patch-20261008`, destroy ~10-15. SVOS post office is running; its inbox grant is a card for Prime. → `servers/fv-ml1/README.md`

persistent-memory.md

@@ -1,6 +1,6 @@
 # Persistent memory — eshpfi-management
 
-_Last updated: 2026-10-06 ~2145 PT (fv-ml1 KV pins + GPU 0/1 rebalance by usage with contexts at published maximums, gen-small output cap; backup checker triple fixed, destroyed-guest residue pruned, pbs-nh3 mirror retention matched; Scriberr diarization fixed with caches on /tank and no real token; albok-service 0.1.11 → 0.1.12; gateway keeps no content (arbo incognito) + irv-ml1 restic exclude; jekyll moved to sec-reasoning.)_
+_Last updated: 2026-10-08 ~0740 PT (patch night: fv-ml1 kernel 6.12.111 + NVIDIA 580.178.04 + ZFS 2.3.9, Docker 29.8.2 on nh3-docker + esh-docker-vm; nh3-svos on SVOS a737a06 with knowledge/terminal/post office live; albok-service 0.1.13 → 0.1.14; gateway glm-5.3 + gen-frontier on 5.3; heid's runaway 76 GB sweep killed; Hermes launchers repaired.)_
 
 > **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its
 > `Written:` stamp is under **8 hours** old, read it (it carries the in-flight
@@ -113,24 +113,30 @@ no longer deployed sidecars here. See Recent decisions.)
 
 ## Current state / in-flight
 
-_As of 2026-10-06 ~2145 PT. The newest subsection is first; older subsections carry their own dates. What was DONE on 10-05/06 lives in Recent decisions; this section keeps only what is open, scheduled or being watched._
+_As of 2026-10-08 ~0740 PT. The newest subsection is first; older subsections carry their own dates. What was DONE lives in Recent decisions; this section keeps only what is open, scheduled or being watched._
 
-### 2026-10-06: live now (as of 2026-10-06 ~2145 PT)
+### 2026-10-08: live now (as of 2026-10-08 ~0740 PT)
 
-- **Nothing is mid-action.** No claims held (`scripts/ops-log claims`). eshpfi is pushed through `0f6c7a0` (origin/main == HEAD at ~2140). ⚠ The working tree AND index are shared with infra-hermes and subagents: commit with `git commit -- <paths>` (auto-memory `feedback_shared_git_index_commit_pathspecs`). `graphify-out/GRAPH_REPORT.md` stays modified and uncommitted on purpose (auto-regenerated).
+- **Nothing is mid-action.** No claims held (`scripts/ops-log claims`). eshpfi is pushed through this snapshot's commit. ⚠ The working tree AND index are shared with infra-hermes and subagents: commit with `git commit -- <paths>` (auto-memory `feedback_shared_git_index_commit_pathspecs`). `graphify-out/GRAPH_REPORT.md` stays modified and uncommitted on purpose (auto-regenerated).
-- **Verified 2026-10-07/08 (closed):** the 10-07 0800 backup check read all fresh with no DESTROYED/DEGRADED lines; the pbs-nh3 mirror prune ran OK at 00:00 on 10-07 and 10-08 (nh3-nas still 79% at 2350, since GC frees chunks only after the 24 h atime cutoff; re-read after the 10-08 GC); `pre-pve9` snapshots are gone on pfi-pve and esh-pve. **Still owed:** the fv-ml1 GPU 0/1 plateau read (Beszel `stats.g.<n>.mu` × 1.024); sfsrv-ana `pre-pbs4` delete unverified (infra-ops has no key there; ask infra-hermes).
+- **Owed by me (no input needed):**
-- **Prime 2026-10-07 late:** SVOS post office bridge handle = `svos` (he declares it on the fleet page; then enable `svos-postoffice` and request its `ns:notifications` write grant as a card for him). **No z.ai vision alias** on the gateway.
+  - albok `.browse/` exposure test before ANY share: pick a surface (sshfs `follow_symlinks` is the lead), rename `.browse/` on nh3-albok, and a fresh open through the client's cached listing must FAIL; also record the failed-rename case. Report on albok-dev's thread; the operator's user needs the restricted wings' groups on the share host. → `stacks/albok-service/README.md`
+  - fv-ml1 GPU 0/1 plateau read (Beszel `stats.g.<n>.mu` × 1.024), now ~10-10, since every seat restarted at 0034 10-08.
+  - Destroy `zroot/ROOT/debian@pre-patch-20261008` on fv-ml1 ~10-15 if nothing regressed.
+  - sfsrv-ana `pre-pbs4` snapshot delete is unverified (infra-ops has no key there; ask infra-hermes).
+  - Re-read nh3-nas usage after the 10-08 GC (79% at 10-07 2350; the mirror prune ran OK).
+  - Install SVOS kernel 0.0.7 (`Type=notify`, `NotifyAccess=main`: the thaw-race fix Prime accepted) with the next nh3-svos refresh, when svos-dev says it is on main.
+- **Verified 2026-10-07/08 (closed):** the 10-07 0800 backup check read all fresh with no DESTROYED/DEGRADED lines; the pbs-nh3 mirror prune ran OK at 00:00 on 10-07 and 10-08; `pre-pve9` snapshots are gone on pfi-pve and esh-pve.
 - **Watching: jekyll on `sec-reasoning` (cyberprev, GPU 0)** since ~1435 10-06 (Prime moved it off gen-large). sec holds 337k KV tokens. jekyll's ~180k context plus a long sec prompt (up to 192k) can overflow it, as gen-large did (preemptions; decode down to 2–16 tok/s). jekyll's context grows ~1k/call toward the 262k ceiling. It runs on the shared all-agents-local key; a key of its own would make its traffic attributable.
 - **Waiting on Prime:**
   - albok's 200-object load check, 2.01× against a 2× bar (albok-dev: the ruling is the operator's).
   - The cleanup bundle offered 10-05: close stale parks 49/46/40; refresh the fv-ml1 README stack tables plus the serial-keyed NVMe table and `zpool replace` line (the 2084-media-error drive is now `/dev/nvme6`, not nvme7); wire ZED pool alerts into althing-alert-bridge; prune the exited one-shot containers on fv-ml1 (keep the `parakeet` rollback).
   - Push Booth commit `ce095c1`. A trusted HTTPS name for ESH Homepage. nh3-dev reboot (kernel 6.1.0-53 installed).
   - ⚠ **irv-ml1: DO NOT PATCH (Prime)**; config changes are fine. ana-nas is out of scope.
-- **Scheduled / unchecked:** the 10-05 0815 + 0900 infra-hermes jobs (sfsrv-ana `pre-pbs4` snapshot delete; drop `pve/root-pre-pve9` on pfi-pve + esh-pve) left no ops-log lines; ask infra-hermes or read the hosts. esh-nas-pve `nvme/ROOT@pre-pve9` destroy ~10-10 (ours). 10-11 0540: first weekly sudo-io archiver run (check it). 10-11 0900: infra-hermes deletes the two one-off PBS VM vzdumps. Never `zpool upgrade` any pool yet. Next scheduled tank scrub on fv-ml1: Sun 10-11 0024 (the September slot was missed); check it ran.
+- **Scheduled / unchecked:** esh-nas-pve `nvme/ROOT@pre-pve9` destroy ~10-10 (ours). 10-11 0540: first weekly sudo-io archiver run (check it). 10-11 0900: infra-hermes deletes the two one-off PBS VM vzdumps. Never `zpool upgrade` any pool yet. Next scheduled tank scrub on fv-ml1: Sun 10-11 0024 (the September slot was missed); check it ran.
 - **Worldtree memory gate:** every FAIL since 09-30 (0930T0809, 1004, 1007) is ONE fixture, `u-negative-ephemeral` FILED in one run — the #416 ephemeral residual, ~3 of 42 fixture-runs since bbcc1788 (~1 batch in 5 FAILs on it), NOT a recall regression; recall (filing/user median 0.83–0.92, opener/cross_session/inline_core 100%) has held throughout. Negative is a structural class, so one run's leak fails the aggregate (`_run_structural_failed`); filing alone is judged by median. 10-07: infra-hermes told to classify FAILs by failing row (msg to its skill's daily-duty section); worldtree-dev told #416 is not closed (pull-only, unread 4 days) — reopen is theirs/Prime's.
-- **Small clean-ups, ours, untracked:** stale LiteLLM aliases `erp-tune-v2` and `gemma4-26b-a4b-it-base`; ana-docker's Traefik (`readTimeout` 10m) and CrowdSec (`use_wal`, bouncer `updateMaxFailure: 3`) configs live only on the host, not in `stacks/`; TEI weights revision unpinned on esh-ml1 + nh3-ml1; a stray nginx `.bak` on pfi-tacticalrmm; meromero's Homepage label still says "Dormant / rollback"; Scriberr's `patches/0003` lands only at the next `scripts/scriberr-rebuild`.
+- **Small clean-ups, ours, untracked:** stale LiteLLM aliases `erp-tune-v2` and `gemma4-26b-a4b-it-base`; ana-docker's Traefik (`readTimeout` 10m) and CrowdSec (`use_wal`, bouncer `updateMaxFailure: 3`) configs live only on the host, not in `stacks/`; TEI weights revision unpinned on esh-ml1 + nh3-ml1; a stray nginx `.bak` on pfi-tacticalrmm; meromero's Homepage label still says "Dormant / rollback"; Scriberr's `patches/0003` lands only at the next `scripts/scriberr-rebuild`. The LiteLLM compose.yaml on ana-docker differs from `stacks/litellm/compose.yaml` (found 10-07; only the conf side was pushed). nh3-dev holds ~33 GB of unused TAGGED Docker images (rollbacks; not pressing at 66%).
 - **Open, Prime's calls (unchanged from 10-03):** QNAP LCD visual check (`services/qnap-lcd/`); NH3 Mac mini (`Mac-mini`, 00:c5:85:13:e0:92, 10.100.10.121, asleep since 10-03 ~1407) wake fix on site: `sudo pmset -a sleep 0 disksleep 0 displaysleep 10 womp 1 autorestart 1 powernap 0`, ideally Ethernet; Booth unreachable from 10.100.10.166 (Prime's Wi-Fi MacBook; resume only on his word; lead: SSID client isolation / proxy-ARP); Uptime Kuma has nothing on Miranda's chain (svos :8770, hermes-gateway); mask `openipmi.service` on esh-nas-pve; run `playbooks/pve-nag-patch.yaml` on esh-pve-2 when it is plugged back in.
-- **Older open items, one line each:** esh-matter: Prime shares the Aqara W200 into HA via Matter multi-admin (if commissioning misbehaves, the UniFi mDNS reflector on esh-iot is the first knob). Restic: the rest-server password rotation stays Prime's (the units no longer leak it). esh-ml1: the nh3-docker Dozzle agent has been stopped by hand since ~2026-04 (revive or drop), and the Beszel superuser password rotation was offered to Prime. Blender: agent-drawn CAD Sketcher geometry and MeasureIt overlays in MCP screenshots are still open (`stacks/blender/README.md`). Worldtree reward path: confirm personal's image now includes d8726a13 (it ran `51168a2e` on 09-27). nh3-dev's root disk sits near the Beszel 85% line. ESH has a single outside route (esh-scale on esh-pve), noted and untracked.
+- **Older open items, one line each:** esh-matter: Prime shares the Aqara W200 into HA via Matter multi-admin (if commissioning misbehaves, the UniFi mDNS reflector on esh-iot is the first knob). Restic: the rest-server password rotation stays Prime's (the units no longer leak it). esh-ml1: the nh3-docker Dozzle agent has been stopped by hand since ~2026-04 (revive or drop), and the Beszel superuser password rotation was offered to Prime. Blender: agent-drawn CAD Sketcher geometry and MeasureIt overlays in MCP screenshots are still open (`stacks/blender/README.md`). Worldtree reward path: confirm personal's image now includes d8726a13 (it ran `51168a2e` on 09-27). nh3-dev's root disk is at ~66% since heid's runaway sweep was killed 10-08 (it flapped on the Beszel 85% line before). ESH has a single outside route (esh-scale on esh-pve), noted and untracked.
 
 ### fv-ml1 GPU layout (as of 2026-10-05 2249)
 
@@ -181,14 +187,17 @@ Every vLLM seat on a shared card (GPU 0, GPU 1) has byte-pinned KV (`--kv-cache-
 
 ## Recent decisions
 
+- `[2026-10-08]` **heid's runaway sweep killed on Prime's order: two `heidnotes/sweep.py` processes had written ~76 GB into a deleted-but-open file; nh3-dev root 86% → 66%** — the sweep read its own output. heid fixed it at the source. → `persistent-memory.d/2026-10-08-heid-runaway-sweep-killed.md`
+- `[2026-10-08]` **SVOS post office LIVE on nh3-svos** — Prime declared `svos`; its `ns:notifications` write grant was approved on his word at 0731 (card `card:4d3a8b26c17f472d828b06fd495192db`); a test message was ingested and delivered; `reach` ok, unread 0. Reported to svos-dev.
+- `[2026-10-07]` **No z.ai vision alias on the gateway (Prime).** glm-5.x is text-only at z.ai; glm-4.6v and glm-5.3-flash read images on our plan but stay off the gateway.
 - `[2026-10-06]` **Scriberr diarization fixed: gated pyannote, a torchcodec/torch mismatch, and WhisperX's CPU defaults; model caches on /tank, HF_TOKEN a placeholder (Prime), GPU+speaker profiles.** Patch 0003 waits for the next rebuild (tracked at `stacks/scriberr/patches/README.md`). → `persistent-memory.d/2026-10-06-scriberr-diarization-fixed.md`
 - `[2026-10-06]` **Backup checker: destroyed guests reported as residue, an ssh stdin swallow fixed, snapshot age from the name; residue pruned (Prime); pbs-nh3 mirror keeps pbs-ana's history (Prime "match history"), first prune 10-07 00:00.** → `persistent-memory.d/2026-10-06-backup-checker-fixes-and-mirror-retention.md`
-- `[2026-10-08]` **Patch night (Prime's go): fv-ml1 → kernel 6.12.111, ZFS 2.3.9, NVIDIA 580.178.04, toolkit 1.20.1 (one reboot, 13/13 seats healthy in ~7 min, every alias answered); Docker 29.4.1 → 29.8.2 on esh-docker-vm and nh3-docker (post office ~25 s blip), all containers back.** ⚠ fv-ml1 boots ZFSBootMenu, not GRUB; trixie's apt rejects NVIDIA's debian12 repo key (SHA1 binding), so driver debs are gpgv/sha256-verified by hand. Holds re-applied on all three. Rollback snapshot `zroot/ROOT/debian@pre-patch-20261008`, destroy ~10-15. SVOS post office is running; its inbox grant is a card for Prime. → `servers/fv-ml1/README.md`
+- `[2026-10-08]` **Patch night (Prime's go): fv-ml1 kernel/NVIDIA/ZFS/toolkit + Docker 29.8.2 on esh-docker-vm and nh3-docker, all verified** — ZBM boot, the hand-verified NVIDIA driver path, holds re-applied. → `persistent-memory.d/2026-10-08-patch-night-fv-ml1-kernel-nvidia-zfs-docker-hosts.md`
-- `[2026-10-07]` **nh3-svos refreshed to SVOS a737a06 (schema 11) at svos-dev's ask (Prime's 2240 decision).** Live: app-knowledge (Albok token per A111), plus app-terminal, degraded by design (no seat on the box), on its own origin `svos-terminal.phasefinal.com` (no Basic auth, mesh-only). app-postoffice is declared and stopped (⚠ the bridge handle is Prime's to declare); Miranda's packages are only staged, for U13. Acceptance: 17/17 HTTPS checks and the Chromium phone/desk checks passed. ⚠ The interface fails once on thaw (it starts before the kernel listens); the restart recovers it, but the hook's single status check can race it. Reported to svos-dev. → `servers/nh3-svos/README.md`
+- `[2026-10-07]` **nh3-svos refreshed to SVOS a737a06 (schema 11): knowledge, terminal (own origin) and post office apps** — acceptance 17/17 + Chromium; the thaw race. → `persistent-memory.d/2026-10-07-nh3-svos-refresh-a737a06-apps.md`
-- `[2026-10-07]` **Gateway: glm-5.3 added; gen-frontier / gen-frontier-reasoning moved 5.2 → 5.3 (Prime); every GLM 5.2/5.3 entry advertises 1,048,576 in / 131,072 out with max_tokens 131,072 as the default.** 5.3 cannot turn reasoning off, so the gen-frontier pair is now effort low / high (measured ~260 vs ~1,490 reasoning tokens). ⚠ z.ai's `thinking: disabled` is light thinking, not zero, on 5.1/5.2/5.3 alike (220–340 reasoning tokens on a hard prompt). Restarted 1725 (61 s), held 2 h for albok-dev's load A/B. The gate script's first two runs could never pass: `grep -c` exits 1 on a zero count. Images: glm-5.x is text only (z.ai rejects image parts); glm-4.6v and glm-5.3-flash read images on our plan, but neither is on the gateway. → `stacks/litellm/conf/config.yaml`
+- `[2026-10-07]` **Gateway: glm-5.3 added, gen-frontier pair moved to 5.3 (effort low/high), 1M/128K advertised on GLM 5.2/5.3 (Prime)** — `thinking: disabled` is light thinking, not zero. → `persistent-memory.d/2026-10-07-gateway-glm-5-3-gen-frontier.md`
 - `[2026-10-07]` **Hermes launchers repaired (Miranda relay 8462):** svos-dev's scratch probe had rewritten `~/.hermes/hermes-agent/.hermes/bin/{hermes,hermes-acp}` to exec a /tmp scratch Python. Repointed to `~/.hermes/tools/python-3.14.7…`; the gateway was not restarted. ⚠ Don't run `git` in `~/.hermes/hermes-agent`: a `status` there looped on auto-gc.
-- `[2026-10-07]` **albok-service 0.1.14 (unit 4.3: sidecar listing, byte-less ingest, a `.browse/` view; detector 2) deployed at 1804 at albok-dev's ask 8524.** `leaks` went from 8 to 6 when the placeholder false positives cleared. Reindexes ran one per wing afterwards to render the view. ⚠ **OWED: the `.browse/` exposure test.** Pick a share surface (sshfs `follow_symlinks` is the lead), rename `.browse/` on the server, and check that a fresh open through the client's cached listing FAILS. Nothing is shared until it passes; report to albok-dev. → `stacks/albok-service/README.md`
+- `[2026-10-07]` **albok-service 0.1.14 (unit 4.3) deployed + all four wings reindexed** — the `.browse/` view exists on the box only; the share test is owed. → `persistent-memory.d/2026-10-07-albok-service-0-1-14.md`
-- `[2026-10-07]` **albok-service 0.1.13 (unit 4.2, the sensitivity gate) deployed on nh3-albok at 1135 at albok-dev's ask 8413.** Rollback is NOT free: the first-start scan writes `sensitivity` sidecars, which 0.1.12 cannot load, so rolling back needs the strip snippet first. The census found `leaks` 8: six hold the shared all-agents key, two are false positives on a doc placeholder, nothing to rotate. → `stacks/albok-service/README.md`
+- `[2026-10-07]` **albok-service 0.1.13 (unit 4.2, the sensitivity gate) deployed** — rollback to 0.1.12 needs the strip snippet first. → `persistent-memory.d/2026-10-07-albok-service-0-1-13-sensitivity-gate.md`
 - `[2026-10-06]` **albok-service 0.1.11 (0202) then 0.1.12 (1254) deployed on nh3-albok at albok-dev's asks 7991/8199**, built from clean archives, pinned by digest, single-file upload (deploy-stack's `--delete` would wipe the on-host rollback copies). Every wing is now migrated to generation collections, so a 0.1.10 rollback needs a full reindex. albok's 2.01×-vs-2× load check awaits Prime. → `stacks/albok-service/README.md`
 - `[2026-10-06]` **The LiteLLM gateway keeps no request/response content, and `store_prompts_in_spend_logs: false` is now a marked consumer guarantee (comfy-dev, arbo incognito).** Proved with a canary plus positive controls; per-request `no-log` does NOT cover the proxy's own spend-log callback (v1.97 source). irv-ml1 restic also excludes `input/incognito-*` (dry-run proved with a canary and a control). `847424c`, `0f6c7a0`.
 - `[2026-10-06]` **jekyll moved from gen-large to sec-reasoning (Prime)** after its ~180k-token agent loop and Prime's ~140k session overflowed flash-next's KV. → `persistent-memory.d/2026-10-06-gen-large-contention-jekyll.md`
@@ -280,22 +289,8 @@ Every vLLM seat on a shared card (GPU 0, GPU 1) has byte-pinned KV (`--kv-cache-
 - `[2026-09-25]` Created empty private repo `corviduo/norn` (git@gitea.phasefinal.com:corviduo/norn.git) as claude-bot, at brokkr-smithy-dev's relay of the operator's Norn ruling; the `norn-dev` handle is the operator's to declare.
 - `[2026-09-25]` **Reward seat audited (Skywork-Reward-V2-Llama-3.1-8B still #1 of 188 on RewardBench 2; our AWQ ≈ bf16 within noise; double BOS costs ~2.7 pts) and moved fv-ml1 → esh-ml1; esh-ml1 monitoring wired (Beszel+GPU, Kuma, Homepage, Dozzle); Dozzle hub's stale agent IPs fixed; nh3-dev Beszel agent revived.** → `persistent-memory.d/2026-09-25-reward-move-and-esh-ml1-monitoring.md`
 - `[2026-09-25]` **TEI adopted as the fleet embed/rerank engine (Prime); esh-ml1 is the sole gateway backend for `qwen3-embedding` + `reranker`; fv-ml1's vLLM embed/rerank seats retired (~6.1 GB freed); parakeet stays on fv-ml1.** → `persistent-memory.d/2026-09-25-tei-fleet-embed-rerank.md`
-- `[2026-09-24]` **esh-ml1 built — RTX 2000E Ada as CT 110 on esh-pve (host driver 580.178.04 DKMS, loaded live, no reboot), vLLM embed+rerank at measured parity with fv-ml1, wired as LiteLLM `order: 2` failover; dead DB alias `reranker-a3-bge-v2-m3` (pre-relocation IP) repaired.** → `persistent-memory.d/2026-09-24-esh-ml1-embed-rerank.md`
-- `[2026-09-24]` **esh-pve: VM 102 retired, a 14-day hung VFIO process found and cleared, T400 → RTX 2000 Ada; Prime chose LXC + host driver for embed/rerank** (implementation deferred to next session, tracked here + `servers/esh-pve/README.md`). → `persistent-memory.d/2026-09-24-esh-pve-vm102-retired-gpu-swap.md`
-- `[2026-09-24]` **pfi-gx10 AC-restore UEFI patch applied but NOT validated; box OFF until Prime's AC pull 2026-09-25** (tracking: `servers/pfi-gx10/README.md`, `e5197a3`). — `[2026-09-25]` ✅ **VALIDATED**: Prime pulled and replugged AC, and it came up by itself at 1108:54 (`2b49be4`). → `persistent-memory.d/2026-09-24-gx10-ac-restore-patch.md`
-- `[2026-09-24]` **NH3 power outage recovered** — pbs-nh3 had no `onboot` (set), NFS boot race fixed with automount (`1cbde50`), every other Claude session on nh3-dev died. → `persistent-memory.d/2026-09-24-nh3-power-outage-recovery.md`
-- `[2026-09-24]` **Miranda standing order is a repo CLAUDE.md operating parameter** — (`4b29492`, aligned to the global send protocol in `bcf3342`): high-urgency matters go to her, fixed or not… → `persistent-memory.d/2026-09-24-miranda-standing-order-is-a-repo-claude-md-operating.md`
-- `[2026-09-24]` **task-board mothballed** (Prime): container removed on ana-docker, data/image/compose kept, Kuma monitor deleted, task_* instructions removed from CLAUDE.md and the fork template (`e6da607`). Its hooks had sent no traffic in 30 days.
-- `[2026-09-24]` **Military 24-hour Pacific clock times** carried into the Codex/Grok shared bootstrap `docs/fleettools/AGENT-BOOTSTRAP.md` (`ad2b4d9`); Claude seats get it from the global CLAUDE.md.
 - `[2026-09-24]` **Worldtree `admin.memory.forget` stays OFF on demo/personal** until an instance needs it — a destructive erase; enabling it is a per-change operator yes via deploy-wt-config.
-- `[2026-09-24]` **A git checkout under `root:docker` needs `safe.directory` for its deploy user** — the 09-14 normalization (`826a63b`) silently broke yt-voice-clipper's webhook deploy until v0.3.13; fixed on irv-ml1 and recorded in fleet conventions (`eea9eb2`). Sweep found no other case.
-- `[2026-09-23]` **elway sudo uploads land root:root, validated and staged; fleet ownership audit built; 33 mis-owned root files fixed on 10 hosts.** → `persistent-memory.d/2026-09-23-elway-ownership-fix-fleet-audit.md`
-- `[2026-09-23]` **headscale-ddns hardened** — (`fedd4b6`): Cloudflare calls retry and validate every body (`pick()`), no write without both IDs, the run… → `persistent-memory.d/2026-09-23-headscale-ddns-hardened.md`
-- `[2026-09-23]` **esh-docker-vm restic was skipped 09-22..23 by my own Kuma move** — a dead uptime-kuma lookup aborted `pre-backup.sh` under `set -e` (`25e41d2`). → `persistent-memory.d/2026-09-23-esh-docker-vm-restic-was-skipped-09-22-23-by-my-own-kuma.md`
-- `[2026-09-23]` **hermes-gateway restart exit-1 is a Hermes race, not a crash** — the planned-stop watcher consumes the marker before SIGTERM re-runs the handler; drop-in… → `persistent-memory.d/2026-09-23-hermes-gateway-restart-exit-1-is-a-hermes-race-not-a-crash.md`
-- `[2026-09-23]` **Booth link board cleared to 14 durable links** (208 removed: drops, release posts, research links, token-bearing URLs; Scriberr reposted at `scriberr.fv.internal:8080`). Prime's rule: durable debug links only.
 
-- `[2026-09-22]` **safe-rm installed on nh3-dev and delegated fleet-wide to infra-hermes.** — ⚠ The package installs INERT and looks fine — Debian's `/etc/zsh/zprofile` has 0 non-comment lines so the… → `persistent-memory.d/2026-09-22-safe-rm-installed-on-nh3-dev-and-delegated-fleet-wide-to.md`
 
 - `[2026-09-17]` **PARKED lv-krakauer, and the reason is a selection criterion the line was missing: ask whether the author HAS a voice before investigating whether the…** → `persistent-memory.d/2026-09-17-parked-lv-krakauer-and-the-reason-is-a-selection-criterion.md`
 - `[2026-09-17]` **The althing route-declaring SessionStart hook is documented but NOT installed on nh3-dev** — `dev_launch.py` has zero occurrences of "route", no hook declares one, and every live route was hand-declared… → `persistent-memory.d/2026-09-17-the-althing-route-declaring-sessionstart-hook-is-documented.md`
@@ -308,10 +303,14 @@ Every vLLM seat on a shared card (GPU 0, GPU 1) has byte-pinned KV (`--kv-cache-
 
 - `[2026-08-24]` **`nconnect=8` on `/mnt/smithy` — approved but DEFERRED at operator instruction** → `persistent-memory.d/2026-08-24-nconnect-8-on-mnt-smithy-approved-but-deferred-at.md`
 
-_214 older entries archived to archival-memory.md._
+_228 older entries archived to archival-memory.md._
 
 ## Tried and abandoned
 
+- `[2026-10-08]` **NVIDIA's network debian12 repo on trixie via `cuda-keyring`** — trixie's sqv rejects the key's SHA1 binding signature, so `apt update` errors until the list is gone (`dpkg -r` keeps the conffile; it took `dpkg -P`). Fetch the debs and verify InRelease (gpgv) → Packages.gz → sha256 instead. → `servers/fv-ml1/README.md`
+- `[2026-10-08]` **`secret get <name> | head` "to see its shape"** — a single-value secret IS its shape; it printed the fv-ml1 BMC password into the local transcript. Check with `secret list` or `wc -c`, never by printing.
+- `[2026-10-07]` **An idle gate counting with `grep -c … || echo ERR`** — `grep -c` exits 1 on a ZERO count, so the value was "0\nERR" and never equalled "0"; two 75-min LiteLLM restart gates were blind to any quiet window. Use `|| true` and test the gate against a known-idle state first.
+- `[2026-10-07]` **`pgrep -f <pattern>` to check that a process died** — it matched the checking shell's own command line and reported a survivor; check with `ps -eo … | grep '[x]…'`, as the same self-match bit `pkill -f` before.
 - `[2026-10-06]` **Restarting a user-facing service after an idle check that only PRINTS** — `nvidia-smi … | wc -l` said 1 and the same command restarted Scriberr anyway, killing Prime's transcript run. Make the check gate (`&&` / exit). auto-memory `feedback_idle_check_must_gate_the_restart`.
 - `[2026-10-06]` **Aging PBS snapshots by directory mtime** — verify jobs and notes edits rewrite `index.json.blob`, so a 10-04 snapshot read as 10-06. Age by the snapshot NAME. auto-memory `reference_pbs_snapshot_mtime_moves_on_verify`.
 - `[2026-10-06]` **`deploy-stack.sh` on a stack whose host dir holds rollback copies** (albok-service `compose.yaml.pre-*`) — its rsync `--delete` removes them. Upload the single changed file with elway instead.
@@ -338,7 +337,5 @@ _214 older entries archived to archival-memory.md._
 - `[2026-09-25]` **Concluding "not on the UDM" from a port table read 15 s after link-up** — UniFi polls (~60 s); AMT was on port 6 as Prime said. Wait a poll, and prefer a which-port-lists-the-MAC check. auto-memory `feedback_polled_stats_lag_the_event`.
 - `[2026-09-25]` **The esh-pve NVIDIA DKMS recipe on nh3-pve** — it built, then the kernel refused to load it: nh3-pve has SECURE BOOT ON (esh-pve, the same MS-01, has it off). The installer rolled back. The playbook now pre-flights `mokutil --sb-state` (`7ddd116`).
 - `[2026-09-25]` **elway against an unpinned host-key name** — every `when:` hit ssh rc 255, so all 9 steps showed SKIPPED, not failed. Only the verify phase went red. Pin the name (the IP key matched) or use the IP. Untracked elway defect: rc 255 in `when:` should fail.
-- `[2026-09-24]` **Testing "Restore AC Power Loss" with an OS shutdown** — a shutdown stays off BY DESIGN; only pulling and restoring AC tests it. A community README claimed the two are indistinguishable; trusting it cost Prime two trips to the pfi-gx10 power button.
-- `[2026-09-23]` **`booth link --help`** — there is no help flag; it posts `--help` to the operator's link board as a link. Read `booth` with no args for usage.
 
-_121 older entries archived to archival-memory.md._
+_123 older entries archived to archival-memory.md._