memory: snapshot — 2026-10-08 patch night, nh3-svos a737a06 + post office live, albok 0.1.13/0.1.14, gateway glm-5.3, heid sweep incident; 16 entries archived

archival-memory.md

@@ -14735,3 +14735,275 @@ longer takes a full-card seat. That is the trade; it is the operator's call, and
 gateway should move with it either way.
   _Archived 2026-10-06._
 
+## Recent decisions (archived 2026-10-08 batch)
+
+# `[2026-09-24]` esh-ml1 built: RTX 2000E Ada serving embed + rerank, LiteLLM order-2 failover behind fv-ml1
+
+Executes Prime's 2026-09-24 decision (LXC + host driver, not a VFIO VM). Done
+2204–2240 PT the same night, **with no esh-pve reboot**.
+
+**Host (esh-pve).** NVIDIA **580.178.04** open modules via DKMS from NVIDIA's
+`-no-compat32.run` (sha256 checked against NVIDIA's published sum), applied
+live by `playbooks/pve-nvidia-host.yaml`: nouveau was never loaded and the
+T400's vfio ids matched nothing, so nothing held the card. Our own
+`nvidia-persistenced.service` creates `/dev/nvidia{0,ctl,-uvm,-uvm-tools}`
+**before `pve-guests`** (else `pct start` refuses the `devN:` entries). Retired
+`vfio.conf` (moved to `/root/nvidia/`) and `blacklist nvidia`. Initramfs NOT
+rebuilt on purpose (boot path untouched). 580 because the fleet's vLLM
+`v0.24.0` image is CUDA 13.0 (driver ≥ 580); fv-ml1 runs 580.65.06. `.run`
+not an apt repo: one file on both sides makes the host-module/container-lib
+version match true by construction; apt would let an upgrade move one side.
+
+**CT 110 `esh-ml1`** (`playbooks/gpu-lxc.yaml`): unprivileged Debian 12,
+nesting+keyctl, 6c/16G/80G `local-lvm`, `10.0.50.80` VLAN 50 (static; UDM pool
+is `.150–.250`), `startup order=30` after esh-scale/VMs. NVIDIA userspace from
+the same `.run` with `--no-kernel-modules`; docker-ce; nvidia-container-toolkit
+with `no-cgroups=true`. **Docker-in-unprivileged-LXC with the GPU worked first
+try on lxc-pve 6.0.0-2** (no AppArmor workaround needed). Fleet ids per
+`docs/pfi/fleet-conventions.md` (infra-ops 850, docker 851, vh 1000 — vh has
+no key/password yet). **Not in the vzdump job, on purpose** (rebuildable).
+DNS `esh-ml1.esh.internal` synced to all three resolvers.
+
+**Stack `stacks/embed-rerank`:** `vllm-embed` :8001 + `vllm-rerank-bge` :8013,
+same models/flags/vLLM digest (`251eba5cc7c1`) as fv-ml1, 0.20 mem-util each →
+4,823 MiB used of 16,380.
+
+**Parity (the load-bearing measurement; embeddings are model-specific).** 11
+texts × 2 runs per site. Embed cosine FV-vs-ESH median 0.999908 / min 0.999772,
+inside both self-noise floors (FV-vs-FV 0.999927 / 0.999791; ESH-vs-ESH
+0.999911 / 0.999809). Negative control (different texts) 0.07–0.38. Sensitivity
+floor ~2×10⁻⁴ cosine. Rerank |score| max 0.000145 vs FV-FV floor 0.000181,
+identical ranking.
+
+**Gateway.** `qwen3-embedding` and `reranker` each got a second deployment
+(`order: 2`, esh-ml1); fv-ml1 is `order: 1`. LiteLLM v1.97 order-fallback
+proven with throwaway groups (created + deleted): primary refusing → ESH in
++0.15 s (3/3); primary host-down (no ARP) → ESH but **~18.7 s on every call,
+no cooldown** (3/3); dead-only negative control → 500 (3/3). Post-restart
+production traffic is served by fv-ml1 (header `x-litellm-model-api-base`).
+`/health?model=` returned 503 with empty lists for both groups — not
+investigated. Gateway restart: liveliness back in ~52 s.
+
+**Found + fixed:** DB-only alias `reranker-a3-bge-v2-m3` pointed at ana-ml2's
+pre-relocation IP `10.250.50.54` — dead since 2026-09-12 (500 after ~23 s), no
+callers in 7-day spend logs. PATCHed to `10.251.50.54:8013`. Sweep of
+`/model/info` found no other `10.250.50.54` targets.
+
+**Who actually uses these names (spend logs, 7 days to 2026-09-25 05:35Z):**
+`qwen3-embedding` — worldtree-gateway 481 (nh3-dev + corviduo-dev), nevermore
+20; `reranker` — nevermore 13. **No ESH-side consumer called either in 7 days**
+(Open WebUI's RAG embeds only on document upload). So today esh-ml1's value is
+an FV-outage failover for Worldtree + nevermore, not local service for ESH.
+
+**Open:** a direct ESH→esh-ml1 path (survives a mesh outage) would help only
+ESH consumers, which are idle — recommended parked. esh-ml1 not yet in
+Homepage docker.yaml or Beszel. The ~18.7 s host-down failover penalty is
+untuned (router retries/cooldown).
+
+**Speed A/B vs fv-ml1 (2026-09-25 0628–0640 PT, table in `servers/esh-ml1/README.md`).**
+Single queries: a wash (on-box ESH ~9 ms vs FV ~12 ms; from ana-docker ESH is
+faster because it is network-closer). Bulk: ESH 3–10× slower (bulk embed ~50 vs
+~403 passages/s; rerank-20 ~170 vs ~51 ms p50, ~5.7 vs ~55 req/s at conc 8).
+Noise floor 3–5% on throughput. ⇒ failover role confirmed; load-sharing would
+roughly double rerank latency for half the calls.
+
+**TEI bake-off (2026-09-25, `stacks/tei-bakeoff/README.md`).** Prime asked to
+evaluate a purpose-built engine instead of vLLM → HF Text Embeddings Inference
+1.9.4 (Infinity ruled out: last release 2025-08-22, dropped by us in May for no
+Qwen3). Verdict: TEI **matches** (embed cosine vs vLLM-FV median 0.999925, inside
+noise; TEI queries against the OLD vLLM index overlap@10 0.988 vs 0.986 self-noise
+→ no re-embed needed; rerank decisions identical, tail scores move ≤0.019) but is
+**not faster** on the Ada (novel 40 s vs 31 s; tiny queries ~25% faster). It is
+**much lighter**: ~2.6 GB VRAM for both vs vLLM's 4.8 GB budget, 8 GB image vs 30,
+~4 s restart vs ~24 s. Gateway: embed drop-in via `hosted_vllm/`; rerank needs the
+`huggingface/` provider. Both engines left running side by side pending Prime's call.
+  _Archived 2026-10-08._
+
+# `[2026-09-24]` esh-pve: VM 102 retired, a 14-day hung VFIO process found, T400 → RTX 2000 Ada
+
+**VM 102 `esh-vm-workstation` retired permanently** (Prime: "not the machine I
+want running a virtual windows machine"). Final backup
+`pbs-ana:backup/vm/102/2026-09-10T10:32:20Z` (verification ok) set
+**protected** so prune cannot take it; `qm destroy 102 --purge`.
+
+⚠⚠ **Found: `kvm -id 102` had been hung in D state since the 2026-09-10 0332
+vzdump.** vzdump starts a stopped VM (here with the T400 passed through,
+`x-vga=1`) to read its disks; that process never exited. It held **16 GB of
+locked RAM**, the GPU, and two LVs, while `qm status 102` said `stopped` the
+whole time. Same class as the 2026-08-19 hard freeze: VFIO + this box. Only a
+reboot cleared it.
+
+**GPU swap, Prime on site, 2119–2137.** Guests shut down cleanly; host powered
+off; the T400 was removed and an **NVIDIA RTX 2000 / 2000E Ada** installed
+(`01:00.0`, `10de:28b0`, subsystem `10de:1871`). Host back 2136:49; leftover
+LVs removed; all guests auto-started; esh-docker-vm NFS, traefik (0×403) and
+AdGuard verified. The new card has **no driver bound**; `/etc/modprobe.d` still
+lists the T400's vfio ids (`10de:1ff2,10de:10fa`), now matching nothing.
+Commits `719e3fa`, `7cbd4c3`.
+
+**Decision (Prime, 2026-09-24): the card serves embedding + reranking as an
+LXC with the NVIDIA driver on the host — NOT a VFIO VM.** Rationale: both
+esh-pve hangs this year came from VFIO passthrough; LXC avoids it and pins no
+VM RAM. Accepted cost: a driver on the hypervisor, rebuilt on PVE kernel
+updates, host/container versions must match. **Implementation deferred to the
+next session** (tracking: this entry + `servers/esh-pve/README.md`).
+
+**ESH single-route observation:** esh-scale (CT 108, ESH's only mesh subnet
+router) lives on esh-pve, so while esh-pve was down *all* of ESH looked dark
+from outside, including esh-pve-nas and vm-esh-nas, which were up. Untracked by
+operator choice so far — a second ESH route is an idea, not a decision.
+
+**vPro:** the copper cable is in the AMT-capable port (**I226-LM**, `enp89s0`,
+MAC `58:47:ca:76:99:32`), verified at 2500 Mb/s; the I226-V (`enp88s0`) is
+not AMT-capable. Both links set back admin-down on the host. **MEBx
+provisioning deferred by Prime** ("some other time"): Ctrl+P at boot, enable
+manageability, static IP, KVM on with User Opt-in None, activate network
+access; MeshCentral should live on ana-docker/nh3-docker, never on esh-pve.
+  _Archived 2026-10-08._
+
+# `[2026-09-24]` pfi-gx10 "power on after AC loss" — UEFI patch applied, NOT validated
+
+pfi-gx10 (ASUS Ascent GX10, BIOS `GX10DGX.0105.2026.0505.1153`, no BMC) stayed
+off after the NH3 outage. With Prime's approval (after the permission
+classifier first refused it as externally-sourced firmware code), bytes 3–4
+of the 81-byte UEFI var `Setup-ec87d643-eba4-4bb5-a1e5-3f3e36b20da9` were
+changed `0x1d,0x00 → 0x1c,0x01` via `efivar -w`; readback shows exactly those
+two bytes changed. Recipe: community tool `Fheodoor01/DGX-Spark---Power-on-boot-`
+(diff-derived from a power-on unit vs a stay-off unit; author-verified on
+GX10DGX.0103/.0104). **Byte 3's meaning is unknown**; the resulting pair
+matches factory power-on units.
+
+**Original saved twice:** `/var/lib/efi-setup-backup/Setup-ec87d643-20260924-pre-poweron.bin`
+on the box and `servers/pfi-gx10/efi-Setup-ec87d643-20260924-pre-poweron.bin`
+(sha256 `6d6f6f77…`, raw incl. the 4-byte attr header). Revert procedure in
+`servers/pfi-gx10/README.md`. Commit `e5197a3`.
+
+⚠⚠ **An OS shutdown is NOT a test of this setting.** It means "start when
+power RETURNS after a cut"; a `poweroff` is supposed to stay off by design.
+The tool's README claims the two are indistinguishable — false here. I
+trusted it and ran two poweroff "tests" (1421, and 1435 after the reboot the
+README says the firmware needs); both stayed off, each costing Prime a trip to
+the button, and both prove nothing. Prime called it out directly.
+
+**Status:** the box is OFF (the 1435 shutdown) with the patch in place, and
+Prime left NH3. **He will pull and restore AC on 2026-09-25.** Comes up alone
+→ works; drop "NOT YET VALIDATED" from the README. Stays off → he presses the
+button, revert from the backup, and set it in firmware via a PiKVM (he intends
+to add one with ATX control, which also covers remote power-on regardless).
+
+**Validated 2026-09-25 (Prime, relayed by Miranda 1333):** Prime replugged AC and
+the box came up by itself at 1108:54 PT, with no button press. The patch works.
+  _Archived 2026-10-08._
+
+# `[2026-09-24]` NH3 power outage — recovered; three latent gaps closed, one needs a hand
+
+Power at NH3 dropped and returned; every NH3 box booted at 1233–1234 PT. The
+core recovered unaided (post office, headscale, egress proxy, DNS, WAN IP
+unchanged at `70.230.226.88`). Three things did not, each a latent config gap
+that only a site-wide power event exposes:
+
+1. **pbs-nh3 (VM 105) had no `onboot` flag** and stayed down. Set `onboot=1`
+   and started it; datastore NFS mounted (its line uses `bg`). `qm guest exec
+   105` works (guest agent on) — the root path, since infra-ops is not
+   provisioned there.
+2. **NFS boot race.** nh3-nas is the slowest box to serve NFS, so every plain
+   fstab NFS line failed at boot (nh3-docker `/mnt/compose` `/mnt/backup`,
+   nh3-dev `/mnt/backup`). nh3-dev's `/mnt/smithy` already had
+   `x-systemd.automount` and self-healed on its next access (1234 failed, 1309
+   mounted, no hand) — the existence proof. `playbooks/nh3-nfs-automount.yaml`
+   brought the rest to that shape (`_netdev,nofail,x-systemd.automount,
+   x-systemd.mount-timeout=30`), commit `1cbde50`. ⚠ nh3-dev's installer
+   cdrom fstab line is a pre-existing `findmnt --verify` error; the playbook
+   judges only errors a rewrite ADDS, after its first run aborted on it.
+3. **pfi-gx10 (bare metal, no BMC) did not power on** — ASUS ships "Restore AC
+   Power Loss" = stay off. See `[[2026-09-24-gx10-ac-restore-patch]]`.
+
+Also: **every Claude session on nh3-dev died** with the reboot except
+infra-ops, infra-hermes, jekyll. Relaunch is Prime's call (dev-launch).
+`nh3-laser` (VM 104) is on-demand and **stays off** by Prime's ruling;
+`servers/nh3-pve/README.md` now carries an expected-after-power-loss column per
+guest so the next triage does not flag it.
+
+The headscale-ddns failures 1141–1202 were the WAN being down, correctly
+refused (no public v4); every run since 1247 reads `unchanged`.
+  _Archived 2026-10-08._
+
+# `[2026-09-24]` Miranda standing order is a repo CLAUDE.md operating parameter
+
+**Miranda standing order is a repo CLAUDE.md operating parameter** (`4b29492`, aligned to the global send protocol in `bcf3342`): high-urgency matters go to her, fixed or not; URGENT only when it cannot wait (she phones Prime). Channel verified end to end (thread `01M3A0RP4Q8T0KNGH8TMFSNDA6`); it depends on svos + hermes-gateway. Prime's callsign **PRiMe / papa romeo mike** is a name, not an authenticator (`617b759`, `62817a2`).
+  _Archived 2026-10-08._
+
+- `[2026-09-24]` **task-board mothballed** (Prime): container removed on ana-docker, data/image/compose kept, Kuma monitor deleted, task_* instructions removed from CLAUDE.md and the fork template (`e6da607`). Its hooks had sent no traffic in 30 days.
+  _Archived 2026-10-08._
+
+- `[2026-09-24]` **Military 24-hour Pacific clock times** carried into the Codex/Grok shared bootstrap `docs/fleettools/AGENT-BOOTSTRAP.md` (`ad2b4d9`); Claude seats get it from the global CLAUDE.md.
+  _Archived 2026-10-08._
+
+- `[2026-09-24]` **A git checkout under `root:docker` needs `safe.directory` for its deploy user** — the 09-14 normalization (`826a63b`) silently broke yt-voice-clipper's webhook deploy until v0.3.13; fixed on irv-ml1 and recorded in fleet conventions (`eea9eb2`). Sweep found no other case.
+  _Archived 2026-10-08._
+
+# `[2026-09-23]` elway sudo uploads land root:root; fleet ownership audit; 33 files fixed
+
+elway's sudo upload was scp-as-user then `sudo mv`, and mv keeps the owner, so
+every file it installed as root (systemd units, `/etc` configs, root-run
+scripts) landed owned by infra-ops or lkraven. A sudoers drop-in installed that
+way would break sudo outright. Commit `466f7aa`:
+
+- sudo uploads chown to `root:root` by default; `upload.owner:` / `--owner`
+  override; owner refused on non-sudo uploads.
+- chown+chmod act on the STAGED file, then one `mv` publishes it, so a failed
+  chown cannot leave the live path mis-owned. A trap removes the staged file on
+  every exit. A directory dest is refused before anything moves.
+- `mode` was spliced unquoted into the remote root shell; now octal-validated
+  and quoted. `mode`/`owner` must be quoted YAML strings (bare `0644` → 420).
+- `preflight()` resolves every step before any remote action.
+- 24 unit tests; heid bug-hunt "Puck" (Gróa + seat) folded.
+
+`scripts/fleet-ownership-audit.sh` (read-only; exit 0/4/5): tier A root-parsed
+paths (symlinks judged by target), tier X root-run unit Exec paths incl.
+drop-ins, tier B /opt summary. Unprivileged or partial hosts report INCOMPLETE,
+never clean; completion is nonce-marked. Positive controls on nh3-dev fired for
+each tier. **Six hosts have no infra-ops identity** (ana-wg, nh3-nas, pbs-ana,
+pbs-nh3, pfi-postgres, vm-esh-nas) and audit unprivileged.
+
+**33 files chowned to root on 10 hosts** with Prime's approval (the first
+attempt was blocked by the permission classifier): 31 elway-placed (beszel
+agent trio on 6 hosts, esh-vm-db/ana-docker restic hooks and retry drop-ins,
+irv-ml1 units, nh3-dev alert bridge, fv-ml1 retired file) plus 2 non-elway
+root-exec holes (ana-docker `/usr/local/bin/resticprofile` owned by llmuser;
+esh-pve-nas `/usr/local/lib/libigdgmm.so.11.3.1343` uid 1000). Re-audit: 0 of
+the 33 remain; all affected services still active. Left as low-risk and
+reported: ana-docker `/root/.nvm` (lkraven), irv-ml1 `uv`/`uvx`, fv-ml1
+`btop` and a root uv-cache python.
+  _Archived 2026-10-08._
+
+# `[2026-09-23]` headscale-ddns hardened
+
+**headscale-ddns hardened** (`fedd4b6`): Cloudflare calls retry and validate every body (`pick()`), no write without both IDs, the run ends on a confirmation, only a global v4 is published, `curl -q`. Three of four weekly failures were empty zone lookups. heid bug-hunt "Talus" folded (one finding was my own regression).
+  _Archived 2026-10-08._
+
+# `[2026-09-23]` esh-docker-vm restic was skipped 09-22..23 by my own Kuma move
+
+**esh-docker-vm restic was skipped 09-22..23 by my own Kuma move** — a dead uptime-kuma lookup aborted `pre-backup.sh` under `set -e` (`25e41d2`). Then, by Prime's decision, the redundant Paperless pg_dump went too: it had failed auth every night since 2026-04-24 and left a 0-byte dump in every snapshot; the DB is covered at source by esh-vm-db's pg_dumpall (`6e8da46`).
+  _Archived 2026-10-08._
+
+# `[2026-09-23]` hermes-gateway restart exit-1 is a Hermes race, not a crash
+
+**hermes-gateway restart exit-1 is a Hermes race, not a crash** — the planned-stop watcher consumes the marker before SIGTERM re-runs the handler; drop-in `SuccessExitStatus=1` stops the false OnFailure page at zero coverage cost (`c2b0a05`). svos had its own stop-timeout (an open board SSE tab), fixed by svos-dev with `timeout_graceful_shutdown=5`.
+  _Archived 2026-10-08._
+
+- `[2026-09-23]` **Booth link board cleared to 14 durable links** (208 removed: drops, release posts, research links, token-bearing URLs; Scriberr reposted at `scriberr.fv.internal:8080`). Prime's rule: durable debug links only.
+  _Archived 2026-10-08._
+
+# `[2026-09-22]` safe-rm installed on nh3-dev and delegated fleet-wide to infra-hermes.
+
+⭐ **safe-rm installed on nh3-dev and delegated fleet-wide to infra-hermes.** ⚠ The package installs INERT and looks fine — Debian's `/etc/zsh/zprofile` has 0 non-comment lines so the shipped `profile.d` hook never fires under zsh; verify with `command -v rm`, never `dpkg -l`. ⚠ And it does NOT cover the habit that prompted it: measured, it refuses `rm -rf /home` and deletes an unset-variable path without complaint. Blacklist, not heuristic; `set -u` is the actual cover. Wiring ruled: `/etc/bash.bashrc` above the `case $-` guard, not `/etc/environment` (an rc self-guards, smaller blast radius, covers bash-over-ssh).
+  _Archived 2026-10-08._
+
+## Tried and abandoned (archived 2026-10-08 batch)
+
+- `[2026-09-24]` **Testing "Restore AC Power Loss" with an OS shutdown** — a shutdown stays off BY DESIGN; only pulling and restoring AC tests it. A community README claimed the two are indistinguishable; trusting it cost Prime two trips to the pfi-gx10 power button.
+  _Archived 2026-10-08._
+
+- `[2026-09-23]` **`booth link --help`** — there is no help flag; it posts `--help` to the operator's link board as a link. Read `booth` with no args for usage.
+  _Archived 2026-10-08._