ops(pfi-pve): upgraded to PVE 9.2.21 / kernel 7.0.14-20; reboot test + post-check recorded; pre-reboot checks corrected for PVE 9

CLAUDE.md

@@ -86,8 +86,10 @@ sent 08:32, her ack at 08:33:29).** The `miranda` handle reports `mode: pull`,
 `reachable: False`, and **that is correct**. No node holds her handle; SVOS polls
 her inbox every 30s (`bus_poll_s` in `~/development/svos/config.yaml`) and relays
 it into her conversation. ⚠ **So this channel depends on `svos.service` (:8770)
-and `hermes-gateway` on nh3-dev.** If an incident takes either of them down,
+and `hermes-gateway` on nh3-dev, and on the LiteLLM gateway on ana-docker**
-Miranda cannot hear you, and even a routine svos restart is about 100s of
+(`10.250.50.70:4000` is the model `base_url` in both `~/.hermes/config.yaml` and the svos config; found during
+the 2026-10-04 pfi-pve upgrade, where every pfi-pve reboot took the gateway out for ~20 min). If an incident takes
+any of them down, Miranda cannot hear you, and even a routine svos restart is about 100s of
 deafness (about 70s of that is vault reads). Reach Prime another way as well (the
 Booth board, or say it in this session).
 

docs/runbooks/pfi-pve-pve9-upgrade-plan.md

@@ -1,5 +1,8 @@
 # pfi-pve (Anaheim): Proxmox VE 8 → 9 upgrade
 
+**STATUS: DONE 2026-10-04 0715 — PVE 9.2.21 / kernel 7.0.14-20, post-check passed. Open: drop the root snapshot
+on/after 2026-10-05 (see Log).**
+
 Requested by Prime 2026-10-04 ~0550 ("drive the ana pve upgrade to proxmox 9"; "proceed" ~0600).
 Siblings: `docs/runbooks/esh-pve-cluster-pve9-upgrade-plan.md` (esh-nas-pve, done 2026-10-03) and
 nh3-pve (done 2026-10-03, `playbooks/nh3-pve-pve9-postfix.yaml`). Official guide:
@@ -99,3 +102,41 @@ execute ping 10.250.250.30     # iDRAC
   `grubx64.efi` rewritten 06:08 (md5 a4a8d8d1… → d9d0a989…, BOOTX64.EFI identical), `/boot/grub/x86_64-efi` refreshed,
   BootOrder 0003 (proxmox) first, four NIC pins in the -43 and -8 initramfs. pve8to9 --full: 0 FAIL, 1 WARN (running
   guests), NOTICE local-lvm autoactivation (local storage, left as is). No guest touched.
+- 0611:51 **PVE 8 reboot test** issued (FortiGate-side ping watcher running, independent of ana-scale). ana-docker
+  stopped answering 0614:20; ana-scale + ana-nas 0617:39; host DOWN 0617:58; **host UP 0620:14** (R750xs POST ~2 min);
+  mesh back 0621:23; ana-docker 0624:22; all 14 guests running by ~0635. **Passed:** booted 6.8.12-43 via the new
+  GRUB, NIC names held, `.31` by DHCP, microcode 0xd000410, pools healthy, no failed units on the host.
+  Services: gitea/vaultwarden 200, PBS 200, ana-nas NFS active (ana-docker `/mnt/backup` remounted), Worldtree
+  demo/personal 200 0635, pinned 0636. ana-docker 44/45 Up (missing only `buildx_buildkit_worldtree-builder0`, an
+  on-demand buildx builder with no restart policy). ⚠ **LiteLLM took until 0634:51** (container up 0627:56, then
+  ~6.5 min in `prisma` toolchain prep + `migrate deploy` under boot-storm load, load avg 11): the gateway, and with
+  it Miranda/Hermes, was out ~20 min, not ~10. Pre-existing, not ours: pfi-pteradactyl `apache2` fails every boot
+  (nginx owns :80) + `systemd-journal-flush` failed.
+- 0637 upgrade playbook: (a first run went to a mistyped `10.250.50.31` and died at elway's sudo probe, nothing
+  executed). 0638 real run: pve8to9 0 FAIL; root snapshot `pve/root-pre-pve9` 15G; trixie sources; plan read =
+  770 installs, 53 removals (t64 library renames + python3-distutils/future/lib2to3; isc-dhcp-client, srvadmin
+  stay); detached dist-upgrade started 0638 (`pve9-dist-upgrade.service`). Targets: pve-manager 9.2.21, kernel
+  7.0.14-20, GRUB 2.12-9+pmx2, ZFS 2.4.4 userland (⚠ no `zpool upgrade`), ceph-common 19.2.6, lxcfs 7.0.
+  sshd restarted mid-upgrade (~0640, connection reset for a few seconds), as expected.
+- 0647 dist-upgrade EXIT 0 (770 packages; GRUB 2.12-9+pmx2 postinst: "Installing for x86_64-efi platform …
+  Installation finished. No error reported."). Pre-reboot check: two of my checks were wrong for PVE 9 and were
+  fixed, not waived: (1) the GRUB binary carries no greppable version string, so the postinst's own report is the
+  proof; (2) ifupdown2 3.3 flags PVE's own `bridge-fd 0` as out of range and `ifreload -a -s` exits 1 on that
+  warning alone. Positive control: nh3-pve and esh-nas-pve (same ifupdown2 3.3.0-1+pmx12, same line) show the same
+  warning and run vmbr0 fine (STP off, forward_delay 0). The check now fails on any OTHER ifupdown2 output (tested
+  both ways). Re-run: 7/7 OK. Enterprise repo set `Enabled: false`. Snapshot COW 40% (≈6 G) after the upgrade.
+- 0648:56 **PVE 9 reboot** issued. ana-docker stopped answering 0650:32; mesh down 0653:51; host DOWN 0654:10;
+  **host UP 0656:45**; mesh back 0657:52; all 14 guests running 0704 (boot storm: host load 37, ospool HDD
+  read-IOPS-bound with a cold ARC); dockerd on ana-docker active 0708; LiteLLM 0711:25; Worldtree ×3 0712:55;
+  OpenWebUI healthy 0715.
+- **Post-check PASSED (0715):** pve-manager 9.2.21, kernel 7.0.14-20, booted via GRUB 2.12; NIC names held, `.31`
+  by DHCP, vmbr0 STP off; NASPool + ospool ONLINE (⚠ ZFS 2.4.4 now — do NOT `zpool upgrade` until 9 is kept);
+  no failed host units; chrony synced; all 5 storages active incl. the pbs-ana datastore; pve8to9 0 FAIL / 0 WARN
+  (NOTICE: local-lvm autoactivation, local storage, left). Gateway embed (dim 1024) + `gen-small` chat OK;
+  gitea/vaultwarden 200; PBS 200; TacticalRMM 200; pfi-postgres :5432, rest-server-ana :8000 open; ana-nas NFS
+  active; ana-docker 44/45 Up (again only the on-demand `buildx_buildkit_worldtree-builder0`); nag patch re-applied.
+- **Outage actually paid, per reboot:** NH3/ESH↔Anaheim ~4 min; host ~2.5 min; the LiteLLM gateway (and with it
+  Miranda/Hermes/SVOS) **~20-21 min** each time (0614-0635, 0650-0711), because ana-docker's dockerd and LiteLLM's
+  prisma start are slow under the 14-guest boot storm on HDD-backed ospool.
+- Open: drop `pve/root-pre-pve9` (`lvremove -y pve/root-pre-pve9`) after ~24 h of clean running, i.e. on/after
+  2026-10-05; sooner if `lvs` Data% passes 80 (a full thick snapshot only invalidates itself, origin unharmed).

persistent-memory.md

@@ -1,6 +1,6 @@
 # Persistent memory — eshpfi-management
 
-_Last updated: 2026-10-04 ~0546 PT (albok-service 0.1.6 live after Prime's push go; esh-nas-pve + nh3-pve on PVE 9; QNAP LCD fixed, awaiting Prime's look; NH3 Mac mini asleep, fixed on site today.)_
+_Last updated: 2026-10-04 ~0720 PT (pfi-pve on PVE 9; albok-service 0.1.6 live after Prime's push go; esh-nas-pve + nh3-pve on PVE 9; QNAP LCD fixed, awaiting Prime's look; NH3 Mac mini asleep, fixed on site today.)_
 
 > **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its
 > `Written:` stamp is under **8 hours** old, read it (it carries the in-flight
@@ -119,6 +119,7 @@ _As of 2026-10-04 ~0546 PT. The newest subsection is first; older subsections ca
 
 ### 2026-10-03/04: live now (as of 2026-10-04 ~0546 PT)
 
+- **pfi-pve (Anaheim): DONE, PVE 9.2.21 / kernel 7.0.14-20 since 0656 2026-10-04** (Prime: "drive it", "proceed"). Prep fixed a real boot trap: the box boots UEFI but had `grub-pc`, so the ESP GRUB was the 2023 install's; now `grub-efi-amd64`. NIC pins, microcode, a PVE 8 reboot test (passed), then trixie. All 14 guests + services verified 0715. ⚠ Each pfi-pve reboot = ~4 min of NH3/ESH↔ANA (ana-scale) and **~20 min of LiteLLM, which also silences Miranda**. Break-glass = FortiGate WAN SSH `infra-ops@38.120.12.42` (verified). ⚠ ifupdown2 3.3 `ifreload -a -s` exits 1 on PVE's own `bridge-fd 0` — cosmetic (same on nh3-pve/esh-nas-pve). **Open: `lvremove -y pve/root-pre-pve9` on/after 2026-10-05** (asked infra-hermes); no `zpool upgrade`. → `docs/runbooks/pfi-pve-pve9-upgrade-plan.md`
 - **nh3-pve: DONE. ON PVE 9.2.21 / kernel 7.0.14-20 (booted 1402).** Post-check: every guest is back (the stopped 104/108 are as before), ZFS healthy, DNS, post office, albok, svos/hermes, mesh and AMT all OK. ⚠ **nh3-ml1 (CT 109) is DOWN:** no `proxmox-headers-7.0` was installed, so the nvidia 580.178.04 DKMS module was never built for 7.0, `nvidia-persistenced` failed, and CT 109 has no /dev/nvidia*. **FIXED 1409 (Prime go, `playbooks/nh3-pve-pve9-postfix.yaml`):** installed `proxmox-default-headers` + headers-7.0.14-20, nvidia DKMS built for 7.0 (580.178.04 compiles fine), nh3-ml1 back (TEI :8001/:8013 healthy, embed dim 1024); intel-microcode installed (non-free-firmware added; takes effect next boot); `pve-enterprise.sources` off; pve8to9 now 0 FAIL / 1 WARN (dkms, benign). VM 102 snapshot `pre-rootgrow-20261002` DELETED (Prime go). No `pre-pve9` snapshot had been taken. **POST-CHECK DONE: nh3-pve clean.** ⚠ Lesson: on a PVE host with DKMS, check that `proxmox-default-headers` is installed BEFORE a major upgrade, or the new kernel boots with no module.
 - **nh3-pve-2** (MS-03, NH3) is live at 10.100.250.62 on nh3-mgmt:
   - X710 `nic3` on USW port 25;

playbooks/pfi-pve-pve9-prereboot.yaml

@@ -45,7 +45,8 @@ verify:
       set -e
       ls -la --time-style=long-iso /boot/efi/EFI/proxmox/ /boot/efi/EFI/BOOT/
       test "$(md5sum < /boot/efi/EFI/proxmox/grubx64.efi)" = "$(md5sum < /boot/efi/EFI/BOOT/BOOTX64.EFI)"
-      strings /boot/efi/EFI/proxmox/grubx64.efi | grep -m1 -E '^2\.1[2-9]' || strings /boot/efi/EFI/proxmox/grubx64.efi | grep -m1 -i 'grub 2\.1'
+      # The binary carries no greppable version string; the trixie grub-efi-amd64 postinst's own report is the proof.
+      grep -q 'Installation finished. No error reported.' /root/pve9-upgrade.log
       first=$(efibootmgr | sed -n 's/^BootOrder: \([0-9A-F]*\).*/\1/p')
       efibootmgr -v | grep -E "^Boot${first}\*? " | grep -qi 'EFI\\proxmox\\grubx64.efi'
     changed_when: "false"
@@ -55,7 +56,13 @@ verify:
       set -e
       dpkg-query -W -f='${Status}' isc-dhcp-client | grep -q 'install ok installed' || dpkg-query -W -f='${Status}' dhcpcd-base | grep -q 'install ok installed'
       dpkg-query -W -f='${Status}' ifupdown2 | grep -q 'install ok installed'
-      ifreload -a -s
+      # ifupdown2 3.3 (PVE 9) flags PVE's own `bridge-fd 0` as out of range and `-s` exits 1 on that warning alone.
+      # nh3-pve and esh-nas-pve show the same warning and booted with vmbr0 fine (STP off, forward_delay 0), so
+      # only a warning OTHER than that one fails this check.
+      out=$(ifreload -a -s 2>&1) || true
+      echo "$out"
+      # (an `if`, not a bare `! pipeline`: set -e ignores a negated pipeline, so that form could never fail)
+      if echo "$out" | grep -v 'bridge-fd: value of out range "0"' | grep -q .; then echo "unexpected ifupdown2 output"; exit 1; fi
       test -z "$(grep -vE '^\s*(#|$)' /etc/sysctl.conf 2>/dev/null)"
     changed_when: "false"
   - name: pve8to9 --full (post-upgrade checks)

servers/pfi-pve/README.md

@@ -16,7 +16,11 @@ Proxmox VE hypervisor for the Anaheim colo (`pve.phasefinal.com`). Hosts every V
 - **Chassis:** Dell PowerEdge R750xs (2U rackmount)
 - **CPU:** Intel Xeon Silver 4310 @ 2.10 GHz
 - **RAM:** 188 GB
-- **Kernel:** `6.8.12-8-pve` (Proxmox 8.x)
+- **PVE / kernel:** 9.2.21 / `7.0.14-20-pve` since 2026-10-04 (upgraded from 8.3.5; `docs/runbooks/pfi-pve-pve9-upgrade-plan.md`)
+- **Boot:** UEFI → `\EFI\proxmox\grubx64.efi` on the BOSS card; `grub-efi-amd64` (was wrongly `grub-pc` until 2026-10-04). NIC names pinned by MAC (`/etc/systemd/network/10-pin-*.link`). Mgmt IP is DHCP on `vmbr0.250` (FortiGate lease)
+- **iDRAC9:** Enterprise licence (virtual console), fw 6.10.30.00; creds `secret get pfi-pve/idrac-infra-ops` (user `infra-ops`)
+- ⚠ **PSU1 has no input power** (repeatedly since 2026-04-23, latest 2026-09-11); running on PSU2 alone
+- ⚠ **Rebooting this host takes all of Anaheim off the fleet for ~4 min** (CT 114 ana-scale is the only NH3/ESH↔ANA path) **and the LiteLLM gateway off for ~20 min** (boot storm). Out-of-band way in: FortiGate WAN SSH `infra-ops@38.120.12.42` (see the upgrade plan § Break-glass)
 - **Storage:** local `pve-root` at 65G (49% used) + `pve` LVM pool for VM disks
 
 ## What it runs