ops(pfi-pve): upgraded to PVE 9.2.21 / kernel 7.0.14-20; reboot test + post-check recorded; pre-reboot checks corrected for PVE 9

docs/runbooks/pfi-pve-pve9-upgrade-plan.md

@@ -1,5 +1,8 @@
 # pfi-pve (Anaheim): Proxmox VE 8 → 9 upgrade
 
+**STATUS: DONE 2026-10-04 0715 — PVE 9.2.21 / kernel 7.0.14-20, post-check passed. Open: drop the root snapshot
+on/after 2026-10-05 (see Log).**
+
 Requested by Prime 2026-10-04 ~0550 ("drive the ana pve upgrade to proxmox 9"; "proceed" ~0600).
 Siblings: `docs/runbooks/esh-pve-cluster-pve9-upgrade-plan.md` (esh-nas-pve, done 2026-10-03) and
 nh3-pve (done 2026-10-03, `playbooks/nh3-pve-pve9-postfix.yaml`). Official guide:
@@ -99,3 +102,41 @@ execute ping 10.250.250.30     # iDRAC
   `grubx64.efi` rewritten 06:08 (md5 a4a8d8d1… → d9d0a989…, BOOTX64.EFI identical), `/boot/grub/x86_64-efi` refreshed,
   BootOrder 0003 (proxmox) first, four NIC pins in the -43 and -8 initramfs. pve8to9 --full: 0 FAIL, 1 WARN (running
   guests), NOTICE local-lvm autoactivation (local storage, left as is). No guest touched.
+- 0611:51 **PVE 8 reboot test** issued (FortiGate-side ping watcher running, independent of ana-scale). ana-docker
+  stopped answering 0614:20; ana-scale + ana-nas 0617:39; host DOWN 0617:58; **host UP 0620:14** (R750xs POST ~2 min);
+  mesh back 0621:23; ana-docker 0624:22; all 14 guests running by ~0635. **Passed:** booted 6.8.12-43 via the new
+  GRUB, NIC names held, `.31` by DHCP, microcode 0xd000410, pools healthy, no failed units on the host.
+  Services: gitea/vaultwarden 200, PBS 200, ana-nas NFS active (ana-docker `/mnt/backup` remounted), Worldtree
+  demo/personal 200 0635, pinned 0636. ana-docker 44/45 Up (missing only `buildx_buildkit_worldtree-builder0`, an
+  on-demand buildx builder with no restart policy). ⚠ **LiteLLM took until 0634:51** (container up 0627:56, then
+  ~6.5 min in `prisma` toolchain prep + `migrate deploy` under boot-storm load, load avg 11): the gateway, and with
+  it Miranda/Hermes, was out ~20 min, not ~10. Pre-existing, not ours: pfi-pteradactyl `apache2` fails every boot
+  (nginx owns :80) + `systemd-journal-flush` failed.
+- 0637 upgrade playbook: (a first run went to a mistyped `10.250.50.31` and died at elway's sudo probe, nothing
+  executed). 0638 real run: pve8to9 0 FAIL; root snapshot `pve/root-pre-pve9` 15G; trixie sources; plan read =
+  770 installs, 53 removals (t64 library renames + python3-distutils/future/lib2to3; isc-dhcp-client, srvadmin
+  stay); detached dist-upgrade started 0638 (`pve9-dist-upgrade.service`). Targets: pve-manager 9.2.21, kernel
+  7.0.14-20, GRUB 2.12-9+pmx2, ZFS 2.4.4 userland (⚠ no `zpool upgrade`), ceph-common 19.2.6, lxcfs 7.0.
+  sshd restarted mid-upgrade (~0640, connection reset for a few seconds), as expected.
+- 0647 dist-upgrade EXIT 0 (770 packages; GRUB 2.12-9+pmx2 postinst: "Installing for x86_64-efi platform …
+  Installation finished. No error reported."). Pre-reboot check: two of my checks were wrong for PVE 9 and were
+  fixed, not waived: (1) the GRUB binary carries no greppable version string, so the postinst's own report is the
+  proof; (2) ifupdown2 3.3 flags PVE's own `bridge-fd 0` as out of range and `ifreload -a -s` exits 1 on that
+  warning alone. Positive control: nh3-pve and esh-nas-pve (same ifupdown2 3.3.0-1+pmx12, same line) show the same
+  warning and run vmbr0 fine (STP off, forward_delay 0). The check now fails on any OTHER ifupdown2 output (tested
+  both ways). Re-run: 7/7 OK. Enterprise repo set `Enabled: false`. Snapshot COW 40% (≈6 G) after the upgrade.
+- 0648:56 **PVE 9 reboot** issued. ana-docker stopped answering 0650:32; mesh down 0653:51; host DOWN 0654:10;
+  **host UP 0656:45**; mesh back 0657:52; all 14 guests running 0704 (boot storm: host load 37, ospool HDD
+  read-IOPS-bound with a cold ARC); dockerd on ana-docker active 0708; LiteLLM 0711:25; Worldtree ×3 0712:55;
+  OpenWebUI healthy 0715.
+- **Post-check PASSED (0715):** pve-manager 9.2.21, kernel 7.0.14-20, booted via GRUB 2.12; NIC names held, `.31`
+  by DHCP, vmbr0 STP off; NASPool + ospool ONLINE (⚠ ZFS 2.4.4 now — do NOT `zpool upgrade` until 9 is kept);
+  no failed host units; chrony synced; all 5 storages active incl. the pbs-ana datastore; pve8to9 0 FAIL / 0 WARN
+  (NOTICE: local-lvm autoactivation, local storage, left). Gateway embed (dim 1024) + `gen-small` chat OK;
+  gitea/vaultwarden 200; PBS 200; TacticalRMM 200; pfi-postgres :5432, rest-server-ana :8000 open; ana-nas NFS
+  active; ana-docker 44/45 Up (again only the on-demand `buildx_buildkit_worldtree-builder0`); nag patch re-applied.
+- **Outage actually paid, per reboot:** NH3/ESH↔Anaheim ~4 min; host ~2.5 min; the LiteLLM gateway (and with it
+  Miranda/Hermes/SVOS) **~20-21 min** each time (0614-0635, 0650-0711), because ana-docker's dockerd and LiteLLM's
+  prisma start are slow under the 14-guest boot storm on HDD-backed ospool.
+- Open: drop `pve/root-pre-pve9` (`lvremove -y pve/root-pre-pve9`) after ~24 h of clean running, i.e. on/after
+  2026-10-05; sooner if `lvs` Data% passes 80 (a full thick snapshot only invalidates itself, origin unharmed).