esh-pfi-infrastructure/persistent-memory.d/2026-10-04-fleet-on-pve-9-and-pbs-4.md

[2026-10-04] The whole fleet is on PVE 9.2.21 and PBS 4.2.7 — what each host taught

Every hypervisor we run is on PVE 9.2.21 / kernel 7.0.14-20, and both backup servers are on PBS 4.2.7 / Debian 13.7. nh3-pve + esh-nas-pve 2026-10-03; pfi-pve (0656), esh-pve (0742), pbs-nh3 + pbs-ana (1131 / 1140) 2026-10-04; nh3-pve-2 and esh-pve-2 were installed on 9. sfsrv-ana (SureFire, PVE 8) is the client's and was not touched. Plans + logs: docs/runbooks/pfi-pve-pve9-upgrade-plan.md, docs/runbooks/esh-pve-cluster-pve9-upgrade-plan.md, docs/runbooks/pbs-pbs4-upgrade-plan.md. Commits 075ba09, b67ab48 (pfi-pve), 9a6b235, 73129a0 (esh-pve), d0c1bc6 (PBS).

What each one taught

• nh3-pve (10-03): no proxmox-headers-7.0 was installed, so the nvidia DKMS module never built for 7.0 and nh3-ml1 (CT 109) came up with no GPU. Fixed with proxmox-default-headers + headers + dkms autoinstall (playbooks/nh3-pve-pve9-postfix.yaml). On any DKMS host, confirm proxmox-default-headers BEFORE the upgrade and dkms status for the new kernel BEFORE the reboot.
• esh-nas-pve (10-03): /boot is a separate 488 MB LV; with four kernels the second 7.0 initramfs hit "No space left" mid-dist-upgrade. Trim old kernels and check df /boot first.
• pfi-pve booted UEFI from a 2023 GRUB because grub-pc was the installed package (the PVE 7.4 install was never switched to grub-efi-amd64). That is the wiki's "UEFI + root on LVM may fail to boot" trap; prep swapped it and a PVE 8 reboot test proved it before trixie. Check dpkg -l grub-pc grub-efi-amd64 + efibootmgr -v on any PVE host before its next major upgrade.
• A pfi-pve reboot = ~4 min of NH3/ESH↔ANA (ana-scale CT 114 is the only site-to-site path) and ~20 min of the LiteLLM gateway (14-guest boot storm on HDD-backed ospool; LiteLLM's prisma start is slow under it), which also silences Miranda (her base_url is the ANA gateway; recorded in CLAUDE.md, fallback parked as park 98). Break-glass verified: FortiGate WAN SSH infra-ops@38.120.12.42 is tunnel-independent and reaches the iDRAC (Enterprise licence).
• esh-pve's vmbr0 still listed enp87s0, dead since the 2026-09-24 GPU renumbered the igc ports; every boot logged "bridge configuration failed". Fixed to the ports actually forwarding before the upgrade. One reboot (Prime on site, his call over a PVE 8 test): mesh ~1 min, DNS/HA/Zigbee ~2 min; nvidia DKMS built for 7.0 because proxmox-default-headers was already in (the nh3-pve trap).
• ifupdown2 3.3 (PVE 9) makes ifreload -a -s exit 1 on PVE's own bridge-fd 0. Cosmetic: nh3-pve and esh-nas-pve show it and run fine. Pre-reboot checks now fail only on OTHER ifupdown2 output.
• PBS were "PBS on plain Debian" (no proxmox-backup meta package), so pbs3to4 FAILs one check by design; the real requirement was asserted directly. Driven as root through qm guest exec (no infra-ops login then; added after, Prime) — and the first in-band apt died mid-dpkg on BOTH VMs when qemu-guest-agent restarted itself (auto-memory feedback_no_inband_apt_through_guest_agent). Proven after: v3→v4 sync, PVE 9→PBS 4 write (ct/113), v4→v4 sync.
• The PVE 9 vzdump warns on Windows VMs without the Microsoft 2023 Secure Boot certs (ms-cert=2023k). Moot for now: Prime retired VM 101 (DC) and VM 104 (nh3-laser) the same day; enroll with qm enroll-efi-keys before any start.