# `[2026-10-04]` The whole fleet is on PVE 9.2.21 and PBS 4.2.7 — what each host taught

**Every hypervisor we run is on PVE 9.2.21 / kernel 7.0.14-20, and both backup servers are on PBS 4.2.7 / Debian 13.7.**
nh3-pve + esh-nas-pve 2026-10-03; pfi-pve (0656), esh-pve (0742), pbs-nh3 + pbs-ana (1131 / 1140) 2026-10-04; nh3-pve-2
and esh-pve-2 were installed on 9. sfsrv-ana (SureFire, PVE 8) is the client's and was not touched. Plans + logs:
`docs/runbooks/pfi-pve-pve9-upgrade-plan.md`, `docs/runbooks/esh-pve-cluster-pve9-upgrade-plan.md`,
`docs/runbooks/pbs-pbs4-upgrade-plan.md`. Commits 075ba09, b67ab48 (pfi-pve), 9a6b235, 73129a0 (esh-pve), d0c1bc6 (PBS).

## What each one taught

- **nh3-pve (10-03): no `proxmox-headers-7.0` was installed, so the nvidia DKMS module never built for 7.0 and nh3-ml1
  (CT 109) came up with no GPU.** Fixed with `proxmox-default-headers` + headers + `dkms autoinstall`
  (`playbooks/nh3-pve-pve9-postfix.yaml`). On any DKMS host, confirm `proxmox-default-headers` BEFORE the upgrade and
  `dkms status` for the new kernel BEFORE the reboot.
- **esh-nas-pve (10-03): `/boot` is a separate 488 MB LV**; with four kernels the second 7.0 initramfs hit "No space left"
  mid-dist-upgrade. Trim old kernels and check `df /boot` first.
- **pfi-pve booted UEFI from a 2023 GRUB because `grub-pc` was the installed package** (the PVE 7.4 install was never
  switched to `grub-efi-amd64`). That is the wiki's "UEFI + root on LVM may fail to boot" trap; prep swapped it and a
  PVE 8 reboot test proved it before trixie. Check `dpkg -l grub-pc grub-efi-amd64` + `efibootmgr -v` on any PVE host
  before its next major upgrade.
- **A pfi-pve reboot = ~4 min of NH3/ESH↔ANA (ana-scale CT 114 is the only site-to-site path) and ~20 min of the LiteLLM
  gateway** (14-guest boot storm on HDD-backed ospool; LiteLLM's prisma start is slow under it), which also silences
  Miranda (her `base_url` is the ANA gateway; recorded in CLAUDE.md, fallback parked as park 98). Break-glass verified:
  FortiGate WAN SSH `infra-ops@38.120.12.42` is tunnel-independent and reaches the iDRAC (Enterprise licence).
- **esh-pve's vmbr0 still listed `enp87s0`**, dead since the 2026-09-24 GPU renumbered the igc ports; every boot logged
  "bridge configuration failed". Fixed to the ports actually forwarding before the upgrade. One reboot (Prime on site,
  his call over a PVE 8 test): mesh ~1 min, DNS/HA/Zigbee ~2 min; nvidia DKMS built for 7.0 because
  `proxmox-default-headers` was already in (the nh3-pve trap).
- **ifupdown2 3.3 (PVE 9) makes `ifreload -a -s` exit 1 on PVE's own `bridge-fd 0`.** Cosmetic: nh3-pve and esh-nas-pve
  show it and run fine. Pre-reboot checks now fail only on OTHER ifupdown2 output.
- **PBS were "PBS on plain Debian"** (no `proxmox-backup` meta package), so `pbs3to4` FAILs one check by design; the
  real requirement was asserted directly. Driven as root through `qm guest exec` (no infra-ops login then; added after,
  Prime) — and the first in-band `apt` died mid-dpkg on BOTH VMs when qemu-guest-agent restarted itself (auto-memory
  `feedback_no_inband_apt_through_guest_agent`). Proven after: v3→v4 sync, PVE 9→PBS 4 write (ct/113), v4→v4 sync.
- **The PVE 9 vzdump warns on Windows VMs without the Microsoft 2023 Secure Boot certs** (`ms-cert=2023k`). Moot for
  now: Prime retired VM 101 (DC) and VM 104 (nh3-laser) the same day; enroll with `qm enroll-efi-keys` before any start.