vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity
Files
8758c359bc2c7bfbc0a78e18bfd0316efd435f51
esh-pfi-infrastructure/docs/runbooks/pfi-pve-pve9-upgrade-plan.md
T

12 KiB
Raw Blame History

pfi-pve (Anaheim): Proxmox VE 8 → 9 upgrade

STATUS: DONE 2026-10-04 0715 — PVE 9.2.21 / kernel 7.0.14-20, post-check passed. Open: drop the root snapshot on/after 2026-10-05 (see Log).

Requested by Prime 2026-10-04 ~0550 ("drive the ana pve upgrade to proxmox 9"; "proceed" ~0600). Siblings: docs/runbooks/esh-pve-cluster-pve9-upgrade-plan.md (esh-nas-pve, done 2026-10-03) and nh3-pve (done 2026-10-03, playbooks/nh3-pve-pve9-postfix.yaml). Official guide: pve.proxmox.com/wiki/Upgrade_from_8_to_9 (re-read 2026-10-04).

The host (read live 2026-10-04 0550-0610)
HW Dell PowerEdge R750xs, Xeon Silver 4310, 188 GB, BIOS 1.9.2; iDRAC9 6.10.30.00 at 10.250.250.30, Enterprise licence (virtual console)
PVE / kernel 8.3.5 / 6.8.12-8, up 22 days (booted ~2026-09-11)
Cluster single node (no corosync.conf)
Root ext4 on LVM pve/root 65.6 G (31 G free), /boot is on root; VG pve 16 G free
Boot UEFI, GRUB from \EFI\proxmox\grubx64.efi on the BOSS card (sdk2), but the grub-pc (BIOS) meta package was installed — the EFI binary and /boot/grub/x86_64-efi dated from the 2023-03-30 PVE 7.4 install. Secure Boot off
Network vmbr0 = eno8303 + eno8403 (tg3) + eno12399np0 + eno12409np1 (bnxt_en), STP off, only eno8303 has carrier. Mgmt IP 10.250.250.31 is DHCP on vmbr0.250 (FortiGate lease, isc-dhcp-client)
Storage ZFS ospool (every guest disk but pbs-ana's), ZFS NASPool raidz2 (rebuilt 2026-09-05), local-lvm thin (pbs-ana), local (19 G of ISOs)
Extras No DKMS, no NVIDIA. Dell srvadmin-hapi + srvadmin-idracadm7 11.4 (local racadm) from a hand-installed .deb; Debian ceph-common 16.2 (client only)
Health PSU1 has had no input power since 2026-09-11 23:31 (and repeatedly since 2026-04-23); running on PSU2 alone. Known and accepted (pfi-pve-naspool-rebuild.md). Not a blocker; a reboot does not change it
Backups vzdump 03:00 all guests except 100/109/114 OK today; CT 109 22:00 OK last night

Guests (all onboot=1): VMs 100 pbs-ana, 101 PFI-ANA-DC (Win), 102 ana-docker, 103 SlaveBot (Win), 104 Mongo, 105 Postgres, 106 corviduo-dev, 107 pteradactyl, 110 webhost, 111 tacticalrmm; CTs 109 ana-nas (NFS, order 1), 112 ana-filebot, 113 ana-wg, 114 ana-scale (order 1).

What a pfi-pve reboot takes down — all of Anaheim, for ~15 min each time

  • NH3 and ESH lose Anaheim entirely. CT 114 ana-scale is the only site-to-site path since the 2026-09-06 headscale cutover (IPsec dormant). That includes the iDRAC and the FortiGate's LAN side.
  • ana-docker: the LiteLLM gateway (every agent's model calls, including Miranda/Hermes and SVOS, whose base_url is 10.250.50.70:4000 — Miranda is deaf for the duration), Traefik public endpoints, gitea, vaultwarden, rest-server-ana, Langfuse, Beszel/Dozzle hubs, RustDesk.
  • pfi-postgres + Mongo, corviduo-dev (Worldtree demo/personal/pinned), PBS-ANA, TacticalRMM/MeshCentral, AD DC.
  • Not affected: the post office (nh3-docker), nh3-dev, ESH and NH3 local services, fv-ml1.

Findings and what prep did about them (playbooks/pfi-pve-pve9-prep.yaml, no downtime)

  1. grub-pc on a UEFI, root-on-LVM box = the wiki's "UEFI systems with root on LVM may fail to boot". Prep installs grub-efi-amd64 (removes grub-pc) with force_efi_extra_removable, so the ESP binary and the removable copy are rewritten and stay in step with /boot/grub from now on. Kept: tar of /etc, /boot/grub and the ESP in /root/pve9-prep-20261004/; the 2023 binary also at EFI/proxmox-2023/ (last resort only — an old core loading new modules can fail).
  2. NIC names pinned by MAC (/etc/systemd/network/10-pin-*.link + initramfs), all four ports. The bnxt np0/np1 suffix is the kind of name a new kernel changes; a renamed eno8303 = vmbr0 without carrier.
  3. Latest 8.4 (8.4.21, kernel 6.8.12-43), non-free-firmware + intel-microcode. systemd-boot was never installed.
  4. Not changed: the DHCP mgmt address. The pre-reboot check asserts a DHCP client survives the upgrade.

Sequence

  1. Prep (no downtime). ✅ see Log.
  2. Reboot test on PVE 8 (~15 min outage). Validates the new GRUB, the pins, kernel -43 and microcode while the box is still on 8 and /boot still matches a bookworm GRUB, so a failure there is a smaller fix.
  3. playbooks/pfi-pve-pve9-upgrade.yaml (guests stay up): guards, LVM snapshot pve/root-pre-pve9 (15 G), trixie sources (deb822 proxmox.sources), reads the dist-upgrade plan and refuses if it would remove a boot/network package, then a DETACHED dist-upgrade (/root/pve9-upgrade.log).
  4. playbooks/pfi-pve-pve9-prereboot.yaml: enterprise repo off; asserts EXIT 0, dpkg clean, PVE 9, initramfs has the pins + dm-snapshot, ESP rewritten + proxmox first in BootOrder, DHCP client present, ifreload -a -s parses, pve8to9.
  5. Reboot (~15 min outage). Post-check: all 14 guests, ana-scale/mesh, NFS, PBS datastore, LiteLLM, gitea, vaultwarden, Worldtree ×3, container counts against the pre-reboot baseline.
  6. Drop the LVM snapshot once 9 is confirmed (it slows writes to / while it exists).

Break-glass if a reboot does not come back (verified reachable 2026-10-04 0603)

Nothing in Anaheim answers from NH3 while ana-scale is down, so the way in is the FortiGate's WAN SSH (the 2026-09-06 safety net: wan1 allowaccess ping ssh, infra-ops trusthost = NH3 70.230.226.88 + ESH):

SSHPASS=$(secret get fortigate/ana-gw-infra-ops-password) sshpass -e ssh infra-ops@38.120.12.42
execute ping 10.250.250.31     # host up?          (0603: FortiGate→iDRAC 0.3 ms)
execute ping 10.250.50.45      # ana-scale up?
execute ping 10.250.250.30     # iDRAC
  • Host up, ana-scale not: wait for CT 114 (order 1); if it stays down, start it via the iDRAC/host once reachable.
  • Host not up: restore the old NH3↔ANA IPsec path (the documented un-dormant procedure, docs/pfi/headscale-mesh-plan.md § CUTOVER EXECUTED): on the FortiGate config router static / edit 20 / set status disable (the 10.100.0.0/16 → 10.250.50.45 mesh route; route 8 via pfi-ana-nh3 then wins), and on the NH3 UDM enable pfi-nh3-ana and disable the 10.250.0.0/16 → 10.100.50.46 static route. Tunnel and mesh route must never be live for the same /16 at once. Then the iDRAC (Redfish creds pfi-pve/idrac-infra-ops, verified 0601) gives power, SEL, one-time boot and screen capture; Prime can use the virtual console. Undo both edges in reverse once ana-scale is back.

Rollback

  • After step 3, before or after the reboot: lvconvert --merge pve/root-pre-pve9, reboot (the merge runs at activation; from a rescue ISO via iDRAC virtual media if 9 does not boot). Covers / and /boot, not the ESP: if GRUB on the ESP is the problem, boot the rescue ISO and grub-install from the merged root.
  • Guest storage (ospool, NASPool, local-lvm) is not touched by the host upgrade. Never zpool upgrade ospool or NASPool after the upgrade until 9 has been kept.

Log (PT)

  • 0550-0610 recon (read-only); plan written. Positive controls: iDRAC Redfish login OK 0601; FortiGate WAN SSH login OK 0603; FortiGate → iDRAC + ana-scale ping OK.
  • 0601-0614 prep done (pfi-pve-pve9-prep.yaml, elway overall CHANGED, 0 failed): PVE 8.4.21, kernel 6.8.12-43 installed (grub.cfg default), intel-microcode 3.20251111.1, grub-efi-amd64 2.06-13+pmx7 in / grub-pc out, ESP grubx64.efi rewritten 06:08 (md5 a4a8d8d1… → d9d0a989…, BOOTX64.EFI identical), /boot/grub/x86_64-efi refreshed, BootOrder 0003 (proxmox) first, four NIC pins in the -43 and -8 initramfs. pve8to9 --full: 0 FAIL, 1 WARN (running guests), NOTICE local-lvm autoactivation (local storage, left as is). No guest touched.
  • 0611:51 PVE 8 reboot test issued (FortiGate-side ping watcher running, independent of ana-scale). ana-docker stopped answering 0614:20; ana-scale + ana-nas 0617:39; host DOWN 0617:58; host UP 0620:14 (R750xs POST ~2 min); mesh back 0621:23; ana-docker 0624:22; all 14 guests running by ~0635. Passed: booted 6.8.12-43 via the new GRUB, NIC names held, .31 by DHCP, microcode 0xd000410, pools healthy, no failed units on the host. Services: gitea/vaultwarden 200, PBS 200, ana-nas NFS active (ana-docker /mnt/backup remounted), Worldtree demo/personal 200 0635, pinned 0636. ana-docker 44/45 Up (missing only buildx_buildkit_worldtree-builder0, an on-demand buildx builder with no restart policy). ⚠ LiteLLM took until 0634:51 (container up 0627:56, then ~6.5 min in prisma toolchain prep + migrate deploy under boot-storm load, load avg 11): the gateway, and with it Miranda/Hermes, was out ~20 min, not ~10. Pre-existing, not ours: pfi-pteradactyl apache2 fails every boot (nginx owns :80) + systemd-journal-flush failed.
  • 0637 upgrade playbook: (a first run went to a mistyped 10.250.50.31 and died at elway's sudo probe, nothing executed). 0638 real run: pve8to9 0 FAIL; root snapshot pve/root-pre-pve9 15G; trixie sources; plan read = 770 installs, 53 removals (t64 library renames + python3-distutils/future/lib2to3; isc-dhcp-client, srvadmin stay); detached dist-upgrade started 0638 (pve9-dist-upgrade.service). Targets: pve-manager 9.2.21, kernel 7.0.14-20, GRUB 2.12-9+pmx2, ZFS 2.4.4 userland (⚠ no zpool upgrade), ceph-common 19.2.6, lxcfs 7.0. sshd restarted mid-upgrade (~0640, connection reset for a few seconds), as expected.
  • 0647 dist-upgrade EXIT 0 (770 packages; GRUB 2.12-9+pmx2 postinst: "Installing for x86_64-efi platform … Installation finished. No error reported."). Pre-reboot check: two of my checks were wrong for PVE 9 and were fixed, not waived: (1) the GRUB binary carries no greppable version string, so the postinst's own report is the proof; (2) ifupdown2 3.3 flags PVE's own bridge-fd 0 as out of range and ifreload -a -s exits 1 on that warning alone. Positive control: nh3-pve and esh-nas-pve (same ifupdown2 3.3.0-1+pmx12, same line) show the same warning and run vmbr0 fine (STP off, forward_delay 0). The check now fails on any OTHER ifupdown2 output (tested both ways). Re-run: 7/7 OK. Enterprise repo set Enabled: false. Snapshot COW 40% (≈6 G) after the upgrade.
  • 0648:56 PVE 9 reboot issued. ana-docker stopped answering 0650:32; mesh down 0653:51; host DOWN 0654:10; host UP 0656:45; mesh back 0657:52; all 14 guests running 0704 (boot storm: host load 37, ospool HDD read-IOPS-bound with a cold ARC); dockerd on ana-docker active 0708; LiteLLM 0711:25; Worldtree ×3 0712:55; OpenWebUI healthy 0715.
  • Post-check PASSED (0715): pve-manager 9.2.21, kernel 7.0.14-20, booted via GRUB 2.12; NIC names held, .31 by DHCP, vmbr0 STP off; NASPool + ospool ONLINE (⚠ ZFS 2.4.4 now — do NOT zpool upgrade until 9 is kept); no failed host units; chrony synced; all 5 storages active incl. the pbs-ana datastore; pve8to9 0 FAIL / 0 WARN (NOTICE: local-lvm autoactivation, local storage, left). Gateway embed (dim 1024) + gen-small chat OK; gitea/vaultwarden 200; PBS 200; TacticalRMM 200; pfi-postgres :5432, rest-server-ana :8000 open; ana-nas NFS active; ana-docker 44/45 Up (again only the on-demand buildx_buildkit_worldtree-builder0); nag patch re-applied.
  • Outage actually paid, per reboot: NH3/ESH↔Anaheim ~4 min; host ~2.5 min; the LiteLLM gateway (and with it Miranda/Hermes/SVOS) ~20-21 min each time (0614-0635, 0650-0711), because ana-docker's dockerd and LiteLLM's prisma start are slow under the 14-guest boot storm on HDD-backed ospool.
  • Open: drop pve/root-pre-pve9 (lvremove -y pve/root-pre-pve9) after ~24 h of clean running, i.e. on/after 2026-10-05; sooner if lvs Data% passes 80 (a full thick snapshot only invalidates itself, origin unharmed).
Reference in New Issue View Git Blame Copy Permalink