# pfi-pve (Anaheim): Proxmox VE 8 → 9 upgrade

**STATUS: DONE 2026-10-04 0715 — PVE 9.2.21 / kernel 7.0.14-20, post-check passed. Open: drop the root snapshot
on/after 2026-10-05 (see Log).**

Requested by Prime 2026-10-04 ~0550 ("drive the ana pve upgrade to proxmox 9"; "proceed" ~0600).
Siblings: `docs/runbooks/esh-pve-cluster-pve9-upgrade-plan.md` (esh-nas-pve, done 2026-10-03) and
nh3-pve (done 2026-10-03, `playbooks/nh3-pve-pve9-postfix.yaml`). Official guide:
pve.proxmox.com/wiki/Upgrade_from_8_to_9 (re-read 2026-10-04).

## The host (read live 2026-10-04 0550-0610)

| | |
|---|---|
| HW | Dell PowerEdge R750xs, Xeon Silver 4310, 188 GB, BIOS 1.9.2; iDRAC9 6.10.30.00 at 10.250.250.30, **Enterprise licence** (virtual console) |
| PVE / kernel | 8.3.5 / 6.8.12-8, up 22 days (booted ~2026-09-11) |
| Cluster | **single node** (no corosync.conf) |
| Root | ext4 on LVM `pve/root` 65.6 G (31 G free), `/boot` is on root; VG `pve` 16 G free |
| Boot | **UEFI, GRUB from `\EFI\proxmox\grubx64.efi` on the BOSS card (sdk2), but the `grub-pc` (BIOS) meta package was installed** — the EFI binary and `/boot/grub/x86_64-efi` dated from the 2023-03-30 PVE 7.4 install. Secure Boot off |
| Network | `vmbr0` = eno8303 + eno8403 (tg3) + eno12399np0 + eno12409np1 (bnxt_en), STP off, only eno8303 has carrier. Mgmt IP 10.250.250.31 is **DHCP** on `vmbr0.250` (FortiGate lease, isc-dhcp-client) |
| Storage | ZFS `ospool` (every guest disk but pbs-ana's), ZFS `NASPool` raidz2 (rebuilt 2026-09-05), `local-lvm` thin (pbs-ana), `local` (19 G of ISOs) |
| Extras | No DKMS, no NVIDIA. Dell `srvadmin-hapi` + `srvadmin-idracadm7` 11.4 (local racadm) from a hand-installed .deb; Debian `ceph-common` 16.2 (client only) |
| Health | **PSU1 has had no input power since 2026-09-11 23:31** (and repeatedly since 2026-04-23); running on PSU2 alone. Known and accepted (`pfi-pve-naspool-rebuild.md`). Not a blocker; a reboot does not change it |
| Backups | vzdump 03:00 all guests except 100/109/114 OK today; CT 109 22:00 OK last night |

Guests (all `onboot=1`): VMs 100 pbs-ana, 101 PFI-ANA-DC (Win), 102 ana-docker, 103 SlaveBot (Win),
104 Mongo, 105 Postgres, 106 corviduo-dev, 107 pteradactyl, 110 webhost, 111 tacticalrmm;
CTs 109 ana-nas (NFS, order 1), 112 ana-filebot, 113 ana-wg, **114 ana-scale (order 1)**.

## What a pfi-pve reboot takes down — all of Anaheim, for ~15 min each time

- **NH3 and ESH lose Anaheim entirely.** CT 114 ana-scale is the only site-to-site path since the
  2026-09-06 headscale cutover (IPsec dormant). That includes the iDRAC and the FortiGate's LAN side.
- **ana-docker:** the LiteLLM gateway (every agent's model calls, **including Miranda/Hermes and SVOS**,
  whose `base_url` is `10.250.50.70:4000` — Miranda is deaf for the duration), Traefik public endpoints,
  gitea, vaultwarden, rest-server-ana, Langfuse, Beszel/Dozzle hubs, RustDesk.
- pfi-postgres + Mongo, corviduo-dev (Worldtree demo/personal/pinned), PBS-ANA, TacticalRMM/MeshCentral, AD DC.
- Not affected: the post office (nh3-docker), nh3-dev, ESH and NH3 local services, fv-ml1.

## Findings and what prep did about them (`playbooks/pfi-pve-pve9-prep.yaml`, no downtime)

1. **grub-pc on a UEFI, root-on-LVM box** = the wiki's "UEFI systems with root on LVM may fail to boot".
   Prep installs `grub-efi-amd64` (removes `grub-pc`) with `force_efi_extra_removable`, so the ESP binary
   and the removable copy are rewritten and stay in step with `/boot/grub` from now on. Kept: tar of `/etc`,
   `/boot/grub` and the ESP in `/root/pve9-prep-20261004/`; the 2023 binary also at `EFI/proxmox-2023/`
   (last resort only — an old core loading new modules can fail).
2. **NIC names pinned by MAC** (`/etc/systemd/network/10-pin-*.link` + initramfs), all four ports. The bnxt
   `np0/np1` suffix is the kind of name a new kernel changes; a renamed eno8303 = vmbr0 without carrier.
3. Latest 8.4 (8.4.21, kernel 6.8.12-43), `non-free-firmware` + `intel-microcode`. systemd-boot was never installed.
4. Not changed: the DHCP mgmt address. The pre-reboot check asserts a DHCP client survives the upgrade.

## Sequence

1. Prep (no downtime). ✅ see Log.
2. **Reboot test on PVE 8** (~15 min outage). Validates the new GRUB, the pins, kernel -43 and microcode while
   the box is still on 8 and `/boot` still matches a bookworm GRUB, so a failure there is a smaller fix.
3. `playbooks/pfi-pve-pve9-upgrade.yaml` (guests stay up): guards, LVM snapshot `pve/root-pre-pve9` (15 G),
   trixie sources (deb822 `proxmox.sources`), reads the dist-upgrade plan and refuses if it would remove a
   boot/network package, then a DETACHED dist-upgrade (`/root/pve9-upgrade.log`).
4. `playbooks/pfi-pve-pve9-prereboot.yaml`: enterprise repo off; asserts EXIT 0, dpkg clean, PVE 9,
   initramfs has the pins + dm-snapshot, ESP rewritten + proxmox first in BootOrder, DHCP client present,
   `ifreload -a -s` parses, pve8to9.
5. Reboot (~15 min outage). Post-check: all 14 guests, ana-scale/mesh, NFS, PBS datastore, LiteLLM, gitea,
   vaultwarden, Worldtree ×3, container counts against the pre-reboot baseline.
6. Drop the LVM snapshot once 9 is confirmed (it slows writes to `/` while it exists).

## Break-glass if a reboot does not come back (verified reachable 2026-10-04 0603)

Nothing in Anaheim answers from NH3 while ana-scale is down, so the way in is the FortiGate's WAN SSH
(the 2026-09-06 safety net: `wan1 allowaccess ping ssh`, `infra-ops` trusthost = NH3 70.230.226.88 + ESH):

```
SSHPASS=$(secret get fortigate/ana-gw-infra-ops-password) sshpass -e ssh infra-ops@38.120.12.42
execute ping 10.250.250.31     # host up?          (0603: FortiGate→iDRAC 0.3 ms)
execute ping 10.250.50.45      # ana-scale up?
execute ping 10.250.250.30     # iDRAC
```

- Host up, ana-scale not: wait for CT 114 (order 1); if it stays down, start it via the iDRAC/host once reachable.
- Host not up: restore the old NH3↔ANA IPsec path (the documented un-dormant procedure,
  `docs/pfi/headscale-mesh-plan.md` § CUTOVER EXECUTED): on the FortiGate `config router static / edit 20 /
  set status disable` (the `10.100.0.0/16 → 10.250.50.45` mesh route; route 8 via `pfi-ana-nh3` then wins),
  and on the NH3 UDM enable `pfi-nh3-ana` and disable the `10.250.0.0/16 → 10.100.50.46` static route. Tunnel
  and mesh route must never be live for the same /16 at once. Then the iDRAC (Redfish creds
  `pfi-pve/idrac-infra-ops`, verified 0601) gives power, SEL, one-time boot and screen capture; Prime can use
  the virtual console. Undo both edges in reverse once ana-scale is back.

## Rollback

- After step 3, before or after the reboot: `lvconvert --merge pve/root-pre-pve9`, reboot (the merge runs at
  activation; from a rescue ISO via iDRAC virtual media if 9 does not boot). Covers `/` and `/boot`, not the
  ESP: if GRUB on the ESP is the problem, boot the rescue ISO and `grub-install` from the merged root.
- Guest storage (ospool, NASPool, local-lvm) is not touched by the host upgrade. Never `zpool upgrade`
  ospool or NASPool after the upgrade until 9 has been kept.

## Log (PT)

- 0550-0610 recon (read-only); plan written. Positive controls: iDRAC Redfish login OK 0601; FortiGate WAN SSH
  login OK 0603; FortiGate → iDRAC + ana-scale ping OK.
- 0601-0614 **prep done** (`pfi-pve-pve9-prep.yaml`, elway overall CHANGED, 0 failed): PVE 8.4.21, kernel 6.8.12-43
  installed (grub.cfg default), intel-microcode 3.20251111.1, `grub-efi-amd64` 2.06-13+pmx7 in / `grub-pc` out, ESP
  `grubx64.efi` rewritten 06:08 (md5 a4a8d8d1… → d9d0a989…, BOOTX64.EFI identical), `/boot/grub/x86_64-efi` refreshed,
  BootOrder 0003 (proxmox) first, four NIC pins in the -43 and -8 initramfs. pve8to9 --full: 0 FAIL, 1 WARN (running
  guests), NOTICE local-lvm autoactivation (local storage, left as is). No guest touched.
- 0611:51 **PVE 8 reboot test** issued (FortiGate-side ping watcher running, independent of ana-scale). ana-docker
  stopped answering 0614:20; ana-scale + ana-nas 0617:39; host DOWN 0617:58; **host UP 0620:14** (R750xs POST ~2 min);
  mesh back 0621:23; ana-docker 0624:22; all 14 guests running by ~0635. **Passed:** booted 6.8.12-43 via the new
  GRUB, NIC names held, `.31` by DHCP, microcode 0xd000410, pools healthy, no failed units on the host.
  Services: gitea/vaultwarden 200, PBS 200, ana-nas NFS active (ana-docker `/mnt/backup` remounted), Worldtree
  demo/personal 200 0635, pinned 0636. ana-docker 44/45 Up (missing only `buildx_buildkit_worldtree-builder0`, an
  on-demand buildx builder with no restart policy). ⚠ **LiteLLM took until 0634:51** (container up 0627:56, then
  ~6.5 min in `prisma` toolchain prep + `migrate deploy` under boot-storm load, load avg 11): the gateway, and with
  it Miranda/Hermes, was out ~20 min, not ~10. Pre-existing, not ours: pfi-pteradactyl `apache2` fails every boot
  (nginx owns :80) + `systemd-journal-flush` failed.
- 0637 upgrade playbook: (a first run went to a mistyped `10.250.50.31` and died at elway's sudo probe, nothing
  executed). 0638 real run: pve8to9 0 FAIL; root snapshot `pve/root-pre-pve9` 15G; trixie sources; plan read =
  770 installs, 53 removals (t64 library renames + python3-distutils/future/lib2to3; isc-dhcp-client, srvadmin
  stay); detached dist-upgrade started 0638 (`pve9-dist-upgrade.service`). Targets: pve-manager 9.2.21, kernel
  7.0.14-20, GRUB 2.12-9+pmx2, ZFS 2.4.4 userland (⚠ no `zpool upgrade`), ceph-common 19.2.6, lxcfs 7.0.
  sshd restarted mid-upgrade (~0640, connection reset for a few seconds), as expected.
- 0647 dist-upgrade EXIT 0 (770 packages; GRUB 2.12-9+pmx2 postinst: "Installing for x86_64-efi platform …
  Installation finished. No error reported."). Pre-reboot check: two of my checks were wrong for PVE 9 and were
  fixed, not waived: (1) the GRUB binary carries no greppable version string, so the postinst's own report is the
  proof; (2) ifupdown2 3.3 flags PVE's own `bridge-fd 0` as out of range and `ifreload -a -s` exits 1 on that
  warning alone. Positive control: nh3-pve and esh-nas-pve (same ifupdown2 3.3.0-1+pmx12, same line) show the same
  warning and run vmbr0 fine (STP off, forward_delay 0). The check now fails on any OTHER ifupdown2 output (tested
  both ways). Re-run: 7/7 OK. Enterprise repo set `Enabled: false`. Snapshot COW 40% (≈6 G) after the upgrade.
- 0648:56 **PVE 9 reboot** issued. ana-docker stopped answering 0650:32; mesh down 0653:51; host DOWN 0654:10;
  **host UP 0656:45**; mesh back 0657:52; all 14 guests running 0704 (boot storm: host load 37, ospool HDD
  read-IOPS-bound with a cold ARC); dockerd on ana-docker active 0708; LiteLLM 0711:25; Worldtree ×3 0712:55;
  OpenWebUI healthy 0715.
- **Post-check PASSED (0715):** pve-manager 9.2.21, kernel 7.0.14-20, booted via GRUB 2.12; NIC names held, `.31`
  by DHCP, vmbr0 STP off; NASPool + ospool ONLINE (⚠ ZFS 2.4.4 now — do NOT `zpool upgrade` until 9 is kept);
  no failed host units; chrony synced; all 5 storages active incl. the pbs-ana datastore; pve8to9 0 FAIL / 0 WARN
  (NOTICE: local-lvm autoactivation, local storage, left). Gateway embed (dim 1024) + `gen-small` chat OK;
  gitea/vaultwarden 200; PBS 200; TacticalRMM 200; pfi-postgres :5432, rest-server-ana :8000 open; ana-nas NFS
  active; ana-docker 44/45 Up (again only the on-demand `buildx_buildkit_worldtree-builder0`); nag patch re-applied.
- **Outage actually paid, per reboot:** NH3/ESH↔Anaheim ~4 min; host ~2.5 min; the LiteLLM gateway (and with it
  Miranda/Hermes/SVOS) **~20-21 min** each time (0614-0635, 0650-0711), because ana-docker's dockerd and LiteLLM's
  prisma start are slow under the 14-guest boot storm on HDD-backed ospool.
- Open: drop `pve/root-pre-pve9` (`lvremove -y pve/root-pre-pve9`) after ~24 h of clean running, i.e. on/after
  2026-10-05; sooner if `lvs` Data% passes 80 (a full thick snapshot only invalidates itself, origin unharmed).