143 lines
12 KiB
Markdown
143 lines
12 KiB
Markdown
# pfi-pve (Anaheim): Proxmox VE 8 → 9 upgrade
|
||
|
||
**STATUS: DONE 2026-10-04 0715 — PVE 9.2.21 / kernel 7.0.14-20, post-check passed. Open: drop the root snapshot
|
||
on/after 2026-10-05 (see Log).**
|
||
|
||
Requested by Prime 2026-10-04 ~0550 ("drive the ana pve upgrade to proxmox 9"; "proceed" ~0600).
|
||
Siblings: `docs/runbooks/esh-pve-cluster-pve9-upgrade-plan.md` (esh-nas-pve, done 2026-10-03) and
|
||
nh3-pve (done 2026-10-03, `playbooks/nh3-pve-pve9-postfix.yaml`). Official guide:
|
||
pve.proxmox.com/wiki/Upgrade_from_8_to_9 (re-read 2026-10-04).
|
||
|
||
## The host (read live 2026-10-04 0550-0610)
|
||
|
||
| | |
|
||
|---|---|
|
||
| HW | Dell PowerEdge R750xs, Xeon Silver 4310, 188 GB, BIOS 1.9.2; iDRAC9 6.10.30.00 at 10.250.250.30, **Enterprise licence** (virtual console) |
|
||
| PVE / kernel | 8.3.5 / 6.8.12-8, up 22 days (booted ~2026-09-11) |
|
||
| Cluster | **single node** (no corosync.conf) |
|
||
| Root | ext4 on LVM `pve/root` 65.6 G (31 G free), `/boot` is on root; VG `pve` 16 G free |
|
||
| Boot | **UEFI, GRUB from `\EFI\proxmox\grubx64.efi` on the BOSS card (sdk2), but the `grub-pc` (BIOS) meta package was installed** — the EFI binary and `/boot/grub/x86_64-efi` dated from the 2023-03-30 PVE 7.4 install. Secure Boot off |
|
||
| Network | `vmbr0` = eno8303 + eno8403 (tg3) + eno12399np0 + eno12409np1 (bnxt_en), STP off, only eno8303 has carrier. Mgmt IP 10.250.250.31 is **DHCP** on `vmbr0.250` (FortiGate lease, isc-dhcp-client) |
|
||
| Storage | ZFS `ospool` (every guest disk but pbs-ana's), ZFS `NASPool` raidz2 (rebuilt 2026-09-05), `local-lvm` thin (pbs-ana), `local` (19 G of ISOs) |
|
||
| Extras | No DKMS, no NVIDIA. Dell `srvadmin-hapi` + `srvadmin-idracadm7` 11.4 (local racadm) from a hand-installed .deb; Debian `ceph-common` 16.2 (client only) |
|
||
| Health | **PSU1 has had no input power since 2026-09-11 23:31** (and repeatedly since 2026-04-23); running on PSU2 alone. Known and accepted (`pfi-pve-naspool-rebuild.md`). Not a blocker; a reboot does not change it |
|
||
| Backups | vzdump 03:00 all guests except 100/109/114 OK today; CT 109 22:00 OK last night |
|
||
|
||
Guests (all `onboot=1`): VMs 100 pbs-ana, 101 PFI-ANA-DC (Win), 102 ana-docker, 103 SlaveBot (Win),
|
||
104 Mongo, 105 Postgres, 106 corviduo-dev, 107 pteradactyl, 110 webhost, 111 tacticalrmm;
|
||
CTs 109 ana-nas (NFS, order 1), 112 ana-filebot, 113 ana-wg, **114 ana-scale (order 1)**.
|
||
|
||
## What a pfi-pve reboot takes down — all of Anaheim, for ~15 min each time
|
||
|
||
- **NH3 and ESH lose Anaheim entirely.** CT 114 ana-scale is the only site-to-site path since the
|
||
2026-09-06 headscale cutover (IPsec dormant). That includes the iDRAC and the FortiGate's LAN side.
|
||
- **ana-docker:** the LiteLLM gateway (every agent's model calls, **including Miranda/Hermes and SVOS**,
|
||
whose `base_url` is `10.250.50.70:4000` — Miranda is deaf for the duration), Traefik public endpoints,
|
||
gitea, vaultwarden, rest-server-ana, Langfuse, Beszel/Dozzle hubs, RustDesk.
|
||
- pfi-postgres + Mongo, corviduo-dev (Worldtree demo/personal/pinned), PBS-ANA, TacticalRMM/MeshCentral, AD DC.
|
||
- Not affected: the post office (nh3-docker), nh3-dev, ESH and NH3 local services, fv-ml1.
|
||
|
||
## Findings and what prep did about them (`playbooks/pfi-pve-pve9-prep.yaml`, no downtime)
|
||
|
||
1. **grub-pc on a UEFI, root-on-LVM box** = the wiki's "UEFI systems with root on LVM may fail to boot".
|
||
Prep installs `grub-efi-amd64` (removes `grub-pc`) with `force_efi_extra_removable`, so the ESP binary
|
||
and the removable copy are rewritten and stay in step with `/boot/grub` from now on. Kept: tar of `/etc`,
|
||
`/boot/grub` and the ESP in `/root/pve9-prep-20261004/`; the 2023 binary also at `EFI/proxmox-2023/`
|
||
(last resort only — an old core loading new modules can fail).
|
||
2. **NIC names pinned by MAC** (`/etc/systemd/network/10-pin-*.link` + initramfs), all four ports. The bnxt
|
||
`np0/np1` suffix is the kind of name a new kernel changes; a renamed eno8303 = vmbr0 without carrier.
|
||
3. Latest 8.4 (8.4.21, kernel 6.8.12-43), `non-free-firmware` + `intel-microcode`. systemd-boot was never installed.
|
||
4. Not changed: the DHCP mgmt address. The pre-reboot check asserts a DHCP client survives the upgrade.
|
||
|
||
## Sequence
|
||
|
||
1. Prep (no downtime). ✅ see Log.
|
||
2. **Reboot test on PVE 8** (~15 min outage). Validates the new GRUB, the pins, kernel -43 and microcode while
|
||
the box is still on 8 and `/boot` still matches a bookworm GRUB, so a failure there is a smaller fix.
|
||
3. `playbooks/pfi-pve-pve9-upgrade.yaml` (guests stay up): guards, LVM snapshot `pve/root-pre-pve9` (15 G),
|
||
trixie sources (deb822 `proxmox.sources`), reads the dist-upgrade plan and refuses if it would remove a
|
||
boot/network package, then a DETACHED dist-upgrade (`/root/pve9-upgrade.log`).
|
||
4. `playbooks/pfi-pve-pve9-prereboot.yaml`: enterprise repo off; asserts EXIT 0, dpkg clean, PVE 9,
|
||
initramfs has the pins + dm-snapshot, ESP rewritten + proxmox first in BootOrder, DHCP client present,
|
||
`ifreload -a -s` parses, pve8to9.
|
||
5. Reboot (~15 min outage). Post-check: all 14 guests, ana-scale/mesh, NFS, PBS datastore, LiteLLM, gitea,
|
||
vaultwarden, Worldtree ×3, container counts against the pre-reboot baseline.
|
||
6. Drop the LVM snapshot once 9 is confirmed (it slows writes to `/` while it exists).
|
||
|
||
## Break-glass if a reboot does not come back (verified reachable 2026-10-04 0603)
|
||
|
||
Nothing in Anaheim answers from NH3 while ana-scale is down, so the way in is the FortiGate's WAN SSH
|
||
(the 2026-09-06 safety net: `wan1 allowaccess ping ssh`, `infra-ops` trusthost = NH3 70.230.226.88 + ESH):
|
||
|
||
```
|
||
SSHPASS=$(secret get fortigate/ana-gw-infra-ops-password) sshpass -e ssh infra-ops@38.120.12.42
|
||
execute ping 10.250.250.31 # host up? (0603: FortiGate→iDRAC 0.3 ms)
|
||
execute ping 10.250.50.45 # ana-scale up?
|
||
execute ping 10.250.250.30 # iDRAC
|
||
```
|
||
|
||
- Host up, ana-scale not: wait for CT 114 (order 1); if it stays down, start it via the iDRAC/host once reachable.
|
||
- Host not up: restore the old NH3↔ANA IPsec path (the documented un-dormant procedure,
|
||
`docs/pfi/headscale-mesh-plan.md` § CUTOVER EXECUTED): on the FortiGate `config router static / edit 20 /
|
||
set status disable` (the `10.100.0.0/16 → 10.250.50.45` mesh route; route 8 via `pfi-ana-nh3` then wins),
|
||
and on the NH3 UDM enable `pfi-nh3-ana` and disable the `10.250.0.0/16 → 10.100.50.46` static route. Tunnel
|
||
and mesh route must never be live for the same /16 at once. Then the iDRAC (Redfish creds
|
||
`pfi-pve/idrac-infra-ops`, verified 0601) gives power, SEL, one-time boot and screen capture; Prime can use
|
||
the virtual console. Undo both edges in reverse once ana-scale is back.
|
||
|
||
## Rollback
|
||
|
||
- After step 3, before or after the reboot: `lvconvert --merge pve/root-pre-pve9`, reboot (the merge runs at
|
||
activation; from a rescue ISO via iDRAC virtual media if 9 does not boot). Covers `/` and `/boot`, not the
|
||
ESP: if GRUB on the ESP is the problem, boot the rescue ISO and `grub-install` from the merged root.
|
||
- Guest storage (ospool, NASPool, local-lvm) is not touched by the host upgrade. Never `zpool upgrade`
|
||
ospool or NASPool after the upgrade until 9 has been kept.
|
||
|
||
## Log (PT)
|
||
|
||
- 0550-0610 recon (read-only); plan written. Positive controls: iDRAC Redfish login OK 0601; FortiGate WAN SSH
|
||
login OK 0603; FortiGate → iDRAC + ana-scale ping OK.
|
||
- 0601-0614 **prep done** (`pfi-pve-pve9-prep.yaml`, elway overall CHANGED, 0 failed): PVE 8.4.21, kernel 6.8.12-43
|
||
installed (grub.cfg default), intel-microcode 3.20251111.1, `grub-efi-amd64` 2.06-13+pmx7 in / `grub-pc` out, ESP
|
||
`grubx64.efi` rewritten 06:08 (md5 a4a8d8d1… → d9d0a989…, BOOTX64.EFI identical), `/boot/grub/x86_64-efi` refreshed,
|
||
BootOrder 0003 (proxmox) first, four NIC pins in the -43 and -8 initramfs. pve8to9 --full: 0 FAIL, 1 WARN (running
|
||
guests), NOTICE local-lvm autoactivation (local storage, left as is). No guest touched.
|
||
- 0611:51 **PVE 8 reboot test** issued (FortiGate-side ping watcher running, independent of ana-scale). ana-docker
|
||
stopped answering 0614:20; ana-scale + ana-nas 0617:39; host DOWN 0617:58; **host UP 0620:14** (R750xs POST ~2 min);
|
||
mesh back 0621:23; ana-docker 0624:22; all 14 guests running by ~0635. **Passed:** booted 6.8.12-43 via the new
|
||
GRUB, NIC names held, `.31` by DHCP, microcode 0xd000410, pools healthy, no failed units on the host.
|
||
Services: gitea/vaultwarden 200, PBS 200, ana-nas NFS active (ana-docker `/mnt/backup` remounted), Worldtree
|
||
demo/personal 200 0635, pinned 0636. ana-docker 44/45 Up (missing only `buildx_buildkit_worldtree-builder0`, an
|
||
on-demand buildx builder with no restart policy). ⚠ **LiteLLM took until 0634:51** (container up 0627:56, then
|
||
~6.5 min in `prisma` toolchain prep + `migrate deploy` under boot-storm load, load avg 11): the gateway, and with
|
||
it Miranda/Hermes, was out ~20 min, not ~10. Pre-existing, not ours: pfi-pteradactyl `apache2` fails every boot
|
||
(nginx owns :80) + `systemd-journal-flush` failed.
|
||
- 0637 upgrade playbook: (a first run went to a mistyped `10.250.50.31` and died at elway's sudo probe, nothing
|
||
executed). 0638 real run: pve8to9 0 FAIL; root snapshot `pve/root-pre-pve9` 15G; trixie sources; plan read =
|
||
770 installs, 53 removals (t64 library renames + python3-distutils/future/lib2to3; isc-dhcp-client, srvadmin
|
||
stay); detached dist-upgrade started 0638 (`pve9-dist-upgrade.service`). Targets: pve-manager 9.2.21, kernel
|
||
7.0.14-20, GRUB 2.12-9+pmx2, ZFS 2.4.4 userland (⚠ no `zpool upgrade`), ceph-common 19.2.6, lxcfs 7.0.
|
||
sshd restarted mid-upgrade (~0640, connection reset for a few seconds), as expected.
|
||
- 0647 dist-upgrade EXIT 0 (770 packages; GRUB 2.12-9+pmx2 postinst: "Installing for x86_64-efi platform …
|
||
Installation finished. No error reported."). Pre-reboot check: two of my checks were wrong for PVE 9 and were
|
||
fixed, not waived: (1) the GRUB binary carries no greppable version string, so the postinst's own report is the
|
||
proof; (2) ifupdown2 3.3 flags PVE's own `bridge-fd 0` as out of range and `ifreload -a -s` exits 1 on that
|
||
warning alone. Positive control: nh3-pve and esh-nas-pve (same ifupdown2 3.3.0-1+pmx12, same line) show the same
|
||
warning and run vmbr0 fine (STP off, forward_delay 0). The check now fails on any OTHER ifupdown2 output (tested
|
||
both ways). Re-run: 7/7 OK. Enterprise repo set `Enabled: false`. Snapshot COW 40% (≈6 G) after the upgrade.
|
||
- 0648:56 **PVE 9 reboot** issued. ana-docker stopped answering 0650:32; mesh down 0653:51; host DOWN 0654:10;
|
||
**host UP 0656:45**; mesh back 0657:52; all 14 guests running 0704 (boot storm: host load 37, ospool HDD
|
||
read-IOPS-bound with a cold ARC); dockerd on ana-docker active 0708; LiteLLM 0711:25; Worldtree ×3 0712:55;
|
||
OpenWebUI healthy 0715.
|
||
- **Post-check PASSED (0715):** pve-manager 9.2.21, kernel 7.0.14-20, booted via GRUB 2.12; NIC names held, `.31`
|
||
by DHCP, vmbr0 STP off; NASPool + ospool ONLINE (⚠ ZFS 2.4.4 now — do NOT `zpool upgrade` until 9 is kept);
|
||
no failed host units; chrony synced; all 5 storages active incl. the pbs-ana datastore; pve8to9 0 FAIL / 0 WARN
|
||
(NOTICE: local-lvm autoactivation, local storage, left). Gateway embed (dim 1024) + `gen-small` chat OK;
|
||
gitea/vaultwarden 200; PBS 200; TacticalRMM 200; pfi-postgres :5432, rest-server-ana :8000 open; ana-nas NFS
|
||
active; ana-docker 44/45 Up (again only the on-demand `buildx_buildkit_worldtree-builder0`); nag patch re-applied.
|
||
- **Outage actually paid, per reboot:** NH3/ESH↔Anaheim ~4 min; host ~2.5 min; the LiteLLM gateway (and with it
|
||
Miranda/Hermes/SVOS) **~20-21 min** each time (0614-0635, 0650-0711), because ana-docker's dockerd and LiteLLM's
|
||
prisma start are slow under the 14-guest boot storm on HDD-backed ospool.
|
||
- Open: drop `pve/root-pre-pve9` (`lvremove -y pve/root-pre-pve9`) after ~24 h of clean running, i.e. on/after
|
||
2026-10-05; sooner if `lvs` Data% passes 80 (a full thick snapshot only invalidates itself, origin unharmed).
|