102 lines
7.5 KiB
Markdown
102 lines
7.5 KiB
Markdown
# pfi-pve (Anaheim): Proxmox VE 8 → 9 upgrade
|
||
|
||
Requested by Prime 2026-10-04 ~0550 ("drive the ana pve upgrade to proxmox 9"; "proceed" ~0600).
|
||
Siblings: `docs/runbooks/esh-pve-cluster-pve9-upgrade-plan.md` (esh-nas-pve, done 2026-10-03) and
|
||
nh3-pve (done 2026-10-03, `playbooks/nh3-pve-pve9-postfix.yaml`). Official guide:
|
||
pve.proxmox.com/wiki/Upgrade_from_8_to_9 (re-read 2026-10-04).
|
||
|
||
## The host (read live 2026-10-04 0550-0610)
|
||
|
||
| | |
|
||
|---|---|
|
||
| HW | Dell PowerEdge R750xs, Xeon Silver 4310, 188 GB, BIOS 1.9.2; iDRAC9 6.10.30.00 at 10.250.250.30, **Enterprise licence** (virtual console) |
|
||
| PVE / kernel | 8.3.5 / 6.8.12-8, up 22 days (booted ~2026-09-11) |
|
||
| Cluster | **single node** (no corosync.conf) |
|
||
| Root | ext4 on LVM `pve/root` 65.6 G (31 G free), `/boot` is on root; VG `pve` 16 G free |
|
||
| Boot | **UEFI, GRUB from `\EFI\proxmox\grubx64.efi` on the BOSS card (sdk2), but the `grub-pc` (BIOS) meta package was installed** — the EFI binary and `/boot/grub/x86_64-efi` dated from the 2023-03-30 PVE 7.4 install. Secure Boot off |
|
||
| Network | `vmbr0` = eno8303 + eno8403 (tg3) + eno12399np0 + eno12409np1 (bnxt_en), STP off, only eno8303 has carrier. Mgmt IP 10.250.250.31 is **DHCP** on `vmbr0.250` (FortiGate lease, isc-dhcp-client) |
|
||
| Storage | ZFS `ospool` (every guest disk but pbs-ana's), ZFS `NASPool` raidz2 (rebuilt 2026-09-05), `local-lvm` thin (pbs-ana), `local` (19 G of ISOs) |
|
||
| Extras | No DKMS, no NVIDIA. Dell `srvadmin-hapi` + `srvadmin-idracadm7` 11.4 (local racadm) from a hand-installed .deb; Debian `ceph-common` 16.2 (client only) |
|
||
| Health | **PSU1 has had no input power since 2026-09-11 23:31** (and repeatedly since 2026-04-23); running on PSU2 alone. Known and accepted (`pfi-pve-naspool-rebuild.md`). Not a blocker; a reboot does not change it |
|
||
| Backups | vzdump 03:00 all guests except 100/109/114 OK today; CT 109 22:00 OK last night |
|
||
|
||
Guests (all `onboot=1`): VMs 100 pbs-ana, 101 PFI-ANA-DC (Win), 102 ana-docker, 103 SlaveBot (Win),
|
||
104 Mongo, 105 Postgres, 106 corviduo-dev, 107 pteradactyl, 110 webhost, 111 tacticalrmm;
|
||
CTs 109 ana-nas (NFS, order 1), 112 ana-filebot, 113 ana-wg, **114 ana-scale (order 1)**.
|
||
|
||
## What a pfi-pve reboot takes down — all of Anaheim, for ~15 min each time
|
||
|
||
- **NH3 and ESH lose Anaheim entirely.** CT 114 ana-scale is the only site-to-site path since the
|
||
2026-09-06 headscale cutover (IPsec dormant). That includes the iDRAC and the FortiGate's LAN side.
|
||
- **ana-docker:** the LiteLLM gateway (every agent's model calls, **including Miranda/Hermes and SVOS**,
|
||
whose `base_url` is `10.250.50.70:4000` — Miranda is deaf for the duration), Traefik public endpoints,
|
||
gitea, vaultwarden, rest-server-ana, Langfuse, Beszel/Dozzle hubs, RustDesk.
|
||
- pfi-postgres + Mongo, corviduo-dev (Worldtree demo/personal/pinned), PBS-ANA, TacticalRMM/MeshCentral, AD DC.
|
||
- Not affected: the post office (nh3-docker), nh3-dev, ESH and NH3 local services, fv-ml1.
|
||
|
||
## Findings and what prep did about them (`playbooks/pfi-pve-pve9-prep.yaml`, no downtime)
|
||
|
||
1. **grub-pc on a UEFI, root-on-LVM box** = the wiki's "UEFI systems with root on LVM may fail to boot".
|
||
Prep installs `grub-efi-amd64` (removes `grub-pc`) with `force_efi_extra_removable`, so the ESP binary
|
||
and the removable copy are rewritten and stay in step with `/boot/grub` from now on. Kept: tar of `/etc`,
|
||
`/boot/grub` and the ESP in `/root/pve9-prep-20261004/`; the 2023 binary also at `EFI/proxmox-2023/`
|
||
(last resort only — an old core loading new modules can fail).
|
||
2. **NIC names pinned by MAC** (`/etc/systemd/network/10-pin-*.link` + initramfs), all four ports. The bnxt
|
||
`np0/np1` suffix is the kind of name a new kernel changes; a renamed eno8303 = vmbr0 without carrier.
|
||
3. Latest 8.4 (8.4.21, kernel 6.8.12-43), `non-free-firmware` + `intel-microcode`. systemd-boot was never installed.
|
||
4. Not changed: the DHCP mgmt address. The pre-reboot check asserts a DHCP client survives the upgrade.
|
||
|
||
## Sequence
|
||
|
||
1. Prep (no downtime). ✅ see Log.
|
||
2. **Reboot test on PVE 8** (~15 min outage). Validates the new GRUB, the pins, kernel -43 and microcode while
|
||
the box is still on 8 and `/boot` still matches a bookworm GRUB, so a failure there is a smaller fix.
|
||
3. `playbooks/pfi-pve-pve9-upgrade.yaml` (guests stay up): guards, LVM snapshot `pve/root-pre-pve9` (15 G),
|
||
trixie sources (deb822 `proxmox.sources`), reads the dist-upgrade plan and refuses if it would remove a
|
||
boot/network package, then a DETACHED dist-upgrade (`/root/pve9-upgrade.log`).
|
||
4. `playbooks/pfi-pve-pve9-prereboot.yaml`: enterprise repo off; asserts EXIT 0, dpkg clean, PVE 9,
|
||
initramfs has the pins + dm-snapshot, ESP rewritten + proxmox first in BootOrder, DHCP client present,
|
||
`ifreload -a -s` parses, pve8to9.
|
||
5. Reboot (~15 min outage). Post-check: all 14 guests, ana-scale/mesh, NFS, PBS datastore, LiteLLM, gitea,
|
||
vaultwarden, Worldtree ×3, container counts against the pre-reboot baseline.
|
||
6. Drop the LVM snapshot once 9 is confirmed (it slows writes to `/` while it exists).
|
||
|
||
## Break-glass if a reboot does not come back (verified reachable 2026-10-04 0603)
|
||
|
||
Nothing in Anaheim answers from NH3 while ana-scale is down, so the way in is the FortiGate's WAN SSH
|
||
(the 2026-09-06 safety net: `wan1 allowaccess ping ssh`, `infra-ops` trusthost = NH3 70.230.226.88 + ESH):
|
||
|
||
```
|
||
SSHPASS=$(secret get fortigate/ana-gw-infra-ops-password) sshpass -e ssh infra-ops@38.120.12.42
|
||
execute ping 10.250.250.31 # host up? (0603: FortiGate→iDRAC 0.3 ms)
|
||
execute ping 10.250.50.45 # ana-scale up?
|
||
execute ping 10.250.250.30 # iDRAC
|
||
```
|
||
|
||
- Host up, ana-scale not: wait for CT 114 (order 1); if it stays down, start it via the iDRAC/host once reachable.
|
||
- Host not up: restore the old NH3↔ANA IPsec path (the documented un-dormant procedure,
|
||
`docs/pfi/headscale-mesh-plan.md` § CUTOVER EXECUTED): on the FortiGate `config router static / edit 20 /
|
||
set status disable` (the `10.100.0.0/16 → 10.250.50.45` mesh route; route 8 via `pfi-ana-nh3` then wins),
|
||
and on the NH3 UDM enable `pfi-nh3-ana` and disable the `10.250.0.0/16 → 10.100.50.46` static route. Tunnel
|
||
and mesh route must never be live for the same /16 at once. Then the iDRAC (Redfish creds
|
||
`pfi-pve/idrac-infra-ops`, verified 0601) gives power, SEL, one-time boot and screen capture; Prime can use
|
||
the virtual console. Undo both edges in reverse once ana-scale is back.
|
||
|
||
## Rollback
|
||
|
||
- After step 3, before or after the reboot: `lvconvert --merge pve/root-pre-pve9`, reboot (the merge runs at
|
||
activation; from a rescue ISO via iDRAC virtual media if 9 does not boot). Covers `/` and `/boot`, not the
|
||
ESP: if GRUB on the ESP is the problem, boot the rescue ISO and `grub-install` from the merged root.
|
||
- Guest storage (ospool, NASPool, local-lvm) is not touched by the host upgrade. Never `zpool upgrade`
|
||
ospool or NASPool after the upgrade until 9 has been kept.
|
||
|
||
## Log (PT)
|
||
|
||
- 0550-0610 recon (read-only); plan written. Positive controls: iDRAC Redfish login OK 0601; FortiGate WAN SSH
|
||
login OK 0603; FortiGate → iDRAC + ana-scale ping OK.
|
||
- 0601-0614 **prep done** (`pfi-pve-pve9-prep.yaml`, elway overall CHANGED, 0 failed): PVE 8.4.21, kernel 6.8.12-43
|
||
installed (grub.cfg default), intel-microcode 3.20251111.1, `grub-efi-amd64` 2.06-13+pmx7 in / `grub-pc` out, ESP
|
||
`grubx64.efi` rewritten 06:08 (md5 a4a8d8d1… → d9d0a989…, BOOTX64.EFI identical), `/boot/grub/x86_64-efi` refreshed,
|
||
BootOrder 0003 (proxmox) first, four NIC pins in the -43 and -8 initramfs. pve8to9 --full: 0 FAIL, 1 WARN (running
|
||
guests), NOTICE local-lvm autoactivation (local storage, left as is). No guest touched.
|