vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity
Files
bf46beaa96c2eec7b1f7a012e006c8e567f08cdf
esh-pfi-infrastructure/docs/runbooks/esh-pve-cluster-pve9-upgrade-plan.md
T

16 KiB
Raw Blame History

esh-pve-cluster: Proxmox VE 8 → 9 upgrade plan — DONE (esh-nas-pve 2026-10-03, esh-pve 2026-10-04)

Requested by Prime via Miranda, 2026-10-02 (thread 01M3YX5QGQ5V8BQXYM46E5T3PB). Do not execute without a further green light from Prime. Facts below were read live on 2026-10-02 ~1930 PT, read-only (including pve8to9 --full on both nodes). Re-run pve8to9 --full and check the official wiki (pve.proxmox.com/wiki/Upgrade_from_8_to_9) on the day; anything that disagrees wins.

The cluster today

pve (fleet: esh-pve) esh-nas-pve (fleet: esh-pve-nas)
Mgmt IP 10.0.250.35 10.0.50.55
HW Minisforum MS-01, i9-13900H, 62 GiB Xeon W-1250, 125 GiB
PVE / kernel 8.4.20 / 6.8.12-42 (‑43 pending) 8.4.20 / 6.8.12-13 (up 6 weeks; ‑43 pending)
Root ext4 on LVM pve/root (96 G, 69 G free), VG free 16 G ZFS nvme/ROOT/pve-1 (pool nvme)
Boot UEFI, GRUB (proxmox-boot-tool not in use) UEFI, GRUB reading ZFS (proxmox-boot-tool not in use)
Extras NVIDIA 580.178.04 DKMS (RTX 2000E Ada → CT 110 esh-ml1) ZFS pools nvme, ssd, tank (175 T raidz2×2; healthy since 2026-10-03, was DEGRADED)
Guests 100 esh-vm-docker, 101 esh-vm-db, 108 esh-scale, 110 esh-ml1, 111 esh-matter 103 esh-nas (NFS server 10.0.50.50), 104 vm-esh-nas, 105 plex, 106 filebot, 107 jellyfin

Repos: pve-no-subscription + Debian bookworm main/contrib, no Ceph, no HA resources. Quorum: 2 nodes × 1 vote, no QDevice, no two_node.

⚠ Findings that come before the upgrade

  1. ✅ RESOLVED 2026-10-03 0629, so no longer a blocker. Two full scrubs ran. The first repaired 198 G, the residue of the Aug 20 resilver, which had never been followed by a full scrub. The second repaired 0 B with 0 errors. The disk was kept, and zpool status -x says healthy. The original finding follows. tank on esh-nas-pve is DEGRADED, and has been since about 2026-08-20. In raidz2-0, disk wwn-0x5000c500c91df554 shows 36 CKSUM errors ("too many errors"). The last resilver (Aug 20, 3.11 T) logged 6,676,313 errors. "No known data errors" and SMART is clean (0 reallocated, 0 pending, 0 CRC, 11,006 h). That vdev is down to one disk of redundancy. Fix first: zpool clear tank + a full scrub; if errors return, replace the disk. Do not upgrade this node with the pool degraded.
  2. Remote access to ESH depends on CT 108 (esh-scale) on pve. Measured path from NH3: 10.100.50.46 → 100.64.0.2 (esh-scale) → 10.0.50.1 → ESH. When pve reboots, NH3/ANA lose ESH until 108 is back. If pve does not come back, there is no remote recovery path (esh-pve's AMT is not set up). Either someone is at ESH for pve's reboot, or set up esh-pve's AMT (+ phone-home, as done for nh3-pve) first.
  3. pve8to9 FAIL on both nodes: the systemd-boot meta-package is installed. Both nodes boot via GRUB, so remove it (apt remove systemd-boot; systemd-boot-efi stays).

esh-nas-pve facts re-read 2026-10-03 (Prime asked whether infra-ops can upgrade it unattended)

  • QNAP hardware (QNAP Systems, Inc., Comet Lake HECI present, but no AMT/IPMI we can use), so there is no remote console.
  • PVE 8.4.20, but still running kernel 6.8.12-13, up since 2026-08-18. Its first reboot in ~7 weeks is itself a risk: do a reboot test on PVE 8 first, so a reboot problem is not mistaken for an upgrade problem.
  • No NIC pins (/etc/systemd/network/ is empty); the uplink is enp5s0f0. Pin before the upgrade.
  • No DKMS on this node (the NVIDIA headers lesson applies to pve, not here).
  • Infra-ops can do the prep remotely. The upgrade reboot should happen only with someone able to reach ESH, or with Prime's explicit acceptance that a failed boot means an outage until someone is on site.

esh-nas-pve: UPGRADED to PVE 9.2.21 / kernel 7.0.14-20, 2026-10-03 1552-1606 (Prime on site)

playbooks/esh-nas-pve-pve9-upgrade.yaml: root snapshot nvme/ROOT@pre-pve9 (keep about a week, then destroy), trixie sources, dist-upgrade run DETACHED (log /root/pve9-upgrade.log). What it took:

  • ⚠ /boot is a separate 488 MB ext4 LV (/dev/mapper/pve-boot); ZFS holds only /. With four kernels it hit 91%, and the second mkinitramfs for 7.0 failed with "No space left on device" (dist-upgrade EXIT 100, initramfs-tools half-configured). Fix: apt remove the two oldest 6.8 kernels → 49%, dpkg --configure -a, re-run dist-upgrade → clean. For pve: check df /boot and trim old kernels BEFORE the dist-upgrade.
  • The upgrade adds pve-enterprise.sources, enabled. Set Enabled: false.
  • chrony failed right after the upgrade ("Could not send notification to $NOTIFY_SOCKET", new chronyd under the old running systemd/kernel). It was fine after the reboot: synced and active.
  • The two old community de-nag hooks were retired to /root/pve9-prep-20261003/; pve-nag-patch re-patched the 9.x UI.
  • Reboot 16:01:06, down 16:01:31, up 16:04:14; pinned names held; pools ONLINE; guests + NFS back 16:05:28.
  • VM 100 stayed up (stopped calibre-web-automated + beszel-agent, unmounted /mnt/books + /mnt/backup, remounted after), so ESH DNS never went down. VM 101 was down 16:00:52-16:05:47 and started first try this time.
  • pve8to9 after: 0 FAIL, 1 WARN (2-node cluster). openipmi.service fails at boot (QNAP has no IPMI); harmless, can be masked.

Reboot test on PVE 8: DONE 2026-10-03 1533-1546 (Prime on site)

Prep: playbooks/esh-nas-pve-pve9-prep.yaml (8.4.21, kernel 6.8.12-43, intel-microcode, systemd-boot removed, NIC pins); one-off backups of 103-107 to pbs-ana (1530-1532); pve8to9 0 FAIL. Then:

  • VMs 100/101 on pve were shut down cleanly (15 s each).
  • esh-nas-pve: reboot 15:33:23, down at 15:34:12, up at 15:36:58 on 6.8.12-43.
  • Pinned names held and vmbr0 came back on .55; pools ONLINE; microcode active.
  • Guests 103-107 started one after another and were all up by ~15:38; NFS (:2049) was back at 15:38.
  • ⚠ VM 101's first start timed out (qm start "got timeout" at 15:38:29): its disk is qcow2 on the esh-nas NFS storage, still recovering. Retry after a few minutes; it started cleanly at 15:45 (postgres + mongod active).
  • ⚠ mosquitto on VM 100 did NOT come back by itself: it exited 0 at shutdown, and although it is unless-stopped, dockerd did not restart it. zigbee2mqtt restart-looped without its broker until mosquitto was started by hand (compose up -d, 15:45). After any VM 100 reboot, check mosquitto and zigbee2mqtt.
  • Total ESH impact: DNS ~6 min (15:33-15:39); DB ~12 min; Zigbee ~13 min; NAS/media ~4 min.
  • Better for the real esh-nas-pve reboot: do NOT shut VM 100 down. That is what took ESH DNS out. Only two of its containers use the NAS shares: calibre-web-automated (/mnt/books) and beszel-agent (stats of /mnt/books and /mnt/backup). Stop those two, umount /mnt/books /mnt/backup inside VM 100, reboot the NAS host, then remount and start them. DNS, Z2M/mosquitto and the rest stay up. VM 101 still has to shut down (its disk is qcow2 on the esh-nas NFS storage).

Pre-flight (no guest downtime)

  1. Backups: one-off vzdump of all ten guests to pbs-ana in snapshot mode, right before the window, including 108 and 110, which the nightly job does not back up. Check each new snapshot is listed. Plus a host config backup per node: /etc and /etc/pve tar to PBS or the NAS.
  2. Bring both nodes to the latest 8.4 (apt update && apt full-upgrade), then pve8to9 --full must show no FAIL.
  3. Clear the pve8to9 items:
    • both: apt remove systemd-boot;
    • pve: echo 'grub-efi-amd64 grub2/force_efi_extra_removable boolean true' | debconf-set-selections -v -u;
    • both, optional but recommended: enable non-free-firmware, apt install intel-microcode.
  4. Pin NIC names by MAC (systemd .link) on both nodes before the upgrade. Debian 13's systemd may change predictable names, and a renamed uplink leaves vmbr0 with no link: the whole node is off the network. pve8to9 did not flag it; this is a precaution. Template: playbooks/nh3-pve-pin-nic-names.yaml. Uplinks today: pve enp3s0f0np0 (X710), esh-nas-pve enp5s0f0.
  5. NVIDIA on pve: first make sure proxmox-default-headers is installed. On nh3-pve (2026-10-03) the upgrade put in kernel 7.0 with NO 7.0 headers, so DKMS never built the module and the GPU container did not start. The fix was apt install proxmox-default-headers proxmox-headers-<new> + dkms autoinstall. 580.178.04 does compile on 7.0.14. (playbooks/nh3-pve-pve9-postfix.yaml.) 580.178.04 is DKMS-built for 6.8. After the dist-upgrade and before rebooting, dkms status must show the module built for the new kernel; if not, reinstall the driver for trixie before rebooting. esh-ml1 (CT 110) is the fleet's only embed/rerank backend behind LiteLLM, and albok-service now depends on it. Put nh3-ml1 (its twin) behind the gateway first, so embeddings survive pve's window.
  6. Root snapshots for rollback:
    • esh-nas-pve: zfs snapshot -r nvme/ROOT@pre-pve9.
    • pve: lvcreate -s -L 15G -n root-pre-pve9 pve/root (VG has 16 G free).

Order and window

Recommended: esh-nas-pve first, then pve, in one evening window of about 2–2.5 h. Doing the NAS node first keeps the remote path (CT 108 on pve) up while the data-heavy node is worked on, so anything that goes wrong there is still diagnosable from outside.

  1. On pve, cleanly shut down VM 101 (esh-vm-db, disk on NFS from 10.0.50.50) and VM 100 (esh-vm-docker, hard NFS mounts). A NAS reboot under them means I/O hangs or unkillable D-state (2026 incident).
  2. esh-nas-pve: switch repos, dist-upgrade, reboot. Verify that all pools import, tank state, NFS exports, and that guests 103–107 are up.
  3. Start VMs 100 and 101 on pve; verify the DB and Docker services.
  4. pve: switch repos, dist-upgrade, check dkms status, reboot. Verify CT 108 (mesh), CT 110 (TEI answering through LiteLLM), CT 111 (Matter), VMs 100/101.
  5. pve8to9 --full on both (it has post-upgrade checks), pvecm status (quorate), guest list.

Repo switch, per node (verify against the wiki on the day):

sed -i 's/bookworm/trixie/g' /etc/apt/sources.list            # Debian main/updates/security
# replace the bookworm pve-no-subscription line with deb822 /etc/apt/sources.list.d/proxmox.sources:
#   Types: deb
#   URIs: http://download.proxmox.com/debian/pve
#   Suites: trixie
#   Components: pve-no-subscription
#   Signed-By: /usr/share/keyrings/proxmox-archive-keyring.gpg
apt update && apt dist-upgrade        # keep local config where asked unless the wiki says otherwise

Expected downtime and guest behaviour

  • The apt phase (15–30 min per node): guests keep running.
  • The reboot phase: that node's guests stop and restart on boot (all are onboot=1):
    • roughly 5–10 min for pve;
    • 10–20 min for esh-nas-pve (the 175 T tank import, NFS);
    • VMs 100/101 are down from step 1 until step 3, roughly 30–45 min.
  • No migration: no HA and no live migration (local storage on both nodes, cpu=host, different CPUs). Guests are stopped, not moved.
  • Quorum: while either node reboots, the other loses quorum. Its running guests keep running, but you cannot start, stop or reconfigure guests and /etc/pve is read-only until the node is back. Do not use pvecm expected 1 unless something is stuck.
  • During pve's reboot:
    • ESH loses its mesh router (no remote path; the Matter/HA bridge drops);
    • fleet embeddings go down unless nh3-ml1 is behind the gateway;
    • the ESH AdGuard resolver (on esh-vm-docker) is down; the DNS ring falls back to the other sites.

Rollback: honest version

There is no supported in-place downgrade from PVE 9 / Debian 13 back to 8 / 12. What exists:

  • esh-nas-pve: zfs rollback nvme/ROOT/pve-1@pre-pve9 from a rescue shell (PVE ISO, or console). ⚠ Never run zpool upgrade on nvme after the upgrade. GRUB reads that pool directly, and new feature flags can make it unbootable and break this rollback.
  • pve: merge the LVM snapshot (lvconvert --merge pve/root-pre-pve9) from rescue, then reboot. The snapshot only holds while its 15 G covers the changes; drop it once 9 is confirmed good.
  • Last resort: reinstall 8.4 and restore /etc and guests from the pre-flight PBS backups (hours).
  • Guest data is not touched by the host upgrade, so the risk is host availability, not data loss. Both rollbacks need console access, which is why finding 2 matters.

esh-pve: prep DONE 2026-10-04 0729-0733 (playbooks/esh-pve-pve9-prep.yaml, no downtime; Prime: "okay, do esh-pve")

  • 8.4.21, kernel 6.8.12-43 with nvidia 580.178.04 DKMS built for it (proxmox-default-headers was already in, so the nh3-pve headers trap does not apply); intel-microcode + non-free-firmware; systemd-boot removed (the pve8to9 FAIL); force_efi_extra_removable true, so EFI/BOOT now carries the same shim + GRUB as EFI/proxmox (was 2024-05-18).
  • NIC pins by MAC for enp3s0f0np0/enp3s0f1np1 (X710) and enp88s0/enp89s0 (igc).
  • ⚠ Found and fixed: vmbr0's bridge-ports still named enp87s0, which has not existed since the RTX 2000E went in on 2026-09-24 and renumbered the igc ports (enp87s0/enp88s0 → enp88s0/enp89s0). That boot logged "bridge port enp87s0 does not exist … bridge configuration failed (missing ports)"; vmbr0 came up on the X710 anyway. bridge-ports is now enp3s0f0np0 enp3s0f1np1, which is what was actually forwarding (enp88s0 was enslaved but admin-down and stays out: a 2.5G port brought up in an STP-off bridge is an L2 loop if a second cable is ever in). The iface stanzas were renamed to match. File only, no live reload; takes effect at the next boot. Backup /root/pve9-prep-20261004/interfaces.before.
  • pve8to9 --full: 0 FAIL, 3 WARN (2-node cluster, running guests, dkms).
  • Nightly vzdump 03:30 OK (100, 101, 111); 108 (subnet router) excluded by design; 110 esh-ml1 not backed up and not one-off'd: rebuildable from stacks/embed-rerank, and embed/rerank are load-shared with nh3-ml1 since 2026-09-26.

esh-pve: UPGRADED to PVE 9.2.21 / kernel 7.0.14-20, 2026-10-04 0733-0745 (Prime on site; ONE reboot, no PVE 8 test, Prime's call)

  • playbooks/esh-pve-pve9-upgrade.yaml: guards (pve8to9 0 FAIL; prep in; nvidia built for the newest 6.8; VG 16 G free + dm-snapshot in initramfs), root snapshot pve/root-pre-pve9 15 G, trixie sources (ceph.list + pve-enterprise.list moved to /root/pve9-prep-20261004/), plan read (833 installs; removals only t64 libs + python3-distutils/future/lib2to3; proxmox-headers-7.0 came with it), detached dist-upgrade 0734-0740, EXIT 0.
  • playbooks/esh-pve-pve9-prereboot.yaml: all 8 OK — nvidia 580.178.04 DKMS built for 7.0.14-20 ("Building module(s) … done"; the nh3-pve trap did not recur because proxmox-default-headers was in), pins + dm-snapshot in the 7.0 initramfs, shim + GRUB rewritten in both EFI/proxmox and EFI/BOOT ("Installation finished. No error reported."), every bridge port exists, enterprise repo Enabled: false. pve8to9 0 FAIL.
  • Reboot 0740:54. ESH watcher (from nh3-dev): esh-ml1 dark 0741:12, esh-docker-vm 0741:26, esh-pve + esh-scale 0741:36; esh-pve + mesh back 0742:39 (the MS-01 boots in about a minute); esh-ml1 + esh-docker-vm back ~0743:10. ESH impact: mesh ~1 min; DNS / Home Assistant / Zigbee ~2 min.
  • Post-check PASSED 0745: kernel 7.0.14-20; names held; vmbr0 = enp3s0f0np0 + enp3s0f1np1 and the networking journal is clean (the "bridge port enp87s0 does not exist" error is gone); no failed units; chrony, corosync active; cluster quorate, both nodes now 9.2.21; nvidia module loaded, esh-ml1 sees the RTX 2000E (11.8 GiB used), TEI :8001/:8013 200; gateway embeddings 6/6 (dim 1024); esh-docker-vm 18/18 Up (same set as before) with mosquitto and zigbee2mqtt up by themselves (z2m "MQTT publish" from 0743:17; restart counts 0), AdGuard resolving, HA :8123 200, Homepage 200; esh-vm-db postgres + mongod active; esh-matter listening :5580; esh-scale on the mesh. pve8to9 0 FAIL, 2 WARN (2-node, dkms).
  • Open: drop pve/root-pre-pve9 on/after 2026-10-05 (infra-hermes, with pfi-pve's). Never zpool upgrade the ESH pools.
Reference in New Issue View Git Blame Copy Permalink