ops(esh-pve): upgraded to PVE 9.2.21 / kernel 7.0.14-20 (one reboot, Prime on site); ESH cluster uniform on 9

2026-10-04 07:45:05 -07:00
parent 9a6b23581e
commit 73129a0564
5 changed files with 206 additions and 5 deletions
@@ -1,4 +1,4 @@
-# esh-pve-cluster: Proxmox VE 8 → 9 upgrade plan (PLAN ONLY — not executed)
+# esh-pve-cluster: Proxmox VE 8 → 9 upgrade plan — DONE (esh-nas-pve 2026-10-03, esh-pve 2026-10-04)

 Requested by Prime via Miranda, 2026-10-02 (thread `01M3YX5QGQ5V8BQXYM46E5T3PB`). **Do not execute
 without a further green light from Prime.** Facts below were read live on 2026-10-02 ~1930 PT,
@@ -184,3 +184,24 @@ There is **no supported in-place downgrade** from PVE 9 / Debian 13 back to 8 /
 - pve8to9 --full: 0 FAIL, 3 WARN (2-node cluster, running guests, dkms).
 - Nightly vzdump 03:30 OK (100, 101, 111); 108 (subnet router) excluded by design; 110 esh-ml1 not backed up and not
  one-off'd: rebuildable from `stacks/embed-rerank`, and embed/rerank are load-shared with nh3-ml1 since 2026-09-26.
+
+## esh-pve: UPGRADED to PVE 9.2.21 / kernel 7.0.14-20, 2026-10-04 0733-0745 (Prime on site; ONE reboot, no PVE 8 test, Prime's call)
+
+- `playbooks/esh-pve-pve9-upgrade.yaml`: guards (pve8to9 0 FAIL; prep in; nvidia built for the newest 6.8; VG 16 G free +
+  dm-snapshot in initramfs), root snapshot `pve/root-pre-pve9` 15 G, trixie sources (`ceph.list` + `pve-enterprise.list`
+  moved to `/root/pve9-prep-20261004/`), plan read (833 installs; removals only t64 libs + python3-distutils/future/lib2to3;
+  `proxmox-headers-7.0` came with it), detached dist-upgrade 0734-0740, EXIT 0.
+- `playbooks/esh-pve-pve9-prereboot.yaml`: all 8 OK — **nvidia 580.178.04 DKMS built for 7.0.14-20** ("Building module(s)
+  … done"; the nh3-pve trap did not recur because `proxmox-default-headers` was in), pins + dm-snapshot in the 7.0 initramfs,
+  shim + GRUB rewritten in both EFI/proxmox and EFI/BOOT ("Installation finished. No error reported."), every bridge port
+  exists, enterprise repo `Enabled: false`. pve8to9 0 FAIL.
+- Reboot 0740:54. ESH watcher (from nh3-dev): esh-ml1 dark 0741:12, esh-docker-vm 0741:26, esh-pve + esh-scale 0741:36;
+  **esh-pve + mesh back 0742:39** (the MS-01 boots in about a minute); esh-ml1 + esh-docker-vm back ~0743:10.
+  **ESH impact: mesh ~1 min; DNS / Home Assistant / Zigbee ~2 min.**
+- Post-check PASSED 0745: kernel 7.0.14-20; names held; vmbr0 = enp3s0f0np0 + enp3s0f1np1 and **the networking journal is clean**
+  (the "bridge port enp87s0 does not exist" error is gone); no failed units; chrony, corosync active; **cluster quorate, both
+  nodes now 9.2.21**; nvidia module loaded, esh-ml1 sees the RTX 2000E (11.8 GiB used), TEI :8001/:8013 200; gateway
+  embeddings 6/6 (dim 1024); esh-docker-vm 18/18 Up (same set as before) with **mosquitto and zigbee2mqtt up by themselves**
+  (z2m "MQTT publish" from 0743:17; restart counts 0), AdGuard resolving, HA :8123 200, Homepage 200; esh-vm-db postgres +
+  mongod active; esh-matter listening :5580; esh-scale on the mesh. pve8to9 0 FAIL, 2 WARN (2-node, dkms).
+- Open: drop `pve/root-pre-pve9` on/after 2026-10-05 (infra-hermes, with pfi-pve's). Never `zpool upgrade` the ESH pools.