ops(esh-nas-pve): PVE 8 reboot test done; two restart foot-guns recorded

2026-10-03 15:46:37 -07:00
parent f648e18b1a
commit 48f28cfe93
2 changed files with 16 additions and 1 deletions
@@ -49,6 +49,21 @@ Quorum: 2 nodes × 1 vote, no QDevice, no `two_node`.
 - Infra-ops can do the prep remotely. The upgrade reboot should happen only with someone able to reach ESH, or with
  Prime's explicit acceptance that a failed boot means an outage until someone is on site.

+## Reboot test on PVE 8: DONE 2026-10-03 1533-1546 (Prime on site)
+
+Prep: `playbooks/esh-nas-pve-pve9-prep.yaml` (8.4.21, kernel 6.8.12-43, intel-microcode, systemd-boot removed, NIC pins);
+one-off backups of 103-107 to pbs-ana (1530-1532); pve8to9 0 FAIL. Then:
+- VMs 100/101 on `pve` were shut down cleanly (15 s each).
+- esh-nas-pve: reboot 15:33:23, down at 15:34:12, **up at 15:36:58** on 6.8.12-43.
+- **Pinned names held** and vmbr0 came back on .55; pools ONLINE; microcode active.
+- Guests 103-107 started one after another and were all up by ~15:38; NFS (:2049) was back at 15:38.
+- ⚠ **VM 101's first start timed out** (`qm start` "got timeout" at 15:38:29): its disk is qcow2 on the `esh-nas` NFS
+  storage, still recovering. **Retry after a few minutes**; it started cleanly at 15:45 (postgres + mongod active).
+- ⚠ **mosquitto on VM 100 did NOT come back by itself**: it exited 0 at shutdown, and although it is `unless-stopped`,
+  dockerd did not restart it. zigbee2mqtt restart-looped without its broker until mosquitto was started by hand
+  (`compose up -d`, 15:45). **After any VM 100 reboot, check mosquitto and zigbee2mqtt.**
+- Total ESH impact: DNS ~6 min (15:33-15:39); DB ~12 min; Zigbee ~13 min; NAS/media ~4 min.
+
 ## Pre-flight (no guest downtime)

 1. **Backups:** one-off `vzdump` of **all ten** guests to pbs-ana in snapshot mode, right before the