From 48f28cfe93169dbb3dc49a43b9b1ca3b686e4571 Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Sat, 3 Oct 2026 15:46:37 -0700 Subject: [PATCH] ops(esh-nas-pve): PVE 8 reboot test done; two restart foot-guns recorded --- .../runbooks/esh-pve-cluster-pve9-upgrade-plan.md | 15 +++++++++++++++ persistent-memory.md | 2 +- 2 files changed, 16 insertions(+), 1 deletion(-) diff --git a/docs/runbooks/esh-pve-cluster-pve9-upgrade-plan.md b/docs/runbooks/esh-pve-cluster-pve9-upgrade-plan.md index 9dd359b..e2f13f7 100644 --- a/docs/runbooks/esh-pve-cluster-pve9-upgrade-plan.md +++ b/docs/runbooks/esh-pve-cluster-pve9-upgrade-plan.md @@ -49,6 +49,21 @@ Quorum: 2 nodes × 1 vote, no QDevice, no `two_node`. - Infra-ops can do the prep remotely. The upgrade reboot should happen only with someone able to reach ESH, or with Prime's explicit acceptance that a failed boot means an outage until someone is on site. +## Reboot test on PVE 8: DONE 2026-10-03 1533-1546 (Prime on site) + +Prep: `playbooks/esh-nas-pve-pve9-prep.yaml` (8.4.21, kernel 6.8.12-43, intel-microcode, systemd-boot removed, NIC pins); +one-off backups of 103-107 to pbs-ana (1530-1532); pve8to9 0 FAIL. Then: +- VMs 100/101 on `pve` were shut down cleanly (15 s each). +- esh-nas-pve: reboot 15:33:23, down at 15:34:12, **up at 15:36:58** on 6.8.12-43. +- **Pinned names held** and vmbr0 came back on .55; pools ONLINE; microcode active. +- Guests 103-107 started one after another and were all up by ~15:38; NFS (:2049) was back at 15:38. +- ⚠ **VM 101's first start timed out** (`qm start` "got timeout" at 15:38:29): its disk is qcow2 on the `esh-nas` NFS + storage, still recovering. **Retry after a few minutes**; it started cleanly at 15:45 (postgres + mongod active). +- ⚠ **mosquitto on VM 100 did NOT come back by itself**: it exited 0 at shutdown, and although it is `unless-stopped`, + dockerd did not restart it. zigbee2mqtt restart-looped without its broker until mosquitto was started by hand + (`compose up -d`, 15:45). **After any VM 100 reboot, check mosquitto and zigbee2mqtt.** +- Total ESH impact: DNS ~6 min (15:33-15:39); DB ~12 min; Zigbee ~13 min; NAS/media ~4 min. + ## Pre-flight (no guest downtime) 1. **Backups:** one-off `vzdump` of **all ten** guests to pbs-ana in snapshot mode, right before the diff --git a/persistent-memory.md b/persistent-memory.md index 6198c26..22ed81b 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -130,7 +130,7 @@ _As of 2026-10-03 ~1440 PT. The newest subsection is first; older subsections ca - **cira-tunnel-watchdog** on pfi-tacticalrmm has been live since 1232 and drops :4433 tunnels silent >180 s. Watch `journalctl -t cira-tunnel-watchdog`. - **RustDesk server (ana-docker) updated 1.1.14 → 1.1.16 at 1424** (same key; ports 21115-21119 reachable via rustdesk.phasefinal.com). Private key vaulted first as `ana-docker/rustdesk/id_ed25519`. "Change ID" is refused by the OSS server BY DESIGN (Pro feature; `rendezvous_server.rs` answers TCP RegisterPk with NOT_SUPPORT). Workaround, read at client source and untested: stop RustDesk, delete `enc_id` from RustDesk.toml, set a plain `id`, restart. → `servers/ana-docker/README.md` - **Booth vs 10.100.10.166 (Prime, ~1420): that client cannot reach the Booth.** The Booth was up and answered from 4 hosts at 1417; .166 is on nh3-dev's own subnet. Prime interrupted my check, so it is OPEN and resumes only on his word. -- **esh-nas-pve → PVE 9: Prime asked (1440) whether I can do it myself. Assessed read-only and ANSWERED; awaiting his call.** Facts: QNAP hardware, no AMT/IPMI (no remote console); PVE 8.4.20 but running kernel 6.8.12-13; up since 2026-08-18; ZFS root `nvme` booted by GRUB; systemd-boot meta present; NO NIC pins (uplink `enp5s0f0`); no DKMS. My answer: I can do all the prep remotely (vzdump, NIC pin, remove systemd-boot, root snapshot, pve8to9, a reboot test on PVE 8 first). The upgrade reboot should happen only with someone able to reach ESH, or with Prime's explicit acceptance of a no-console failure. VMs 100/101 on `pve` must be shut down first (hard NFS mounts). → `docs/runbooks/esh-pve-cluster-pve9-upgrade-plan.md` +- **esh-nas-pve → PVE 9: PREP + PVE 8 REBOOT TEST DONE (1533-1546, Prime on site).** Prep playbook `playbooks/esh-nas-pve-pve9-prep.yaml`; backups 103-107 on pbs-ana; came back on 6.8.12-43 with the pinned names. Two foot-guns are logged in the plan doc: VM 101's first start times out while its NFS recovers (retry), and mosquitto does not auto-restart on VM 100. **NEXT: the actual 8→9 upgrade on esh-nas-pve, awaiting Prime's go** (root snapshot `nvme/ROOT@pre-pve9` first). (Earlier assessment, kept:) Prime asked (1440) whether I can do it myself. Assessed read-only and ANSWERED. Facts: QNAP hardware, no AMT/IPMI (no remote console); PVE 8.4.20 but running kernel 6.8.12-13; up since 2026-08-18; ZFS root `nvme` booted by GRUB; systemd-boot meta present; NO NIC pins (uplink `enp5s0f0`); no DKMS. My answer: I can do all the prep remotely (vzdump, NIC pin, remove systemd-boot, root snapshot, pve8to9, a reboot test on PVE 8 first). The upgrade reboot should happen only with someone able to reach ESH, or with Prime's explicit acceptance of a no-console failure. VMs 100/101 on `pve` must be shut down first (hard NFS mounts). → `docs/runbooks/esh-pve-cluster-pve9-upgrade-plan.md` - **Done today, nothing open:** - tank verified healthy 0629, disk kept, Miranda told; - albok-service 0.1.2 + the Nemi hourly timer;