ops(esh-nas-pve): PVE 8 reboot test done; two restart foot-guns recorded

docs/runbooks/esh-pve-cluster-pve9-upgrade-plan.md

@@ -49,6 +49,21 @@ Quorum: 2 nodes × 1 vote, no QDevice, no `two_node`.
 - Infra-ops can do the prep remotely. The upgrade reboot should happen only with someone able to reach ESH, or with
   Prime's explicit acceptance that a failed boot means an outage until someone is on site.
 
+## Reboot test on PVE 8: DONE 2026-10-03 1533-1546 (Prime on site)
+
+Prep: `playbooks/esh-nas-pve-pve9-prep.yaml` (8.4.21, kernel 6.8.12-43, intel-microcode, systemd-boot removed, NIC pins);
+one-off backups of 103-107 to pbs-ana (1530-1532); pve8to9 0 FAIL. Then:
+- VMs 100/101 on `pve` were shut down cleanly (15 s each).
+- esh-nas-pve: reboot 15:33:23, down at 15:34:12, **up at 15:36:58** on 6.8.12-43.
+- **Pinned names held** and vmbr0 came back on .55; pools ONLINE; microcode active.
+- Guests 103-107 started one after another and were all up by ~15:38; NFS (:2049) was back at 15:38.
+- ⚠ **VM 101's first start timed out** (`qm start` "got timeout" at 15:38:29): its disk is qcow2 on the `esh-nas` NFS
+  storage, still recovering. **Retry after a few minutes**; it started cleanly at 15:45 (postgres + mongod active).
+- ⚠ **mosquitto on VM 100 did NOT come back by itself**: it exited 0 at shutdown, and although it is `unless-stopped`,
+  dockerd did not restart it. zigbee2mqtt restart-looped without its broker until mosquitto was started by hand
+  (`compose up -d`, 15:45). **After any VM 100 reboot, check mosquitto and zigbee2mqtt.**
+- Total ESH impact: DNS ~6 min (15:33-15:39); DB ~12 min; Zigbee ~13 min; NAS/media ~4 min.
+
 ## Pre-flight (no guest downtime)
 
 1. **Backups:** one-off `vzdump` of **all ten** guests to pbs-ana in snapshot mode, right before the

persistent-memory.md

@@ -130,7 +130,7 @@ _As of 2026-10-03 ~1440 PT. The newest subsection is first; older subsections ca
 - **cira-tunnel-watchdog** on pfi-tacticalrmm has been live since 1232 and drops :4433 tunnels silent >180 s. Watch `journalctl -t cira-tunnel-watchdog`.
 - **RustDesk server (ana-docker) updated 1.1.14 → 1.1.16 at 1424** (same key; ports 21115-21119 reachable via rustdesk.phasefinal.com). Private key vaulted first as `ana-docker/rustdesk/id_ed25519`. "Change ID" is refused by the OSS server BY DESIGN (Pro feature; `rendezvous_server.rs` answers TCP RegisterPk with NOT_SUPPORT). Workaround, read at client source and untested: stop RustDesk, delete `enc_id` from RustDesk.toml, set a plain `id`, restart. → `servers/ana-docker/README.md`
 - **Booth vs 10.100.10.166 (Prime, ~1420): that client cannot reach the Booth.** The Booth was up and answered from 4 hosts at 1417; .166 is on nh3-dev's own subnet. Prime interrupted my check, so it is OPEN and resumes only on his word.
-- **esh-nas-pve → PVE 9: Prime asked (1440) whether I can do it myself. Assessed read-only and ANSWERED; awaiting his call.** Facts: QNAP hardware, no AMT/IPMI (no remote console); PVE 8.4.20 but running kernel 6.8.12-13; up since 2026-08-18; ZFS root `nvme` booted by GRUB; systemd-boot meta present; NO NIC pins (uplink `enp5s0f0`); no DKMS. My answer: I can do all the prep remotely (vzdump, NIC pin, remove systemd-boot, root snapshot, pve8to9, a reboot test on PVE 8 first). The upgrade reboot should happen only with someone able to reach ESH, or with Prime's explicit acceptance of a no-console failure. VMs 100/101 on `pve` must be shut down first (hard NFS mounts). → `docs/runbooks/esh-pve-cluster-pve9-upgrade-plan.md`
+- **esh-nas-pve → PVE 9: PREP + PVE 8 REBOOT TEST DONE (1533-1546, Prime on site).** Prep playbook `playbooks/esh-nas-pve-pve9-prep.yaml`; backups 103-107 on pbs-ana; came back on 6.8.12-43 with the pinned names. Two foot-guns are logged in the plan doc: VM 101's first start times out while its NFS recovers (retry), and mosquitto does not auto-restart on VM 100. **NEXT: the actual 8→9 upgrade on esh-nas-pve, awaiting Prime's go** (root snapshot `nvme/ROOT@pre-pve9` first). (Earlier assessment, kept:) Prime asked (1440) whether I can do it myself. Assessed read-only and ANSWERED. Facts: QNAP hardware, no AMT/IPMI (no remote console); PVE 8.4.20 but running kernel 6.8.12-13; up since 2026-08-18; ZFS root `nvme` booted by GRUB; systemd-boot meta present; NO NIC pins (uplink `enp5s0f0`); no DKMS. My answer: I can do all the prep remotely (vzdump, NIC pin, remove systemd-boot, root snapshot, pve8to9, a reboot test on PVE 8 first). The upgrade reboot should happen only with someone able to reach ESH, or with Prime's explicit acceptance of a no-console failure. VMs 100/101 on `pve` must be shut down first (hard NFS mounts). → `docs/runbooks/esh-pve-cluster-pve9-upgrade-plan.md`
 - **Done today, nothing open:**
   - tank verified healthy 0629, disk kept, Miranda told;
   - albok-service 0.1.2 + the Nemi hourly timer;