ops(esh-nas-pve): upgraded to PVE 9.2.21; /boot-LV-full trap and fixes recorded

docs/runbooks/esh-pve-cluster-pve9-upgrade-plan.md

@@ -49,6 +49,23 @@ Quorum: 2 nodes × 1 vote, no QDevice, no `two_node`.
 - Infra-ops can do the prep remotely. The upgrade reboot should happen only with someone able to reach ESH, or with
   Prime's explicit acceptance that a failed boot means an outage until someone is on site.
 
+## esh-nas-pve: UPGRADED to PVE 9.2.21 / kernel 7.0.14-20, 2026-10-03 1552-1606 (Prime on site)
+
+`playbooks/esh-nas-pve-pve9-upgrade.yaml`: root snapshot `nvme/ROOT@pre-pve9` (keep about a week, then destroy), trixie sources,
+dist-upgrade run DETACHED (log `/root/pve9-upgrade.log`). What it took:
+- ⚠ **`/boot` is a separate 488 MB ext4 LV (`/dev/mapper/pve-boot`)**; ZFS holds only `/`. With four kernels it hit 91%, and
+  the second mkinitramfs for 7.0 failed with "No space left on device" (dist-upgrade EXIT 100, initramfs-tools half-configured).
+  Fix: `apt remove` the two oldest 6.8 kernels → 49%, `dpkg --configure -a`, re-run dist-upgrade → clean.
+  **For `pve`: check `df /boot` and trim old kernels BEFORE the dist-upgrade.**
+- The upgrade adds `pve-enterprise.sources`, enabled. Set `Enabled: false`.
+- chrony failed right after the upgrade ("Could not send notification to $NOTIFY_SOCKET", new chronyd under the old
+  running systemd/kernel). It was fine after the reboot: synced and active.
+- The two old community de-nag hooks were retired to `/root/pve9-prep-20261003/`; `pve-nag-patch` re-patched the 9.x UI.
+- Reboot 16:01:06, down 16:01:31, **up 16:04:14**; pinned names held; pools ONLINE; guests + NFS back 16:05:28.
+- **VM 100 stayed up** (stopped calibre-web-automated + beszel-agent, unmounted /mnt/books + /mnt/backup, remounted
+  after), so **ESH DNS never went down**. VM 101 was down 16:00:52-16:05:47 and started first try this time.
+- pve8to9 after: 0 FAIL, 1 WARN (2-node cluster). `openipmi.service` fails at boot (QNAP has no IPMI); harmless, can be masked.
+
 ## Reboot test on PVE 8: DONE 2026-10-03 1533-1546 (Prime on site)
 
 Prep: `playbooks/esh-nas-pve-pve9-prep.yaml` (8.4.21, kernel 6.8.12-43, intel-microcode, systemd-boot removed, NIC pins);

persistent-memory.md

@@ -130,7 +130,7 @@ _As of 2026-10-03 ~1440 PT. The newest subsection is first; older subsections ca
 - **cira-tunnel-watchdog** on pfi-tacticalrmm has been live since 1232 and drops :4433 tunnels silent >180 s. Watch `journalctl -t cira-tunnel-watchdog`.
 - **RustDesk server (ana-docker) updated 1.1.14 → 1.1.16 at 1424** (same key; ports 21115-21119 reachable via rustdesk.phasefinal.com). Private key vaulted first as `ana-docker/rustdesk/id_ed25519`. "Change ID" is refused by the OSS server BY DESIGN (Pro feature; `rendezvous_server.rs` answers TCP RegisterPk with NOT_SUPPORT). Workaround, read at client source and untested: stop RustDesk, delete `enc_id` from RustDesk.toml, set a plain `id`, restart. → `servers/ana-docker/README.md`
 - **Booth vs 10.100.10.166 (Prime, ~1420): that client cannot reach the Booth.** The Booth was up and answered from 4 hosts at 1417; .166 is on nh3-dev's own subnet. Prime interrupted my check, so it is OPEN and resumes only on his word.
-- **esh-nas-pve → PVE 9: PREP + PVE 8 REBOOT TEST DONE (1533-1546, Prime on site).** Prep playbook `playbooks/esh-nas-pve-pve9-prep.yaml`; backups 103-107 on pbs-ana; came back on 6.8.12-43 with the pinned names. Two foot-guns are logged in the plan doc: VM 101's first start times out while its NFS recovers (retry), and mosquitto does not auto-restart on VM 100. **NEXT: the actual 8→9 upgrade on esh-nas-pve, awaiting Prime's go** (root snapshot `nvme/ROOT@pre-pve9` first). (Earlier assessment, kept:) Prime asked (1440) whether I can do it myself. Assessed read-only and ANSWERED. Facts: QNAP hardware, no AMT/IPMI (no remote console); PVE 8.4.20 but running kernel 6.8.12-13; up since 2026-08-18; ZFS root `nvme` booted by GRUB; systemd-boot meta present; NO NIC pins (uplink `enp5s0f0`); no DKMS. My answer: I can do all the prep remotely (vzdump, NIC pin, remove systemd-boot, root snapshot, pve8to9, a reboot test on PVE 8 first). The upgrade reboot should happen only with someone able to reach ESH, or with Prime's explicit acceptance of a no-console failure. VMs 100/101 on `pve` must be shut down first (hard NFS mounts). → `docs/runbooks/esh-pve-cluster-pve9-upgrade-plan.md`
+- **esh-nas-pve → PVE 9: PREP + PVE 8 REBOOT TEST DONE (1533-1546, Prime on site).** Prep playbook `playbooks/esh-nas-pve-pve9-prep.yaml`; backups 103-107 on pbs-ana; came back on 6.8.12-43 with the pinned names. Two foot-guns are logged in the plan doc: VM 101's first start times out while its NFS recovers (retry), and mosquitto does not auto-restart on VM 100. **DONE 1606: esh-nas-pve is ON PVE 9.2.21 / kernel 7.0.14-20** (Prime go; ESH DNS stayed up; the /boot-LV-full trap is fixed and written up in the plan doc). Keep `nvme/ROOT@pre-pve9` about a week, then destroy it. **`pve` (esh-pve) is still on 8.4.20; its upgrade is a separate, bigger window** (DNS VM, the CT 108 mesh router, esh-ml1 with NVIDIA, so it needs `proxmox-default-headers` + a /boot check first). (Earlier assessment, kept:) Prime asked (1440) whether I can do it myself. Assessed read-only and ANSWERED. Facts: QNAP hardware, no AMT/IPMI (no remote console); PVE 8.4.20 but running kernel 6.8.12-13; up since 2026-08-18; ZFS root `nvme` booted by GRUB; systemd-boot meta present; NO NIC pins (uplink `enp5s0f0`); no DKMS. My answer: I can do all the prep remotely (vzdump, NIC pin, remove systemd-boot, root snapshot, pve8to9, a reboot test on PVE 8 first). The upgrade reboot should happen only with someone able to reach ESH, or with Prime's explicit acceptance of a no-console failure. VMs 100/101 on `pve` must be shut down first (hard NFS mounts). → `docs/runbooks/esh-pve-cluster-pve9-upgrade-plan.md`
 - **Done today, nothing open:**
   - tank verified healthy 0629, disk kept, Miranda told;
   - albok-service 0.1.2 + the Nemi hourly timer;