ops(esh-nas-pve): upgraded to PVE 9.2.21; /boot-LV-full trap and fixes recorded

This commit is contained in:
vh
2026-10-03 16:06:33 -07:00
parent ca18b04a22
commit 9ba8962885
2 changed files with 18 additions and 1 deletions
@@ -49,6 +49,23 @@ Quorum: 2 nodes × 1 vote, no QDevice, no `two_node`.
- Infra-ops can do the prep remotely. The upgrade reboot should happen only with someone able to reach ESH, or with
Prime's explicit acceptance that a failed boot means an outage until someone is on site.
## esh-nas-pve: UPGRADED to PVE 9.2.21 / kernel 7.0.14-20, 2026-10-03 1552-1606 (Prime on site)
`playbooks/esh-nas-pve-pve9-upgrade.yaml`: root snapshot `nvme/ROOT@pre-pve9` (keep about a week, then destroy), trixie sources,
dist-upgrade run DETACHED (log `/root/pve9-upgrade.log`). What it took:
- ⚠ **`/boot` is a separate 488 MB ext4 LV (`/dev/mapper/pve-boot`)**; ZFS holds only `/`. With four kernels it hit 91%, and
the second mkinitramfs for 7.0 failed with "No space left on device" (dist-upgrade EXIT 100, initramfs-tools half-configured).
Fix: `apt remove` the two oldest 6.8 kernels → 49%, `dpkg --configure -a`, re-run dist-upgrade → clean.
**For `pve`: check `df /boot` and trim old kernels BEFORE the dist-upgrade.**
- The upgrade adds `pve-enterprise.sources`, enabled. Set `Enabled: false`.
- chrony failed right after the upgrade ("Could not send notification to $NOTIFY_SOCKET", new chronyd under the old
running systemd/kernel). It was fine after the reboot: synced and active.
- The two old community de-nag hooks were retired to `/root/pve9-prep-20261003/`; `pve-nag-patch` re-patched the 9.x UI.
- Reboot 16:01:06, down 16:01:31, **up 16:04:14**; pinned names held; pools ONLINE; guests + NFS back 16:05:28.
- **VM 100 stayed up** (stopped calibre-web-automated + beszel-agent, unmounted /mnt/books + /mnt/backup, remounted
after), so **ESH DNS never went down**. VM 101 was down 16:00:52-16:05:47 and started first try this time.
- pve8to9 after: 0 FAIL, 1 WARN (2-node cluster). `openipmi.service` fails at boot (QNAP has no IPMI); harmless, can be masked.
## Reboot test on PVE 8: DONE 2026-10-03 1533-1546 (Prime on site)
Prep: `playbooks/esh-nas-pve-pve9-prep.yaml` (8.4.21, kernel 6.8.12-43, intel-microcode, systemd-boot removed, NIC pins);