pve-manager 8.4.11 -> 8.4.20, corosync 3.1.9 -> 3.1.10-pve2, and kernel
6.8.12-42 staged on the /boot LV. dpkg clean, nothing outstanding for
apt -f install, all PVE services active, cluster quorate, no unapplied
conffiles. Reboot deliberately deferred at operator request, so the host
still runs 6.8.12-13 until a chosen window.
This validates the GRUB fix from 061c4b7 under the exact condition it
was written for. update-grub regenerated entries for the new kernel and
entry 0 -- what GRUB_DEFAULT=0 selects -- is now
/vmlinuz-6.8.12-42-pve with root=ZFS=nvme/ROOT/pve-1, supplied by the
grub.d drop-in since grub-mkconfig cannot derive the pool name itself.
The old kernel keeps correct entries as a fallback and the ext4 rollback
entry is untouched. Had the fix not landed first, saved_entry would
still be pinned to 6.8.12-13 and the host would boot the old kernel
indefinitely -- 161 security updates installed and never run.
/boot holds both kernel sets at 176M used of 488M, confirming the 512M
LV carved out of swap was sized correctly.
Adds a ZFS snapshot step to the upgrade playbook, taken automatically on
ZFS-root nodes before any package lands. That is the first real use of
the boot-environment upside the migration was meant to unlock: rollback
for this upgrade is now `zfs rollback -r
nvme/ROOT/pve-1@pre-upgrade-20260818T141652Z && reboot` rather than
archaeology in dpkg. Also documents that the corosync bump restarts
corosync mid-upgrade, which on a 2-node cluster is a brief quorum event.
171 packages, pve-manager 8.4.11 -> 8.4.20, kernel 6.8.12-16 -> 6.8.12-42,
corosync 3.1.9 -> 3.1.10-pve2. dpkg clean, no unapplied conffiles, no
failed units, cluster quorate with both nodes visible after the reboot.
Adds a reusable pve-node-upgrade playbook (upgrade only -- reboot stays a
separate deliberate step, since it has cluster and NFS consequences the
playbook cannot see). It guards on quorum and free space, snapshots
/etc/pve and friends first, uses --force-confdef/--force-confold, and
surfaces any .dpkg-dist files that policy left unapplied so they are not
silently ignored.
The reboot needed a forced guest stop, operator-authorised after the risk
was surfaced. Two obstacles, only one of them ours:
- A vzdump had been hung since 14 April -- 126 days, stalled at 0% of
256 GiB -- holding lock: backup on VM 102, which had therefore been
sitting in QEMU prelaunch that entire time. Killed by explicit PID; 102
is now cleanly stopped rather than half-alive.
- esh-vm-db would not shut down: its guest agent had died and ACPI went
unanswered. Most likely ours -- it hard-mounts /mnt/backup from CT 103,
which we deliberately left mounted through the NAS reboots.
PostgreSQL survived the hard stop. It had checkpointed five minutes
prior, so recovery replayed 56 bytes of WAL in 0.02s and came up ready;
all four databases present and queryable. That was lucky timing as much
as anything -- a hard stop mid-checkpoint on a busy database would not
read the same way.
The reboot also repaired esh-vm-db, which had silently lost sshd, mongod
and its guest agent. All three are back.
vh