From 8ddc87c852adc8b173e9d4f47b45488f89a3b165 Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Mon, 17 Aug 2026 21:40:24 -0700 Subject: [PATCH] docs(esh-pve-nas): record the blocked-patching driver and the upgrade ordering The operator-visible symptom is that PVE cannot be updated on this box for lack of room. Measured: 225 packages pending, 161 carrying deb12uN/Debian-Security bumps including ssh, against esh-pve's 8.4.14 versus this host's 8.4.11 and 20 weeks of uptime. Records the ordering explicitly -- migrate first, upgrade after. The pending set includes proxmox-kernel-6.8.12-42-pve-signed, roughly 250 MB of kernel plus initramfs landing in /boot which is on root with 1.3 GB free. Unpacking 225 packages including dpkg and perl into that headroom risks filling the disk mid-transaction and wedging dpkg on a hypervisor running five guests. Notes the apt archive-dir redirect as a partial escape hatch if patching cannot wait, and that zfs-initramfs 2.2.8 is fully capable of root-on-ZFS so there is no need to upgrade ZFS before migrating. --- docs/runbooks/esh-pve-nas-boot-migration.md | 22 ++++++++++++++++++++- 1 file changed, 21 insertions(+), 1 deletion(-) diff --git a/docs/runbooks/esh-pve-nas-boot-migration.md b/docs/runbooks/esh-pve-nas-boot-migration.md index 10da43d..5bff988 100644 --- a/docs/runbooks/esh-pve-nas-boot-migration.md +++ b/docs/runbooks/esh-pve-nas-boot-migration.md @@ -19,6 +19,25 @@ unremarkable and **wear is not the driver**. The actual problems: filesystem* out from under a running hypervisor while its guests keep going. 2. **6 GB has no headroom** — `/usr` alone is 3.7 GB. 3. **Unmirrored**, while 928 GB of mirrored NVMe sits 96% empty. +4. **It has blocked patching for months.** This is the operator-visible symptom + and the real urgency: `apt-get -s dist-upgrade` shows **225 packages pending, + 161 of them carrying `deb12uN` / Debian-Security bumps** — including `ssh + 1:9.2p1-2+deb12u10`. The host sits on `pve-manager/8.4.11` while its sibling + esh-pve is on 8.4.14, and it has 20 weeks of uptime because it cannot take a + kernel. + + ⚠ **Do not attempt the upgrade before the migration.** The pending set + includes `proxmox-kernel-6.8.12-42-pve-signed` (from -13) — a signed kernel + plus initramfs is ~250 MB, and **`/boot` is on root**, which has 1.3 GB free. + 225 packages unpacking (dpkg, perl, glibc-adjacent) into that headroom risks + filling the disk mid-transaction and leaving a broken dpkg state on a + hypervisor running five guests. Recovering a wedged dpkg on a full root is + far worse than waiting for the reboot. + + If patching genuinely cannot wait, the escape hatch is to keep downloads off + root — `apt-get -o Dir::Cache::Archives=/nvme/tmp/apt-archives dist-upgrade` + — but the kernel still lands in `/boot` on root, so this reduces the risk + rather than removing it. Migrating first is the shorter path to safety. ## The design: boot on the DOM, root on ZFS @@ -63,7 +82,8 @@ Verified on the host 2026-08-17: - **UEFI** firmware, `grub-efi-amd64 2.06-13+pmx7` installed - **`zfs-initramfs 2.2.8-pve1` is already installed**, and the running initrd already carries **76 ZFS files** — the pivot capability exists today, no new - packages + packages. (The pending upgrade would take ZFS to 2.2.10-pve1; 2.2.8 is fully + capable of root-on-ZFS, so migrate on what is installed and upgrade after.) - `/boot` is currently *part of* root (108 MB), so it must be split out onto its own ext4 filesystem on the DOM as part of this work - root is only **4.3 GB** to copy