esh-pfi-infrastructure/docs/runbooks/esh-pve-nas-boot-migration.md

esh-pve-nas — moving PVE root off the USB DOM

Status: DONE — cut over 2026-08-18. Root is nvme/ROOT/pve-1 on the mirrored NVMe; /boot is ext4 on the DOM; the DOM is out of the runtime I/O path. All five guests healthy, all three pools ONLINE, systemctl is-system-running = running. The ext4 root (pve-root) is intact, unmounted, and still carries its own kernel and initrd as the rollback.

Post-cutover boot config: saved_entry=pve-zfs-root, no next_entry. If grubenv were ever unreadable GRUB falls through to menu entry 0, which the /etc/default/grub.d/zfs-root.cfg drop-in also points at root=ZFS=nvme/ROOT/pve-1 — so every path boots ZFS.

⚠ The window cost an unplanned outage, caused by a bug in this runbook's own tooling, not by the migration. Read § The mount-propagation incident before running anything like this again. Two other findings — the blast radius being more than double what was documented, and the one-shot rollback not actually working — are recorded in § The pool-name bug's neighbours below.

Staging is two playbooks, both rerunnable:

phase	playbook	what it did
1	playbooks/esh-pve-nas-stage-zfs-root.yaml	carved the /boot LV out of swap, populated it, rsynced the root into nvme/ROOT/pve-1, wrote the copy's fstab
2	playbooks/esh-pve-nas-stage-bootloader.yaml	ZFS initramfs, grub.cfg, rollback entry, grubenv — without grub-install

Plan revised 2026-08-17 from "reinstall to a mirrored-NVMe ZFS root" to "split the boot chain from the root filesystem" — operator's proposal, and it is strictly better. The original reinstall plan is kept at the bottom as the fallback.

Why

PVE root lives on a USB Disk-on-Module — sdq, 7.3 GB, ID_BUS=usb, ID_VENDOR=NORELSYS — as a 6 GB ext4 root plus 768 MB swap and a 512 MB ESP.

A DOM is SLC/pSLC with a real controller, so the 284 GB written since boot is unremarkable and wear is not the driver. The actual problems:

1. It is on the USB bus. A bus reset or re-enumeration drops the root filesystem out from under a running hypervisor while its guests keep going.

2. 6 GB has no headroom — /usr alone is 3.7 GB.

3. Unmirrored, while 928 GB of mirrored NVMe sits 96% empty.

4. It has blocked patching for months. This is the operator-visible symptom and the real urgency: apt-get -s dist-upgrade shows 225 packages pending, 161 of them carrying deb12uN / Debian-Security bumps — including ssh 1:9.2p1-2+deb12u10. The host sits on pve-manager/8.4.11 while its sibling esh-pve is on 8.4.14, and it has 20 weeks of uptime because it cannot take a kernel.

   ⚠ Do not attempt the upgrade before the migration. The pending set includes proxmox-kernel-6.8.12-42-pve-signed (from -13) — a signed kernel plus initramfs is ~250 MB, and /boot is on root, which has 1.3 GB free. 225 packages unpacking (dpkg, perl, glibc-adjacent) into that headroom risks filling the disk mid-transaction and leaving a broken dpkg state on a hypervisor running five guests. Recovering a wedged dpkg on a full root is far worse than waiting for the reboot.

   If patching genuinely cannot wait, the escape hatch is to keep downloads off root — apt-get -o Dir::Cache::Archives=/nvme/tmp/apt-archives dist-upgrade — but the kernel still lands in /boot on root, so this reduces the risk rather than removing it. Migrating first is the shorter path to safety.

The design: boot on the DOM, root on ZFS

Boot and root do not have to live on the same device. Split them:

	device	contents	written when
boot	DOM sdq	ESP + /boot (ext4): GRUB, kernels, initramfs	only on kernel/GRUB updates
root	nvme pool	nvme/ROOT/pve-1 — everything else	constantly, on mirrored NVMe

GRUB reads the kernel and initrd from ext4 on the DOM, so GRUB never has to read ZFS — which matters, because the nvme pool has encryption, large_dnode and zstd_compress enabled and GRUB cannot read those. The initramfs then imports the pool and pivots to root=ZFS=nvme/ROOT/pve-1.

Why this beats the reinstall

• The nvme pool is not destroyed. The root dataset is created inside the existing pool. No guest migration, no zpool export/import of ssd/tank, no reinstall.
• Downtime is one reboot, not half a day.
• Rollback is a GRUB menu entry. The existing ext4 root stays on the DOM, untouched. If ZFS root fails to come up, pick the old entry and you are back in a minute. That is a far better rollback than "reinstall and restore."
• The #1 risk is actually retired. Once booted, root is on NVMe — a USB bus reset mid-run no longer takes the running system down. The DOM becomes read-mostly.
• Free upside: boot environments. zfs snapshot nvme/ROOT/pve-1@pre-upgrade before an apt run, roll back if it breaks.

What it does NOT fix

The DOM remains the only boot path. If it dies, the machine will not boot until the image is restored — though the ZFS root, with all config and guests, stays intact. Mitigation is a cloned fallback image (dd of sdq, ~7 GB, refreshed after kernel updates), kept off-box next to the config snapshot.

Preconditions — all already satisfied

Verified on the host 2026-08-17:

• UEFI firmware, grub-efi-amd64 2.06-13+pmx7 installed
• zfs-initramfs 2.2.8-pve1 is already installed, and the running initrd already carries 76 ZFS files — the pivot capability exists today, no new packages. (The pending upgrade would take ZFS to 2.2.10-pve1; 2.2.8 is fully capable of root-on-ZFS, so migrate on what is installed and upgrade after.)
• /boot is currently part of root (108 MB), so it must be split out onto its own ext4 filesystem on the DOM as part of this work
• root is only 4.3 GB to copy
• swap is 767 MB with 123 MB used against 125 GB of RAM — irrelevant; leave it on the DOM LV. Do not put swap on a zvol (deadlock risk)

⚠ cachefile is none and /etc/zfs/zpool.cache is 0 bytes — pools import by scan today (verified: zfs-import-scan.service active, zfs-import-cache.service inactive). For root-on-ZFS this must be deterministic, or the pool may not be imported early enough to find root.

⚠⚠ Set the cachefile on ALL THREE pools, not just nvme. An earlier draft of this runbook said zpool set cachefile=/etc/zfs/zpool.cache nvme, and that one-pool form is a trap. Populating a cachefile flips the host from import-by-scan to import-by-cache — so a cache containing only nvme means ssd and tank never get imported at boot. CT 103 esh-nas has twelve bind mounts spanning all three pools (/tank/media, /ssd/compose, /nvme/nvme-pvestore, …), so the NAS would come up with every export empty and both NFS clients would hang on hard mounts. The scoped-looking command is more dangerous than the broad one.

Done 2026-08-18 for nvme, ssd and tank; verified all three present in the resulting 11,976-byte cache via zdb -C -U /etc/zfs/zpool.cache. Phase 1's third guard step re-asserts this on every run.

⚠ Blast radius — unchanged, and still the gating constraint

CT 103 esh-nas (10.0.50.50) is the NAS, and it runs on this host. Two dependents mount it over hard NFS — they do not fail, they hang unkillably:

client	mounts
esh-docker-vm (10.0.50.45)	/mnt/books, /mnt/backup
esh-pve (10.0.250.35)	/mnt/pve/esh-nas, /mnt/pve/tank-vmbu

This is a known incident shape: the only remedy for esh-docker-vm's D-state is a host reboot, and /mnt/books was deliberately left hard because calibre's SQLite risks corruption under soft.

The reboot in this plan is brief, but it is still a reboot — quiesce the clients first.

Where the /boot LV came from — the VG was full

The original step 7 said /boot could "stay inside the DOM's existing LVM as its own small ext4 LV, or reuse the freed space once root moves off." Neither was available: VG pve had 4 MB free, and the 6 GB root is mounted ext4, which cannot shrink online — freeing space from it needs a rescue boot, which would have cost the "one reboot" property the whole design rests on.

The only space reclaimable live was the 768 MB swap LV (123 MB in use against 125 GB of RAM). Operator's call 2026-08-17: shrink swap rather than drop it. Final layout:

LV	size	role
pve-root	6.04 G	ext4 — untouched, the rollback root
pve-boot	512 M	ext4 — the new /boot (NEW)
pve-swap	256 M	swap (was 768 M)

Rejected alternatives: dropping swap outright (more kernel headroom, no OOM cushion); proxmox-boot-tool on the 512 MB ESP (PVE-native and no LVM surgery, but it reformats the ESP and downgrades rollback from "pick a menu entry" to "restore the DOM image"); rescue-boot to shrink root (keeps swap whole, costs a second reboot and an offline resize of the filesystem we are fleeing).

Sequence

Pre-flight (no downtime) — done 2026-08-18

1. dd the DOM to an off-box image. Crash-consistent, not clean — the root LV is live during the read, so a restore replays the ext4 journal. That is fine for its purpose (boot-chain insurance) and is what a snapshot backup does anyway. Not fixable with an LVM snapshot: the VG has no free extents.
2. Refresh the config snapshot (nh3-dev:~/backups/esh-pve-nas/).
3. zpool set cachefile=/etc/zfs/zpool.cache on nvme, ssd AND tank (see the precondition warning above — the one-pool form breaks the NAS).

Phase 1 — playbooks/esh-pve-nas-stage-zfs-root.yaml (live, no disruption) 4. zfs create -o mountpoint=none nvme/ROOT, then nvme/ROOT/pve-1 with canmount=noauto, compression=zstd, xattr=sa, acltype=posixacl. Create it with mountpoint=none and only set / at the very end — canmount=noauto alone is the documented guard, but never having a dataset that claims / while the ext4 root is live is the guard that cannot misfire. 5. Reclaim the swap LV into pve-boot, mkfs, populate from /boot. 6. Mount the dataset at /mnt/newroot and rsync the live root in. --one-file-system does the exclusion work: every path the old plan listed by hand (/proc /sys /dev /run /nvme /ssd /tank /var/log/journal /boot) is already a separate mount, so it is skipped structurally rather than by a list that can drift. 7. Write the copy's /etc/fstab: no root line (the initramfs mounts it), plus /dev/pve/boot /boot ext4, the ESP, and swap.

Phase 2 — playbooks/esh-pve-nas-stage-bootloader.yaml (live, no disruption) 8. Chroot into the copy with the boot LV and ESP mounted, then update-initramfs -u -k all + update-grub. 9. ⚠⚠ grub-mkconfig gets the ZFS root WRONG here, silently. Override it. See § The pool-name bug below — this is the single most dangerous thing found during staging. 10. GRUB_DEFAULT=saved, plus 40_custom carrying both boot paths as hand-authored entries with stable ids (pve-zfs-root, pve-ext4-rollback), with grubenv pinned to the rollback, not to ZFS (see § Cutover for why).

⚠ The pool-name bug — the near-miss worth reading

Left to itself, update-grub on this host produces:

linux /vmlinuz-6.8.12-13-pve root=ZFS=/ROOT/pve-1 ro quiet intel_iommu=on

The pool name is missing. It should be root=ZFS=nvme/ROOT/pve-1. That boots to an initramfs prompt — with CT 103 esh-nas down and both NFS clients hanging on hard mounts, at whatever hour the window happens to be.

It is not a typo, and it is not random. Debian's /etc/grub.d/10_linux builds the ZFS root as ${rpool}${bootfs}:

part	from	value here
rpool	grub-probe --device <dev> --target=fs_label	empty
bootfs	make_system_path_relative_to_its_root /	/ROOT/pve-1

grub-probe --target=fs / fails outright on this pool — grub-probe: error: unknown filesystem — because GRUB's own ZFS reader cannot open a pool with encryption, large_dnode and zstd_compress enabled. So rpool comes back empty and concatenates to nothing.

That is the same feature set that forced /boot to stay ext4 on the DOM. The design already accounted for GRUB being unable to read the pool; what was missed is that the same limitation also corrupts the kernel command line — and does it without an error, because grub-probe's failure is swallowed by 2>/dev/null || true.

The fix, in two layers:

1. /etc/default/grub.d/zfs-root.cfg sets GRUB_CMDLINE_LINUX="root=ZFS=nvme/ROOT/pve-1 boot=zfs". This is appended after the bogus value, and both the kernel and the zfs initramfs script take the last root= on the line — so every auto-generated entry becomes correct. A drop-in, not an edit to /etc/default/grub, so a grub package upgrade cannot revert it in a conffile merge.
2. 40_custom carries an explicit pve-zfs-root entry with a single clean root= and a stable id. That is what cutover's grub-reboot targets — the auto-generated ids are derived from pool member device paths (gnulinux-simple-/dev/nvme0n1p1_/dev/nvme1n1p1) and would shift if the mirror ever changed.

The general lesson, which is the transferable part: the phase-2 verify step originally grepped for root=ZFS=nvme/ROOT/pve-1 appearing somewhere in grub.cfg. Once the drop-in was added that grep passes — while pool-less entries sit in the menu untouched. The check that actually holds walks every linux line, takes the last root= on it, and asserts it against a known-good set. Assert the effective value, not the presence of a substring. 10. grub-install is deliberately NOT run during staging. The ESP stub still points at the old /boot inside the ext4 root, so the host's boot path stays byte-identical to what it has been for 140 days. Everything error-prone is built and verified in advance; the ESP rewrite is a two-second idempotent command held back to the window.

Cutover — the remaining work, § Cutover below.

The transferable lessons from this migration live in docs/pfi/ops-lessons-playbook.md — the ops sibling to the quantization playbook. Everything below is the ESH-specific narrative; the rules that would bite on any host are collected there.

⚠ The mount-propagation incident — the expensive lesson of 2026-08-18

What broke. The staging chroot was built with mount --rbind /dev and /sys and no --make-rslave. On a systemd host / has shared propagation, so those binds propagate in both directions. When the cutover tore the chroot down with umount -R, the unmounts propagated back into the live host and removed the real /sys/fs/cgroup, /dev/pts and /dev/shm.

With cgroup2 gone, systemd-logind could no longer create a session. The result is a host that:

• answers ping, accepts TCP, and completes SSH authentication
• keeps serving from daemons already resident in memory (pveproxy returned a clean HTTP 401 throughout)
• hangs on every new exec, including /sbin/reboot — so the reboot that was supposed to end the window never ran

Why it cost so much time: it is a near-perfect impostor of failing root-disk I/O. Both present as "host is up, daemons answer, nothing new can start." The session diagnosed it as the USB DOM dying and told the operator to walk to the machine. That was wrong, and the operator caught it: the DOM had been reliable for years and the wedge began immediately after a change.

The evidence that settles it, and was available the whole time — from dmesg, obtainable in the brief windows when exec did succeed:

line	says
[16.00] sd 56:0:0:0: [sdq] Attached SCSI removable disk	DOM enumerated cleanly, no errors
[12114881.98] systemd[1]: nvme-varlog-stage.mount: Deactivated	timestamp is 140 days — this is the ORIGINAL boot

That second line is the whole answer: the machine never rebooted. A down-detector loop had also never once reported the host down; that was read as a fast reboot rather than as no reboot at all.

Rules that follow:

1. Always mount --make-rslave after mount --rbind into a chroot. Phase 2 now does this and carries a guard that refuses to continue if any bind still reports shared propagation.
2. A reboot is not confirmed until the host is observed DOWN. Poll for disappearance, not just for reappearance. "Never went down" and "went down and came back fast" are indistinguishable if you only watch for the host to answer.
3. Before blaming hardware for a wedge that began right after a change, get dmesg and check the boot timestamp. Diagnose the change first; hardware is the explanation of last resort, not first.

Recovery took no console access. Windows where exec briefly succeeded were enough to land an idempotent remount of cgroup2 / devpts / shm, after which systemctl reset-failed returned the host to running. Total data loss: none. The root filesystem, the DOM and all three pools were never at risk — this was a mount-namespace fault, not a storage one.

The pool-name bug's neighbours — two more corrections

The blast radius was more than double what was documented. The runbook named two NFS dependents. ss -tn '( sport = :2049 )' inside CT 103 showed five:

client	mount	disposition
10.0.50.45 esh-docker-vm	/mnt/books, /mnt/backup — hard	quiesced
10.0.250.35 esh-pve	esh-nas, tank-vmbu — hard	quiesced
10.0.50.60 esh-vm-db	/mnt/backup — hard	left mounted deliberately
10.0.50.154 vm-esh-nas	—	is VM 104 on this host; stops with it
10.100.10.50 nh3-dev	/mnt/books — soft,ro	safe, errors instead of blocking

Ask the server who its clients are. A runbook's list of dependents is a snapshot that rots; ss on the NFS server is ground truth.

esh-vm-db was left mounted on purpose and came through read-write — a hard mount with no active user blocks and resumes, which is what hard is for. Its backup timers were ~19h out, and unmounting would have meant an unmount/remount cycle over the qemu guest agent on a host with no ssh access.

The one-shot rollback does not work, and the warning was right. grub-reboot printed "Detected GRUB environment block on lvm device — will remain the default boot entry until manually cleared." Confirmed empirically: after the successful ZFS boot, next_entry=pve-zfs-root was still set. GRUB can read grubenv on LVM but cannot write it, so boot_once degrades to a sticky default. There is no auto-fallback on this host. A failed boot must be corrected at the console.

The steady-state config therefore does not rely on it: saved_entry=pve-zfs-root with next_entry cleared. Restoring a real one-shot would mean relocating grubenv onto the ESP (vfat on a plain partition, which GRUB can write) — parked, not required.

Cutover

The only remaining work. Everything below the quiesce is minutes.

1. Quiesce the NFS clients (see § Blast radius). On esh-docker-vm (10.0.50.45) stop whatever holds /mnt/books and /mnt/backup and unmount them; on esh-pve (10.0.250.35) disable the esh-nas and tank-vmbu storages. Do this first and confirm it — a hard mount left live turns a brief reboot into an unkillable D-state needing a reboot of that host too.
2. Shut down the five guests.
3. Point the ESP at the new /boot and arm the one-shot:

   chroot /mnt/newroot grub-install --target=x86_64-efi \
       --efi-directory=/boot/efi --bootloader-id=proxmox
   chroot /mnt/newroot grub-reboot '<zfs entry id — phase 2's verify prints it>'
   

4. Set the dataset's final mountpoint, then reboot:

   zfs set mountpoint=/ nvme/ROOT/pve-1     # canmount stays noauto
   reboot
   

Why grub-reboot and not a new default. GRUB_DEFAULT=saved with grubenv pinned to the ext4 rollback means the ZFS entry is tried exactly once. If it fails, the next reboot returns to ext4 by itself — no console, no hands. That matters more here than on a normal host: a boot that hangs at an initramfs prompt takes CT 103 esh-nas down with it, and the NFS clients hang rather than fail. Only after the second successful ZFS boot (§ Verification) should the saved default move to the ZFS entry with grub-set-default.

Verification

• findmnt -no SOURCE,FSTYPE / → nvme/ROOT/pve-1 zfs
• df -h / shows hundreds of GB, not 5.9
• findmnt /boot → ext4 on the DOM; /boot/efi mounted
• all five guests running; CT 103 serving NFS (pct exec 103 -- exportfs -v)
• esh-docker-vm remounted and healthy; esh-pve storages green
• a second reboot to prove it was not a one-off
• only then: refresh the DOM image, since /boot has changed

Rollback

Instant and cheap at every stage: the ext4 root on the DOM is never modified, and its GRUB entry stays in the menu. Worst case is a boot to initramfs → reboot → pick the old entry. Keep the ext4 root for at least a few weeks of normal operation before reclaiming it.

Open decisions

• Second boot device? The split fixes runtime fragility but not boot-time single-point-of-failure. A cloned DOM/USB as a cold spare is the cheap answer.
• esh-filebot (CT 106) is an empty container — 80 GB quota, six passthrough mounts, nothing running since March. Retire rather than carry it.
• Reclaiming the old ext4 root once the ZFS root has proven itself.

---

Fallback plan: full reinstall to a mirrored-NVMe ZFS root

Only if the split above proves unworkable. Fresh PVE install to ZFS RAID1 across both NVMes — mirrored boot with proper ESPs under proxmox-boot-tool, no USB in the path at all.

Costs: the nvme pool must be destroyed, so its 32 GB of guest rootfs moves to ssd (1.42 T free) first; ssd and tank must be cleanly exported so the installer cannot touch them; guest configs restore from the snapshot plus the 8 PBS backups per guest. Half a day, and rollback after the install step is "reinstall and restore".

Note both NVMes are whole-disk ZFS members (partition 1 spans all 931.5 GiB, 1.7 MiB free), so adding an ESP to them without destroying the pool is impossible — which is what forces the reinstall in this variant, and what the split plan avoids entirely.