feat(esh-pve-nas): stage the PVE root migration off the USB DOM

Everything but the reboot. Two rerunnable elway playbooks; the host is
still running from the ext4 root and its boot path is byte-identical to
the last 140 days, because grub-install is deliberately held back to the
cutover window.

Phase 1 (esh-pve-nas-stage-zfs-root.yaml): carve a 512 MB /boot LV out
of the 768 MB swap LV, populate it, rsync the 4.3 GB ext4 root into
nvme/ROOT/pve-1, write the copy's fstab.

Phase 2 (esh-pve-nas-stage-bootloader.yaml): ZFS initramfs, grub.cfg,
explicit pve-zfs-root and pve-ext4-rollback entries with stable ids,
grubenv pinned to the rollback so cutover's grub-reboot is a one-shot.

Three landmines the plan did not predict, all caught by verify steps
asserting effective state rather than by reading the plan:

- The /boot LV had nowhere to live. VG pve had 4 MB free and mounted
  ext4 cannot shrink; freeing space from root needs a rescue boot, which
  costs the one-reboot property. Space came from swap (768M -> 256M).

- The runbook's `zpool set cachefile=... nvme` would have broken the
  NAS. Populating a cachefile flips the host from import-by-scan to
  import-by-cache, so a one-pool cache leaves ssd and tank unimported --
  and CT 103 esh-nas has twelve bind mounts spanning all three pools.
  Set on all three instead, verified in the resulting cache.

- update-grub silently emitted a pool-less root=ZFS=/ROOT/pve-1, which
  boots to an initramfs prompt. Debian's 10_linux builds ${rpool}${bootfs}
  and rpool comes from grub-probe --target=fs_label, which returns empty
  because GRUB's ZFS reader cannot open a pool with encryption,
  large_dnode and zstd_compress -- the same feature set that forced /boot
  to stay ext4. The probe failure is swallowed by `2>/dev/null || true`.
  Fixed with a /etc/default/grub.d drop-in plus explicit menu entries.

The transferable lesson: the original verify grepped for the correct
root= string appearing somewhere in grub.cfg, which passes while every
menu entry is still broken. Assert the effective value, not the presence
of a substring.

persistent-memory.d/2026-08-17-esh-pve-nas-dom.md

@@ -118,7 +118,77 @@ Recorded in `servers/esh-pve-nas/README.md` as a never-reboot-casually warning.
   migrate.
 - Both ESH hypervisors have **20 weeks uptime** and differing PVE patch levels.
 
-Runbook: `docs/runbooks/esh-pve-nas-boot-migration.md`. Config snapshot off-box at
+## Staging executed 2026-08-18 — everything but the reboot
+
+Two rerunnable elway playbooks, 0 failed steps, 17/17 verify green:
+`playbooks/esh-pve-nas-stage-zfs-root.yaml` (LV surgery, `/boot` populate,
+4.3 GB root rsync in 228 s, fstab) and `playbooks/esh-pve-nas-stage-bootloader.yaml`
+(ZFS initramfs, grub.cfg, both menu entries, grubenv).
+
+**`grub-install` is deliberately NOT run.** The ESP stub still points at the old
+`/boot` inside the ext4 root, so the host's boot path is byte-identical to the
+last 140 days and an unplanned reboot mid-staging is a non-event. Cutover is
+`grub-install` + `grub-reboot pve-zfs-root` + `zfs set mountpoint=/` + reboot.
+
+Final DOM layout: `pve-root` 6.04 G (untouched, the rollback) + `pve-boot` 512 M
+(new) + `pve-swap` 256 M (was 768 M).
+
+### The three landmines staging found
+
+1. **The `/boot` LV had nowhere to live.** VG `pve` had **4 MB free**, and
+   mounted ext4 cannot shrink — freeing space from root needs a rescue boot,
+   which costs the "one reboot" property the design rests on. Only live source
+   was the swap LV. Operator chose shrink-to-256M over drop-entirely.
+2. **The one-pool cachefile would have broken the NAS.** `zpool set
+   cachefile=… nvme` looks scoped and safe; it is the opposite. Populating a
+   cachefile flips the host from `zfs-import-scan` to `zfs-import-cache`
+   (verified: scan active, cache inactive beforehand), so a cache holding only
+   `nvme` leaves `ssd` and `tank` unimported at boot — and CT 103 has twelve
+   bind mounts spanning all three pools. Every export would come up empty and
+   both `hard` NFS clients would hang.
+3. **`update-grub` silently emitted a pool-less `root=ZFS=/ROOT/pve-1`.**
+   Debian's `10_linux` builds `${rpool}${bootfs}`; `rpool` comes from
+   `grub-probe --target=fs_label`, which returns empty because GRUB's ZFS reader
+   cannot open a pool with `encryption`/`large_dnode`/`zstd_compress` — and the
+   probe failure is swallowed by `2>/dev/null || true`. The same feature set
+   that forced `/boot` to stay ext4 also corrupts the kernel command line, which
+   the design did not anticipate. Fixed with a `/etc/default/grub.d/zfs-root.cfg`
+   drop-in (last `root=` wins) plus explicit `pve-zfs-root` and
+   `pve-ext4-rollback` entries carrying stable ids — the auto-generated ids are
+   derived from pool member device paths and would shift if the mirror changed.
+
+**The transferable lesson from (3):** the original verify grepped for
+`root=ZFS=nvme/ROOT/pve-1` *appearing somewhere* in grub.cfg. Once the drop-in
+was added that grep passes — while pool-less entries sit in the menu untouched.
+The check that holds walks every `linux` line, takes the **last** `root=`, and
+asserts it against a known-good set. **Assert the effective value, not the
+presence of a substring.**
+
+### One-shot boot, not a new default
+
+`GRUB_DEFAULT=saved` with grubenv pinned to `pve-ext4-rollback`, and cutover uses
+`grub-reboot pve-zfs-root` so ZFS is tried **exactly once**. A failed ZFS boot
+returns to ext4 by itself on the next reboot — no console, no hands. That matters
+more here than on a normal host: a hang at an initramfs prompt takes CT 103 down
+and the NFS clients hang rather than fail. Only after a second clean ZFS boot
+should the saved default move.
+
+### Off-box artifacts (`nh3-dev:~/backups/esh-pve-nas/`)
+
+- `dom-sdq-20260818.img.zst` — full DOM image, 7,837,450,240 B raw / 2.38 GiB
+  compressed, zstd XXH64 verified. ⚠ **Crash-consistent, not clean** — the root
+  LV was live during the read, so a restore replays the ext4 journal. Not
+  fixable with an LVM snapshot: the VG has no free extents.
+- `bootchain-20260818.tar.gz` — clean, consistent tar of `/boot` + ESP (88 MB,
+  644 entries, full proxmox shim/grub EFI chain). This is the higher-quality
+  boot-chain artifact; the dd image is the belt-and-braces full-device restore.
+- `pve-config-snapshot-20260818T051*.tar.gz` — 147 entries incl. the new
+  grub.cfg, fstab, LVM/ZFS/blkid state.
+  ⚠ Building this the first time produced a **corrupt archive**: `pvs; vgs; lvs >
+  file` redirects only the last command, so `pvs`/`vgs` output leaked into the
+  tar stream on stdout. Group with `{ …; } > file`.
+
+Runbook: `docs/runbooks/esh-pve-nas-boot-migration.md`. Earlier config snapshot at
 `nh3-dev:~/backups/esh-pve-nas/pve-config-snapshot-20260818T043027Z.tar.gz` (0600,
 sha256 `dc312793d027dc43…`) — `/etc/pve`, network, fstab, apt, authorized_keys plus
 captured `zpool`/`zfs`/`disk-by-id`/`lsblk`-with-serials/`pvesm`/`dpkg` state and