feat(esh-pve-nas): stage the PVE root migration off the USB DOM

Everything but the reboot. Two rerunnable elway playbooks; the host is
still running from the ext4 root and its boot path is byte-identical to
the last 140 days, because grub-install is deliberately held back to the
cutover window.

Phase 1 (esh-pve-nas-stage-zfs-root.yaml): carve a 512 MB /boot LV out
of the 768 MB swap LV, populate it, rsync the 4.3 GB ext4 root into
nvme/ROOT/pve-1, write the copy's fstab.

Phase 2 (esh-pve-nas-stage-bootloader.yaml): ZFS initramfs, grub.cfg,
explicit pve-zfs-root and pve-ext4-rollback entries with stable ids,
grubenv pinned to the rollback so cutover's grub-reboot is a one-shot.

Three landmines the plan did not predict, all caught by verify steps
asserting effective state rather than by reading the plan:

- The /boot LV had nowhere to live. VG pve had 4 MB free and mounted
  ext4 cannot shrink; freeing space from root needs a rescue boot, which
  costs the one-reboot property. Space came from swap (768M -> 256M).

- The runbook's `zpool set cachefile=... nvme` would have broken the
  NAS. Populating a cachefile flips the host from import-by-scan to
  import-by-cache, so a one-pool cache leaves ssd and tank unimported --
  and CT 103 esh-nas has twelve bind mounts spanning all three pools.
  Set on all three instead, verified in the resulting cache.

- update-grub silently emitted a pool-less root=ZFS=/ROOT/pve-1, which
  boots to an initramfs prompt. Debian's 10_linux builds ${rpool}${bootfs}
  and rpool comes from grub-probe --target=fs_label, which returns empty
  because GRUB's ZFS reader cannot open a pool with encryption,
  large_dnode and zstd_compress -- the same feature set that forced /boot
  to stay ext4. The probe failure is swallowed by `2>/dev/null || true`.
  Fixed with a /etc/default/grub.d drop-in plus explicit menu entries.

The transferable lesson: the original verify grepped for the correct
root= string appearing somewhere in grub.cfg, which passes while every
menu entry is still broken. Assert the effective value, not the presence
of a substring.
This commit is contained in:
vh
2026-08-17 22:11:14 -07:00
parent d3e1cc4a41
commit c4b2278e7d
6 changed files with 724 additions and 36 deletions
+27 -6
View File
@@ -17,11 +17,27 @@ Media services on this box sit at `10.0.50.56` (Plex) and `10.0.50.57` (Jellyfin
- **RAM:** 125.6 GB
- **Kernel:** `6.8.12-13-pve` (Proxmox 8.x)
- **Storage:** ⚠ **PVE root is on a USB Disk-on-Module** (`sdq`, 7.3 GB, `ID_BUS=usb`,
NORELSYS 1081) — 6 GB ext4 root + 768 MB swap + a 512 MB ESP that is the host's
**only** boot path. Not a wear problem (a DOM is SLC/pSLC), but it is on the USB
bus, has no headroom, and is unmirrored. Mitigated 2026-08-17 (90% → 76%: journald
capped, journal relocated to ZFS); migration planned in
[`docs/runbooks/esh-pve-nas-boot-migration.md`](../../docs/runbooks/esh-pve-nas-boot-migration.md).
NORELSYS 1081), whose 512 MB ESP is the host's **only** boot path. Not a wear
problem (a DOM is SLC/pSLC), but it is on the USB bus, has no headroom, and is
unmirrored. Mitigated 2026-08-17 (90% → 76%: journald capped, journal relocated
to ZFS). **Migration STAGED 2026-08-18 — awaiting only the reboot window**;
runbook [`docs/runbooks/esh-pve-nas-boot-migration.md`](../../docs/runbooks/esh-pve-nas-boot-migration.md).
DOM LVM layout after staging (was root 6.04 G + swap 768 M):
| LV | size | role |
|---|---|---|
| `pve-root` | 6.04 G | ext4 — the **live root today**, and the rollback after cutover |
| `pve-boot` | 512 M | ext4 — the new `/boot`, carved out of swap |
| `pve-swap` | 256 M | swap, shrunk to make room |
Post-cutover root is `nvme/ROOT/pve-1` on the mirrored NVMe pool. `/boot` stays
ext4 on the DOM on purpose: GRUB cannot read the `nvme` pool, which has
`encryption`, `large_dnode` and `zstd_compress` enabled.
- ⚠ **Never set the ZFS cachefile on one pool.** `zpool set cachefile=…` flips the
host from import-by-scan to import-by-cache; a cache holding only `nvme` leaves
`ssd` and `tank` unimported at boot, which empties every CT 103 export. Set it on
all three or none.
- **Pools:** `nvme` (2× 931 GB NVMe mirror — 32 G used, 867 G free, holds every guest
rootfs), `ssd` (4× 894 GB Intel SATA, 2 mirrors — 1.42 T free), `tank`
(12× 14.6 TB raidz2 ×2 — 40 T of 175 T). Plus NFS `/mnt/pve/tank-vmbu` for VM backups.
@@ -31,7 +47,12 @@ Media services on this box sit at `10.0.50.56` (Plex) and `10.0.50.57` (Jellyfin
wedges esh-docker-vm into unkillable D-state unless those clients are quiesced
first. Never reboot this host casually.
> **Root-fs pressure:** at 87% used on a 5.9 GB root partition, there's not much room for package upgrades or logs. Worth cleaning up or growing the root if this host is staying in production for a while.
> **Root-fs pressure:** 77% of a 5.9 GB root (1.3 GB free) after the 2026-08-17
> mitigation. Still not enough for the pending upgrade — **225 packages, 161
> carrying `deb12uN`/security bumps, including a ~250 MB signed kernel that lands
> in `/boot`.** ⚠ **Migrate first, patch after:** unpacking that into 1.3 GB of
> headroom risks wedging dpkg on a hypervisor running five guests. The host is on
> `pve-manager/8.4.11` vs esh-pve's 8.4.14 for exactly this reason.
## What it runs