feat(esh-pve-nas): stage the PVE root migration off the USB DOM
Everything but the reboot. Two rerunnable elway playbooks; the host is
still running from the ext4 root and its boot path is byte-identical to
the last 140 days, because grub-install is deliberately held back to the
cutover window.
Phase 1 (esh-pve-nas-stage-zfs-root.yaml): carve a 512 MB /boot LV out
of the 768 MB swap LV, populate it, rsync the 4.3 GB ext4 root into
nvme/ROOT/pve-1, write the copy's fstab.
Phase 2 (esh-pve-nas-stage-bootloader.yaml): ZFS initramfs, grub.cfg,
explicit pve-zfs-root and pve-ext4-rollback entries with stable ids,
grubenv pinned to the rollback so cutover's grub-reboot is a one-shot.
Three landmines the plan did not predict, all caught by verify steps
asserting effective state rather than by reading the plan:
- The /boot LV had nowhere to live. VG pve had 4 MB free and mounted
ext4 cannot shrink; freeing space from root needs a rescue boot, which
costs the one-reboot property. Space came from swap (768M -> 256M).
- The runbook's `zpool set cachefile=... nvme` would have broken the
NAS. Populating a cachefile flips the host from import-by-scan to
import-by-cache, so a one-pool cache leaves ssd and tank unimported --
and CT 103 esh-nas has twelve bind mounts spanning all three pools.
Set on all three instead, verified in the resulting cache.
- update-grub silently emitted a pool-less root=ZFS=/ROOT/pve-1, which
boots to an initramfs prompt. Debian's 10_linux builds ${rpool}${bootfs}
and rpool comes from grub-probe --target=fs_label, which returns empty
because GRUB's ZFS reader cannot open a pool with encryption,
large_dnode and zstd_compress -- the same feature set that forced /boot
to stay ext4. The probe failure is swallowed by `2>/dev/null || true`.
Fixed with a /etc/default/grub.d drop-in plus explicit menu entries.
The transferable lesson: the original verify grepped for the correct
root= string appearing somewhere in grub.cfg, which passes while every
menu entry is still broken. Assert the effective value, not the presence
of a substring.
This commit is contained in:
@@ -17,11 +17,27 @@ Media services on this box sit at `10.0.50.56` (Plex) and `10.0.50.57` (Jellyfin
|
||||
- **RAM:** 125.6 GB
|
||||
- **Kernel:** `6.8.12-13-pve` (Proxmox 8.x)
|
||||
- **Storage:** ⚠ **PVE root is on a USB Disk-on-Module** (`sdq`, 7.3 GB, `ID_BUS=usb`,
|
||||
NORELSYS 1081) — 6 GB ext4 root + 768 MB swap + a 512 MB ESP that is the host's
|
||||
**only** boot path. Not a wear problem (a DOM is SLC/pSLC), but it is on the USB
|
||||
bus, has no headroom, and is unmirrored. Mitigated 2026-08-17 (90% → 76%: journald
|
||||
capped, journal relocated to ZFS); migration planned in
|
||||
[`docs/runbooks/esh-pve-nas-boot-migration.md`](../../docs/runbooks/esh-pve-nas-boot-migration.md).
|
||||
NORELSYS 1081), whose 512 MB ESP is the host's **only** boot path. Not a wear
|
||||
problem (a DOM is SLC/pSLC), but it is on the USB bus, has no headroom, and is
|
||||
unmirrored. Mitigated 2026-08-17 (90% → 76%: journald capped, journal relocated
|
||||
to ZFS). **Migration STAGED 2026-08-18 — awaiting only the reboot window**;
|
||||
runbook [`docs/runbooks/esh-pve-nas-boot-migration.md`](../../docs/runbooks/esh-pve-nas-boot-migration.md).
|
||||
|
||||
DOM LVM layout after staging (was root 6.04 G + swap 768 M):
|
||||
|
||||
| LV | size | role |
|
||||
|---|---|---|
|
||||
| `pve-root` | 6.04 G | ext4 — the **live root today**, and the rollback after cutover |
|
||||
| `pve-boot` | 512 M | ext4 — the new `/boot`, carved out of swap |
|
||||
| `pve-swap` | 256 M | swap, shrunk to make room |
|
||||
|
||||
Post-cutover root is `nvme/ROOT/pve-1` on the mirrored NVMe pool. `/boot` stays
|
||||
ext4 on the DOM on purpose: GRUB cannot read the `nvme` pool, which has
|
||||
`encryption`, `large_dnode` and `zstd_compress` enabled.
|
||||
- ⚠ **Never set the ZFS cachefile on one pool.** `zpool set cachefile=…` flips the
|
||||
host from import-by-scan to import-by-cache; a cache holding only `nvme` leaves
|
||||
`ssd` and `tank` unimported at boot, which empties every CT 103 export. Set it on
|
||||
all three or none.
|
||||
- **Pools:** `nvme` (2× 931 GB NVMe mirror — 32 G used, 867 G free, holds every guest
|
||||
rootfs), `ssd` (4× 894 GB Intel SATA, 2 mirrors — 1.42 T free), `tank`
|
||||
(12× 14.6 TB raidz2 ×2 — 40 T of 175 T). Plus NFS `/mnt/pve/tank-vmbu` for VM backups.
|
||||
@@ -31,7 +47,12 @@ Media services on this box sit at `10.0.50.56` (Plex) and `10.0.50.57` (Jellyfin
|
||||
wedges esh-docker-vm into unkillable D-state unless those clients are quiesced
|
||||
first. Never reboot this host casually.
|
||||
|
||||
> **Root-fs pressure:** at 87% used on a 5.9 GB root partition, there's not much room for package upgrades or logs. Worth cleaning up or growing the root if this host is staying in production for a while.
|
||||
> **Root-fs pressure:** 77% of a 5.9 GB root (1.3 GB free) after the 2026-08-17
|
||||
> mitigation. Still not enough for the pending upgrade — **225 packages, 161
|
||||
> carrying `deb12uN`/security bumps, including a ~250 MB signed kernel that lands
|
||||
> in `/boot`.** ⚠ **Migrate first, patch after:** unpacking that into 1.3 GB of
|
||||
> headroom risks wedging dpkg on a hypervisor running five guests. The host is on
|
||||
> `pve-manager/8.4.11` vs esh-pve's 8.4.14 for exactly this reason.
|
||||
|
||||
## What it runs
|
||||
|
||||
|
||||
Reference in New Issue
Block a user