vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity

feat(esh-pve-nas): stage the PVE root migration off the USB DOM

Browse Source 
Everything but the reboot. Two rerunnable elway playbooks; the host is
still running from the ext4 root and its boot path is byte-identical to
the last 140 days, because grub-install is deliberately held back to the
cutover window.

Phase 1 (esh-pve-nas-stage-zfs-root.yaml): carve a 512 MB /boot LV out
of the 768 MB swap LV, populate it, rsync the 4.3 GB ext4 root into
nvme/ROOT/pve-1, write the copy's fstab.

Phase 2 (esh-pve-nas-stage-bootloader.yaml): ZFS initramfs, grub.cfg,
explicit pve-zfs-root and pve-ext4-rollback entries with stable ids,
grubenv pinned to the rollback so cutover's grub-reboot is a one-shot.

Three landmines the plan did not predict, all caught by verify steps
asserting effective state rather than by reading the plan:

- The /boot LV had nowhere to live. VG pve had 4 MB free and mounted
  ext4 cannot shrink; freeing space from root needs a rescue boot, which
  costs the one-reboot property. Space came from swap (768M -> 256M).

- The runbook's `zpool set cachefile=... nvme` would have broken the
  NAS. Populating a cachefile flips the host from import-by-scan to
  import-by-cache, so a one-pool cache leaves ssd and tank unimported --
  and CT 103 esh-nas has twelve bind mounts spanning all three pools.
  Set on all three instead, verified in the resulting cache.

- update-grub silently emitted a pool-less root=ZFS=/ROOT/pve-1, which
  boots to an initramfs prompt. Debian's 10_linux builds ${rpool}${bootfs}
  and rpool comes from grub-probe --target=fs_label, which returns empty
  because GRUB's ZFS reader cannot open a pool with encryption,
  large_dnode and zstd_compress -- the same feature set that forced /boot
  to stay ext4. The probe failure is swallowed by `2>/dev/null || true`.
  Fixed with a /etc/default/grub.d drop-in plus explicit menu entries.

The transferable lesson: the original verify grepped for the correct
root= string appearing somewhere in grub.cfg, which passes while every
menu entry is still broken. Assert the effective value, not the presence
of a substring.
This commit is contained in:
vh
2026-08-17 22:11:14 -07:00
parent d3e1cc4a41
commit c4b2278e7d
6 changed files with 724 additions and 36 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/runbooks/esh-pve-nas-boot-migration.md
+169 -27
View File
@@ -1,6 +1,16 @@
# esh-pve-nas — moving PVE root off the USB DOM
**Status: PLANNED, not executed.** Mitigation landed 2026-08-17 (root 90% → 76%).
**Status: STAGED — everything but the reboot is done.** Mitigation landed
2026-08-17 (root 90% → 76%); staging landed 2026-08-18. The host is still
running from the ext4 root on the DOM and its boot path is unchanged; what
remains is the cutover window (§ Cutover).
Staging is two playbooks, both rerunnable:
| phase | playbook | what it did |
|---|---|---|
| 1 | `playbooks/esh-pve-nas-stage-zfs-root.yaml` | carved the `/boot` LV out of swap, populated it, rsynced the root into `nvme/ROOT/pve-1`, wrote the copy's fstab |
| 2 | `playbooks/esh-pve-nas-stage-bootloader.yaml` | ZFS initramfs, grub.cfg, rollback entry, grubenv — **without** `grub-install` |
**Plan revised 2026-08-17** from "reinstall to a mirrored-NVMe ZFS root" to
**"split the boot chain from the root filesystem"** — operator's proposal, and it
@@ -91,9 +101,23 @@ Verified on the host 2026-08-17:
on the DOM LV. **Do not put swap on a zvol** (deadlock risk)
⚠ **`cachefile` is `none` and `/etc/zfs/zpool.cache` is 0 bytes** — pools import
by scan today. For root-on-ZFS this must be deterministic: set
`zpool set cachefile=/etc/zfs/zpool.cache nvme` and rebuild the initramfs, or the
pool may not be imported early enough to find root.
by scan today (verified: `zfs-import-scan.service` active,
`zfs-import-cache.service` inactive). For root-on-ZFS this must be
deterministic, or the pool may not be imported early enough to find root.
⚠⚠ **Set the cachefile on ALL THREE pools, not just `nvme`.** An earlier draft
of this runbook said `zpool set cachefile=/etc/zfs/zpool.cache nvme`, and that
one-pool form is a trap. Populating a cachefile flips the host from
import-by-scan to import-by-cache — so a cache containing only `nvme` means
**`ssd` and `tank` never get imported at boot.** CT 103 `esh-nas` has twelve
bind mounts spanning all three pools (`/tank/media`, `/ssd/compose`,
`/nvme/nvme-pvestore`, …), so the NAS would come up with every export empty and
both NFS clients would hang on `hard` mounts. The scoped-looking command is more
dangerous than the broad one.
Done 2026-08-18 for `nvme`, `ssd` and `tank`; verified all three present in the
resulting 11,976-byte cache via `zdb -C -U /etc/zfs/zpool.cache`. Phase 1's
third guard step re-asserts this on every run.
## ⚠ Blast radius — unchanged, and still the gating constraint
@@ -112,36 +136,154 @@ SQLite risks corruption under `soft`.
The reboot in this plan is brief, but it is still a reboot — quiesce the clients
first.
## Where the `/boot` LV came from — the VG was full
The original step 7 said `/boot` could "stay inside the DOM's existing LVM as
its own small ext4 LV, or reuse the freed space once root moves off." Neither
was available: **VG `pve` had 4 MB free**, and the 6 GB root is mounted ext4,
which cannot shrink online — freeing space from it needs a rescue boot, which
would have cost the "one reboot" property the whole design rests on.
The only space reclaimable live was the **768 MB swap LV** (123 MB in use
against 125 GB of RAM). Operator's call 2026-08-17: **shrink swap rather than
drop it.** Final layout:
| LV | size | role |
|---|---|---|
| `pve-root` | 6.04 G | ext4 — **untouched**, the rollback root |
| `pve-boot` | 512 M | ext4 — the new `/boot` (NEW) |
| `pve-swap` | 256 M | swap (was 768 M) |
Rejected alternatives: dropping swap outright (more kernel headroom, no OOM
cushion); `proxmox-boot-tool` on the 512 MB ESP (PVE-native and no LVM surgery,
but it reformats the ESP and downgrades rollback from "pick a menu entry" to
"restore the DOM image"); rescue-boot to shrink root (keeps swap whole, costs a
second reboot and an offline resize of the filesystem we are fleeing).
## Sequence
**Pre-flight (no downtime)**
1. `dd` the DOM to an image, off-box. This is the rollback for the boot chain.
**Pre-flight (no downtime)** — done 2026-08-18
1. `dd` the DOM to an off-box image. **Crash-consistent, not clean** — the root
LV is live during the read, so a restore replays the ext4 journal. That is
fine for its purpose (boot-chain insurance) and is what a snapshot backup
does anyway. Not fixable with an LVM snapshot: the VG has no free extents.
2. Refresh the config snapshot (`nh3-dev:~/backups/esh-pve-nas/`).
3. `zpool set cachefile=/etc/zfs/zpool.cache nvme`.
3. `zpool set cachefile=/etc/zfs/zpool.cache` on **`nvme`, `ssd` AND `tank`**
(see the precondition warning above — the one-pool form breaks the NAS).
**Build the ZFS root (live, no disruption)**
4. `zfs create -o mountpoint=none nvme/ROOT`
`zfs create -o mountpoint=/ -o canmount=noauto nvme/ROOT/pve-1`
`canmount=noauto` is essential — otherwise ZFS tries to mount it over the
running root.
5. Mount it somewhere temporary and `rsync -aHAX --numeric-ids` the live root
into it, excluding `/proc /sys /dev /run /mnt /nvme /ssd /tank /var/log/journal`.
6. In the copy: edit `/etc/fstab` to drop the root line, keep the ESP and swap
lines, and add the new `/boot` entry.
**Phase 1 — `playbooks/esh-pve-nas-stage-zfs-root.yaml`** (live, no disruption)
4. `zfs create -o mountpoint=none nvme/ROOT`, then `nvme/ROOT/pve-1` with
`canmount=noauto`, `compression=zstd`, `xattr=sa`, `acltype=posixacl`.
Create it with `mountpoint=none` and only set `/` at the very end —
`canmount=noauto` alone is the documented guard, but never having a dataset
that claims `/` while the ext4 root is live is the guard that cannot misfire.
5. Reclaim the swap LV into `pve-boot`, mkfs, populate from `/boot`.
6. Mount the dataset at `/mnt/newroot` and rsync the live root in.
`--one-file-system` does the exclusion work: every path the old plan listed
by hand (`/proc /sys /dev /run /nvme /ssd /tank /var/log/journal /boot`) is
already a separate mount, so it is skipped structurally rather than by a
list that can drift.
7. Write the copy's `/etc/fstab`: no root line (the initramfs mounts it), plus
`/dev/pve/boot /boot ext4`, the ESP, and swap.
**Split `/boot` onto the DOM**
7. Shrink/repartition is not needed — `/boot` can stay inside the DOM's existing
LVM as its own small ext4 LV, or reuse the freed space once root moves off.
Copy `/boot` into it and mount at `/boot`.
**Phase 2 — `playbooks/esh-pve-nas-stage-bootloader.yaml`** (live, no disruption)
8. Chroot into the copy with the boot LV and ESP mounted, then
`update-initramfs -u -k all` + `update-grub`.
9. ⚠⚠ **`grub-mkconfig` gets the ZFS root WRONG here, silently. Override it.**
See § The pool-name bug below — this is the single most dangerous thing
found during staging.
10. `GRUB_DEFAULT=saved`, plus `40_custom` carrying **both** boot paths as
hand-authored entries with stable ids (`pve-zfs-root`, `pve-ext4-rollback`),
with grubenv pinned to the **rollback**, not to ZFS (see § Cutover for why).
**Bootloader**
8. `GRUB_CMDLINE_LINUX="root=ZFS=nvme/ROOT/pve-1"`, `update-initramfs -u -k all`,
`update-grub`. **Keep the existing ext4 entry in the menu** and leave
`GRUB_TIMEOUT=5` so it is selectable.
## ⚠ The pool-name bug — the near-miss worth reading
**Cut over**
9. Quiesce the NFS clients (see blast radius), shut down guests, reboot.
10. Verify (below). If it fails, reboot and choose the ext4 entry.
Left to itself, `update-grub` on this host produces:
```
linux /vmlinuz-6.8.12-13-pve root=ZFS=/ROOT/pve-1 ro quiet intel_iommu=on
```
**The pool name is missing.** It should be `root=ZFS=nvme/ROOT/pve-1`. That
boots to an initramfs prompt — with CT 103 `esh-nas` down and both NFS clients
hanging on `hard` mounts, at whatever hour the window happens to be.
It is not a typo, and it is not random. Debian's `/etc/grub.d/10_linux` builds
the ZFS root as `${rpool}${bootfs}`:
| part | from | value here |
|---|---|---|
| `rpool` | `grub-probe --device <dev> --target=fs_label` | **empty** |
| `bootfs` | `make_system_path_relative_to_its_root /` | `/ROOT/pve-1` |
`grub-probe --target=fs /` fails outright on this pool — `grub-probe: error:
unknown filesystem` — because **GRUB's own ZFS reader cannot open a pool with
`encryption`, `large_dnode` and `zstd_compress` enabled.** So `rpool` comes back
empty and concatenates to nothing.
That is the *same* feature set that forced `/boot` to stay ext4 on the DOM. The
design already accounted for GRUB being unable to read the pool; what was missed
is that the same limitation also corrupts the kernel command line — and does it
**without an error**, because `grub-probe`'s failure is swallowed by
`2>/dev/null || true`.
**The fix, in two layers:**
1. `/etc/default/grub.d/zfs-root.cfg` sets
`GRUB_CMDLINE_LINUX="root=ZFS=nvme/ROOT/pve-1 boot=zfs"`. This is appended
*after* the bogus value, and both the kernel and the zfs initramfs script
take the **last** `root=` on the line — so every auto-generated entry becomes
correct. A drop-in, not an edit to `/etc/default/grub`, so a grub package
upgrade cannot revert it in a conffile merge.
2. `40_custom` carries an explicit `pve-zfs-root` entry with a single clean
`root=` and a stable id. That is what cutover's `grub-reboot` targets — the
auto-generated ids are derived from pool member device paths
(`gnulinux-simple-/dev/nvme0n1p1_/dev/nvme1n1p1`) and would shift if the
mirror ever changed.
**The general lesson, which is the transferable part:** the phase-2 verify step
originally grepped for `root=ZFS=nvme/ROOT/pve-1` *appearing somewhere* in
grub.cfg. Once the drop-in was added that grep passes — while pool-less entries
sit in the menu untouched. The check that actually holds walks every `linux`
line, takes the **last** `root=` on it, and asserts it against a known-good set.
Assert the effective value, not the presence of a substring.
10. **`grub-install` is deliberately NOT run during staging.** The ESP stub
still points at the old `/boot` inside the ext4 root, so the host's boot
path stays byte-identical to what it has been for 140 days. Everything
error-prone is built and verified in advance; the ESP rewrite is a
two-second idempotent command held back to the window.
**Cutover** — the remaining work, § Cutover below.
## Cutover
The only remaining work. Everything below the quiesce is minutes.
1. **Quiesce the NFS clients** (see § Blast radius). On **esh-docker-vm**
(10.0.50.45) stop whatever holds `/mnt/books` and `/mnt/backup` and unmount
them; on **esh-pve** (10.0.250.35) disable the `esh-nas` and `tank-vmbu`
storages. Do this first and confirm it — a `hard` mount left live turns a
brief reboot into an unkillable D-state needing a reboot of *that* host too.
2. Shut down the five guests.
3. Point the ESP at the new `/boot` and arm the one-shot:
```
chroot /mnt/newroot grub-install --target=x86_64-efi \
--efi-directory=/boot/efi --bootloader-id=proxmox
chroot /mnt/newroot grub-reboot '<zfs entry id — phase 2's verify prints it>'
```
4. Set the dataset's final mountpoint, then reboot:
```
zfs set mountpoint=/ nvme/ROOT/pve-1 # canmount stays noauto
reboot
```
**Why `grub-reboot` and not a new default.** `GRUB_DEFAULT=saved` with grubenv
pinned to the ext4 rollback means the ZFS entry is tried **exactly once**. If it
fails, the next reboot returns to ext4 *by itself* — no console, no hands. That
matters more here than on a normal host: a boot that hangs at an initramfs
prompt takes CT 103 `esh-nas` down with it, and the NFS clients hang rather than
fail. Only after the second successful ZFS boot (§ Verification) should the
saved default move to the ZFS entry with `grub-set-default`.
## Verification