esh-pfi-infrastructure/docs/runbooks/esh-pve-nas-boot-migration.md

# esh-pve-nas — moving PVE root off the USB DOM

**Status: DONE — cut over 2026-08-18.** Root is `nvme/ROOT/pve-1` on the mirrored
NVMe; `/boot` is ext4 on the DOM; the DOM is out of the runtime I/O path. All five
guests healthy, all three pools ONLINE, `systemctl is-system-running` = `running`.
The ext4 root (`pve-root`) is intact, unmounted, and still carries its own kernel
and initrd as the rollback.

Post-cutover boot config: `saved_entry=pve-zfs-root`, no `next_entry`. If grubenv
were ever unreadable GRUB falls through to menu entry 0, which the
`/etc/default/grub.d/zfs-root.cfg` drop-in also points at `root=ZFS=nvme/ROOT/pve-1`
— so every path boots ZFS.

⚠ **The window cost an unplanned outage, caused by a bug in this runbook's own
tooling, not by the migration.** Read § The mount-propagation incident before
running anything like this again. Two other findings — the blast radius being
more than double what was documented, and the one-shot rollback not actually
working — are recorded in § The pool-name bug's neighbours below.

Staging is two playbooks, both rerunnable:

| phase | playbook | what it did |
|---|---|---|
| 1 | `playbooks/esh-pve-nas-stage-zfs-root.yaml` | carved the `/boot` LV out of swap, populated it, rsynced the root into `nvme/ROOT/pve-1`, wrote the copy's fstab |
| 2 | `playbooks/esh-pve-nas-stage-bootloader.yaml` | ZFS initramfs, grub.cfg, rollback entry, grubenv — **without** `grub-install` |

**Plan revised 2026-08-17** from "reinstall to a mirrored-NVMe ZFS root" to
**"split the boot chain from the root filesystem"** — operator's proposal, and it
is strictly better. The original reinstall plan is kept at the bottom as the
fallback.

## Why

PVE root lives on a **USB Disk-on-Module** — `sdq`, 7.3 GB, `ID_BUS=usb`,
`ID_VENDOR=NORELSYS` — as a 6 GB ext4 root plus 768 MB swap and a 512 MB ESP.

A DOM is SLC/pSLC with a real controller, so the 284 GB written since boot is
unremarkable and **wear is not the driver**. The actual problems:

1. **It is on the USB bus.** A bus reset or re-enumeration drops the *root
   filesystem* out from under a running hypervisor while its guests keep going.
2. **6 GB has no headroom** — `/usr` alone is 3.7 GB.
3. **Unmirrored**, while 928 GB of mirrored NVMe sits 96% empty.
4. **It has blocked patching for months.** This is the operator-visible symptom
   and the real urgency: `apt-get -s dist-upgrade` shows **225 packages pending,
   161 of them carrying `deb12uN` / Debian-Security bumps** — including `ssh
   1:9.2p1-2+deb12u10`. The host sits on `pve-manager/8.4.11` while its sibling
   esh-pve is on 8.4.14, and it has 20 weeks of uptime because it cannot take a
   kernel.

   ⚠ **Do not attempt the upgrade before the migration.** The pending set
   includes `proxmox-kernel-6.8.12-42-pve-signed` (from -13) — a signed kernel
   plus initramfs is ~250 MB, and **`/boot` is on root**, which has 1.3 GB free.
   225 packages unpacking (dpkg, perl, glibc-adjacent) into that headroom risks
   filling the disk mid-transaction and leaving a broken dpkg state on a
   hypervisor running five guests. Recovering a wedged dpkg on a full root is
   far worse than waiting for the reboot.

   If patching genuinely cannot wait, the escape hatch is to keep downloads off
   root — `apt-get -o Dir::Cache::Archives=/nvme/tmp/apt-archives dist-upgrade`
   — but the kernel still lands in `/boot` on root, so this reduces the risk
   rather than removing it. Migrating first is the shorter path to safety.

## The design: boot on the DOM, root on ZFS

Boot and root do not have to live on the same device. Split them:

| | device | contents | written when |
|---|---|---|---|
| **boot** | DOM `sdq` | ESP + `/boot` (ext4): GRUB, kernels, initramfs | **only on kernel/GRUB updates** |
| **root** | `nvme` pool | `nvme/ROOT/pve-1` — everything else | constantly, on mirrored NVMe |

GRUB reads the kernel and initrd from **ext4 on the DOM**, so GRUB never has to
read ZFS — which matters, because the `nvme` pool has `encryption`,
`large_dnode` and `zstd_compress` enabled and GRUB cannot read those. The
initramfs then imports the pool and pivots to `root=ZFS=nvme/ROOT/pve-1`.

### Why this beats the reinstall

- **The `nvme` pool is not destroyed.** The root dataset is created *inside* the
  existing pool. No guest migration, no `zpool export/import` of `ssd`/`tank`,
  no reinstall.
- **Downtime is one reboot**, not half a day.
- **Rollback is a GRUB menu entry.** The existing ext4 root stays on the DOM,
  untouched. If ZFS root fails to come up, pick the old entry and you are back in
  a minute. That is a far better rollback than "reinstall and restore."
- **The #1 risk is actually retired.** Once booted, root is on NVMe — a USB bus
  reset mid-run no longer takes the running system down. The DOM becomes
  read-mostly.
- **Free upside: boot environments.** `zfs snapshot nvme/ROOT/pve-1@pre-upgrade`
  before an apt run, roll back if it breaks.

### What it does NOT fix

The DOM remains the **only boot path**. If it dies, the machine will not boot
until the image is restored — though the ZFS root, with all config and guests,
stays intact. Mitigation is a **cloned fallback image** (`dd` of `sdq`, ~7 GB,
refreshed after kernel updates), kept off-box next to the config snapshot.

## Preconditions — all already satisfied

Verified on the host 2026-08-17:

- **UEFI** firmware, `grub-efi-amd64 2.06-13+pmx7` installed
- **`zfs-initramfs 2.2.8-pve1` is already installed**, and the running initrd
  already carries **76 ZFS files** — the pivot capability exists today, no new
  packages. (The pending upgrade would take ZFS to 2.2.10-pve1; 2.2.8 is fully
  capable of root-on-ZFS, so migrate on what is installed and upgrade after.)
- `/boot` is currently *part of* root (108 MB), so it must be split out onto its
  own ext4 filesystem on the DOM as part of this work
- root is only **4.3 GB** to copy
- swap is 767 MB with 123 MB used against 125 GB of RAM — irrelevant; leave it
  on the DOM LV. **Do not put swap on a zvol** (deadlock risk)

⚠ **`cachefile` is `none` and `/etc/zfs/zpool.cache` is 0 bytes** — pools import
by scan today (verified: `zfs-import-scan.service` active,
`zfs-import-cache.service` inactive). For root-on-ZFS this must be
deterministic, or the pool may not be imported early enough to find root.

⚠⚠ **Set the cachefile on ALL THREE pools, not just `nvme`.** An earlier draft
of this runbook said `zpool set cachefile=/etc/zfs/zpool.cache nvme`, and that
one-pool form is a trap. Populating a cachefile flips the host from
import-by-scan to import-by-cache — so a cache containing only `nvme` means
**`ssd` and `tank` never get imported at boot.** CT 103 `esh-nas` has twelve
bind mounts spanning all three pools (`/tank/media`, `/ssd/compose`,
`/nvme/nvme-pvestore`, …), so the NAS would come up with every export empty and
both NFS clients would hang on `hard` mounts. The scoped-looking command is more
dangerous than the broad one.

Done 2026-08-18 for `nvme`, `ssd` and `tank`; verified all three present in the
resulting 11,976-byte cache via `zdb -C -U /etc/zfs/zpool.cache`. Phase 1's
third guard step re-asserts this on every run.

## ⚠ Blast radius — unchanged, and still the gating constraint

**CT 103 `esh-nas` (10.0.50.50) is the NAS, and it runs on this host.** Two
dependents mount it over **`hard`** NFS — they do not fail, they hang unkillably:

| client | mounts |
|---|---|
| **esh-docker-vm** (10.0.50.45) | `/mnt/books`, `/mnt/backup` |
| **esh-pve** (10.0.250.35) | `/mnt/pve/esh-nas`, `/mnt/pve/tank-vmbu` |

This is a known incident shape: the only remedy for esh-docker-vm's D-state is a
host reboot, and `/mnt/books` was *deliberately* left `hard` because calibre's
SQLite risks corruption under `soft`.

The reboot in this plan is brief, but it is still a reboot — quiesce the clients
first.

## Where the `/boot` LV came from — the VG was full

The original step 7 said `/boot` could "stay inside the DOM's existing LVM as
its own small ext4 LV, or reuse the freed space once root moves off." Neither
was available: **VG `pve` had 4 MB free**, and the 6 GB root is mounted ext4,
which cannot shrink online — freeing space from it needs a rescue boot, which
would have cost the "one reboot" property the whole design rests on.

The only space reclaimable live was the **768 MB swap LV** (123 MB in use
against 125 GB of RAM). Operator's call 2026-08-17: **shrink swap rather than
drop it.** Final layout:

| LV | size | role |
|---|---|---|
| `pve-root` | 6.04 G | ext4 — **untouched**, the rollback root |
| `pve-boot` | 512 M | ext4 — the new `/boot` (NEW) |
| `pve-swap` | 256 M | swap (was 768 M) |

Rejected alternatives: dropping swap outright (more kernel headroom, no OOM
cushion); `proxmox-boot-tool` on the 512 MB ESP (PVE-native and no LVM surgery,
but it reformats the ESP and downgrades rollback from "pick a menu entry" to
"restore the DOM image"); rescue-boot to shrink root (keeps swap whole, costs a
second reboot and an offline resize of the filesystem we are fleeing).

## Sequence

**Pre-flight (no downtime)** — done 2026-08-18
1. `dd` the DOM to an off-box image. **Crash-consistent, not clean** — the root
   LV is live during the read, so a restore replays the ext4 journal. That is
   fine for its purpose (boot-chain insurance) and is what a snapshot backup
   does anyway. Not fixable with an LVM snapshot: the VG has no free extents.
2. Refresh the config snapshot (`nh3-dev:~/backups/esh-pve-nas/`).
3. `zpool set cachefile=/etc/zfs/zpool.cache` on **`nvme`, `ssd` AND `tank`**
   (see the precondition warning above — the one-pool form breaks the NAS).

**Phase 1 — `playbooks/esh-pve-nas-stage-zfs-root.yaml`** (live, no disruption)
4. `zfs create -o mountpoint=none nvme/ROOT`, then `nvme/ROOT/pve-1` with
   `canmount=noauto`, `compression=zstd`, `xattr=sa`, `acltype=posixacl`.
   Create it with `mountpoint=none` and only set `/` at the very end —
   `canmount=noauto` alone is the documented guard, but never having a dataset
   that claims `/` while the ext4 root is live is the guard that cannot misfire.
5. Reclaim the swap LV into `pve-boot`, mkfs, populate from `/boot`.
6. Mount the dataset at `/mnt/newroot` and rsync the live root in.
   `--one-file-system` does the exclusion work: every path the old plan listed
   by hand (`/proc /sys /dev /run /nvme /ssd /tank /var/log/journal /boot`) is
   already a separate mount, so it is skipped structurally rather than by a
   list that can drift.
7. Write the copy's `/etc/fstab`: no root line (the initramfs mounts it), plus
   `/dev/pve/boot /boot ext4`, the ESP, and swap.

**Phase 2 — `playbooks/esh-pve-nas-stage-bootloader.yaml`** (live, no disruption)
8. Chroot into the copy with the boot LV and ESP mounted, then
   `update-initramfs -u -k all` + `update-grub`.
9. ⚠⚠ **`grub-mkconfig` gets the ZFS root WRONG here, silently. Override it.**
   See § The pool-name bug below — this is the single most dangerous thing
   found during staging.
10. `GRUB_DEFAULT=saved`, plus `40_custom` carrying **both** boot paths as
    hand-authored entries with stable ids (`pve-zfs-root`, `pve-ext4-rollback`),
    with grubenv pinned to the **rollback**, not to ZFS (see § Cutover for why).

## ⚠ The pool-name bug — the near-miss worth reading

Left to itself, `update-grub` on this host produces:

```
linux /vmlinuz-6.8.12-13-pve root=ZFS=/ROOT/pve-1 ro quiet intel_iommu=on
```

**The pool name is missing.** It should be `root=ZFS=nvme/ROOT/pve-1`. That
boots to an initramfs prompt — with CT 103 `esh-nas` down and both NFS clients
hanging on `hard` mounts, at whatever hour the window happens to be.

It is not a typo, and it is not random. Debian's `/etc/grub.d/10_linux` builds
the ZFS root as `${rpool}${bootfs}`:

| part | from | value here |
|---|---|---|
| `rpool` | `grub-probe --device <dev> --target=fs_label` | **empty** |
| `bootfs` | `make_system_path_relative_to_its_root /` | `/ROOT/pve-1` |

`grub-probe --target=fs /` fails outright on this pool — `grub-probe: error:
unknown filesystem` — because **GRUB's own ZFS reader cannot open a pool with
`encryption`, `large_dnode` and `zstd_compress` enabled.** So `rpool` comes back
empty and concatenates to nothing.

That is the *same* feature set that forced `/boot` to stay ext4 on the DOM. The
design already accounted for GRUB being unable to read the pool; what was missed
is that the same limitation also corrupts the kernel command line — and does it
**without an error**, because `grub-probe`'s failure is swallowed by
`2>/dev/null || true`.

**The fix, in two layers:**

1. `/etc/default/grub.d/zfs-root.cfg` sets
   `GRUB_CMDLINE_LINUX="root=ZFS=nvme/ROOT/pve-1 boot=zfs"`. This is appended
   *after* the bogus value, and both the kernel and the zfs initramfs script
   take the **last** `root=` on the line — so every auto-generated entry becomes
   correct. A drop-in, not an edit to `/etc/default/grub`, so a grub package
   upgrade cannot revert it in a conffile merge.
2. `40_custom` carries an explicit `pve-zfs-root` entry with a single clean
   `root=` and a stable id. That is what cutover's `grub-reboot` targets — the
   auto-generated ids are derived from pool member device paths
   (`gnulinux-simple-/dev/nvme0n1p1_/dev/nvme1n1p1`) and would shift if the
   mirror ever changed.

**The general lesson, which is the transferable part:** the phase-2 verify step
originally grepped for `root=ZFS=nvme/ROOT/pve-1` *appearing somewhere* in
grub.cfg. Once the drop-in was added that grep passes — while pool-less entries
sit in the menu untouched. The check that actually holds walks every `linux`
line, takes the **last** `root=` on it, and asserts it against a known-good set.
Assert the effective value, not the presence of a substring.
10. **`grub-install` is deliberately NOT run during staging.** The ESP stub
    still points at the old `/boot` inside the ext4 root, so the host's boot
    path stays byte-identical to what it has been for 140 days. Everything
    error-prone is built and verified in advance; the ESP rewrite is a
    two-second idempotent command held back to the window.

**Cutover** — the remaining work, § Cutover below.

> **The transferable lessons from this migration live in**
> [`docs/pfi/ops-lessons-playbook.md`](../pfi/ops-lessons-playbook.md) — the ops sibling
> to the quantization playbook. Everything below is the ESH-specific narrative;
> the rules that would bite on any host are collected there.

## ⚠ The mount-propagation incident — the expensive lesson of 2026-08-18

**What broke.** The staging chroot was built with `mount --rbind /dev` and `/sys`
and **no `--make-rslave`**. On a systemd host `/` has *shared* propagation, so
those binds propagate in both directions. When the cutover tore the chroot down
with `umount -R`, the unmounts **propagated back into the live host** and removed
the real `/sys/fs/cgroup`, `/dev/pts` and `/dev/shm`.

With cgroup2 gone, `systemd-logind` could no longer create a session. The result
is a host that:

- answers ping, accepts TCP, and **completes SSH authentication**
- keeps serving from daemons already resident in memory (`pveproxy` returned a
  clean HTTP 401 throughout)
- **hangs on every new `exec`**, including `/sbin/reboot` — so the reboot that was
  supposed to end the window never ran

**Why it cost so much time: it is a near-perfect impostor of failing root-disk
I/O.** Both present as "host is up, daemons answer, nothing new can start." The
session diagnosed it as the USB DOM dying and told the operator to walk to the
machine. That was wrong, and the operator caught it: the DOM had been reliable
for years and the wedge began immediately after a change.

**The evidence that settles it, and was available the whole time** — from
`dmesg`, obtainable in the brief windows when exec did succeed:

| line | says |
|---|---|
| `[16.00] sd 56:0:0:0: [sdq] Attached SCSI removable disk` | DOM enumerated **cleanly, no errors** |
| `[12114881.98] systemd[1]: nvme-varlog-stage.mount: Deactivated` | timestamp is **140 days** — this is the ORIGINAL boot |

That second line is the whole answer: **the machine never rebooted.** A
down-detector loop had also never once reported the host down; that was read as a
fast reboot rather than as no reboot at all.

**Rules that follow:**

1. **Always `mount --make-rslave` after `mount --rbind` into a chroot.** Phase 2
   now does this and carries a guard that refuses to continue if any bind still
   reports `shared` propagation.
2. **A reboot is not confirmed until the host is observed DOWN.** Poll for
   disappearance, not just for reappearance. "Never went down" and "went down and
   came back fast" are indistinguishable if you only watch for the host to answer.
3. **Before blaming hardware for a wedge that began right after a change, get
   `dmesg` and check the boot timestamp.** Diagnose the change first; hardware is
   the explanation of last resort, not first.

**Recovery took no console access.** Windows where `exec` briefly succeeded were
enough to land an idempotent remount of cgroup2 / devpts / shm, after which
`systemctl reset-failed` returned the host to `running`. Total data loss: none.
The root filesystem, the DOM and all three pools were never at risk — this was a
mount-namespace fault, not a storage one.

## The pool-name bug's neighbours — two more corrections

**The blast radius was more than double what was documented.** The runbook named
two NFS dependents. `ss -tn '( sport = :2049 )'` inside CT 103 showed **five**:

| client | mount | disposition |
|---|---|---|
| `10.0.50.45` esh-docker-vm | `/mnt/books`, `/mnt/backup` — **hard** | quiesced |
| `10.0.250.35` esh-pve | `esh-nas`, `tank-vmbu` — **hard** | quiesced |
| `10.0.50.60` **esh-vm-db** | `/mnt/backup` — **hard** | **left mounted deliberately** |
| `10.0.50.154` vm-esh-nas | — | is VM 104 *on this host*; stops with it |
| `10.100.10.50` nh3-dev | `/mnt/books` — **soft,ro** | safe, errors instead of blocking |

Ask the *server* who its clients are. A runbook's list of dependents is a snapshot
that rots; `ss` on the NFS server is ground truth.

esh-vm-db was left mounted on purpose and **came through read-write** — a `hard`
mount with no active user blocks and resumes, which is what `hard` is for. Its
backup timers were ~19h out, and unmounting would have meant an unmount/remount
cycle over the qemu guest agent on a host with no ssh access.

**The one-shot rollback does not work, and the warning was right.**
`grub-reboot` printed *"Detected GRUB environment block on lvm device — will
remain the default boot entry until manually cleared."* Confirmed empirically:
after the successful ZFS boot, `next_entry=pve-zfs-root` was **still set**. GRUB
can read grubenv on LVM but cannot write it, so `boot_once` degrades to a sticky
default. **There is no auto-fallback on this host.** A failed boot must be
corrected at the console.

The steady-state config therefore does not rely on it: `saved_entry=pve-zfs-root`
with `next_entry` cleared. Restoring a real one-shot would mean relocating grubenv
onto the ESP (vfat on a plain partition, which GRUB *can* write) — parked, not
required.

## Cutover

The only remaining work. Everything below the quiesce is minutes.

1. **Quiesce the NFS clients** (see § Blast radius). On **esh-docker-vm**
   (10.0.50.45) stop whatever holds `/mnt/books` and `/mnt/backup` and unmount
   them; on **esh-pve** (10.0.250.35) disable the `esh-nas` and `tank-vmbu`
   storages. Do this first and confirm it — a `hard` mount left live turns a
   brief reboot into an unkillable D-state needing a reboot of *that* host too.
2. Shut down the five guests.
3. Point the ESP at the new `/boot` and arm the one-shot:
   ```
   chroot /mnt/newroot grub-install --target=x86_64-efi \
       --efi-directory=/boot/efi --bootloader-id=proxmox
   chroot /mnt/newroot grub-reboot '<zfs entry id — phase 2's verify prints it>'
   ```
4. Set the dataset's final mountpoint, then reboot:
   ```
   zfs set mountpoint=/ nvme/ROOT/pve-1     # canmount stays noauto
   reboot
   ```

**Why `grub-reboot` and not a new default.** `GRUB_DEFAULT=saved` with grubenv
pinned to the ext4 rollback means the ZFS entry is tried **exactly once**. If it
fails, the next reboot returns to ext4 *by itself* — no console, no hands. That
matters more here than on a normal host: a boot that hangs at an initramfs
prompt takes CT 103 `esh-nas` down with it, and the NFS clients hang rather than
fail. Only after the second successful ZFS boot (§ Verification) should the
saved default move to the ZFS entry with `grub-set-default`.

## Verification

- `findmnt -no SOURCE,FSTYPE /` → `nvme/ROOT/pve-1  zfs`
- `df -h /` shows hundreds of GB, not 5.9
- `findmnt /boot` → ext4 on the DOM; `/boot/efi` mounted
- all five guests running; CT 103 serving NFS (`pct exec 103 -- exportfs -v`)
- esh-docker-vm remounted and healthy; esh-pve storages green
- **a second reboot** to prove it was not a one-off
- only then: refresh the DOM image, since `/boot` has changed

## Rollback

Instant and cheap at every stage: the ext4 root on the DOM is never modified, and
its GRUB entry stays in the menu. Worst case is a boot to initramfs → reboot →
pick the old entry. Keep the ext4 root for at least a few weeks of normal
operation before reclaiming it.

## Open decisions

- **Second boot device?** The split fixes runtime fragility but not boot-time
  single-point-of-failure. A cloned DOM/USB as a cold spare is the cheap answer.
- **`esh-filebot` (CT 106)** is an empty container — 80 GB quota, six passthrough
  mounts, nothing running since March. Retire rather than carry it.
- **Reclaiming the old ext4 root** once the ZFS root has proven itself.

---

## Fallback plan: full reinstall to a mirrored-NVMe ZFS root

Only if the split above proves unworkable. Fresh PVE install to ZFS RAID1 across
both NVMes — mirrored boot with proper ESPs under `proxmox-boot-tool`, no USB in
the path at all.

Costs: the `nvme` pool must be destroyed, so its **32 GB of guest rootfs** moves
to `ssd` (1.42 T free) first; `ssd` and `tank` must be cleanly exported so the
installer cannot touch them; guest configs restore from the snapshot plus the 8
PBS backups per guest. Half a day, and rollback after the install step is
"reinstall and restore".

Note both NVMes are *whole-disk* ZFS members (partition 1 spans all 931.5 GiB,
1.7 MiB free), so adding an ESP to them without destroying the pool is
impossible — which is what forces the reinstall in this variant, and what the
split plan avoids entirely.