vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity

docs(esh-pve-nas): split boot from root instead of reinstalling

Browse Source 
Operator's proposal, and it is strictly better than the reinstall plan.

Boot and root do not have to share a device. Keep the ESP and /boot on the DOM
as ext4 -- so GRUB never has to read ZFS, which matters because the nvme pool
has encryption, large_dnode and zstd_compress enabled and GRUB cannot read
those -- and move root to nvme/ROOT/pve-1. The initramfs imports the pool and
pivots.

What this buys over the reinstall: the nvme pool survives, so no guest
migration, no export/import of ssd and tank, no reinstall. Downtime is one
reboot rather than half a day. Rollback is a GRUB menu entry, because the ext4
root stays on the DOM untouched. And it retires the actual top risk -- with
root on NVMe, a USB bus reset mid-run no longer takes the running system down;
the DOM becomes read-mostly, written only on kernel updates.

Preconditions verified and already met: UEFI with grub-efi, zfs-initramfs
2.2.8-pve1 installed with 76 ZFS files already in the running initrd, root only
4.3 GB to copy, swap negligible against 125 GB RAM.

Two traps recorded: canmount=noauto on the root dataset or ZFS tries to mount
over the running root; and cachefile is currently none with a 0-byte
zpool.cache, so the pool imports by scan today and must be given a cachefile
before the initramfs is rebuilt.

The reinstall plan is retained as the fallback.
This commit is contained in:
vh
2026-08-17 21:36:30 -07:00
parent 2275e11be0
commit 3e311756d7
1 changed files with 138 additions and 119 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/runbooks/esh-pve-nas-boot-migration.md
+138 -119
View File
@@ -1,149 +1,168 @@
# esh-pve-nas — migrating PVE off the USB DOM
# esh-pve-nas — moving PVE root off the USB DOM
**Status: PLANNED, not executed.** Mitigation landed 2026-08-17 (root 90% → 76%);
this is the structural fix.
**Status: PLANNED, not executed.** Mitigation landed 2026-08-17 (root 90% → 76%).
**Plan revised 2026-08-17** from "reinstall to a mirrored-NVMe ZFS root" to
**"split the boot chain from the root filesystem"** — operator's proposal, and it
is strictly better. The original reinstall plan is kept at the bottom as the
fallback.
## Why
PVE root lives on a **USB Disk-on-Module** — `sdq`, 7.3 GB, `ID_BUS=usb`,
`ID_VENDOR=NORELSYS`, model 1081 — carved into a 512 MB ESP, a 768 MB swap LV
and a **6 GB ext4 root**.
`ID_VENDOR=NORELSYS` — as a 6 GB ext4 root plus 768 MB swap and a 512 MB ESP.
A DOM is not a thumb drive: SLC/pSLC NAND with a real controller, so the
284 GB written since boot is unremarkable and **wear is not the driver**. What
is:
A DOM is SLC/pSLC with a real controller, so the 284 GB written since boot is
unremarkable and **wear is not the driver**. The actual problems:
1. **It is on the USB bus.** NAND quality is irrelevant to a bus reset or a
re-enumeration event — either drops the root filesystem out from under a
running hypervisor while its guests keep executing.
2. **6 GB has no headroom.** It hit 90% (571 MB free) with nothing pathological
on it — `/usr` alone is 3.7 GB. Tonight's mitigation bought 1.4 GB, not room
to grow.
3. **It is single and unmirrored** — the one device the host cannot run without
has no redundancy, while 928 GB of *mirrored* NVMe sits 96% empty.
1. **It is on the USB bus.** A bus reset or re-enumeration drops the *root
filesystem* out from under a running hypervisor while its guests keep going.
2. **6 GB has no headroom** — `/usr` alone is 3.7 GB.
3. **Unmirrored**, while 928 GB of mirrored NVMe sits 96% empty.
## Current state (captured 2026-08-18)
## The design: boot on the DOM, root on ZFS
Off-box snapshot: `nh3-dev:/home/lkraven/backups/esh-pve-nas/pve-config-snapshot-20260818T043027Z.tar.gz`
(0600, sha256 `dc312793d027dc43…`). Contains `/etc/pve`, `/etc/network/interfaces`,
`/etc/fstab`, apt config, root's `authorized_keys`, plus captured live state:
`zpool status/list/properties`, `zfs list`, `/dev/disk/by-id`, `lsblk` with
serials + WWNs, `pvesm status`, `pveversion -v`, `dpkg --get-selections`, and
every CT/VM config. **The newest on-disk copy before this was June 2024.**
Boot and root do not have to live on the same device. Split them:
| | detail |
| | device | contents | written when |
|---|---|---|---|
| **boot** | DOM `sdq` | ESP + `/boot` (ext4): GRUB, kernels, initramfs | **only on kernel/GRUB updates** |
| **root** | `nvme` pool | `nvme/ROOT/pve-1` — everything else | constantly, on mirrored NVMe |
GRUB reads the kernel and initrd from **ext4 on the DOM**, so GRUB never has to
read ZFS — which matters, because the `nvme` pool has `encryption`,
`large_dnode` and `zstd_compress` enabled and GRUB cannot read those. The
initramfs then imports the pool and pivots to `root=ZFS=nvme/ROOT/pve-1`.
### Why this beats the reinstall
- **The `nvme` pool is not destroyed.** The root dataset is created *inside* the
existing pool. No guest migration, no `zpool export/import` of `ssd`/`tank`,
no reinstall.
- **Downtime is one reboot**, not half a day.
- **Rollback is a GRUB menu entry.** The existing ext4 root stays on the DOM,
untouched. If ZFS root fails to come up, pick the old entry and you are back in
a minute. That is a far better rollback than "reinstall and restore."
- **The #1 risk is actually retired.** Once booted, root is on NVMe — a USB bus
reset mid-run no longer takes the running system down. The DOM becomes
read-mostly.
- **Free upside: boot environments.** `zfs snapshot nvme/ROOT/pve-1@pre-upgrade`
before an apt run, roll back if it breaks.
### What it does NOT fix
The DOM remains the **only boot path**. If it dies, the machine will not boot
until the image is restored — though the ZFS root, with all config and guests,
stays intact. Mitigation is a **cloned fallback image** (`dd` of `sdq`, ~7 GB,
refreshed after kernel updates), kept off-box next to the config snapshot.
## Preconditions — all already satisfied
Verified on the host 2026-08-17:
- **UEFI** firmware, `grub-efi-amd64 2.06-13+pmx7` installed
- **`zfs-initramfs 2.2.8-pve1` is already installed**, and the running initrd
already carries **76 ZFS files** — the pivot capability exists today, no new
packages
- `/boot` is currently *part of* root (108 MB), so it must be split out onto its
own ext4 filesystem on the DOM as part of this work
- root is only **4.3 GB** to copy
- swap is 767 MB with 123 MB used against 125 GB of RAM — irrelevant; leave it
on the DOM LV. **Do not put swap on a zvol** (deadlock risk)
⚠ **`cachefile` is `none` and `/etc/zfs/zpool.cache` is 0 bytes** — pools import
by scan today. For root-on-ZFS this must be deterministic: set
`zpool set cachefile=/etc/zfs/zpool.cache nvme` and rebuild the initramfs, or the
pool may not be imported early enough to find root.
## ⚠ Blast radius — unchanged, and still the gating constraint
**CT 103 `esh-nas` (10.0.50.50) is the NAS, and it runs on this host.** Two
dependents mount it over **`hard`** NFS — they do not fail, they hang unkillably:
| client | mounts |
|---|---|
| node | `esh-nas-pve.esteban.net`, PVE 8.4.11, kernel 6.8.12-13 |
| network | `vmbr0` static **10.0.50.55/24**, gw 10.0.50.1, bridge-port `enp5s0f0`, VLAN-aware 2-4094 |
| `nvme` pool | mirror, 2× 931.5 GB NVMe — **32.1 G used, 867 G free, no snapshots** |
| `ssd` pool | 2 mirrors, 4× 894 GB Intel SATA — 263 G used, 1.42 T free |
| `tank` pool | 2× raidz2, 12× 14.6 TB — 40.2 T of 175 T |
| guests | CT103 `esh-nas`, CT105 `vm-plex`, CT106 `esh-filebot`, CT107 `vm-jellyfin`, VM104 `vm-esh-nas` |
| backups | 8 per guest on `pbs-ana` |
| **esh-docker-vm** (10.0.50.45) | `/mnt/books`, `/mnt/backup` |
| **esh-pve** (10.0.250.35) | `/mnt/pve/esh-nas`, `/mnt/pve/tank-vmbu` |
**Every guest rootfs is on `nvme`, and nothing else is.** Media and data live on
`tank` via LXC passthrough mounts. So the pool that must be cleared holds
**~32 GB** — `ssd` absorbs that without noticing.
This is a known incident shape: the only remedy for esh-docker-vm's D-state is a
host reboot, and `/mnt/books` was *deliberately* left `hard` because calibre's
SQLite risks corruption under `soft`.
## ⚠ The blast radius — read this before scheduling
**CT 103 `esh-nas` (10.0.50.50) is the NAS, and it runs on the box being
rebuilt.** Two dependents mount it over **`hard`** NFS, which means they do not
fail — they hang, unkillably:
| client | mounts | mode |
|---|---|---|
| **esh-docker-vm** (10.0.50.45) | `/mnt/books`, `/mnt/backup` | `hard` |
| **esh-pve** (10.0.250.35) | `/mnt/pve/esh-nas`, `/mnt/pve/tank-vmbu` | `hard` |
This is a known incident shape — hard NFS from 10.0.50.50 puts esh-docker-vm
into unkillable D-state whose only remedy is a host reboot, and `/mnt/books`
was *deliberately* left `hard` because calibre's SQLite would risk corruption
under `soft`. There is a watchdog on esh-pve for exactly this.
**Consequence: taking this host down stalls the other hypervisor's storage
layer and wedges esh-docker-vm unless its NFS is quiesced first.** That is the
single largest risk in this migration and it is not obvious from the host being
worked on.
## Target design (recommended)
**Fresh PVE install to ZFS RAID1 across the two NVMes.** The installer builds
`rpool` with proper ESPs on both devices, managed by `proxmox-boot-tool` —
mirrored boot, no USB in the path, 900 GB of headroom. `rpool` then serves both
the OS and the 32 GB of guest rootfs, collapsing the separate `nvme` pool.
Why not the alternatives:
- **In-place conversion** (LVM-ext4 root → ZFS root) is not a supported path and
fails into a rebuild anyway.
- **Add an ESP to the existing NVMes** is impossible: both are *whole-disk* ZFS
members (partition 1 spans all 931.5 GiB, type BF01) with **1.7 MiB free**.
`proxmox-boot-tool status` confirms nothing is managed — the DOM's ESP is the
only boot path.
- **Install to 2 of the 4 Intel SSDs** works but destroys one `ssd` mirror and
spends 894 GB on a boot device.
- **Replace the DOM** leaves every structural problem intact.
The reboot in this plan is brief, but it is still a reboot — quiesce the clients
first.
## Sequence
**Pre-flight (no downtime)**
1. Verify PBS backups are current for all five guests; take fresh ones if not.
2. Re-pull the config snapshot so it is same-day.
3. Record `zpool status` GUIDs and the by-id paths for **`ssd` and `tank`** — the
installer must never see these disks. 16 SATA devices stay untouched.
4. Confirm install media + out-of-band console access to the QNAP.
1. `dd` the DOM to an image, off-box. This is the rollback for the boot chain.
2. Refresh the config snapshot (`nh3-dev:~/backups/esh-pve-nas/`).
3. `zpool set cachefile=/etc/zfs/zpool.cache nvme`.
**Quiesce (start of window)**
5. On **esh-docker-vm**: stop the containers touching NFS, then `umount` `/mnt/books`
and `/mnt/backup`. Do not skip — this is the wedge.
6. On **esh-pve**: disable the `esh-nas` and `tank-vmbu` storages in the PVE UI
(Datacenter → Storage), so pvestatd stops probing them.
7. Shut down all five guests on esh-pve-nas.
**Build the ZFS root (live, no disruption)**
4. `zfs create -o mountpoint=none nvme/ROOT`
`zfs create -o mountpoint=/ -o canmount=noauto nvme/ROOT/pve-1`
`canmount=noauto` is essential — otherwise ZFS tries to mount it over the
running root.
5. Mount it somewhere temporary and `rsync -aHAX --numeric-ids` the live root
into it, excluding `/proc /sys /dev /run /mnt /nvme /ssd /tank /var/log/journal`.
6. In the copy: edit `/etc/fstab` to drop the root line, keep the ESP and swap
lines, and add the new `/boot` entry.
**Move guest rootfs off `nvme`**
8. `zfs send | zfs recv` each of the six datasets to `ssd` (~32 GB), or restore
from PBS after the install. Sending is faster and does not depend on backups
being complete.
9. Update the guest configs to reference `ssd:` rather than `nvme:`.
**Split `/boot` onto the DOM**
7. Shrink/repartition is not needed — `/boot` can stay inside the DOM's existing
LVM as its own small ext4 LV, or reuse the freed space once root moves off.
Copy `/boot` into it and mount at `/boot`.
**Rebuild**
10. `zpool export ssd tank` — clean export so the installer cannot disturb them.
11. Install PVE to **ZFS RAID1 on nvme0n1 + nvme1n1 only**. Same hostname, same
IP, same PVE version family.
12. `zpool import ssd tank`; restore `/etc/pve/storage.cfg` entries; restore
`/etc/network/interfaces` verbatim (VLAN-aware bridge on `enp5s0f0`).
13. Restore guest configs; start CT 103 **first** — it is the NAS the others
need.
**Bootloader**
8. `GRUB_CMDLINE_LINUX="root=ZFS=nvme/ROOT/pve-1"`, `update-initramfs -u -k all`,
`update-grub`. **Keep the existing ext4 entry in the menu** and leave
`GRUB_TIMEOUT=5` so it is selectable.
**Unquiesce**
14. Re-enable storages on esh-pve; remount NFS on esh-docker-vm; restart its
containers.
15. Physically remove or blank the DOM so it can never be booted from again.
**Cut over**
9. Quiesce the NFS clients (see blast radius), shut down guests, reboot.
10. Verify (below). If it fails, reboot and choose the ext4 entry.
## Verification
- `proxmox-boot-tool status` lists **both** NVMe ESPs
- root is on `rpool/ROOT/pve-1`, and `df -h /` shows hundreds of GB
- all five guests running; `pct exec 103 -- exportfs -v` serving
- esh-docker-vm: `/mnt/books` and `/mnt/backup` mounted, containers healthy
- esh-pve: both NFS storages green in the UI
- a reboot test **before** declaring done — boot order is the thing being changed
- `findmnt -no SOURCE,FSTYPE /` → `nvme/ROOT/pve-1 zfs`
- `df -h /` shows hundreds of GB, not 5.9
- `findmnt /boot` → ext4 on the DOM; `/boot/efi` mounted
- all five guests running; CT 103 serving NFS (`pct exec 103 -- exportfs -v`)
- esh-docker-vm remounted and healthy; esh-pve storages green
- **a second reboot** to prove it was not a one-off
- only then: refresh the DOM image, since `/boot` has changed
## Rollback
Until step 11 everything is reversible: the DOM is untouched, so leaving the
NVMes alone and re-importing pools returns the old system. **After step 11 the
old root is gone** — rollback becomes "reinstall and restore from the snapshot
plus PBS." Keep the DOM physically intact and unmodified until verification
passes; it *is* the rollback.
Instant and cheap at every stage: the ext4 root on the DOM is never modified, and
its GRUB entry stays in the menu. Worst case is a boot to initramfs → reboot →
pick the old entry. Keep the ext4 root for at least a few weeks of normal
operation before reclaiming it.
## Open decisions
- **Window length.** Realistically half a day, dominated by install and
verification, not by the 32 GB move.
- **What to do with the freed DOM slot** — leave empty, or keep the DOM as a
cold spare.
- **`esh-filebot` (CT 106)** is an empty container: 80 GB quota, six passthrough
mounts, nothing running since March. Migrating it is wasted work — decide
whether it comes across at all.
- **Second boot device?** The split fixes runtime fragility but not boot-time
single-point-of-failure. A cloned DOM/USB as a cold spare is the cheap answer.
- **`esh-filebot` (CT 106)** is an empty container — 80 GB quota, six passthrough
mounts, nothing running since March. Retire rather than carry it.
- **Reclaiming the old ext4 root** once the ZFS root has proven itself.
---
## Fallback plan: full reinstall to a mirrored-NVMe ZFS root
Only if the split above proves unworkable. Fresh PVE install to ZFS RAID1 across
both NVMes — mirrored boot with proper ESPs under `proxmox-boot-tool`, no USB in
the path at all.
Costs: the `nvme` pool must be destroyed, so its **32 GB of guest rootfs** moves
to `ssd` (1.42 T free) first; `ssd` and `tank` must be cleanly exported so the
installer cannot touch them; guest configs restore from the snapshot plus the 8
PBS backups per guest. Half a day, and rollback after the install step is
"reinstall and restore".
Note both NVMes are *whole-disk* ZFS members (partition 1 spans all 931.5 GiB,
1.7 MiB free), so adding an ESP to them without destroying the pool is
impossible — which is what forces the reinstall in this variant, and what the
split plan avoids entirely.