docs(esh-pve-nas): split boot from root instead of reinstalling
Operator's proposal, and it is strictly better than the reinstall plan. Boot and root do not have to share a device. Keep the ESP and /boot on the DOM as ext4 -- so GRUB never has to read ZFS, which matters because the nvme pool has encryption, large_dnode and zstd_compress enabled and GRUB cannot read those -- and move root to nvme/ROOT/pve-1. The initramfs imports the pool and pivots. What this buys over the reinstall: the nvme pool survives, so no guest migration, no export/import of ssd and tank, no reinstall. Downtime is one reboot rather than half a day. Rollback is a GRUB menu entry, because the ext4 root stays on the DOM untouched. And it retires the actual top risk -- with root on NVMe, a USB bus reset mid-run no longer takes the running system down; the DOM becomes read-mostly, written only on kernel updates. Preconditions verified and already met: UEFI with grub-efi, zfs-initramfs 2.2.8-pve1 installed with 76 ZFS files already in the running initrd, root only 4.3 GB to copy, swap negligible against 125 GB RAM. Two traps recorded: canmount=noauto on the root dataset or ZFS tries to mount over the running root; and cachefile is currently none with a 0-byte zpool.cache, so the pool imports by scan today and must be given a cachefile before the initramfs is rebuilt. The reinstall plan is retained as the fallback.
This commit is contained in:
@@ -1,149 +1,168 @@
|
|||||||
# esh-pve-nas — migrating PVE off the USB DOM
|
# esh-pve-nas — moving PVE root off the USB DOM
|
||||||
|
|
||||||
**Status: PLANNED, not executed.** Mitigation landed 2026-08-17 (root 90% → 76%);
|
**Status: PLANNED, not executed.** Mitigation landed 2026-08-17 (root 90% → 76%).
|
||||||
this is the structural fix.
|
|
||||||
|
**Plan revised 2026-08-17** from "reinstall to a mirrored-NVMe ZFS root" to
|
||||||
|
**"split the boot chain from the root filesystem"** — operator's proposal, and it
|
||||||
|
is strictly better. The original reinstall plan is kept at the bottom as the
|
||||||
|
fallback.
|
||||||
|
|
||||||
## Why
|
## Why
|
||||||
|
|
||||||
PVE root lives on a **USB Disk-on-Module** — `sdq`, 7.3 GB, `ID_BUS=usb`,
|
PVE root lives on a **USB Disk-on-Module** — `sdq`, 7.3 GB, `ID_BUS=usb`,
|
||||||
`ID_VENDOR=NORELSYS`, model 1081 — carved into a 512 MB ESP, a 768 MB swap LV
|
`ID_VENDOR=NORELSYS` — as a 6 GB ext4 root plus 768 MB swap and a 512 MB ESP.
|
||||||
and a **6 GB ext4 root**.
|
|
||||||
|
|
||||||
A DOM is not a thumb drive: SLC/pSLC NAND with a real controller, so the
|
A DOM is SLC/pSLC with a real controller, so the 284 GB written since boot is
|
||||||
284 GB written since boot is unremarkable and **wear is not the driver**. What
|
unremarkable and **wear is not the driver**. The actual problems:
|
||||||
is:
|
|
||||||
|
|
||||||
1. **It is on the USB bus.** NAND quality is irrelevant to a bus reset or a
|
1. **It is on the USB bus.** A bus reset or re-enumeration drops the *root
|
||||||
re-enumeration event — either drops the root filesystem out from under a
|
filesystem* out from under a running hypervisor while its guests keep going.
|
||||||
running hypervisor while its guests keep executing.
|
2. **6 GB has no headroom** — `/usr` alone is 3.7 GB.
|
||||||
2. **6 GB has no headroom.** It hit 90% (571 MB free) with nothing pathological
|
3. **Unmirrored**, while 928 GB of mirrored NVMe sits 96% empty.
|
||||||
on it — `/usr` alone is 3.7 GB. Tonight's mitigation bought 1.4 GB, not room
|
|
||||||
to grow.
|
|
||||||
3. **It is single and unmirrored** — the one device the host cannot run without
|
|
||||||
has no redundancy, while 928 GB of *mirrored* NVMe sits 96% empty.
|
|
||||||
|
|
||||||
## Current state (captured 2026-08-18)
|
## The design: boot on the DOM, root on ZFS
|
||||||
|
|
||||||
Off-box snapshot: `nh3-dev:/home/lkraven/backups/esh-pve-nas/pve-config-snapshot-20260818T043027Z.tar.gz`
|
Boot and root do not have to live on the same device. Split them:
|
||||||
(0600, sha256 `dc312793d027dc43…`). Contains `/etc/pve`, `/etc/network/interfaces`,
|
|
||||||
`/etc/fstab`, apt config, root's `authorized_keys`, plus captured live state:
|
|
||||||
`zpool status/list/properties`, `zfs list`, `/dev/disk/by-id`, `lsblk` with
|
|
||||||
serials + WWNs, `pvesm status`, `pveversion -v`, `dpkg --get-selections`, and
|
|
||||||
every CT/VM config. **The newest on-disk copy before this was June 2024.**
|
|
||||||
|
|
||||||
| | detail |
|
| | device | contents | written when |
|
||||||
|
|---|---|---|---|
|
||||||
|
| **boot** | DOM `sdq` | ESP + `/boot` (ext4): GRUB, kernels, initramfs | **only on kernel/GRUB updates** |
|
||||||
|
| **root** | `nvme` pool | `nvme/ROOT/pve-1` — everything else | constantly, on mirrored NVMe |
|
||||||
|
|
||||||
|
GRUB reads the kernel and initrd from **ext4 on the DOM**, so GRUB never has to
|
||||||
|
read ZFS — which matters, because the `nvme` pool has `encryption`,
|
||||||
|
`large_dnode` and `zstd_compress` enabled and GRUB cannot read those. The
|
||||||
|
initramfs then imports the pool and pivots to `root=ZFS=nvme/ROOT/pve-1`.
|
||||||
|
|
||||||
|
### Why this beats the reinstall
|
||||||
|
|
||||||
|
- **The `nvme` pool is not destroyed.** The root dataset is created *inside* the
|
||||||
|
existing pool. No guest migration, no `zpool export/import` of `ssd`/`tank`,
|
||||||
|
no reinstall.
|
||||||
|
- **Downtime is one reboot**, not half a day.
|
||||||
|
- **Rollback is a GRUB menu entry.** The existing ext4 root stays on the DOM,
|
||||||
|
untouched. If ZFS root fails to come up, pick the old entry and you are back in
|
||||||
|
a minute. That is a far better rollback than "reinstall and restore."
|
||||||
|
- **The #1 risk is actually retired.** Once booted, root is on NVMe — a USB bus
|
||||||
|
reset mid-run no longer takes the running system down. The DOM becomes
|
||||||
|
read-mostly.
|
||||||
|
- **Free upside: boot environments.** `zfs snapshot nvme/ROOT/pve-1@pre-upgrade`
|
||||||
|
before an apt run, roll back if it breaks.
|
||||||
|
|
||||||
|
### What it does NOT fix
|
||||||
|
|
||||||
|
The DOM remains the **only boot path**. If it dies, the machine will not boot
|
||||||
|
until the image is restored — though the ZFS root, with all config and guests,
|
||||||
|
stays intact. Mitigation is a **cloned fallback image** (`dd` of `sdq`, ~7 GB,
|
||||||
|
refreshed after kernel updates), kept off-box next to the config snapshot.
|
||||||
|
|
||||||
|
## Preconditions — all already satisfied
|
||||||
|
|
||||||
|
Verified on the host 2026-08-17:
|
||||||
|
|
||||||
|
- **UEFI** firmware, `grub-efi-amd64 2.06-13+pmx7` installed
|
||||||
|
- **`zfs-initramfs 2.2.8-pve1` is already installed**, and the running initrd
|
||||||
|
already carries **76 ZFS files** — the pivot capability exists today, no new
|
||||||
|
packages
|
||||||
|
- `/boot` is currently *part of* root (108 MB), so it must be split out onto its
|
||||||
|
own ext4 filesystem on the DOM as part of this work
|
||||||
|
- root is only **4.3 GB** to copy
|
||||||
|
- swap is 767 MB with 123 MB used against 125 GB of RAM — irrelevant; leave it
|
||||||
|
on the DOM LV. **Do not put swap on a zvol** (deadlock risk)
|
||||||
|
|
||||||
|
⚠ **`cachefile` is `none` and `/etc/zfs/zpool.cache` is 0 bytes** — pools import
|
||||||
|
by scan today. For root-on-ZFS this must be deterministic: set
|
||||||
|
`zpool set cachefile=/etc/zfs/zpool.cache nvme` and rebuild the initramfs, or the
|
||||||
|
pool may not be imported early enough to find root.
|
||||||
|
|
||||||
|
## ⚠ Blast radius — unchanged, and still the gating constraint
|
||||||
|
|
||||||
|
**CT 103 `esh-nas` (10.0.50.50) is the NAS, and it runs on this host.** Two
|
||||||
|
dependents mount it over **`hard`** NFS — they do not fail, they hang unkillably:
|
||||||
|
|
||||||
|
| client | mounts |
|
||||||
|---|---|
|
|---|---|
|
||||||
| node | `esh-nas-pve.esteban.net`, PVE 8.4.11, kernel 6.8.12-13 |
|
| **esh-docker-vm** (10.0.50.45) | `/mnt/books`, `/mnt/backup` |
|
||||||
| network | `vmbr0` static **10.0.50.55/24**, gw 10.0.50.1, bridge-port `enp5s0f0`, VLAN-aware 2-4094 |
|
| **esh-pve** (10.0.250.35) | `/mnt/pve/esh-nas`, `/mnt/pve/tank-vmbu` |
|
||||||
| `nvme` pool | mirror, 2× 931.5 GB NVMe — **32.1 G used, 867 G free, no snapshots** |
|
|
||||||
| `ssd` pool | 2 mirrors, 4× 894 GB Intel SATA — 263 G used, 1.42 T free |
|
|
||||||
| `tank` pool | 2× raidz2, 12× 14.6 TB — 40.2 T of 175 T |
|
|
||||||
| guests | CT103 `esh-nas`, CT105 `vm-plex`, CT106 `esh-filebot`, CT107 `vm-jellyfin`, VM104 `vm-esh-nas` |
|
|
||||||
| backups | 8 per guest on `pbs-ana` |
|
|
||||||
|
|
||||||
**Every guest rootfs is on `nvme`, and nothing else is.** Media and data live on
|
This is a known incident shape: the only remedy for esh-docker-vm's D-state is a
|
||||||
`tank` via LXC passthrough mounts. So the pool that must be cleared holds
|
host reboot, and `/mnt/books` was *deliberately* left `hard` because calibre's
|
||||||
**~32 GB** — `ssd` absorbs that without noticing.
|
SQLite risks corruption under `soft`.
|
||||||
|
|
||||||
## ⚠ The blast radius — read this before scheduling
|
The reboot in this plan is brief, but it is still a reboot — quiesce the clients
|
||||||
|
first.
|
||||||
**CT 103 `esh-nas` (10.0.50.50) is the NAS, and it runs on the box being
|
|
||||||
rebuilt.** Two dependents mount it over **`hard`** NFS, which means they do not
|
|
||||||
fail — they hang, unkillably:
|
|
||||||
|
|
||||||
| client | mounts | mode |
|
|
||||||
|---|---|---|
|
|
||||||
| **esh-docker-vm** (10.0.50.45) | `/mnt/books`, `/mnt/backup` | `hard` |
|
|
||||||
| **esh-pve** (10.0.250.35) | `/mnt/pve/esh-nas`, `/mnt/pve/tank-vmbu` | `hard` |
|
|
||||||
|
|
||||||
This is a known incident shape — hard NFS from 10.0.50.50 puts esh-docker-vm
|
|
||||||
into unkillable D-state whose only remedy is a host reboot, and `/mnt/books`
|
|
||||||
was *deliberately* left `hard` because calibre's SQLite would risk corruption
|
|
||||||
under `soft`. There is a watchdog on esh-pve for exactly this.
|
|
||||||
|
|
||||||
**Consequence: taking this host down stalls the other hypervisor's storage
|
|
||||||
layer and wedges esh-docker-vm unless its NFS is quiesced first.** That is the
|
|
||||||
single largest risk in this migration and it is not obvious from the host being
|
|
||||||
worked on.
|
|
||||||
|
|
||||||
## Target design (recommended)
|
|
||||||
|
|
||||||
**Fresh PVE install to ZFS RAID1 across the two NVMes.** The installer builds
|
|
||||||
`rpool` with proper ESPs on both devices, managed by `proxmox-boot-tool` —
|
|
||||||
mirrored boot, no USB in the path, 900 GB of headroom. `rpool` then serves both
|
|
||||||
the OS and the 32 GB of guest rootfs, collapsing the separate `nvme` pool.
|
|
||||||
|
|
||||||
Why not the alternatives:
|
|
||||||
|
|
||||||
- **In-place conversion** (LVM-ext4 root → ZFS root) is not a supported path and
|
|
||||||
fails into a rebuild anyway.
|
|
||||||
- **Add an ESP to the existing NVMes** is impossible: both are *whole-disk* ZFS
|
|
||||||
members (partition 1 spans all 931.5 GiB, type BF01) with **1.7 MiB free**.
|
|
||||||
`proxmox-boot-tool status` confirms nothing is managed — the DOM's ESP is the
|
|
||||||
only boot path.
|
|
||||||
- **Install to 2 of the 4 Intel SSDs** works but destroys one `ssd` mirror and
|
|
||||||
spends 894 GB on a boot device.
|
|
||||||
- **Replace the DOM** leaves every structural problem intact.
|
|
||||||
|
|
||||||
## Sequence
|
## Sequence
|
||||||
|
|
||||||
**Pre-flight (no downtime)**
|
**Pre-flight (no downtime)**
|
||||||
1. Verify PBS backups are current for all five guests; take fresh ones if not.
|
1. `dd` the DOM to an image, off-box. This is the rollback for the boot chain.
|
||||||
2. Re-pull the config snapshot so it is same-day.
|
2. Refresh the config snapshot (`nh3-dev:~/backups/esh-pve-nas/`).
|
||||||
3. Record `zpool status` GUIDs and the by-id paths for **`ssd` and `tank`** — the
|
3. `zpool set cachefile=/etc/zfs/zpool.cache nvme`.
|
||||||
installer must never see these disks. 16 SATA devices stay untouched.
|
|
||||||
4. Confirm install media + out-of-band console access to the QNAP.
|
|
||||||
|
|
||||||
**Quiesce (start of window)**
|
**Build the ZFS root (live, no disruption)**
|
||||||
5. On **esh-docker-vm**: stop the containers touching NFS, then `umount` `/mnt/books`
|
4. `zfs create -o mountpoint=none nvme/ROOT`
|
||||||
and `/mnt/backup`. Do not skip — this is the wedge.
|
`zfs create -o mountpoint=/ -o canmount=noauto nvme/ROOT/pve-1`
|
||||||
6. On **esh-pve**: disable the `esh-nas` and `tank-vmbu` storages in the PVE UI
|
`canmount=noauto` is essential — otherwise ZFS tries to mount it over the
|
||||||
(Datacenter → Storage), so pvestatd stops probing them.
|
running root.
|
||||||
7. Shut down all five guests on esh-pve-nas.
|
5. Mount it somewhere temporary and `rsync -aHAX --numeric-ids` the live root
|
||||||
|
into it, excluding `/proc /sys /dev /run /mnt /nvme /ssd /tank /var/log/journal`.
|
||||||
|
6. In the copy: edit `/etc/fstab` to drop the root line, keep the ESP and swap
|
||||||
|
lines, and add the new `/boot` entry.
|
||||||
|
|
||||||
**Move guest rootfs off `nvme`**
|
**Split `/boot` onto the DOM**
|
||||||
8. `zfs send | zfs recv` each of the six datasets to `ssd` (~32 GB), or restore
|
7. Shrink/repartition is not needed — `/boot` can stay inside the DOM's existing
|
||||||
from PBS after the install. Sending is faster and does not depend on backups
|
LVM as its own small ext4 LV, or reuse the freed space once root moves off.
|
||||||
being complete.
|
Copy `/boot` into it and mount at `/boot`.
|
||||||
9. Update the guest configs to reference `ssd:` rather than `nvme:`.
|
|
||||||
|
|
||||||
**Rebuild**
|
**Bootloader**
|
||||||
10. `zpool export ssd tank` — clean export so the installer cannot disturb them.
|
8. `GRUB_CMDLINE_LINUX="root=ZFS=nvme/ROOT/pve-1"`, `update-initramfs -u -k all`,
|
||||||
11. Install PVE to **ZFS RAID1 on nvme0n1 + nvme1n1 only**. Same hostname, same
|
`update-grub`. **Keep the existing ext4 entry in the menu** and leave
|
||||||
IP, same PVE version family.
|
`GRUB_TIMEOUT=5` so it is selectable.
|
||||||
12. `zpool import ssd tank`; restore `/etc/pve/storage.cfg` entries; restore
|
|
||||||
`/etc/network/interfaces` verbatim (VLAN-aware bridge on `enp5s0f0`).
|
|
||||||
13. Restore guest configs; start CT 103 **first** — it is the NAS the others
|
|
||||||
need.
|
|
||||||
|
|
||||||
**Unquiesce**
|
**Cut over**
|
||||||
14. Re-enable storages on esh-pve; remount NFS on esh-docker-vm; restart its
|
9. Quiesce the NFS clients (see blast radius), shut down guests, reboot.
|
||||||
containers.
|
10. Verify (below). If it fails, reboot and choose the ext4 entry.
|
||||||
15. Physically remove or blank the DOM so it can never be booted from again.
|
|
||||||
|
|
||||||
## Verification
|
## Verification
|
||||||
|
|
||||||
- `proxmox-boot-tool status` lists **both** NVMe ESPs
|
- `findmnt -no SOURCE,FSTYPE /` → `nvme/ROOT/pve-1 zfs`
|
||||||
- root is on `rpool/ROOT/pve-1`, and `df -h /` shows hundreds of GB
|
- `df -h /` shows hundreds of GB, not 5.9
|
||||||
- all five guests running; `pct exec 103 -- exportfs -v` serving
|
- `findmnt /boot` → ext4 on the DOM; `/boot/efi` mounted
|
||||||
- esh-docker-vm: `/mnt/books` and `/mnt/backup` mounted, containers healthy
|
- all five guests running; CT 103 serving NFS (`pct exec 103 -- exportfs -v`)
|
||||||
- esh-pve: both NFS storages green in the UI
|
- esh-docker-vm remounted and healthy; esh-pve storages green
|
||||||
- a reboot test **before** declaring done — boot order is the thing being changed
|
- **a second reboot** to prove it was not a one-off
|
||||||
|
- only then: refresh the DOM image, since `/boot` has changed
|
||||||
|
|
||||||
## Rollback
|
## Rollback
|
||||||
|
|
||||||
Until step 11 everything is reversible: the DOM is untouched, so leaving the
|
Instant and cheap at every stage: the ext4 root on the DOM is never modified, and
|
||||||
NVMes alone and re-importing pools returns the old system. **After step 11 the
|
its GRUB entry stays in the menu. Worst case is a boot to initramfs → reboot →
|
||||||
old root is gone** — rollback becomes "reinstall and restore from the snapshot
|
pick the old entry. Keep the ext4 root for at least a few weeks of normal
|
||||||
plus PBS." Keep the DOM physically intact and unmodified until verification
|
operation before reclaiming it.
|
||||||
passes; it *is* the rollback.
|
|
||||||
|
|
||||||
## Open decisions
|
## Open decisions
|
||||||
|
|
||||||
- **Window length.** Realistically half a day, dominated by install and
|
- **Second boot device?** The split fixes runtime fragility but not boot-time
|
||||||
verification, not by the 32 GB move.
|
single-point-of-failure. A cloned DOM/USB as a cold spare is the cheap answer.
|
||||||
- **What to do with the freed DOM slot** — leave empty, or keep the DOM as a
|
- **`esh-filebot` (CT 106)** is an empty container — 80 GB quota, six passthrough
|
||||||
cold spare.
|
mounts, nothing running since March. Retire rather than carry it.
|
||||||
- **`esh-filebot` (CT 106)** is an empty container: 80 GB quota, six passthrough
|
- **Reclaiming the old ext4 root** once the ZFS root has proven itself.
|
||||||
mounts, nothing running since March. Migrating it is wasted work — decide
|
|
||||||
whether it comes across at all.
|
---
|
||||||
|
|
||||||
|
## Fallback plan: full reinstall to a mirrored-NVMe ZFS root
|
||||||
|
|
||||||
|
Only if the split above proves unworkable. Fresh PVE install to ZFS RAID1 across
|
||||||
|
both NVMes — mirrored boot with proper ESPs under `proxmox-boot-tool`, no USB in
|
||||||
|
the path at all.
|
||||||
|
|
||||||
|
Costs: the `nvme` pool must be destroyed, so its **32 GB of guest rootfs** moves
|
||||||
|
to `ssd` (1.42 T free) first; `ssd` and `tank` must be cleanly exported so the
|
||||||
|
installer cannot touch them; guest configs restore from the snapshot plus the 8
|
||||||
|
PBS backups per guest. Half a day, and rollback after the install step is
|
||||||
|
"reinstall and restore".
|
||||||
|
|
||||||
|
Note both NVMes are *whole-disk* ZFS members (partition 1 spans all 931.5 GiB,
|
||||||
|
1.7 MiB free), so adding an ESP to them without destroying the pool is
|
||||||
|
impossible — which is what forces the reinstall in this variant, and what the
|
||||||
|
split plan avoids entirely.
|
||||||
|
|||||||
Reference in New Issue
Block a user