diff --git a/docs/runbooks/esh-pve-nas-boot-migration.md b/docs/runbooks/esh-pve-nas-boot-migration.md index cc1a81b..10da43d 100644 --- a/docs/runbooks/esh-pve-nas-boot-migration.md +++ b/docs/runbooks/esh-pve-nas-boot-migration.md @@ -1,149 +1,168 @@ -# esh-pve-nas — migrating PVE off the USB DOM +# esh-pve-nas — moving PVE root off the USB DOM -**Status: PLANNED, not executed.** Mitigation landed 2026-08-17 (root 90% → 76%); -this is the structural fix. +**Status: PLANNED, not executed.** Mitigation landed 2026-08-17 (root 90% → 76%). + +**Plan revised 2026-08-17** from "reinstall to a mirrored-NVMe ZFS root" to +**"split the boot chain from the root filesystem"** — operator's proposal, and it +is strictly better. The original reinstall plan is kept at the bottom as the +fallback. ## Why PVE root lives on a **USB Disk-on-Module** — `sdq`, 7.3 GB, `ID_BUS=usb`, -`ID_VENDOR=NORELSYS`, model 1081 — carved into a 512 MB ESP, a 768 MB swap LV -and a **6 GB ext4 root**. +`ID_VENDOR=NORELSYS` — as a 6 GB ext4 root plus 768 MB swap and a 512 MB ESP. -A DOM is not a thumb drive: SLC/pSLC NAND with a real controller, so the -284 GB written since boot is unremarkable and **wear is not the driver**. What -is: +A DOM is SLC/pSLC with a real controller, so the 284 GB written since boot is +unremarkable and **wear is not the driver**. The actual problems: -1. **It is on the USB bus.** NAND quality is irrelevant to a bus reset or a - re-enumeration event — either drops the root filesystem out from under a - running hypervisor while its guests keep executing. -2. **6 GB has no headroom.** It hit 90% (571 MB free) with nothing pathological - on it — `/usr` alone is 3.7 GB. Tonight's mitigation bought 1.4 GB, not room - to grow. -3. **It is single and unmirrored** — the one device the host cannot run without - has no redundancy, while 928 GB of *mirrored* NVMe sits 96% empty. +1. **It is on the USB bus.** A bus reset or re-enumeration drops the *root + filesystem* out from under a running hypervisor while its guests keep going. +2. **6 GB has no headroom** — `/usr` alone is 3.7 GB. +3. **Unmirrored**, while 928 GB of mirrored NVMe sits 96% empty. -## Current state (captured 2026-08-18) +## The design: boot on the DOM, root on ZFS -Off-box snapshot: `nh3-dev:/home/lkraven/backups/esh-pve-nas/pve-config-snapshot-20260818T043027Z.tar.gz` -(0600, sha256 `dc312793d027dc43…`). Contains `/etc/pve`, `/etc/network/interfaces`, -`/etc/fstab`, apt config, root's `authorized_keys`, plus captured live state: -`zpool status/list/properties`, `zfs list`, `/dev/disk/by-id`, `lsblk` with -serials + WWNs, `pvesm status`, `pveversion -v`, `dpkg --get-selections`, and -every CT/VM config. **The newest on-disk copy before this was June 2024.** +Boot and root do not have to live on the same device. Split them: -| | detail | +| | device | contents | written when | +|---|---|---|---| +| **boot** | DOM `sdq` | ESP + `/boot` (ext4): GRUB, kernels, initramfs | **only on kernel/GRUB updates** | +| **root** | `nvme` pool | `nvme/ROOT/pve-1` — everything else | constantly, on mirrored NVMe | + +GRUB reads the kernel and initrd from **ext4 on the DOM**, so GRUB never has to +read ZFS — which matters, because the `nvme` pool has `encryption`, +`large_dnode` and `zstd_compress` enabled and GRUB cannot read those. The +initramfs then imports the pool and pivots to `root=ZFS=nvme/ROOT/pve-1`. + +### Why this beats the reinstall + +- **The `nvme` pool is not destroyed.** The root dataset is created *inside* the + existing pool. No guest migration, no `zpool export/import` of `ssd`/`tank`, + no reinstall. +- **Downtime is one reboot**, not half a day. +- **Rollback is a GRUB menu entry.** The existing ext4 root stays on the DOM, + untouched. If ZFS root fails to come up, pick the old entry and you are back in + a minute. That is a far better rollback than "reinstall and restore." +- **The #1 risk is actually retired.** Once booted, root is on NVMe — a USB bus + reset mid-run no longer takes the running system down. The DOM becomes + read-mostly. +- **Free upside: boot environments.** `zfs snapshot nvme/ROOT/pve-1@pre-upgrade` + before an apt run, roll back if it breaks. + +### What it does NOT fix + +The DOM remains the **only boot path**. If it dies, the machine will not boot +until the image is restored — though the ZFS root, with all config and guests, +stays intact. Mitigation is a **cloned fallback image** (`dd` of `sdq`, ~7 GB, +refreshed after kernel updates), kept off-box next to the config snapshot. + +## Preconditions — all already satisfied + +Verified on the host 2026-08-17: + +- **UEFI** firmware, `grub-efi-amd64 2.06-13+pmx7` installed +- **`zfs-initramfs 2.2.8-pve1` is already installed**, and the running initrd + already carries **76 ZFS files** — the pivot capability exists today, no new + packages +- `/boot` is currently *part of* root (108 MB), so it must be split out onto its + own ext4 filesystem on the DOM as part of this work +- root is only **4.3 GB** to copy +- swap is 767 MB with 123 MB used against 125 GB of RAM — irrelevant; leave it + on the DOM LV. **Do not put swap on a zvol** (deadlock risk) + +⚠ **`cachefile` is `none` and `/etc/zfs/zpool.cache` is 0 bytes** — pools import +by scan today. For root-on-ZFS this must be deterministic: set +`zpool set cachefile=/etc/zfs/zpool.cache nvme` and rebuild the initramfs, or the +pool may not be imported early enough to find root. + +## ⚠ Blast radius — unchanged, and still the gating constraint + +**CT 103 `esh-nas` (10.0.50.50) is the NAS, and it runs on this host.** Two +dependents mount it over **`hard`** NFS — they do not fail, they hang unkillably: + +| client | mounts | |---|---| -| node | `esh-nas-pve.esteban.net`, PVE 8.4.11, kernel 6.8.12-13 | -| network | `vmbr0` static **10.0.50.55/24**, gw 10.0.50.1, bridge-port `enp5s0f0`, VLAN-aware 2-4094 | -| `nvme` pool | mirror, 2× 931.5 GB NVMe — **32.1 G used, 867 G free, no snapshots** | -| `ssd` pool | 2 mirrors, 4× 894 GB Intel SATA — 263 G used, 1.42 T free | -| `tank` pool | 2× raidz2, 12× 14.6 TB — 40.2 T of 175 T | -| guests | CT103 `esh-nas`, CT105 `vm-plex`, CT106 `esh-filebot`, CT107 `vm-jellyfin`, VM104 `vm-esh-nas` | -| backups | 8 per guest on `pbs-ana` | +| **esh-docker-vm** (10.0.50.45) | `/mnt/books`, `/mnt/backup` | +| **esh-pve** (10.0.250.35) | `/mnt/pve/esh-nas`, `/mnt/pve/tank-vmbu` | -**Every guest rootfs is on `nvme`, and nothing else is.** Media and data live on -`tank` via LXC passthrough mounts. So the pool that must be cleared holds -**~32 GB** — `ssd` absorbs that without noticing. +This is a known incident shape: the only remedy for esh-docker-vm's D-state is a +host reboot, and `/mnt/books` was *deliberately* left `hard` because calibre's +SQLite risks corruption under `soft`. -## ⚠ The blast radius — read this before scheduling - -**CT 103 `esh-nas` (10.0.50.50) is the NAS, and it runs on the box being -rebuilt.** Two dependents mount it over **`hard`** NFS, which means they do not -fail — they hang, unkillably: - -| client | mounts | mode | -|---|---|---| -| **esh-docker-vm** (10.0.50.45) | `/mnt/books`, `/mnt/backup` | `hard` | -| **esh-pve** (10.0.250.35) | `/mnt/pve/esh-nas`, `/mnt/pve/tank-vmbu` | `hard` | - -This is a known incident shape — hard NFS from 10.0.50.50 puts esh-docker-vm -into unkillable D-state whose only remedy is a host reboot, and `/mnt/books` -was *deliberately* left `hard` because calibre's SQLite would risk corruption -under `soft`. There is a watchdog on esh-pve for exactly this. - -**Consequence: taking this host down stalls the other hypervisor's storage -layer and wedges esh-docker-vm unless its NFS is quiesced first.** That is the -single largest risk in this migration and it is not obvious from the host being -worked on. - -## Target design (recommended) - -**Fresh PVE install to ZFS RAID1 across the two NVMes.** The installer builds -`rpool` with proper ESPs on both devices, managed by `proxmox-boot-tool` — -mirrored boot, no USB in the path, 900 GB of headroom. `rpool` then serves both -the OS and the 32 GB of guest rootfs, collapsing the separate `nvme` pool. - -Why not the alternatives: - -- **In-place conversion** (LVM-ext4 root → ZFS root) is not a supported path and - fails into a rebuild anyway. -- **Add an ESP to the existing NVMes** is impossible: both are *whole-disk* ZFS - members (partition 1 spans all 931.5 GiB, type BF01) with **1.7 MiB free**. - `proxmox-boot-tool status` confirms nothing is managed — the DOM's ESP is the - only boot path. -- **Install to 2 of the 4 Intel SSDs** works but destroys one `ssd` mirror and - spends 894 GB on a boot device. -- **Replace the DOM** leaves every structural problem intact. +The reboot in this plan is brief, but it is still a reboot — quiesce the clients +first. ## Sequence **Pre-flight (no downtime)** -1. Verify PBS backups are current for all five guests; take fresh ones if not. -2. Re-pull the config snapshot so it is same-day. -3. Record `zpool status` GUIDs and the by-id paths for **`ssd` and `tank`** — the - installer must never see these disks. 16 SATA devices stay untouched. -4. Confirm install media + out-of-band console access to the QNAP. +1. `dd` the DOM to an image, off-box. This is the rollback for the boot chain. +2. Refresh the config snapshot (`nh3-dev:~/backups/esh-pve-nas/`). +3. `zpool set cachefile=/etc/zfs/zpool.cache nvme`. -**Quiesce (start of window)** -5. On **esh-docker-vm**: stop the containers touching NFS, then `umount` `/mnt/books` - and `/mnt/backup`. Do not skip — this is the wedge. -6. On **esh-pve**: disable the `esh-nas` and `tank-vmbu` storages in the PVE UI - (Datacenter → Storage), so pvestatd stops probing them. -7. Shut down all five guests on esh-pve-nas. +**Build the ZFS root (live, no disruption)** +4. `zfs create -o mountpoint=none nvme/ROOT` + `zfs create -o mountpoint=/ -o canmount=noauto nvme/ROOT/pve-1` + `canmount=noauto` is essential — otherwise ZFS tries to mount it over the + running root. +5. Mount it somewhere temporary and `rsync -aHAX --numeric-ids` the live root + into it, excluding `/proc /sys /dev /run /mnt /nvme /ssd /tank /var/log/journal`. +6. In the copy: edit `/etc/fstab` to drop the root line, keep the ESP and swap + lines, and add the new `/boot` entry. -**Move guest rootfs off `nvme`** -8. `zfs send | zfs recv` each of the six datasets to `ssd` (~32 GB), or restore - from PBS after the install. Sending is faster and does not depend on backups - being complete. -9. Update the guest configs to reference `ssd:` rather than `nvme:`. +**Split `/boot` onto the DOM** +7. Shrink/repartition is not needed — `/boot` can stay inside the DOM's existing + LVM as its own small ext4 LV, or reuse the freed space once root moves off. + Copy `/boot` into it and mount at `/boot`. -**Rebuild** -10. `zpool export ssd tank` — clean export so the installer cannot disturb them. -11. Install PVE to **ZFS RAID1 on nvme0n1 + nvme1n1 only**. Same hostname, same - IP, same PVE version family. -12. `zpool import ssd tank`; restore `/etc/pve/storage.cfg` entries; restore - `/etc/network/interfaces` verbatim (VLAN-aware bridge on `enp5s0f0`). -13. Restore guest configs; start CT 103 **first** — it is the NAS the others - need. +**Bootloader** +8. `GRUB_CMDLINE_LINUX="root=ZFS=nvme/ROOT/pve-1"`, `update-initramfs -u -k all`, + `update-grub`. **Keep the existing ext4 entry in the menu** and leave + `GRUB_TIMEOUT=5` so it is selectable. -**Unquiesce** -14. Re-enable storages on esh-pve; remount NFS on esh-docker-vm; restart its - containers. -15. Physically remove or blank the DOM so it can never be booted from again. +**Cut over** +9. Quiesce the NFS clients (see blast radius), shut down guests, reboot. +10. Verify (below). If it fails, reboot and choose the ext4 entry. ## Verification -- `proxmox-boot-tool status` lists **both** NVMe ESPs -- root is on `rpool/ROOT/pve-1`, and `df -h /` shows hundreds of GB -- all five guests running; `pct exec 103 -- exportfs -v` serving -- esh-docker-vm: `/mnt/books` and `/mnt/backup` mounted, containers healthy -- esh-pve: both NFS storages green in the UI -- a reboot test **before** declaring done — boot order is the thing being changed +- `findmnt -no SOURCE,FSTYPE /` → `nvme/ROOT/pve-1 zfs` +- `df -h /` shows hundreds of GB, not 5.9 +- `findmnt /boot` → ext4 on the DOM; `/boot/efi` mounted +- all five guests running; CT 103 serving NFS (`pct exec 103 -- exportfs -v`) +- esh-docker-vm remounted and healthy; esh-pve storages green +- **a second reboot** to prove it was not a one-off +- only then: refresh the DOM image, since `/boot` has changed ## Rollback -Until step 11 everything is reversible: the DOM is untouched, so leaving the -NVMes alone and re-importing pools returns the old system. **After step 11 the -old root is gone** — rollback becomes "reinstall and restore from the snapshot -plus PBS." Keep the DOM physically intact and unmodified until verification -passes; it *is* the rollback. +Instant and cheap at every stage: the ext4 root on the DOM is never modified, and +its GRUB entry stays in the menu. Worst case is a boot to initramfs → reboot → +pick the old entry. Keep the ext4 root for at least a few weeks of normal +operation before reclaiming it. ## Open decisions -- **Window length.** Realistically half a day, dominated by install and - verification, not by the 32 GB move. -- **What to do with the freed DOM slot** — leave empty, or keep the DOM as a - cold spare. -- **`esh-filebot` (CT 106)** is an empty container: 80 GB quota, six passthrough - mounts, nothing running since March. Migrating it is wasted work — decide - whether it comes across at all. +- **Second boot device?** The split fixes runtime fragility but not boot-time + single-point-of-failure. A cloned DOM/USB as a cold spare is the cheap answer. +- **`esh-filebot` (CT 106)** is an empty container — 80 GB quota, six passthrough + mounts, nothing running since March. Retire rather than carry it. +- **Reclaiming the old ext4 root** once the ZFS root has proven itself. + +--- + +## Fallback plan: full reinstall to a mirrored-NVMe ZFS root + +Only if the split above proves unworkable. Fresh PVE install to ZFS RAID1 across +both NVMes — mirrored boot with proper ESPs under `proxmox-boot-tool`, no USB in +the path at all. + +Costs: the `nvme` pool must be destroyed, so its **32 GB of guest rootfs** moves +to `ssd` (1.42 T free) first; `ssd` and `tank` must be cleanly exported so the +installer cannot touch them; guest configs restore from the snapshot plus the 8 +PBS backups per guest. Half a day, and rollback after the install step is +"reinstall and restore". + +Note both NVMes are *whole-disk* ZFS members (partition 1 spans all 931.5 GiB, +1.7 MiB free), so adding an ESP to them without destroying the pool is +impossible — which is what forces the reinstall in this variant, and what the +split plan avoids entirely.