From 2275e11be0b24cfe6101b8ce8afe46f403608f33 Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Mon, 17 Aug 2026 21:32:33 -0700 Subject: [PATCH] docs(esh-pve-nas): plan the migration off the USB DOM; flag the NFS blast radius PVE root on esh-pve-nas is a USB Disk-on-Module: 6 GB ext4 with the host's only ESP. A DOM is SLC/pSLC so wear is not the driver -- the problems are that it is on the USB bus (a reset drops root under a running hypervisor), has no headroom, and is unmirrored while 928 GB of mirrored NVMe sits 96% empty. Runbook targets a fresh PVE install to ZFS RAID1 across both NVMes. In-place conversion is unsupported, and adding an ESP to the existing NVMes is impossible -- both are whole-disk ZFS members with 1.7 MiB free and proxmox-boot-tool manages nothing today. The headline risk is not on the host being rebuilt: CT 103 esh-nas IS the NAS at 10.0.50.50, and both esh-docker-vm and esh-pve mount it hard. Taking this box down stalls esh-pve's storage layer and wedges esh-docker-vm into the D-state whose only remedy is a host reboot -- the incident shape already on record. Quiescing those clients is step one of the window, and the README now warns against casual reboots. Config snapshot captured off-box to nh3-dev (0600) with /etc/pve, network and fstab config plus zpool/zfs/disk-by-id/guest state; the newest on-disk copy before this was June 2024. --- docs/runbooks/esh-pve-nas-boot-migration.md | 149 ++++++++++++++++++++ servers/esh-pve-nas/README.md | 15 +- 2 files changed, 163 insertions(+), 1 deletion(-) create mode 100644 docs/runbooks/esh-pve-nas-boot-migration.md diff --git a/docs/runbooks/esh-pve-nas-boot-migration.md b/docs/runbooks/esh-pve-nas-boot-migration.md new file mode 100644 index 0000000..cc1a81b --- /dev/null +++ b/docs/runbooks/esh-pve-nas-boot-migration.md @@ -0,0 +1,149 @@ +# esh-pve-nas — migrating PVE off the USB DOM + +**Status: PLANNED, not executed.** Mitigation landed 2026-08-17 (root 90% → 76%); +this is the structural fix. + +## Why + +PVE root lives on a **USB Disk-on-Module** — `sdq`, 7.3 GB, `ID_BUS=usb`, +`ID_VENDOR=NORELSYS`, model 1081 — carved into a 512 MB ESP, a 768 MB swap LV +and a **6 GB ext4 root**. + +A DOM is not a thumb drive: SLC/pSLC NAND with a real controller, so the +284 GB written since boot is unremarkable and **wear is not the driver**. What +is: + +1. **It is on the USB bus.** NAND quality is irrelevant to a bus reset or a + re-enumeration event — either drops the root filesystem out from under a + running hypervisor while its guests keep executing. +2. **6 GB has no headroom.** It hit 90% (571 MB free) with nothing pathological + on it — `/usr` alone is 3.7 GB. Tonight's mitigation bought 1.4 GB, not room + to grow. +3. **It is single and unmirrored** — the one device the host cannot run without + has no redundancy, while 928 GB of *mirrored* NVMe sits 96% empty. + +## Current state (captured 2026-08-18) + +Off-box snapshot: `nh3-dev:/home/lkraven/backups/esh-pve-nas/pve-config-snapshot-20260818T043027Z.tar.gz` +(0600, sha256 `dc312793d027dc43…`). Contains `/etc/pve`, `/etc/network/interfaces`, +`/etc/fstab`, apt config, root's `authorized_keys`, plus captured live state: +`zpool status/list/properties`, `zfs list`, `/dev/disk/by-id`, `lsblk` with +serials + WWNs, `pvesm status`, `pveversion -v`, `dpkg --get-selections`, and +every CT/VM config. **The newest on-disk copy before this was June 2024.** + +| | detail | +|---|---| +| node | `esh-nas-pve.esteban.net`, PVE 8.4.11, kernel 6.8.12-13 | +| network | `vmbr0` static **10.0.50.55/24**, gw 10.0.50.1, bridge-port `enp5s0f0`, VLAN-aware 2-4094 | +| `nvme` pool | mirror, 2× 931.5 GB NVMe — **32.1 G used, 867 G free, no snapshots** | +| `ssd` pool | 2 mirrors, 4× 894 GB Intel SATA — 263 G used, 1.42 T free | +| `tank` pool | 2× raidz2, 12× 14.6 TB — 40.2 T of 175 T | +| guests | CT103 `esh-nas`, CT105 `vm-plex`, CT106 `esh-filebot`, CT107 `vm-jellyfin`, VM104 `vm-esh-nas` | +| backups | 8 per guest on `pbs-ana` | + +**Every guest rootfs is on `nvme`, and nothing else is.** Media and data live on +`tank` via LXC passthrough mounts. So the pool that must be cleared holds +**~32 GB** — `ssd` absorbs that without noticing. + +## ⚠ The blast radius — read this before scheduling + +**CT 103 `esh-nas` (10.0.50.50) is the NAS, and it runs on the box being +rebuilt.** Two dependents mount it over **`hard`** NFS, which means they do not +fail — they hang, unkillably: + +| client | mounts | mode | +|---|---|---| +| **esh-docker-vm** (10.0.50.45) | `/mnt/books`, `/mnt/backup` | `hard` | +| **esh-pve** (10.0.250.35) | `/mnt/pve/esh-nas`, `/mnt/pve/tank-vmbu` | `hard` | + +This is a known incident shape — hard NFS from 10.0.50.50 puts esh-docker-vm +into unkillable D-state whose only remedy is a host reboot, and `/mnt/books` +was *deliberately* left `hard` because calibre's SQLite would risk corruption +under `soft`. There is a watchdog on esh-pve for exactly this. + +**Consequence: taking this host down stalls the other hypervisor's storage +layer and wedges esh-docker-vm unless its NFS is quiesced first.** That is the +single largest risk in this migration and it is not obvious from the host being +worked on. + +## Target design (recommended) + +**Fresh PVE install to ZFS RAID1 across the two NVMes.** The installer builds +`rpool` with proper ESPs on both devices, managed by `proxmox-boot-tool` — +mirrored boot, no USB in the path, 900 GB of headroom. `rpool` then serves both +the OS and the 32 GB of guest rootfs, collapsing the separate `nvme` pool. + +Why not the alternatives: + +- **In-place conversion** (LVM-ext4 root → ZFS root) is not a supported path and + fails into a rebuild anyway. +- **Add an ESP to the existing NVMes** is impossible: both are *whole-disk* ZFS + members (partition 1 spans all 931.5 GiB, type BF01) with **1.7 MiB free**. + `proxmox-boot-tool status` confirms nothing is managed — the DOM's ESP is the + only boot path. +- **Install to 2 of the 4 Intel SSDs** works but destroys one `ssd` mirror and + spends 894 GB on a boot device. +- **Replace the DOM** leaves every structural problem intact. + +## Sequence + +**Pre-flight (no downtime)** +1. Verify PBS backups are current for all five guests; take fresh ones if not. +2. Re-pull the config snapshot so it is same-day. +3. Record `zpool status` GUIDs and the by-id paths for **`ssd` and `tank`** — the + installer must never see these disks. 16 SATA devices stay untouched. +4. Confirm install media + out-of-band console access to the QNAP. + +**Quiesce (start of window)** +5. On **esh-docker-vm**: stop the containers touching NFS, then `umount` `/mnt/books` + and `/mnt/backup`. Do not skip — this is the wedge. +6. On **esh-pve**: disable the `esh-nas` and `tank-vmbu` storages in the PVE UI + (Datacenter → Storage), so pvestatd stops probing them. +7. Shut down all five guests on esh-pve-nas. + +**Move guest rootfs off `nvme`** +8. `zfs send | zfs recv` each of the six datasets to `ssd` (~32 GB), or restore + from PBS after the install. Sending is faster and does not depend on backups + being complete. +9. Update the guest configs to reference `ssd:` rather than `nvme:`. + +**Rebuild** +10. `zpool export ssd tank` — clean export so the installer cannot disturb them. +11. Install PVE to **ZFS RAID1 on nvme0n1 + nvme1n1 only**. Same hostname, same + IP, same PVE version family. +12. `zpool import ssd tank`; restore `/etc/pve/storage.cfg` entries; restore + `/etc/network/interfaces` verbatim (VLAN-aware bridge on `enp5s0f0`). +13. Restore guest configs; start CT 103 **first** — it is the NAS the others + need. + +**Unquiesce** +14. Re-enable storages on esh-pve; remount NFS on esh-docker-vm; restart its + containers. +15. Physically remove or blank the DOM so it can never be booted from again. + +## Verification + +- `proxmox-boot-tool status` lists **both** NVMe ESPs +- root is on `rpool/ROOT/pve-1`, and `df -h /` shows hundreds of GB +- all five guests running; `pct exec 103 -- exportfs -v` serving +- esh-docker-vm: `/mnt/books` and `/mnt/backup` mounted, containers healthy +- esh-pve: both NFS storages green in the UI +- a reboot test **before** declaring done — boot order is the thing being changed + +## Rollback + +Until step 11 everything is reversible: the DOM is untouched, so leaving the +NVMes alone and re-importing pools returns the old system. **After step 11 the +old root is gone** — rollback becomes "reinstall and restore from the snapshot +plus PBS." Keep the DOM physically intact and unmodified until verification +passes; it *is* the rollback. + +## Open decisions + +- **Window length.** Realistically half a day, dominated by install and + verification, not by the 32 GB move. +- **What to do with the freed DOM slot** — leave empty, or keep the DOM as a + cold spare. +- **`esh-filebot` (CT 106)** is an empty container: 80 GB quota, six passthrough + mounts, nothing running since March. Migrating it is wasted work — decide + whether it comes across at all. diff --git a/servers/esh-pve-nas/README.md b/servers/esh-pve-nas/README.md index 230af4c..3f97bcf 100644 --- a/servers/esh-pve-nas/README.md +++ b/servers/esh-pve-nas/README.md @@ -16,7 +16,20 @@ Media services on this box sit at `10.0.50.56` (Plex) and `10.0.50.57` (Jellyfin - **CPU:** Intel Xeon W-1250 @ 3.30 GHz - **RAM:** 125.6 GB - **Kernel:** `6.8.12-13-pve` (Proxmox 8.x) -- **Storage:** local `pve-root` is tiny (5.9 GB, **87% used — worth watching**) + NFS `/mnt/pve/tank-vmbu` (93 TB) for VM backups +- **Storage:** ⚠ **PVE root is on a USB Disk-on-Module** (`sdq`, 7.3 GB, `ID_BUS=usb`, + NORELSYS 1081) — 6 GB ext4 root + 768 MB swap + a 512 MB ESP that is the host's + **only** boot path. Not a wear problem (a DOM is SLC/pSLC), but it is on the USB + bus, has no headroom, and is unmirrored. Mitigated 2026-08-17 (90% → 76%: journald + capped, journal relocated to ZFS); migration planned in + [`docs/runbooks/esh-pve-nas-boot-migration.md`](../../docs/runbooks/esh-pve-nas-boot-migration.md). +- **Pools:** `nvme` (2× 931 GB NVMe mirror — 32 G used, 867 G free, holds every guest + rootfs), `ssd` (4× 894 GB Intel SATA, 2 mirrors — 1.42 T free), `tank` + (12× 14.6 TB raidz2 ×2 — 40 T of 175 T). Plus NFS `/mnt/pve/tank-vmbu` for VM backups. +- ⚠ **CT 103 `esh-nas` (10.0.50.50) runs on THIS host and serves `hard` NFS** to + esh-docker-vm (`/mnt/books`, `/mnt/backup`) and to esh-pve (`/mnt/pve/esh-nas`, + `/mnt/pve/tank-vmbu`). Taking this box down stalls esh-pve's storage layer and + wedges esh-docker-vm into unkillable D-state unless those clients are quiesced + first. Never reboot this host casually. > **Root-fs pressure:** at 87% used on a 5.9 GB root partition, there's not much room for package upgrades or logs. Worth cleaning up or growing the root if this host is staying in production for a while.