docs(esh-pve-nas): plan the migration off the USB DOM; flag the NFS blast radius

PVE root on esh-pve-nas is a USB Disk-on-Module: 6 GB ext4 with the host's only
ESP. A DOM is SLC/pSLC so wear is not the driver -- the problems are that it is
on the USB bus (a reset drops root under a running hypervisor), has no headroom,
and is unmirrored while 928 GB of mirrored NVMe sits 96% empty.

Runbook targets a fresh PVE install to ZFS RAID1 across both NVMes. In-place
conversion is unsupported, and adding an ESP to the existing NVMes is impossible
-- both are whole-disk ZFS members with 1.7 MiB free and proxmox-boot-tool
manages nothing today.

The headline risk is not on the host being rebuilt: CT 103 esh-nas IS the NAS
at 10.0.50.50, and both esh-docker-vm and esh-pve mount it hard. Taking this box
down stalls esh-pve's storage layer and wedges esh-docker-vm into the D-state
whose only remedy is a host reboot -- the incident shape already on record.
Quiescing those clients is step one of the window, and the README now warns
against casual reboots.

Config snapshot captured off-box to nh3-dev (0600) with /etc/pve, network and
fstab config plus zpool/zfs/disk-by-id/guest state; the newest on-disk copy
before this was June 2024.

docs/runbooks/esh-pve-nas-boot-migration.md

@@ -0,0 +1,149 @@
+# esh-pve-nas — migrating PVE off the USB DOM
+
+**Status: PLANNED, not executed.** Mitigation landed 2026-08-17 (root 90% → 76%);
+this is the structural fix.
+
+## Why
+
+PVE root lives on a **USB Disk-on-Module** — `sdq`, 7.3 GB, `ID_BUS=usb`,
+`ID_VENDOR=NORELSYS`, model 1081 — carved into a 512 MB ESP, a 768 MB swap LV
+and a **6 GB ext4 root**.
+
+A DOM is not a thumb drive: SLC/pSLC NAND with a real controller, so the
+284 GB written since boot is unremarkable and **wear is not the driver**. What
+is:
+
+1. **It is on the USB bus.** NAND quality is irrelevant to a bus reset or a
+   re-enumeration event — either drops the root filesystem out from under a
+   running hypervisor while its guests keep executing.
+2. **6 GB has no headroom.** It hit 90% (571 MB free) with nothing pathological
+   on it — `/usr` alone is 3.7 GB. Tonight's mitigation bought 1.4 GB, not room
+   to grow.
+3. **It is single and unmirrored** — the one device the host cannot run without
+   has no redundancy, while 928 GB of *mirrored* NVMe sits 96% empty.
+
+## Current state (captured 2026-08-18)
+
+Off-box snapshot: `nh3-dev:/home/lkraven/backups/esh-pve-nas/pve-config-snapshot-20260818T043027Z.tar.gz`
+(0600, sha256 `dc312793d027dc43…`). Contains `/etc/pve`, `/etc/network/interfaces`,
+`/etc/fstab`, apt config, root's `authorized_keys`, plus captured live state:
+`zpool status/list/properties`, `zfs list`, `/dev/disk/by-id`, `lsblk` with
+serials + WWNs, `pvesm status`, `pveversion -v`, `dpkg --get-selections`, and
+every CT/VM config. **The newest on-disk copy before this was June 2024.**
+
+| | detail |
+|---|---|
+| node | `esh-nas-pve.esteban.net`, PVE 8.4.11, kernel 6.8.12-13 |
+| network | `vmbr0` static **10.0.50.55/24**, gw 10.0.50.1, bridge-port `enp5s0f0`, VLAN-aware 2-4094 |
+| `nvme` pool | mirror, 2× 931.5 GB NVMe — **32.1 G used, 867 G free, no snapshots** |
+| `ssd` pool | 2 mirrors, 4× 894 GB Intel SATA — 263 G used, 1.42 T free |
+| `tank` pool | 2× raidz2, 12× 14.6 TB — 40.2 T of 175 T |
+| guests | CT103 `esh-nas`, CT105 `vm-plex`, CT106 `esh-filebot`, CT107 `vm-jellyfin`, VM104 `vm-esh-nas` |
+| backups | 8 per guest on `pbs-ana` |
+
+**Every guest rootfs is on `nvme`, and nothing else is.** Media and data live on
+`tank` via LXC passthrough mounts. So the pool that must be cleared holds
+**~32 GB** — `ssd` absorbs that without noticing.
+
+## ⚠ The blast radius — read this before scheduling
+
+**CT 103 `esh-nas` (10.0.50.50) is the NAS, and it runs on the box being
+rebuilt.** Two dependents mount it over **`hard`** NFS, which means they do not
+fail — they hang, unkillably:
+
+| client | mounts | mode |
+|---|---|---|
+| **esh-docker-vm** (10.0.50.45) | `/mnt/books`, `/mnt/backup` | `hard` |
+| **esh-pve** (10.0.250.35) | `/mnt/pve/esh-nas`, `/mnt/pve/tank-vmbu` | `hard` |
+
+This is a known incident shape — hard NFS from 10.0.50.50 puts esh-docker-vm
+into unkillable D-state whose only remedy is a host reboot, and `/mnt/books`
+was *deliberately* left `hard` because calibre's SQLite would risk corruption
+under `soft`. There is a watchdog on esh-pve for exactly this.
+
+**Consequence: taking this host down stalls the other hypervisor's storage
+layer and wedges esh-docker-vm unless its NFS is quiesced first.** That is the
+single largest risk in this migration and it is not obvious from the host being
+worked on.
+
+## Target design (recommended)
+
+**Fresh PVE install to ZFS RAID1 across the two NVMes.** The installer builds
+`rpool` with proper ESPs on both devices, managed by `proxmox-boot-tool` —
+mirrored boot, no USB in the path, 900 GB of headroom. `rpool` then serves both
+the OS and the 32 GB of guest rootfs, collapsing the separate `nvme` pool.
+
+Why not the alternatives:
+
+- **In-place conversion** (LVM-ext4 root → ZFS root) is not a supported path and
+  fails into a rebuild anyway.
+- **Add an ESP to the existing NVMes** is impossible: both are *whole-disk* ZFS
+  members (partition 1 spans all 931.5 GiB, type BF01) with **1.7 MiB free**.
+  `proxmox-boot-tool status` confirms nothing is managed — the DOM's ESP is the
+  only boot path.
+- **Install to 2 of the 4 Intel SSDs** works but destroys one `ssd` mirror and
+  spends 894 GB on a boot device.
+- **Replace the DOM** leaves every structural problem intact.
+
+## Sequence
+
+**Pre-flight (no downtime)**
+1. Verify PBS backups are current for all five guests; take fresh ones if not.
+2. Re-pull the config snapshot so it is same-day.
+3. Record `zpool status` GUIDs and the by-id paths for **`ssd` and `tank`** — the
+   installer must never see these disks. 16 SATA devices stay untouched.
+4. Confirm install media + out-of-band console access to the QNAP.
+
+**Quiesce (start of window)**
+5. On **esh-docker-vm**: stop the containers touching NFS, then `umount` `/mnt/books`
+   and `/mnt/backup`. Do not skip — this is the wedge.
+6. On **esh-pve**: disable the `esh-nas` and `tank-vmbu` storages in the PVE UI
+   (Datacenter → Storage), so pvestatd stops probing them.
+7. Shut down all five guests on esh-pve-nas.
+
+**Move guest rootfs off `nvme`**
+8. `zfs send | zfs recv` each of the six datasets to `ssd` (~32 GB), or restore
+   from PBS after the install. Sending is faster and does not depend on backups
+   being complete.
+9. Update the guest configs to reference `ssd:` rather than `nvme:`.
+
+**Rebuild**
+10. `zpool export ssd tank` — clean export so the installer cannot disturb them.
+11. Install PVE to **ZFS RAID1 on nvme0n1 + nvme1n1 only**. Same hostname, same
+    IP, same PVE version family.
+12. `zpool import ssd tank`; restore `/etc/pve/storage.cfg` entries; restore
+    `/etc/network/interfaces` verbatim (VLAN-aware bridge on `enp5s0f0`).
+13. Restore guest configs; start CT 103 **first** — it is the NAS the others
+    need.
+
+**Unquiesce**
+14. Re-enable storages on esh-pve; remount NFS on esh-docker-vm; restart its
+    containers.
+15. Physically remove or blank the DOM so it can never be booted from again.
+
+## Verification
+
+- `proxmox-boot-tool status` lists **both** NVMe ESPs
+- root is on `rpool/ROOT/pve-1`, and `df -h /` shows hundreds of GB
+- all five guests running; `pct exec 103 -- exportfs -v` serving
+- esh-docker-vm: `/mnt/books` and `/mnt/backup` mounted, containers healthy
+- esh-pve: both NFS storages green in the UI
+- a reboot test **before** declaring done — boot order is the thing being changed
+
+## Rollback
+
+Until step 11 everything is reversible: the DOM is untouched, so leaving the
+NVMes alone and re-importing pools returns the old system. **After step 11 the
+old root is gone** — rollback becomes "reinstall and restore from the snapshot
+plus PBS." Keep the DOM physically intact and unmodified until verification
+passes; it *is* the rollback.
+
+## Open decisions
+
+- **Window length.** Realistically half a day, dominated by install and
+  verification, not by the 32 GB move.
+- **What to do with the freed DOM slot** — leave empty, or keep the DOM as a
+  cold spare.
+- **`esh-filebot` (CT 106)** is an empty container: 80 GB quota, six passthrough
+  mounts, nothing running since March. Migrating it is wasted work — decide
+  whether it comes across at all.

servers/esh-pve-nas/README.md

@@ -16,7 +16,20 @@ Media services on this box sit at `10.0.50.56` (Plex) and `10.0.50.57` (Jellyfin
 - **CPU:** Intel Xeon W-1250 @ 3.30 GHz
 - **RAM:** 125.6 GB
 - **Kernel:** `6.8.12-13-pve` (Proxmox 8.x)
-- **Storage:** local `pve-root` is tiny (5.9 GB, **87% used — worth watching**) + NFS `/mnt/pve/tank-vmbu` (93 TB) for VM backups
+- **Storage:** ⚠ **PVE root is on a USB Disk-on-Module** (`sdq`, 7.3 GB, `ID_BUS=usb`,
+  NORELSYS 1081) — 6 GB ext4 root + 768 MB swap + a 512 MB ESP that is the host's
+  **only** boot path. Not a wear problem (a DOM is SLC/pSLC), but it is on the USB
+  bus, has no headroom, and is unmirrored. Mitigated 2026-08-17 (90% → 76%: journald
+  capped, journal relocated to ZFS); migration planned in
+  [`docs/runbooks/esh-pve-nas-boot-migration.md`](../../docs/runbooks/esh-pve-nas-boot-migration.md).
+- **Pools:** `nvme` (2× 931 GB NVMe mirror — 32 G used, 867 G free, holds every guest
+  rootfs), `ssd` (4× 894 GB Intel SATA, 2 mirrors — 1.42 T free), `tank`
+  (12× 14.6 TB raidz2 ×2 — 40 T of 175 T). Plus NFS `/mnt/pve/tank-vmbu` for VM backups.
+- ⚠ **CT 103 `esh-nas` (10.0.50.50) runs on THIS host and serves `hard` NFS** to
+  esh-docker-vm (`/mnt/books`, `/mnt/backup`) and to esh-pve (`/mnt/pve/esh-nas`,
+  `/mnt/pve/tank-vmbu`). Taking this box down stalls esh-pve's storage layer and
+  wedges esh-docker-vm into unkillable D-state unless those clients are quiesced
+  first. Never reboot this host casually.
 
 > **Root-fs pressure:** at 87% used on a 5.9 GB root partition, there's not much room for package upgrades or logs. Worth cleaning up or growing the root if this host is staying in production for a while.