vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity

docs(esh-pve-nas): plan the migration off the USB DOM; flag the NFS blast radius

Browse Source 
PVE root on esh-pve-nas is a USB Disk-on-Module: 6 GB ext4 with the host's only
ESP. A DOM is SLC/pSLC so wear is not the driver -- the problems are that it is
on the USB bus (a reset drops root under a running hypervisor), has no headroom,
and is unmirrored while 928 GB of mirrored NVMe sits 96% empty.

Runbook targets a fresh PVE install to ZFS RAID1 across both NVMes. In-place
conversion is unsupported, and adding an ESP to the existing NVMes is impossible
-- both are whole-disk ZFS members with 1.7 MiB free and proxmox-boot-tool
manages nothing today.

The headline risk is not on the host being rebuilt: CT 103 esh-nas IS the NAS
at 10.0.50.50, and both esh-docker-vm and esh-pve mount it hard. Taking this box
down stalls esh-pve's storage layer and wedges esh-docker-vm into the D-state
whose only remedy is a host reboot -- the incident shape already on record.
Quiescing those clients is step one of the window, and the README now warns
against casual reboots.

Config snapshot captured off-box to nh3-dev (0600) with /etc/pve, network and
fstab config plus zpool/zfs/disk-by-id/guest state; the newest on-disk copy
before this was June 2024.
This commit is contained in:
vh
2026-08-17 21:32:33 -07:00
parent ca8c0a318e
commit 2275e11be0
2 changed files with 163 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files
servers/esh-pve-nas/README.md
+14 -1
View File
@@ -16,7 +16,20 @@ Media services on this box sit at `10.0.50.56` (Plex) and `10.0.50.57` (Jellyfin
- **CPU:** Intel Xeon W-1250 @ 3.30 GHz
- **RAM:** 125.6 GB
- **Kernel:** `6.8.12-13-pve` (Proxmox 8.x)
- **Storage:** local `pve-root` is tiny (5.9 GB, **87% used — worth watching**) + NFS `/mnt/pve/tank-vmbu` (93 TB) for VM backups
- **Storage:** ⚠ **PVE root is on a USB Disk-on-Module** (`sdq`, 7.3 GB, `ID_BUS=usb`,
NORELSYS 1081) — 6 GB ext4 root + 768 MB swap + a 512 MB ESP that is the host's
**only** boot path. Not a wear problem (a DOM is SLC/pSLC), but it is on the USB
bus, has no headroom, and is unmirrored. Mitigated 2026-08-17 (90% → 76%: journald
capped, journal relocated to ZFS); migration planned in
[`docs/runbooks/esh-pve-nas-boot-migration.md`](../../docs/runbooks/esh-pve-nas-boot-migration.md).
- **Pools:** `nvme` (2× 931 GB NVMe mirror — 32 G used, 867 G free, holds every guest
rootfs), `ssd` (4× 894 GB Intel SATA, 2 mirrors — 1.42 T free), `tank`
(12× 14.6 TB raidz2 ×2 — 40 T of 175 T). Plus NFS `/mnt/pve/tank-vmbu` for VM backups.
- ⚠ **CT 103 `esh-nas` (10.0.50.50) runs on THIS host and serves `hard` NFS** to
esh-docker-vm (`/mnt/books`, `/mnt/backup`) and to esh-pve (`/mnt/pve/esh-nas`,
`/mnt/pve/tank-vmbu`). Taking this box down stalls esh-pve's storage layer and
wedges esh-docker-vm into unkillable D-state unless those clients are quiesced
first. Never reboot this host casually.
> **Root-fs pressure:** at 87% used on a 5.9 GB root partition, there's not much room for package upgrades or logs. Worth cleaning up or growing the root if this host is staying in production for a while.