vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity

feat(esh-pve-nas): cut PVE root over to ZFS on the mirrored NVMe

Browse Source 
Root is now nvme/ROOT/pve-1. The USB DOM keeps the ESP and /boot but is
out of the runtime I/O path, so a bus reset can no longer drop root from
under a running hypervisor. All five guests healthy, three pools ONLINE,
system running, ext4 pve-root intact and unmounted as the rollback with
its own kernel and initrd. zfs-import-cache is now the active import
path -- the all-three-pools cachefile fix doing its job.

The window cost an unplanned outage, and the cause was this repo's own
tooling rather than the migration.

The staging chroot ran `mount --rbind /dev` and /sys with no
--make-rslave. On systemd `/` has shared propagation, so the cutover's
`umount -R` propagated back into the live host and removed the real
/sys/fs/cgroup, /dev/pts and /dev/shm. With cgroup2 gone systemd-logind
could not create a session: ping fine, TCP fine, SSH authentication
succeeded, resident daemons kept serving -- and every new exec hung,
including /sbin/reboot, so the reboot never ran at all.

It impersonates failing root-disk I/O almost perfectly, and I called it
as the DOM dying. That was wrong. dmesg had the answer throughout: the
DOM attached cleanly with no errors, and the last log timestamp was
12114881s -- 140 days -- meaning this was still the original boot. A
down-detector had also never reported the host down, which I read as a
fast reboot rather than as no reboot.

Fixes and guards:

- --make-rslave after every rbind, plus a guard that refuses to proceed
  while any chroot bind still reports shared propagation.
- Confirm a reboot by observing the host DOWN, not by watching for it to
  come back. Those two states are indistinguishable otherwise.
- Blast radius now measured from the server: `ss` inside CT 103 found
  five NFS clients, not the two documented. The new one that mattered is
  esh-vm-db, hard-mounted and unreachable by ssh. Left mounted on
  purpose and it came through read-write.
- grub-reboot's one-shot does NOT work here: grubenv sits on an LVM LV
  which GRUB reads but cannot write, so next_entry survived the boot
  that consumed it. Steady state is saved_entry=pve-zfs-root with no
  next_entry. There is no auto-fallback on this host and no IPMI.

Recovery needed no console: an idempotent cgroup2/devpts/shm remount
landed in the brief windows where exec succeeded. No data was lost, and
neither the DOM nor any pool was ever at risk.
This commit is contained in:
vh
2026-08-18 06:39:34 -07:00
parent b637947ffd
commit 5f11d1b3cb
8 changed files with 480 additions and 24 deletions
Download Patch File Download Diff File Expand all files Collapse all files
servers/esh-pve-nas/README.md
+39 -17
View File
@@ -16,24 +16,34 @@ Media services on this box sit at `10.0.50.56` (Plex) and `10.0.50.57` (Jellyfin
- **CPU:** Intel Xeon W-1250 @ 3.30 GHz
- **RAM:** 125.6 GB
- **Kernel:** `6.8.12-13-pve` (Proxmox 8.x)
- **Storage:** ⚠ **PVE root is on a USB Disk-on-Module** (`sdq`, 7.3 GB, `ID_BUS=usb`,
NORELSYS 1081), whose 512 MB ESP is the host's **only** boot path. Not a wear
problem (a DOM is SLC/pSLC), but it is on the USB bus, has no headroom, and is
unmirrored. Mitigated 2026-08-17 (90% → 76%: journald capped, journal relocated
to ZFS). **Migration STAGED 2026-08-18 — awaiting only the reboot window**;
runbook [`docs/runbooks/esh-pve-nas-boot-migration.md`](../../docs/runbooks/esh-pve-nas-boot-migration.md).
- **Storage: root is `nvme/ROOT/pve-1` on the mirrored NVMe pool** since the
2026-08-18 migration. The USB Disk-on-Module (7.3 GB, `ID_BUS=usb`, NORELSYS
1081) still holds the ESP and `/boot`, but is **out of the runtime I/O path** —
a USB bus reset no longer drops root from under a running hypervisor. Runbook:
[`docs/runbooks/esh-pve-nas-boot-migration.md`](../../docs/runbooks/esh-pve-nas-boot-migration.md).
DOM LVM layout after staging (was root 6.04 G + swap 768 M):
DOM LVM layout:
| LV | size | role |
|---|---|---|
| `pve-root` | 6.04 G | ext4 — the **live root today**, and the rollback after cutover |
| `pve-boot` | 512 M | ext4 — the new `/boot`, carved out of swap |
| `pve-root` | 6.04 G | ext4 — **the rollback**, intact and unmounted, keeps its own kernel + initrd |
| `pve-boot` | 512 M | ext4 — `/boot`, carved out of swap |
| `pve-swap` | 256 M | swap, shrunk to make room |
Post-cutover root is `nvme/ROOT/pve-1` on the mirrored NVMe pool. `/boot` stays
ext4 on the DOM on purpose: GRUB cannot read the `nvme` pool, which has
`encryption`, `large_dnode` and `zstd_compress` enabled.
`/boot` stays ext4 on the DOM on purpose: GRUB cannot read the `nvme` pool,
which has `encryption`, `large_dnode` and `zstd_compress` enabled.
⚠ **Device letters are not stable** — the DOM was `sdq` before the reboot and
`sdl` after. `fstab` uses `/dev/pve/*` and UUIDs; never write a rule against a
bare `sdX` on this host.
⚠ **There is no auto-fallback if a boot fails.** grubenv lives on an LVM LV,
which GRUB can read but not write, so `grub-reboot`'s one-shot degrades to a
sticky default (verified 2026-08-18: `next_entry` survived the boot that
consumed it). Steady state is `saved_entry=pve-zfs-root` with no `next_entry`.
A failed boot needs the console — and this box has **no IPMI, no BMC, no serial
console**. Recovery is selecting `Proxmox VE - ROLLBACK: ext4 root on the USB
DOM` at the GRUB menu.
- ⚠ **Never set the ZFS cachefile on one pool.** `zpool set cachefile=…` flips the
host from import-by-scan to import-by-cache; a cache holding only `nvme` leaves
`ssd` and `tank` unimported at boot, which empties every CT 103 export. Set it on
@@ -41,11 +51,23 @@ Media services on this box sit at `10.0.50.56` (Plex) and `10.0.50.57` (Jellyfin
- **Pools:** `nvme` (2× 931 GB NVMe mirror — 32 G used, 867 G free, holds every guest
rootfs), `ssd` (4× 894 GB Intel SATA, 2 mirrors — 1.42 T free), `tank`
(12× 14.6 TB raidz2 ×2 — 40 T of 175 T). Plus NFS `/mnt/pve/tank-vmbu` for VM backups.
- ⚠ **CT 103 `esh-nas` (10.0.50.50) runs on THIS host and serves `hard` NFS** to
esh-docker-vm (`/mnt/books`, `/mnt/backup`) and to esh-pve (`/mnt/pve/esh-nas`,
`/mnt/pve/tank-vmbu`). Taking this box down stalls esh-pve's storage layer and
wedges esh-docker-vm into unkillable D-state unless those clients are quiesced
first. Never reboot this host casually.
- ⚠ **CT 103 `esh-nas` (10.0.50.50) runs on THIS host and serves `hard` NFS.**
Never reboot this host casually — quiesce the clients first. Measured from the
server on 2026-08-18 (`ss -tn '( sport = :2049 )'` inside CT 103), there are
**five** clients, not the two long documented here:
| client | mounts | opts |
|---|---|---|
| esh-docker-vm `10.0.50.45` | `/mnt/books`, `/mnt/backup` | **hard** — quiesce |
| esh-pve `10.0.250.35` | `/mnt/pve/esh-nas`, `/mnt/pve/tank-vmbu` | **hard** — quiesce |
| esh-vm-db `10.0.50.60` | `/mnt/backup` | **hard** — no ssh; reach it via `qm guest exec 101` on esh-pve |
| vm-esh-nas `10.0.50.154` | — | is VM 104 *on this host*; stops with it |
| nh3-dev `10.100.10.50` | `/mnt/books` | soft,ro — safe, errors rather than blocks |
**Ask the server who its clients are; don't trust this table.** It rots. `ss` on
CT 103 is ground truth, and that is how the three undocumented clients surfaced.
Playbooks: `esh-cutover-1-quiesce-docker-vm.yaml`,
`esh-cutover-2-quiesce-esh-pve.yaml`, `esh-cutover-5-restore-docker-vm.yaml`.
> **Root-fs pressure:** 77% of a 5.9 GB root (1.3 GB free) after the 2026-08-17
> mitigation. Still not enough for the pending upgrade — **225 packages, 161