vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity

feat(esh-pve-nas): cut PVE root over to ZFS on the mirrored NVMe

Browse Source 
Root is now nvme/ROOT/pve-1. The USB DOM keeps the ESP and /boot but is
out of the runtime I/O path, so a bus reset can no longer drop root from
under a running hypervisor. All five guests healthy, three pools ONLINE,
system running, ext4 pve-root intact and unmounted as the rollback with
its own kernel and initrd. zfs-import-cache is now the active import
path -- the all-three-pools cachefile fix doing its job.

The window cost an unplanned outage, and the cause was this repo's own
tooling rather than the migration.

The staging chroot ran `mount --rbind /dev` and /sys with no
--make-rslave. On systemd `/` has shared propagation, so the cutover's
`umount -R` propagated back into the live host and removed the real
/sys/fs/cgroup, /dev/pts and /dev/shm. With cgroup2 gone systemd-logind
could not create a session: ping fine, TCP fine, SSH authentication
succeeded, resident daemons kept serving -- and every new exec hung,
including /sbin/reboot, so the reboot never ran at all.

It impersonates failing root-disk I/O almost perfectly, and I called it
as the DOM dying. That was wrong. dmesg had the answer throughout: the
DOM attached cleanly with no errors, and the last log timestamp was
12114881s -- 140 days -- meaning this was still the original boot. A
down-detector had also never reported the host down, which I read as a
fast reboot rather than as no reboot.

Fixes and guards:

- --make-rslave after every rbind, plus a guard that refuses to proceed
  while any chroot bind still reports shared propagation.
- Confirm a reboot by observing the host DOWN, not by watching for it to
  come back. Those two states are indistinguishable otherwise.
- Blast radius now measured from the server: `ss` inside CT 103 found
  five NFS clients, not the two documented. The new one that mattered is
  esh-vm-db, hard-mounted and unreachable by ssh. Left mounted on
  purpose and it came through read-write.
- grub-reboot's one-shot does NOT work here: grubenv sits on an LVM LV
  which GRUB reads but cannot write, so next_entry survived the boot
  that consumed it. Steady state is saved_entry=pve-zfs-root with no
  next_entry. There is no auto-fallback on this host and no IPMI.

Recovery needed no console: an idempotent cgroup2/devpts/shm remount
landed in the brief windows where exec succeeded. No data was lost, and
neither the DOM nor any pool was ever at risk.
This commit is contained in:
vh
2026-08-18 06:39:34 -07:00
parent b637947ffd
commit 5f11d1b3cb
8 changed files with 480 additions and 24 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/runbooks/esh-pve-nas-boot-migration.md
+103 -4
View File
@@ -1,9 +1,21 @@
# esh-pve-nas — moving PVE root off the USB DOM
**Status: STAGED — everything but the reboot is done.** Mitigation landed
2026-08-17 (root 90% → 76%); staging landed 2026-08-18. The host is still
running from the ext4 root on the DOM and its boot path is unchanged; what
remains is the cutover window (§ Cutover).
**Status: DONE — cut over 2026-08-18.** Root is `nvme/ROOT/pve-1` on the mirrored
NVMe; `/boot` is ext4 on the DOM; the DOM is out of the runtime I/O path. All five
guests healthy, all three pools ONLINE, `systemctl is-system-running` = `running`.
The ext4 root (`pve-root`) is intact, unmounted, and still carries its own kernel
and initrd as the rollback.
Post-cutover boot config: `saved_entry=pve-zfs-root`, no `next_entry`. If grubenv
were ever unreadable GRUB falls through to menu entry 0, which the
`/etc/default/grub.d/zfs-root.cfg` drop-in also points at `root=ZFS=nvme/ROOT/pve-1`
— so every path boots ZFS.
⚠ **The window cost an unplanned outage, caused by a bug in this runbook's own
tooling, not by the migration.** Read § The mount-propagation incident before
running anything like this again. Two other findings — the blast radius being
more than double what was documented, and the one-shot rollback not actually
working — are recorded in § The pool-name bug's neighbours below.
Staging is two playbooks, both rerunnable:
@@ -255,6 +267,93 @@ Assert the effective value, not the presence of a substring.
**Cutover** — the remaining work, § Cutover below.
## ⚠ The mount-propagation incident — the expensive lesson of 2026-08-18
**What broke.** The staging chroot was built with `mount --rbind /dev` and `/sys`
and **no `--make-rslave`**. On a systemd host `/` has *shared* propagation, so
those binds propagate in both directions. When the cutover tore the chroot down
with `umount -R`, the unmounts **propagated back into the live host** and removed
the real `/sys/fs/cgroup`, `/dev/pts` and `/dev/shm`.
With cgroup2 gone, `systemd-logind` could no longer create a session. The result
is a host that:
- answers ping, accepts TCP, and **completes SSH authentication**
- keeps serving from daemons already resident in memory (`pveproxy` returned a
clean HTTP 401 throughout)
- **hangs on every new `exec`**, including `/sbin/reboot` — so the reboot that was
supposed to end the window never ran
**Why it cost so much time: it is a near-perfect impostor of failing root-disk
I/O.** Both present as "host is up, daemons answer, nothing new can start." The
session diagnosed it as the USB DOM dying and told the operator to walk to the
machine. That was wrong, and the operator caught it: the DOM had been reliable
for years and the wedge began immediately after a change.
**The evidence that settles it, and was available the whole time** — from
`dmesg`, obtainable in the brief windows when exec did succeed:
| line | says |
|---|---|
| `[16.00] sd 56:0:0:0: [sdq] Attached SCSI removable disk` | DOM enumerated **cleanly, no errors** |
| `[12114881.98] systemd[1]: nvme-varlog-stage.mount: Deactivated` | timestamp is **140 days** — this is the ORIGINAL boot |
That second line is the whole answer: **the machine never rebooted.** A
down-detector loop had also never once reported the host down; that was read as a
fast reboot rather than as no reboot at all.
**Rules that follow:**
1. **Always `mount --make-rslave` after `mount --rbind` into a chroot.** Phase 2
now does this and carries a guard that refuses to continue if any bind still
reports `shared` propagation.
2. **A reboot is not confirmed until the host is observed DOWN.** Poll for
disappearance, not just for reappearance. "Never went down" and "went down and
came back fast" are indistinguishable if you only watch for the host to answer.
3. **Before blaming hardware for a wedge that began right after a change, get
`dmesg` and check the boot timestamp.** Diagnose the change first; hardware is
the explanation of last resort, not first.
**Recovery took no console access.** Windows where `exec` briefly succeeded were
enough to land an idempotent remount of cgroup2 / devpts / shm, after which
`systemctl reset-failed` returned the host to `running`. Total data loss: none.
The root filesystem, the DOM and all three pools were never at risk — this was a
mount-namespace fault, not a storage one.
## The pool-name bug's neighbours — two more corrections
**The blast radius was more than double what was documented.** The runbook named
two NFS dependents. `ss -tn '( sport = :2049 )'` inside CT 103 showed **five**:
| client | mount | disposition |
|---|---|---|
| `10.0.50.45` esh-docker-vm | `/mnt/books`, `/mnt/backup` — **hard** | quiesced |
| `10.0.250.35` esh-pve | `esh-nas`, `tank-vmbu` — **hard** | quiesced |
| `10.0.50.60` **esh-vm-db** | `/mnt/backup` — **hard** | **left mounted deliberately** |
| `10.0.50.154` vm-esh-nas | — | is VM 104 *on this host*; stops with it |
| `10.100.10.50` nh3-dev | `/mnt/books` — **soft,ro** | safe, errors instead of blocking |
Ask the *server* who its clients are. A runbook's list of dependents is a snapshot
that rots; `ss` on the NFS server is ground truth.
esh-vm-db was left mounted on purpose and **came through read-write** — a `hard`
mount with no active user blocks and resumes, which is what `hard` is for. Its
backup timers were ~19h out, and unmounting would have meant an unmount/remount
cycle over the qemu guest agent on a host with no ssh access.
**The one-shot rollback does not work, and the warning was right.**
`grub-reboot` printed *"Detected GRUB environment block on lvm device — will
remain the default boot entry until manually cleared."* Confirmed empirically:
after the successful ZFS boot, `next_entry=pve-zfs-root` was **still set**. GRUB
can read grubenv on LVM but cannot write it, so `boot_once` degrades to a sticky
default. **There is no auto-fallback on this host.** A failed boot must be
corrected at the console.
The steady-state config therefore does not rely on it: `saved_entry=pve-zfs-root`
with `next_entry` cleared. Restoring a real one-shot would mean relocating grubenv
onto the ESP (vfat on a plain partition, which GRUB *can* write) — parked, not
required.
## Cutover
The only remaining work. Everything below the quiesce is minutes.