feat(esh-pve-nas): cut PVE root over to ZFS on the mirrored NVMe
Root is now nvme/ROOT/pve-1. The USB DOM keeps the ESP and /boot but is out of the runtime I/O path, so a bus reset can no longer drop root from under a running hypervisor. All five guests healthy, three pools ONLINE, system running, ext4 pve-root intact and unmounted as the rollback with its own kernel and initrd. zfs-import-cache is now the active import path -- the all-three-pools cachefile fix doing its job. The window cost an unplanned outage, and the cause was this repo's own tooling rather than the migration. The staging chroot ran `mount --rbind /dev` and /sys with no --make-rslave. On systemd `/` has shared propagation, so the cutover's `umount -R` propagated back into the live host and removed the real /sys/fs/cgroup, /dev/pts and /dev/shm. With cgroup2 gone systemd-logind could not create a session: ping fine, TCP fine, SSH authentication succeeded, resident daemons kept serving -- and every new exec hung, including /sbin/reboot, so the reboot never ran at all. It impersonates failing root-disk I/O almost perfectly, and I called it as the DOM dying. That was wrong. dmesg had the answer throughout: the DOM attached cleanly with no errors, and the last log timestamp was 12114881s -- 140 days -- meaning this was still the original boot. A down-detector had also never reported the host down, which I read as a fast reboot rather than as no reboot. Fixes and guards: - --make-rslave after every rbind, plus a guard that refuses to proceed while any chroot bind still reports shared propagation. - Confirm a reboot by observing the host DOWN, not by watching for it to come back. Those two states are indistinguishable otherwise. - Blast radius now measured from the server: `ss` inside CT 103 found five NFS clients, not the two documented. The new one that mattered is esh-vm-db, hard-mounted and unreachable by ssh. Left mounted on purpose and it came through read-write. - grub-reboot's one-shot does NOT work here: grubenv sits on an LVM LV which GRUB reads but cannot write, so next_entry survived the boot that consumed it. Steady state is saved_entry=pve-zfs-root with no next_entry. There is no auto-fallback on this host and no IPMI. Recovery needed no console: an idempotent cgroup2/devpts/shm remount landed in the brief windows where exec succeeded. No data was lost, and neither the DOM nor any pool was ever at risk.
This commit is contained in:
@@ -0,0 +1,51 @@
|
||||
# esh-pve-nas cutover, step 2 of 5 — quiesce esh-pve's hard NFS storages.
|
||||
#
|
||||
# Run: scripts/elway root@10.0.250.35 --playbook playbooks/esh-cutover-2-quiesce-esh-pve.yaml
|
||||
#
|
||||
# esh-pve mounts two `hard` NFS storages from CT 103 on esh-pve-nas:
|
||||
# esh-nas -> 10.0.50.50:/mnt/pvestore at /mnt/pve/esh-nas
|
||||
# tank-vmbu -> 10.0.50.50:/mnt/tank-vmbu at /mnt/pve/tank-vmbu
|
||||
#
|
||||
# Disabling the storage first matters: if the storage stays enabled, pvestatd
|
||||
# keeps stat()ing the path and will re-trigger the mount (and then block on it)
|
||||
# the moment the server disappears. Disable, THEN unmount.
|
||||
#
|
||||
# Measured 2026-08-18: esh-nas holds 2.9 MB of 96 TB and no running guest has a
|
||||
# disk on either storage — all three (100 esh-vm-docker, 101 esh-vm-db,
|
||||
# 102 esh-vm-workstation) live on local-lvm. So this quiesce costs backup targets
|
||||
# for the duration, not guest availability. Guests are deliberately left running.
|
||||
#
|
||||
# Reversed by playbooks/esh-cutover-5-restore.yaml.
|
||||
|
||||
steps:
|
||||
- name: Disable the esh-nas storage so pvestatd stops touching it
|
||||
shell: pvesm set esh-nas --disable 1
|
||||
when: "pvesm status 2>/dev/null | awk '$1==\"esh-nas\"{print $3}' | grep -q active"
|
||||
|
||||
- name: Disable the tank-vmbu storage
|
||||
shell: pvesm set tank-vmbu --disable 1
|
||||
when: "grep -q '^nfs: tank-vmbu' /etc/pve/storage.cfg && ! grep -A8 '^nfs: tank-vmbu' /etc/pve/storage.cfg | grep -q 'disable'"
|
||||
|
||||
- name: Give pvestatd a moment to let go before unmounting
|
||||
shell: sleep 5
|
||||
changed_when: "false"
|
||||
|
||||
- name: Unmount /mnt/pve/esh-nas
|
||||
shell: umount /mnt/pve/esh-nas || umount -l /mnt/pve/esh-nas
|
||||
when: "mountpoint -q /mnt/pve/esh-nas"
|
||||
|
||||
- name: Unmount /mnt/pve/tank-vmbu
|
||||
shell: umount /mnt/pve/tank-vmbu || umount -l /mnt/pve/tank-vmbu
|
||||
when: "mountpoint -q /mnt/pve/tank-vmbu"
|
||||
|
||||
verify:
|
||||
- name: Neither esh-nas-backed NFS mount remains
|
||||
shell: "! findmnt -t nfs,nfs4 -o SOURCE | grep -q '10\\.0\\.50\\.50'"
|
||||
changed_when: "false"
|
||||
|
||||
- name: All three guests are still running
|
||||
shell: |
|
||||
n=$(qm list | awk 'NR>1 && $3=="running"' | wc -l)
|
||||
echo "$n VMs running"
|
||||
test "$n" -eq 3
|
||||
changed_when: "false"
|
||||
Reference in New Issue
Block a user