vh
5f11d1b3cb
Root is now nvme/ROOT/pve-1. The USB DOM keeps the ESP and /boot but is out of the runtime I/O path, so a bus reset can no longer drop root from under a running hypervisor. All five guests healthy, three pools ONLINE, system running, ext4 pve-root intact and unmounted as the rollback with its own kernel and initrd. zfs-import-cache is now the active import path -- the all-three-pools cachefile fix doing its job. The window cost an unplanned outage, and the cause was this repo's own tooling rather than the migration. The staging chroot ran `mount --rbind /dev` and /sys with no --make-rslave. On systemd `/` has shared propagation, so the cutover's `umount -R` propagated back into the live host and removed the real /sys/fs/cgroup, /dev/pts and /dev/shm. With cgroup2 gone systemd-logind could not create a session: ping fine, TCP fine, SSH authentication succeeded, resident daemons kept serving -- and every new exec hung, including /sbin/reboot, so the reboot never ran at all. It impersonates failing root-disk I/O almost perfectly, and I called it as the DOM dying. That was wrong. dmesg had the answer throughout: the DOM attached cleanly with no errors, and the last log timestamp was 12114881s -- 140 days -- meaning this was still the original boot. A down-detector had also never reported the host down, which I read as a fast reboot rather than as no reboot. Fixes and guards: - --make-rslave after every rbind, plus a guard that refuses to proceed while any chroot bind still reports shared propagation. - Confirm a reboot by observing the host DOWN, not by watching for it to come back. Those two states are indistinguishable otherwise. - Blast radius now measured from the server: `ss` inside CT 103 found five NFS clients, not the two documented. The new one that mattered is esh-vm-db, hard-mounted and unreachable by ssh. Left mounted on purpose and it came through read-write. - grub-reboot's one-shot does NOT work here: grubenv sits on an LVM LV which GRUB reads but cannot write, so next_entry survived the boot that consumed it. Steady state is saved_entry=pve-zfs-root with no next_entry. There is no auto-fallback on this host and no IPMI. Recovery needed no console: an idempotent cgroup2/devpts/shm remount landed in the brief windows where exec succeeded. No data was lost, and neither the DOM nor any pool was ever at risk.
52 lines
2.1 KiB
YAML
52 lines
2.1 KiB
YAML
# esh-pve-nas cutover, step 2 of 5 — quiesce esh-pve's hard NFS storages.
|
|
#
|
|
# Run: scripts/elway root@10.0.250.35 --playbook playbooks/esh-cutover-2-quiesce-esh-pve.yaml
|
|
#
|
|
# esh-pve mounts two `hard` NFS storages from CT 103 on esh-pve-nas:
|
|
# esh-nas -> 10.0.50.50:/mnt/pvestore at /mnt/pve/esh-nas
|
|
# tank-vmbu -> 10.0.50.50:/mnt/tank-vmbu at /mnt/pve/tank-vmbu
|
|
#
|
|
# Disabling the storage first matters: if the storage stays enabled, pvestatd
|
|
# keeps stat()ing the path and will re-trigger the mount (and then block on it)
|
|
# the moment the server disappears. Disable, THEN unmount.
|
|
#
|
|
# Measured 2026-08-18: esh-nas holds 2.9 MB of 96 TB and no running guest has a
|
|
# disk on either storage — all three (100 esh-vm-docker, 101 esh-vm-db,
|
|
# 102 esh-vm-workstation) live on local-lvm. So this quiesce costs backup targets
|
|
# for the duration, not guest availability. Guests are deliberately left running.
|
|
#
|
|
# Reversed by playbooks/esh-cutover-5-restore.yaml.
|
|
|
|
steps:
|
|
- name: Disable the esh-nas storage so pvestatd stops touching it
|
|
shell: pvesm set esh-nas --disable 1
|
|
when: "pvesm status 2>/dev/null | awk '$1==\"esh-nas\"{print $3}' | grep -q active"
|
|
|
|
- name: Disable the tank-vmbu storage
|
|
shell: pvesm set tank-vmbu --disable 1
|
|
when: "grep -q '^nfs: tank-vmbu' /etc/pve/storage.cfg && ! grep -A8 '^nfs: tank-vmbu' /etc/pve/storage.cfg | grep -q 'disable'"
|
|
|
|
- name: Give pvestatd a moment to let go before unmounting
|
|
shell: sleep 5
|
|
changed_when: "false"
|
|
|
|
- name: Unmount /mnt/pve/esh-nas
|
|
shell: umount /mnt/pve/esh-nas || umount -l /mnt/pve/esh-nas
|
|
when: "mountpoint -q /mnt/pve/esh-nas"
|
|
|
|
- name: Unmount /mnt/pve/tank-vmbu
|
|
shell: umount /mnt/pve/tank-vmbu || umount -l /mnt/pve/tank-vmbu
|
|
when: "mountpoint -q /mnt/pve/tank-vmbu"
|
|
|
|
verify:
|
|
- name: Neither esh-nas-backed NFS mount remains
|
|
shell: "! findmnt -t nfs,nfs4 -o SOURCE | grep -q '10\\.0\\.50\\.50'"
|
|
changed_when: "false"
|
|
|
|
- name: All three guests are still running
|
|
shell: |
|
|
n=$(qm list | awk 'NR>1 && $3=="running"' | wc -l)
|
|
echo "$n VMs running"
|
|
test "$n" -eq 3
|
|
changed_when: "false"
|