vh
5f11d1b3cb
Root is now nvme/ROOT/pve-1. The USB DOM keeps the ESP and /boot but is out of the runtime I/O path, so a bus reset can no longer drop root from under a running hypervisor. All five guests healthy, three pools ONLINE, system running, ext4 pve-root intact and unmounted as the rollback with its own kernel and initrd. zfs-import-cache is now the active import path -- the all-three-pools cachefile fix doing its job. The window cost an unplanned outage, and the cause was this repo's own tooling rather than the migration. The staging chroot ran `mount --rbind /dev` and /sys with no --make-rslave. On systemd `/` has shared propagation, so the cutover's `umount -R` propagated back into the live host and removed the real /sys/fs/cgroup, /dev/pts and /dev/shm. With cgroup2 gone systemd-logind could not create a session: ping fine, TCP fine, SSH authentication succeeded, resident daemons kept serving -- and every new exec hung, including /sbin/reboot, so the reboot never ran at all. It impersonates failing root-disk I/O almost perfectly, and I called it as the DOM dying. That was wrong. dmesg had the answer throughout: the DOM attached cleanly with no errors, and the last log timestamp was 12114881s -- 140 days -- meaning this was still the original boot. A down-detector had also never reported the host down, which I read as a fast reboot rather than as no reboot. Fixes and guards: - --make-rslave after every rbind, plus a guard that refuses to proceed while any chroot bind still reports shared propagation. - Confirm a reboot by observing the host DOWN, not by watching for it to come back. Those two states are indistinguishable otherwise. - Blast radius now measured from the server: `ss` inside CT 103 found five NFS clients, not the two documented. The new one that mattered is esh-vm-db, hard-mounted and unreachable by ssh. Left mounted on purpose and it came through read-write. - grub-reboot's one-shot does NOT work here: grubenv sits on an LVM LV which GRUB reads but cannot write, so next_entry survived the boot that consumed it. Steady state is saved_entry=pve-zfs-root with no next_entry. There is no auto-fallback on this host and no IPMI. Recovery needed no console: an idempotent cgroup2/devpts/shm remount landed in the brief windows where exec succeeded. No data was lost, and neither the DOM nor any pool was ever at risk.
56 lines
2.2 KiB
YAML
56 lines
2.2 KiB
YAML
# esh-pve-nas cutover, step 1 of 5 — quiesce esh-docker-vm's hard NFS mounts.
|
|
#
|
|
# Run: scripts/elway infra-ops@10.0.50.45 --playbook playbooks/esh-cutover-1-quiesce-docker-vm.yaml
|
|
#
|
|
# Why this is first and why it is not optional: /mnt/books and /mnt/backup are
|
|
# `hard` NFS from CT 103 on esh-pve-nas. A hard mount does not fail when the
|
|
# server goes away — it blocks forever in D-state, and the only known remedy is
|
|
# rebooting THIS host. /mnt/books was deliberately left hard because calibre's
|
|
# SQLite risks corruption under `soft`, so the mount option is not the fix; the
|
|
# quiesce is.
|
|
#
|
|
# Measured 2026-08-18: exactly one container binds these paths
|
|
# (calibre-web-automated -> /mnt/books/calibre/{ingest,calibre_library}) and
|
|
# /mnt/backup has no container consumers at all. The blast radius is one service,
|
|
# not the seventeen containers on this host.
|
|
#
|
|
# Reversed by playbooks/esh-cutover-5-restore.yaml.
|
|
|
|
steps:
|
|
# No --format here: elway substitutes {{ ... }}, so Go template braces in a
|
|
# shell command are a booby trap. --filter + -q avoids them entirely.
|
|
- name: Stop the only container holding the NFS mounts
|
|
shell: sudo -n docker stop calibre-web-automated
|
|
when: "test -n \"$(sudo -n docker ps -q --filter name=^calibre-web-automated$)\""
|
|
|
|
- name: Confirm nothing else has files open under the mounts
|
|
shell: |
|
|
busy=$(sudo -n lsof +D /mnt/books +D /mnt/backup 2>/dev/null | tail -n +2 | wc -l)
|
|
if [ "$busy" -ne 0 ]; then
|
|
echo "STILL BUSY — refusing to unmount:"
|
|
sudo -n lsof +D /mnt/books +D /mnt/backup 2>/dev/null | head -20
|
|
exit 1
|
|
fi
|
|
echo "no open files under either mount"
|
|
changed_when: "false"
|
|
|
|
- name: Unmount /mnt/books
|
|
shell: sudo -n umount /mnt/books
|
|
when: "mountpoint -q /mnt/books"
|
|
|
|
- name: Unmount /mnt/backup
|
|
shell: sudo -n umount /mnt/backup
|
|
when: "mountpoint -q /mnt/backup"
|
|
|
|
verify:
|
|
- name: Neither NFS mount remains
|
|
shell: "! findmnt -t nfs,nfs4 -o TARGET | grep -qE '/mnt/(books|backup)'"
|
|
changed_when: "false"
|
|
|
|
- name: The other sixteen containers are still up
|
|
shell: |
|
|
n=$(sudo -n docker ps -q | wc -l)
|
|
echo "$n containers still running"
|
|
test "$n" -ge 10
|
|
changed_when: "false"
|