From 5f11d1b3cb0c8cefe39ab7545586f3a6847d6b99 Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Tue, 18 Aug 2026 06:39:34 -0700 Subject: [PATCH] feat(esh-pve-nas): cut PVE root over to ZFS on the mirrored NVMe Root is now nvme/ROOT/pve-1. The USB DOM keeps the ESP and /boot but is out of the runtime I/O path, so a bus reset can no longer drop root from under a running hypervisor. All five guests healthy, three pools ONLINE, system running, ext4 pve-root intact and unmounted as the rollback with its own kernel and initrd. zfs-import-cache is now the active import path -- the all-three-pools cachefile fix doing its job. The window cost an unplanned outage, and the cause was this repo's own tooling rather than the migration. The staging chroot ran `mount --rbind /dev` and /sys with no --make-rslave. On systemd `/` has shared propagation, so the cutover's `umount -R` propagated back into the live host and removed the real /sys/fs/cgroup, /dev/pts and /dev/shm. With cgroup2 gone systemd-logind could not create a session: ping fine, TCP fine, SSH authentication succeeded, resident daemons kept serving -- and every new exec hung, including /sbin/reboot, so the reboot never ran at all. It impersonates failing root-disk I/O almost perfectly, and I called it as the DOM dying. That was wrong. dmesg had the answer throughout: the DOM attached cleanly with no errors, and the last log timestamp was 12114881s -- 140 days -- meaning this was still the original boot. A down-detector had also never reported the host down, which I read as a fast reboot rather than as no reboot. Fixes and guards: - --make-rslave after every rbind, plus a guard that refuses to proceed while any chroot bind still reports shared propagation. - Confirm a reboot by observing the host DOWN, not by watching for it to come back. Those two states are indistinguishable otherwise. - Blast radius now measured from the server: `ss` inside CT 103 found five NFS clients, not the two documented. The new one that mattered is esh-vm-db, hard-mounted and unreachable by ssh. Left mounted on purpose and it came through read-write. - grub-reboot's one-shot does NOT work here: grubenv sits on an LVM LV which GRUB reads but cannot write, so next_entry survived the boot that consumed it. Steady state is saved_entry=pve-zfs-root with no next_entry. There is no auto-fallback on this host and no IPMI. Recovery needed no console: an idempotent cgroup2/devpts/shm remount landed in the brief windows where exec succeeded. No data was lost, and neither the DOM nor any pool was ever at risk. --- docs/runbooks/esh-pve-nas-boot-migration.md | 107 +++++++++++- persistent-memory.md | 5 +- .../esh-cutover-1-quiesce-docker-vm.yaml | 55 +++++++ playbooks/esh-cutover-2-quiesce-esh-pve.yaml | 51 ++++++ playbooks/esh-cutover-3-esh-pve-nas.yaml | 154 ++++++++++++++++++ .../esh-cutover-5-restore-docker-vm.yaml | 45 +++++ playbooks/esh-pve-nas-stage-bootloader.yaml | 31 +++- servers/esh-pve-nas/README.md | 56 +++++-- 8 files changed, 480 insertions(+), 24 deletions(-) create mode 100644 playbooks/esh-cutover-1-quiesce-docker-vm.yaml create mode 100644 playbooks/esh-cutover-2-quiesce-esh-pve.yaml create mode 100644 playbooks/esh-cutover-3-esh-pve-nas.yaml create mode 100644 playbooks/esh-cutover-5-restore-docker-vm.yaml diff --git a/docs/runbooks/esh-pve-nas-boot-migration.md b/docs/runbooks/esh-pve-nas-boot-migration.md index 13beb8b..678cb1a 100644 --- a/docs/runbooks/esh-pve-nas-boot-migration.md +++ b/docs/runbooks/esh-pve-nas-boot-migration.md @@ -1,9 +1,21 @@ # esh-pve-nas — moving PVE root off the USB DOM -**Status: STAGED — everything but the reboot is done.** Mitigation landed -2026-08-17 (root 90% → 76%); staging landed 2026-08-18. The host is still -running from the ext4 root on the DOM and its boot path is unchanged; what -remains is the cutover window (§ Cutover). +**Status: DONE — cut over 2026-08-18.** Root is `nvme/ROOT/pve-1` on the mirrored +NVMe; `/boot` is ext4 on the DOM; the DOM is out of the runtime I/O path. All five +guests healthy, all three pools ONLINE, `systemctl is-system-running` = `running`. +The ext4 root (`pve-root`) is intact, unmounted, and still carries its own kernel +and initrd as the rollback. + +Post-cutover boot config: `saved_entry=pve-zfs-root`, no `next_entry`. If grubenv +were ever unreadable GRUB falls through to menu entry 0, which the +`/etc/default/grub.d/zfs-root.cfg` drop-in also points at `root=ZFS=nvme/ROOT/pve-1` +— so every path boots ZFS. + +⚠ **The window cost an unplanned outage, caused by a bug in this runbook's own +tooling, not by the migration.** Read § The mount-propagation incident before +running anything like this again. Two other findings — the blast radius being +more than double what was documented, and the one-shot rollback not actually +working — are recorded in § The pool-name bug's neighbours below. Staging is two playbooks, both rerunnable: @@ -255,6 +267,93 @@ Assert the effective value, not the presence of a substring. **Cutover** — the remaining work, § Cutover below. +## ⚠ The mount-propagation incident — the expensive lesson of 2026-08-18 + +**What broke.** The staging chroot was built with `mount --rbind /dev` and `/sys` +and **no `--make-rslave`**. On a systemd host `/` has *shared* propagation, so +those binds propagate in both directions. When the cutover tore the chroot down +with `umount -R`, the unmounts **propagated back into the live host** and removed +the real `/sys/fs/cgroup`, `/dev/pts` and `/dev/shm`. + +With cgroup2 gone, `systemd-logind` could no longer create a session. The result +is a host that: + +- answers ping, accepts TCP, and **completes SSH authentication** +- keeps serving from daemons already resident in memory (`pveproxy` returned a + clean HTTP 401 throughout) +- **hangs on every new `exec`**, including `/sbin/reboot` — so the reboot that was + supposed to end the window never ran + +**Why it cost so much time: it is a near-perfect impostor of failing root-disk +I/O.** Both present as "host is up, daemons answer, nothing new can start." The +session diagnosed it as the USB DOM dying and told the operator to walk to the +machine. That was wrong, and the operator caught it: the DOM had been reliable +for years and the wedge began immediately after a change. + +**The evidence that settles it, and was available the whole time** — from +`dmesg`, obtainable in the brief windows when exec did succeed: + +| line | says | +|---|---| +| `[16.00] sd 56:0:0:0: [sdq] Attached SCSI removable disk` | DOM enumerated **cleanly, no errors** | +| `[12114881.98] systemd[1]: nvme-varlog-stage.mount: Deactivated` | timestamp is **140 days** — this is the ORIGINAL boot | + +That second line is the whole answer: **the machine never rebooted.** A +down-detector loop had also never once reported the host down; that was read as a +fast reboot rather than as no reboot at all. + +**Rules that follow:** + +1. **Always `mount --make-rslave` after `mount --rbind` into a chroot.** Phase 2 + now does this and carries a guard that refuses to continue if any bind still + reports `shared` propagation. +2. **A reboot is not confirmed until the host is observed DOWN.** Poll for + disappearance, not just for reappearance. "Never went down" and "went down and + came back fast" are indistinguishable if you only watch for the host to answer. +3. **Before blaming hardware for a wedge that began right after a change, get + `dmesg` and check the boot timestamp.** Diagnose the change first; hardware is + the explanation of last resort, not first. + +**Recovery took no console access.** Windows where `exec` briefly succeeded were +enough to land an idempotent remount of cgroup2 / devpts / shm, after which +`systemctl reset-failed` returned the host to `running`. Total data loss: none. +The root filesystem, the DOM and all three pools were never at risk — this was a +mount-namespace fault, not a storage one. + +## The pool-name bug's neighbours — two more corrections + +**The blast radius was more than double what was documented.** The runbook named +two NFS dependents. `ss -tn '( sport = :2049 )'` inside CT 103 showed **five**: + +| client | mount | disposition | +|---|---|---| +| `10.0.50.45` esh-docker-vm | `/mnt/books`, `/mnt/backup` — **hard** | quiesced | +| `10.0.250.35` esh-pve | `esh-nas`, `tank-vmbu` — **hard** | quiesced | +| `10.0.50.60` **esh-vm-db** | `/mnt/backup` — **hard** | **left mounted deliberately** | +| `10.0.50.154` vm-esh-nas | — | is VM 104 *on this host*; stops with it | +| `10.100.10.50` nh3-dev | `/mnt/books` — **soft,ro** | safe, errors instead of blocking | + +Ask the *server* who its clients are. A runbook's list of dependents is a snapshot +that rots; `ss` on the NFS server is ground truth. + +esh-vm-db was left mounted on purpose and **came through read-write** — a `hard` +mount with no active user blocks and resumes, which is what `hard` is for. Its +backup timers were ~19h out, and unmounting would have meant an unmount/remount +cycle over the qemu guest agent on a host with no ssh access. + +**The one-shot rollback does not work, and the warning was right.** +`grub-reboot` printed *"Detected GRUB environment block on lvm device — will +remain the default boot entry until manually cleared."* Confirmed empirically: +after the successful ZFS boot, `next_entry=pve-zfs-root` was **still set**. GRUB +can read grubenv on LVM but cannot write it, so `boot_once` degrades to a sticky +default. **There is no auto-fallback on this host.** A failed boot must be +corrected at the console. + +The steady-state config therefore does not rely on it: `saved_entry=pve-zfs-root` +with `next_entry` cleared. Restoring a real one-shot would mean relocating grubenv +onto the ESP (vfat on a plain partition, which GRUB *can* write) — parked, not +required. + ## Cutover The only remaining work. Everything below the quiesce is minutes. diff --git a/persistent-memory.md b/persistent-memory.md index eb4a043..a347599 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -133,7 +133,10 @@ _As of 2026-08-18 — **gen seat SWAPPED to `absolute-heresy` and operator-confi - **⏳ WATCHING: DavidAU's HERETIC build of Qwen3.8-27B — the one worth waiting for.** `DavidAU/Qwen3.8-27B-Cold-Fusion-GAIN-V1.1` examined 2026-08-17 and **NOT adopted**: it is a capability/efficiency finetune of **stock** Qwen3.8 and every bench row on its card is labelled **`[non heretic]`** — adopting it would reintroduce base refusals the current seat does not have. ⚠ **Easy to misread as uncensored** (operator did, and it is a fair mistake): DavidAU's back catalog is almost all `Uncensored-Heretic` builds — Fable-Fusion 711, Qwen3.5-9B Cold-Fusion — so the naming pattern implies it. This one simply has not had that stage run yet; the card's roadmap says the HERETIC version is **IN PROGRESS from base**. **That** is the release to watch, not this one. What makes it worth watching: third-party benches (Nightmedia, mxfp8) beat stock Qwen3.8 by **+0.064 arc/c, +0.056 arc/e, +0.056 obkqa**; claimed **MTP acceptance 55.7% (record 59.9%)** vs our measured 47.2%; **thinking tokens cut to 1/10–1/2**; PPL *dropped* vs base. Same GAIN/Cold-Fusion pipeline that produced Fable-Fusion 711, which we already serve on `char-rp-reasoning` — proven in-fleet, not just claimed. Structurally clean (1199 tensors, 15 mtp in shard 18, 333 visual). ⚠ MTP/speed figures are **GGUF/llama.cpp on a 5090, not vLLM** — may not transfer; and because its MTP head was likely *trained*, the free CPU-hash shortcut would NOT apply (it won't match base) so a real acceptance gate would be needed. -- **🟢 esh-pve-nas — migration STAGED 2026-08-18; ONLY THE REBOOT WINDOW REMAINS.** PVE root ran off a **NORELSYS USB Disk-on-Module** (`sdq`). Wear was NOT the driver (a DOM is SLC/pSLC — operator corrected my first read); the drivers are the **USB bus** (a reset drops root under a running hypervisor), no headroom, no mirror, and **blocked patching: 225 packages, 161 with `deb12uN`/security bumps**, stuck on PVE 8.4.11 vs esh-pve's 8.4.14. Mitigated 2026-08-17 (90% → 76%); **staged 2026-08-18 via two rerunnable elway playbooks** (`esh-pve-nas-stage-zfs-root.yaml`, `esh-pve-nas-stage-bootloader.yaml`), 0 failed, 17/17 verify green. +- **🟢 esh-pve-nas — MIGRATION DONE 2026-08-18. Root is `nvme/ROOT/pve-1` on mirrored NVMe; the USB DOM is out of the runtime I/O path.** All five guests healthy, three pools ONLINE, system `running`, ext4 `pve-root` intact+unmounted as rollback with its own kernel. Boot config: `saved_entry=pve-zfs-root`, no `next_entry`; if grubenv were unreadable GRUB falls to entry 0 which the `/etc/default/grub.d/zfs-root.cfg` drop-in also points at ZFS. **`zfs-import-cache.service` is now the active import path** (the all-three-pools cachefile fix working as intended); vestigial `zfs-import@nvme.service` disabled — it failed every boot as redundant. ⚠ **Device letters shift across reboots** (DOM was `sdq`, now `sdl`) — never key anything to a bare `sdX` here. ⚠ **NO auto-fallback on a failed boot, and no IPMI/BMC/serial** — grubenv on LVM is readable but not writable by GRUB, so `grub-reboot`'s one-shot degrades to a sticky default (verified: `next_entry` survived the boot that consumed it). Recovery = pick the ROLLBACK entry at the console. **⏳ REMAINING: second reboot to prove repeatability (needs another quiesce window), refresh the DOM image since `/boot` changed, then patch the 225 pending packages — the whole point of the migration.** + ⚠⚠ **THE WINDOW COST AN UNPLANNED OUTAGE, caused by our own tooling, not the migration.** The staging chroot did `mount --rbind /dev` + `/sys` with **no `--make-rslave`**; on systemd `/` is *shared*, so the cutover's `umount -R` **propagated back into the live host** and stripped the real `/sys/fs/cgroup`, `/dev/pts`, `/dev/shm`. logind could then create no sessions: ping fine, TCP fine, **SSH authenticates**, resident daemons keep serving (pveproxy returned clean 401s) — and **every new exec hangs, including `/sbin/reboot`**, so the reboot never ran. **It is a near-perfect impostor of failing root-disk I/O**, and I misdiagnosed it as the USB DOM dying and told the operator to walk to the machine. **Operator caught it** — the DOM had been fine for years and the wedge began right after a change. The settling evidence was in `dmesg` all along: `[16.00] [sdq] Attached SCSI removable disk` (clean, no errors) and a last-line timestamp of **`[12114881]` = 140 days = the ORIGINAL boot** — the machine had never rebooted. My down-detector never once reported the host down and I read that as a fast reboot rather than *no* reboot. Recovered with **no console access** by hammering an idempotent cgroup2/devpts/shm remount into the brief windows where exec succeeded. Zero data loss. **RULES: (1) always `--make-rslave` after `--rbind` (playbook now guards on `PROPAGATION != shared`); (2) a reboot is not confirmed until the host is observed DOWN — poll for disappearance, not reappearance; (3) before blaming hardware for a wedge that started right after a change, get `dmesg` and check the boot timestamp.** + ⚠ **Blast radius was more than DOUBLE what the runbook said** — `ss -tn '( sport = :2049 )'` in CT 103 found **five** NFS clients, not two: +**esh-vm-db `10.0.50.60`** (`/mnt/backup`, **hard**, no ssh — reach via `qm guest exec 101` on esh-pve), +vm-esh-nas (is VM 104 on the host itself), +nh3-dev (soft,ro — safe). **Ask the server who its clients are; a documented dependent list rots.** esh-vm-db was deliberately left mounted (backup target, timers ~19h out) and **came through read-write** — a `hard` mount with no active user blocks and resumes, which is what `hard` is for. + _(historical, pre-cutover context below)_ **migration STAGED 2026-08-18.** PVE root ran off a **NORELSYS USB Disk-on-Module** (`sdq`). Wear was NOT the driver (a DOM is SLC/pSLC — operator corrected my first read); the drivers are the **USB bus** (a reset drops root under a running hypervisor), no headroom, no mirror, and **blocked patching: 225 packages, 161 with `deb12uN`/security bumps**, stuck on PVE 8.4.11 vs esh-pve's 8.4.14. Mitigated 2026-08-17 (90% → 76%); **staged 2026-08-18 via two rerunnable elway playbooks** (`esh-pve-nas-stage-zfs-root.yaml`, `esh-pve-nas-stage-bootloader.yaml`), 0 failed, 17/17 verify green. **Host is still on ext4 and its boot path is byte-identical to the last 140 days** — `grub-install` is deliberately held back to the window, so an unplanned reboot mid-staging is a non-event. **CUTOVER = quiesce NFS clients → stop guests → `grub-install` → `grub-reboot pve-zfs-root` → `zfs set mountpoint=/ nvme/ROOT/pve-1` → reboot.** ⚠⚠ **THE POOL-NAME BUG — the near-miss worth remembering.** `update-grub` silently emits `root=ZFS=/ROOT/pve-1` with **the pool name missing** → boots to an initramfs prompt. Cause: Debian's `10_linux` builds `${rpool}${bootfs}` where `rpool = grub-probe --target=fs_label`, and **GRUB's ZFS reader cannot open a pool with `encryption`/`large_dnode`/`zstd_compress`** — so it returns empty and the failure is swallowed by `2>/dev/null || true`. The *same* feature set that forced `/boot` to stay ext4 also corrupts the kernel cmdline, which the design did not anticipate. Fixed with a `/etc/default/grub.d/zfs-root.cfg` drop-in (last `root=` wins) plus explicit `pve-zfs-root` / `pve-ext4-rollback` entries with stable ids. **Durable lesson: assert the EFFECTIVE value, not the presence of a substring** — the original verify grepped for the right string *appearing somewhere* and would have passed with every menu entry still broken. ⚠ **Never set the ZFS cachefile on one pool.** The runbook's `zpool set cachefile=… nvme` was a trap: populating a cache flips the host from import-by-scan to import-by-cache, so a cache holding only `nvme` leaves `ssd`+`tank` unimported and empties every CT 103 export. Set on all three 2026-08-18, verified in the 11,976-byte cache. diff --git a/playbooks/esh-cutover-1-quiesce-docker-vm.yaml b/playbooks/esh-cutover-1-quiesce-docker-vm.yaml new file mode 100644 index 0000000..baf7d76 --- /dev/null +++ b/playbooks/esh-cutover-1-quiesce-docker-vm.yaml @@ -0,0 +1,55 @@ +# esh-pve-nas cutover, step 1 of 5 — quiesce esh-docker-vm's hard NFS mounts. +# +# Run: scripts/elway infra-ops@10.0.50.45 --playbook playbooks/esh-cutover-1-quiesce-docker-vm.yaml +# +# Why this is first and why it is not optional: /mnt/books and /mnt/backup are +# `hard` NFS from CT 103 on esh-pve-nas. A hard mount does not fail when the +# server goes away — it blocks forever in D-state, and the only known remedy is +# rebooting THIS host. /mnt/books was deliberately left hard because calibre's +# SQLite risks corruption under `soft`, so the mount option is not the fix; the +# quiesce is. +# +# Measured 2026-08-18: exactly one container binds these paths +# (calibre-web-automated -> /mnt/books/calibre/{ingest,calibre_library}) and +# /mnt/backup has no container consumers at all. The blast radius is one service, +# not the seventeen containers on this host. +# +# Reversed by playbooks/esh-cutover-5-restore.yaml. + +steps: + # No --format here: elway substitutes {{ ... }}, so Go template braces in a + # shell command are a booby trap. --filter + -q avoids them entirely. + - name: Stop the only container holding the NFS mounts + shell: sudo -n docker stop calibre-web-automated + when: "test -n \"$(sudo -n docker ps -q --filter name=^calibre-web-automated$)\"" + + - name: Confirm nothing else has files open under the mounts + shell: | + busy=$(sudo -n lsof +D /mnt/books +D /mnt/backup 2>/dev/null | tail -n +2 | wc -l) + if [ "$busy" -ne 0 ]; then + echo "STILL BUSY — refusing to unmount:" + sudo -n lsof +D /mnt/books +D /mnt/backup 2>/dev/null | head -20 + exit 1 + fi + echo "no open files under either mount" + changed_when: "false" + + - name: Unmount /mnt/books + shell: sudo -n umount /mnt/books + when: "mountpoint -q /mnt/books" + + - name: Unmount /mnt/backup + shell: sudo -n umount /mnt/backup + when: "mountpoint -q /mnt/backup" + +verify: + - name: Neither NFS mount remains + shell: "! findmnt -t nfs,nfs4 -o TARGET | grep -qE '/mnt/(books|backup)'" + changed_when: "false" + + - name: The other sixteen containers are still up + shell: | + n=$(sudo -n docker ps -q | wc -l) + echo "$n containers still running" + test "$n" -ge 10 + changed_when: "false" diff --git a/playbooks/esh-cutover-2-quiesce-esh-pve.yaml b/playbooks/esh-cutover-2-quiesce-esh-pve.yaml new file mode 100644 index 0000000..666ccf0 --- /dev/null +++ b/playbooks/esh-cutover-2-quiesce-esh-pve.yaml @@ -0,0 +1,51 @@ +# esh-pve-nas cutover, step 2 of 5 — quiesce esh-pve's hard NFS storages. +# +# Run: scripts/elway root@10.0.250.35 --playbook playbooks/esh-cutover-2-quiesce-esh-pve.yaml +# +# esh-pve mounts two `hard` NFS storages from CT 103 on esh-pve-nas: +# esh-nas -> 10.0.50.50:/mnt/pvestore at /mnt/pve/esh-nas +# tank-vmbu -> 10.0.50.50:/mnt/tank-vmbu at /mnt/pve/tank-vmbu +# +# Disabling the storage first matters: if the storage stays enabled, pvestatd +# keeps stat()ing the path and will re-trigger the mount (and then block on it) +# the moment the server disappears. Disable, THEN unmount. +# +# Measured 2026-08-18: esh-nas holds 2.9 MB of 96 TB and no running guest has a +# disk on either storage — all three (100 esh-vm-docker, 101 esh-vm-db, +# 102 esh-vm-workstation) live on local-lvm. So this quiesce costs backup targets +# for the duration, not guest availability. Guests are deliberately left running. +# +# Reversed by playbooks/esh-cutover-5-restore.yaml. + +steps: + - name: Disable the esh-nas storage so pvestatd stops touching it + shell: pvesm set esh-nas --disable 1 + when: "pvesm status 2>/dev/null | awk '$1==\"esh-nas\"{print $3}' | grep -q active" + + - name: Disable the tank-vmbu storage + shell: pvesm set tank-vmbu --disable 1 + when: "grep -q '^nfs: tank-vmbu' /etc/pve/storage.cfg && ! grep -A8 '^nfs: tank-vmbu' /etc/pve/storage.cfg | grep -q 'disable'" + + - name: Give pvestatd a moment to let go before unmounting + shell: sleep 5 + changed_when: "false" + + - name: Unmount /mnt/pve/esh-nas + shell: umount /mnt/pve/esh-nas || umount -l /mnt/pve/esh-nas + when: "mountpoint -q /mnt/pve/esh-nas" + + - name: Unmount /mnt/pve/tank-vmbu + shell: umount /mnt/pve/tank-vmbu || umount -l /mnt/pve/tank-vmbu + when: "mountpoint -q /mnt/pve/tank-vmbu" + +verify: + - name: Neither esh-nas-backed NFS mount remains + shell: "! findmnt -t nfs,nfs4 -o SOURCE | grep -q '10\\.0\\.50\\.50'" + changed_when: "false" + + - name: All three guests are still running + shell: | + n=$(qm list | awk 'NR>1 && $3=="running"' | wc -l) + echo "$n VMs running" + test "$n" -eq 3 + changed_when: "false" diff --git a/playbooks/esh-cutover-3-esh-pve-nas.yaml b/playbooks/esh-cutover-3-esh-pve-nas.yaml new file mode 100644 index 0000000..56823fa --- /dev/null +++ b/playbooks/esh-cutover-3-esh-pve-nas.yaml @@ -0,0 +1,154 @@ +# esh-pve-nas cutover, step 3 of 5 — point the ESP at the new /boot and reboot. +# +# Run: scripts/elway root@esh-pve-nas --playbook playbooks/esh-cutover-3-esh-pve-nas.yaml +# +# PRECONDITION: steps 1 and 2 must have run. Both NFS clients hold `hard` mounts +# from CT 103 which lives on this host; taking it down with them mounted wedges +# esh-docker-vm in unkillable D-state. A guard below refuses to proceed if either +# client is still mounted. +# +# ⚠ ORDERING TRAP, and it is the reason this is a playbook and not four commands: +# `zfs set mountpoint=/` on a dataset that is CURRENTLY MOUNTED makes ZFS unmount +# and REMOUNT it at the new location — i.e. it would try to mount the ZFS root +# over the live ext4 root of a running hypervisor. canmount=noauto does not save +# you; that governs automatic mounting at import, not an explicit property change +# on a mounted dataset. The dataset must be UNMOUNTED first, which means the +# chroot binds have to come down first, which means grub-install and grub-reboot +# have to happen BEFORE any of that. Hence the sequence below is not negotiable. +# +# This playbook ENDS BY REBOOTING THE HOST. elway will lose the connection; that +# is expected, not a failure. + +vars: + newroot: /mnt/newroot + root_dataset: nvme/ROOT/pve-1 + quiesced: "no" # caller MUST pass --var quiesced=yes after verifying both clients + +steps: + # ---------- guards ---------- + + - name: GUARD — still on the ext4 root (not already cut over) + shell: | + test "$(findmnt -no FSTYPE /)" = "ext4" || { echo "already on ZFS; refusing"; exit 1; } + changed_when: "false" + + # This host has no ssh keys to the NFS clients, so the caller verifies their + # mount tables and attests via --var quiesced=yes. + # + # ⚠ THE RUNBOOK'S BLAST RADIUS WAS WRONG. It named two dependents. `ss` on CT 103 + # showed FIVE distinct clients on 2026-08-18: + # 10.0.50.45 esh-docker-vm hard -> quiesced by step 1 + # 10.0.250.35 esh-pve hard -> quiesced by step 2 + # 10.0.50.60 esh-vm-db hard -> DELIBERATELY LEFT MOUNTED (see below) + # 10.0.50.154 vm-esh-nas n/a -> is VM 104 on THIS host; dies with it + # 10.100.10.50 nh3-dev soft,ro -> errors instead of blocking; safe + # + # esh-vm-db is left mounted on purpose. It is a backup TARGET with no live user: + # resticprofile-backup and postgresql-dump next fire ~19h out, and a hard mount + # with nothing actively using it blocks and then resumes when the server returns + # — that is what `hard` is for. Unmounting it would mean an unmount/remount cycle + # over the qemu guest agent on a host with no ssh access, where a failed remount + # breaks backups silently. Leaving it is the lower-risk branch, not the lazy one. + # The gate is `quiesced`, which the CALLER sets only after checking each client's + # mount table directly (this host has no ssh to them; see step 1/2 playbooks). + # + # ⚠ It deliberately does NOT gate on server-side NFS session count. Measured + # 2026-08-18: esh-docker-vm's sessions drained within ~90s, but esh-pve held 11 + # established connections to :2049 indefinitely with NO mounts in either + # `findmnt` or `/proc/mounts` and nothing holding a cwd there. That is the Linux + # NFSv4 client keeping its transport alive past the last unmount, and it is the + # wrong thing to gate on: the failure this whole runbook exists to prevent is a + # process blocking on a MOUNTED hard filesystem when the server vanishes. With no + # mount there is nothing to block on — an idle socket to a departing server just + # resets. Gating on sessions would have stalled the window forever on a condition + # that never clears and never mattered. + - name: GUARD — caller has confirmed both hard-NFS clients are unmounted + shell: | + test "{{ quiesced }}" = "yes" || { + echo "run playbooks 1 and 2 and confirm client mount tables first"; exit 1; } + echo "caller attests: esh-docker-vm and esh-pve carry no esh-nas mounts" + echo "--- server-side sessions, informational only ---" + pct exec 103 -- ss -tnH state established '( sport = :2049 )' 2>/dev/null \ + | awk '{print $4}' | sed 's/:[0-9]*$//' | sort | uniq -c || true + changed_when: "false" + + - name: GUARD — staging artifacts are all present + shell: | + mountpoint -q {{ newroot }} || { echo "{{ newroot }} not mounted"; exit 1; } + mountpoint -q {{ newroot }}/boot || { echo "boot LV not in the chroot"; exit 1; } + grep -q pve-zfs-root {{ newroot }}/boot/grub/grub.cfg || { echo "no ZFS entry"; exit 1; } + grep -q 'saved_entry=pve-ext4-rollback' {{ newroot }}/boot/grub/grubenv || { echo "grubenv not pinned to rollback"; exit 1; } + changed_when: "false" + + # ---------- stop the guests, NAS last ---------- + + - name: Stop the guests (reverse of startup order — CT 103, the NAS, goes last) + shell: | + for v in 105 106 107; do pct status $v 2>/dev/null | grep -q running && pct shutdown $v --timeout 90 || true; done + qm status 104 2>/dev/null | grep -q running && qm shutdown 104 --timeout 90 || true + for i in $(seq 1 30); do + running=$( (pct list | awk 'NR>1 && $2=="running"'; qm list | awk 'NR>1 && $3=="running"') | wc -l ) + [ "$running" -le 1 ] && break + sleep 3 + done + pct status 103 2>/dev/null | grep -q running && pct shutdown 103 --timeout 90 || true + sleep 3 + echo "--- remaining ---"; pct list; qm list + changed_when: "true" + + # ---------- the actual cutover ---------- + + - name: Point the ESP at the new /boot LV + shell: | + chroot {{ newroot }} grub-install --target=x86_64-efi \ + --efi-directory=/boot/efi --bootloader-id=proxmox + changed_when: "true" + + - name: Verify the ESP stub now points at the /boot LV, not the ext4 root + shell: | + BOOT_UUID=$(blkid -s UUID -o value /dev/mapper/pve-boot) + grep -q "$BOOT_UUID" {{ newroot }}/boot/efi/EFI/proxmox/grub.cfg || { + echo "ESP stub does NOT reference the boot LV — aborting before reboot"; exit 1; } + echo "ESP stub -> boot LV $BOOT_UUID" + changed_when: "false" + + - name: Arm the ONE-SHOT ZFS boot (default stays pinned to the ext4 rollback) + shell: | + chroot {{ newroot }} grub-reboot pve-zfs-root + grep -o 'next_entry=.*' {{ newroot }}/boot/grub/grubenv + grep -o 'saved_entry=.*' {{ newroot }}/boot/grub/grubenv + changed_when: "true" + + # ---------- tear the chroot down so the dataset can be unmounted ---------- + + - name: Unmount the chroot, innermost first + shell: | + for m in proc/sys/fs/binfmt_misc proc sys dev/pts dev/shm dev/mqueue dev/hugepages dev boot/efi boot; do + mountpoint -q {{ newroot }}/$m && umount -R {{ newroot }}/$m 2>/dev/null || true + done + findmnt -R {{ newroot }} -o TARGET | tail -n +2 || echo " (nothing left under {{ newroot }})" + changed_when: "true" + + - name: Unmount the ZFS root dataset BEFORE changing its mountpoint + shell: zfs unmount {{ root_dataset }} + when: "mountpoint -q {{ newroot }}" + + - name: Set the dataset's final mountpoint (safe only now that it is unmounted) + shell: | + zfs set mountpoint=/ {{ root_dataset }} + zfs get -H -o value mountpoint,canmount {{ root_dataset }} | tr '\n' ' '; echo + # paranoia: the live root must STILL be the ext4 LV at this instant + test "$(findmnt -no SOURCE /)" = "/dev/mapper/pve-root" || { + echo "ZFS MOUNTED OVER THE LIVE ROOT — do not reboot, investigate"; exit 1; } + changed_when: "true" + + - name: Final pre-reboot assertion + shell: | + echo "root now: $(findmnt -no SOURCE,FSTYPE /)" + echo "dataset: $(zfs get -H -o value mounted {{ root_dataset }}) mounted, canmount=$(zfs get -H -o value canmount {{ root_dataset }})" + echo "next_entry: $(grep -o 'next_entry=.*' {{ newroot }}/boot/grub/grubenv 2>/dev/null || echo '(grubenv not readable — boot LV is unmounted, expected)')" + changed_when: "false" + + - name: REBOOT — connection loss here is expected + shell: systemd-run --on-active=3 --timer-property=AccuracySec=1s /sbin/reboot + changed_when: "true" diff --git a/playbooks/esh-cutover-5-restore-docker-vm.yaml b/playbooks/esh-cutover-5-restore-docker-vm.yaml new file mode 100644 index 0000000..a06c8c0 --- /dev/null +++ b/playbooks/esh-cutover-5-restore-docker-vm.yaml @@ -0,0 +1,45 @@ +# esh-pve-nas cutover, step 5a of 5 — restore esh-docker-vm's NFS mounts. +# +# Run: scripts/elway infra-ops@10.0.50.45 --playbook playbooks/esh-cutover-5-restore-docker-vm.yaml +# +# Reverses playbooks/esh-cutover-1-quiesce-docker-vm.yaml. Mount first, THEN start +# the container: calibre opens its SQLite library on startup, and starting it +# against an unmounted /mnt/books would have it create a fresh empty library on +# the local disk underneath the mountpoint — which then gets shadowed the moment +# the real mount lands, and looks exactly like data loss. + +steps: + - name: Mount /mnt/books + shell: sudo -n mount /mnt/books + when: "! mountpoint -q /mnt/books" + + - name: Mount /mnt/backup + shell: sudo -n mount /mnt/backup + when: "! mountpoint -q /mnt/backup" + + - name: Confirm the library is actually there before starting calibre + shell: | + test -f /mnt/books/calibre/calibre_library/metadata.db || { + echo "calibre library NOT visible — refusing to start the container"; exit 1; } + echo "metadata.db present: $(stat -c %s /mnt/books/calibre/calibre_library/metadata.db) bytes" + changed_when: "false" + + - name: Start calibre-web-automated + shell: sudo -n docker start calibre-web-automated + when: "test -z \"$(sudo -n docker ps -q --filter name=^calibre-web-automated$)\"" + +verify: + - name: Both NFS mounts are back + shell: | + mountpoint -q /mnt/books && mountpoint -q /mnt/backup + findmnt -no SOURCE,OPTIONS /mnt/books | grep -q hard + changed_when: "false" + + - name: calibre-web-automated is running + shell: test -n "$(sudo -n docker ps -q --filter name=^calibre-web-automated$)" + changed_when: "false" + + - name: Full container count is back + shell: | + n=$(sudo -n docker ps -q | wc -l); echo "$n containers running"; test "$n" -ge 17 + changed_when: "false" diff --git a/playbooks/esh-pve-nas-stage-bootloader.yaml b/playbooks/esh-pve-nas-stage-bootloader.yaml index 151265b..7c66c09 100644 --- a/playbooks/esh-pve-nas-stage-bootloader.yaml +++ b/playbooks/esh-pve-nas-stage-bootloader.yaml @@ -86,13 +86,40 @@ steps: mount --bind /boot/efi {{ newroot }}/boot/efi when: "! mountpoint -q {{ newroot }}/boot" - - name: Bind the kernel filesystems into the chroot + # ⚠⚠ --make-rslave IS LOAD-BEARING. Without it this cost a production outage on + # 2026-08-18. + # + # On a systemd host `/` has SHARED mount propagation, so `mount --rbind /dev` + # creates a bind that shares propagation with the original. Every later + # `umount -R` of the chroot copy then propagates BACK to the live system and + # unmounts the REAL /sys/fs/cgroup, /dev/pts and /dev/shm. With cgroup2 gone, + # systemd-logind cannot create a session: sshd still completes authentication + # and already-resident daemons keep serving from memory, but every new exec + # hangs forever. The host looks alive and is unusable, and — this is the part + # that wasted the most time — it looks exactly like failing root-disk I/O. + # + # --make-rslave makes propagation one-way: host -> chroot only. Teardown then + # cannot reach back. + - name: Bind the kernel filesystems into the chroot (SLAVE propagation) shell: | - for d in dev dev/pts proc sys; do + for d in dev proc sys; do mountpoint -q {{ newroot }}/$d || mount --rbind /$d {{ newroot }}/$d + mount --make-rslave {{ newroot }}/$d done + echo "--- propagation (must NOT say shared) ---" + findmnt -o TARGET,PROPAGATION {{ newroot }}/dev {{ newroot }}/sys {{ newroot }}/proc changed_when: "true" + - name: GUARD — refuse to continue if any chroot bind is still shared + shell: | + if findmnt -no PROPAGATION -R {{ newroot }}/dev {{ newroot }}/sys {{ newroot }}/proc \ + 2>/dev/null | grep -q shared; then + echo "chroot binds are SHARED — teardown would unmount the live host's /sys and /dev" + exit 1 + fi + echo "all chroot binds are private/slave — teardown cannot propagate back" + changed_when: "false" + # ---------- build the boot artifacts inside the chroot ---------- - name: Pin the default boot entry to the rollback, not to ZFS diff --git a/servers/esh-pve-nas/README.md b/servers/esh-pve-nas/README.md index 052f628..c3b0e93 100644 --- a/servers/esh-pve-nas/README.md +++ b/servers/esh-pve-nas/README.md @@ -16,24 +16,34 @@ Media services on this box sit at `10.0.50.56` (Plex) and `10.0.50.57` (Jellyfin - **CPU:** Intel Xeon W-1250 @ 3.30 GHz - **RAM:** 125.6 GB - **Kernel:** `6.8.12-13-pve` (Proxmox 8.x) -- **Storage:** ⚠ **PVE root is on a USB Disk-on-Module** (`sdq`, 7.3 GB, `ID_BUS=usb`, - NORELSYS 1081), whose 512 MB ESP is the host's **only** boot path. Not a wear - problem (a DOM is SLC/pSLC), but it is on the USB bus, has no headroom, and is - unmirrored. Mitigated 2026-08-17 (90% → 76%: journald capped, journal relocated - to ZFS). **Migration STAGED 2026-08-18 — awaiting only the reboot window**; - runbook [`docs/runbooks/esh-pve-nas-boot-migration.md`](../../docs/runbooks/esh-pve-nas-boot-migration.md). +- **Storage: root is `nvme/ROOT/pve-1` on the mirrored NVMe pool** since the + 2026-08-18 migration. The USB Disk-on-Module (7.3 GB, `ID_BUS=usb`, NORELSYS + 1081) still holds the ESP and `/boot`, but is **out of the runtime I/O path** — + a USB bus reset no longer drops root from under a running hypervisor. Runbook: + [`docs/runbooks/esh-pve-nas-boot-migration.md`](../../docs/runbooks/esh-pve-nas-boot-migration.md). - DOM LVM layout after staging (was root 6.04 G + swap 768 M): + DOM LVM layout: | LV | size | role | |---|---|---| - | `pve-root` | 6.04 G | ext4 — the **live root today**, and the rollback after cutover | - | `pve-boot` | 512 M | ext4 — the new `/boot`, carved out of swap | + | `pve-root` | 6.04 G | ext4 — **the rollback**, intact and unmounted, keeps its own kernel + initrd | + | `pve-boot` | 512 M | ext4 — `/boot`, carved out of swap | | `pve-swap` | 256 M | swap, shrunk to make room | - Post-cutover root is `nvme/ROOT/pve-1` on the mirrored NVMe pool. `/boot` stays - ext4 on the DOM on purpose: GRUB cannot read the `nvme` pool, which has - `encryption`, `large_dnode` and `zstd_compress` enabled. + `/boot` stays ext4 on the DOM on purpose: GRUB cannot read the `nvme` pool, + which has `encryption`, `large_dnode` and `zstd_compress` enabled. + + ⚠ **Device letters are not stable** — the DOM was `sdq` before the reboot and + `sdl` after. `fstab` uses `/dev/pve/*` and UUIDs; never write a rule against a + bare `sdX` on this host. + + ⚠ **There is no auto-fallback if a boot fails.** grubenv lives on an LVM LV, + which GRUB can read but not write, so `grub-reboot`'s one-shot degrades to a + sticky default (verified 2026-08-18: `next_entry` survived the boot that + consumed it). Steady state is `saved_entry=pve-zfs-root` with no `next_entry`. + A failed boot needs the console — and this box has **no IPMI, no BMC, no serial + console**. Recovery is selecting `Proxmox VE - ROLLBACK: ext4 root on the USB + DOM` at the GRUB menu. - ⚠ **Never set the ZFS cachefile on one pool.** `zpool set cachefile=…` flips the host from import-by-scan to import-by-cache; a cache holding only `nvme` leaves `ssd` and `tank` unimported at boot, which empties every CT 103 export. Set it on @@ -41,11 +51,23 @@ Media services on this box sit at `10.0.50.56` (Plex) and `10.0.50.57` (Jellyfin - **Pools:** `nvme` (2× 931 GB NVMe mirror — 32 G used, 867 G free, holds every guest rootfs), `ssd` (4× 894 GB Intel SATA, 2 mirrors — 1.42 T free), `tank` (12× 14.6 TB raidz2 ×2 — 40 T of 175 T). Plus NFS `/mnt/pve/tank-vmbu` for VM backups. -- ⚠ **CT 103 `esh-nas` (10.0.50.50) runs on THIS host and serves `hard` NFS** to - esh-docker-vm (`/mnt/books`, `/mnt/backup`) and to esh-pve (`/mnt/pve/esh-nas`, - `/mnt/pve/tank-vmbu`). Taking this box down stalls esh-pve's storage layer and - wedges esh-docker-vm into unkillable D-state unless those clients are quiesced - first. Never reboot this host casually. +- ⚠ **CT 103 `esh-nas` (10.0.50.50) runs on THIS host and serves `hard` NFS.** + Never reboot this host casually — quiesce the clients first. Measured from the + server on 2026-08-18 (`ss -tn '( sport = :2049 )'` inside CT 103), there are + **five** clients, not the two long documented here: + + | client | mounts | opts | + |---|---|---| + | esh-docker-vm `10.0.50.45` | `/mnt/books`, `/mnt/backup` | **hard** — quiesce | + | esh-pve `10.0.250.35` | `/mnt/pve/esh-nas`, `/mnt/pve/tank-vmbu` | **hard** — quiesce | + | esh-vm-db `10.0.50.60` | `/mnt/backup` | **hard** — no ssh; reach it via `qm guest exec 101` on esh-pve | + | vm-esh-nas `10.0.50.154` | — | is VM 104 *on this host*; stops with it | + | nh3-dev `10.100.10.50` | `/mnt/books` | soft,ro — safe, errors rather than blocks | + + **Ask the server who its clients are; don't trust this table.** It rots. `ss` on + CT 103 is ground truth, and that is how the three undocumented clients surfaced. + Playbooks: `esh-cutover-1-quiesce-docker-vm.yaml`, + `esh-cutover-2-quiesce-esh-pve.yaml`, `esh-cutover-5-restore-docker-vm.yaml`. > **Root-fs pressure:** 77% of a 5.9 GB root (1.3 GB free) after the 2026-08-17 > mitigation. Still not enough for the pending upgrade — **225 packages, 161