feat(esh-pve-nas): cut PVE root over to ZFS on the mirrored NVMe
Root is now nvme/ROOT/pve-1. The USB DOM keeps the ESP and /boot but is out of the runtime I/O path, so a bus reset can no longer drop root from under a running hypervisor. All five guests healthy, three pools ONLINE, system running, ext4 pve-root intact and unmounted as the rollback with its own kernel and initrd. zfs-import-cache is now the active import path -- the all-three-pools cachefile fix doing its job. The window cost an unplanned outage, and the cause was this repo's own tooling rather than the migration. The staging chroot ran `mount --rbind /dev` and /sys with no --make-rslave. On systemd `/` has shared propagation, so the cutover's `umount -R` propagated back into the live host and removed the real /sys/fs/cgroup, /dev/pts and /dev/shm. With cgroup2 gone systemd-logind could not create a session: ping fine, TCP fine, SSH authentication succeeded, resident daemons kept serving -- and every new exec hung, including /sbin/reboot, so the reboot never ran at all. It impersonates failing root-disk I/O almost perfectly, and I called it as the DOM dying. That was wrong. dmesg had the answer throughout: the DOM attached cleanly with no errors, and the last log timestamp was 12114881s -- 140 days -- meaning this was still the original boot. A down-detector had also never reported the host down, which I read as a fast reboot rather than as no reboot. Fixes and guards: - --make-rslave after every rbind, plus a guard that refuses to proceed while any chroot bind still reports shared propagation. - Confirm a reboot by observing the host DOWN, not by watching for it to come back. Those two states are indistinguishable otherwise. - Blast radius now measured from the server: `ss` inside CT 103 found five NFS clients, not the two documented. The new one that mattered is esh-vm-db, hard-mounted and unreachable by ssh. Left mounted on purpose and it came through read-write. - grub-reboot's one-shot does NOT work here: grubenv sits on an LVM LV which GRUB reads but cannot write, so next_entry survived the boot that consumed it. Steady state is saved_entry=pve-zfs-root with no next_entry. There is no auto-fallback on this host and no IPMI. Recovery needed no console: an idempotent cgroup2/devpts/shm remount landed in the brief windows where exec succeeded. No data was lost, and neither the DOM nor any pool was ever at risk.
This commit is contained in:
@@ -1,9 +1,21 @@
|
||||
# esh-pve-nas — moving PVE root off the USB DOM
|
||||
|
||||
**Status: STAGED — everything but the reboot is done.** Mitigation landed
|
||||
2026-08-17 (root 90% → 76%); staging landed 2026-08-18. The host is still
|
||||
running from the ext4 root on the DOM and its boot path is unchanged; what
|
||||
remains is the cutover window (§ Cutover).
|
||||
**Status: DONE — cut over 2026-08-18.** Root is `nvme/ROOT/pve-1` on the mirrored
|
||||
NVMe; `/boot` is ext4 on the DOM; the DOM is out of the runtime I/O path. All five
|
||||
guests healthy, all three pools ONLINE, `systemctl is-system-running` = `running`.
|
||||
The ext4 root (`pve-root`) is intact, unmounted, and still carries its own kernel
|
||||
and initrd as the rollback.
|
||||
|
||||
Post-cutover boot config: `saved_entry=pve-zfs-root`, no `next_entry`. If grubenv
|
||||
were ever unreadable GRUB falls through to menu entry 0, which the
|
||||
`/etc/default/grub.d/zfs-root.cfg` drop-in also points at `root=ZFS=nvme/ROOT/pve-1`
|
||||
— so every path boots ZFS.
|
||||
|
||||
⚠ **The window cost an unplanned outage, caused by a bug in this runbook's own
|
||||
tooling, not by the migration.** Read § The mount-propagation incident before
|
||||
running anything like this again. Two other findings — the blast radius being
|
||||
more than double what was documented, and the one-shot rollback not actually
|
||||
working — are recorded in § The pool-name bug's neighbours below.
|
||||
|
||||
Staging is two playbooks, both rerunnable:
|
||||
|
||||
@@ -255,6 +267,93 @@ Assert the effective value, not the presence of a substring.
|
||||
|
||||
**Cutover** — the remaining work, § Cutover below.
|
||||
|
||||
## ⚠ The mount-propagation incident — the expensive lesson of 2026-08-18
|
||||
|
||||
**What broke.** The staging chroot was built with `mount --rbind /dev` and `/sys`
|
||||
and **no `--make-rslave`**. On a systemd host `/` has *shared* propagation, so
|
||||
those binds propagate in both directions. When the cutover tore the chroot down
|
||||
with `umount -R`, the unmounts **propagated back into the live host** and removed
|
||||
the real `/sys/fs/cgroup`, `/dev/pts` and `/dev/shm`.
|
||||
|
||||
With cgroup2 gone, `systemd-logind` could no longer create a session. The result
|
||||
is a host that:
|
||||
|
||||
- answers ping, accepts TCP, and **completes SSH authentication**
|
||||
- keeps serving from daemons already resident in memory (`pveproxy` returned a
|
||||
clean HTTP 401 throughout)
|
||||
- **hangs on every new `exec`**, including `/sbin/reboot` — so the reboot that was
|
||||
supposed to end the window never ran
|
||||
|
||||
**Why it cost so much time: it is a near-perfect impostor of failing root-disk
|
||||
I/O.** Both present as "host is up, daemons answer, nothing new can start." The
|
||||
session diagnosed it as the USB DOM dying and told the operator to walk to the
|
||||
machine. That was wrong, and the operator caught it: the DOM had been reliable
|
||||
for years and the wedge began immediately after a change.
|
||||
|
||||
**The evidence that settles it, and was available the whole time** — from
|
||||
`dmesg`, obtainable in the brief windows when exec did succeed:
|
||||
|
||||
| line | says |
|
||||
|---|---|
|
||||
| `[16.00] sd 56:0:0:0: [sdq] Attached SCSI removable disk` | DOM enumerated **cleanly, no errors** |
|
||||
| `[12114881.98] systemd[1]: nvme-varlog-stage.mount: Deactivated` | timestamp is **140 days** — this is the ORIGINAL boot |
|
||||
|
||||
That second line is the whole answer: **the machine never rebooted.** A
|
||||
down-detector loop had also never once reported the host down; that was read as a
|
||||
fast reboot rather than as no reboot at all.
|
||||
|
||||
**Rules that follow:**
|
||||
|
||||
1. **Always `mount --make-rslave` after `mount --rbind` into a chroot.** Phase 2
|
||||
now does this and carries a guard that refuses to continue if any bind still
|
||||
reports `shared` propagation.
|
||||
2. **A reboot is not confirmed until the host is observed DOWN.** Poll for
|
||||
disappearance, not just for reappearance. "Never went down" and "went down and
|
||||
came back fast" are indistinguishable if you only watch for the host to answer.
|
||||
3. **Before blaming hardware for a wedge that began right after a change, get
|
||||
`dmesg` and check the boot timestamp.** Diagnose the change first; hardware is
|
||||
the explanation of last resort, not first.
|
||||
|
||||
**Recovery took no console access.** Windows where `exec` briefly succeeded were
|
||||
enough to land an idempotent remount of cgroup2 / devpts / shm, after which
|
||||
`systemctl reset-failed` returned the host to `running`. Total data loss: none.
|
||||
The root filesystem, the DOM and all three pools were never at risk — this was a
|
||||
mount-namespace fault, not a storage one.
|
||||
|
||||
## The pool-name bug's neighbours — two more corrections
|
||||
|
||||
**The blast radius was more than double what was documented.** The runbook named
|
||||
two NFS dependents. `ss -tn '( sport = :2049 )'` inside CT 103 showed **five**:
|
||||
|
||||
| client | mount | disposition |
|
||||
|---|---|---|
|
||||
| `10.0.50.45` esh-docker-vm | `/mnt/books`, `/mnt/backup` — **hard** | quiesced |
|
||||
| `10.0.250.35` esh-pve | `esh-nas`, `tank-vmbu` — **hard** | quiesced |
|
||||
| `10.0.50.60` **esh-vm-db** | `/mnt/backup` — **hard** | **left mounted deliberately** |
|
||||
| `10.0.50.154` vm-esh-nas | — | is VM 104 *on this host*; stops with it |
|
||||
| `10.100.10.50` nh3-dev | `/mnt/books` — **soft,ro** | safe, errors instead of blocking |
|
||||
|
||||
Ask the *server* who its clients are. A runbook's list of dependents is a snapshot
|
||||
that rots; `ss` on the NFS server is ground truth.
|
||||
|
||||
esh-vm-db was left mounted on purpose and **came through read-write** — a `hard`
|
||||
mount with no active user blocks and resumes, which is what `hard` is for. Its
|
||||
backup timers were ~19h out, and unmounting would have meant an unmount/remount
|
||||
cycle over the qemu guest agent on a host with no ssh access.
|
||||
|
||||
**The one-shot rollback does not work, and the warning was right.**
|
||||
`grub-reboot` printed *"Detected GRUB environment block on lvm device — will
|
||||
remain the default boot entry until manually cleared."* Confirmed empirically:
|
||||
after the successful ZFS boot, `next_entry=pve-zfs-root` was **still set**. GRUB
|
||||
can read grubenv on LVM but cannot write it, so `boot_once` degrades to a sticky
|
||||
default. **There is no auto-fallback on this host.** A failed boot must be
|
||||
corrected at the console.
|
||||
|
||||
The steady-state config therefore does not rely on it: `saved_entry=pve-zfs-root`
|
||||
with `next_entry` cleared. Restoring a real one-shot would mean relocating grubenv
|
||||
onto the ESP (vfat on a plain partition, which GRUB *can* write) — parked, not
|
||||
required.
|
||||
|
||||
## Cutover
|
||||
|
||||
The only remaining work. Everything below the quiesce is minutes.
|
||||
|
||||
@@ -133,7 +133,10 @@ _As of 2026-08-18 — **gen seat SWAPPED to `absolute-heresy` and operator-confi
|
||||
|
||||
- **⏳ WATCHING: DavidAU's HERETIC build of Qwen3.8-27B — the one worth waiting for.** `DavidAU/Qwen3.8-27B-Cold-Fusion-GAIN-V1.1` examined 2026-08-17 and **NOT adopted**: it is a capability/efficiency finetune of **stock** Qwen3.8 and every bench row on its card is labelled **`[non heretic]`** — adopting it would reintroduce base refusals the current seat does not have. ⚠ **Easy to misread as uncensored** (operator did, and it is a fair mistake): DavidAU's back catalog is almost all `Uncensored-Heretic` builds — Fable-Fusion 711, Qwen3.5-9B Cold-Fusion — so the naming pattern implies it. This one simply has not had that stage run yet; the card's roadmap says the HERETIC version is **IN PROGRESS from base**. **That** is the release to watch, not this one. What makes it worth watching: third-party benches (Nightmedia, mxfp8) beat stock Qwen3.8 by **+0.064 arc/c, +0.056 arc/e, +0.056 obkqa**; claimed **MTP acceptance 55.7% (record 59.9%)** vs our measured 47.2%; **thinking tokens cut to 1/10–1/2**; PPL *dropped* vs base. Same GAIN/Cold-Fusion pipeline that produced Fable-Fusion 711, which we already serve on `char-rp-reasoning` — proven in-fleet, not just claimed. Structurally clean (1199 tensors, 15 mtp in shard 18, 333 visual). ⚠ MTP/speed figures are **GGUF/llama.cpp on a 5090, not vLLM** — may not transfer; and because its MTP head was likely *trained*, the free CPU-hash shortcut would NOT apply (it won't match base) so a real acceptance gate would be needed.
|
||||
|
||||
- **🟢 esh-pve-nas — migration STAGED 2026-08-18; ONLY THE REBOOT WINDOW REMAINS.** PVE root ran off a **NORELSYS USB Disk-on-Module** (`sdq`). Wear was NOT the driver (a DOM is SLC/pSLC — operator corrected my first read); the drivers are the **USB bus** (a reset drops root under a running hypervisor), no headroom, no mirror, and **blocked patching: 225 packages, 161 with `deb12uN`/security bumps**, stuck on PVE 8.4.11 vs esh-pve's 8.4.14. Mitigated 2026-08-17 (90% → 76%); **staged 2026-08-18 via two rerunnable elway playbooks** (`esh-pve-nas-stage-zfs-root.yaml`, `esh-pve-nas-stage-bootloader.yaml`), 0 failed, 17/17 verify green.
|
||||
- **🟢 esh-pve-nas — MIGRATION DONE 2026-08-18. Root is `nvme/ROOT/pve-1` on mirrored NVMe; the USB DOM is out of the runtime I/O path.** All five guests healthy, three pools ONLINE, system `running`, ext4 `pve-root` intact+unmounted as rollback with its own kernel. Boot config: `saved_entry=pve-zfs-root`, no `next_entry`; if grubenv were unreadable GRUB falls to entry 0 which the `/etc/default/grub.d/zfs-root.cfg` drop-in also points at ZFS. **`zfs-import-cache.service` is now the active import path** (the all-three-pools cachefile fix working as intended); vestigial `zfs-import@nvme.service` disabled — it failed every boot as redundant. ⚠ **Device letters shift across reboots** (DOM was `sdq`, now `sdl`) — never key anything to a bare `sdX` here. ⚠ **NO auto-fallback on a failed boot, and no IPMI/BMC/serial** — grubenv on LVM is readable but not writable by GRUB, so `grub-reboot`'s one-shot degrades to a sticky default (verified: `next_entry` survived the boot that consumed it). Recovery = pick the ROLLBACK entry at the console. **⏳ REMAINING: second reboot to prove repeatability (needs another quiesce window), refresh the DOM image since `/boot` changed, then patch the 225 pending packages — the whole point of the migration.**
|
||||
⚠⚠ **THE WINDOW COST AN UNPLANNED OUTAGE, caused by our own tooling, not the migration.** The staging chroot did `mount --rbind /dev` + `/sys` with **no `--make-rslave`**; on systemd `/` is *shared*, so the cutover's `umount -R` **propagated back into the live host** and stripped the real `/sys/fs/cgroup`, `/dev/pts`, `/dev/shm`. logind could then create no sessions: ping fine, TCP fine, **SSH authenticates**, resident daemons keep serving (pveproxy returned clean 401s) — and **every new exec hangs, including `/sbin/reboot`**, so the reboot never ran. **It is a near-perfect impostor of failing root-disk I/O**, and I misdiagnosed it as the USB DOM dying and told the operator to walk to the machine. **Operator caught it** — the DOM had been fine for years and the wedge began right after a change. The settling evidence was in `dmesg` all along: `[16.00] [sdq] Attached SCSI removable disk` (clean, no errors) and a last-line timestamp of **`[12114881]` = 140 days = the ORIGINAL boot** — the machine had never rebooted. My down-detector never once reported the host down and I read that as a fast reboot rather than *no* reboot. Recovered with **no console access** by hammering an idempotent cgroup2/devpts/shm remount into the brief windows where exec succeeded. Zero data loss. **RULES: (1) always `--make-rslave` after `--rbind` (playbook now guards on `PROPAGATION != shared`); (2) a reboot is not confirmed until the host is observed DOWN — poll for disappearance, not reappearance; (3) before blaming hardware for a wedge that started right after a change, get `dmesg` and check the boot timestamp.**
|
||||
⚠ **Blast radius was more than DOUBLE what the runbook said** — `ss -tn '( sport = :2049 )'` in CT 103 found **five** NFS clients, not two: +**esh-vm-db `10.0.50.60`** (`/mnt/backup`, **hard**, no ssh — reach via `qm guest exec 101` on esh-pve), +vm-esh-nas (is VM 104 on the host itself), +nh3-dev (soft,ro — safe). **Ask the server who its clients are; a documented dependent list rots.** esh-vm-db was deliberately left mounted (backup target, timers ~19h out) and **came through read-write** — a `hard` mount with no active user blocks and resumes, which is what `hard` is for.
|
||||
_(historical, pre-cutover context below)_ **migration STAGED 2026-08-18.** PVE root ran off a **NORELSYS USB Disk-on-Module** (`sdq`). Wear was NOT the driver (a DOM is SLC/pSLC — operator corrected my first read); the drivers are the **USB bus** (a reset drops root under a running hypervisor), no headroom, no mirror, and **blocked patching: 225 packages, 161 with `deb12uN`/security bumps**, stuck on PVE 8.4.11 vs esh-pve's 8.4.14. Mitigated 2026-08-17 (90% → 76%); **staged 2026-08-18 via two rerunnable elway playbooks** (`esh-pve-nas-stage-zfs-root.yaml`, `esh-pve-nas-stage-bootloader.yaml`), 0 failed, 17/17 verify green.
|
||||
**Host is still on ext4 and its boot path is byte-identical to the last 140 days** — `grub-install` is deliberately held back to the window, so an unplanned reboot mid-staging is a non-event. **CUTOVER = quiesce NFS clients → stop guests → `grub-install` → `grub-reboot pve-zfs-root` → `zfs set mountpoint=/ nvme/ROOT/pve-1` → reboot.**
|
||||
⚠⚠ **THE POOL-NAME BUG — the near-miss worth remembering.** `update-grub` silently emits `root=ZFS=/ROOT/pve-1` with **the pool name missing** → boots to an initramfs prompt. Cause: Debian's `10_linux` builds `${rpool}${bootfs}` where `rpool = grub-probe --target=fs_label`, and **GRUB's ZFS reader cannot open a pool with `encryption`/`large_dnode`/`zstd_compress`** — so it returns empty and the failure is swallowed by `2>/dev/null || true`. The *same* feature set that forced `/boot` to stay ext4 also corrupts the kernel cmdline, which the design did not anticipate. Fixed with a `/etc/default/grub.d/zfs-root.cfg` drop-in (last `root=` wins) plus explicit `pve-zfs-root` / `pve-ext4-rollback` entries with stable ids. **Durable lesson: assert the EFFECTIVE value, not the presence of a substring** — the original verify grepped for the right string *appearing somewhere* and would have passed with every menu entry still broken.
|
||||
⚠ **Never set the ZFS cachefile on one pool.** The runbook's `zpool set cachefile=… nvme` was a trap: populating a cache flips the host from import-by-scan to import-by-cache, so a cache holding only `nvme` leaves `ssd`+`tank` unimported and empties every CT 103 export. Set on all three 2026-08-18, verified in the 11,976-byte cache.
|
||||
|
||||
@@ -0,0 +1,55 @@
|
||||
# esh-pve-nas cutover, step 1 of 5 — quiesce esh-docker-vm's hard NFS mounts.
|
||||
#
|
||||
# Run: scripts/elway infra-ops@10.0.50.45 --playbook playbooks/esh-cutover-1-quiesce-docker-vm.yaml
|
||||
#
|
||||
# Why this is first and why it is not optional: /mnt/books and /mnt/backup are
|
||||
# `hard` NFS from CT 103 on esh-pve-nas. A hard mount does not fail when the
|
||||
# server goes away — it blocks forever in D-state, and the only known remedy is
|
||||
# rebooting THIS host. /mnt/books was deliberately left hard because calibre's
|
||||
# SQLite risks corruption under `soft`, so the mount option is not the fix; the
|
||||
# quiesce is.
|
||||
#
|
||||
# Measured 2026-08-18: exactly one container binds these paths
|
||||
# (calibre-web-automated -> /mnt/books/calibre/{ingest,calibre_library}) and
|
||||
# /mnt/backup has no container consumers at all. The blast radius is one service,
|
||||
# not the seventeen containers on this host.
|
||||
#
|
||||
# Reversed by playbooks/esh-cutover-5-restore.yaml.
|
||||
|
||||
steps:
|
||||
# No --format here: elway substitutes {{ ... }}, so Go template braces in a
|
||||
# shell command are a booby trap. --filter + -q avoids them entirely.
|
||||
- name: Stop the only container holding the NFS mounts
|
||||
shell: sudo -n docker stop calibre-web-automated
|
||||
when: "test -n \"$(sudo -n docker ps -q --filter name=^calibre-web-automated$)\""
|
||||
|
||||
- name: Confirm nothing else has files open under the mounts
|
||||
shell: |
|
||||
busy=$(sudo -n lsof +D /mnt/books +D /mnt/backup 2>/dev/null | tail -n +2 | wc -l)
|
||||
if [ "$busy" -ne 0 ]; then
|
||||
echo "STILL BUSY — refusing to unmount:"
|
||||
sudo -n lsof +D /mnt/books +D /mnt/backup 2>/dev/null | head -20
|
||||
exit 1
|
||||
fi
|
||||
echo "no open files under either mount"
|
||||
changed_when: "false"
|
||||
|
||||
- name: Unmount /mnt/books
|
||||
shell: sudo -n umount /mnt/books
|
||||
when: "mountpoint -q /mnt/books"
|
||||
|
||||
- name: Unmount /mnt/backup
|
||||
shell: sudo -n umount /mnt/backup
|
||||
when: "mountpoint -q /mnt/backup"
|
||||
|
||||
verify:
|
||||
- name: Neither NFS mount remains
|
||||
shell: "! findmnt -t nfs,nfs4 -o TARGET | grep -qE '/mnt/(books|backup)'"
|
||||
changed_when: "false"
|
||||
|
||||
- name: The other sixteen containers are still up
|
||||
shell: |
|
||||
n=$(sudo -n docker ps -q | wc -l)
|
||||
echo "$n containers still running"
|
||||
test "$n" -ge 10
|
||||
changed_when: "false"
|
||||
@@ -0,0 +1,51 @@
|
||||
# esh-pve-nas cutover, step 2 of 5 — quiesce esh-pve's hard NFS storages.
|
||||
#
|
||||
# Run: scripts/elway root@10.0.250.35 --playbook playbooks/esh-cutover-2-quiesce-esh-pve.yaml
|
||||
#
|
||||
# esh-pve mounts two `hard` NFS storages from CT 103 on esh-pve-nas:
|
||||
# esh-nas -> 10.0.50.50:/mnt/pvestore at /mnt/pve/esh-nas
|
||||
# tank-vmbu -> 10.0.50.50:/mnt/tank-vmbu at /mnt/pve/tank-vmbu
|
||||
#
|
||||
# Disabling the storage first matters: if the storage stays enabled, pvestatd
|
||||
# keeps stat()ing the path and will re-trigger the mount (and then block on it)
|
||||
# the moment the server disappears. Disable, THEN unmount.
|
||||
#
|
||||
# Measured 2026-08-18: esh-nas holds 2.9 MB of 96 TB and no running guest has a
|
||||
# disk on either storage — all three (100 esh-vm-docker, 101 esh-vm-db,
|
||||
# 102 esh-vm-workstation) live on local-lvm. So this quiesce costs backup targets
|
||||
# for the duration, not guest availability. Guests are deliberately left running.
|
||||
#
|
||||
# Reversed by playbooks/esh-cutover-5-restore.yaml.
|
||||
|
||||
steps:
|
||||
- name: Disable the esh-nas storage so pvestatd stops touching it
|
||||
shell: pvesm set esh-nas --disable 1
|
||||
when: "pvesm status 2>/dev/null | awk '$1==\"esh-nas\"{print $3}' | grep -q active"
|
||||
|
||||
- name: Disable the tank-vmbu storage
|
||||
shell: pvesm set tank-vmbu --disable 1
|
||||
when: "grep -q '^nfs: tank-vmbu' /etc/pve/storage.cfg && ! grep -A8 '^nfs: tank-vmbu' /etc/pve/storage.cfg | grep -q 'disable'"
|
||||
|
||||
- name: Give pvestatd a moment to let go before unmounting
|
||||
shell: sleep 5
|
||||
changed_when: "false"
|
||||
|
||||
- name: Unmount /mnt/pve/esh-nas
|
||||
shell: umount /mnt/pve/esh-nas || umount -l /mnt/pve/esh-nas
|
||||
when: "mountpoint -q /mnt/pve/esh-nas"
|
||||
|
||||
- name: Unmount /mnt/pve/tank-vmbu
|
||||
shell: umount /mnt/pve/tank-vmbu || umount -l /mnt/pve/tank-vmbu
|
||||
when: "mountpoint -q /mnt/pve/tank-vmbu"
|
||||
|
||||
verify:
|
||||
- name: Neither esh-nas-backed NFS mount remains
|
||||
shell: "! findmnt -t nfs,nfs4 -o SOURCE | grep -q '10\\.0\\.50\\.50'"
|
||||
changed_when: "false"
|
||||
|
||||
- name: All three guests are still running
|
||||
shell: |
|
||||
n=$(qm list | awk 'NR>1 && $3=="running"' | wc -l)
|
||||
echo "$n VMs running"
|
||||
test "$n" -eq 3
|
||||
changed_when: "false"
|
||||
@@ -0,0 +1,154 @@
|
||||
# esh-pve-nas cutover, step 3 of 5 — point the ESP at the new /boot and reboot.
|
||||
#
|
||||
# Run: scripts/elway root@esh-pve-nas --playbook playbooks/esh-cutover-3-esh-pve-nas.yaml
|
||||
#
|
||||
# PRECONDITION: steps 1 and 2 must have run. Both NFS clients hold `hard` mounts
|
||||
# from CT 103 which lives on this host; taking it down with them mounted wedges
|
||||
# esh-docker-vm in unkillable D-state. A guard below refuses to proceed if either
|
||||
# client is still mounted.
|
||||
#
|
||||
# ⚠ ORDERING TRAP, and it is the reason this is a playbook and not four commands:
|
||||
# `zfs set mountpoint=/` on a dataset that is CURRENTLY MOUNTED makes ZFS unmount
|
||||
# and REMOUNT it at the new location — i.e. it would try to mount the ZFS root
|
||||
# over the live ext4 root of a running hypervisor. canmount=noauto does not save
|
||||
# you; that governs automatic mounting at import, not an explicit property change
|
||||
# on a mounted dataset. The dataset must be UNMOUNTED first, which means the
|
||||
# chroot binds have to come down first, which means grub-install and grub-reboot
|
||||
# have to happen BEFORE any of that. Hence the sequence below is not negotiable.
|
||||
#
|
||||
# This playbook ENDS BY REBOOTING THE HOST. elway will lose the connection; that
|
||||
# is expected, not a failure.
|
||||
|
||||
vars:
|
||||
newroot: /mnt/newroot
|
||||
root_dataset: nvme/ROOT/pve-1
|
||||
quiesced: "no" # caller MUST pass --var quiesced=yes after verifying both clients
|
||||
|
||||
steps:
|
||||
# ---------- guards ----------
|
||||
|
||||
- name: GUARD — still on the ext4 root (not already cut over)
|
||||
shell: |
|
||||
test "$(findmnt -no FSTYPE /)" = "ext4" || { echo "already on ZFS; refusing"; exit 1; }
|
||||
changed_when: "false"
|
||||
|
||||
# This host has no ssh keys to the NFS clients, so the caller verifies their
|
||||
# mount tables and attests via --var quiesced=yes.
|
||||
#
|
||||
# ⚠ THE RUNBOOK'S BLAST RADIUS WAS WRONG. It named two dependents. `ss` on CT 103
|
||||
# showed FIVE distinct clients on 2026-08-18:
|
||||
# 10.0.50.45 esh-docker-vm hard -> quiesced by step 1
|
||||
# 10.0.250.35 esh-pve hard -> quiesced by step 2
|
||||
# 10.0.50.60 esh-vm-db hard -> DELIBERATELY LEFT MOUNTED (see below)
|
||||
# 10.0.50.154 vm-esh-nas n/a -> is VM 104 on THIS host; dies with it
|
||||
# 10.100.10.50 nh3-dev soft,ro -> errors instead of blocking; safe
|
||||
#
|
||||
# esh-vm-db is left mounted on purpose. It is a backup TARGET with no live user:
|
||||
# resticprofile-backup and postgresql-dump next fire ~19h out, and a hard mount
|
||||
# with nothing actively using it blocks and then resumes when the server returns
|
||||
# — that is what `hard` is for. Unmounting it would mean an unmount/remount cycle
|
||||
# over the qemu guest agent on a host with no ssh access, where a failed remount
|
||||
# breaks backups silently. Leaving it is the lower-risk branch, not the lazy one.
|
||||
# The gate is `quiesced`, which the CALLER sets only after checking each client's
|
||||
# mount table directly (this host has no ssh to them; see step 1/2 playbooks).
|
||||
#
|
||||
# ⚠ It deliberately does NOT gate on server-side NFS session count. Measured
|
||||
# 2026-08-18: esh-docker-vm's sessions drained within ~90s, but esh-pve held 11
|
||||
# established connections to :2049 indefinitely with NO mounts in either
|
||||
# `findmnt` or `/proc/mounts` and nothing holding a cwd there. That is the Linux
|
||||
# NFSv4 client keeping its transport alive past the last unmount, and it is the
|
||||
# wrong thing to gate on: the failure this whole runbook exists to prevent is a
|
||||
# process blocking on a MOUNTED hard filesystem when the server vanishes. With no
|
||||
# mount there is nothing to block on — an idle socket to a departing server just
|
||||
# resets. Gating on sessions would have stalled the window forever on a condition
|
||||
# that never clears and never mattered.
|
||||
- name: GUARD — caller has confirmed both hard-NFS clients are unmounted
|
||||
shell: |
|
||||
test "{{ quiesced }}" = "yes" || {
|
||||
echo "run playbooks 1 and 2 and confirm client mount tables first"; exit 1; }
|
||||
echo "caller attests: esh-docker-vm and esh-pve carry no esh-nas mounts"
|
||||
echo "--- server-side sessions, informational only ---"
|
||||
pct exec 103 -- ss -tnH state established '( sport = :2049 )' 2>/dev/null \
|
||||
| awk '{print $4}' | sed 's/:[0-9]*$//' | sort | uniq -c || true
|
||||
changed_when: "false"
|
||||
|
||||
- name: GUARD — staging artifacts are all present
|
||||
shell: |
|
||||
mountpoint -q {{ newroot }} || { echo "{{ newroot }} not mounted"; exit 1; }
|
||||
mountpoint -q {{ newroot }}/boot || { echo "boot LV not in the chroot"; exit 1; }
|
||||
grep -q pve-zfs-root {{ newroot }}/boot/grub/grub.cfg || { echo "no ZFS entry"; exit 1; }
|
||||
grep -q 'saved_entry=pve-ext4-rollback' {{ newroot }}/boot/grub/grubenv || { echo "grubenv not pinned to rollback"; exit 1; }
|
||||
changed_when: "false"
|
||||
|
||||
# ---------- stop the guests, NAS last ----------
|
||||
|
||||
- name: Stop the guests (reverse of startup order — CT 103, the NAS, goes last)
|
||||
shell: |
|
||||
for v in 105 106 107; do pct status $v 2>/dev/null | grep -q running && pct shutdown $v --timeout 90 || true; done
|
||||
qm status 104 2>/dev/null | grep -q running && qm shutdown 104 --timeout 90 || true
|
||||
for i in $(seq 1 30); do
|
||||
running=$( (pct list | awk 'NR>1 && $2=="running"'; qm list | awk 'NR>1 && $3=="running"') | wc -l )
|
||||
[ "$running" -le 1 ] && break
|
||||
sleep 3
|
||||
done
|
||||
pct status 103 2>/dev/null | grep -q running && pct shutdown 103 --timeout 90 || true
|
||||
sleep 3
|
||||
echo "--- remaining ---"; pct list; qm list
|
||||
changed_when: "true"
|
||||
|
||||
# ---------- the actual cutover ----------
|
||||
|
||||
- name: Point the ESP at the new /boot LV
|
||||
shell: |
|
||||
chroot {{ newroot }} grub-install --target=x86_64-efi \
|
||||
--efi-directory=/boot/efi --bootloader-id=proxmox
|
||||
changed_when: "true"
|
||||
|
||||
- name: Verify the ESP stub now points at the /boot LV, not the ext4 root
|
||||
shell: |
|
||||
BOOT_UUID=$(blkid -s UUID -o value /dev/mapper/pve-boot)
|
||||
grep -q "$BOOT_UUID" {{ newroot }}/boot/efi/EFI/proxmox/grub.cfg || {
|
||||
echo "ESP stub does NOT reference the boot LV — aborting before reboot"; exit 1; }
|
||||
echo "ESP stub -> boot LV $BOOT_UUID"
|
||||
changed_when: "false"
|
||||
|
||||
- name: Arm the ONE-SHOT ZFS boot (default stays pinned to the ext4 rollback)
|
||||
shell: |
|
||||
chroot {{ newroot }} grub-reboot pve-zfs-root
|
||||
grep -o 'next_entry=.*' {{ newroot }}/boot/grub/grubenv
|
||||
grep -o 'saved_entry=.*' {{ newroot }}/boot/grub/grubenv
|
||||
changed_when: "true"
|
||||
|
||||
# ---------- tear the chroot down so the dataset can be unmounted ----------
|
||||
|
||||
- name: Unmount the chroot, innermost first
|
||||
shell: |
|
||||
for m in proc/sys/fs/binfmt_misc proc sys dev/pts dev/shm dev/mqueue dev/hugepages dev boot/efi boot; do
|
||||
mountpoint -q {{ newroot }}/$m && umount -R {{ newroot }}/$m 2>/dev/null || true
|
||||
done
|
||||
findmnt -R {{ newroot }} -o TARGET | tail -n +2 || echo " (nothing left under {{ newroot }})"
|
||||
changed_when: "true"
|
||||
|
||||
- name: Unmount the ZFS root dataset BEFORE changing its mountpoint
|
||||
shell: zfs unmount {{ root_dataset }}
|
||||
when: "mountpoint -q {{ newroot }}"
|
||||
|
||||
- name: Set the dataset's final mountpoint (safe only now that it is unmounted)
|
||||
shell: |
|
||||
zfs set mountpoint=/ {{ root_dataset }}
|
||||
zfs get -H -o value mountpoint,canmount {{ root_dataset }} | tr '\n' ' '; echo
|
||||
# paranoia: the live root must STILL be the ext4 LV at this instant
|
||||
test "$(findmnt -no SOURCE /)" = "/dev/mapper/pve-root" || {
|
||||
echo "ZFS MOUNTED OVER THE LIVE ROOT — do not reboot, investigate"; exit 1; }
|
||||
changed_when: "true"
|
||||
|
||||
- name: Final pre-reboot assertion
|
||||
shell: |
|
||||
echo "root now: $(findmnt -no SOURCE,FSTYPE /)"
|
||||
echo "dataset: $(zfs get -H -o value mounted {{ root_dataset }}) mounted, canmount=$(zfs get -H -o value canmount {{ root_dataset }})"
|
||||
echo "next_entry: $(grep -o 'next_entry=.*' {{ newroot }}/boot/grub/grubenv 2>/dev/null || echo '(grubenv not readable — boot LV is unmounted, expected)')"
|
||||
changed_when: "false"
|
||||
|
||||
- name: REBOOT — connection loss here is expected
|
||||
shell: systemd-run --on-active=3 --timer-property=AccuracySec=1s /sbin/reboot
|
||||
changed_when: "true"
|
||||
@@ -0,0 +1,45 @@
|
||||
# esh-pve-nas cutover, step 5a of 5 — restore esh-docker-vm's NFS mounts.
|
||||
#
|
||||
# Run: scripts/elway infra-ops@10.0.50.45 --playbook playbooks/esh-cutover-5-restore-docker-vm.yaml
|
||||
#
|
||||
# Reverses playbooks/esh-cutover-1-quiesce-docker-vm.yaml. Mount first, THEN start
|
||||
# the container: calibre opens its SQLite library on startup, and starting it
|
||||
# against an unmounted /mnt/books would have it create a fresh empty library on
|
||||
# the local disk underneath the mountpoint — which then gets shadowed the moment
|
||||
# the real mount lands, and looks exactly like data loss.
|
||||
|
||||
steps:
|
||||
- name: Mount /mnt/books
|
||||
shell: sudo -n mount /mnt/books
|
||||
when: "! mountpoint -q /mnt/books"
|
||||
|
||||
- name: Mount /mnt/backup
|
||||
shell: sudo -n mount /mnt/backup
|
||||
when: "! mountpoint -q /mnt/backup"
|
||||
|
||||
- name: Confirm the library is actually there before starting calibre
|
||||
shell: |
|
||||
test -f /mnt/books/calibre/calibre_library/metadata.db || {
|
||||
echo "calibre library NOT visible — refusing to start the container"; exit 1; }
|
||||
echo "metadata.db present: $(stat -c %s /mnt/books/calibre/calibre_library/metadata.db) bytes"
|
||||
changed_when: "false"
|
||||
|
||||
- name: Start calibre-web-automated
|
||||
shell: sudo -n docker start calibre-web-automated
|
||||
when: "test -z \"$(sudo -n docker ps -q --filter name=^calibre-web-automated$)\""
|
||||
|
||||
verify:
|
||||
- name: Both NFS mounts are back
|
||||
shell: |
|
||||
mountpoint -q /mnt/books && mountpoint -q /mnt/backup
|
||||
findmnt -no SOURCE,OPTIONS /mnt/books | grep -q hard
|
||||
changed_when: "false"
|
||||
|
||||
- name: calibre-web-automated is running
|
||||
shell: test -n "$(sudo -n docker ps -q --filter name=^calibre-web-automated$)"
|
||||
changed_when: "false"
|
||||
|
||||
- name: Full container count is back
|
||||
shell: |
|
||||
n=$(sudo -n docker ps -q | wc -l); echo "$n containers running"; test "$n" -ge 17
|
||||
changed_when: "false"
|
||||
@@ -86,13 +86,40 @@ steps:
|
||||
mount --bind /boot/efi {{ newroot }}/boot/efi
|
||||
when: "! mountpoint -q {{ newroot }}/boot"
|
||||
|
||||
- name: Bind the kernel filesystems into the chroot
|
||||
# ⚠⚠ --make-rslave IS LOAD-BEARING. Without it this cost a production outage on
|
||||
# 2026-08-18.
|
||||
#
|
||||
# On a systemd host `/` has SHARED mount propagation, so `mount --rbind /dev`
|
||||
# creates a bind that shares propagation with the original. Every later
|
||||
# `umount -R` of the chroot copy then propagates BACK to the live system and
|
||||
# unmounts the REAL /sys/fs/cgroup, /dev/pts and /dev/shm. With cgroup2 gone,
|
||||
# systemd-logind cannot create a session: sshd still completes authentication
|
||||
# and already-resident daemons keep serving from memory, but every new exec
|
||||
# hangs forever. The host looks alive and is unusable, and — this is the part
|
||||
# that wasted the most time — it looks exactly like failing root-disk I/O.
|
||||
#
|
||||
# --make-rslave makes propagation one-way: host -> chroot only. Teardown then
|
||||
# cannot reach back.
|
||||
- name: Bind the kernel filesystems into the chroot (SLAVE propagation)
|
||||
shell: |
|
||||
for d in dev dev/pts proc sys; do
|
||||
for d in dev proc sys; do
|
||||
mountpoint -q {{ newroot }}/$d || mount --rbind /$d {{ newroot }}/$d
|
||||
mount --make-rslave {{ newroot }}/$d
|
||||
done
|
||||
echo "--- propagation (must NOT say shared) ---"
|
||||
findmnt -o TARGET,PROPAGATION {{ newroot }}/dev {{ newroot }}/sys {{ newroot }}/proc
|
||||
changed_when: "true"
|
||||
|
||||
- name: GUARD — refuse to continue if any chroot bind is still shared
|
||||
shell: |
|
||||
if findmnt -no PROPAGATION -R {{ newroot }}/dev {{ newroot }}/sys {{ newroot }}/proc \
|
||||
2>/dev/null | grep -q shared; then
|
||||
echo "chroot binds are SHARED — teardown would unmount the live host's /sys and /dev"
|
||||
exit 1
|
||||
fi
|
||||
echo "all chroot binds are private/slave — teardown cannot propagate back"
|
||||
changed_when: "false"
|
||||
|
||||
# ---------- build the boot artifacts inside the chroot ----------
|
||||
|
||||
- name: Pin the default boot entry to the rollback, not to ZFS
|
||||
|
||||
@@ -16,24 +16,34 @@ Media services on this box sit at `10.0.50.56` (Plex) and `10.0.50.57` (Jellyfin
|
||||
- **CPU:** Intel Xeon W-1250 @ 3.30 GHz
|
||||
- **RAM:** 125.6 GB
|
||||
- **Kernel:** `6.8.12-13-pve` (Proxmox 8.x)
|
||||
- **Storage:** ⚠ **PVE root is on a USB Disk-on-Module** (`sdq`, 7.3 GB, `ID_BUS=usb`,
|
||||
NORELSYS 1081), whose 512 MB ESP is the host's **only** boot path. Not a wear
|
||||
problem (a DOM is SLC/pSLC), but it is on the USB bus, has no headroom, and is
|
||||
unmirrored. Mitigated 2026-08-17 (90% → 76%: journald capped, journal relocated
|
||||
to ZFS). **Migration STAGED 2026-08-18 — awaiting only the reboot window**;
|
||||
runbook [`docs/runbooks/esh-pve-nas-boot-migration.md`](../../docs/runbooks/esh-pve-nas-boot-migration.md).
|
||||
- **Storage: root is `nvme/ROOT/pve-1` on the mirrored NVMe pool** since the
|
||||
2026-08-18 migration. The USB Disk-on-Module (7.3 GB, `ID_BUS=usb`, NORELSYS
|
||||
1081) still holds the ESP and `/boot`, but is **out of the runtime I/O path** —
|
||||
a USB bus reset no longer drops root from under a running hypervisor. Runbook:
|
||||
[`docs/runbooks/esh-pve-nas-boot-migration.md`](../../docs/runbooks/esh-pve-nas-boot-migration.md).
|
||||
|
||||
DOM LVM layout after staging (was root 6.04 G + swap 768 M):
|
||||
DOM LVM layout:
|
||||
|
||||
| LV | size | role |
|
||||
|---|---|---|
|
||||
| `pve-root` | 6.04 G | ext4 — the **live root today**, and the rollback after cutover |
|
||||
| `pve-boot` | 512 M | ext4 — the new `/boot`, carved out of swap |
|
||||
| `pve-root` | 6.04 G | ext4 — **the rollback**, intact and unmounted, keeps its own kernel + initrd |
|
||||
| `pve-boot` | 512 M | ext4 — `/boot`, carved out of swap |
|
||||
| `pve-swap` | 256 M | swap, shrunk to make room |
|
||||
|
||||
Post-cutover root is `nvme/ROOT/pve-1` on the mirrored NVMe pool. `/boot` stays
|
||||
ext4 on the DOM on purpose: GRUB cannot read the `nvme` pool, which has
|
||||
`encryption`, `large_dnode` and `zstd_compress` enabled.
|
||||
`/boot` stays ext4 on the DOM on purpose: GRUB cannot read the `nvme` pool,
|
||||
which has `encryption`, `large_dnode` and `zstd_compress` enabled.
|
||||
|
||||
⚠ **Device letters are not stable** — the DOM was `sdq` before the reboot and
|
||||
`sdl` after. `fstab` uses `/dev/pve/*` and UUIDs; never write a rule against a
|
||||
bare `sdX` on this host.
|
||||
|
||||
⚠ **There is no auto-fallback if a boot fails.** grubenv lives on an LVM LV,
|
||||
which GRUB can read but not write, so `grub-reboot`'s one-shot degrades to a
|
||||
sticky default (verified 2026-08-18: `next_entry` survived the boot that
|
||||
consumed it). Steady state is `saved_entry=pve-zfs-root` with no `next_entry`.
|
||||
A failed boot needs the console — and this box has **no IPMI, no BMC, no serial
|
||||
console**. Recovery is selecting `Proxmox VE - ROLLBACK: ext4 root on the USB
|
||||
DOM` at the GRUB menu.
|
||||
- ⚠ **Never set the ZFS cachefile on one pool.** `zpool set cachefile=…` flips the
|
||||
host from import-by-scan to import-by-cache; a cache holding only `nvme` leaves
|
||||
`ssd` and `tank` unimported at boot, which empties every CT 103 export. Set it on
|
||||
@@ -41,11 +51,23 @@ Media services on this box sit at `10.0.50.56` (Plex) and `10.0.50.57` (Jellyfin
|
||||
- **Pools:** `nvme` (2× 931 GB NVMe mirror — 32 G used, 867 G free, holds every guest
|
||||
rootfs), `ssd` (4× 894 GB Intel SATA, 2 mirrors — 1.42 T free), `tank`
|
||||
(12× 14.6 TB raidz2 ×2 — 40 T of 175 T). Plus NFS `/mnt/pve/tank-vmbu` for VM backups.
|
||||
- ⚠ **CT 103 `esh-nas` (10.0.50.50) runs on THIS host and serves `hard` NFS** to
|
||||
esh-docker-vm (`/mnt/books`, `/mnt/backup`) and to esh-pve (`/mnt/pve/esh-nas`,
|
||||
`/mnt/pve/tank-vmbu`). Taking this box down stalls esh-pve's storage layer and
|
||||
wedges esh-docker-vm into unkillable D-state unless those clients are quiesced
|
||||
first. Never reboot this host casually.
|
||||
- ⚠ **CT 103 `esh-nas` (10.0.50.50) runs on THIS host and serves `hard` NFS.**
|
||||
Never reboot this host casually — quiesce the clients first. Measured from the
|
||||
server on 2026-08-18 (`ss -tn '( sport = :2049 )'` inside CT 103), there are
|
||||
**five** clients, not the two long documented here:
|
||||
|
||||
| client | mounts | opts |
|
||||
|---|---|---|
|
||||
| esh-docker-vm `10.0.50.45` | `/mnt/books`, `/mnt/backup` | **hard** — quiesce |
|
||||
| esh-pve `10.0.250.35` | `/mnt/pve/esh-nas`, `/mnt/pve/tank-vmbu` | **hard** — quiesce |
|
||||
| esh-vm-db `10.0.50.60` | `/mnt/backup` | **hard** — no ssh; reach it via `qm guest exec 101` on esh-pve |
|
||||
| vm-esh-nas `10.0.50.154` | — | is VM 104 *on this host*; stops with it |
|
||||
| nh3-dev `10.100.10.50` | `/mnt/books` | soft,ro — safe, errors rather than blocks |
|
||||
|
||||
**Ask the server who its clients are; don't trust this table.** It rots. `ss` on
|
||||
CT 103 is ground truth, and that is how the three undocumented clients surfaced.
|
||||
Playbooks: `esh-cutover-1-quiesce-docker-vm.yaml`,
|
||||
`esh-cutover-2-quiesce-esh-pve.yaml`, `esh-cutover-5-restore-docker-vm.yaml`.
|
||||
|
||||
> **Root-fs pressure:** 77% of a 5.9 GB root (1.3 GB free) after the 2026-08-17
|
||||
> mitigation. Still not enough for the pending upgrade — **225 packages, 161
|
||||
|
||||
Reference in New Issue
Block a user