diff --git a/docs/pfi/ops-lessons-playbook.md b/docs/pfi/ops-lessons-playbook.md new file mode 100644 index 0000000..0654a56 --- /dev/null +++ b/docs/pfi/ops-lessons-playbook.md @@ -0,0 +1,181 @@ +# Ops lessons playbook — the transferable ones + +The operational sibling to `model-quantization-playbook.md`, and it exists for the +same reason: hard-won lessons kept dying inside per-host runbooks where nobody +finds them until they have already repeated the mistake. + +**What belongs here:** a lesson that would bite identically on a different host. +**What does not:** anything true only of one machine — that stays in +`servers//README.md` or the relevant runbook. + +Each entry states the rule, what it cost, and how to recognise the situation. +When an entry turns out to be wrong, add a dated row to § Superseded rather than +quietly editing it, so older references stop misleading people. + +--- + +## 1. `mount --rbind` into a chroot needs `--make-rslave` + +**Rule:** after every `mount --rbind /x /target/x`, immediately +`mount --make-rslave /target/x`. Guard on it — refuse to proceed while +`findmnt -o PROPAGATION` reports `shared` for any chroot bind. + +**Why:** on a systemd host `/` has *shared* mount propagation, so an `--rbind` +shares propagation with the original. A later `umount -R` of the chroot copy +**propagates back into the live system** and unmounts the real `/sys/fs/cgroup`, +`/dev/pts`, `/dev/shm`. `--make-rslave` makes propagation one-way (host → chroot), +so teardown cannot reach back. + +**Cost:** an unplanned production outage on esh-pve-nas, 2026-08-18. + +**Recognising it — and this is the valuable part, because it does not look like +what it is.** With cgroup2 gone, `systemd-logind` cannot create sessions, which +produces a host that: + +- answers ping and accepts TCP +- **completes SSH authentication** +- keeps serving from daemons already resident in memory (a PVE box returned clean + HTTP 401s from `pveproxy` throughout) +- **hangs on every new `exec`** — including `/sbin/reboot`, so a reboot issued to + fix it never runs + +That is an almost perfect impostor of **failing root-disk I/O**, and it was +misdiagnosed as exactly that. If you see "daemons answer but nothing new can +start," check `findmnt /sys/fs/cgroup /dev/pts /dev/shm` before you suspect the +disk. + +**Recovery needs no console.** Exec succeeds in brief windows; loop an idempotent +remount until one lands: + +```sh +mountpoint -q /sys/fs/cgroup || mount -t cgroup2 none /sys/fs/cgroup +mountpoint -q /dev/pts || mount -t devpts devpts /dev/pts -o gid=5,mode=620,ptmxmode=666 +mountpoint -q /dev/shm || mount -t tmpfs tmpfs /dev/shm -o mode=1777,nosuid,nodev +``` + +Then `systemctl reset-failed`. Full narrative: +`docs/runbooks/esh-pve-nas-boot-migration.md` § The mount-propagation incident. + +--- + +## 2. A reboot is not confirmed until the host is observed DOWN + +**Rule:** poll for the host's *disappearance* first, then for its return. Never +infer a reboot happened because the host answers. + +**Why:** "never went down" and "went down and came back quickly" are +indistinguishable if you only watch for it to answer. On 2026-08-18 a +down-detector never once reported the host down; that was read as a fast reboot +when in fact `/sbin/reboot` could not exec and the machine never rebooted at all. +Everything diagnosed afterwards was built on that false premise. + +**The cheap confirmation** is the boot timestamp — `uptime -p`, or the last +`dmesg` timestamp. A `dmesg` tail whose last entry sits at `[12114881]` seconds +is telling you the machine has been up 140 days, whatever else you believe. + +```sh +down=0 +while :; do + if ping -c1 -W1 "$H" >/dev/null 2>&1; then + [ $down -eq 1 ] && break || echo "up (not yet down)" + else down=1; echo "DOWN confirmed"; fi + sleep 2 +done +``` + +--- + +## 3. Assert the effective value, not the presence of a substring + +**Rule:** a verification step must check what the system will actually *use*, not +that the correct-looking string appears somewhere in a file. + +**Why:** the check "does `root=ZFS=nvme/ROOT/pve-1` appear in `grub.cfg`?" passed +happily while **every menu entry was still broken** — the correct value had been +appended by a drop-in, and the broken pool-less value was still first on the line. +Since the kernel takes the *last* `root=`, only a check that extracts the last one +per entry and compares it against a known-good set proves anything. + +```awk +/^[[:space:]]*linux[[:space:]]/ { + r=""; for (i=1;i<=NF;i++) if ($i ~ /^root=/) r=$i; + if (r != "root=ZFS=pool/dataset" && r != "root=/dev/mapper/x") { print "BAD: " r; bad=1 } +} END { exit bad?1:0 } +``` + +Generalises well beyond GRUB: last-wins config keys, layered drop-ins, anything +with override semantics. **Grep proves presence; only evaluation proves effect.** + +--- + +## 4. Ask the server who its clients are + +**Rule:** before taking a service down, enumerate its dependents **from the +service**, not from documentation. + +**Why:** a runbook named two NFS dependents. `ss` on the NFS server found five — +including a database VM with a `hard` mount and no SSH access. Documented +dependent lists rot silently because nothing forces them to be updated when a new +client mounts. + +```sh +# NFS server: who is actually connected right now +ss -tnH state established '( sport = :2049 )' | awk '{print $4}' | sed 's/:[0-9]*$//' | sort | uniq -c +``` + +Equivalents worth reaching for: `ss -tnp` by port for any service, `docker ps` +plus mount inspection for bind-mount consumers, `pvesm status` for storage. + +**Corollary on `hard` NFS mounts:** a `hard` mount with **no active user** blocks +and then resumes when the server returns — that is what `hard` is for, and it came +through read-write across two server reboots. The disaster case is a *process +actively using* the mount. So quiescing means stopping the consumers, not +necessarily unmounting; and when unmounting is expensive or risky (a host you +cannot SSH to), leaving an idle hard mount is often the lower-risk branch. + +--- + +## 5. The scoped-looking command can be the dangerous one + +**Rule:** when a command names one member of a set, ask what happens to the +members it does not name. + +**Why:** `zpool set cachefile=/etc/zfs/zpool.cache nvme` looks careful and +narrow. It is not: populating a cachefile flips the host from import-by-scan to +import-by-**cache**, so a cache containing only `nvme` leaves `ssd` and `tank` +unimported at boot. On a host whose NAS container had twelve bind mounts spanning +all three pools, that empties every export. The broad form — setting it on all +three — is the safe one. + +--- + +## 6. Long uptime hides breakage; a forced look is worth more than it seems + +Not a rule so much as a calibration. One migration on a pair of hosts with 20 +weeks of uptime surfaced, none of it caused by the work: + +| found | dead for | +|---|---| +| `pvestatd` SEGV'd (node rendered dark in the UI, otherwise healthy) | 82 days | +| a `vzdump` hung at 0% of 256 GiB, holding `lock: backup` | 126 days | +| a VM stuck in QEMU `prelaunch` behind that lock | ~4 months | +| a VM silently missing `sshd`, `mongod` and its guest agent | unknown | +| an undocumented 2-node cluster, and 3 undocumented NFS clients | always | + +**When a host has not been rebooted or audited in months, budget for finding +unrelated breakage, and treat that as part of the value rather than as scope +creep.** Several of these were invisible precisely because nothing had forced +anyone to look. + +Corollary: **a cosmetic-only symptom can hide for a very long time.** Nothing +alerted on `pvestatd`; its sole symptom was a grey tile in a UI nobody had reason +to stare at. Worth a watchdog on anything whose failure mode is "the dashboard +quietly stops being true." + +--- + +## Superseded claims + +| date | claim | correction | +|---|---|---| +| — | _(none yet)_ | | diff --git a/docs/runbooks/esh-pve-nas-boot-migration.md b/docs/runbooks/esh-pve-nas-boot-migration.md index 678cb1a..0d5ff51 100644 --- a/docs/runbooks/esh-pve-nas-boot-migration.md +++ b/docs/runbooks/esh-pve-nas-boot-migration.md @@ -267,6 +267,11 @@ Assert the effective value, not the presence of a substring. **Cutover** — the remaining work, § Cutover below. +> **The transferable lessons from this migration live in** +> [`docs/pfi/ops-lessons-playbook.md`](../pfi/ops-lessons-playbook.md) — the ops sibling +> to the quantization playbook. Everything below is the ESH-specific narrative; +> the rules that would bite on any host are collected there. + ## ⚠ The mount-propagation incident — the expensive lesson of 2026-08-18 **What broke.** The staging chroot was built with `mount --rbind /dev` and `/sys`