vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity

docs(pfi): add an ops-lessons playbook for the transferable failures

Browse Source 
Sibling to model-quantization-playbook.md, and it exists for the same
reason that one does: hard-won lessons were dying inside per-host
runbooks where nobody finds them until after repeating the mistake.

Six entries seeded from the esh-pve-nas migration, all of which would
bite identically on any other host:

1. mount --rbind into a chroot needs --make-rslave, and losing cgroup2
   impersonates failing root-disk I/O closely enough that it was
   misdiagnosed as exactly that.
2. A reboot is not confirmed until the host is observed DOWN; "never
   rebooted" and "rebooted fast" are indistinguishable otherwise.
3. Assert the effective value, not the presence of a substring. Grep
   proves presence; only evaluation proves effect.
4. Ask the server who its clients are -- documented dependent lists rot.
   Plus the corollary that an idle hard NFS mount blocks and resumes, so
   quiescing means stopping consumers, not always unmounting.
5. The scoped-looking command can be the dangerous one; setting a ZFS
   cachefile on one pool of three would have stopped the other two from
   importing at boot.
6. Long uptime hides breakage, and a forced look is worth more than it
   appears -- one migration surfaced an 82-day-dead pvestatd, a 126-day
   hung vzdump, a VM in prelaunch for four months, and an undocumented
   cluster, none of them caused by the work.

Carries a superseded-claims table so corrections are dated rather than
silently edited, same discipline as the quantization playbook. The ESH
runbook now links here so the general rules are reachable from the
specific story and vice versa.
This commit is contained in:
vh
2026-08-18 07:14:48 -07:00
parent 7d27ec9d41
commit 0559e12a2d
2 changed files with 186 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/pfi/ops-lessons-playbook.md
+181
View File
@@ -0,0 +1,181 @@
# Ops lessons playbook — the transferable ones
The operational sibling to `model-quantization-playbook.md`, and it exists for the
same reason: hard-won lessons kept dying inside per-host runbooks where nobody
finds them until they have already repeated the mistake.
**What belongs here:** a lesson that would bite identically on a different host.
**What does not:** anything true only of one machine — that stays in
`servers/<host>/README.md` or the relevant runbook.
Each entry states the rule, what it cost, and how to recognise the situation.
When an entry turns out to be wrong, add a dated row to § Superseded rather than
quietly editing it, so older references stop misleading people.
---
## 1. `mount --rbind` into a chroot needs `--make-rslave`
**Rule:** after every `mount --rbind /x /target/x`, immediately
`mount --make-rslave /target/x`. Guard on it — refuse to proceed while
`findmnt -o PROPAGATION` reports `shared` for any chroot bind.
**Why:** on a systemd host `/` has *shared* mount propagation, so an `--rbind`
shares propagation with the original. A later `umount -R` of the chroot copy
**propagates back into the live system** and unmounts the real `/sys/fs/cgroup`,
`/dev/pts`, `/dev/shm`. `--make-rslave` makes propagation one-way (host → chroot),
so teardown cannot reach back.
**Cost:** an unplanned production outage on esh-pve-nas, 2026-08-18.
**Recognising it — and this is the valuable part, because it does not look like
what it is.** With cgroup2 gone, `systemd-logind` cannot create sessions, which
produces a host that:
- answers ping and accepts TCP
- **completes SSH authentication**
- keeps serving from daemons already resident in memory (a PVE box returned clean
HTTP 401s from `pveproxy` throughout)
- **hangs on every new `exec`** — including `/sbin/reboot`, so a reboot issued to
fix it never runs
That is an almost perfect impostor of **failing root-disk I/O**, and it was
misdiagnosed as exactly that. If you see "daemons answer but nothing new can
start," check `findmnt /sys/fs/cgroup /dev/pts /dev/shm` before you suspect the
disk.
**Recovery needs no console.** Exec succeeds in brief windows; loop an idempotent
remount until one lands:
```sh
mountpoint -q /sys/fs/cgroup || mount -t cgroup2 none /sys/fs/cgroup
mountpoint -q /dev/pts || mount -t devpts devpts /dev/pts -o gid=5,mode=620,ptmxmode=666
mountpoint -q /dev/shm || mount -t tmpfs tmpfs /dev/shm -o mode=1777,nosuid,nodev
```
Then `systemctl reset-failed`. Full narrative:
`docs/runbooks/esh-pve-nas-boot-migration.md` § The mount-propagation incident.
---
## 2. A reboot is not confirmed until the host is observed DOWN
**Rule:** poll for the host's *disappearance* first, then for its return. Never
infer a reboot happened because the host answers.
**Why:** "never went down" and "went down and came back quickly" are
indistinguishable if you only watch for it to answer. On 2026-08-18 a
down-detector never once reported the host down; that was read as a fast reboot
when in fact `/sbin/reboot` could not exec and the machine never rebooted at all.
Everything diagnosed afterwards was built on that false premise.
**The cheap confirmation** is the boot timestamp — `uptime -p`, or the last
`dmesg` timestamp. A `dmesg` tail whose last entry sits at `[12114881]` seconds
is telling you the machine has been up 140 days, whatever else you believe.
```sh
down=0
while :; do
if ping -c1 -W1 "$H" >/dev/null 2>&1; then
[ $down -eq 1 ] && break || echo "up (not yet down)"
else down=1; echo "DOWN confirmed"; fi
sleep 2
done
```
---
## 3. Assert the effective value, not the presence of a substring
**Rule:** a verification step must check what the system will actually *use*, not
that the correct-looking string appears somewhere in a file.
**Why:** the check "does `root=ZFS=nvme/ROOT/pve-1` appear in `grub.cfg`?" passed
happily while **every menu entry was still broken** — the correct value had been
appended by a drop-in, and the broken pool-less value was still first on the line.
Since the kernel takes the *last* `root=`, only a check that extracts the last one
per entry and compares it against a known-good set proves anything.
```awk
/^[[:space:]]*linux[[:space:]]/ {
r=""; for (i=1;i<=NF;i++) if ($i ~ /^root=/) r=$i;
if (r != "root=ZFS=pool/dataset" && r != "root=/dev/mapper/x") { print "BAD: " r; bad=1 }
} END { exit bad?1:0 }
```
Generalises well beyond GRUB: last-wins config keys, layered drop-ins, anything
with override semantics. **Grep proves presence; only evaluation proves effect.**
---
## 4. Ask the server who its clients are
**Rule:** before taking a service down, enumerate its dependents **from the
service**, not from documentation.
**Why:** a runbook named two NFS dependents. `ss` on the NFS server found five —
including a database VM with a `hard` mount and no SSH access. Documented
dependent lists rot silently because nothing forces them to be updated when a new
client mounts.
```sh
# NFS server: who is actually connected right now
ss -tnH state established '( sport = :2049 )' | awk '{print $4}' | sed 's/:[0-9]*$//' | sort | uniq -c
```
Equivalents worth reaching for: `ss -tnp` by port for any service, `docker ps`
plus mount inspection for bind-mount consumers, `pvesm status` for storage.
**Corollary on `hard` NFS mounts:** a `hard` mount with **no active user** blocks
and then resumes when the server returns — that is what `hard` is for, and it came
through read-write across two server reboots. The disaster case is a *process
actively using* the mount. So quiescing means stopping the consumers, not
necessarily unmounting; and when unmounting is expensive or risky (a host you
cannot SSH to), leaving an idle hard mount is often the lower-risk branch.
---
## 5. The scoped-looking command can be the dangerous one
**Rule:** when a command names one member of a set, ask what happens to the
members it does not name.
**Why:** `zpool set cachefile=/etc/zfs/zpool.cache nvme` looks careful and
narrow. It is not: populating a cachefile flips the host from import-by-scan to
import-by-**cache**, so a cache containing only `nvme` leaves `ssd` and `tank`
unimported at boot. On a host whose NAS container had twelve bind mounts spanning
all three pools, that empties every export. The broad form — setting it on all
three — is the safe one.
---
## 6. Long uptime hides breakage; a forced look is worth more than it seems
Not a rule so much as a calibration. One migration on a pair of hosts with 20
weeks of uptime surfaced, none of it caused by the work:
| found | dead for |
|---|---|
| `pvestatd` SEGV'd (node rendered dark in the UI, otherwise healthy) | 82 days |
| a `vzdump` hung at 0% of 256 GiB, holding `lock: backup` | 126 days |
| a VM stuck in QEMU `prelaunch` behind that lock | ~4 months |
| a VM silently missing `sshd`, `mongod` and its guest agent | unknown |
| an undocumented 2-node cluster, and 3 undocumented NFS clients | always |
**When a host has not been rebooted or audited in months, budget for finding
unrelated breakage, and treat that as part of the value rather than as scope
creep.** Several of these were invisible precisely because nothing had forced
anyone to look.
Corollary: **a cosmetic-only symptom can hide for a very long time.** Nothing
alerted on `pvestatd`; its sole symptom was a grey tile in a UI nobody had reason
to stare at. Worth a watchdog on anything whose failure mode is "the dashboard
quietly stops being true."
---
## Superseded claims
| date | claim | correction |
|---|---|---|
| — | _(none yet)_ | |
docs/runbooks/esh-pve-nas-boot-migration.md
+5
View File
@@ -267,6 +267,11 @@ Assert the effective value, not the presence of a substring.
**Cutover** — the remaining work, § Cutover below.
> **The transferable lessons from this migration live in**
> [`docs/pfi/ops-lessons-playbook.md`](../pfi/ops-lessons-playbook.md) — the ops sibling
> to the quantization playbook. Everything below is the ESH-specific narrative;
> the rules that would bite on any host are collected there.
## ⚠ The mount-propagation incident — the expensive lesson of 2026-08-18
**What broke.** The staging chroot was built with `mount --rbind /dev` and `/sys`