Files
esh-pfi-infrastructure/docs/runbooks
vh 0559e12a2d docs(pfi): add an ops-lessons playbook for the transferable failures
Sibling to model-quantization-playbook.md, and it exists for the same
reason that one does: hard-won lessons were dying inside per-host
runbooks where nobody finds them until after repeating the mistake.

Six entries seeded from the esh-pve-nas migration, all of which would
bite identically on any other host:

1. mount --rbind into a chroot needs --make-rslave, and losing cgroup2
   impersonates failing root-disk I/O closely enough that it was
   misdiagnosed as exactly that.
2. A reboot is not confirmed until the host is observed DOWN; "never
   rebooted" and "rebooted fast" are indistinguishable otherwise.
3. Assert the effective value, not the presence of a substring. Grep
   proves presence; only evaluation proves effect.
4. Ask the server who its clients are -- documented dependent lists rot.
   Plus the corollary that an idle hard NFS mount blocks and resumes, so
   quiescing means stopping consumers, not always unmounting.
5. The scoped-looking command can be the dangerous one; setting a ZFS
   cachefile on one pool of three would have stopped the other two from
   importing at boot.
6. Long uptime hides breakage, and a forced look is worth more than it
   appears -- one migration surfaced an 82-day-dead pvestatd, a 126-day
   hung vzdump, a VM in prelaunch for four months, and an undocumented
   cluster, none of them caused by the work.

Carries a superseded-claims table so corrections are dated rather than
silently edited, same discipline as the quantization playbook. The ESH
runbook now links here so the general rules are reachable from the
specific story and vice versa.
2026-08-18 07:14:48 -07:00
..