vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity

docs(pfi): add an ops-lessons playbook for the transferable failures

Browse Source 
Sibling to model-quantization-playbook.md, and it exists for the same
reason that one does: hard-won lessons were dying inside per-host
runbooks where nobody finds them until after repeating the mistake.

Six entries seeded from the esh-pve-nas migration, all of which would
bite identically on any other host:

1. mount --rbind into a chroot needs --make-rslave, and losing cgroup2
   impersonates failing root-disk I/O closely enough that it was
   misdiagnosed as exactly that.
2. A reboot is not confirmed until the host is observed DOWN; "never
   rebooted" and "rebooted fast" are indistinguishable otherwise.
3. Assert the effective value, not the presence of a substring. Grep
   proves presence; only evaluation proves effect.
4. Ask the server who its clients are -- documented dependent lists rot.
   Plus the corollary that an idle hard NFS mount blocks and resumes, so
   quiescing means stopping consumers, not always unmounting.
5. The scoped-looking command can be the dangerous one; setting a ZFS
   cachefile on one pool of three would have stopped the other two from
   importing at boot.
6. Long uptime hides breakage, and a forced look is worth more than it
   appears -- one migration surfaced an 82-day-dead pvestatd, a 126-day
   hung vzdump, a VM in prelaunch for four months, and an undocumented
   cluster, none of them caused by the work.

Carries a superseded-claims table so corrections are dated rather than
silently edited, same discipline as the quantization playbook. The ESH
runbook now links here so the general rules are reachable from the
specific story and vice versa.
This commit is contained in:
vh
2026-08-18 07:14:48 -07:00
parent 7d27ec9d41
commit 0559e12a2d
2 changed files with 186 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/runbooks/esh-pve-nas-boot-migration.md
+5
View File
@@ -267,6 +267,11 @@ Assert the effective value, not the presence of a substring.
**Cutover** — the remaining work, § Cutover below.
> **The transferable lessons from this migration live in**
> [`docs/pfi/ops-lessons-playbook.md`](../pfi/ops-lessons-playbook.md) — the ops sibling
> to the quantization playbook. Everything below is the ESH-specific narrative;
> the rules that would bite on any host are collected there.
## ⚠ The mount-propagation incident — the expensive lesson of 2026-08-18
**What broke.** The staging chroot was built with `mount --rbind /dev` and `/sys`