esh-pfi-infrastructure/docs/pfi/ops-lessons-playbook.md

Ops lessons playbook — the transferable ones

The operational sibling to model-quantization-playbook.md, and it exists for the same reason: hard-won lessons kept dying inside per-host runbooks where nobody finds them until they have already repeated the mistake.

What belongs here: a lesson that would bite identically on a different host. What does not: anything true only of one machine — that stays in servers/<host>/README.md or the relevant runbook.

Each entry states the rule, what it cost, and how to recognise the situation. When an entry turns out to be wrong, add a dated row to § Superseded rather than quietly editing it, so older references stop misleading people.

---

1. mount --rbind into a chroot needs --make-rslave

Rule: after every mount --rbind /x /target/x, immediately mount --make-rslave /target/x. Guard on it — refuse to proceed while findmnt -o PROPAGATION reports shared for any chroot bind.

Why: on a systemd host / has shared mount propagation, so an --rbind shares propagation with the original. A later umount -R of the chroot copy propagates back into the live system and unmounts the real /sys/fs/cgroup, /dev/pts, /dev/shm. --make-rslave makes propagation one-way (host → chroot), so teardown cannot reach back.

Cost: an unplanned production outage on esh-pve-nas, 2026-08-18.

Recognising it — and this is the valuable part, because it does not look like what it is. With cgroup2 gone, systemd-logind cannot create sessions, which produces a host that:

• answers ping and accepts TCP
• completes SSH authentication
• keeps serving from daemons already resident in memory (a PVE box returned clean HTTP 401s from pveproxy throughout)
• hangs on every new exec — including /sbin/reboot, so a reboot issued to fix it never runs

That is an almost perfect impostor of failing root-disk I/O, and it was misdiagnosed as exactly that. If you see "daemons answer but nothing new can start," check findmnt /sys/fs/cgroup /dev/pts /dev/shm before you suspect the disk.

Recovery needs no console. Exec succeeds in brief windows; loop an idempotent remount until one lands:

mountpoint -q /sys/fs/cgroup || mount -t cgroup2 none /sys/fs/cgroup
mountpoint -q /dev/pts       || mount -t devpts devpts /dev/pts -o gid=5,mode=620,ptmxmode=666
mountpoint -q /dev/shm       || mount -t tmpfs tmpfs /dev/shm -o mode=1777,nosuid,nodev

Then systemctl reset-failed. Full narrative: docs/runbooks/esh-pve-nas-boot-migration.md § The mount-propagation incident.

---

2. A reboot is not confirmed until the host is observed DOWN

Rule: poll for the host's disappearance first, then for its return. Never infer a reboot happened because the host answers.

Why: "never went down" and "went down and came back quickly" are indistinguishable if you only watch for it to answer. On 2026-08-18 a down-detector never once reported the host down; that was read as a fast reboot when in fact /sbin/reboot could not exec and the machine never rebooted at all. Everything diagnosed afterwards was built on that false premise.

The cheap confirmation is the boot timestamp — uptime -p, or the last dmesg timestamp. A dmesg tail whose last entry sits at [12114881] seconds is telling you the machine has been up 140 days, whatever else you believe.

down=0
while :; do
  if ping -c1 -W1 "$H" >/dev/null 2>&1; then
    [ $down -eq 1 ] && break || echo "up (not yet down)"
  else down=1; echo "DOWN confirmed"; fi
  sleep 2
done

---

3. Assert the effective value, not the presence of a substring

Rule: a verification step must check what the system will actually use, not that the correct-looking string appears somewhere in a file.

Why: the check "does root=ZFS=nvme/ROOT/pve-1 appear in grub.cfg?" passed happily while every menu entry was still broken — the correct value had been appended by a drop-in, and the broken pool-less value was still first on the line. Since the kernel takes the last root=, only a check that extracts the last one per entry and compares it against a known-good set proves anything.

/^[[:space:]]*linux[[:space:]]/ {
  r=""; for (i=1;i<=NF;i++) if ($i ~ /^root=/) r=$i;
  if (r != "root=ZFS=pool/dataset" && r != "root=/dev/mapper/x") { print "BAD: " r; bad=1 }
} END { exit bad?1:0 }

Generalises well beyond GRUB: last-wins config keys, layered drop-ins, anything with override semantics. Grep proves presence; only evaluation proves effect.

---

4. Ask the server who its clients are

Rule: before taking a service down, enumerate its dependents from the service, not from documentation.

Why: a runbook named two NFS dependents. ss on the NFS server found five — including a database VM with a hard mount and no SSH access. Documented dependent lists rot silently because nothing forces them to be updated when a new client mounts.

# NFS server: who is actually connected right now
ss -tnH state established '( sport = :2049 )' | awk '{print $4}' | sed 's/:[0-9]*$//' | sort | uniq -c

Equivalents worth reaching for: ss -tnp by port for any service, docker ps plus mount inspection for bind-mount consumers, pvesm status for storage.

Corollary on hard NFS mounts: a hard mount with no active user blocks and then resumes when the server returns — that is what hard is for, and it came through read-write across two server reboots. The disaster case is a process actively using the mount. So quiescing means stopping the consumers, not necessarily unmounting; and when unmounting is expensive or risky (a host you cannot SSH to), leaving an idle hard mount is often the lower-risk branch.

---

5. The scoped-looking command can be the dangerous one

Rule: when a command names one member of a set, ask what happens to the members it does not name.

Why: zpool set cachefile=/etc/zfs/zpool.cache nvme looks careful and narrow. It is not: populating a cachefile flips the host from import-by-scan to import-by-cache, so a cache containing only nvme leaves ssd and tank unimported at boot. On a host whose NAS container had twelve bind mounts spanning all three pools, that empties every export. The broad form — setting it on all three — is the safe one.

---

6. Long uptime hides breakage; a forced look is worth more than it seems

Not a rule so much as a calibration. One migration on a pair of hosts with 20 weeks of uptime surfaced, none of it caused by the work:

found	dead for
pvestatd SEGV'd (node rendered dark in the UI, otherwise healthy)	82 days
a vzdump hung at 0% of 256 GiB, holding lock: backup	126 days
a VM stuck in QEMU prelaunch behind that lock	~4 months
a VM silently missing sshd, mongod and its guest agent	unknown
an undocumented 2-node cluster, and 3 undocumented NFS clients	always

When a host has not been rebooted or audited in months, budget for finding unrelated breakage, and treat that as part of the value rather than as scope creep. Several of these were invisible precisely because nothing had forced anyone to look.

Corollary: a cosmetic-only symptom can hide for a very long time. Nothing alerted on pvestatd; its sole symptom was a grey tile in a UI nobody had reason to stare at. Worth a watchdog on anything whose failure mode is "the dashboard quietly stops being true."

---

7. Verify a "this will break X" premise before building around it

Rule: when a risk is asserted but never tested, test it — especially before it justifies a body of work.

Why: fleet IPv6 work was justified largely by "ESH fiber behind CGNAT will break Site Magic on IPv4." The fiber cutover tested it for free: Cox was unplugged, ESH failed over to 5G on 192.168.200.111 — RFC1918, double-NAT, no inbound path, strictly worse than CGNAT — and the tunnel held, carrying real traffic to all four ESH hosts.

The mechanism was discoverable in advance and made the outcome predictable: Site Magic is WireGuard, and the far side (NH3) has a public endpoint, so the NAT'd side dials out and never needs reachability. Ten minutes of reading the device config would have graded the risk correctly.

How to apply: for any "X will break Y" belief, ask what protocol Y actually uses and which side must be reachable. NAT breaks inbound reachability; it does not break outbound-initiated tunnels with keepalives. Beliefs that gate real work deserve a test or an explicit "untested" label — and when they do get tested, record the result where the belief lived, not only where the test happened.

Related: Site Magic has no WAN binding — magic_site_to_site_vpn on the gateway is just enabled plus a keypair, peers orchestrated in the UniFi cloud. It rides whichever uplink is active, so the only lever is failover priority, and that moves all site traffic rather than just the tunnel.

---

8. A result proven for one protocol does not transfer to another

Rule: when a test clears a risk, state which mechanism it cleared it for, and check whether every affected system shares that mechanism.

Why: proving that NAT does not break Site Magic (WireGuard, outbound-dialed to a public peer) I wrote up as "no addressing outcome threatens the inter-site tunnel." But the fleet has two inter-site links with opposite NAT behaviour, and the other one — IPsec to the colo FortiGate — was already broken at that exact moment, traffic leaking unencapsulated to the carrier. The operator caught it; the test I had just run would have caught it too, had I run it against both links instead of one.

How to apply: ask what property made the test pass — here, "outbound-initiated, peer needs no inbound reachability" — and then ask which systems lack it. IPsec site-to-site pins a peer IP and expects a routable address; WireGuard does not. Same NAT, opposite outcome. Enumerate the affected set before generalising, and name the mechanism in the conclusion so the scope is visible to the next reader.

---

9. IPsec to a NAT'd site: dialup peer + NAT-T, and you cannot convert in place

Rule: a site-to-site IPsec tunnel to any endpoint that might sit behind NAT needs type dynamic (dialup responder) and nattraversal enable. Both. Neither alone is sufficient.

Why: ESH↔colo died the moment ESH stopped having a public IP. Two independent causes, and the second was invisible until the first was investigated:

setting	broken tunnel	working tunnel
type	static, remote-gw 70.181.90.232 (a dead address)	ddns
nattraversal	disable	disable — but NH3 is publicly addressed, so it never mattered

The static peer IP is the obvious failure. The subtle one is that nattraversal disable would have kept the tunnel down even with the correct peer IP, because ESP cannot traverse NAT without UDP-4500 encapsulation. A "just re-pin the IP" fix would have failed and looked mysterious.

⚠ FortiOS refuses set type dynamic on an existing tunnel — "Cannot change tunnel type once configured", with a clean rollback. So the fix is not an edit.

Prefer building the replacement ALONGSIDE the broken one, not recreating it. Deleting a phase1 cascades into its phase2, its static routes and every policy referencing the interface — on the affected box that was 1 + 2 + 10 objects. A new phase1 + phase2 + one route + two consolidated policies is additive, leaves the old config intact as rollback, and cannot break what still works.

Confirming it worked — the tunnel summary line says everything:

'ana-eshudm-dyn_0'  97.170.236.56:4500  selectors(total,up): 1/1
 ^^^ _0 = dialup child      ^^^ carrier IP    ^^^ :4500 = NAT-T

_0 means the peer was accepted without being known in advance; :4500 means NAT-T is carrying ESP; the address is the carrier's, which could never have been pinned. And traceroute drops from "8 hops wandering the carrier" to "gateway → peer → destination".

⚠ Residual fragility on the UniFi end. The UDM's ipsec_local_ip must hold a literal address — "" is rejected with api.err.InvalidPayload — so it still needs updating whenever that site's WAN address changes. The gateway end is now address-agnostic; the UniFi end is not.

---

10. IPv6 collapses two independent exposure controls into one, and it fails open

Rule: before enabling IPv6 on any segment carrying real hosts, write explicit default-deny inbound policy for that segment and verify it from off-net. Reading the ruleset is not verification.

Why — the asymmetry, which is the part worth internalising. Under IPv4 with NAT, exposing an internal host required two affirmative acts: a DNAT/port forward and an accept rule. Miss either and the host stays dark. There is no v4 misconfiguration that accidentally exposes an internal host, because without the translation there is no path at all. NAT was load-bearing security whether or not it was designed as such.

Under IPv6 the path exists inherently — the address is routable from birth. The firewall is now the only control, so two independent things that both had to succeed become one thing that must not fail. The failure mode inverts from fail-closed to fail-open.

Concrete ways it bites:

failure	v4 consequence	v6 consequence
permissive rule ordered above the deny	harmless, no forward exists	immediate exposure
ruleset silently only matches one address family	v4 covered, v6 ungoverned	whole segment on default
new VLAN added, firewall not updated	just a VLAN	live on the internet at first RA
ISP re-delegates a different prefix	n/a	address-literal rules stop matching

How to apply:

• Key rules on interface/zone, not address literals — a re-delegated prefix must not be able to silently unmatch a rule.
• Treat "enable v6 on a segment" as a change requiring the policy to exist first, not as a networking toggle followed by cleanup.
• Verify from outside. Probe the segment's v6 addresses from off-net and confirm the denies hold. This is §3's "assert the effective value, not the presence of a substring" applied to firewall policy: a ruleset that says deny is not evidence that packets are dropped.

Operator position on the ESH fleet (2026-08-19): no 1:1 inbound pass-through. The policy work is writing and proving default-deny, not deciding what to expose.

---

11. Check the writer and the reader together — and name the failure's DIRECTION

A guard's predicate has to be read against what its writer actually produces. Each line is individually reasonable; the mismatch only exists when you hold both. A guard whose test disagrees with its writer's contract has quietly stopped guarding, in whichever direction the mismatch runs.

Two of these surfaced within an hour on 2026-09-09, on opposite sides of the same gate.

The dangerous half (ours). scripts/refresh-server-info.sh and its Proxmox sibling promoted the SSH capture with an unconditional mv whenever ssh exited 0. Every reader downstream tests that snapshot with -s. So a host that connected fine and emitted nothing — inspect script dying before its first write, output swallowed by a remote wrapper — would replace a good snapshot with an empty file and report ok (0 bytes). The header claimed "a failed run never clobbers the previous good snapshot", which was true only for a failed connection; the succeeded-but-empty case is the half nobody inspects, because the line reads reassuring. Fixed 9b9f062: empty capture refused, previous snapshot kept, host counted as failed and the exit code carries it. A capture that collapses to under a quarter of the previous one still promotes but is flagged — a host really can shed services, and a script that guesses there will eventually guess wrong on a real one.

The annoying half (brokkr-smithy-dev's, same day). A launch guard tested a completion sentinel with -s; the writer creates it with touch, so it is zero bytes. The precondition could never pass, whatever the upstream job did. Worse shape than a regression: from the outside it is indistinguishable from a legitimate refusal, so it sends you hunting a problem that does not exist.

The three directions, because the mitigations differ

Same root — the instrument observed something adjacent to what it was named after — but these do not collapse into "the instrument was wrong":

direction	specimen (all 2026-09-09)	what it wants
False reassurance	the empty-snapshot promote; pgrep -f base_window_r7 over ssh matching its own argv, reporting a peer's job "alive" for 2.5 h while blind to it	an independent observation of the object — the seat's own request log (Running: N reqs), the artifact itself
False refusal	-s on a touched sentinel	a predicate that matches its writer's contract
False alarm	an error scan reporting 2 hits by matching the word "refusal" in a log	a pattern that matches the thing, not a word appearing near it

False reassurance is the one that kills you quietly. The other two announce themselves: they waste attention and misdirect, but they cannot silently destroy a good artifact. A post-mortem that lumps all three together loses the half that decides what to do about it.

Filter on the ARTIFACT, not on the name pointing at it

Measured 2026-09-09 while quantifying how much traffic had reached a tune that failed a safety gate. The gateway alias trial had pointed at three different artifacts across the day, so the obvious query — "how many calls to trial?" — answers a question about a name, not about the thing:

filtered on	rows
model_group = 'trial' (the alias)	363
model = 'hosted_vllm/erp-tune-v7-nvfp4a16' (the artifact)	77

Wrong by 4.7x, in the direction that looks careful. Reporting 363 would have overstated the operator's own exposure nearly fivefold, and nothing about the query would have looked sloppy — an alias is what a caller types, so counting it feels like counting usage.

The rule: when the question is "what did this artifact do", filter on the artifact's identity, never on a mutable pointer to it. A name that has been repointed carries the history of everything it ever pointed at. Same family as §3 (identity, not resemblance): an alias resembles the thing and is not it.

A fourth variant: the instrument read a surface MID-TRANSITION

Added 2026-09-09 from a near-miss brokkr-smithy-dev caught and did not send. Verifying the trial alias removal, its first read returned 34 aliases with the alias still present — which looked exactly like the fix had not taken. It had. The read had raced the gateway restart. The tell was that the next three reads came back non-JSON, because the service was mid-restart; waiting for it to settle returned 33 and no alias.

Had that first read been sent, it would have been a false alarm during an incident, and the cost is specific: the other party goes back to re-verify a fix that was already correct, on the word of an observer who sampled a surface at a moment nobody meant to ask about. The instrument answered honestly about the wrong instant.

The rule: a disagreement between two observers is not a finding until the boring explanation is ruled out — a race, a restart, a cache, a stale read. During an incident the pressure runs the other way, because a discrepancy feels urgent and urgency argues for sending it immediately. Read twice, let the surface settle, and prefer the explanation that requires nothing to be wrong.

How to apply. When you write or review a guard, open its writer in the same pass and state the contract out loud — touch → exists-but-empty; mv on exit-0 → may be empty; > → may be truncated; pgrep -f <literal> → matches your own argv. Then ask which direction this predicate fails toward. Sibling of §3 (identity, not resemblance) and §2 (observe the state, don't infer it).

Superseded claims

date	claim	correction
2026-08-18	"ESH behind CGNAT will break the inter-site tunnels, so IPv6 is the escape hatch"	Half true, and the halves matter. Tested live on RFC1918 double-NAT (192.168.200.111): Site Magic (NH3↔ESH, WireGuard) HELD — it dials out to NH3's public edge and never needs inbound reachability. IPsec (colo↔ESH, ana-gw FortiGate) BROKE — traceroute showed traffic unencapsulated, leaking to the carrier. IPv6 keeps its justification on the IPsec link only.
2026-08-18	(my own, same day) "no addressing outcome on the fiber threatens the inter-site tunnel"	Over-generalised. I proved it for WireGuard and wrote it as if it covered every link. Operator caught it. See lesson 8.