esh-pfi-infrastructure/docs/pfi/ops-lessons-playbook.md

# Ops lessons playbook — the transferable ones

The operational sibling to `model-quantization-playbook.md`, and it exists for the
same reason: hard-won lessons kept dying inside per-host runbooks where nobody
finds them until they have already repeated the mistake.

**What belongs here:** a lesson that would bite identically on a different host.
**What does not:** anything true only of one machine — that stays in
`servers/<host>/README.md` or the relevant runbook.

Each entry states the rule, what it cost, and how to recognise the situation.
When an entry turns out to be wrong, add a dated row to § Superseded rather than
quietly editing it, so older references stop misleading people.

---

## 1. `mount --rbind` into a chroot needs `--make-rslave`

**Rule:** after every `mount --rbind /x /target/x`, immediately
`mount --make-rslave /target/x`. Guard on it — refuse to proceed while
`findmnt -o PROPAGATION` reports `shared` for any chroot bind.

**Why:** on a systemd host `/` has *shared* mount propagation, so an `--rbind`
shares propagation with the original. A later `umount -R` of the chroot copy
**propagates back into the live system** and unmounts the real `/sys/fs/cgroup`,
`/dev/pts`, `/dev/shm`. `--make-rslave` makes propagation one-way (host → chroot),
so teardown cannot reach back.

**Cost:** an unplanned production outage on esh-pve-nas, 2026-08-18.

**Recognising it — and this is the valuable part, because it does not look like
what it is.** With cgroup2 gone, `systemd-logind` cannot create sessions, which
produces a host that:

- answers ping and accepts TCP
- **completes SSH authentication**
- keeps serving from daemons already resident in memory (a PVE box returned clean
  HTTP 401s from `pveproxy` throughout)
- **hangs on every new `exec`** — including `/sbin/reboot`, so a reboot issued to
  fix it never runs

That is an almost perfect impostor of **failing root-disk I/O**, and it was
misdiagnosed as exactly that. If you see "daemons answer but nothing new can
start," check `findmnt /sys/fs/cgroup /dev/pts /dev/shm` before you suspect the
disk.

**Recovery needs no console.** Exec succeeds in brief windows; loop an idempotent
remount until one lands:

```sh
mountpoint -q /sys/fs/cgroup || mount -t cgroup2 none /sys/fs/cgroup
mountpoint -q /dev/pts       || mount -t devpts devpts /dev/pts -o gid=5,mode=620,ptmxmode=666
mountpoint -q /dev/shm       || mount -t tmpfs tmpfs /dev/shm -o mode=1777,nosuid,nodev
```

Then `systemctl reset-failed`. Full narrative:
`docs/runbooks/esh-pve-nas-boot-migration.md` § The mount-propagation incident.

---

## 2. A reboot is not confirmed until the host is observed DOWN

**Rule:** poll for the host's *disappearance* first, then for its return. Never
infer a reboot happened because the host answers.

**Why:** "never went down" and "went down and came back quickly" are
indistinguishable if you only watch for it to answer. On 2026-08-18 a
down-detector never once reported the host down; that was read as a fast reboot
when in fact `/sbin/reboot` could not exec and the machine never rebooted at all.
Everything diagnosed afterwards was built on that false premise.

**The cheap confirmation** is the boot timestamp — `uptime -p`, or the last
`dmesg` timestamp. A `dmesg` tail whose last entry sits at `[12114881]` seconds
is telling you the machine has been up 140 days, whatever else you believe.

```sh
down=0
while :; do
  if ping -c1 -W1 "$H" >/dev/null 2>&1; then
    [ $down -eq 1 ] && break || echo "up (not yet down)"
  else down=1; echo "DOWN confirmed"; fi
  sleep 2
done
```

---

## 3. Assert the effective value, not the presence of a substring

**Rule:** a verification step must check what the system will actually *use*, not
that the correct-looking string appears somewhere in a file.

**Why:** the check "does `root=ZFS=nvme/ROOT/pve-1` appear in `grub.cfg`?" passed
happily while **every menu entry was still broken** — the correct value had been
appended by a drop-in, and the broken pool-less value was still first on the line.
Since the kernel takes the *last* `root=`, only a check that extracts the last one
per entry and compares it against a known-good set proves anything.

```awk
/^[[:space:]]*linux[[:space:]]/ {
  r=""; for (i=1;i<=NF;i++) if ($i ~ /^root=/) r=$i;
  if (r != "root=ZFS=pool/dataset" && r != "root=/dev/mapper/x") { print "BAD: " r; bad=1 }
} END { exit bad?1:0 }
```

Generalises well beyond GRUB: last-wins config keys, layered drop-ins, anything
with override semantics. **Grep proves presence; only evaluation proves effect.**

---

## 4. Ask the server who its clients are

**Rule:** before taking a service down, enumerate its dependents **from the
service**, not from documentation.

**Why:** a runbook named two NFS dependents. `ss` on the NFS server found five —
including a database VM with a `hard` mount and no SSH access. Documented
dependent lists rot silently because nothing forces them to be updated when a new
client mounts.

```sh
# NFS server: who is actually connected right now
ss -tnH state established '( sport = :2049 )' | awk '{print $4}' | sed 's/:[0-9]*$//' | sort | uniq -c
```

Equivalents worth reaching for: `ss -tnp` by port for any service, `docker ps`
plus mount inspection for bind-mount consumers, `pvesm status` for storage.

**Corollary on `hard` NFS mounts:** a `hard` mount with **no active user** blocks
and then resumes when the server returns — that is what `hard` is for, and it came
through read-write across two server reboots. The disaster case is a *process
actively using* the mount. So quiescing means stopping the consumers, not
necessarily unmounting; and when unmounting is expensive or risky (a host you
cannot SSH to), leaving an idle hard mount is often the lower-risk branch.

---

## 5. The scoped-looking command can be the dangerous one

**Rule:** when a command names one member of a set, ask what happens to the
members it does not name.

**Why:** `zpool set cachefile=/etc/zfs/zpool.cache nvme` looks careful and
narrow. It is not: populating a cachefile flips the host from import-by-scan to
import-by-**cache**, so a cache containing only `nvme` leaves `ssd` and `tank`
unimported at boot. On a host whose NAS container had twelve bind mounts spanning
all three pools, that empties every export. The broad form — setting it on all
three — is the safe one.

---

## 6. Long uptime hides breakage; a forced look is worth more than it seems

Not a rule so much as a calibration. One migration on a pair of hosts with 20
weeks of uptime surfaced, none of it caused by the work:

| found | dead for |
|---|---|
| `pvestatd` SEGV'd (node rendered dark in the UI, otherwise healthy) | 82 days |
| a `vzdump` hung at 0% of 256 GiB, holding `lock: backup` | 126 days |
| a VM stuck in QEMU `prelaunch` behind that lock | ~4 months |
| a VM silently missing `sshd`, `mongod` and its guest agent | unknown |
| an undocumented 2-node cluster, and 3 undocumented NFS clients | always |

**When a host has not been rebooted or audited in months, budget for finding
unrelated breakage, and treat that as part of the value rather than as scope
creep.** Several of these were invisible precisely because nothing had forced
anyone to look.

Corollary: **a cosmetic-only symptom can hide for a very long time.** Nothing
alerted on `pvestatd`; its sole symptom was a grey tile in a UI nobody had reason
to stare at. Worth a watchdog on anything whose failure mode is "the dashboard
quietly stops being true."

---

## 7. Verify a "this will break X" premise before building around it

**Rule:** when a risk is asserted but never tested, test it — especially before it
justifies a body of work.

**Why:** fleet IPv6 work was justified largely by "ESH fiber behind CGNAT will
break Site Magic on IPv4." The fiber cutover tested it for free: Cox was
unplugged, ESH failed over to 5G on `192.168.200.111` — **RFC1918, double-NAT,
no inbound path, strictly worse than CGNAT** — and the tunnel held, carrying real
traffic to all four ESH hosts.

The mechanism was discoverable in advance and made the outcome predictable:
Site Magic is **WireGuard**, and the far side (NH3) has a public endpoint, so the
NAT'd side dials out and never needs reachability. Ten minutes of reading the
device config would have graded the risk correctly.

**How to apply:** for any "X will break Y" belief, ask what protocol Y actually
uses and which side must be reachable. NAT breaks *inbound* reachability; it does
not break outbound-initiated tunnels with keepalives. Beliefs that gate real work
deserve a test or an explicit "untested" label — and when they do get tested,
record the result where the belief lived, not only where the test happened.

**Related:** Site Magic has **no WAN binding** — `magic_site_to_site_vpn` on the
gateway is just `enabled` plus a keypair, peers orchestrated in the UniFi cloud.
It rides whichever uplink is active, so the only lever is failover priority, and
that moves *all* site traffic rather than just the tunnel.

---

## 8. A result proven for one protocol does not transfer to another

**Rule:** when a test clears a risk, state **which mechanism** it cleared it for,
and check whether every affected system shares that mechanism.

**Why:** proving that NAT does not break **Site Magic** (WireGuard, outbound-dialed
to a public peer) I wrote up as "no addressing outcome threatens the inter-site
tunnel." But the fleet has *two* inter-site links with opposite NAT behaviour, and
the other one — **IPsec** to the colo FortiGate — was **already broken at that
exact moment**, traffic leaking unencapsulated to the carrier. The operator caught
it; the test I had just run would have caught it too, had I run it against both
links instead of one.

**How to apply:** ask what property made the test pass — here, "outbound-initiated,
peer needs no inbound reachability" — and then ask which systems *lack* it. IPsec
site-to-site pins a peer IP and expects a routable address; WireGuard does not.
Same NAT, opposite outcome. Enumerate the affected set before generalising, and
name the mechanism in the conclusion so the scope is visible to the next reader.

---

## 9. IPsec to a NAT'd site: dialup peer + NAT-T, and you cannot convert in place

**Rule:** a site-to-site IPsec tunnel to any endpoint that might sit behind NAT
needs **`type dynamic`** (dialup responder) **and `nattraversal enable`**. Both.
Neither alone is sufficient.

**Why:** ESH↔colo died the moment ESH stopped having a public IP. Two independent
causes, and the second was invisible until the first was investigated:

| setting | broken tunnel | working tunnel |
|---|---|---|
| `type` | `static`, `remote-gw 70.181.90.232` (a dead address) | `ddns` |
| `nattraversal` | `disable` | `disable` — but NH3 is **publicly addressed**, so it never mattered |

The static peer IP is the obvious failure. The subtle one is that **`nattraversal
disable` would have kept the tunnel down even with the correct peer IP**, because
ESP cannot traverse NAT without UDP-4500 encapsulation. A "just re-pin the IP"
fix would have failed and looked mysterious.

⚠ **FortiOS refuses `set type dynamic` on an existing tunnel** — *"Cannot change
tunnel type once configured"*, with a clean rollback. So the fix is not an edit.

**Prefer building the replacement ALONGSIDE the broken one, not recreating it.**
Deleting a phase1 cascades into its phase2, its static routes and every policy
referencing the interface — on the affected box that was 1 + 2 + 10 objects.
A new `phase1` + `phase2` + one route + two consolidated policies is additive,
leaves the old config intact as rollback, and cannot break what still works.

**Confirming it worked** — the tunnel summary line says everything:

```
'ana-eshudm-dyn_0'  97.170.236.56:4500  selectors(total,up): 1/1
 ^^^ _0 = dialup child      ^^^ carrier IP    ^^^ :4500 = NAT-T
```

`_0` means the peer was accepted without being known in advance; `:4500` means
NAT-T is carrying ESP; the address is the carrier's, which could never have been
pinned. And traceroute drops from "8 hops wandering the carrier" to "gateway →
peer → destination".

⚠ **Residual fragility on the UniFi end.** The UDM's `ipsec_local_ip` must hold a
literal address — `""` is rejected with `api.err.InvalidPayload` — so it still
needs updating whenever that site's WAN address changes. The gateway end is now
address-agnostic; the UniFi end is not.

---

## 10. IPv6 collapses two independent exposure controls into one, and it fails open

**Rule:** before enabling IPv6 on any segment carrying real hosts, write explicit
default-deny inbound policy for that segment **and verify it from off-net**.
Reading the ruleset is not verification.

**Why — the asymmetry, which is the part worth internalising.** Under IPv4 with
NAT, exposing an internal host required **two** affirmative acts: a DNAT/port
forward *and* an accept rule. Miss either and the host stays dark. There is no
v4 misconfiguration that accidentally exposes an internal host, because without
the translation there is no path at all. NAT was load-bearing security whether or
not it was designed as such.

Under IPv6 the path exists inherently — the address is routable from birth. The
firewall is now the *only* control, so two independent things that both had to
succeed become one thing that must not fail. **The failure mode inverts from
fail-closed to fail-open.**

**Concrete ways it bites:**

| failure | v4 consequence | v6 consequence |
|---|---|---|
| permissive rule ordered above the deny | harmless, no forward exists | immediate exposure |
| ruleset silently only matches one address family | v4 covered, v6 ungoverned | whole segment on default |
| new VLAN added, firewall not updated | just a VLAN | live on the internet at first RA |
| ISP re-delegates a different prefix | n/a | address-literal rules stop matching |

**How to apply:**
- Key rules on **interface/zone, not address literals** — a re-delegated prefix
  must not be able to silently unmatch a rule.
- Treat "enable v6 on a segment" as a change requiring the policy to exist
  *first*, not as a networking toggle followed by cleanup.
- **Verify from outside.** Probe the segment's v6 addresses from off-net and
  confirm the denies hold. This is §3's "assert the effective value, not the
  presence of a substring" applied to firewall policy: a ruleset that *says*
  deny is not evidence that packets are dropped.

Operator position on the ESH fleet (2026-08-19): **no 1:1 inbound pass-through.**
The policy work is writing and proving default-deny, not deciding what to expose.

---

## 11. Check the writer and the reader together — and name the failure's DIRECTION

A guard's predicate has to be read against what its writer actually produces.
Each line is individually reasonable; the mismatch only exists when you hold
both. **A guard whose test disagrees with its writer's contract has quietly
stopped guarding, in whichever direction the mismatch runs.**

Two of these surfaced within an hour on 2026-09-09, on opposite sides of the same
gate.

**The dangerous half (ours).** `scripts/refresh-server-info.sh` and its Proxmox
sibling promoted the SSH capture with an unconditional `mv` whenever ssh exited
0. Every reader downstream tests that snapshot with `-s`. So a host that
connected fine and emitted nothing — inspect script dying before its first write,
output swallowed by a remote wrapper — would **replace a good snapshot with an
empty file and report `ok (0 bytes)`.** The header claimed "a failed run never
clobbers the previous good snapshot", which was true only for a failed
*connection*; the succeeded-but-empty case is the half nobody inspects, because
the line reads reassuring. Fixed `9b9f062`: empty capture refused, previous
snapshot kept, host counted as failed and the exit code carries it. A capture
that collapses to under a quarter of the previous one still **promotes but is
flagged** — a host really can shed services, and a script that guesses there will
eventually guess wrong on a real one.

**The annoying half (brokkr-smithy-dev's, same day).** A launch guard tested a
completion sentinel with `-s`; the writer creates it with `touch`, so it is zero
bytes. The precondition could never pass, whatever the upstream job did. Worse
*shape* than a regression: from the outside it is indistinguishable from a
legitimate refusal, so it sends you hunting a problem that does not exist.

### The three directions, because the mitigations differ

Same root — the instrument observed something *adjacent* to what it was named
after — but these do not collapse into "the instrument was wrong":

| direction | specimen (all 2026-09-09) | what it wants |
|---|---|---|
| **False reassurance** | the empty-snapshot promote; `pgrep -f base_window_r7` over ssh matching its own argv, reporting a peer's job "alive" for 2.5 h while blind to it | an **independent observation of the object** — the seat's own request log (`Running: N reqs`), the artifact itself |
| **False refusal** | `-s` on a `touch`ed sentinel | a **predicate that matches its writer's contract** |
| **False alarm** | an error scan reporting 2 hits by matching the word "refusal" in a log | a **pattern that matches the thing**, not a word appearing near it |

**False reassurance is the one that kills you quietly.** The other two announce
themselves: they waste attention and misdirect, but they cannot silently destroy
a good artifact. A post-mortem that lumps all three together loses the half that
decides what to do about it.

### Filter on the ARTIFACT, not on the name pointing at it

Measured 2026-09-09 while quantifying how much traffic had reached a tune that
failed a safety gate. The gateway alias `trial` had pointed at three different
artifacts across the day, so the obvious query — *"how many calls to `trial`?"* —
answers a question about a **name**, not about the **thing**:

| filtered on | rows |
|---|---|
| `model_group = 'trial'` (the alias) | **363** |
| `model = 'hosted_vllm/erp-tune-v7-nvfp4a16'` (the artifact) | **77** |

**Wrong by 4.7x, in the direction that looks careful.** Reporting 363 would have
overstated the operator's own exposure nearly fivefold, and nothing about the
query would have looked sloppy — an alias is what a caller types, so counting it
feels like counting usage.

**The rule:** when the question is *"what did this artifact do"*, filter on the
artifact's identity, never on a mutable pointer to it. A name that has been
repointed carries the history of everything it ever pointed at. Same family as §3
(identity, not resemblance): an alias resembles the thing and is not it.

### A fourth variant: the instrument read a surface MID-TRANSITION

Added 2026-09-09 from a near-miss brokkr-smithy-dev caught and did not send.
Verifying the `trial` alias removal, its first read returned 34 aliases with the
alias **still present** — which looked exactly like the fix had not taken. It
had. The read had raced the gateway restart. The tell was that the next three
reads came back non-JSON, because the service was mid-restart; waiting for it to
settle returned 33 and no alias.

Had that first read been sent, it would have been a **false alarm during an
incident**, and the cost is specific: the other party goes back to re-verify a
fix that was already correct, on the word of an observer who sampled a surface at
a moment nobody meant to ask about. The instrument answered honestly about the
wrong instant.

**The rule: a disagreement between two observers is not a finding until the
boring explanation is ruled out** — a race, a restart, a cache, a stale read.
During an incident the pressure runs the other way, because a discrepancy feels
urgent and urgency argues for sending it immediately. Read twice, let the surface
settle, and prefer the explanation that requires nothing to be wrong.

**How to apply.** When you write or review a guard, open its writer in the same
pass and state the contract out loud — `touch` → exists-but-empty; `mv` on
exit-0 → may be empty; `>` → may be truncated; `pgrep -f <literal>` → matches
your own argv. Then ask which direction this predicate fails toward. Sibling of
§3 (identity, not resemblance) and §2 (observe the state, don't infer it).

## Superseded claims

| date | claim | correction |
|---|---|---|
| 2026-08-18 | "ESH behind CGNAT will break the inter-site tunnels, so IPv6 is the escape hatch" | **Half true, and the halves matter.** Tested live on RFC1918 double-NAT (`192.168.200.111`): **Site Magic (NH3↔ESH, WireGuard) HELD** — it dials out to NH3's public edge and never needs inbound reachability. **IPsec (colo↔ESH, ana-gw FortiGate) BROKE** — traceroute showed traffic unencapsulated, leaking to the carrier. IPv6 keeps its justification on the IPsec link only. |
| 2026-08-18 | *(my own, same day)* "no addressing outcome on the fiber threatens the inter-site tunnel" | **Over-generalised.** I proved it for WireGuard and wrote it as if it covered every link. Operator caught it. See lesson 8. |