runbook(fv-outage): a NAT change 34 min earlier means power is not established — and power.log settles it for free

Another session applied a scoped Tailscale SNAT rule to the FV gateway at ~06:22Z, 34
minutes before the site went dark (docs/runbooks/fv-to-ana-nat.md, not my work, left
uncommitted). That makes the UPS-overload theory a hypothesis rather than a finding,
and nobody should buy hardware on it until the discriminator below has been read.

On the evidence that change is the wrong shape to have caused this, and it is recorded
as such so the visit is not wasted chasing it: one OUTBOUND SNAT rule scoped to a
single source /32 and a single destination /16 cannot stop the gateway, the BMC or the
public WAN address from answering inbound; no routes, filter rules, WAN settings or
subnet advertisements were touched; pfctl -sr came back byte-identical; and it was
verified bidirectional afterwards including ANA->FV SSH with Beszel 18/18 up.

Their BMC datapoint used 10.251.50.50, which is not the BMC -- that is 10.251.250.50,
a different subnet. They correctly declined to claim BMC health, but the observation is
void rather than negative and should not be reasoned from.

The discriminator costs nothing and is already on disk: power.log is written locally to
/tank every 10 s by a shell loop on the box and does not depend on the network.
Entries past 06:56Z mean the machine never lost power, which makes this a routing fault
and the UPS innocent; entries stopping at 06:56Z confirm power. Cross-check with uptime
and journalctl --list-boots -- continuous uptime across 06:56Z kills the UPS theory
outright.

So the first action on site is now to READ, not to fix. The two hypotheses lead to
completely different remediations and only one of them needs a new UPS.
This commit is contained in:
vh
2026-09-13 00:16:25 -07:00
parent 524da702ef
commit 59ddedd980
2 changed files with 54 additions and 2 deletions
+11 -1
View File
@@ -116,7 +116,17 @@ _As of 2026-09-12 ~19:30 PT._
Every Fountain Valley address is unreachable, **including the BMC** — the OOB path
goes through the same OPNsense gateway, which is also dark. All other sites healthy.
Leading cause: the 1500 VA Eaton UPS overloaded and died under two-card GPU load.
Leading cause: the 1500 VA Eaton UPS overloaded and died under two-card GPU load —
**but NOT established.** ⚠ Another session applied a Tailscale SNAT rule to the FV
gateway at 06:22Z, 34 min before the outage (`docs/runbooks/fv-to-ana-nat.md`); on the
evidence it is the wrong shape to have caused this (outbound-only, routes and filter
untouched, verified bidirectional afterwards incl. ANA→FV SSH and Beszel 18/18), but it
means power is not the only candidate. ⭐⭐ **The discriminator is free: `power.log`
writes locally every 10 s and does not need the network. Entries past 06:56Z ⇒ the box
never lost power and the UPS is innocent; stopping at 06:56Z ⇒ power.** So the FIRST
action on site is `uptime` + `journalctl --list-boots` + `tail power.log` — READ before
fixing, because the two hypotheses need completely different remediations and only one
of them needs new hardware.
**It will not self-recover; do not poll FV addresses.** 19 of 30 gateway aliases are
down with no local fallback (every free local model was on fv-ml1; irv-ml1 runs no
chat seat). On recovery: recover `power.log` first, mask Docker before the network,