runbook(fv-outage): a NAT change 34 min earlier means power is not established — and power.log settles it for free
Another session applied a scoped Tailscale SNAT rule to the FV gateway at ~06:22Z, 34 minutes before the site went dark (docs/runbooks/fv-to-ana-nat.md, not my work, left uncommitted). That makes the UPS-overload theory a hypothesis rather than a finding, and nobody should buy hardware on it until the discriminator below has been read. On the evidence that change is the wrong shape to have caused this, and it is recorded as such so the visit is not wasted chasing it: one OUTBOUND SNAT rule scoped to a single source /32 and a single destination /16 cannot stop the gateway, the BMC or the public WAN address from answering inbound; no routes, filter rules, WAN settings or subnet advertisements were touched; pfctl -sr came back byte-identical; and it was verified bidirectional afterwards including ANA->FV SSH with Beszel 18/18 up. Their BMC datapoint used 10.251.50.50, which is not the BMC -- that is 10.251.250.50, a different subnet. They correctly declined to claim BMC health, but the observation is void rather than negative and should not be reasoned from. The discriminator costs nothing and is already on disk: power.log is written locally to /tank every 10 s by a shell loop on the box and does not depend on the network. Entries past 06:56Z mean the machine never lost power, which makes this a routing fault and the UPS innocent; entries stopping at 06:56Z confirm power. Cross-check with uptime and journalctl --list-boots -- continuous uptime across 06:56Z kills the UPS theory outright. So the first action on site is now to READ, not to fix. The two hypotheses lead to completely different remediations and only one of them needs a new UPS.
This commit is contained in:
+11
-1
@@ -116,7 +116,17 @@ _As of 2026-09-12 ~19:30 PT._
|
||||
|
||||
Every Fountain Valley address is unreachable, **including the BMC** — the OOB path
|
||||
goes through the same OPNsense gateway, which is also dark. All other sites healthy.
|
||||
Leading cause: the 1500 VA Eaton UPS overloaded and died under two-card GPU load.
|
||||
Leading cause: the 1500 VA Eaton UPS overloaded and died under two-card GPU load —
|
||||
**but NOT established.** ⚠ Another session applied a Tailscale SNAT rule to the FV
|
||||
gateway at 06:22Z, 34 min before the outage (`docs/runbooks/fv-to-ana-nat.md`); on the
|
||||
evidence it is the wrong shape to have caused this (outbound-only, routes and filter
|
||||
untouched, verified bidirectional afterwards incl. ANA→FV SSH and Beszel 18/18), but it
|
||||
means power is not the only candidate. ⭐⭐ **The discriminator is free: `power.log`
|
||||
writes locally every 10 s and does not need the network. Entries past 06:56Z ⇒ the box
|
||||
never lost power and the UPS is innocent; stopping at 06:56Z ⇒ power.** So the FIRST
|
||||
action on site is `uptime` + `journalctl --list-boots` + `tail power.log` — READ before
|
||||
fixing, because the two hypotheses need completely different remediations and only one
|
||||
of them needs new hardware.
|
||||
**It will not self-recover; do not poll FV addresses.** 19 of 30 gateway aliases are
|
||||
down with no local fallback (every free local model was on fv-ml1; irv-ml1 runs no
|
||||
chat seat). On recovery: recover `power.log` first, mask Docker before the network,
|
||||
|
||||
Reference in New Issue
Block a user