runbook(fv-outage): a NAT change 34 min earlier means power is not established — and power.log settles it for free

Another session applied a scoped Tailscale SNAT rule to the FV gateway at ~06:22Z, 34
minutes before the site went dark (docs/runbooks/fv-to-ana-nat.md, not my work, left
uncommitted). That makes the UPS-overload theory a hypothesis rather than a finding,
and nobody should buy hardware on it until the discriminator below has been read.

On the evidence that change is the wrong shape to have caused this, and it is recorded
as such so the visit is not wasted chasing it: one OUTBOUND SNAT rule scoped to a
single source /32 and a single destination /16 cannot stop the gateway, the BMC or the
public WAN address from answering inbound; no routes, filter rules, WAN settings or
subnet advertisements were touched; pfctl -sr came back byte-identical; and it was
verified bidirectional afterwards including ANA->FV SSH with Beszel 18/18 up.

Their BMC datapoint used 10.251.50.50, which is not the BMC -- that is 10.251.250.50,
a different subnet. They correctly declined to claim BMC health, but the observation is
void rather than negative and should not be reasoned from.

The discriminator costs nothing and is already on disk: power.log is written locally to
/tank every 10 s by a shell loop on the box and does not depend on the network.
Entries past 06:56Z mean the machine never lost power, which makes this a routing fault
and the UPS innocent; entries stopping at 06:56Z confirm power. Cross-check with uptime
and journalctl --list-boots -- continuous uptime across 06:56Z kills the UPS theory
outright.

So the first action on site is now to READ, not to fix. The two hypotheses lead to
completely different remediations and only one of them needs a new UPS.

docs/runbooks/fv-site-dark-20260913.md

@@ -72,7 +72,49 @@ the means to bypass it.
 and is recoverable only from `power.log`. Do not let anyone put a wattage in a
 purchasing decision until that file has been read.
 
-## Three candidate causes, and what distinguishes them
+## ⚠⚠ A FOURTH CANDIDATE, and the test that settles all of them for free
+
+**Another session applied a Tailscale SNAT rule to the FV gateway at ~06:22Z — 34
+minutes before the site went dark.** See `docs/runbooks/fv-to-ana-nat.md` (uncommitted
+as of this writing; not my work, left alone). So "we overloaded the power" is no longer
+the only live hypothesis, and the UPS should not be replaced on the strength of a theory
+until the one below is run.
+
+**On the evidence, that change is the WRONG SHAPE to have caused this**, and I want that
+on the record so nobody wastes the visit chasing it:
+
+- It is one **outbound** SNAT rule, source-scoped to `10.251.50.54/32`, destination-
+  scoped to `10.250.0.0/16`. An outbound NAT rule cannot stop the gateway, the BMC or
+  the public WAN address from answering **inbound**.
+- The runbook states no routes, filter rules, WAN settings or subnet advertisements were
+  touched, and that `pfctl -sr` came back byte-identical.
+- It was verified working in **both** directions afterwards: FV→hub HTTP 200, FV→ANA
+  TCP 5432, FV internet HTTPS 200, **ANA→FV SSH reachable**, gateway management intact,
+  Beszel **18/18 up**.
+
+⚠ Note their BMC observation used **`10.251.50.50`**, which is not the BMC — the BMC is
+**`10.251.250.50`**, a different subnet. They correctly declined to claim BMC health, but
+the datapoint is *void*, not negative. Do not reason from it either way.
+
+### ⭐⭐ THE DISCRIMINATOR — run this before forming any conclusion
+
+`power.log` is written **locally on `/tank`, every 10 seconds, by a shell loop on the
+box.** It does not depend on the network. So:
+
+| `power.log` last entry | what it means |
+|---|---|
+| **past 06:56Z** | the box **never lost power**. This is a routing/gateway fault, and the UPS is innocent. |
+| **stops at ~06:56Z** | the box lost power. UPS/circuit confirmed. |
+
+Cross-check with `uptime` and `journalctl --list-boots` the moment there is a console:
+**continuous uptime across 06:56Z kills the UPS theory outright.**
+
+⚠ **So the FIRST action on site is to read, not to fix.** `uptime`,
+`journalctl --list-boots`, then `tail power.log`. Establish whether the machine ever
+went down before anyone buys hardware or flips anything — the two hypotheses lead to
+completely different remediations and only one of them needs a new UPS.
+
+## Three further candidate causes, and what distinguishes them
 
 Cannot be distinguished remotely, because every FV path — including the BMC —
 traverses the OPNsense gateway, and the gateway is also dark.

persistent-memory.md

@@ -116,7 +116,17 @@ _As of 2026-09-12 ~19:30 PT._
 
 Every Fountain Valley address is unreachable, **including the BMC** — the OOB path
 goes through the same OPNsense gateway, which is also dark. All other sites healthy.
-Leading cause: the 1500 VA Eaton UPS overloaded and died under two-card GPU load.
+Leading cause: the 1500 VA Eaton UPS overloaded and died under two-card GPU load —
+**but NOT established.** ⚠ Another session applied a Tailscale SNAT rule to the FV
+gateway at 06:22Z, 34 min before the outage (`docs/runbooks/fv-to-ana-nat.md`); on the
+evidence it is the wrong shape to have caused this (outbound-only, routes and filter
+untouched, verified bidirectional afterwards incl. ANA→FV SSH and Beszel 18/18), but it
+means power is not the only candidate. ⭐⭐ **The discriminator is free: `power.log`
+writes locally every 10 s and does not need the network. Entries past 06:56Z ⇒ the box
+never lost power and the UPS is innocent; stopping at 06:56Z ⇒ power.** So the FIRST
+action on site is `uptime` + `journalctl --list-boots` + `tail power.log` — READ before
+fixing, because the two hypotheses need completely different remediations and only one
+of them needs new hardware.
 **It will not self-recover; do not poll FV addresses.** 19 of 30 gateway aliases are
 down with no local fallback (every free local model was on fv-ml1; irv-ml1 runs no
 chat seat). On recovery: recover `power.log` first, mask Docker before the network,