diff --git a/docs/runbooks/fv-site-dark-20260913.md b/docs/runbooks/fv-site-dark-20260913.md index d0ee7c7..9d1ceb7 100644 --- a/docs/runbooks/fv-site-dark-20260913.md +++ b/docs/runbooks/fv-site-dark-20260913.md @@ -72,7 +72,49 @@ the means to bypass it. and is recoverable only from `power.log`. Do not let anyone put a wattage in a purchasing decision until that file has been read. -## Three candidate causes, and what distinguishes them +## ⚠⚠ A FOURTH CANDIDATE, and the test that settles all of them for free + +**Another session applied a Tailscale SNAT rule to the FV gateway at ~06:22Z — 34 +minutes before the site went dark.** See `docs/runbooks/fv-to-ana-nat.md` (uncommitted +as of this writing; not my work, left alone). So "we overloaded the power" is no longer +the only live hypothesis, and the UPS should not be replaced on the strength of a theory +until the one below is run. + +**On the evidence, that change is the WRONG SHAPE to have caused this**, and I want that +on the record so nobody wastes the visit chasing it: + +- It is one **outbound** SNAT rule, source-scoped to `10.251.50.54/32`, destination- + scoped to `10.250.0.0/16`. An outbound NAT rule cannot stop the gateway, the BMC or + the public WAN address from answering **inbound**. +- The runbook states no routes, filter rules, WAN settings or subnet advertisements were + touched, and that `pfctl -sr` came back byte-identical. +- It was verified working in **both** directions afterwards: FV→hub HTTP 200, FV→ANA + TCP 5432, FV internet HTTPS 200, **ANA→FV SSH reachable**, gateway management intact, + Beszel **18/18 up**. + +⚠ Note their BMC observation used **`10.251.50.50`**, which is not the BMC — the BMC is +**`10.251.250.50`**, a different subnet. They correctly declined to claim BMC health, but +the datapoint is *void*, not negative. Do not reason from it either way. + +### ⭐⭐ THE DISCRIMINATOR — run this before forming any conclusion + +`power.log` is written **locally on `/tank`, every 10 seconds, by a shell loop on the +box.** It does not depend on the network. So: + +| `power.log` last entry | what it means | +|---|---| +| **past 06:56Z** | the box **never lost power**. This is a routing/gateway fault, and the UPS is innocent. | +| **stops at ~06:56Z** | the box lost power. UPS/circuit confirmed. | + +Cross-check with `uptime` and `journalctl --list-boots` the moment there is a console: +**continuous uptime across 06:56Z kills the UPS theory outright.** + +⚠ **So the FIRST action on site is to read, not to fix.** `uptime`, +`journalctl --list-boots`, then `tail power.log`. Establish whether the machine ever +went down before anyone buys hardware or flips anything — the two hypotheses lead to +completely different remediations and only one of them needs a new UPS. + +## Three further candidate causes, and what distinguishes them Cannot be distinguished remotely, because every FV path — including the BMC — traverses the OPNsense gateway, and the gateway is also dark. diff --git a/persistent-memory.md b/persistent-memory.md index 699dd31..4da694c 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -116,7 +116,17 @@ _As of 2026-09-12 ~19:30 PT._ Every Fountain Valley address is unreachable, **including the BMC** — the OOB path goes through the same OPNsense gateway, which is also dark. All other sites healthy. -Leading cause: the 1500 VA Eaton UPS overloaded and died under two-card GPU load. +Leading cause: the 1500 VA Eaton UPS overloaded and died under two-card GPU load — +**but NOT established.** ⚠ Another session applied a Tailscale SNAT rule to the FV +gateway at 06:22Z, 34 min before the outage (`docs/runbooks/fv-to-ana-nat.md`); on the +evidence it is the wrong shape to have caused this (outbound-only, routes and filter +untouched, verified bidirectional afterwards incl. ANA→FV SSH and Beszel 18/18), but it +means power is not the only candidate. ⭐⭐ **The discriminator is free: `power.log` +writes locally every 10 s and does not need the network. Entries past 06:56Z ⇒ the box +never lost power and the UPS is innocent; stopping at 06:56Z ⇒ power.** So the FIRST +action on site is `uptime` + `journalctl --list-boots` + `tail power.log` — READ before +fixing, because the two hypotheses need completely different remediations and only one +of them needs new hardware. **It will not self-recover; do not poll FV addresses.** 19 of 30 gateway aliases are down with no local fallback (every free local model was on fv-ml1; irv-ml1 runs no chat seat). On recovery: recover `power.log` first, mask Docker before the network,