runbook(fv-outage): operator ruling — undersized UPS; NAT demoted; ammeter protocol for the visit

Operator's reasoning, accepted and better than the hypothesis-space argument it
replaces: the NAT change went effective, was verified bidirectional, and then ran
correctly for twenty minutes before the site died the moment GPU load was applied. A
working config change does not spontaneously fail under an unrelated physical variable.
The load correlation is tight; the NAT correlation is merely adjacent in time.
Undersized UPS is the only candidate that explains the trigger. NAT material retained
as record, and the power.log/uptime check demoted from decision point to free
confirmation.

Adds the measurement protocol, since the operator is bringing a PDU and an ammeter.
The load-bearing caveat: power.log is GPU-ONLY -- nvidia-smi per-card, excluding CPU,
566 GB of RAM, drives, fans and PSU conversion losses -- so the ammeter at the plug is
the primary instrument and power.log only cross-checks the GPU share. Four states to
capture (idle, one card, two cards, four cards), and capture PEAK rather than average:
UPS overload protection responds to short-term overload, so an average-only reading
that hides transients will mis-size the replacement exactly the way the present unit
got mis-sized, and must be recorded as a floor rather than as the draw.

The four-card figure is earmarked for servers/fv-ml1/README.md, because it closes the
cutover's own open question -- that the FV circuit was likely specced against half the
real draw, back when every record still said the box had two GPUs.

persistent-memory.md

@@ -116,17 +116,22 @@ _As of 2026-09-12 ~19:30 PT._
 
 Every Fountain Valley address is unreachable, **including the BMC** — the OOB path
 goes through the same OPNsense gateway, which is also dark. All other sites healthy.
-Leading cause: the 1500 VA Eaton UPS overloaded and died under two-card GPU load —
-**but NOT established.** ⚠ Another session applied a Tailscale SNAT rule to the FV
-gateway at 06:22Z, 34 min before the outage (`docs/runbooks/fv-to-ana-nat.md`); on the
-evidence it is the wrong shape to have caused this (outbound-only, routes and filter
-untouched, verified bidirectional afterwards incl. ANA→FV SSH and Beszel 18/18), but it
-means power is not the only candidate. ⭐⭐ **The discriminator is free: `power.log`
-writes locally every 10 s and does not need the network. Entries past 06:56Z ⇒ the box
-never lost power and the UPS is innocent; stopping at 06:56Z ⇒ power.** So the FIRST
-action on site is `uptime` + `journalctl --list-boots` + `tail power.log` — READ before
-fixing, because the two hypotheses need completely different remediations and only one
-of them needs new hardware.
+⭐ **Cause, per operator ruling 2026-09-13: the 1500 VA Eaton UPS was UNDERSIZED** for
+this chassis and gave way under two-card GPU load. A Tailscale SNAT change on the FV
+gateway 34 min earlier was considered and **DEMOTED** — it went effective, was verified
+bidirectional, and then ran correctly for 20 minutes before the site died the moment
+load was applied; a working config change does not spontaneously fail under someone
+else's GPU load. The load correlation is tight, the NAT correlation merely adjacent.
+⚠ **Do NOT use a UPS's surge-only outlets to exceed its rating** — both banks share one
+NEMA 5-15P inlet rated 12 A total; the surge bank bypasses the inverter, not the current
+limit. Bypass entirely instead. Operator bringing a **PDU + ammeter** 2026-09-14.
+⚠ **`power.log` is GPU-ONLY** (nvidia-smi per-card; excludes CPU, 566 GB RAM, drives,
+fans, PSU losses) — the ammeter at the plug is the primary instrument, power.log is a
+cross-check on the GPU share. Measure idle / 1-card / **2-card** / **4-card**, and
+⚠⚠ **capture PEAK not average** (UPS overload protection responds to short-term
+overload; an average-only meter's figure is a FLOOR, not the draw). ⭐ The 4-card number
+goes into `servers/fv-ml1/README.md` permanently — it closes the cutover's open
+"circuit specced against half the real draw" question with a measurement.
 **It will not self-recover; do not poll FV addresses.** 19 of 30 gateway aliases are
 down with no local fallback (every free local model was on fv-ml1; irv-ml1 runs no
 chat seat). On recovery: recover `power.log` first, mask Docker before the network,