runbook(fv-outage): operator ruling — undersized UPS; NAT demoted; ammeter protocol for the visit

Operator's reasoning, accepted and better than the hypothesis-space argument it
replaces: the NAT change went effective, was verified bidirectional, and then ran
correctly for twenty minutes before the site died the moment GPU load was applied. A
working config change does not spontaneously fail under an unrelated physical variable.
The load correlation is tight; the NAT correlation is merely adjacent in time.
Undersized UPS is the only candidate that explains the trigger. NAT material retained
as record, and the power.log/uptime check demoted from decision point to free
confirmation.

Adds the measurement protocol, since the operator is bringing a PDU and an ammeter.
The load-bearing caveat: power.log is GPU-ONLY -- nvidia-smi per-card, excluding CPU,
566 GB of RAM, drives, fans and PSU conversion losses -- so the ammeter at the plug is
the primary instrument and power.log only cross-checks the GPU share. Four states to
capture (idle, one card, two cards, four cards), and capture PEAK rather than average:
UPS overload protection responds to short-term overload, so an average-only reading
that hides transients will mis-size the replacement exactly the way the present unit
got mis-sized, and must be recorded as a floor rather than as the draw.

The four-card figure is earmarked for servers/fv-ml1/README.md, because it closes the
cutover's own open question -- that the FV circuit was likely specced against half the
real draw, back when every record still said the box had two GPUs.

docs/runbooks/fv-site-dark-20260913.md

@@ -72,7 +72,54 @@ the means to bypass it.
 and is recoverable only from `power.log`. Do not let anyone put a wattage in a
 purchasing decision until that file has been read.
 
-## ⚠⚠ A FOURTH CANDIDATE, and the test that settles all of them for free
+## ⭐ OPERATOR RULING 2026-09-13: undersized UPS. NAT hypothesis DEMOTED.
+
+> "highly doubt the nat hypothesis, it went effective (connectivity confirmed for
+> previously dark path) -- then 20 minutes later, during load, site went dark. that's
+> pretty unlikely to be the cause. for sure I think the ups was undersized."
+
+Accepted, and the reasoning is better than the hypothesis-space argument below: **a
+config change that went effective, was verified bidirectional, and then ran correctly
+for twenty minutes does not spontaneously fail when an unrelated physical variable --
+someone else's GPU load -- is introduced.** The load correlation is tight; the NAT
+correlation is merely adjacent in time. Undersized UPS is the only candidate that
+explains the *trigger*.
+
+The NAT material below is retained as record, not as a live competing theory, and the
+`power.log` / `uptime` check is retained as **free confirmation** rather than as a
+decision point.
+
+## Measurement plan for the site visit (operator bringing a PDU + ammeter)
+
+⚠ **`power.log` is GPU-ONLY.** It samples `nvidia-smi` per-card draw and does **not**
+include the host: CPU, 566 GB of RAM, drives, fans, or PSU conversion losses. The
+number that matters against a UPS rating is the whole chassis **at the plug**. The
+ammeter is therefore the primary instrument and `power.log` is a cross-check on the
+GPU share.
+
+Capture four states -- this is the first real sizing data that has ever existed for
+this box:
+
+| state | why it matters |
+|---|---|
+| all seats down, idle | the floor (GPU idle measured 3.80 / 3.88 / 14.27 / 6.75 W) |
+| one card loaded | the condition that ran fine for a day |
+| **two cards loaded** | the condition that took the site down |
+| **four cards loaded** | the only number that can size a replacement honestly |
+
+⚠⚠ **CAPTURE PEAK, NOT AVERAGE.** GPU power has fast transients and UPS overload
+protection responds to short-term overload, so a 1-second-average reading can
+under-read peaks badly. Use max-hold/peak capture if the meter has it. A figure like
+"1100 W average" that hides 1600 W spikes will mis-size the replacement the same way
+the current unit got mis-sized. **If the meter is average-only, record the number as a
+FLOOR, not as the draw.**
+
+⭐ **The four-card figure goes into `servers/fv-ml1/README.md` permanently.** The
+cutover notes flagged that the FV circuit was "likely specced against half the real
+draw" while every record still said two GPUs; this closes that with a measurement
+instead of an assumption.
+
+## The NAT change, retained as record (DEMOTED — see the ruling above)
 
 **Another session applied a Tailscale SNAT rule to the FV gateway at ~06:22Z — 34
 minutes before the site went dark.** See `docs/runbooks/fv-to-ana-nat.md` (uncommitted

persistent-memory.md

@@ -116,17 +116,22 @@ _As of 2026-09-12 ~19:30 PT._
 
 Every Fountain Valley address is unreachable, **including the BMC** — the OOB path
 goes through the same OPNsense gateway, which is also dark. All other sites healthy.
-Leading cause: the 1500 VA Eaton UPS overloaded and died under two-card GPU load —
+⭐ **Cause, per operator ruling 2026-09-13: the 1500 VA Eaton UPS was UNDERSIZED** for
-**but NOT established.** ⚠ Another session applied a Tailscale SNAT rule to the FV
+this chassis and gave way under two-card GPU load. A Tailscale SNAT change on the FV
-gateway at 06:22Z, 34 min before the outage (`docs/runbooks/fv-to-ana-nat.md`); on the
+gateway 34 min earlier was considered and **DEMOTED** — it went effective, was verified
-evidence it is the wrong shape to have caused this (outbound-only, routes and filter
+bidirectional, and then ran correctly for 20 minutes before the site died the moment
-untouched, verified bidirectional afterwards incl. ANA→FV SSH and Beszel 18/18), but it
+load was applied; a working config change does not spontaneously fail under someone
-means power is not the only candidate. ⭐⭐ **The discriminator is free: `power.log`
+else's GPU load. The load correlation is tight, the NAT correlation merely adjacent.
-writes locally every 10 s and does not need the network. Entries past 06:56Z ⇒ the box
+⚠ **Do NOT use a UPS's surge-only outlets to exceed its rating** — both banks share one
-never lost power and the UPS is innocent; stopping at 06:56Z ⇒ power.** So the FIRST
+NEMA 5-15P inlet rated 12 A total; the surge bank bypasses the inverter, not the current
-action on site is `uptime` + `journalctl --list-boots` + `tail power.log` — READ before
+limit. Bypass entirely instead. Operator bringing a **PDU + ammeter** 2026-09-14.
-fixing, because the two hypotheses need completely different remediations and only one
+⚠ **`power.log` is GPU-ONLY** (nvidia-smi per-card; excludes CPU, 566 GB RAM, drives,
-of them needs new hardware.
+fans, PSU losses) — the ammeter at the plug is the primary instrument, power.log is a
+cross-check on the GPU share. Measure idle / 1-card / **2-card** / **4-card**, and
+⚠⚠ **capture PEAK not average** (UPS overload protection responds to short-term
+overload; an average-only meter's figure is a FLOOR, not the draw). ⭐ The 4-card number
+goes into `servers/fv-ml1/README.md` permanently — it closes the cutover's open
+"circuit specced against half the real draw" question with a measurement.
 **It will not self-recover; do not poll FV addresses.** 19 of 30 gateway aliases are
 down with no local fallback (every free local model was on fv-ml1; irv-ml1 runs no
 chat seat). On recovery: recover `power.log` first, mask Docker before the network,