runbook(fv-outage): operator ruling — undersized UPS; NAT demoted; ammeter protocol for the visit

Operator's reasoning, accepted and better than the hypothesis-space argument it
replaces: the NAT change went effective, was verified bidirectional, and then ran
correctly for twenty minutes before the site died the moment GPU load was applied. A
working config change does not spontaneously fail under an unrelated physical variable.
The load correlation is tight; the NAT correlation is merely adjacent in time.
Undersized UPS is the only candidate that explains the trigger. NAT material retained
as record, and the power.log/uptime check demoted from decision point to free
confirmation.

Adds the measurement protocol, since the operator is bringing a PDU and an ammeter.
The load-bearing caveat: power.log is GPU-ONLY -- nvidia-smi per-card, excluding CPU,
566 GB of RAM, drives, fans and PSU conversion losses -- so the ammeter at the plug is
the primary instrument and power.log only cross-checks the GPU share. Four states to
capture (idle, one card, two cards, four cards), and capture PEAK rather than average:
UPS overload protection responds to short-term overload, so an average-only reading
that hides transients will mis-size the replacement exactly the way the present unit
got mis-sized, and must be recorded as a floor rather than as the draw.

The four-card figure is earmarked for servers/fv-ml1/README.md, because it closes the
cutover's own open question -- that the FV circuit was likely specced against half the
real draw, back when every record still said the box had two GPUs.
This commit is contained in:
vh
2026-09-13 00:19:26 -07:00
parent 59ddedd980
commit 00b842bb9b
2 changed files with 64 additions and 12 deletions
+48 -1
View File
@@ -72,7 +72,54 @@ the means to bypass it.
and is recoverable only from `power.log`. Do not let anyone put a wattage in a
purchasing decision until that file has been read.
## ⚠⚠ A FOURTH CANDIDATE, and the test that settles all of them for free
## ⭐ OPERATOR RULING 2026-09-13: undersized UPS. NAT hypothesis DEMOTED.
> "highly doubt the nat hypothesis, it went effective (connectivity confirmed for
> previously dark path) -- then 20 minutes later, during load, site went dark. that's
> pretty unlikely to be the cause. for sure I think the ups was undersized."
Accepted, and the reasoning is better than the hypothesis-space argument below: **a
config change that went effective, was verified bidirectional, and then ran correctly
for twenty minutes does not spontaneously fail when an unrelated physical variable --
someone else's GPU load -- is introduced.** The load correlation is tight; the NAT
correlation is merely adjacent in time. Undersized UPS is the only candidate that
explains the *trigger*.
The NAT material below is retained as record, not as a live competing theory, and the
`power.log` / `uptime` check is retained as **free confirmation** rather than as a
decision point.
## Measurement plan for the site visit (operator bringing a PDU + ammeter)
⚠ **`power.log` is GPU-ONLY.** It samples `nvidia-smi` per-card draw and does **not**
include the host: CPU, 566 GB of RAM, drives, fans, or PSU conversion losses. The
number that matters against a UPS rating is the whole chassis **at the plug**. The
ammeter is therefore the primary instrument and `power.log` is a cross-check on the
GPU share.
Capture four states -- this is the first real sizing data that has ever existed for
this box:
| state | why it matters |
|---|---|
| all seats down, idle | the floor (GPU idle measured 3.80 / 3.88 / 14.27 / 6.75 W) |
| one card loaded | the condition that ran fine for a day |
| **two cards loaded** | the condition that took the site down |
| **four cards loaded** | the only number that can size a replacement honestly |
⚠⚠ **CAPTURE PEAK, NOT AVERAGE.** GPU power has fast transients and UPS overload
protection responds to short-term overload, so a 1-second-average reading can
under-read peaks badly. Use max-hold/peak capture if the meter has it. A figure like
"1100 W average" that hides 1600 W spikes will mis-size the replacement the same way
the current unit got mis-sized. **If the meter is average-only, record the number as a
FLOOR, not as the draw.**
⭐ **The four-card figure goes into `servers/fv-ml1/README.md` permanently.** The
cutover notes flagged that the FV circuit was "likely specced against half the real
draw" while every record still said two GPUs; this closes that with a measurement
instead of an assumption.
## The NAT change, retained as record (DEMOTED — see the ruling above)
**Another session applied a Tailscale SNAT rule to the FV gateway at ~06:22Z — 34
minutes before the site went dark.** See `docs/runbooks/fv-to-ana-nat.md` (uncommitted