From 00b842bb9b72a1d801a9a83fa1ce12fa43288b91 Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Sun, 13 Sep 2026 00:19:26 -0700 Subject: [PATCH] =?UTF-8?q?runbook(fv-outage):=20operator=20ruling=20?= =?UTF-8?q?=E2=80=94=20undersized=20UPS;=20NAT=20demoted;=20ammeter=20prot?= =?UTF-8?q?ocol=20for=20the=20visit?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Operator's reasoning, accepted and better than the hypothesis-space argument it replaces: the NAT change went effective, was verified bidirectional, and then ran correctly for twenty minutes before the site died the moment GPU load was applied. A working config change does not spontaneously fail under an unrelated physical variable. The load correlation is tight; the NAT correlation is merely adjacent in time. Undersized UPS is the only candidate that explains the trigger. NAT material retained as record, and the power.log/uptime check demoted from decision point to free confirmation. Adds the measurement protocol, since the operator is bringing a PDU and an ammeter. The load-bearing caveat: power.log is GPU-ONLY -- nvidia-smi per-card, excluding CPU, 566 GB of RAM, drives, fans and PSU conversion losses -- so the ammeter at the plug is the primary instrument and power.log only cross-checks the GPU share. Four states to capture (idle, one card, two cards, four cards), and capture PEAK rather than average: UPS overload protection responds to short-term overload, so an average-only reading that hides transients will mis-size the replacement exactly the way the present unit got mis-sized, and must be recorded as a floor rather than as the draw. The four-card figure is earmarked for servers/fv-ml1/README.md, because it closes the cutover's own open question -- that the FV circuit was likely specced against half the real draw, back when every record still said the box had two GPUs. --- docs/runbooks/fv-site-dark-20260913.md | 49 +++++++++++++++++++++++++- persistent-memory.md | 27 ++++++++------ 2 files changed, 64 insertions(+), 12 deletions(-) diff --git a/docs/runbooks/fv-site-dark-20260913.md b/docs/runbooks/fv-site-dark-20260913.md index 9d1ceb7..3780d56 100644 --- a/docs/runbooks/fv-site-dark-20260913.md +++ b/docs/runbooks/fv-site-dark-20260913.md @@ -72,7 +72,54 @@ the means to bypass it. and is recoverable only from `power.log`. Do not let anyone put a wattage in a purchasing decision until that file has been read. -## ⚠⚠ A FOURTH CANDIDATE, and the test that settles all of them for free +## ⭐ OPERATOR RULING 2026-09-13: undersized UPS. NAT hypothesis DEMOTED. + +> "highly doubt the nat hypothesis, it went effective (connectivity confirmed for +> previously dark path) -- then 20 minutes later, during load, site went dark. that's +> pretty unlikely to be the cause. for sure I think the ups was undersized." + +Accepted, and the reasoning is better than the hypothesis-space argument below: **a +config change that went effective, was verified bidirectional, and then ran correctly +for twenty minutes does not spontaneously fail when an unrelated physical variable -- +someone else's GPU load -- is introduced.** The load correlation is tight; the NAT +correlation is merely adjacent in time. Undersized UPS is the only candidate that +explains the *trigger*. + +The NAT material below is retained as record, not as a live competing theory, and the +`power.log` / `uptime` check is retained as **free confirmation** rather than as a +decision point. + +## Measurement plan for the site visit (operator bringing a PDU + ammeter) + +⚠ **`power.log` is GPU-ONLY.** It samples `nvidia-smi` per-card draw and does **not** +include the host: CPU, 566 GB of RAM, drives, fans, or PSU conversion losses. The +number that matters against a UPS rating is the whole chassis **at the plug**. The +ammeter is therefore the primary instrument and `power.log` is a cross-check on the +GPU share. + +Capture four states -- this is the first real sizing data that has ever existed for +this box: + +| state | why it matters | +|---|---| +| all seats down, idle | the floor (GPU idle measured 3.80 / 3.88 / 14.27 / 6.75 W) | +| one card loaded | the condition that ran fine for a day | +| **two cards loaded** | the condition that took the site down | +| **four cards loaded** | the only number that can size a replacement honestly | + +⚠⚠ **CAPTURE PEAK, NOT AVERAGE.** GPU power has fast transients and UPS overload +protection responds to short-term overload, so a 1-second-average reading can +under-read peaks badly. Use max-hold/peak capture if the meter has it. A figure like +"1100 W average" that hides 1600 W spikes will mis-size the replacement the same way +the current unit got mis-sized. **If the meter is average-only, record the number as a +FLOOR, not as the draw.** + +⭐ **The four-card figure goes into `servers/fv-ml1/README.md` permanently.** The +cutover notes flagged that the FV circuit was "likely specced against half the real +draw" while every record still said two GPUs; this closes that with a measurement +instead of an assumption. + +## The NAT change, retained as record (DEMOTED — see the ruling above) **Another session applied a Tailscale SNAT rule to the FV gateway at ~06:22Z — 34 minutes before the site went dark.** See `docs/runbooks/fv-to-ana-nat.md` (uncommitted diff --git a/persistent-memory.md b/persistent-memory.md index 4da694c..086c42e 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -116,17 +116,22 @@ _As of 2026-09-12 ~19:30 PT._ Every Fountain Valley address is unreachable, **including the BMC** — the OOB path goes through the same OPNsense gateway, which is also dark. All other sites healthy. -Leading cause: the 1500 VA Eaton UPS overloaded and died under two-card GPU load — -**but NOT established.** ⚠ Another session applied a Tailscale SNAT rule to the FV -gateway at 06:22Z, 34 min before the outage (`docs/runbooks/fv-to-ana-nat.md`); on the -evidence it is the wrong shape to have caused this (outbound-only, routes and filter -untouched, verified bidirectional afterwards incl. ANA→FV SSH and Beszel 18/18), but it -means power is not the only candidate. ⭐⭐ **The discriminator is free: `power.log` -writes locally every 10 s and does not need the network. Entries past 06:56Z ⇒ the box -never lost power and the UPS is innocent; stopping at 06:56Z ⇒ power.** So the FIRST -action on site is `uptime` + `journalctl --list-boots` + `tail power.log` — READ before -fixing, because the two hypotheses need completely different remediations and only one -of them needs new hardware. +⭐ **Cause, per operator ruling 2026-09-13: the 1500 VA Eaton UPS was UNDERSIZED** for +this chassis and gave way under two-card GPU load. A Tailscale SNAT change on the FV +gateway 34 min earlier was considered and **DEMOTED** — it went effective, was verified +bidirectional, and then ran correctly for 20 minutes before the site died the moment +load was applied; a working config change does not spontaneously fail under someone +else's GPU load. The load correlation is tight, the NAT correlation merely adjacent. +⚠ **Do NOT use a UPS's surge-only outlets to exceed its rating** — both banks share one +NEMA 5-15P inlet rated 12 A total; the surge bank bypasses the inverter, not the current +limit. Bypass entirely instead. Operator bringing a **PDU + ammeter** 2026-09-14. +⚠ **`power.log` is GPU-ONLY** (nvidia-smi per-card; excludes CPU, 566 GB RAM, drives, +fans, PSU losses) — the ammeter at the plug is the primary instrument, power.log is a +cross-check on the GPU share. Measure idle / 1-card / **2-card** / **4-card**, and +⚠⚠ **capture PEAK not average** (UPS overload protection responds to short-term +overload; an average-only meter's figure is a FLOOR, not the draw). ⭐ The 4-card number +goes into `servers/fv-ml1/README.md` permanently — it closes the cutover's open +"circuit specced against half the real draw" question with a measurement. **It will not self-recover; do not poll FV addresses.** 19 of 30 gateway aliases are down with no local fallback (every free local model was on fv-ml1; irv-ml1 runs no chat seat). On recovery: recover `power.log` first, mask Docker before the network,