runbook(fv-outage): UPS overload as leading hypothesis, site-visit bring-list, no-local-fallback correction

Operator's read is that the UPS the box was plugged into overloaded and died, and it
fits better than the breaker-trip theory: a UPS's output rating sits far below the
circuit's, so it is the first protective device to give -- which explains why the site
let go at TWO cards loaded rather than four, and why the ~25 W firewall died with it.

Records the operationally important consequence: a tripped UPS resets, an overloaded
one can kill its output stage permanently. If it is dead, nothing on site can be reset
back to life, so the visit needs the means to BYPASS the UPS or it is wasted.

Elevates recovery of /tank/.../power.log to the first action on site. It sampled all
four cards every 10 s up to the cut, lives on /tank rather than in a container, and is
the only measurement of what the load actually drew -- without it a replacement UPS
gets sized by guesswork. Also states that no load figure exists yet, only idle.

Corrects an earlier claim of mine in this session: there is NO local fallback for the
19 dark aliases. Probed -- every free local model is on fv-ml1, and irv-ml1 runs no
chat seat at all, only TTS/ComfyUI/arbo/clipper work on two partly-occupied Ampere
cards. The only non-fv chat backends are paid. Any paid coverage must go under a new
opt-in alias name rather than a silent repoint of summarizer/gen/classifier.

docs/runbooks/fv-site-dark-20260913.md

@@ -32,6 +32,43 @@ answer with 0% loss, so this is FV-local, not an isolation of the observer.
 So the site went dark inside a ~2.5 minute window, roughly five minutes into
 sustained bench load on GPU 3 with GPU 2 also resident/serving.
 
+## ⭐ LEADING HYPOTHESIS (operator, 2026-09-13): the UPS overloaded and died
+
+Operator's read, and it fits the evidence **better** than the breaker-trip theory
+below, for a reason worth writing down:
+
+**A UPS's output rating is far below the circuit's, so the UPS is the FIRST protective
+device to give.** A common 1500 VA rack unit delivers ~900-1000 W; a 15 A circuit
+carries ~1800 W. Two of these cards under load, plus a host carrying 566 GB of RAM,
+plus the firewall, plausibly clears the UPS rating while staying comfortably under the
+breaker. That explains the thing a breaker trip explains poorly: **why it let go at
+TWO cards loaded and not four.** It also explains the OPNsense box dying -- same UPS.
+
+⚠ **"Died" is the operationally important word.** A tripped UPS resets; an overloaded
+one can kill its output stage or battery pack permanently. If it is dead rather than
+tripped, **nothing on site can be reset back to life** and the trip is wasted without
+the means to bypass it.
+
+### Bring / check list for the site visit
+
+- **The means to BYPASS the UPS entirely** -- chassis and firewall straight to the
+  PDU or wall. Do this first, get the site back, diagnose the UPS after.
+- **Read off the UPS before moving it:** make/model, VA and W rating, fault LEDs, and
+  any LCD or event log entry (many units record "overload" explicitly).
+- **Note which outlets were battery-backed vs surge-only.** Mixed-bank units are
+  common, and a GPU chassis on the battery-backed bank overloads soonest.
+- ⭐ **Recover `/tank/aimodels/flash-next-mtp-bench/power.log` FIRST, before any
+  bring-up.** It sampled all four cards every 10 s right up to the cut and is the
+  ONLY measurement of what the load actually drew. It lives on `/tank`, not in a
+  container, so it survived. Without it, the replacement UPS gets sized by guesswork.
+- ⚠ **Size the replacement for FOUR cards under load, not two.** Anything sized to
+  today's failure just relocates the trip point to the next person who loads all four.
+
+⚠ **No measured load figure exists yet.** Idle draw was measured (GPU0 3.80 W, GPU1
+3.88 W, GPU2 14.27 W, GPU3 6.75 W at 06:51:17Z) but the load figure died with the box
+and is recoverable only from `power.log`. Do not let anyone put a wattage in a
+purchasing decision until that file has been read.
+
 ## Three candidate causes, and what distinguishes them
 
 Cannot be distinguished remotely, because every FV path — including the BMC —
@@ -63,8 +100,21 @@ measured it (the power log on fv-ml1) died with the box.
     qwen-image-bench, qwen3-embedding, reranker, sec, sec-reasoning, summarizer,
     summarizer-large
 
-Still served: the paid passthroughs (`gen-frontier*`, `glm*`, `kimi*`) and the irv-ml1
-locals. The gateway itself on ana-docker is healthy — it is the backends that are gone.
+⚠⚠ **THERE IS NO LOCAL FALLBACK, and an earlier note in this session wrongly implied
+there was.** Probed 2026-09-13: every free local model on the gateway lives on
+fv-ml1. irv-ml1 runs **no LLM chat seat at all** -- it carries TTS (tts-gateway,
+breeze-tts, voice-studio, omnivoice-ref), ComfyUI, arbo, yt-voice-clipper, bragi and
+waterland-studio, and its two Ampere cards are partly occupied by them. The only
+non-fv chat backends on the gateway are **paid**: api.z.ai (9 aliases) and
+Moonshot/Kimi (2).
+
+So the choice during the outage is: leave the 19 aliases failing loudly, or spend
+vendor credits. ⚠ **If credits are spent, it must be under a NEW alias name that
+callers opt into** -- never by silently repointing `summarizer`/`gen`/`classifier` at
+GLM. Silent model substitution behind a familiar name is a standing prohibition here
+and has already been violated twice; an outage is not an exemption.
+
+The gateway itself on ana-docker is healthy — it is the backends that are gone.
 
 ## ⚠⚠ RECOVERY — do NOT just reset the breaker and walk away