diff --git a/docs/runbooks/fv-site-dark-20260913.md b/docs/runbooks/fv-site-dark-20260913.md index ed23efd..127d0b1 100644 --- a/docs/runbooks/fv-site-dark-20260913.md +++ b/docs/runbooks/fv-site-dark-20260913.md @@ -32,6 +32,43 @@ answer with 0% loss, so this is FV-local, not an isolation of the observer. So the site went dark inside a ~2.5 minute window, roughly five minutes into sustained bench load on GPU 3 with GPU 2 also resident/serving. +## ⭐ LEADING HYPOTHESIS (operator, 2026-09-13): the UPS overloaded and died + +Operator's read, and it fits the evidence **better** than the breaker-trip theory +below, for a reason worth writing down: + +**A UPS's output rating is far below the circuit's, so the UPS is the FIRST protective +device to give.** A common 1500 VA rack unit delivers ~900-1000 W; a 15 A circuit +carries ~1800 W. Two of these cards under load, plus a host carrying 566 GB of RAM, +plus the firewall, plausibly clears the UPS rating while staying comfortably under the +breaker. That explains the thing a breaker trip explains poorly: **why it let go at +TWO cards loaded and not four.** It also explains the OPNsense box dying -- same UPS. + +⚠ **"Died" is the operationally important word.** A tripped UPS resets; an overloaded +one can kill its output stage or battery pack permanently. If it is dead rather than +tripped, **nothing on site can be reset back to life** and the trip is wasted without +the means to bypass it. + +### Bring / check list for the site visit + +- **The means to BYPASS the UPS entirely** -- chassis and firewall straight to the + PDU or wall. Do this first, get the site back, diagnose the UPS after. +- **Read off the UPS before moving it:** make/model, VA and W rating, fault LEDs, and + any LCD or event log entry (many units record "overload" explicitly). +- **Note which outlets were battery-backed vs surge-only.** Mixed-bank units are + common, and a GPU chassis on the battery-backed bank overloads soonest. +- ⭐ **Recover `/tank/aimodels/flash-next-mtp-bench/power.log` FIRST, before any + bring-up.** It sampled all four cards every 10 s right up to the cut and is the + ONLY measurement of what the load actually drew. It lives on `/tank`, not in a + container, so it survived. Without it, the replacement UPS gets sized by guesswork. +- ⚠ **Size the replacement for FOUR cards under load, not two.** Anything sized to + today's failure just relocates the trip point to the next person who loads all four. + +⚠ **No measured load figure exists yet.** Idle draw was measured (GPU0 3.80 W, GPU1 +3.88 W, GPU2 14.27 W, GPU3 6.75 W at 06:51:17Z) but the load figure died with the box +and is recoverable only from `power.log`. Do not let anyone put a wattage in a +purchasing decision until that file has been read. + ## Three candidate causes, and what distinguishes them Cannot be distinguished remotely, because every FV path — including the BMC — @@ -63,8 +100,21 @@ measured it (the power log on fv-ml1) died with the box. qwen-image-bench, qwen3-embedding, reranker, sec, sec-reasoning, summarizer, summarizer-large -Still served: the paid passthroughs (`gen-frontier*`, `glm*`, `kimi*`) and the irv-ml1 -locals. The gateway itself on ana-docker is healthy — it is the backends that are gone. +⚠⚠ **THERE IS NO LOCAL FALLBACK, and an earlier note in this session wrongly implied +there was.** Probed 2026-09-13: every free local model on the gateway lives on +fv-ml1. irv-ml1 runs **no LLM chat seat at all** -- it carries TTS (tts-gateway, +breeze-tts, voice-studio, omnivoice-ref), ComfyUI, arbo, yt-voice-clipper, bragi and +waterland-studio, and its two Ampere cards are partly occupied by them. The only +non-fv chat backends on the gateway are **paid**: api.z.ai (9 aliases) and +Moonshot/Kimi (2). + +So the choice during the outage is: leave the 19 aliases failing loudly, or spend +vendor credits. ⚠ **If credits are spent, it must be under a NEW alias name that +callers opt into** -- never by silently repointing `summarizer`/`gen`/`classifier` at +GLM. Silent model substitution behind a familiar name is a standing prohibition here +and has already been violated twice; an outage is not an exemption. + +The gateway itself on ana-docker is healthy — it is the backends that are gone. ## ⚠⚠ RECOVERY — do NOT just reset the breaker and walk away