From b6335bf6ada91556528879d86f78ec19cb9139ba Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Sun, 13 Sep 2026 00:21:36 -0700 Subject: [PATCH] =?UTF-8?q?runbook(fv-outage):=20the=20circuit=20case=20?= =?UTF-8?q?=E2=80=94=20split=20power=20survives=20a=20trip=20on=20battery,?= =?UTF-8?q?=20but=20only=20if=20the=20colo=20handoff=20does?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Operator: 'unless of course the thing trips the circuit anyway.' Correct, and it splits into two halves with different answers. A breaker trip is the event the split-power proposal survives: firewall + BMC is 25-40 W on a 1500 VA unit, which is hours of battery, and on a trip the UPS stops being a load-bearing supply and goes back to being what it is for. What it does NOT cover is the colo's own handoff -- their switch, ONT or demarc. If that sits on the circuit we just tripped, the outcome is a firewall running on battery with nothing upstream to talk to and the drive happens anyway. Added as a question for the facility, because it decides whether split power delivers remote diagnosis or merely feels like it does. Records the case where none of it matters: removing an undersized UPS does not remove the constraint, it promotes the next one -- UPS ~900-1200 W to circuit ~1800 W at 15 A or ~2400 W at 20 A. Which side the four-card figure lands on decides everything, which is what makes that single ammeter reading the load-bearing measurement of the visit. Surfaces the lever that may avoid an electrician entirely: nvidia-smi -pl caps per-card TGP, so the box can be made to fit its feed at a throughput cost rather than a rewiring cost. Read nvidia-smi -q -d POWER for the enforced range before assuming how much room the dial has, and persist any cap -- one that evaporates on reboot will hold right up until the next power event and then silently stop holding. --- docs/runbooks/fv-site-dark-20260913.md | 49 ++++++++++++++++++++++++++ persistent-memory.md | 11 ++++++ 2 files changed, 60 insertions(+) diff --git a/docs/runbooks/fv-site-dark-20260913.md b/docs/runbooks/fv-site-dark-20260913.md index 3780d56..9cb4bd2 100644 --- a/docs/runbooks/fv-site-dark-20260913.md +++ b/docs/runbooks/fv-site-dark-20260913.md @@ -253,6 +253,55 @@ Expect `[]`. Before the outage that query returned 16 entries. **Do not reboot the OPNsense firewall** (standing operator directive; its reboot API 403s anyway). +## ⚠ THE CIRCUIT CASE — what split power does and does not buy (operator, 2026-09-13) + +> "unless of course the thing trips the circuit anyway." + +**It still helps, but only halfway, and the halfway matters.** + +- ✅ **A breaker trip is exactly what the split survives.** Firewall + BMC on the UPS is + ~25-40 W of load on a 1500 VA unit — hours of battery, not minutes. On a trip the UPS + stops being a load-bearing supply and goes back to being what it is for. +- ❌ **A live firewall is useless if the path OUT of the site is dead.** Our UPS covers + our gear; it does not cover the **colo's handoff** — their switch, ONT or demarc. If + that sits on the circuit we just tripped, the result is a firewall running happily on + battery with nothing upstream to talk to, and the drive happens anyway. + ⭐ **ASK THE FACILITY: is the network handoff on our circuit or theirs, and is theirs + on facility UPS?** This is the question that decides whether split power actually + delivers remote diagnosis or merely feels like it does. + +### ⚠⚠ And the case where none of the above matters + +**If four cards plus host exceeds the circuit, no UPS arrangement helps** — the box does +not fit its feed. Removing an undersized UPS does not remove the constraint, it promotes +the next one: + + UPS ~900-1200 W (the one that just gave way) + circuit ~1800 W @ 15 A / ~2400 W @ 20 A + +Which side of those the four-card figure lands on decides everything, which is why that +single ammeter reading is the load-bearing measurement of the visit. + +### ⭐ The lever that may avoid an electrician: per-card power limits + +`nvidia-smi -pl ` caps TGP per card. The box can be made to fit whatever the feed +turns out to be, at a **throughput** cost rather than a **rewiring** cost — four capped +cards on a 15 A circuit is a dial we control today, where a 20 A drop is a ticket and a +site visit. + +- Read `nvidia-smi -q -d POWER` first for the enforced min/max range per card; do not + assume how much room the dial has. +- ⚠ **If capping is the answer it MUST be persisted** (systemd unit, or an `if-up` + equivalent). A limit that evaporates on reboot is worse than no limit, because it will + hold right up until the next power event and then silently stop holding. + +### Three questions for the site visit + +1. What is the **breaker rating** on that circuit? +2. Is the circuit **dedicated** to us, or shared with other racks/tenants? +3. Is the **network handoff** on our circuit or the facility's, and is the facility's on + their UPS? + ## ⚠ PROPOSED, NOT RATIFIED — split the power so the management plane survives **This is a recommendation awaiting the operator's call, not settled intent.** Written diff --git a/persistent-memory.md b/persistent-memory.md index 086c42e..f0d5ee6 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -132,6 +132,17 @@ cross-check on the GPU share. Measure idle / 1-card / **2-card** / **4-card**, a overload; an average-only meter's figure is a FLOOR, not the draw). ⭐ The 4-card number goes into `servers/fv-ml1/README.md` permanently — it closes the cutover's open "circuit specced against half the real draw" question with a measurement. +⚠⚠ **Removing an undersized UPS does not remove the constraint, it promotes the next +one** (UPS ~900-1200 W → circuit ~1800 W @15 A / ~2400 W @20 A). If 4 cards + host +exceeds the circuit, NO UPS arrangement helps. ⭐ **The lever is `nvidia-smi -pl`** — +cap per-card TGP so the box fits its feed at a throughput cost instead of a rewiring +cost; read `nvidia-smi -q -d POWER` for the enforced range first, and **persist the cap** +(a limit that evaporates on reboot holds until the next power event and then does not). +⭐ Split power (firewall+BMC on UPS, chassis on PDU) survives a *breaker* trip on +battery — 25-40 W on a 1500 VA unit is hours — **but only delivers remote access if the +COLO'S HANDOFF survives too**; ask the facility whether the handoff is on our circuit or +theirs. Three questions for the visit: breaker rating, is the circuit dedicated, whose +gear is the handoff on. **It will not self-recover; do not poll FV addresses.** 19 of 30 gateway aliases are down with no local fallback (every free local model was on fv-ml1; irv-ml1 runs no chat seat). On recovery: recover `power.log` first, mask Docker before the network,