runbook(fv-outage): the circuit case — split power survives a trip on battery, but only if the colo handoff does
Operator: 'unless of course the thing trips the circuit anyway.' Correct, and it splits into two halves with different answers. A breaker trip is the event the split-power proposal survives: firewall + BMC is 25-40 W on a 1500 VA unit, which is hours of battery, and on a trip the UPS stops being a load-bearing supply and goes back to being what it is for. What it does NOT cover is the colo's own handoff -- their switch, ONT or demarc. If that sits on the circuit we just tripped, the outcome is a firewall running on battery with nothing upstream to talk to and the drive happens anyway. Added as a question for the facility, because it decides whether split power delivers remote diagnosis or merely feels like it does. Records the case where none of it matters: removing an undersized UPS does not remove the constraint, it promotes the next one -- UPS ~900-1200 W to circuit ~1800 W at 15 A or ~2400 W at 20 A. Which side the four-card figure lands on decides everything, which is what makes that single ammeter reading the load-bearing measurement of the visit. Surfaces the lever that may avoid an electrician entirely: nvidia-smi -pl caps per-card TGP, so the box can be made to fit its feed at a throughput cost rather than a rewiring cost. Read nvidia-smi -q -d POWER for the enforced range before assuming how much room the dial has, and persist any cap -- one that evaporates on reboot will hold right up until the next power event and then silently stop holding.
This commit is contained in:
@@ -132,6 +132,17 @@ cross-check on the GPU share. Measure idle / 1-card / **2-card** / **4-card**, a
|
||||
overload; an average-only meter's figure is a FLOOR, not the draw). ⭐ The 4-card number
|
||||
goes into `servers/fv-ml1/README.md` permanently — it closes the cutover's open
|
||||
"circuit specced against half the real draw" question with a measurement.
|
||||
⚠⚠ **Removing an undersized UPS does not remove the constraint, it promotes the next
|
||||
one** (UPS ~900-1200 W → circuit ~1800 W @15 A / ~2400 W @20 A). If 4 cards + host
|
||||
exceeds the circuit, NO UPS arrangement helps. ⭐ **The lever is `nvidia-smi -pl`** —
|
||||
cap per-card TGP so the box fits its feed at a throughput cost instead of a rewiring
|
||||
cost; read `nvidia-smi -q -d POWER` for the enforced range first, and **persist the cap**
|
||||
(a limit that evaporates on reboot holds until the next power event and then does not).
|
||||
⭐ Split power (firewall+BMC on UPS, chassis on PDU) survives a *breaker* trip on
|
||||
battery — 25-40 W on a 1500 VA unit is hours — **but only delivers remote access if the
|
||||
COLO'S HANDOFF survives too**; ask the facility whether the handoff is on our circuit or
|
||||
theirs. Three questions for the visit: breaker rating, is the circuit dedicated, whose
|
||||
gear is the handoff on.
|
||||
**It will not self-recover; do not poll FV addresses.** 19 of 30 gateway aliases are
|
||||
down with no local fallback (every free local model was on fv-ml1; irv-ml1 runs no
|
||||
chat seat). On recovery: recover `power.log` first, mask Docker before the network,
|
||||
|
||||
Reference in New Issue
Block a user