memory: run 3c goes to the GX10 — I priced the failure in the wrong units
The operator overruled my ana-ml2 recommendation and was right. I had step times in hand, so I priced a breaker trip as eleven minutes of lost training. That is the recompute cost and it is the cheapest component of the loss. A breaker trip at Anaheim is a forty-minute drive each way on the operator's time, whenever he happens to notice, with thirteen hosts dark until he arrives -- including the hypervisor most of them run on, the fleet's primary backup server, and three SureFire client machines that are a customer's production hosts under a hosting agreement. So save_steps 100 to 50 caps the recompute, not the outage, and it was never the mitigation I claimed. Thirteen hours unattended on a desk in NH3 beats two and a half hours that can put a client's hosts dark, especially when nothing is waiting on this run. Recorded the general form at length because it is the transferable part: when recommending between options, check whether you priced the failure mode in whatever units you happened to be measuring. A metric in hand will volunteer itself as the unit of risk. Also measured and dismissed the obvious third option: the RTX PRO 6000s have a 250 W floor against a 300 W default, so capping both saves 100 W on a box drawing about a kilowatt. Not enough to matter, and it costs throughput to buy.
This commit is contained in:
@@ -332,13 +332,44 @@ traffic by design). Confirming it needs a bandwidth-bound microbenchmark that ha
|
||||
⚠ **THE GX10 IS A CAPACITY BOX, NOT A THROUGHPUT BOX.** 121 GB holds models nothing else in the
|
||||
fleet can. It is **not** the box to move a time-sensitive run *to*.
|
||||
|
||||
### ⚠ THE RECOMMENDATION REVERSES THE PLAN ON FILE — run 3c belongs on ana-ml2
|
||||
### ✅ RUN 3c BELONGS ON THE GX10 — the plan on file was right and infra-ops' reversal was WRONG
|
||||
|
||||
The standing plan was "move run 3c to the GX10; it is the power answer rather than a power
|
||||
triage." **Measurement kills that.** Run 3c did not die because ana-ml2 is unreliable — it died
|
||||
because `save_steps` was **100** and the 2026-08-26 breaker trip landed at **step 80**, so no
|
||||
checkpoint existed. **`save_steps` is now 50.** A power event costs **≤50 steps ≈ 11 min** and
|
||||
resumes. Trading 2.5 h for 13.3 h buys insurance against a risk already engineered out.
|
||||
⚠⚠ **infra-ops recommended ana-ml2 and the operator overruled it, correctly.** The error is
|
||||
worth more than the recommendation: **I costed the failure in the units I was measuring.** I had
|
||||
step times, so I priced a breaker trip as *"≤50 steps ≈ 11 minutes of lost training"* — the
|
||||
recompute cost. That is not what a breaker trip at Anaheim costs.
|
||||
|
||||
**Operator, 2026-09-02:** *"when the breaker trips at ana, that's a 40 minute drive to reset it.
|
||||
it's not 25 minutes, it's a giant pain in the ass."*
|
||||
|
||||
**The real cost of an Anaheim breaker trip:**
|
||||
|
||||
a 40-minute drive EACH WAY, on the operator's time, whenever he notices
|
||||
13 hosts down: ana-ml2, ana-docker, pfi-pve (hypervisor for most of them),
|
||||
pfi-postgres, pbs-ana (the fleet's PRIMARY backup server), ana-wg (WireGuard),
|
||||
pfi-ana-webhost, ana-filebot, pfi-pteradactyl, pfi-tacticalrmm
|
||||
⚠ including THREE SureFire CLIENT hosts (sfsrv-ana, sf-ana-container, sf-r630)
|
||||
-- a client's production machines, under a hosting agreement
|
||||
|
||||
⚠ **`save_steps` 100 → 50 caps the RECOMPUTE, not the OUTAGE.** It was never the mitigation I
|
||||
claimed; it only bounds the cheapest component of the loss.
|
||||
|
||||
**So: 13.3 h unattended on a desk in NH3, drawing desktop-class power and unable to take down
|
||||
anything, beats 2.5 h that can put a client's hosts dark until someone drives to Anaheim.** The
|
||||
GX10's slowness is close to free here — nothing is waiting on this run — and the risk it removes
|
||||
is expensive and lands on the operator personally.
|
||||
|
||||
⚠ **GENERAL FORM, and the reason this is recorded at length: when you recommend between
|
||||
options, check whether you priced the failure mode in the units you happened to be measuring.**
|
||||
A metric in hand will volunteer itself as the unit of risk.
|
||||
|
||||
### ⚠ Power-capping ana-ml2 is a WEAK lever — measured, do not spend time on it
|
||||
|
||||
2x RTX PRO 6000 Blackwell Max-Q: limit 300 W, default 300 W, min 250 W, max 325 W
|
||||
|
||||
Capping both GPUs to the floor saves **100 W total** on a box drawing ~1 kW. Not nothing, but
|
||||
nowhere near enough to make an Anaheim training run safe, and it costs throughput to buy it.
|
||||
Recorded so nobody re-derives it as a clever third option.
|
||||
|
||||
### Probe methodology — five launches, five distinct causes, all recorded
|
||||
|
||||
|
||||
Reference in New Issue
Block a user