memory: run 3c goes to the GX10 — I priced the failure in the wrong units

The operator overruled my ana-ml2 recommendation and was right. I had step
times in hand, so I priced a breaker trip as eleven minutes of lost
training. That is the recompute cost and it is the cheapest component of
the loss.

A breaker trip at Anaheim is a forty-minute drive each way on the
operator's time, whenever he happens to notice, with thirteen hosts dark
until he arrives -- including the hypervisor most of them run on, the
fleet's primary backup server, and three SureFire client machines that are
a customer's production hosts under a hosting agreement.

So save_steps 100 to 50 caps the recompute, not the outage, and it was
never the mitigation I claimed. Thirteen hours unattended on a desk in NH3
beats two and a half hours that can put a client's hosts dark, especially
when nothing is waiting on this run.

Recorded the general form at length because it is the transferable part:
when recommending between options, check whether you priced the failure
mode in whatever units you happened to be measuring. A metric in hand will
volunteer itself as the unit of risk.

Also measured and dismissed the obvious third option: the RTX PRO 6000s
have a 250 W floor against a 300 W default, so capping both saves 100 W on
a box drawing about a kilowatt. Not enough to matter, and it costs
throughput to buy.
This commit is contained in:
vh
2026-09-02 01:17:49 -07:00
parent 5a24d77f12
commit dce261335e
@@ -332,13 +332,44 @@ traffic by design). Confirming it needs a bandwidth-bound microbenchmark that ha
⚠ **THE GX10 IS A CAPACITY BOX, NOT A THROUGHPUT BOX.** 121 GB holds models nothing else in the
fleet can. It is **not** the box to move a time-sensitive run *to*.
### ⚠ THE RECOMMENDATION REVERSES THE PLAN ON FILE — run 3c belongs on ana-ml2
### ✅ RUN 3c BELONGS ON THE GX10 — the plan on file was right and infra-ops' reversal was WRONG
The standing plan was "move run 3c to the GX10; it is the power answer rather than a power
triage." **Measurement kills that.** Run 3c did not die because ana-ml2 is unreliable — it died
because `save_steps` was **100** and the 2026-08-26 breaker trip landed at **step 80**, so no
checkpoint existed. **`save_steps` is now 50.** A power event costs **≤50 steps ≈ 11 min** and
resumes. Trading 2.5 h for 13.3 h buys insurance against a risk already engineered out.
⚠⚠ **infra-ops recommended ana-ml2 and the operator overruled it, correctly.** The error is
worth more than the recommendation: **I costed the failure in the units I was measuring.** I had
step times, so I priced a breaker trip as *"≤50 steps ≈ 11 minutes of lost training"* — the
recompute cost. That is not what a breaker trip at Anaheim costs.
**Operator, 2026-09-02:** *"when the breaker trips at ana, that's a 40 minute drive to reset it.
it's not 25 minutes, it's a giant pain in the ass."*
**The real cost of an Anaheim breaker trip:**
a 40-minute drive EACH WAY, on the operator's time, whenever he notices
13 hosts down: ana-ml2, ana-docker, pfi-pve (hypervisor for most of them),
pfi-postgres, pbs-ana (the fleet's PRIMARY backup server), ana-wg (WireGuard),
pfi-ana-webhost, ana-filebot, pfi-pteradactyl, pfi-tacticalrmm
⚠ including THREE SureFire CLIENT hosts (sfsrv-ana, sf-ana-container, sf-r630)
-- a client's production machines, under a hosting agreement
⚠ **`save_steps` 100 → 50 caps the RECOMPUTE, not the OUTAGE.** It was never the mitigation I
claimed; it only bounds the cheapest component of the loss.
**So: 13.3 h unattended on a desk in NH3, drawing desktop-class power and unable to take down
anything, beats 2.5 h that can put a client's hosts dark until someone drives to Anaheim.** The
GX10's slowness is close to free here — nothing is waiting on this run — and the risk it removes
is expensive and lands on the operator personally.
⚠ **GENERAL FORM, and the reason this is recorded at length: when you recommend between
options, check whether you priced the failure mode in the units you happened to be measuring.**
A metric in hand will volunteer itself as the unit of risk.
### ⚠ Power-capping ana-ml2 is a WEAK lever — measured, do not spend time on it
2x RTX PRO 6000 Blackwell Max-Q: limit 300 W, default 300 W, min 250 W, max 325 W
Capping both GPUs to the floor saves **100 W total** on a box drawing ~1 kW. Not nothing, but
nowhere near enough to make an Anaheim training run safe, and it costs throughput to buy it.
Recorded so nobody re-derives it as a clever third option.
### Probe methodology — five launches, five distinct causes, all recorded