diff --git a/persistent-memory.d/2026-09-01-pfi-gx10-onboarding.md b/persistent-memory.d/2026-09-01-pfi-gx10-onboarding.md index 058b13e..85b6b9d 100644 --- a/persistent-memory.d/2026-09-01-pfi-gx10-onboarding.md +++ b/persistent-memory.d/2026-09-01-pfi-gx10-onboarding.md @@ -332,13 +332,44 @@ traffic by design). Confirming it needs a bandwidth-bound microbenchmark that ha ⚠ **THE GX10 IS A CAPACITY BOX, NOT A THROUGHPUT BOX.** 121 GB holds models nothing else in the fleet can. It is **not** the box to move a time-sensitive run *to*. -### ⚠ THE RECOMMENDATION REVERSES THE PLAN ON FILE — run 3c belongs on ana-ml2 +### ✅ RUN 3c BELONGS ON THE GX10 — the plan on file was right and infra-ops' reversal was WRONG -The standing plan was "move run 3c to the GX10; it is the power answer rather than a power -triage." **Measurement kills that.** Run 3c did not die because ana-ml2 is unreliable — it died -because `save_steps` was **100** and the 2026-08-26 breaker trip landed at **step 80**, so no -checkpoint existed. **`save_steps` is now 50.** A power event costs **≤50 steps ≈ 11 min** and -resumes. Trading 2.5 h for 13.3 h buys insurance against a risk already engineered out. +⚠⚠ **infra-ops recommended ana-ml2 and the operator overruled it, correctly.** The error is +worth more than the recommendation: **I costed the failure in the units I was measuring.** I had +step times, so I priced a breaker trip as *"≤50 steps ≈ 11 minutes of lost training"* — the +recompute cost. That is not what a breaker trip at Anaheim costs. + +**Operator, 2026-09-02:** *"when the breaker trips at ana, that's a 40 minute drive to reset it. +it's not 25 minutes, it's a giant pain in the ass."* + +**The real cost of an Anaheim breaker trip:** + + a 40-minute drive EACH WAY, on the operator's time, whenever he notices + 13 hosts down: ana-ml2, ana-docker, pfi-pve (hypervisor for most of them), + pfi-postgres, pbs-ana (the fleet's PRIMARY backup server), ana-wg (WireGuard), + pfi-ana-webhost, ana-filebot, pfi-pteradactyl, pfi-tacticalrmm + ⚠ including THREE SureFire CLIENT hosts (sfsrv-ana, sf-ana-container, sf-r630) + -- a client's production machines, under a hosting agreement + +⚠ **`save_steps` 100 → 50 caps the RECOMPUTE, not the OUTAGE.** It was never the mitigation I +claimed; it only bounds the cheapest component of the loss. + +**So: 13.3 h unattended on a desk in NH3, drawing desktop-class power and unable to take down +anything, beats 2.5 h that can put a client's hosts dark until someone drives to Anaheim.** The +GX10's slowness is close to free here — nothing is waiting on this run — and the risk it removes +is expensive and lands on the operator personally. + +⚠ **GENERAL FORM, and the reason this is recorded at length: when you recommend between +options, check whether you priced the failure mode in the units you happened to be measuring.** +A metric in hand will volunteer itself as the unit of risk. + +### ⚠ Power-capping ana-ml2 is a WEAK lever — measured, do not spend time on it + + 2x RTX PRO 6000 Blackwell Max-Q: limit 300 W, default 300 W, min 250 W, max 325 W + +Capping both GPUs to the floor saves **100 W total** on a box drawing ~1 kW. Not nothing, but +nowhere near enough to make an Anaheim training run safe, and it costs throughput to buy it. +Recorded so nobody re-derives it as a clever third option. ### Probe methodology — five launches, five distinct causes, all recorded