From dce261335e6fddbd26475f1d946b852a75db7031 Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Wed, 2 Sep 2026 01:17:49 -0700 Subject: [PATCH] =?UTF-8?q?memory:=20run=203c=20goes=20to=20the=20GX10=20?= =?UTF-8?q?=E2=80=94=20I=20priced=20the=20failure=20in=20the=20wrong=20uni?= =?UTF-8?q?ts?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The operator overruled my ana-ml2 recommendation and was right. I had step times in hand, so I priced a breaker trip as eleven minutes of lost training. That is the recompute cost and it is the cheapest component of the loss. A breaker trip at Anaheim is a forty-minute drive each way on the operator's time, whenever he happens to notice, with thirteen hosts dark until he arrives -- including the hypervisor most of them run on, the fleet's primary backup server, and three SureFire client machines that are a customer's production hosts under a hosting agreement. So save_steps 100 to 50 caps the recompute, not the outage, and it was never the mitigation I claimed. Thirteen hours unattended on a desk in NH3 beats two and a half hours that can put a client's hosts dark, especially when nothing is waiting on this run. Recorded the general form at length because it is the transferable part: when recommending between options, check whether you priced the failure mode in whatever units you happened to be measuring. A metric in hand will volunteer itself as the unit of risk. Also measured and dismissed the obvious third option: the RTX PRO 6000s have a 250 W floor against a 300 W default, so capping both saves 100 W on a box drawing about a kilowatt. Not enough to matter, and it costs throughput to buy. --- .../2026-09-01-pfi-gx10-onboarding.md | 43 ++++++++++++++++--- 1 file changed, 37 insertions(+), 6 deletions(-) diff --git a/persistent-memory.d/2026-09-01-pfi-gx10-onboarding.md b/persistent-memory.d/2026-09-01-pfi-gx10-onboarding.md index 058b13e..85b6b9d 100644 --- a/persistent-memory.d/2026-09-01-pfi-gx10-onboarding.md +++ b/persistent-memory.d/2026-09-01-pfi-gx10-onboarding.md @@ -332,13 +332,44 @@ traffic by design). Confirming it needs a bandwidth-bound microbenchmark that ha ⚠ **THE GX10 IS A CAPACITY BOX, NOT A THROUGHPUT BOX.** 121 GB holds models nothing else in the fleet can. It is **not** the box to move a time-sensitive run *to*. -### ⚠ THE RECOMMENDATION REVERSES THE PLAN ON FILE — run 3c belongs on ana-ml2 +### ✅ RUN 3c BELONGS ON THE GX10 — the plan on file was right and infra-ops' reversal was WRONG -The standing plan was "move run 3c to the GX10; it is the power answer rather than a power -triage." **Measurement kills that.** Run 3c did not die because ana-ml2 is unreliable — it died -because `save_steps` was **100** and the 2026-08-26 breaker trip landed at **step 80**, so no -checkpoint existed. **`save_steps` is now 50.** A power event costs **≤50 steps ≈ 11 min** and -resumes. Trading 2.5 h for 13.3 h buys insurance against a risk already engineered out. +⚠⚠ **infra-ops recommended ana-ml2 and the operator overruled it, correctly.** The error is +worth more than the recommendation: **I costed the failure in the units I was measuring.** I had +step times, so I priced a breaker trip as *"≤50 steps ≈ 11 minutes of lost training"* — the +recompute cost. That is not what a breaker trip at Anaheim costs. + +**Operator, 2026-09-02:** *"when the breaker trips at ana, that's a 40 minute drive to reset it. +it's not 25 minutes, it's a giant pain in the ass."* + +**The real cost of an Anaheim breaker trip:** + + a 40-minute drive EACH WAY, on the operator's time, whenever he notices + 13 hosts down: ana-ml2, ana-docker, pfi-pve (hypervisor for most of them), + pfi-postgres, pbs-ana (the fleet's PRIMARY backup server), ana-wg (WireGuard), + pfi-ana-webhost, ana-filebot, pfi-pteradactyl, pfi-tacticalrmm + ⚠ including THREE SureFire CLIENT hosts (sfsrv-ana, sf-ana-container, sf-r630) + -- a client's production machines, under a hosting agreement + +⚠ **`save_steps` 100 → 50 caps the RECOMPUTE, not the OUTAGE.** It was never the mitigation I +claimed; it only bounds the cheapest component of the loss. + +**So: 13.3 h unattended on a desk in NH3, drawing desktop-class power and unable to take down +anything, beats 2.5 h that can put a client's hosts dark until someone drives to Anaheim.** The +GX10's slowness is close to free here — nothing is waiting on this run — and the risk it removes +is expensive and lands on the operator personally. + +⚠ **GENERAL FORM, and the reason this is recorded at length: when you recommend between +options, check whether you priced the failure mode in the units you happened to be measuring.** +A metric in hand will volunteer itself as the unit of risk. + +### ⚠ Power-capping ana-ml2 is a WEAK lever — measured, do not spend time on it + + 2x RTX PRO 6000 Blackwell Max-Q: limit 300 W, default 300 W, min 250 W, max 325 W + +Capping both GPUs to the floor saves **100 W total** on a box drawing ~1 kW. Not nothing, but +nowhere near enough to make an Anaheim training run safe, and it costs throughput to buy it. +Recorded so nobody re-derives it as a clever third option. ### Probe methodology — five launches, five distinct causes, all recorded