From 5a24d77f12c868ecb8b5ae68c5c357b970d0c308 Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Wed, 2 Sep 2026 01:16:20 -0700 Subject: [PATCH] =?UTF-8?q?memory:=20run-3c=20probe=20answered=20=E2=80=94?= =?UTF-8?q?=2079.3=20s/it,=20and=20it=20reverses=20the=20plan=20on=20file?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The GX10 runs run 3c at about 79.3 s/it with 35 GB of headroom, which puts 604 steps at 13.3 hours against ana-ml2's 2.2 to 2.7. Six times slower where raw compute predicts 2.7, which points at memory bandwidth rather than FLOPs -- recorded as a hypothesis, since confirming it needs a bandwidth-bound microbenchmark nobody has run. That reverses the standing plan. Moving run 3c here was framed as the power answer, but the run did not die because ana-ml2 is unreliable. It died because save_steps was 100 and the breaker tripped at step 80, so no checkpoint existed. save_steps is now 50, which caps a power event at about eleven minutes. Trading 2.5 hours for 13.3 buys insurance against a risk already engineered out. Also records the five launch failures and their causes, and the one that matters most: the attention-backend trap was present and I first declared it absent. I checked whether flash-attn was installed, which is the wrong discriminator; the harness sets flex_attention explicitly in code. Absence of an alternative is not evidence of the default. The probe now reads the resolved backend back off the loaded model, and flex_attention does compile and run on sm_121. --- .../2026-09-01-pfi-gx10-onboarding.md | 68 +++++++++++++++++++ 1 file changed, 68 insertions(+) diff --git a/persistent-memory.d/2026-09-01-pfi-gx10-onboarding.md b/persistent-memory.d/2026-09-01-pfi-gx10-onboarding.md index 8cbf13c..058b13e 100644 --- a/persistent-memory.d/2026-09-01-pfi-gx10-onboarding.md +++ b/persistent-memory.d/2026-09-01-pfi-gx10-onboarding.md @@ -312,3 +312,71 @@ padding on the real run is 0.0% so the corpus does not change per-step cost mate direct because ana-ml2's key is not authorised for `infra-ops@pfi-gx10` — and relaying costs nothing extra, since the Wi-Fi leg is the bottleneck either way. **Deliberately did not add an access grant just to save a hop.** + +## `[2026-09-02]` RUN-3c THROUGHPUT PROBE — ANSWERED. ~79.3 s/it, and the box is 6x slower than ana-ml2 + +### The number + + pfi-gx10 79.30 / 79.37 / 79.33 / 79.45 s/it (0.19% spread across timed steps) + -> 604 steps ~= 13.3 h + memory 86 / 121 GB, ~35 GB headroom -- NOT memory-constrained + ana-ml2 10.8-15.8 s/it (run-03c.log, the real run) + -> 604 steps ~= 2.2-2.7 h + +⚠ **6x slower where raw compute predicts 2.7x** (93 TFLOP/s bf16 measured here vs an RTX PRO +6000 Blackwell's ~250). **Hypothesis, not a measurement:** the limiter is **memory bandwidth** +— GB10's unified LPDDR5X against dedicated graphics memory, on a workload built to be +bandwidth-hungry (seq 16384 with gradient checkpointing, which trades compute for memory +traffic by design). Confirming it needs a bandwidth-bound microbenchmark that has not been run. + +⚠ **THE GX10 IS A CAPACITY BOX, NOT A THROUGHPUT BOX.** 121 GB holds models nothing else in the +fleet can. It is **not** the box to move a time-sensitive run *to*. + +### ⚠ THE RECOMMENDATION REVERSES THE PLAN ON FILE — run 3c belongs on ana-ml2 + +The standing plan was "move run 3c to the GX10; it is the power answer rather than a power +triage." **Measurement kills that.** Run 3c did not die because ana-ml2 is unreliable — it died +because `save_steps` was **100** and the 2026-08-26 breaker trip landed at **step 80**, so no +checkpoint existed. **`save_steps` is now 50.** A power event costs **≤50 steps ≈ 11 min** and +resumes. Trading 2.5 h for 13.3 h buys insurance against a risk already engineered out. + +### Probe methodology — five launches, five distinct causes, all recorded + + 1. reaped SSH long remote job piped through a reapable foreground ssh. + FIX: setsid nohup, log ON the box. Work survived; output did not. + 2. bare LoRA target names q_proj/k_proj/... also match Gemma-4's VISION and AUDIO towers, + whose projections are Gemma4ClippableLinear -- PEFT cannot wrap + them. FIX: the real run's 205 fully-qualified names, sha recorded. + 3. missing python3-dev Triton JITs a C shim at FIRST USE and needs Python.h. Torch + imports, model loads, LoRA attaches, THEN step 1 dies with a bare + CalledProcessError -- Triton devnulls the compiler's stdout. + 4. naive-loss OOM (32 GiB) the 262,144-wide vocab x 32,768 positions logits tensor in fp32. + ⚠ NOT a GX10 limit -- the harness docstring says it OOMs "on a + 95.60 GiB card (measured)", i.e. on ana-ml2 too. + 5. self-inflicted VRAM leak a stuck orphan held 80 GB while PyTorch reported 0 bytes + allocated; every relaunch was doomed before it started and blamed + the NEW run. ⚠ ASSERT THE GPU IS CLEAR BEFORE EVERY LAUNCH. + +### ⚠ THE BACKEND-DELTA TRAP WAS PRESENT AND I FIRST DECLARED IT ABSENT + +CLAUDE.md records two frontier-panel conclusions voided by a benchmark and a trainer having +silently different attention backends. I checked for it by asking whether **flash-attn was +installed** — the wrong discriminator entirely — and reported "no delta". The harness sets +**`ATTN_IMPLEMENTATION = "flex_attention"`** explicitly in code, with tuned kernel options; its +own comment records peak 74.3 GiB under flex vs sdpa's 79.71. + +⚠ **Read the configured value in the code. Absence of an alternative is not evidence of the +default.** The probe now requests flex_attention and reads `_attn_implementation` back off the +loaded model. **RESOLVED: flex_attention — it compiles and runs on sm_121**, which was a real +open risk (flex compiles through Triton; sm_121 is not in torch's arch list). + +### Faithfulness of the probe (what makes the number comparable) + + LoRA the real run's 205 fully-qualified modules, sha256 c9ee070c0d67eb08 + loss path copied from the harness: trunk hidden states, DROP IGNORE_INDEX before the + head, chunk lm_head+CE at loss_chunk_tokens=1024 under checkpoint, softcap 30.0 + loss ratio 28.4% of positions masked to match the real 13,310,930 / 18,600,057 loss/ctx + ratio -- observed 23,351-23,638 loss tokens/step against a 23,449 target + tokens sampled [1000, 250000) to stay below every multimodal special token + (boi 255999, boa 256000, image 258880, audio 258881, eoi/eoa/video 258882-4) + warmup first 2 steps discarded; sm_121 JITs from sm_120 PTX on first use