vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity

memory: run-3c probe answered — 79.3 s/it, and it reverses the plan on file

Browse Source 
The GX10 runs run 3c at about 79.3 s/it with 35 GB of headroom, which puts
604 steps at 13.3 hours against ana-ml2's 2.2 to 2.7. Six times slower
where raw compute predicts 2.7, which points at memory bandwidth rather
than FLOPs -- recorded as a hypothesis, since confirming it needs a
bandwidth-bound microbenchmark nobody has run.

That reverses the standing plan. Moving run 3c here was framed as the
power answer, but the run did not die because ana-ml2 is unreliable. It
died because save_steps was 100 and the breaker tripped at step 80, so no
checkpoint existed. save_steps is now 50, which caps a power event at
about eleven minutes. Trading 2.5 hours for 13.3 buys insurance against a
risk already engineered out.

Also records the five launch failures and their causes, and the one that
matters most: the attention-backend trap was present and I first declared
it absent. I checked whether flash-attn was installed, which is the wrong
discriminator; the harness sets flex_attention explicitly in code. Absence
of an alternative is not evidence of the default. The probe now reads the
resolved backend back off the loaded model, and flex_attention does
compile and run on sm_121.
This commit is contained in:
Vuong Hoang
2026-09-02 01:16:20 -07:00
parent 84349d7a0e
commit 5a24d77f12
1 changed files with 68 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
persistent-memory.d/2026-09-01-pfi-gx10-onboarding.md
+68
View File
@@ -312,3 +312,71 @@ padding on the real run is 0.0% so the corpus does not change per-step cost mate
direct because ana-ml2's key is not authorised for `infra-ops@pfi-gx10` — and relaying costs
nothing extra, since the Wi-Fi leg is the bottleneck either way. **Deliberately did not add an
access grant just to save a hop.**
## `[2026-09-02]` RUN-3c THROUGHPUT PROBE — ANSWERED. ~79.3 s/it, and the box is 6x slower than ana-ml2
### The number
pfi-gx10 79.30 / 79.37 / 79.33 / 79.45 s/it (0.19% spread across timed steps)
-> 604 steps ~= 13.3 h
memory 86 / 121 GB, ~35 GB headroom -- NOT memory-constrained
ana-ml2 10.8-15.8 s/it (run-03c.log, the real run)
-> 604 steps ~= 2.2-2.7 h
⚠ **6x slower where raw compute predicts 2.7x** (93 TFLOP/s bf16 measured here vs an RTX PRO
6000 Blackwell's ~250). **Hypothesis, not a measurement:** the limiter is **memory bandwidth**
— GB10's unified LPDDR5X against dedicated graphics memory, on a workload built to be
bandwidth-hungry (seq 16384 with gradient checkpointing, which trades compute for memory
traffic by design). Confirming it needs a bandwidth-bound microbenchmark that has not been run.
⚠ **THE GX10 IS A CAPACITY BOX, NOT A THROUGHPUT BOX.** 121 GB holds models nothing else in the
fleet can. It is **not** the box to move a time-sensitive run *to*.
### ⚠ THE RECOMMENDATION REVERSES THE PLAN ON FILE — run 3c belongs on ana-ml2
The standing plan was "move run 3c to the GX10; it is the power answer rather than a power
triage." **Measurement kills that.** Run 3c did not die because ana-ml2 is unreliable — it died
because `save_steps` was **100** and the 2026-08-26 breaker trip landed at **step 80**, so no
checkpoint existed. **`save_steps` is now 50.** A power event costs **≤50 steps ≈ 11 min** and
resumes. Trading 2.5 h for 13.3 h buys insurance against a risk already engineered out.
### Probe methodology — five launches, five distinct causes, all recorded
1. reaped SSH long remote job piped through a reapable foreground ssh.
FIX: setsid nohup, log ON the box. Work survived; output did not.
2. bare LoRA target names q_proj/k_proj/... also match Gemma-4's VISION and AUDIO towers,
whose projections are Gemma4ClippableLinear -- PEFT cannot wrap
them. FIX: the real run's 205 fully-qualified names, sha recorded.
3. missing python3-dev Triton JITs a C shim at FIRST USE and needs Python.h. Torch
imports, model loads, LoRA attaches, THEN step 1 dies with a bare
CalledProcessError -- Triton devnulls the compiler's stdout.
4. naive-loss OOM (32 GiB) the 262,144-wide vocab x 32,768 positions logits tensor in fp32.
⚠ NOT a GX10 limit -- the harness docstring says it OOMs "on a
95.60 GiB card (measured)", i.e. on ana-ml2 too.
5. self-inflicted VRAM leak a stuck orphan held 80 GB while PyTorch reported 0 bytes
allocated; every relaunch was doomed before it started and blamed
the NEW run. ⚠ ASSERT THE GPU IS CLEAR BEFORE EVERY LAUNCH.
### ⚠ THE BACKEND-DELTA TRAP WAS PRESENT AND I FIRST DECLARED IT ABSENT
CLAUDE.md records two frontier-panel conclusions voided by a benchmark and a trainer having
silently different attention backends. I checked for it by asking whether **flash-attn was
installed** — the wrong discriminator entirely — and reported "no delta". The harness sets
**`ATTN_IMPLEMENTATION = "flex_attention"`** explicitly in code, with tuned kernel options; its
own comment records peak 74.3 GiB under flex vs sdpa's 79.71.
⚠ **Read the configured value in the code. Absence of an alternative is not evidence of the
default.** The probe now requests flex_attention and reads `_attn_implementation` back off the
loaded model. **RESOLVED: flex_attention — it compiles and runs on sm_121**, which was a real
open risk (flex compiles through Triton; sm_121 is not in torch's arch list).
### Faithfulness of the probe (what makes the number comparable)
LoRA the real run's 205 fully-qualified modules, sha256 c9ee070c0d67eb08
loss path copied from the harness: trunk hidden states, DROP IGNORE_INDEX before the
head, chunk lm_head+CE at loss_chunk_tokens=1024 under checkpoint, softcap 30.0
loss ratio 28.4% of positions masked to match the real 13,310,930 / 18,600,057 loss/ctx
ratio -- observed 23,351-23,638 loss tokens/step against a 23,449 target
tokens sampled [1000, 250000) to stay below every multimodal special token
(boi 255999, boa 256000, image 258880, audio 258881, eoi/eoa/video 258882-4)
warmup first 2 steps discarded; sm_121 JITs from sm_120 PTX on first use