memory: run-3c probe answered — 79.3 s/it, and it reverses the plan on file

The GX10 runs run 3c at about 79.3 s/it with 35 GB of headroom, which puts
604 steps at 13.3 hours against ana-ml2's 2.2 to 2.7. Six times slower
where raw compute predicts 2.7, which points at memory bandwidth rather
than FLOPs -- recorded as a hypothesis, since confirming it needs a
bandwidth-bound microbenchmark nobody has run.

That reverses the standing plan. Moving run 3c here was framed as the
power answer, but the run did not die because ana-ml2 is unreliable. It
died because save_steps was 100 and the breaker tripped at step 80, so no
checkpoint existed. save_steps is now 50, which caps a power event at
about eleven minutes. Trading 2.5 hours for 13.3 buys insurance against a
risk already engineered out.

Also records the five launch failures and their causes, and the one that
matters most: the attention-backend trap was present and I first declared
it absent. I checked whether flash-attn was installed, which is the wrong
discriminator; the harness sets flex_attention explicitly in code. Absence
of an alternative is not evidence of the default. The probe now reads the
resolved backend back off the loaded model, and flex_attention does
compile and run on sm_121.

persistent-memory.d/2026-09-01-pfi-gx10-onboarding.md

@@ -312,3 +312,71 @@ padding on the real run is 0.0% so the corpus does not change per-step cost mate
 direct because ana-ml2's key is not authorised for `infra-ops@pfi-gx10` — and relaying costs
 nothing extra, since the Wi-Fi leg is the bottleneck either way. **Deliberately did not add an
 access grant just to save a hop.**
+
+## `[2026-09-02]` RUN-3c THROUGHPUT PROBE — ANSWERED. ~79.3 s/it, and the box is 6x slower than ana-ml2
+
+### The number
+
+    pfi-gx10    79.30 / 79.37 / 79.33 / 79.45 s/it    (0.19% spread across timed steps)
+                -> 604 steps ~= 13.3 h
+                memory 86 / 121 GB, ~35 GB headroom -- NOT memory-constrained
+    ana-ml2     10.8-15.8 s/it (run-03c.log, the real run)
+                -> 604 steps ~= 2.2-2.7 h
+
+⚠ **6x slower where raw compute predicts 2.7x** (93 TFLOP/s bf16 measured here vs an RTX PRO
+6000 Blackwell's ~250). **Hypothesis, not a measurement:** the limiter is **memory bandwidth**
+— GB10's unified LPDDR5X against dedicated graphics memory, on a workload built to be
+bandwidth-hungry (seq 16384 with gradient checkpointing, which trades compute for memory
+traffic by design). Confirming it needs a bandwidth-bound microbenchmark that has not been run.
+
+⚠ **THE GX10 IS A CAPACITY BOX, NOT A THROUGHPUT BOX.** 121 GB holds models nothing else in the
+fleet can. It is **not** the box to move a time-sensitive run *to*.
+
+### ⚠ THE RECOMMENDATION REVERSES THE PLAN ON FILE — run 3c belongs on ana-ml2
+
+The standing plan was "move run 3c to the GX10; it is the power answer rather than a power
+triage." **Measurement kills that.** Run 3c did not die because ana-ml2 is unreliable — it died
+because `save_steps` was **100** and the 2026-08-26 breaker trip landed at **step 80**, so no
+checkpoint existed. **`save_steps` is now 50.** A power event costs **≤50 steps ≈ 11 min** and
+resumes. Trading 2.5 h for 13.3 h buys insurance against a risk already engineered out.
+
+### Probe methodology — five launches, five distinct causes, all recorded
+
+    1. reaped SSH             long remote job piped through a reapable foreground ssh.
+                              FIX: setsid nohup, log ON the box. Work survived; output did not.
+    2. bare LoRA target names q_proj/k_proj/... also match Gemma-4's VISION and AUDIO towers,
+                              whose projections are Gemma4ClippableLinear -- PEFT cannot wrap
+                              them. FIX: the real run's 205 fully-qualified names, sha recorded.
+    3. missing python3-dev    Triton JITs a C shim at FIRST USE and needs Python.h. Torch
+                              imports, model loads, LoRA attaches, THEN step 1 dies with a bare
+                              CalledProcessError -- Triton devnulls the compiler's stdout.
+    4. naive-loss OOM (32 GiB) the 262,144-wide vocab x 32,768 positions logits tensor in fp32.
+                              ⚠ NOT a GX10 limit -- the harness docstring says it OOMs "on a
+                              95.60 GiB card (measured)", i.e. on ana-ml2 too.
+    5. self-inflicted VRAM leak a stuck orphan held 80 GB while PyTorch reported 0 bytes
+                              allocated; every relaunch was doomed before it started and blamed
+                              the NEW run. ⚠ ASSERT THE GPU IS CLEAR BEFORE EVERY LAUNCH.
+
+### ⚠ THE BACKEND-DELTA TRAP WAS PRESENT AND I FIRST DECLARED IT ABSENT
+
+CLAUDE.md records two frontier-panel conclusions voided by a benchmark and a trainer having
+silently different attention backends. I checked for it by asking whether **flash-attn was
+installed** — the wrong discriminator entirely — and reported "no delta". The harness sets
+**`ATTN_IMPLEMENTATION = "flex_attention"`** explicitly in code, with tuned kernel options; its
+own comment records peak 74.3 GiB under flex vs sdpa's 79.71.
+
+⚠ **Read the configured value in the code. Absence of an alternative is not evidence of the
+default.** The probe now requests flex_attention and reads `_attn_implementation` back off the
+loaded model. **RESOLVED: flex_attention — it compiles and runs on sm_121**, which was a real
+open risk (flex compiles through Triton; sm_121 is not in torch's arch list).
+
+### Faithfulness of the probe (what makes the number comparable)
+
+    LoRA         the real run's 205 fully-qualified modules, sha256 c9ee070c0d67eb08
+    loss path    copied from the harness: trunk hidden states, DROP IGNORE_INDEX before the
+                 head, chunk lm_head+CE at loss_chunk_tokens=1024 under checkpoint, softcap 30.0
+    loss ratio   28.4% of positions masked to match the real 13,310,930 / 18,600,057 loss/ctx
+                 ratio -- observed 23,351-23,638 loss tokens/step against a 23,449 target
+    tokens       sampled [1000, 250000) to stay below every multimodal special token
+                 (boi 255999, boa 256000, image 258880, audio 258881, eoi/eoa/video 258882-4)
+    warmup       first 2 steps discarded; sm_121 JITs from sm_120 PTX on first use