diff --git a/persistent-memory.d/2026-09-01-pfi-gx10-onboarding.md b/persistent-memory.d/2026-09-01-pfi-gx10-onboarding.md index de98742..8cbf13c 100644 --- a/persistent-memory.d/2026-09-01-pfi-gx10-onboarding.md +++ b/persistent-memory.d/2026-09-01-pfi-gx10-onboarding.md @@ -253,3 +253,62 @@ Still outstanding, all needing root: - racking (playbooks/gx10-rack-network.yaml) -- optional, NOT a prerequisite **Nothing above blocks the run-3c throughput probe**, which needs only the driver and torch. + +## `[2026-09-01 23:5x]` Run-3c throughput probe — IN FLIGHT, and the ana-ml2 baseline is now on record + +### The baseline nobody had written down: ana-ml2's real s/it for run 3c + +Extracted from `/tank/erp-tune/run-03c.log` (the actual run, before the breaker killed it): + + step 10 10.81 s/it step 20 12.88 s/it step 24 15.76 s/it + => 604 steps on ana-ml2 ~= 2.2-2.7 h + +**This is the number the GX10 has to be compared against**, and it was sitting in a log +nobody had reduced. Anything under ~45 s/it on the GX10 makes 604 steps an overnight run. + +### Exact run-3c geometry (from run-03c.json + a real adapter_config.json) + + base /tank/aimodels/gemma4-26b-a4b-it-bf16 49 GB + gemma4, 30 layers, hidden 2816, 16 heads / 8 KV, MoE (A4B), + vocab 262144, sliding_window 1024, multimodal (vision+audio configs) + LoRA r 64, alpha 128, dropout 0.0, bias none, 205 target modules + (q/k/v/o + gate/up/down across all 30 language-model layers) + shape max_seq_len 16384, per_device_batch 2, grad_accum 8 (effective 16) + other gradient_checkpointing true, bf16, cosine, warmup 0.1, wd 0.01 + length 604 optimizer steps, 1 epoch, save_steps 50 + +### ⚠ THE BACKEND-DELTA TRAP — CHECKED, and it is NOT present + +CLAUDE.md warns that two frontier-panel conclusions were voided because a benchmark and a +trainer had silently different attention backends. Enumerated both sides **before** running: + + ana-ml2 flash_attn NOT installed in /tank/erp-tune/venv -> falls back to sdpa + pfi-gx10 flash_attn not available for aarch64/sm_121 -> falls back to sdpa + +Same backend both sides. ⚠ **Library versions DO differ and it is recorded rather than +assumed harmless:** + + ana-ml2 torch 2.13.0 transformers 5.15.1 peft 0.20.0 accelerate 1.14.0 + pfi-gx10 torch 2.14.0+cu130 transformers 5.16.1 peft 0.20.0 accelerate 1.14.0 + +The probe reads the **resolved** `_attn_implementation` back off the loaded model rather than +trusting the request, so the comparison is checkable rather than assumed. + +### Probe design (§4-disciplined) + +`~/ml/probe_3c.py` on the GX10. Real geometry, synthetic batches — it measures step cost, and +padding on the real run is 0.0% so the corpus does not change per-step cost materially. + +- **12 steps, first 2 discarded as warmup.** Non-negotiable here: an unwarmed benchmark on + this box already read 27 TFLOP/s when the true figure was 93, because it was timing the PTX + JIT. sm_121 is not in torch's arch list. +- Reports the **step-time distribution** (median, min/max, every step), not a mean beside a + settings dump. +- Emits **every provenance key always, null when unused** — present-and-null, not absent. + +### Transfer + +49 GB relayed **ana-ml2 -> nh3-dev -> pfi-gx10**, ~33 MB/s, ~25 min. Relayed rather than +direct because ana-ml2's key is not authorised for `infra-ops@pfi-gx10` — and relaying costs +nothing extra, since the Wi-Fi leg is the bottleneck either way. **Deliberately did not add an +access grant just to save a hop.**