memory: reframe the GX10 work as what it was — a baseline and a tooling check

Operator clarified the purpose, so the record now leads with it: this was
a baseline for the box and a check that the tooling loads, not a decision
about where run 3c runs. The placement reasoning stays because it is
sound, but it is marked as a byproduct rather than the deliverable.

Two things were actually delivered. The box trains: aarch64 and sm_121
run torch 2.14.0+cu130 with transformers, accelerate, peft, trl, datasets,
safetensors and bitsandbytes, plus the harness's own flex_attention
backend and chunked-loss path, and nothing beyond python3-dev was needed.
And the baseline is 79.35 s/it median across seven timed steps with a
0.19% spread.

Also recorded the port scope without executing it, so nobody re-derives
it: about 2.5 GB of data, a venv rebuild on aarch64, no encode cache worth
moving since the encode runs in 14 seconds, and copy the corpus rather
than mounting NFS on a desk box that will be unattended for hours.
This commit is contained in:
2026-09-02 01:20:26 -07:00
parent dce261335e
commit b039aa19e8
2 changed files with 47 additions and 4 deletions
@@ -313,11 +313,24 @@ direct because ana-ml2's key is not authorised for `infra-ops@pfi-gx10` — and
nothing extra, since the Wi-Fi leg is the bottleneck either way. **Deliberately did not add an
access grant just to save a hop.**
## `[2026-09-02]` RUN-3c THROUGHPUT PROBE — ANSWERED. ~79.3 s/it, and the box is 6x slower than ana-ml2
## `[2026-09-02]` GX10 BASELINE + TOOLING VERIFIED — ~79.35 s/it on the run-3c shape
⚠ **PURPOSE, stated by the operator so it is not misread later: this was a BASELINE for the box
and a check that the tooling loads.** It was NOT a decision about where run 3c runs, and no port
was undertaken. The placement reasoning below is a *byproduct* — real and worth keeping, but the
deliverable was the number and the working stack.
**Deliverable 1 — the tooling loads and runs.** aarch64 + sm_121 runs the full training stack:
torch 2.14.0+cu130, transformers, accelerate, peft, trl, datasets, safetensors, bitsandbytes,
**plus the harness's own `flex_attention` backend and chunked-loss path**. Nothing exotic was
needed beyond `python3-dev`. This box can train.
**Deliverable 2 — the baseline.**
### The number
pfi-gx10 79.30 / 79.37 / 79.33 / 79.45 s/it (0.19% spread across timed steps)
pfi-gx10 79.30 79.37 79.33 79.45 79.38 79.35 79.35 s/it -> MEDIAN 79.35
(0.19% spread across seven timed steps; warmup 85.90 then 79.32, discarded)
-> 604 steps ~= 13.3 h
memory 86 / 121 GB, ~35 GB headroom -- NOT memory-constrained
ana-ml2 10.8-15.8 s/it (run-03c.log, the real run)
@@ -332,7 +345,7 @@ traffic by design). Confirming it needs a bandwidth-bound microbenchmark that ha
**THE GX10 IS A CAPACITY BOX, NOT A THROUGHPUT BOX.** 121 GB holds models nothing else in the
fleet can. It is **not** the box to move a time-sensitive run *to*.
### ✅ RUN 3c BELONGS ON THE GX10 the plan on file was right and infra-ops' reversal was WRONG
### Byproduct: IF run 3c ever moves here, the GX10 is the right home — and infra-ops' reversal was WRONG
⚠⚠ **infra-ops recommended ana-ml2 and the operator overruled it, correctly.** The error is
worth more than the recommendation: **I costed the failure in the units I was measuring.** I had
@@ -411,3 +424,25 @@ open risk (flex compiles through Triton; sm_121 is not in torch's arch list).
tokens sampled [1000, 250000) to stay below every multimodal special token
(boi 255999, boa 256000, image 258880, audio 258881, eoi/eoa/video 258882-4)
warmup first 2 steps discarded; sm_121 JITs from sm_120 PTX on first use
## ⚠ NOT DONE, and deliberately: the run-3c port
Scoped but **not executed** — operator stood it down; the baseline was the goal. Scope, so it
does not have to be re-derived:
harness eitri-smithy 26 MB corpus /mnt/smithy/.../derived 2.3 GB
recipe-r3 485 KB holdout 82 MB
run-03c manifests 70 MB -> ~2.5 GB, ~75 s at 33 MB/s
venv 3.2 GB NOT portable (x86-64) -- rebuild on aarch64
**No encode cache to preserve.** The launch script's "609 MB encode cache / ~4.3 hours"
comment carries its own correction to 145.5 s, and run 3c's log shows **`[encode] done in
14.4s`** on 32 workers. Nothing worth moving.
**Copy the corpus; do NOT mount `/mnt/smithy` over NFS on this box.** A desk machine on Wi-Fi
is the wrong place for a hard NFS dependency under a 13-hour unattended run — the fleet has
already lost a host to a hard mount wedging into unkillable D-state
([[incident_esh_docker_nfs_boot_race]]).
The only untested piece is the harness's own 26 MB of Python; every library it leans on is
verified above.
+9 -1
View File
@@ -110,7 +110,15 @@ no longer deployed sidecars here. See Recent decisions.)
_As of 2026-09-01 — **the GX10 is on the operator's desk, NOT racked. Standing it up as an inference AND training box is the next action.** Nothing is training._
- **▶ NEXT: pfi-gx10 as an inference + training box.** The operator forgot to rack it; it is
- ** pfi-gx10 BASELINED AND OUTFITTED (2026-09-02) — bare metal, training stack verified.**
`uv` + torch 2.14.0+cu130 + transformers/peft/trl/datasets/bitsandbytes all clean on aarch64;
nvcc, NVIDIA Container Toolkit, docker group, NOPASSWD sudo. **Baseline: ~79.35 s/it** on the
run-3c shape (seq 16384, batch 2, LoRA r64, flex_attention, harness chunked loss) -> 604 steps
≈ 13.3 h vs ana-ml2's 2.2-2.7 h. ⚠ **6x slower where compute predicts 2.7x** — likely memory
bandwidth; **capacity box, not throughput box.** Port NOT done (operator stood it down; the
baseline was the goal). Still on Wi-Fi at `10.100.10.226`, not racked.
`persistent-memory.d/2026-09-01-pfi-gx10-onboarding.md`
- **▶ (superseded) pfi-gx10 as an inference + training box.** The operator forgot to rack it; it is
still on his desk on Wi-Fi at `10.100.10.226` (temp DHCP). Already headless and registered.
**The box is BARE — no torch, no nvcc, no CUDA stack.** Purpose is local inference *and*
**run 3c**, whose 49 GB bf16 base fits 121 GB unified with room where ana-ml2 was tight.