memory: reframe the GX10 work as what it was — a baseline and a tooling check
Operator clarified the purpose, so the record now leads with it: this was a baseline for the box and a check that the tooling loads, not a decision about where run 3c runs. The placement reasoning stays because it is sound, but it is marked as a byproduct rather than the deliverable. Two things were actually delivered. The box trains: aarch64 and sm_121 run torch 2.14.0+cu130 with transformers, accelerate, peft, trl, datasets, safetensors and bitsandbytes, plus the harness's own flex_attention backend and chunked-loss path, and nothing beyond python3-dev was needed. And the baseline is 79.35 s/it median across seven timed steps with a 0.19% spread. Also recorded the port scope without executing it, so nobody re-derives it: about 2.5 GB of data, a venv rebuild on aarch64, no encode cache worth moving since the encode runs in 14 seconds, and copy the corpus rather than mounting NFS on a desk box that will be unattended for hours.
This commit is contained in:
@@ -313,11 +313,24 @@ direct because ana-ml2's key is not authorised for `infra-ops@pfi-gx10` — and
|
||||
nothing extra, since the Wi-Fi leg is the bottleneck either way. **Deliberately did not add an
|
||||
access grant just to save a hop.**
|
||||
|
||||
## `[2026-09-02]` RUN-3c THROUGHPUT PROBE — ANSWERED. ~79.3 s/it, and the box is 6x slower than ana-ml2
|
||||
## `[2026-09-02]` GX10 BASELINE + TOOLING VERIFIED — ~79.35 s/it on the run-3c shape
|
||||
|
||||
⚠ **PURPOSE, stated by the operator so it is not misread later: this was a BASELINE for the box
|
||||
and a check that the tooling loads.** It was NOT a decision about where run 3c runs, and no port
|
||||
was undertaken. The placement reasoning below is a *byproduct* — real and worth keeping, but the
|
||||
deliverable was the number and the working stack.
|
||||
|
||||
**Deliverable 1 — the tooling loads and runs.** aarch64 + sm_121 runs the full training stack:
|
||||
torch 2.14.0+cu130, transformers, accelerate, peft, trl, datasets, safetensors, bitsandbytes,
|
||||
**plus the harness's own `flex_attention` backend and chunked-loss path**. Nothing exotic was
|
||||
needed beyond `python3-dev`. This box can train.
|
||||
|
||||
**Deliverable 2 — the baseline.**
|
||||
|
||||
### The number
|
||||
|
||||
pfi-gx10 79.30 / 79.37 / 79.33 / 79.45 s/it (0.19% spread across timed steps)
|
||||
pfi-gx10 79.30 79.37 79.33 79.45 79.38 79.35 79.35 s/it -> MEDIAN 79.35
|
||||
(0.19% spread across seven timed steps; warmup 85.90 then 79.32, discarded)
|
||||
-> 604 steps ~= 13.3 h
|
||||
memory 86 / 121 GB, ~35 GB headroom -- NOT memory-constrained
|
||||
ana-ml2 10.8-15.8 s/it (run-03c.log, the real run)
|
||||
@@ -332,7 +345,7 @@ traffic by design). Confirming it needs a bandwidth-bound microbenchmark that ha
|
||||
⚠ **THE GX10 IS A CAPACITY BOX, NOT A THROUGHPUT BOX.** 121 GB holds models nothing else in the
|
||||
fleet can. It is **not** the box to move a time-sensitive run *to*.
|
||||
|
||||
### ✅ RUN 3c BELONGS ON THE GX10 — the plan on file was right and infra-ops' reversal was WRONG
|
||||
### Byproduct: IF run 3c ever moves here, the GX10 is the right home — and infra-ops' reversal was WRONG
|
||||
|
||||
⚠⚠ **infra-ops recommended ana-ml2 and the operator overruled it, correctly.** The error is
|
||||
worth more than the recommendation: **I costed the failure in the units I was measuring.** I had
|
||||
@@ -411,3 +424,25 @@ open risk (flex compiles through Triton; sm_121 is not in torch's arch list).
|
||||
tokens sampled [1000, 250000) to stay below every multimodal special token
|
||||
(boi 255999, boa 256000, image 258880, audio 258881, eoi/eoa/video 258882-4)
|
||||
warmup first 2 steps discarded; sm_121 JITs from sm_120 PTX on first use
|
||||
|
||||
## ⚠ NOT DONE, and deliberately: the run-3c port
|
||||
|
||||
Scoped but **not executed** — operator stood it down; the baseline was the goal. Scope, so it
|
||||
does not have to be re-derived:
|
||||
|
||||
harness eitri-smithy 26 MB corpus /mnt/smithy/.../derived 2.3 GB
|
||||
recipe-r3 485 KB holdout 82 MB
|
||||
run-03c manifests 70 MB -> ~2.5 GB, ~75 s at 33 MB/s
|
||||
venv 3.2 GB NOT portable (x86-64) -- rebuild on aarch64
|
||||
|
||||
⚠ **No encode cache to preserve.** The launch script's "609 MB encode cache / ~4.3 hours"
|
||||
comment carries its own correction to 145.5 s, and run 3c's log shows **`[encode] done in
|
||||
14.4s`** on 32 workers. Nothing worth moving.
|
||||
|
||||
⚠ **Copy the corpus; do NOT mount `/mnt/smithy` over NFS on this box.** A desk machine on Wi-Fi
|
||||
is the wrong place for a hard NFS dependency under a 13-hour unattended run — the fleet has
|
||||
already lost a host to a hard mount wedging into unkillable D-state
|
||||
([[incident_esh_docker_nfs_boot_race]]).
|
||||
|
||||
The only untested piece is the harness's own 26 MB of Python; every library it leans on is
|
||||
verified above.
|
||||
|
||||
@@ -110,7 +110,15 @@ no longer deployed sidecars here. See Recent decisions.)
|
||||
|
||||
_As of 2026-09-01 — **the GX10 is on the operator's desk, NOT racked. Standing it up as an inference AND training box is the next action.** Nothing is training._
|
||||
|
||||
- **▶ NEXT: pfi-gx10 as an inference + training box.** The operator forgot to rack it; it is
|
||||
- **✅ pfi-gx10 BASELINED AND OUTFITTED (2026-09-02) — bare metal, training stack verified.**
|
||||
`uv` + torch 2.14.0+cu130 + transformers/peft/trl/datasets/bitsandbytes all clean on aarch64;
|
||||
nvcc, NVIDIA Container Toolkit, docker group, NOPASSWD sudo. **Baseline: ~79.35 s/it** on the
|
||||
run-3c shape (seq 16384, batch 2, LoRA r64, flex_attention, harness chunked loss) -> 604 steps
|
||||
≈ 13.3 h vs ana-ml2's 2.2-2.7 h. ⚠ **6x slower where compute predicts 2.7x** — likely memory
|
||||
bandwidth; **capacity box, not throughput box.** Port NOT done (operator stood it down; the
|
||||
baseline was the goal). Still on Wi-Fi at `10.100.10.226`, not racked.
|
||||
→ `persistent-memory.d/2026-09-01-pfi-gx10-onboarding.md`
|
||||
- **▶ (superseded) pfi-gx10 as an inference + training box.** The operator forgot to rack it; it is
|
||||
still on his desk on Wi-Fi at `10.100.10.226` (temp DHCP). Already headless and registered.
|
||||
**The box is BARE — no torch, no nvcc, no CUDA stack.** Purpose is local inference *and*
|
||||
**run 3c**, whose 49 GB bf16 base fits 121 GB unified with room where ana-ml2 was tight.
|
||||
|
||||
Reference in New Issue
Block a user