diff --git a/persistent-memory.d/2026-09-01-pfi-gx10-onboarding.md b/persistent-memory.d/2026-09-01-pfi-gx10-onboarding.md index 85b6b9d..8f3572d 100644 --- a/persistent-memory.d/2026-09-01-pfi-gx10-onboarding.md +++ b/persistent-memory.d/2026-09-01-pfi-gx10-onboarding.md @@ -313,11 +313,24 @@ direct because ana-ml2's key is not authorised for `infra-ops@pfi-gx10` — and nothing extra, since the Wi-Fi leg is the bottleneck either way. **Deliberately did not add an access grant just to save a hop.** -## `[2026-09-02]` RUN-3c THROUGHPUT PROBE — ANSWERED. ~79.3 s/it, and the box is 6x slower than ana-ml2 +## `[2026-09-02]` GX10 BASELINE + TOOLING VERIFIED — ~79.35 s/it on the run-3c shape + +⚠ **PURPOSE, stated by the operator so it is not misread later: this was a BASELINE for the box +and a check that the tooling loads.** It was NOT a decision about where run 3c runs, and no port +was undertaken. The placement reasoning below is a *byproduct* — real and worth keeping, but the +deliverable was the number and the working stack. + +**Deliverable 1 — the tooling loads and runs.** aarch64 + sm_121 runs the full training stack: +torch 2.14.0+cu130, transformers, accelerate, peft, trl, datasets, safetensors, bitsandbytes, +**plus the harness's own `flex_attention` backend and chunked-loss path**. Nothing exotic was +needed beyond `python3-dev`. This box can train. + +**Deliverable 2 — the baseline.** ### The number - pfi-gx10 79.30 / 79.37 / 79.33 / 79.45 s/it (0.19% spread across timed steps) + pfi-gx10 79.30 79.37 79.33 79.45 79.38 79.35 79.35 s/it -> MEDIAN 79.35 + (0.19% spread across seven timed steps; warmup 85.90 then 79.32, discarded) -> 604 steps ~= 13.3 h memory 86 / 121 GB, ~35 GB headroom -- NOT memory-constrained ana-ml2 10.8-15.8 s/it (run-03c.log, the real run) @@ -332,7 +345,7 @@ traffic by design). Confirming it needs a bandwidth-bound microbenchmark that ha ⚠ **THE GX10 IS A CAPACITY BOX, NOT A THROUGHPUT BOX.** 121 GB holds models nothing else in the fleet can. It is **not** the box to move a time-sensitive run *to*. -### ✅ RUN 3c BELONGS ON THE GX10 — the plan on file was right and infra-ops' reversal was WRONG +### Byproduct: IF run 3c ever moves here, the GX10 is the right home — and infra-ops' reversal was WRONG ⚠⚠ **infra-ops recommended ana-ml2 and the operator overruled it, correctly.** The error is worth more than the recommendation: **I costed the failure in the units I was measuring.** I had @@ -411,3 +424,25 @@ open risk (flex compiles through Triton; sm_121 is not in torch's arch list). tokens sampled [1000, 250000) to stay below every multimodal special token (boi 255999, boa 256000, image 258880, audio 258881, eoi/eoa/video 258882-4) warmup first 2 steps discarded; sm_121 JITs from sm_120 PTX on first use + +## ⚠ NOT DONE, and deliberately: the run-3c port + +Scoped but **not executed** — operator stood it down; the baseline was the goal. Scope, so it +does not have to be re-derived: + + harness eitri-smithy 26 MB corpus /mnt/smithy/.../derived 2.3 GB + recipe-r3 485 KB holdout 82 MB + run-03c manifests 70 MB -> ~2.5 GB, ~75 s at 33 MB/s + venv 3.2 GB NOT portable (x86-64) -- rebuild on aarch64 + +⚠ **No encode cache to preserve.** The launch script's "609 MB encode cache / ~4.3 hours" +comment carries its own correction to 145.5 s, and run 3c's log shows **`[encode] done in +14.4s`** on 32 workers. Nothing worth moving. + +⚠ **Copy the corpus; do NOT mount `/mnt/smithy` over NFS on this box.** A desk machine on Wi-Fi +is the wrong place for a hard NFS dependency under a 13-hour unattended run — the fleet has +already lost a host to a hard mount wedging into unkillable D-state +([[incident_esh_docker_nfs_boot_race]]). + +The only untested piece is the harness's own 26 MB of Python; every library it leans on is +verified above. diff --git a/persistent-memory.md b/persistent-memory.md index 0648c29..b9983a0 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -110,7 +110,15 @@ no longer deployed sidecars here. See Recent decisions.) _As of 2026-09-01 — **the GX10 is on the operator's desk, NOT racked. Standing it up as an inference AND training box is the next action.** Nothing is training._ -- **▶ NEXT: pfi-gx10 as an inference + training box.** The operator forgot to rack it; it is +- **✅ pfi-gx10 BASELINED AND OUTFITTED (2026-09-02) — bare metal, training stack verified.** + `uv` + torch 2.14.0+cu130 + transformers/peft/trl/datasets/bitsandbytes all clean on aarch64; + nvcc, NVIDIA Container Toolkit, docker group, NOPASSWD sudo. **Baseline: ~79.35 s/it** on the + run-3c shape (seq 16384, batch 2, LoRA r64, flex_attention, harness chunked loss) -> 604 steps + ≈ 13.3 h vs ana-ml2's 2.2-2.7 h. ⚠ **6x slower where compute predicts 2.7x** — likely memory + bandwidth; **capacity box, not throughput box.** Port NOT done (operator stood it down; the + baseline was the goal). Still on Wi-Fi at `10.100.10.226`, not racked. + → `persistent-memory.d/2026-09-01-pfi-gx10-onboarding.md` +- **▶ (superseded) pfi-gx10 as an inference + training box.** The operator forgot to rack it; it is still on his desk on Wi-Fi at `10.100.10.226` (temp DHCP). Already headless and registered. **The box is BARE — no torch, no nvcc, no CUDA stack.** Purpose is local inference *and* **run 3c**, whose 49 GB bf16 base fits 121 GB unified with room where ana-ml2 was tight.