diff --git a/persistent-memory.d/2026-09-01-pfi-gx10-onboarding.md b/persistent-memory.d/2026-09-01-pfi-gx10-onboarding.md index 8f3572d..aef77cd 100644 --- a/persistent-memory.d/2026-09-01-pfi-gx10-onboarding.md +++ b/persistent-memory.d/2026-09-01-pfi-gx10-onboarding.md @@ -329,10 +329,13 @@ needed beyond `python3-dev`. This box can train. ### The number - pfi-gx10 79.30 79.37 79.33 79.45 79.38 79.35 79.35 s/it -> MEDIAN 79.35 - (0.19% spread across seven timed steps; warmup 85.90 then 79.32, discarded) + pfi-gx10 MEDIAN 79.36 s/it min/max 79.30 / 79.45 (10 timed steps, 0.19% spread; + warmup 85.90 then 79.32, discarded) + peak memory 75.1 / 121.6 GiB -- 46 GiB SPARE, comfortably not memory-bound + attn_resolved: flex_attention (matched to the harness, read off the model) + FINAL, from the probe's own provenance block on the box -> 604 steps ~= 13.3 h - memory 86 / 121 GB, ~35 GB headroom -- NOT memory-constrained + peak 75.1 / 121.6 GiB (PyTorch max_memory_allocated) -- 46 GiB spare ana-ml2 10.8-15.8 s/it (run-03c.log, the real run) -> 604 steps ~= 2.2-2.7 h diff --git a/persistent-memory.md b/persistent-memory.md index 20c7d99..72f561e 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -151,7 +151,7 @@ not arrived._ ## Recent decisions -- `[2026-09-02]` **pfi-gx10 BASELINED: ~79.35 s/it on the run-3c shape, and the training stack works on aarch64/sm_121.** Median across 7 timed steps, 0.19% spread, 86/121 GB. **6× slower than ana-ml2 where compute predicts 2.7×** → likely memory-bandwidth-bound; **capacity box, not throughput box.** Ruled **bare metal, not Proxmox** (no aarch64 PVE; the GPU is on-package and cache-coherent, so passthrough would partition the unified memory that is the whole point). ⚠ `sm_121` is NOT in torch's arch list — everything JITs from sm_120 PTX, so **warm up before timing anything** (an unwarmed bench read 27 TFLOP/s against a true 93). → `persistent-memory.d/2026-09-01-pfi-gx10-onboarding.md` +- `[2026-09-02]` **pfi-gx10 BASELINED: 79.36 s/it median on the run-3c shape, and the training stack works on aarch64/sm_121.** Median across 10 timed steps, 0.19% spread, **peak 75.1 / 121.6 GiB — 46 GiB spare**, `attn_resolved: flex_attention`. **6× slower than ana-ml2 where compute predicts 2.7×** → likely memory-bandwidth-bound; **capacity box, not throughput box.** Ruled **bare metal, not Proxmox** (no aarch64 PVE; the GPU is on-package and cache-coherent, so passthrough would partition the unified memory that is the whole point). ⚠ `sm_121` is NOT in torch's arch list — everything JITs from sm_120 PTX, so **warm up before timing anything** (an unwarmed bench read 27 TFLOP/s against a true 93). → `persistent-memory.d/2026-09-01-pfi-gx10-onboarding.md` - `[2026-09-02]` **I priced a failure in the units I happened to be measuring — operator overruled me, correctly.** Recommended run 3c to ana-ml2 by costing a breaker trip as "≤50 steps ≈ 11 min of recompute". It is a **40-minute drive each way** with **13 Anaheim hosts dark, three of them SureFire CLIENT machines**. `save_steps` caps the recompute, never the outage. ⚠ **General form: a metric in hand will volunteer itself as the unit of risk.** → `persistent-memory.d/2026-09-01-pfi-gx10-onboarding.md` - `[2026-09-02]` **althing 3.2.0→3.2.4 deployed, and ALTHING DEPLOY IS FOUR SURFACES not three.** The fourth (plugin) had no runbook step and was frozen at Aug 28 — **missing the SessionStart/SessionEnd hooks and `pane-route.sh` entirely**, so "CC seats re-declare automatically" was never true here. Now one command (`scripts/deploy-althing.sh`). ⚠ `uv tool install .` **without `--force` is a silent no-op**. ⚠ **A missing deploy surface presents as "the migration needs manual work", not as an error.** → `persistent-memory.d/2026-09-01-althing-320-deploy.md` - `[2026-09-01]` **irv-ml1 GPU resident map, and `dots-tts` holds 14,430 MiB against a ~6 GB baseline** — tts-dev's prompt-feature cache, capped at 32 entries after two incidents; the cap still permits a long way of growth. 3090 at 76% behind a warn-only watchdog. ⚠ Restates the GPU-ordering foot-gun: `device_ids: ["1"]` is the A6000 in a container, but a bare native `CUDA_VISIBLE_DEVICES=1` gets the **3090**. → `persistent-memory.d/2026-09-01-irv-ml1-gpu-residents.md`