diff --git a/persistent-memory.d/2026-09-01-pfi-gx10-onboarding.md b/persistent-memory.d/2026-09-01-pfi-gx10-onboarding.md index d7b0555..189ace1 100644 --- a/persistent-memory.d/2026-09-01-pfi-gx10-onboarding.md +++ b/persistent-memory.d/2026-09-01-pfi-gx10-onboarding.md @@ -114,3 +114,87 @@ So the throughput probe (ARM64 torch, load base, ten steps, report s/it) can pro desk, today**, without racking first. Racking remains one command (`playbooks/gx10-rack-network.yaml`, VLAN 50, static `10.100.50.60`) and is worth doing for permanence, but it does not gate the decision the probe exists to answer. + + +## `[2026-09-01 23:2x]` OUTFITTED (userspace) — CUDA WORKS on sm_121, full training stack imports + +### ✅ RULING: BARE METAL, not Proxmox/VM. Settled, do not re-litigate. + +1. **Proxmox VE has no aarch64 build.** It ships x86_64 only; the ARM ports are unofficial + community work. Running an unsupported hypervisor under a compute asset is a bad trade. +2. **The GPU cannot be cleanly passed through.** GB10 is a superchip: the GPU sits on NVIDIA's + own on-package root complex (bridges `22ce`/`22d0`/`22d1`, GPU at `000f:01:00.0`) and is + **cache-coherent with the CPU over NVLink-C2C, sharing the same physical LPDDR5X.** Passing + it to a guest means partitioning unified memory — which destroys the entire value + proposition, since **121.6 GiB GPU-addressable** is the reason this box exists. +3. **Vendor's supported path is bare metal** — it already runs NVIDIA's own kernel flavour + (`6.17.0-1031-nvidia`). +4. **Fleet convention agrees:** ana-ml2 and irv-ml1 are both bare-metal GPU hosts. Hypervisors + are separate machines in this fleet, never under the accelerators. +5. **Single tenant.** VM isolation buys multi-tenancy nobody needs; the rollback benefit is + better served by **containers + `uv` venvs**, which is what the rest of the fleet does. + +### Installed and verified (all userspace, NO sudo needed) + + uv 0.12.9 (aarch64), ~/.local/bin, wired into .bashrc + .profile + venv ~/ml/.venv (python 3.12) + torch 2.14.0+cu130 cuda runtime 13.0, cudnn 92400, triton 3.8.0 + stack numpy 2.5.2, transformers 5.16.1, accelerate 1.14.0, peft 0.20.0, + trl 1.12.0, datasets 5.0.1, safetensors 0.8.0, hf_hub 1.29.0, + bitsandbytes 0.50.2 -- ALL import clean on aarch64 + +**The "per-arch unknowns" warning did not materialise for the training stack.** Everything +imported first try. + +### ✅ MEASURED GPU COMPUTE — it works, and the memory is the point + + device NVIDIA GB10, capability sm_121, driver 580.173.02 + GPU memory 121.6 GiB total / 110.8 GiB free <- unified, fully CUDA-addressable + bf16 8192^3 ~93 TFLOP/s fp16 ~93 TFLOP/s + fp32+tf32 38.2 TFLOP/s fp32 (no tf32) 19.5 TFLOP/s + bf16/fp32 4.9x -> tensor cores ARE engaged + +~93 TFLOP/s dense bf16 is roughly **A6000-class throughput with 2.5x the memory**. Capacity, +not speed, is what this box buys. A 49 GB bf16 base fits with ~70 GB to spare. + +### ⚠⚠ I NEARLY REPORTED A PHANTOM PROBLEM — NO WARMUP MEANS YOU MEASURE THE JIT + +First benchmark (10 iterations, **no warmup**) read **27.0 TFLOP/s bf16** — slower than a 3090 +— and I was one step from reporting "tensor cores are not engaging on sm_121." Adding 5 warmup +iterations moved it to **93 TFLOP/s. The first measurement was timing the PTX JIT compile.** + +⚠ **This is exactly `docs/pfi/training-throughput-playbook.md` §4: a run that completes, reports +plausible numbers, and is wrong.** It would have sent us chasing a cuBLAS fallback that does not +exist. **Always warm up before timing anything on this box** — see the JIT note below for why it +bites harder here than elsewhere. + +### ⚠ sm_121 IS NOT IN TORCH'S COMPILED ARCH LIST — everything JITs from sm_120 PTX + + torch.cuda.get_arch_list() -> ['sm_80','sm_90','sm_100','sm_110','sm_120'] + device capability -> sm_121 + +There is **no sm_121 cubin in the wheel**; it runs by PTX forward-compat JIT from sm_120. +Consequences: + +- **First use of every kernel pays a JIT compile** (cached afterward in `~/.nv/ComputeCache`). + This is what corrupted the benchmark above. +- ⚠ **Any library shipping cubin-only binaries with no PTX will simply fail here.** That is the + real shape of the "per-arch unknown" risk — not source builds, but prebuilt kernels. + +### 🔴 BLOCKED ON SUDO — the privileged half is not done + +⚠ **`infra-ops` does NOT have NOPASSWD sudo on pfi-gx10** (`sudo -n -v` -> "a password is +required"), unlike the rest of the fleet — see [[reference_infra_ops_sudo_identity]], which +should be corrected: gx10 is an exception. **No gx10 credential is in the vault** (`secret list +--prefix pfi-gx10` -> 0 items), and `scripts/elway` prompts via `getpass`, which a +non-interactive session cannot answer. + +Still outstanding, all needing root: + + - CUDA toolkit / nvcc (absent -- needed to COMPILE kernels, not to run torch) + - NVIDIA Container Toolkit + add infra-ops to the docker group + (docker daemon runs; infra-ops gets "permission denied" on the socket) + - NOPASSWD sudo for infra-ops, so this box stops being a fleet exception + - racking (playbooks/gx10-rack-network.yaml) -- optional, NOT a prerequisite + +**Nothing above blocks the run-3c throughput probe**, which needs only the driver and torch. diff --git a/persistent-memory.md b/persistent-memory.md index fe26081..a10f158 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -123,6 +123,18 @@ _As of 2026-09-01 — **the GX10 is on the operator's desk, NOT racked. Standing cable. ⚠ **Racking is NOT a prerequisite** — the Wi-Fi 7 link measures **67 MB/s over SSH** (49 GB base in ~12 min), so the throughput probe can run from the desk today. ⚠ `nvidia-smi` FB Memory `N/A` is CORRECT for GB10 unified memory, not a fault. + ✅ **OUTFITTED 2026-09-01 (userspace)** — **RULING: BARE METAL, not Proxmox** (no aarch64 PVE + build; the GPU is on-package and cache-coherent over NVLink-C2C so passthrough would partition + the unified memory that is the whole point; fleet GPU hosts are all bare metal). + Installed `uv` 0.12.9 + `~/ml/.venv` with torch **2.14.0+cu130** and the full training stack + (transformers/accelerate/peft/trl/datasets/bitsandbytes) — **all import clean on aarch64**. + **CUDA WORKS: sm_121, 121.6 GiB addressable, ~93 TFLOP/s bf16** (A6000-class speed, 2.5x the + memory). ⚠ **sm_121 is NOT in torch's arch list** — everything JITs from sm_120 PTX, so + **first use of every kernel pays a JIT compile** and cubin-only libraries will fail. + ⚠⚠ **ALWAYS WARM UP BEFORE TIMING** — an unwarmed benchmark read 27 TFLOP/s (it was timing the + JIT) and nearly became a phantom "tensor cores are broken" report. + 🔴 **BLOCKED ON SUDO** for nvcc / container toolkit / docker group: **infra-ops has NO NOPASSWD + sudo on gx10** (fleet exception) and **no credential is vaulted**. → `persistent-memory.d/2026-09-01-pfi-gx10-onboarding.md` - **▶ ADA INFERENCE SERVER (R750xa `JPJ1ZP3`) — awaiting an iDRAC inventory.** 2× RTX 6000 Ada to be fitted; ComfyUI's new home at NH3. **Order regardless: 8× `M04W6` RDIMM (restores