memory: snapshot — GX10 outfitted userspace; bare metal ruled; CUDA works on sm_121
Ruling on the architecture question: bare metal, not Proxmox. Proxmox VE has no aarch64 build, and more fundamentally the GB10's GPU sits on an on-package root complex cache-coherent with the CPU over NVLink-C2C, sharing the same LPDDR5X. Passing it to a guest would mean partitioning the unified memory that is the entire reason for the box. The fleet's other GPU hosts are bare metal for the same class of reason. Installed uv and a venv with torch 2.14.0+cu130 plus the full training stack, and every one of transformers, accelerate, peft, trl, datasets, safetensors, huggingface_hub and bitsandbytes imports clean on aarch64. The per-arch unknowns warning did not materialise for any of them. CUDA works: sm_121, 121.6 GiB addressable, about 93 TFLOP/s dense bf16 with tensor cores confirmed engaged by the bf16-to-fp32 ratio. That is A6000-class throughput with two and a half times the memory, so capacity rather than speed is what this box buys. Two warnings worth keeping. sm_121 is not in torch's compiled arch list, so everything runs by PTX JIT from sm_120: first use of every kernel pays a compile, and any library shipping cubins without PTX will fail outright. And an unwarmed benchmark read 27 TFLOP/s because it was timing that JIT, which nearly became a phantom report that tensor cores were broken -- the playbook's section 4 shape exactly, a run that completes and reports plausible numbers and is wrong. The privileged half is blocked: infra-ops has no NOPASSWD sudo on this box, unlike the rest of the fleet, and no credential is vaulted. That gates nvcc, the container toolkit and the docker group, but not the run-3c throughput probe.
This commit is contained in:
@@ -114,3 +114,87 @@ So the throughput probe (ARM64 torch, load base, ten steps, report s/it) can pro
|
||||
desk, today**, without racking first. Racking remains one command
|
||||
(`playbooks/gx10-rack-network.yaml`, VLAN 50, static `10.100.50.60`) and is worth doing for
|
||||
permanence, but it does not gate the decision the probe exists to answer.
|
||||
|
||||
|
||||
## `[2026-09-01 23:2x]` OUTFITTED (userspace) — CUDA WORKS on sm_121, full training stack imports
|
||||
|
||||
### ✅ RULING: BARE METAL, not Proxmox/VM. Settled, do not re-litigate.
|
||||
|
||||
1. **Proxmox VE has no aarch64 build.** It ships x86_64 only; the ARM ports are unofficial
|
||||
community work. Running an unsupported hypervisor under a compute asset is a bad trade.
|
||||
2. **The GPU cannot be cleanly passed through.** GB10 is a superchip: the GPU sits on NVIDIA's
|
||||
own on-package root complex (bridges `22ce`/`22d0`/`22d1`, GPU at `000f:01:00.0`) and is
|
||||
**cache-coherent with the CPU over NVLink-C2C, sharing the same physical LPDDR5X.** Passing
|
||||
it to a guest means partitioning unified memory — which destroys the entire value
|
||||
proposition, since **121.6 GiB GPU-addressable** is the reason this box exists.
|
||||
3. **Vendor's supported path is bare metal** — it already runs NVIDIA's own kernel flavour
|
||||
(`6.17.0-1031-nvidia`).
|
||||
4. **Fleet convention agrees:** ana-ml2 and irv-ml1 are both bare-metal GPU hosts. Hypervisors
|
||||
are separate machines in this fleet, never under the accelerators.
|
||||
5. **Single tenant.** VM isolation buys multi-tenancy nobody needs; the rollback benefit is
|
||||
better served by **containers + `uv` venvs**, which is what the rest of the fleet does.
|
||||
|
||||
### Installed and verified (all userspace, NO sudo needed)
|
||||
|
||||
uv 0.12.9 (aarch64), ~/.local/bin, wired into .bashrc + .profile
|
||||
venv ~/ml/.venv (python 3.12)
|
||||
torch 2.14.0+cu130 cuda runtime 13.0, cudnn 92400, triton 3.8.0
|
||||
stack numpy 2.5.2, transformers 5.16.1, accelerate 1.14.0, peft 0.20.0,
|
||||
trl 1.12.0, datasets 5.0.1, safetensors 0.8.0, hf_hub 1.29.0,
|
||||
bitsandbytes 0.50.2 -- ALL import clean on aarch64
|
||||
|
||||
**The "per-arch unknowns" warning did not materialise for the training stack.** Everything
|
||||
imported first try.
|
||||
|
||||
### ✅ MEASURED GPU COMPUTE — it works, and the memory is the point
|
||||
|
||||
device NVIDIA GB10, capability sm_121, driver 580.173.02
|
||||
GPU memory 121.6 GiB total / 110.8 GiB free <- unified, fully CUDA-addressable
|
||||
bf16 8192^3 ~93 TFLOP/s fp16 ~93 TFLOP/s
|
||||
fp32+tf32 38.2 TFLOP/s fp32 (no tf32) 19.5 TFLOP/s
|
||||
bf16/fp32 4.9x -> tensor cores ARE engaged
|
||||
|
||||
~93 TFLOP/s dense bf16 is roughly **A6000-class throughput with 2.5x the memory**. Capacity,
|
||||
not speed, is what this box buys. A 49 GB bf16 base fits with ~70 GB to spare.
|
||||
|
||||
### ⚠⚠ I NEARLY REPORTED A PHANTOM PROBLEM — NO WARMUP MEANS YOU MEASURE THE JIT
|
||||
|
||||
First benchmark (10 iterations, **no warmup**) read **27.0 TFLOP/s bf16** — slower than a 3090
|
||||
— and I was one step from reporting "tensor cores are not engaging on sm_121." Adding 5 warmup
|
||||
iterations moved it to **93 TFLOP/s. The first measurement was timing the PTX JIT compile.**
|
||||
|
||||
⚠ **This is exactly `docs/pfi/training-throughput-playbook.md` §4: a run that completes, reports
|
||||
plausible numbers, and is wrong.** It would have sent us chasing a cuBLAS fallback that does not
|
||||
exist. **Always warm up before timing anything on this box** — see the JIT note below for why it
|
||||
bites harder here than elsewhere.
|
||||
|
||||
### ⚠ sm_121 IS NOT IN TORCH'S COMPILED ARCH LIST — everything JITs from sm_120 PTX
|
||||
|
||||
torch.cuda.get_arch_list() -> ['sm_80','sm_90','sm_100','sm_110','sm_120']
|
||||
device capability -> sm_121
|
||||
|
||||
There is **no sm_121 cubin in the wheel**; it runs by PTX forward-compat JIT from sm_120.
|
||||
Consequences:
|
||||
|
||||
- **First use of every kernel pays a JIT compile** (cached afterward in `~/.nv/ComputeCache`).
|
||||
This is what corrupted the benchmark above.
|
||||
- ⚠ **Any library shipping cubin-only binaries with no PTX will simply fail here.** That is the
|
||||
real shape of the "per-arch unknown" risk — not source builds, but prebuilt kernels.
|
||||
|
||||
### 🔴 BLOCKED ON SUDO — the privileged half is not done
|
||||
|
||||
⚠ **`infra-ops` does NOT have NOPASSWD sudo on pfi-gx10** (`sudo -n -v` -> "a password is
|
||||
required"), unlike the rest of the fleet — see [[reference_infra_ops_sudo_identity]], which
|
||||
should be corrected: gx10 is an exception. **No gx10 credential is in the vault** (`secret list
|
||||
--prefix pfi-gx10` -> 0 items), and `scripts/elway` prompts via `getpass`, which a
|
||||
non-interactive session cannot answer.
|
||||
|
||||
Still outstanding, all needing root:
|
||||
|
||||
- CUDA toolkit / nvcc (absent -- needed to COMPILE kernels, not to run torch)
|
||||
- NVIDIA Container Toolkit + add infra-ops to the docker group
|
||||
(docker daemon runs; infra-ops gets "permission denied" on the socket)
|
||||
- NOPASSWD sudo for infra-ops, so this box stops being a fleet exception
|
||||
- racking (playbooks/gx10-rack-network.yaml) -- optional, NOT a prerequisite
|
||||
|
||||
**Nothing above blocks the run-3c throughput probe**, which needs only the driver and torch.
|
||||
|
||||
@@ -123,6 +123,18 @@ _As of 2026-09-01 — **the GX10 is on the operator's desk, NOT racked. Standing
|
||||
cable. ⚠ **Racking is NOT a prerequisite** — the Wi-Fi 7 link measures **67 MB/s over SSH**
|
||||
(49 GB base in ~12 min), so the throughput probe can run from the desk today.
|
||||
⚠ `nvidia-smi` FB Memory `N/A` is CORRECT for GB10 unified memory, not a fault.
|
||||
✅ **OUTFITTED 2026-09-01 (userspace)** — **RULING: BARE METAL, not Proxmox** (no aarch64 PVE
|
||||
build; the GPU is on-package and cache-coherent over NVLink-C2C so passthrough would partition
|
||||
the unified memory that is the whole point; fleet GPU hosts are all bare metal).
|
||||
Installed `uv` 0.12.9 + `~/ml/.venv` with torch **2.14.0+cu130** and the full training stack
|
||||
(transformers/accelerate/peft/trl/datasets/bitsandbytes) — **all import clean on aarch64**.
|
||||
**CUDA WORKS: sm_121, 121.6 GiB addressable, ~93 TFLOP/s bf16** (A6000-class speed, 2.5x the
|
||||
memory). ⚠ **sm_121 is NOT in torch's arch list** — everything JITs from sm_120 PTX, so
|
||||
**first use of every kernel pays a JIT compile** and cubin-only libraries will fail.
|
||||
⚠⚠ **ALWAYS WARM UP BEFORE TIMING** — an unwarmed benchmark read 27 TFLOP/s (it was timing the
|
||||
JIT) and nearly became a phantom "tensor cores are broken" report.
|
||||
🔴 **BLOCKED ON SUDO** for nvcc / container toolkit / docker group: **infra-ops has NO NOPASSWD
|
||||
sudo on gx10** (fleet exception) and **no credential is vaulted**.
|
||||
→ `persistent-memory.d/2026-09-01-pfi-gx10-onboarding.md`
|
||||
- **▶ ADA INFERENCE SERVER (R750xa `JPJ1ZP3`) — awaiting an iDRAC inventory.** 2× RTX 6000
|
||||
Ada to be fitted; ComfyUI's new home at NH3. **Order regardless: 8× `M04W6` RDIMM (restores
|
||||
|
||||
Reference in New Issue
Block a user