The probe finished. Median is 79.36 s/it across ten timed steps with a
min/max of 79.30 to 79.45, and peak memory is 75.1 of 121.6 GiB by
PyTorch's own max_memory_allocated -- 46 GiB spare rather than the 35 I
estimated from a live free reading, which was counting fragmentation and
the resident model rather than the allocation high-water mark.
Resolved attention backend recorded as flex_attention, read off the loaded
model rather than trusted from the request, which is the check I got wrong
the first time.
Operator clarified the purpose, so the record now leads with it: this was
a baseline for the box and a check that the tooling loads, not a decision
about where run 3c runs. The placement reasoning stays because it is
sound, but it is marked as a byproduct rather than the deliverable.
Two things were actually delivered. The box trains: aarch64 and sm_121
run torch 2.14.0+cu130 with transformers, accelerate, peft, trl, datasets,
safetensors and bitsandbytes, plus the harness's own flex_attention
backend and chunked-loss path, and nothing beyond python3-dev was needed.
And the baseline is 79.35 s/it median across seven timed steps with a
0.19% spread.
Also recorded the port scope without executing it, so nobody re-derives
it: about 2.5 GB of data, a venv rebuild on aarch64, no encode cache worth
moving since the encode runs in 14 seconds, and copy the corpus rather
than mounting NFS on a desk box that will be unattended for hours.
The operator overruled my ana-ml2 recommendation and was right. I had step
times in hand, so I priced a breaker trip as eleven minutes of lost
training. That is the recompute cost and it is the cheapest component of
the loss.
A breaker trip at Anaheim is a forty-minute drive each way on the
operator's time, whenever he happens to notice, with thirteen hosts dark
until he arrives -- including the hypervisor most of them run on, the
fleet's primary backup server, and three SureFire client machines that are
a customer's production hosts under a hosting agreement.
So save_steps 100 to 50 caps the recompute, not the outage, and it was
never the mitigation I claimed. Thirteen hours unattended on a desk in NH3
beats two and a half hours that can put a client's hosts dark, especially
when nothing is waiting on this run.
Recorded the general form at length because it is the transferable part:
when recommending between options, check whether you priced the failure
mode in whatever units you happened to be measuring. A metric in hand will
volunteer itself as the unit of risk.
Also measured and dismissed the obvious third option: the RTX PRO 6000s
have a 250 W floor against a 300 W default, so capping both saves 100 W on
a box drawing about a kilowatt. Not enough to matter, and it costs
throughput to buy.
The GX10 runs run 3c at about 79.3 s/it with 35 GB of headroom, which puts
604 steps at 13.3 hours against ana-ml2's 2.2 to 2.7. Six times slower
where raw compute predicts 2.7, which points at memory bandwidth rather
than FLOPs -- recorded as a hypothesis, since confirming it needs a
bandwidth-bound microbenchmark nobody has run.
That reverses the standing plan. Moving run 3c here was framed as the
power answer, but the run did not die because ana-ml2 is unreliable. It
died because save_steps was 100 and the breaker tripped at step 80, so no
checkpoint existed. save_steps is now 50, which caps a power event at
about eleven minutes. Trading 2.5 hours for 13.3 buys insurance against a
risk already engineered out.
Also records the five launch failures and their causes, and the one that
matters most: the attention-backend trap was present and I first declared
it absent. I checked whether flash-attn was installed, which is the wrong
discriminator; the harness sets flex_attention explicitly in code. Absence
of an alternative is not evidence of the default. The probe now reads the
resolved backend back off the loaded model, and flex_attention does
compile and run on sm_121.
The comparison number nobody had written down: ana-ml2's real run-3c step
times, pulled out of run-03c.log before the breaker killed it. About 10.8
to 15.8 s/it over the first 24 steps, so 604 steps lands at roughly 2.2
to 2.7 hours. Anything under about 45 s/it on the GX10 makes it an
overnight run.
Also records the exact geometry from run-03c.json and a real
adapter_config.json, so the probe measures the shape that actually ran
rather than an approximation of it.
Checked the backend-delta trap the playbook warns about before running
anything rather than after: flash-attn is installed on neither box, so
both fall back to sdpa. Library versions do differ -- torch 2.13.0 versus
2.14.0, transformers 5.15.1 versus 5.16.1 -- and that is recorded rather
than assumed harmless. The probe reads the resolved attention
implementation back off the loaded model instead of trusting the request.
The probe discards its first two steps as warmup, which is not optional on
this box: an unwarmed benchmark here already read 27 TFLOP/s when the true
figure was 93, because it was timing the PTX JIT.
`sudo -v` refreshes the auth timestamp, and a NOPASSWD-only rule creates
no timestamp to refresh, so on sudo >= 1.9.15 `sudo -n -v` returns
non-zero while every real command runs passwordless. Measured:
pfi-gx10 sudo 1.9.15p5 sudo -n -v rc=1 sudo -n true rc=0
nh3-docker sudo 1.9.13p3 sudo -n -v rc=0 sudo -n true rc=0
ana-docker sudo 1.9.13p3 sudo -n -v rc=0 sudo -n true rc=0
irv-ml1 sudo 1.9.13p3 sudo -n -v rc=0 sudo -n true rc=0
Only pfi-gx10 is new enough to hit it today, but every host does as it
moves past 1.9.13, and the failure mode is bad: elway prompts for a
password on a host with working NOPASSWD sudo, which in a
non-interactive run is an EOFError partway through a playbook.
The same probe in my own notes cost this session directly. gx10 looked
like a fleet exception with no NOPASSWD sudo when it had it from account
creation, and the operator was asked for a password that was never
needed. Corrected in auto-memory too.
Also lands the gx10 privileged outfit playbook, now green at 5/5:
NOPASSWD sudo, nvcc, docker group, a CUDA container seeing the GB10, and
the userspace torch stack still working afterward.
Ruling on the architecture question: bare metal, not Proxmox. Proxmox VE
has no aarch64 build, and more fundamentally the GB10's GPU sits on an
on-package root complex cache-coherent with the CPU over NVLink-C2C,
sharing the same LPDDR5X. Passing it to a guest would mean partitioning
the unified memory that is the entire reason for the box. The fleet's
other GPU hosts are bare metal for the same class of reason.
Installed uv and a venv with torch 2.14.0+cu130 plus the full training
stack, and every one of transformers, accelerate, peft, trl, datasets,
safetensors, huggingface_hub and bitsandbytes imports clean on aarch64.
The per-arch unknowns warning did not materialise for any of them.
CUDA works: sm_121, 121.6 GiB addressable, about 93 TFLOP/s dense bf16
with tensor cores confirmed engaged by the bf16-to-fp32 ratio. That is
A6000-class throughput with two and a half times the memory, so capacity
rather than speed is what this box buys.
Two warnings worth keeping. sm_121 is not in torch's compiled arch list,
so everything runs by PTX JIT from sm_120: first use of every kernel pays
a compile, and any library shipping cubins without PTX will fail outright.
And an unwarmed benchmark read 27 TFLOP/s because it was timing that JIT,
which nearly became a phantom report that tensor cores were broken -- the
playbook's section 4 shape exactly, a run that completes and reports
plausible numbers and is wrong.
The privileged half is blocked: infra-ops has no NOPASSWD sudo on this
box, unlike the rest of the fleet, and no credential is vaulted. That
gates nvcc, the container toolkit and the docker group, but not the
run-3c throughput probe.
Probed the box read-only. Alive and idle at 11h48m uptime, 118 of 121 GB
memory free, 822 GB disk free, and completely unchanged since onboarding:
no torch, no nvcc, no uv, and infra-ops is not in the docker group.
The useful finding is a negative one. I assumed the temporary Wi-Fi would
gate getting a 49 GB base model onto the box and it does not. The link is
Wi-Fi 7 on 6 GHz at 2401.9 Mbit/s with a -48 dBm signal, and a measured
300 MB transfer ran at 67 MB/s over SSH, which puts the full base at
about twelve minutes. SSH's cipher is the limiter there, not the radio.
So the throughput probe can run from the desk today and racking is worth
doing for permanence rather than as a blocker.
Also recorded that nvidia-smi reporting FB Memory and BAR1 as N/A is
correct for GB10 rather than a driver fault, since the Grace Blackwell
superchip shares unified LPDDR5X between CPU and GPU and has no discrete
VRAM figure to report.
Run 3c's new intended home is the GX10 rather than a power triage on ana-ml2: a ~240 W
appliance instead of the kilowatt-class box that tripped the breaker, and 121 GB unified
holds the 49 GB bf16 base comfortably where ana-ml2 was tight. The box is bare, so the
first move is a throughput probe rather than a harness port.
Also banked: the Ada migration settling on zfs send with branch (b) ruled out by irv-ml1
keeping its eight services; the Synapse 39-release upgrade with its one-way schema
migration, the appservice namespace opening and the admin API lockdown; the ratified room
alias convention; and a named failure class — a correct check aimed at the wrong object —
with six instances from one day across three sessions.
vh