vh
7142657749
Run 3c's new intended home is the GX10 rather than a power triage on ana-ml2: a ~240 W appliance instead of the kilowatt-class box that tripped the breaker, and 121 GB unified holds the 49 GB bf16 base comfortably where ana-ml2 was tight. The box is bare, so the first move is a throughput probe rather than a harness port. Also banked: the Ada migration settling on zfs send with branch (b) ruled out by irv-ml1 keeping its eight services; the Synapse 39-release upgrade with its one-way schema migration, the appservice namespace opening and the admin API lockdown; the ratified room alias convention; and a named failure class — a correct check aimed at the wrong object — with six instances from one day across three sessions.
4.5 KiB
[2026-09-01] pfi-gx10 (ASUS Ascent GX10) onboarded headless — and it is the intended new home for run 3c
ASUS Ascent GX10 arrived and was registered, converted to headless, and given a rack-move playbook. It was NOT racked — the operator ran out of day. It is still on his desk, on Wi-Fi, on a temporary DHCP lease.
| GPU | NVIDIA GB10, driver 580.173.02, compute capability 12.1 (sm_121) |
| CPU | 20 cores, aarch64 |
| Memory | 121 GB UNIFIED — CPU and GPU share it; not 121 GB plus VRAM |
| Storage | 916 GB NVMe, 6% used |
| OS | Ubuntu 24.04.4, kernel 6.17.0-1031-nvidia |
| Access | infra-ops NOPASSWD sudo (operator-bootstrapped). lkraven has key auth but needs a password to escalate — automation must connect as infra-ops. |
Purpose (operator, 2026-09-01)
Local inference experiments and the failed training — run 3c. That is the whole point: run 3c died on ana-ml2 when a kilowatt-class training box tripped a breaker (2× 300 W GPUs + dual EPYC 9254). The GX10 is a ~240 W appliance, roughly a fifth of the draw, on a different site's circuits.
The memory arithmetic favours it strongly. Run 3c is a LoRA (r=64, batch 2 × accum 8,
gradient checkpointing) over gemma4-26b-a4b-it-bf16 — 49 GB of base weights, working
set roughly 55–65 GB. The gemma4-charrp compose warns in capitals that "48.10 GiB of BF16
weights CANNOT be served here" on ana-ml2's shared GPU0. 121 GB unified turns that
constraint into a non-issue.
What has NOT been established — do not assume any of it
- The box is bare. No torch, no nvcc, no CUDA stack. Docker 29.2.1 and 822 GB free.
- aarch64 dependency risk. torch 2.9.1+cu130 ARM64 wheels exist; transformers/TRL/PEFT are pure Python. Compiled deps — flash-attn, bitsandbytes, liger, xformers — are open questions per-arch.
- Triton has NO sm_121 support (established independently by comfy-dev the same day).
Anything reaching for
torch.compileor Triton-backed kernels is closed on this silicon. - Throughput is unmeasured. Run 3c ran 16.45 s/it on a 300 W RTX PRO 6000. GB10 will be slower; how much decides whether 604 steps is an overnight run or two days. Measure this before porting anything — the recommended first move is a probe: install ARM64 torch, load the base, run ten steps, report s/it.
- Model transfer: 49 GB from ana-ml2 over the Anaheim↔NH3 link at a measured 32.4 MB/s — about 25 minutes. Note that is a third of the irv-ml1↔NH3 link's 99 MB/s.
The headless conversion, and the lesson inside it
playbooks/gx10-headless.yaml (1b596c8): multi-user.target, gnome-remote-desktop stopped,
sleep/suspend/hibernate masked, logind ignores lid and idle, sshd keepalives, hostname
corrected gx10-a745 → pfi-gx10.
⚠ gdm is a STATIC unit on Ubuntu — pulled in by display-manager.service, never
"enabled". The first version guarded on is-enabled | grep enabled, which always skips, and
the verify tested the same wrong property and passed. Six green verifies having not
stopped the display manager. Both now test is-active. This is an instance of
2026-09-01-wrong-object-measurement.
The playbook refuses to stop GDM while a seat session is held (--var force_dm_stop=true to
override) — automation should not yank a display out from under someone at the machine.
⚠ elway's --sudo applies only to ad-hoc --shell/--upload. Playbook steps run as the
connecting user and must carry their own sudo.
The rack move, written but not run
playbooks/gx10-rack-network.yaml (a0c5fc6) — target settled: nh3-servers VLAN 50,
static 10.100.50.60, clear of .40/.42/.50/.90 and below the .150 DHCP pool.
Requires nothing from the operator beyond racking it. The wired NIC has its own MAC
(30:c5:99:3d:a7:45, distinct from Wi-Fi 50:bb:b5:a2:00:a8), so the post-move address AND
switch port are discoverable from the UDM rather than relayed.
Design property worth preserving: the playbook never leaves itself one path back. Wi-Fi
stays up while the wired interface is configured beside it; Wi-Fi teardown is explicitly a
separate later change. netplan try's auto-rollback needs a TTY that elway cannot provide,
so a live Wi-Fi link is what substitutes for it. Two preconditions are asserted as steps
rather than assumed: carrier must be 1, and the MAC must match (interface names renumber
across kernels; MACs do not).
⚠ The GX10 is NOT the Ada box. Two separate machines — see 2026-09-01-ada-migration-branch-a.