esh-pfi-infrastructure/servers/pfi-gx10

pfi-gx10 — ASUS Ascent GX10 (NVIDIA GB10)

Grace-Blackwell desktop supercomputer. Registered 2026-09-01.

GPU	NVIDIA GB10, driver 580.173.02, compute capability 12.1 (sm_121)
CPU	20 cores, aarch64
Memory	121 GB unified (CPU and GPU share it — not 121 GB plus VRAM)
Storage	916 GB NVMe, 6% used
Kernel	6.17.0-1031-nvidia
Hostname	pfi-gx10 (shipped with static gx10-a745, corrected)

Network — racked, and single-path

Racked 2026-09-03. pfi-gx10.nh3.internal → 10.100.50.60, wired only on enP7s7, VLAN 50 (nh3-servers), UniFi switch port 22.

The address lives on the switch side, not the host — a DHCP reservation against the wired MAC 30:c5:99:3d:a7:45, with the host left on DHCP. Operator ruling: a reservation moves with the box, a netplan static goes stale the moment it does.

⚠ Wi-Fi is deliberately off and there is now exactly ONE path in. If the switch port or the reservation breaks, this is a rack visit. Correct end state for a racked server, but it is a posture change from the desk setup.

Full detail, including the order that made the move safe: docs/runbooks/gx10-rack-network.md.

⚠ The box mounts no NFS, on purpose. Working data is copied to local NVMe — see the training section below.

Access

infra-ops with NOPASSWD sudo (operator-bootstrapped). lkraven also has key auth but needs a password for sudo — automation must connect as infra-ops.

Headless conversion

playbooks/gx10-headless.yaml — run it with the infra-ops@ prefix, since elway's --sudo applies only to ad-hoc commands and playbook steps carry their own.

Ships booting to graphical.target with GDM and GNOME Remote Desktop. The playbook sets multi-user.target, stops the remote-desktop service, masks the sleep/suspend/hibernate targets, tells logind to ignore lid and idle, and adds sshd keepalives so a stalled link does not kill a long job.

⚠ GDM is static on Ubuntu — pulled in by display-manager.service, never "enabled". Guard and verify on is-active, not is-enabled; the latter passes trivially while the desktop is still running.

The playbook will not stop GDM while someone holds a seat session. Override with --var force_dm_stop=true, or just let the rack-install reboot handle it.

Power-on after AC loss — patched, NOT YET VALIDATED (2026-09-24)

There is no BMC, and the ASUS firmware ships with "Restore AC Power Loss" set to stay off, so the box did not come back after the 2026-09-24 NH3 power outage. Prime approved patching the setting from the OS, headless:

• UEFI var Setup-ec87d643-eba4-4bb5-a1e5-3f3e36b20da9, 81 bytes, attrs 0x7, BIOS GX10DGX.0105.2026.0505.1153. Bytes 3–4 changed 0x1d,0x00 → 0x1c,0x01, written with efivar -w. The readback shows exactly those two bytes changed.
• The recipe is community-sourced (Fheodoor01/DGX-Spark---Power-on-boot-, derived by diffing a power-on unit against a stay-off unit; verified by its author on GX10DGX.0103/.0104). Byte 3's meaning is not known. The resulting pair matches factory power-on units.
• Original bytes are saved twice: /var/lib/efi-setup-backup/Setup-ec87d643-20260924-pre-poweron.bin on the box (raw, including the 4-byte attr header) and servers/pfi-gx10/efi-Setup-ec87d643-20260924-pre-poweron.bin here (sha256 6d6f6f77…). To revert, write data[4:] of that file back with efivar -w -n ec87d643-eba4-4bb5-a1e5-3f3e36b20da9-Setup -A 7 -f <file-without-header>.

⚠ A shutdown is NOT a test of this setting. "Power On after AC loss" means the box starts when power RETURNS after a cut. An OS poweroff is supposed to stay off, by design. The community README claims the two are indistinguishable. They are not on this board, and trusting that claim cost two trips to the power button (infra-ops, 1421 and 1435 on 2026-09-24). The only valid test is pulling and restoring AC power without touching the button.

Status at end of 2026-09-24: the box is OFF (the 1435 shutdown), with the patch in place. Prime will pull and restore AC on 2026-09-25. It comes up on its own → the patch works, so delete "NOT YET VALIDATED" above. It stays off → press the button, then revert with the backup and set it in the firmware menu via a PiKVM.

Training — run 3c is staged and ready

The ERP-seat SFT LoRA (run 3c) that died on ana-ml2 at step 24 of 604 to an Anaheim breaker trip is staged here, unchanged, and not launched — that call is the operator's.

ssh infra-ops@10.100.50.60 '~/erp-tune/launch-run-03c.sh'

~79.4 s/it measured on this geometry → 604 steps ≈ 13.3 h, peak 75.1 of 121.6 GiB. Slower than ana-ml2's ~2.5 h and still the right host: this is a ~240 W appliance at NH3 that cannot take a client's machines dark.

Base model and the encoded corpus were both sha256-verified identical to ana-ml2's, so the library delta (transformers 5.15.1 → 5.16.1, x86-64 → aarch64) is measured to be inert rather than assumed harmless. Runbook: docs/runbooks/gx10-run-03c.md; canonical config + launcher in scripts/erp-tune-gx10/.

⚠ Never pkill -f erp_sft_harness over SSH — the pattern is in your own ssh argv and you kill your shell with it. Kill by PID from ~/erp-tune/run-03c.pid.

Relevance to Flash-Next

sm_121, not sm_120. The SGLang fork evaluated for ana-ml2 (henge item 49) narrows to exact SM120 and explicitly excludes SM121/GB10 — it does not apply here. This chip has its own path: the DGX Spark recipe, which mmaps the ~48 GiB PLE n-gram table from NVMe rather than holding it in memory. 121 GB unified and 822 GB of free NVMe make that viable on this box in a way it is not on a 96 GB discrete card.