esh-pfi-infrastructure/servers/pfi-gx10/README.md

# pfi-gx10 — ASUS Ascent GX10 (NVIDIA GB10)

Grace-Blackwell desktop supercomputer. Registered 2026-09-01.

| | |
|---|---|
| GPU | **NVIDIA GB10**, driver 580.173.02, **compute capability 12.1 (`sm_121`)** |
| CPU | 20 cores, **aarch64** |
| Memory | **121 GB unified** (CPU and GPU share it — not 121 GB *plus* VRAM) |
| Storage | 916 GB NVMe, 6% used |
| Kernel | 6.17.0-1031-nvidia |
| Hostname | `pfi-gx10` (shipped with static `gx10-a745`, corrected) |

## Network — racked, and single-path

Racked 2026-09-03. `pfi-gx10.nh3.internal` → **10.100.50.60**, wired only on
`enP7s7`, VLAN 50 (`nh3-servers`), UniFi switch port 22.

**The address lives on the switch side, not the host** — a DHCP *reservation*
against the wired MAC `30:c5:99:3d:a7:45`, with the host left on DHCP. Operator
ruling: a reservation moves with the box, a netplan static goes stale the moment
it does.

⚠ **Wi-Fi is deliberately off and there is now exactly ONE path in.** If the
switch port or the reservation breaks, this is a rack visit. Correct end state
for a racked server, but it is a posture change from the desk setup.

Full detail, including the order that made the move safe:
[`docs/runbooks/gx10-rack-network.md`](../../docs/runbooks/gx10-rack-network.md).

⚠ **The box mounts no NFS, on purpose.** Working data is copied to local NVMe —
see the training section below.

## Access

`infra-ops` with NOPASSWD sudo (operator-bootstrapped). `lkraven` also has key
auth but needs a password for sudo — **automation must connect as `infra-ops`**.

## Headless conversion

`playbooks/gx10-headless.yaml` — run it with the `infra-ops@` prefix, since
elway's `--sudo` applies only to ad-hoc commands and playbook steps carry their
own.

Ships booting to `graphical.target` with GDM and GNOME Remote Desktop. The
playbook sets `multi-user.target`, stops the remote-desktop service, masks the
sleep/suspend/hibernate targets, tells logind to ignore lid and idle, and adds
sshd keepalives so a stalled link does not kill a long job.

⚠ **GDM is `static` on Ubuntu** — pulled in by `display-manager.service`, never
"enabled". Guard and verify on `is-active`, not `is-enabled`; the latter passes
trivially while the desktop is still running.

The playbook will not stop GDM while someone holds a seat session. Override with
`--var force_dm_stop=true`, or just let the rack-install reboot handle it.

## Power-on after AC loss — patched, NOT YET VALIDATED (2026-09-24)

There is **no BMC**, and the ASUS firmware ships with "Restore AC Power Loss" set
to stay off, so the box did not come back after the 2026-09-24 NH3 power outage.
Prime approved patching the setting from the OS, headless:

- UEFI var `Setup-ec87d643-eba4-4bb5-a1e5-3f3e36b20da9`, 81 bytes, attrs `0x7`,
  BIOS `GX10DGX.0105.2026.0505.1153`. Bytes 3–4 changed `0x1d,0x00 → 0x1c,0x01`,
  written with `efivar -w`. The readback shows exactly those two bytes changed.
- The recipe is community-sourced (`Fheodoor01/DGX-Spark---Power-on-boot-`,
  derived by diffing a power-on unit against a stay-off unit; verified by its
  author on GX10DGX.0103/.0104). **Byte 3's meaning is not known.** The resulting
  pair matches factory power-on units.
- **Original bytes are saved twice:** `/var/lib/efi-setup-backup/Setup-ec87d643-20260924-pre-poweron.bin`
  on the box (raw, including the 4-byte attr header) and
  `servers/pfi-gx10/efi-Setup-ec87d643-20260924-pre-poweron.bin` here (sha256
  `6d6f6f77…`). To revert, write `data[4:]` of that file back with
  `efivar -w -n ec87d643-eba4-4bb5-a1e5-3f3e36b20da9-Setup -A 7 -f <file-without-header>`.

⚠ **A shutdown is NOT a test of this setting.** "Power On after AC loss" means the
box starts when power RETURNS after a cut. An OS `poweroff` is supposed to stay
off, by design. The community README claims the two are indistinguishable. They
are not on this board, and trusting that claim cost two trips to the power button
(infra-ops, 1421 and 1435 on 2026-09-24). **The only valid test is pulling and
restoring AC power without touching the button.**

**Status at end of 2026-09-24:** the box is OFF (the 1435 shutdown), with the patch
in place. Prime will pull and restore AC on 2026-09-25. It comes up on its own →
the patch works, so delete "NOT YET VALIDATED" above. It stays off → press the
button, then revert with the backup and set it in the firmware menu via a PiKVM.

## Training — run 3c is staged and ready

The ERP-seat SFT LoRA (run 3c) that died on ana-ml2 at step 24 of 604 to an
Anaheim breaker trip is staged here, unchanged, and **not launched** — that call
is the operator's.

    ssh infra-ops@10.100.50.60 '~/erp-tune/launch-run-03c.sh'

~79.4 s/it measured on this geometry → 604 steps ≈ 13.3 h, peak 75.1 of
121.6 GiB. Slower than ana-ml2's ~2.5 h and still the right host: this is a
~240 W appliance at NH3 that cannot take a client's machines dark.

Base model and the encoded corpus were both sha256-verified identical to
ana-ml2's, so the library delta (transformers 5.15.1 → 5.16.1, x86-64 → aarch64)
is measured to be inert rather than assumed harmless. Runbook:
[`docs/runbooks/gx10-run-03c.md`](../../docs/runbooks/gx10-run-03c.md); canonical
config + launcher in [`scripts/erp-tune-gx10/`](../../scripts/erp-tune-gx10/).

⚠ **Never `pkill -f erp_sft_harness` over SSH** — the pattern is in your own ssh
argv and you kill your shell with it. Kill by PID from `~/erp-tune/run-03c.pid`.

## Relevance to Flash-Next

`sm_121`, not `sm_120`. The SGLang fork evaluated for ana-ml2 (henge item 49)
narrows to **exact SM120 and explicitly excludes SM121/GB10** — it does not apply
here. This chip has its own path: the DGX Spark recipe, which mmaps the ~48 GiB
PLE n-gram table from NVMe rather than holding it in memory. 121 GB unified and
822 GB of free NVMe make that viable on this box in a way it is not on a 96 GB
discrete card.