esh-pfi-infrastructure/servers/pfi-gx10/README.md

# pfi-gx10 — ASUS Ascent GX10 (NVIDIA GB10)

Grace-Blackwell desktop supercomputer. Registered 2026-09-01.

| | |
|---|---|
| GPU | **NVIDIA GB10**, driver 580.173.02, **compute capability 12.1 (`sm_121`)** |
| CPU | 20 cores, **aarch64** |
| Memory | **121 GB unified** (CPU and GPU share it — not 121 GB *plus* VRAM) |
| Storage | 916 GB NVMe, 6% used |
| Kernel | 6.17.0-1031-nvidia |
| Hostname | `pfi-gx10` (shipped with static `gx10-a745`, corrected) |

## Network — racked, and single-path

Racked 2026-09-03. `pfi-gx10.nh3.internal` → **10.100.50.60**, wired only on
`enP7s7`, VLAN 50 (`nh3-servers`), UniFi switch port 22.

**The address lives on the switch side, not the host** — a DHCP *reservation*
against the wired MAC `30:c5:99:3d:a7:45`, with the host left on DHCP. Operator
ruling: a reservation moves with the box, a netplan static goes stale the moment
it does.

⚠ **Wi-Fi is deliberately off and there is now exactly ONE path in.** If the
switch port or the reservation breaks, this is a rack visit. Correct end state
for a racked server, but it is a posture change from the desk setup.

Full detail, including the order that made the move safe:
[`docs/runbooks/gx10-rack-network.md`](../../docs/runbooks/gx10-rack-network.md).

⚠ **The box mounts no NFS, on purpose.** Working data is copied to local NVMe —
see the training section below.

## Access

`infra-ops` with NOPASSWD sudo (operator-bootstrapped). `lkraven` also has key
auth but needs a password for sudo — **automation must connect as `infra-ops`**.

## Headless conversion

`playbooks/gx10-headless.yaml` — run it with the `infra-ops@` prefix, since
elway's `--sudo` applies only to ad-hoc commands and playbook steps carry their
own.

Ships booting to `graphical.target` with GDM and GNOME Remote Desktop. The
playbook sets `multi-user.target`, stops the remote-desktop service, masks the
sleep/suspend/hibernate targets, tells logind to ignore lid and idle, and adds
sshd keepalives so a stalled link does not kill a long job.

⚠ **GDM is `static` on Ubuntu** — pulled in by `display-manager.service`, never
"enabled". Guard and verify on `is-active`, not `is-enabled`; the latter passes
trivially while the desktop is still running.

The playbook will not stop GDM while someone holds a seat session. Override with
`--var force_dm_stop=true`, or just let the rack-install reboot handle it.

## Power-on after AC loss — patched 2026-09-24, VALIDATED 2026-09-25

There is **no BMC**, and the ASUS firmware ships with "Restore AC Power Loss" set
to stay off, so the box did not come back after the 2026-09-24 NH3 power outage.
Prime approved patching the setting from the OS, headless:

- UEFI var `Setup-ec87d643-eba4-4bb5-a1e5-3f3e36b20da9`, 81 bytes, attrs `0x7`,
  BIOS `GX10DGX.0105.2026.0505.1153`. Bytes 3–4 changed `0x1d,0x00 → 0x1c,0x01`,
  written with `efivar -w`. The readback shows exactly those two bytes changed.
- The recipe is community-sourced (`Fheodoor01/DGX-Spark---Power-on-boot-`,
  derived by diffing a power-on unit against a stay-off unit; verified by its
  author on GX10DGX.0103/.0104). **Byte 3's meaning is not known.** The resulting
  pair matches factory power-on units.
- **Original bytes are saved twice:** `/var/lib/efi-setup-backup/Setup-ec87d643-20260924-pre-poweron.bin`
  on the box (raw, including the 4-byte attr header) and
  `servers/pfi-gx10/efi-Setup-ec87d643-20260924-pre-poweron.bin` here (sha256
  `6d6f6f77…`). To revert, write `data[4:]` of that file back with
  `efivar -w -n ec87d643-eba4-4bb5-a1e5-3f3e36b20da9-Setup -A 7 -f <file-without-header>`.

⚠ **A shutdown is NOT a test of this setting.** "Power On after AC loss" means the
box starts when power RETURNS after a cut. An OS `poweroff` is supposed to stay
off, by design. The community README claims the two are indistinguishable. They
are not on this board, and trusting that claim cost two trips to the power button
(infra-ops, 1421 and 1435 on 2026-09-24). **The only valid test is pulling and
restoring AC power without touching the button.**

**Validated 2026-09-25 (Prime, relayed by Miranda):** the box had been off since
the 09-24 1435 shutdown. Prime pulled and replugged AC, and it **came up by itself**,
booting at 1108:54 PT without the button being pressed. The patch works; keep the
backup, since a firmware update may reset `Setup`. Re-check after any BIOS update
the same way, by pulling AC and not by a shutdown.

**Planned OOB (Prime, 2026-09-25):** nh3-pve's NanoKVM moves here on the next NH3
site visit. That also gives this box its first console. See `servers/nh3-pve/README.md`.

## Training — run 3c is staged and ready

The ERP-seat SFT LoRA (run 3c) that died on ana-ml2 at step 24 of 604 to an
Anaheim breaker trip is staged here, unchanged, and **not launched** — that call
is the operator's.

    ssh infra-ops@10.100.50.60 '~/erp-tune/launch-run-03c.sh'

~79.4 s/it measured on this geometry → 604 steps ≈ 13.3 h, peak 75.1 of
121.6 GiB. Slower than ana-ml2's ~2.5 h and still the right host: this is a
~240 W appliance at NH3 that cannot take a client's machines dark.

Base model and the encoded corpus were both sha256-verified identical to
ana-ml2's, so the library delta (transformers 5.15.1 → 5.16.1, x86-64 → aarch64)
is measured to be inert rather than assumed harmless. Runbook:
[`docs/runbooks/gx10-run-03c.md`](../../docs/runbooks/gx10-run-03c.md); canonical
config + launcher in [`scripts/erp-tune-gx10/`](../../scripts/erp-tune-gx10/).

⚠ **Never `pkill -f erp_sft_harness` over SSH** — the pattern is in your own ssh
argv and you kill your shell with it. Kill by PID from `~/erp-tune/run-03c.pid`.

## Relevance to Flash-Next

`sm_121`, not `sm_120`. The SGLang fork evaluated for ana-ml2 (henge item 49)
narrows to **exact SM120 and explicitly excludes SM121/GB10** — it does not apply
here. This chip has its own path: the DGX Spark recipe, which mmaps the ~48 GiB
PLE n-gram table from NVMe rather than holding it in memory. 121 GB unified and
822 GB of free NVMe make that viable on this box in a way it is not on a 96 GB
discrete card.