From 1b596c8c308c52d124cf5d75180cbda1c50a878f Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Tue, 1 Sep 2026 11:36:49 -0700 Subject: [PATCH] feat(pfi-gx10): register the ASUS Ascent GX10 and convert it to headless NVIDIA GB10, aarch64, 121 GB unified, sm_121. Ships booting to graphical.target with GDM and GNOME Remote Desktop running. playbooks/gx10-headless.yaml sets multi-user.target, stops gnome-remote-desktop, masks the sleep/suspend/hibernate targets, makes logind ignore lid and idle, and adds sshd keepalives so a stalled link does not kill a long-running job. Two things learned the hard way and recorded in the playbook: - gdm is a STATIC unit on Ubuntu, pulled in by display-manager.service and never 'enabled'. A guard of always skips, and a verify written the same way passes while the desktop is still running. Both now test is-active. The first run reported six green verifies having not stopped gdm. - elway's --sudo applies only to ad-hoc --shell/--upload. Playbook steps run as the connecting user and must carry their own sudo; connect as infra-ops. The playbook refuses to stop the display manager while a seat session is held, overridable with --var force_dm_stop=true. Networking is deliberately out of scope: the box is on a desk on Wi-Fi with a temporary DHCP lease and no ethernet carrier, and belongs to the rack-install change. --- playbooks/gx10-headless.yaml | 145 +++++++++++++++++++++++++++++++++++ servers/pfi-gx10/README.md | 57 ++++++++++++++ servers/pfi-gx10/ssh-target | 1 + 3 files changed, 203 insertions(+) create mode 100644 playbooks/gx10-headless.yaml create mode 100644 servers/pfi-gx10/README.md create mode 100644 servers/pfi-gx10/ssh-target diff --git a/playbooks/gx10-headless.yaml b/playbooks/gx10-headless.yaml new file mode 100644 index 0000000..7ce4008 --- /dev/null +++ b/playbooks/gx10-headless.yaml @@ -0,0 +1,145 @@ +# pfi-gx10 — convert an ASUS Ascent GX10 (NVIDIA GB10, aarch64) from its +# out-of-box desktop configuration to headless server operation. +# +# scripts/elway infra-ops@10.100.10.226 --playbook playbooks/gx10-headless.yaml +# +# CONNECT AS infra-ops. `--sudo` only applies to ad-hoc --shell/--upload, so +# playbook steps run as the connecting user and carry their own sudo. lkraven +# needs a password for sudo on this box; infra-ops is NOPASSWD. +# +# The box ships booting to graphical.target with GDM and GNOME Remote Desktop +# running. That is correct for a desk appliance and wrong for a rack node: a +# display manager holds a GPU context, and a laptop-derived power profile will +# suspend a machine that is meant to answer at 3am. +# +# OUT OF SCOPE — DELIBERATELY: +# Networking. As of 2026-09-01 this box sits on a desk on Wi-Fi with a +# temporary DHCP lease (10.100.10.226) and NO ethernet carrier (enP7s7 +# carrier=0). Static addressing, VLAN placement and DNS all belong to the +# rack-install change, not to this one. Configuring an address that is about +# to be wrong is worse than leaving it on DHCP. +# +# REVERSIBLE. Every step here undoes with: +# systemctl set-default graphical.target +# systemctl unmask sleep.target suspend.target hibernate.target +# systemctl enable --now gdm +# +# Idempotent: a second run should report ok/skipped throughout. + +vars: + fqdn_hostname: pfi-gx10 + # Stop the display manager even if someone is logged in at the console. + # scripts/elway infra-ops@ -p ... --var force_dm_stop=true + force_dm_stop: "false" + +steps: + # --------------------------------------------------------------------------- + # Identity + # --------------------------------------------------------------------------- + - name: Set the static hostname to match the transient one + # Ships with static=gx10-a745 but transient=pfi-gx10, so `hostname` and + # `hostnamectl` disagree. Anything keying on one of them picks the wrong + # name half the time. pfi-gx10 is the fleet-convention name. + shell: sudo hostnamectl set-hostname {{ fqdn_hostname }} + when: "test \"$(hostnamectl --static)\" != '{{ fqdn_hostname }}'" + + # --------------------------------------------------------------------------- + # Stop booting into a desktop + # --------------------------------------------------------------------------- + - name: Default boot target -> multi-user + shell: sudo systemctl set-default multi-user.target + when: "test \"$(systemctl get-default)\" != 'multi-user.target'" + + - name: Stop GDM + # ⚠ GUARD ON is-active, NOT is-enabled. Ubuntu ships gdm as a STATIC unit — + # it is pulled in by display-manager.service and is never "enabled". A + # `is-enabled | grep enabled` guard therefore always skips, and a verify + # written the same way passes while gdm is still running. Cost one green + # run that had not done the thing. (2026-09-01) + # + # GNOME stays INSTALLED, only stopped: removing it frees little on a 916G + # disk and makes this hard to walk back from a serial console. + # + # Will NOT run while someone holds a seat session, unless force_dm_stop=true. + # Yanking the display out from under a person at the machine is not a thing + # automation should decide to do. + shell: sudo systemctl stop gdm + when: "systemctl is-active --quiet gdm && { [ '{{ force_dm_stop }}' = 'true' ] || ! loginctl list-sessions --no-legend 2>/dev/null | grep -qE ' seat[0-9]+ '; }" + + - name: Stop and disable GNOME Remote Desktop + # Needs a graphical session to be useful; with GDM gone it is a listener + # with nothing behind it. + shell: sudo systemctl disable --now gnome-remote-desktop + when: "systemctl is-enabled gnome-remote-desktop 2>/dev/null | grep -q enabled" + + # --------------------------------------------------------------------------- + # A server must never sleep + # --------------------------------------------------------------------------- + - name: Mask sleep / suspend / hibernate / hybrid-sleep + # Masking, not disabling. These are `static` units pulled in by logind and + # by desktop power policy; disable does not stick, mask does. + shell: sudo systemctl mask sleep.target suspend.target hibernate.target hybrid-sleep.target + when: "! systemctl is-enabled sleep.target 2>/dev/null | grep -q masked" + + - name: logind must ignore lid and idle + shell: | + sudo install -d /etc/systemd/logind.conf.d + sudo tee /etc/systemd/logind.conf.d/10-headless.conf >/dev/null <<'EOF' + [Login] + HandleLidSwitch=ignore + HandleLidSwitchExternalPower=ignore + HandleLidSwitchDocked=ignore + IdleAction=ignore + EOF + creates: /etc/systemd/logind.conf.d/10-headless.conf + + # --------------------------------------------------------------------------- + # Keep long jobs alive across a dropped SSH session + # --------------------------------------------------------------------------- + - name: sshd keepalives so a NAT/Wi-Fi stall does not kill a running job + shell: | + sudo install -d /etc/ssh/sshd_config.d + sudo tee /etc/ssh/sshd_config.d/10-fleet.conf >/dev/null <<'EOF' + ClientAliveInterval 60 + ClientAliveCountMax 10 + TCPKeepAlive yes + EOF + sudo sshd -t + sudo systemctl reload ssh 2>/dev/null || sudo systemctl reload sshd + creates: /etc/ssh/sshd_config.d/10-fleet.conf + +verify: + - name: Boots to multi-user, not graphical + shell: test "$(systemctl get-default)" = "multi-user.target" + changed_when: "false" + + - name: Will boot headless — default target is multi-user + # The DURABLE requirement, true across reboots regardless of what is + # running right now. + shell: test "$(systemctl get-default)" = "multi-user.target" + changed_when: "false" + + - name: GDM is not running + # is-ACTIVE. Expected to FAIL while someone holds a console session and + # force_dm_stop is false — that is an honest red, not a broken playbook. + # A reboot (or the rack install) resolves it. + shell: "! systemctl is-active --quiet gdm" + changed_when: "false" + + - name: Sleep targets are masked + shell: systemctl is-enabled sleep.target 2>/dev/null | grep -q masked + changed_when: "false" + + - name: Static hostname is correct + shell: test "$(hostnamectl --static)" = "{{ fqdn_hostname }}" + changed_when: "false" + + - name: GPU still enumerates after the desktop stack stopped + # The point of the exercise. If disabling GDM cost us the GPU, this catches + # it here rather than the next time someone tries to load a model. + shell: nvidia-smi --query-gpu=name,compute_cap --format=csv,noheader | grep -q GB10 + changed_when: "false" + + - name: sshd config still parses + shell: sudo sshd -t + changed_when: "false" diff --git a/servers/pfi-gx10/README.md b/servers/pfi-gx10/README.md new file mode 100644 index 0000000..5604d3a --- /dev/null +++ b/servers/pfi-gx10/README.md @@ -0,0 +1,57 @@ +# pfi-gx10 — ASUS Ascent GX10 (NVIDIA GB10) + +Grace-Blackwell desktop supercomputer. Registered 2026-09-01. + +| | | +|---|---| +| GPU | **NVIDIA GB10**, driver 580.173.02, **compute capability 12.1 (`sm_121`)** | +| CPU | 20 cores, **aarch64** | +| Memory | **121 GB unified** (CPU and GPU share it — not 121 GB *plus* VRAM) | +| Storage | 916 GB NVMe, 6% used | +| Kernel | 6.17.0-1031-nvidia | +| Hostname | `pfi-gx10` (shipped with static `gx10-a745`, corrected) | + +## ⚠ The address in `ssh-target` is TEMPORARY + +As of 2026-09-01 this box is **on a desk, on Wi-Fi**, holding a DHCP lease at +`10.100.10.226` on `nh3-userland` (VLAN 10). Ethernet `enP7s7` has no carrier. + +It is going into the rack later. At that point it needs a wired link, a static +address, a decision about VLAN placement (a compute node arguably belongs on +`nh3-servers`, VLAN 50, with the rest of them), a `dns/internal.yaml` entry, and +this file plus `ssh-target` updated. **None of that was configured now** — an +address that is about to be wrong is worse than DHCP. + +Expect ~65–80 ms RTT until it is wired. That is Wi-Fi power-save, not a fault. + +## Access + +`infra-ops` with NOPASSWD sudo (operator-bootstrapped). `lkraven` also has key +auth but needs a password for sudo — **automation must connect as `infra-ops`**. + +## Headless conversion + +`playbooks/gx10-headless.yaml` — run it with the `infra-ops@` prefix, since +elway's `--sudo` applies only to ad-hoc commands and playbook steps carry their +own. + +Ships booting to `graphical.target` with GDM and GNOME Remote Desktop. The +playbook sets `multi-user.target`, stops the remote-desktop service, masks the +sleep/suspend/hibernate targets, tells logind to ignore lid and idle, and adds +sshd keepalives so a stalled link does not kill a long job. + +⚠ **GDM is `static` on Ubuntu** — pulled in by `display-manager.service`, never +"enabled". Guard and verify on `is-active`, not `is-enabled`; the latter passes +trivially while the desktop is still running. + +The playbook will not stop GDM while someone holds a seat session. Override with +`--var force_dm_stop=true`, or just let the rack-install reboot handle it. + +## Relevance to Flash-Next + +`sm_121`, not `sm_120`. The SGLang fork evaluated for ana-ml2 (henge item 49) +narrows to **exact SM120 and explicitly excludes SM121/GB10** — it does not apply +here. This chip has its own path: the DGX Spark recipe, which mmaps the ~48 GiB +PLE n-gram table from NVMe rather than holding it in memory. 121 GB unified and +822 GB of free NVMe make that viable on this box in a way it is not on a 96 GB +discrete card. diff --git a/servers/pfi-gx10/ssh-target b/servers/pfi-gx10/ssh-target new file mode 100644 index 0000000..ccea938 --- /dev/null +++ b/servers/pfi-gx10/ssh-target @@ -0,0 +1 @@ +infra-ops@10.100.10.226