memory: snapshot — gx10 unracked and next up as an inference+training box
Run 3c's new intended home is the GX10 rather than a power triage on ana-ml2: a ~240 W appliance instead of the kilowatt-class box that tripped the breaker, and 121 GB unified holds the 49 GB bf16 base comfortably where ana-ml2 was tight. The box is bare, so the first move is a throughput probe rather than a harness port. Also banked: the Ada migration settling on zfs send with branch (b) ruled out by irv-ml1 keeping its eight services; the Synapse 39-release upgrade with its one-way schema migration, the appservice namespace opening and the admin API lockdown; the ratified room alias convention; and a named failure class — a correct check aimed at the wrong object — with six instances from one day across three sessions.
This commit is contained in:
@@ -0,0 +1,79 @@
|
||||
# `[2026-09-01]` pfi-gx10 (ASUS Ascent GX10) onboarded headless — and it is the intended new home for run 3c
|
||||
|
||||
ASUS Ascent GX10 arrived and was registered, converted to headless, and given a rack-move
|
||||
playbook. **It was NOT racked** — the operator ran out of day. It is still on his desk, on
|
||||
Wi-Fi, on a temporary DHCP lease.
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| GPU | **NVIDIA GB10**, driver 580.173.02, **compute capability 12.1 (`sm_121`)** |
|
||||
| CPU | 20 cores, **aarch64** |
|
||||
| Memory | **121 GB UNIFIED** — CPU and GPU share it; not 121 GB *plus* VRAM |
|
||||
| Storage | 916 GB NVMe, 6% used |
|
||||
| OS | Ubuntu 24.04.4, kernel 6.17.0-1031-nvidia |
|
||||
| Access | `infra-ops` NOPASSWD sudo (operator-bootstrapped). `lkraven` has key auth but needs a password to escalate — **automation must connect as `infra-ops`**. |
|
||||
|
||||
## Purpose (operator, 2026-09-01)
|
||||
|
||||
Local inference experiments **and** the failed training — run 3c. That is the whole point:
|
||||
run 3c died on ana-ml2 when a **kilowatt-class** training box tripped a breaker
|
||||
(2× 300 W GPUs + dual EPYC 9254). The GX10 is a ~240 W appliance, roughly a fifth of the
|
||||
draw, on a different site's circuits.
|
||||
|
||||
**The memory arithmetic favours it strongly.** Run 3c is a LoRA (r=64, batch 2 × accum 8,
|
||||
gradient checkpointing) over `gemma4-26b-a4b-it-bf16` — **49 GB of base weights**, working
|
||||
set roughly 55–65 GB. The `gemma4-charrp` compose warns in capitals that "48.10 GiB of BF16
|
||||
weights CANNOT be served here" on ana-ml2's shared GPU0. 121 GB unified turns that
|
||||
constraint into a non-issue.
|
||||
|
||||
## What has NOT been established — do not assume any of it
|
||||
|
||||
1. **The box is bare.** No torch, no nvcc, no CUDA stack. Docker 29.2.1 and 822 GB free.
|
||||
2. **aarch64 dependency risk.** torch 2.9.1+cu130 ARM64 wheels exist; transformers/TRL/PEFT
|
||||
are pure Python. Compiled deps — flash-attn, bitsandbytes, liger, xformers — are open
|
||||
questions per-arch.
|
||||
3. **Triton has NO sm_121 support** (established independently by comfy-dev the same day).
|
||||
Anything reaching for `torch.compile` or Triton-backed kernels is closed on this silicon.
|
||||
4. **Throughput is unmeasured.** Run 3c ran 16.45 s/it on a 300 W RTX PRO 6000. GB10 will be
|
||||
slower; how much decides whether 604 steps is an overnight run or two days.
|
||||
**Measure this before porting anything** — the recommended first move is a probe: install
|
||||
ARM64 torch, load the base, run ten steps, report s/it.
|
||||
5. **Model transfer:** 49 GB from ana-ml2 over the Anaheim↔NH3 link at a **measured
|
||||
32.4 MB/s** — about 25 minutes. Note that is a third of the irv-ml1↔NH3 link's 99 MB/s.
|
||||
|
||||
## The headless conversion, and the lesson inside it
|
||||
|
||||
`playbooks/gx10-headless.yaml` (`1b596c8`): multi-user.target, gnome-remote-desktop stopped,
|
||||
sleep/suspend/hibernate **masked**, logind ignores lid and idle, sshd keepalives, hostname
|
||||
corrected `gx10-a745` → `pfi-gx10`.
|
||||
|
||||
⚠ **`gdm` is a STATIC unit on Ubuntu** — pulled in by `display-manager.service`, never
|
||||
"enabled". The first version guarded on `is-enabled | grep enabled`, which always skips, and
|
||||
**the verify tested the same wrong property and passed**. Six green verifies having not
|
||||
stopped the display manager. Both now test `is-active`. This is an instance of
|
||||
[[2026-09-01-wrong-object-measurement]].
|
||||
|
||||
The playbook refuses to stop GDM while a seat session is held (`--var force_dm_stop=true` to
|
||||
override) — automation should not yank a display out from under someone at the machine.
|
||||
|
||||
⚠ **elway's `--sudo` applies only to ad-hoc `--shell`/`--upload`.** Playbook steps run as the
|
||||
connecting user and must carry their own `sudo`.
|
||||
|
||||
## The rack move, written but not run
|
||||
|
||||
`playbooks/gx10-rack-network.yaml` (`a0c5fc6`) — target settled: **`nh3-servers` VLAN 50,
|
||||
static `10.100.50.60`**, clear of `.40`/`.42`/`.50`/`.90` and below the `.150` DHCP pool.
|
||||
|
||||
Requires nothing from the operator beyond racking it. The wired NIC has its own MAC
|
||||
(`30:c5:99:3d:a7:45`, distinct from Wi-Fi `50:bb:b5:a2:00:a8`), so the post-move address AND
|
||||
switch port are discoverable from the UDM rather than relayed.
|
||||
|
||||
**Design property worth preserving:** the playbook never leaves itself one path back. Wi-Fi
|
||||
stays up while the wired interface is configured beside it; Wi-Fi teardown is explicitly a
|
||||
separate later change. `netplan try`'s auto-rollback needs a TTY that elway cannot provide,
|
||||
so a live Wi-Fi link is what substitutes for it. Two preconditions are asserted as steps
|
||||
rather than assumed: carrier must be 1, and the MAC must match (interface names renumber
|
||||
across kernels; MACs do not).
|
||||
|
||||
⚠ **The GX10 is NOT the Ada box.** Two separate machines — see
|
||||
[[2026-09-01-ada-migration-branch-a]].
|
||||
Reference in New Issue
Block a user