After the 2026-09-24 NH3 outage pfi-gx10 stayed off. It has no BMC, and the
ASUS firmware defaults to staying off after AC loss. With Prime's approval,
bytes 3-4 of the UEFI Setup variable were changed from 0x1d,0x00 to
0x1c,0x01, a community recipe that matches factory power-on units. The
readback shows exactly those two bytes changed. The original 85-byte raw
variable (attr header included) is committed here and also kept on the box.
The README records that an OS shutdown is not a test of this setting. It
stays off by design, and two such "tests" were invalid. The only valid test
is pulling and restoring AC, which Prime will do on 2026-09-25. It also
records the revert procedure.
Rehome run 3c from ana-ml2 to pfi-gx10 unchanged: same corpus, base,
recipe and hyperparameters, different host. Slower (~13.3 h vs ~2.5 h)
and correct — an Anaheim breaker trip costs a 40-minute drive each way
and 13 hosts down, three of them SureFire client machines, while the
GX10 is a ~240 W appliance at NH3 that can take nothing else down.
Verified rather than assumed, because ana-ml2 ran transformers 5.15.1
on x86-64 and this box runs 5.16.1 on aarch64 — the silent
backend-delta class that has already voided conclusions here:
- both 49 GB base shards sha256-match ana-ml2's (size equality is a
weaker claim and was already true)
- a full encode was run into a throwaway dir and the encoded corpus
compared byte-for-byte: 197,360,233 B, sha256 c08bb1fe2ecb0be3,
identical. Every aggregate matched too. That verified artifact is
what the run will train on — it is seeded into run-03c/encode-cache
- the harness's own suite: 122 passed on aarch64
- the config generator asserts key-by-key that no non-path value
differs from run-03c.json
The encode-cache filename differs by design (base_model_path is part of
the key) — an input hash, not an output hash. Documented so it is not
misread as drift, or "fixed" by faking /tank on this box.
Corpus is copied to local NVMe; the box mounts no NFS. nh3-nas is now on
the same subnet, which makes mounting it tempting and still wrong under
a 13 h unattended run.
The launcher refuses on a live pidfile rather than a pgrep: `pgrep -f
erp_sft_harness` invoked over ssh matches the invoking shell's own argv.
That self-match cost a shell during staging.
Not launched. 13.3 h is the operator's call.
pfi-gx10 moved from desk Wi-Fi (10.100.10.226, VLAN 10) to the rack on
10.100.50.60 (nh3-servers, VLAN 50), reachable as pfi-gx10.nh3.internal.
The address is a DHCP RESERVATION on the UDM, not a host static. Operator
ruling during the move, and the better design: a host-side static works
until the box moves and is then a stale netplan file on a machine whose
address you no longer know. The pre-written playbook wrote a host static;
it is kept for its safety ordering and annotated as retired.
The port arrived on the native VLAN, not the server VLAN, so switch port 22
was repointed first. port_overrides is a whole-array PUT — two unrelated
overrides on ports 21 and 23 were read, preserved and written back, with the
original array backed up to a file before the change.
Wi-Fi stayed up as the escape hatch until the wired path was proven from
outside, and was downed last as its own step. The step worth keeping: while
Wi-Fi was up, traffic to nh3-dev still preferred wlP9s9 because that
interface sits directly on the userland subnet, so reachability proved
nothing about the wired path. `ping -I enP7s7` across the VLAN boundary is
what actually settled it before the hatch came down.
Target settled: nh3-servers VLAN 50, static 10.100.50.60. Clear of the four
existing statics and below the .150 DHCP pool where fleet statics live.
The playbook never leaves itself one path back. Wi-Fi stays up throughout while
the wired interface is configured beside it; the new address is verified from
outside before anything is torn down, and Wi-Fi teardown is explicitly a
separate later change. A botched netplan therefore costs a retry over Wi-Fi
rather than a trip to the rack — which is what substitutes for 'netplan try',
whose interactive rollback needs a TTY that elway cannot provide.
Two preconditions are asserted as steps rather than assumed: the interface must
have carrier (writing a static config for a dead NIC and reporting success is
the failure this avoids), and its MAC must match, since interface names can
renumber across kernels but MACs do not.
Requires nothing from the operator beyond racking the box. The wired NIC has a
distinct MAC from the Wi-Fi one, so the post-move address and switch port are
both discoverable from the UDM rather than needing to be relayed.
NVIDIA GB10, aarch64, 121 GB unified, sm_121. Ships booting to graphical.target
with GDM and GNOME Remote Desktop running.
playbooks/gx10-headless.yaml sets multi-user.target, stops gnome-remote-desktop,
masks the sleep/suspend/hibernate targets, makes logind ignore lid and idle, and
adds sshd keepalives so a stalled link does not kill a long-running job.
Two things learned the hard way and recorded in the playbook:
- gdm is a STATIC unit on Ubuntu, pulled in by display-manager.service and never
'enabled'. A guard of always skips, and a verify
written the same way passes while the desktop is still running. Both now test
is-active. The first run reported six green verifies having not stopped gdm.
- elway's --sudo applies only to ad-hoc --shell/--upload. Playbook steps run as
the connecting user and must carry their own sudo; connect as infra-ops.
The playbook refuses to stop the display manager while a seat session is held,
overridable with --var force_dm_stop=true. Networking is deliberately out of
scope: the box is on a desk on Wi-Fi with a temporary DHCP lease and no ethernet
carrier, and belongs to the rack-install change.
vh