Rehome run 3c from ana-ml2 to pfi-gx10 unchanged: same corpus, base,
recipe and hyperparameters, different host. Slower (~13.3 h vs ~2.5 h)
and correct — an Anaheim breaker trip costs a 40-minute drive each way
and 13 hosts down, three of them SureFire client machines, while the
GX10 is a ~240 W appliance at NH3 that can take nothing else down.
Verified rather than assumed, because ana-ml2 ran transformers 5.15.1
on x86-64 and this box runs 5.16.1 on aarch64 — the silent
backend-delta class that has already voided conclusions here:
- both 49 GB base shards sha256-match ana-ml2's (size equality is a
weaker claim and was already true)
- a full encode was run into a throwaway dir and the encoded corpus
compared byte-for-byte: 197,360,233 B, sha256 c08bb1fe2ecb0be3,
identical. Every aggregate matched too. That verified artifact is
what the run will train on — it is seeded into run-03c/encode-cache
- the harness's own suite: 122 passed on aarch64
- the config generator asserts key-by-key that no non-path value
differs from run-03c.json
The encode-cache filename differs by design (base_model_path is part of
the key) — an input hash, not an output hash. Documented so it is not
misread as drift, or "fixed" by faking /tank on this box.
Corpus is copied to local NVMe; the box mounts no NFS. nh3-nas is now on
the same subnet, which makes mounting it tempting and still wrong under
a 13 h unattended run.
The launcher refuses on a live pidfile rather than a pgrep: `pgrep -f
erp_sft_harness` invoked over ssh matches the invoking shell's own argv.
That self-match cost a shell during staging.
Not launched. 13.3 h is the operator's call.
Target settled: nh3-servers VLAN 50, static 10.100.50.60. Clear of the four
existing statics and below the .150 DHCP pool where fleet statics live.
The playbook never leaves itself one path back. Wi-Fi stays up throughout while
the wired interface is configured beside it; the new address is verified from
outside before anything is torn down, and Wi-Fi teardown is explicitly a
separate later change. A botched netplan therefore costs a retry over Wi-Fi
rather than a trip to the rack — which is what substitutes for 'netplan try',
whose interactive rollback needs a TTY that elway cannot provide.
Two preconditions are asserted as steps rather than assumed: the interface must
have carrier (writing a static config for a dead NIC and reporting success is
the failure this avoids), and its MAC must match, since interface names can
renumber across kernels but MACs do not.
Requires nothing from the operator beyond racking the box. The wired NIC has a
distinct MAC from the Wi-Fi one, so the post-move address and switch port are
both discoverable from the UDM rather than needing to be relayed.
NVIDIA GB10, aarch64, 121 GB unified, sm_121. Ships booting to graphical.target
with GDM and GNOME Remote Desktop running.
playbooks/gx10-headless.yaml sets multi-user.target, stops gnome-remote-desktop,
masks the sleep/suspend/hibernate targets, makes logind ignore lid and idle, and
adds sshd keepalives so a stalled link does not kill a long-running job.
Two things learned the hard way and recorded in the playbook:
- gdm is a STATIC unit on Ubuntu, pulled in by display-manager.service and never
'enabled'. A guard of always skips, and a verify
written the same way passes while the desktop is still running. Both now test
is-active. The first run reported six green verifies having not stopped gdm.
- elway's --sudo applies only to ad-hoc --shell/--upload. Playbook steps run as
the connecting user and must carry their own sudo; connect as infra-ops.
The playbook refuses to stop the display manager while a seat session is held,
overridable with --var force_dm_stop=true. Networking is deliberately out of
scope: the box is on a desk on Wi-Fi with a temporary DHCP lease and no ethernet
carrier, and belongs to the rack-install change.
vh