feat(pfi-gx10): stage ERP-seat SFT run 3c — verified, not launched
Rehome run 3c from ana-ml2 to pfi-gx10 unchanged: same corpus, base,
recipe and hyperparameters, different host. Slower (~13.3 h vs ~2.5 h)
and correct — an Anaheim breaker trip costs a 40-minute drive each way
and 13 hosts down, three of them SureFire client machines, while the
GX10 is a ~240 W appliance at NH3 that can take nothing else down.
Verified rather than assumed, because ana-ml2 ran transformers 5.15.1
on x86-64 and this box runs 5.16.1 on aarch64 — the silent
backend-delta class that has already voided conclusions here:
- both 49 GB base shards sha256-match ana-ml2's (size equality is a
weaker claim and was already true)
- a full encode was run into a throwaway dir and the encoded corpus
compared byte-for-byte: 197,360,233 B, sha256 c08bb1fe2ecb0be3,
identical. Every aggregate matched too. That verified artifact is
what the run will train on — it is seeded into run-03c/encode-cache
- the harness's own suite: 122 passed on aarch64
- the config generator asserts key-by-key that no non-path value
differs from run-03c.json
The encode-cache filename differs by design (base_model_path is part of
the key) — an input hash, not an output hash. Documented so it is not
misread as drift, or "fixed" by faking /tank on this box.
Corpus is copied to local NVMe; the box mounts no NFS. nh3-nas is now on
the same subnet, which makes mounting it tempting and still wrong under
a 13 h unattended run.
The launcher refuses on a live pidfile rather than a pgrep: `pgrep -f
erp_sft_harness` invoked over ssh matches the invoking shell's own argv.
That self-match cost a shell during staging.
Not launched. 13.3 h is the operator's call.
This commit is contained in:
+35
-22
@@ -11,33 +11,25 @@ Grace-Blackwell desktop supercomputer. Registered 2026-09-01.
|
||||
| Kernel | 6.17.0-1031-nvidia |
|
||||
| Hostname | `pfi-gx10` (shipped with static `gx10-a745`, corrected) |
|
||||
|
||||
## ⚠ The address in `ssh-target` is TEMPORARY
|
||||
## Network — racked, and single-path
|
||||
|
||||
As of 2026-09-01 this box is **on a desk, on Wi-Fi**, holding a DHCP lease at
|
||||
`10.100.10.226` on `nh3-userland` (VLAN 10). Ethernet `enP7s7` has no carrier.
|
||||
Racked 2026-09-03. `pfi-gx10.nh3.internal` → **10.100.50.60**, wired only on
|
||||
`enP7s7`, VLAN 50 (`nh3-servers`), UniFi switch port 22.
|
||||
|
||||
It is going into the rack later. **Target settled (operator, 2026-09-01):
|
||||
`nh3-servers`, VLAN 50, static `10.100.50.60`** — clear of `.40` nh3-docker,
|
||||
`.42` nh3-extdev, `.50` nh3-nas, `.90` pbs-nh3, and below the `.150` DHCP pool
|
||||
where fleet statics live.
|
||||
**The address lives on the switch side, not the host** — a DHCP *reservation*
|
||||
against the wired MAC `30:c5:99:3d:a7:45`, with the host left on DHCP. Operator
|
||||
ruling: a reservation moves with the box, a netplan static goes stale the moment
|
||||
it does.
|
||||
|
||||
Nothing was configured on the desk — an address that is about to be wrong is
|
||||
worse than DHCP. The move is `playbooks/gx10-rack-network.yaml`.
|
||||
⚠ **Wi-Fi is deliberately off and there is now exactly ONE path in.** If the
|
||||
switch port or the reservation breaks, this is a rack visit. Correct end state
|
||||
for a racked server, but it is a posture change from the desk setup.
|
||||
|
||||
**Nothing is needed from the operator beyond racking it.** The wired NIC has its
|
||||
own MAC (`30:c5:99:3d:a7:45`, distinct from the Wi-Fi `50:bb:b5:a2:00:a8`), so
|
||||
its post-move address is discoverable from the UDM without being told:
|
||||
Full detail, including the order that made the move safe:
|
||||
[`docs/runbooks/gx10-rack-network.md`](../../docs/runbooks/gx10-rack-network.md).
|
||||
|
||||
```bash
|
||||
curl -sk "https://10.100.0.1/proxy/network/api/s/default/stat/sta" \
|
||||
-H "X-API-KEY: $(secret get unifi/pfi-udmse-api-key)" \
|
||||
| python3 -c "import json,sys;[print(c['ip'],c.get('sw_port')) for c in json.load(sys.stdin)['data'] if c['mac']=='30:c5:99:3d:a7:45']"
|
||||
```
|
||||
|
||||
That also returns the switch port, which must be set to the `nh3-servers`
|
||||
network or the box lands back on VLAN 10.
|
||||
|
||||
Expect ~65–80 ms RTT until it is wired. That is Wi-Fi power-save, not a fault.
|
||||
⚠ **The box mounts no NFS, on purpose.** Working data is copied to local NVMe —
|
||||
see the training section below.
|
||||
|
||||
## Access
|
||||
|
||||
@@ -62,6 +54,27 @@ trivially while the desktop is still running.
|
||||
The playbook will not stop GDM while someone holds a seat session. Override with
|
||||
`--var force_dm_stop=true`, or just let the rack-install reboot handle it.
|
||||
|
||||
## Training — run 3c is staged and ready
|
||||
|
||||
The ERP-seat SFT LoRA (run 3c) that died on ana-ml2 at step 24 of 604 to an
|
||||
Anaheim breaker trip is staged here, unchanged, and **not launched** — that call
|
||||
is the operator's.
|
||||
|
||||
ssh infra-ops@10.100.50.60 '~/erp-tune/launch-run-03c.sh'
|
||||
|
||||
~79.4 s/it measured on this geometry → 604 steps ≈ 13.3 h, peak 75.1 of
|
||||
121.6 GiB. Slower than ana-ml2's ~2.5 h and still the right host: this is a
|
||||
~240 W appliance at NH3 that cannot take a client's machines dark.
|
||||
|
||||
Base model and the encoded corpus were both sha256-verified identical to
|
||||
ana-ml2's, so the library delta (transformers 5.15.1 → 5.16.1, x86-64 → aarch64)
|
||||
is measured to be inert rather than assumed harmless. Runbook:
|
||||
[`docs/runbooks/gx10-run-03c.md`](../../docs/runbooks/gx10-run-03c.md); canonical
|
||||
config + launcher in [`scripts/erp-tune-gx10/`](../../scripts/erp-tune-gx10/).
|
||||
|
||||
⚠ **Never `pkill -f erp_sft_harness` over SSH** — the pattern is in your own ssh
|
||||
argv and you kill your shell with it. Kill by PID from `~/erp-tune/run-03c.pid`.
|
||||
|
||||
## Relevance to Flash-Next
|
||||
|
||||
`sm_121`, not `sm_120`. The SGLang fork evaluated for ana-ml2 (henge item 49)
|
||||
|
||||
Reference in New Issue
Block a user