vh
dae77ee118
Rehome run 3c from ana-ml2 to pfi-gx10 unchanged: same corpus, base,
recipe and hyperparameters, different host. Slower (~13.3 h vs ~2.5 h)
and correct — an Anaheim breaker trip costs a 40-minute drive each way
and 13 hosts down, three of them SureFire client machines, while the
GX10 is a ~240 W appliance at NH3 that can take nothing else down.
Verified rather than assumed, because ana-ml2 ran transformers 5.15.1
on x86-64 and this box runs 5.16.1 on aarch64 — the silent
backend-delta class that has already voided conclusions here:
- both 49 GB base shards sha256-match ana-ml2's (size equality is a
weaker claim and was already true)
- a full encode was run into a throwaway dir and the encoded corpus
compared byte-for-byte: 197,360,233 B, sha256 c08bb1fe2ecb0be3,
identical. Every aggregate matched too. That verified artifact is
what the run will train on — it is seeded into run-03c/encode-cache
- the harness's own suite: 122 passed on aarch64
- the config generator asserts key-by-key that no non-path value
differs from run-03c.json
The encode-cache filename differs by design (base_model_path is part of
the key) — an input hash, not an output hash. Documented so it is not
misread as drift, or "fixed" by faking /tank on this box.
Corpus is copied to local NVMe; the box mounts no NFS. nh3-nas is now on
the same subnet, which makes mounting it tempting and still wrong under
a 13 h unattended run.
The launcher refuses on a live pidfile rather than a pgrep: `pgrep -f
erp_sft_harness` invoked over ssh matches the invoking shell's own argv.
That self-match cost a shell during staging.
Not launched. 13.3 h is the operator's call.
5.5 KiB
pfi-gx10 — ERP-seat SFT run 3c, staged and ready to launch
Staged 2026-09-03. Not launched — the launch is the operator's call.
Run 3c is the LoRA that died on ana-ml2 at step 24 of 604 when an Anaheim breaker tripped. It is rehomed to pfi-gx10 unchanged: same corpus, same base, same hyperparameters, different host.
Why this box and not the fast one
ana-ml2 finishes this run in ~2.5 h and pfi-gx10 needs ~13.3 h, and the slow box
is still correct. An Anaheim power event is not priced in lost training steps —
it is a 40-minute drive each way on the operator's time, with 13 hosts down
including pbs-ana (the fleet's primary backup server) and three SureFire
client machines under a hosting agreement. The GX10 is a ~240 W appliance at
NH3 that can take nothing else down, and nothing is waiting on this run.
Power-capping ana-ml2 is not a third option: both RTX PRO 6000s floor at 250 W against a 300 W default, so the whole lever is 100 W on a ~1 kW box.
Layout on the box
/home/infra-ops/models/gemma4-26b-a4b-it-bf16 49 GB base (sha-verified, see below)
/home/infra-ops/erp-tune/
eitri-smithy/ harness, git 0a6bd2e, tracked tree clean
recipe-r3/ recipe, survivors, loss-mask
datasets/derived/ 2.4 GB corpus -- COPIED, see the NFS note
datasets/holdout/ holdout, for the disjointness assertion
run-03c/encode-cache/ pre-seeded with the VERIFIED encode (below)
run-03c-gx10.json the config
launch-run-03c.sh the launcher
/home/infra-ops/ml/.venv torch 2.14.0+cu130, transformers 5.16.1, peft 0.20.0
Canonical copies of the config and launcher live in this repo under
scripts/erp-tune-gx10/.
⚠ The corpus is COPIED, and the box mounts no NFS at all
/mnt/smithy is deliberately not mounted here. A 13-hour unattended run is
the worst possible place for a hard NFS dependency — the fleet has already lost
a host to a hard mount wedging into unkillable D-state. The corpus is 2.4 GB and
copied in 50 s from nh3-dev's mount; there is nothing to gain by mounting it.
Verify before every launch: mount | grep -E 'nfs|cifs' returns nothing.
What was verified, rather than assumed
The base model is byte-identical to ana-ml2's. Both shards sha256-matched across 51.6 GB, not just size-checked:
model-00001-of-00002.safetensors 1127684971bbca40465435a5cad69d67ad603bf5e61c6dfd5561fae4a3bcfdb3
model-00002-of-00002.safetensors aab47033e1e8a492ef8e581efae1cf36478d0433567e7729b3c1728bc8970db7
The tokenizer did not drift across the library delta. ana-ml2 ran transformers 5.15.1 / torch 2.13.0 on x86-64; this box runs 5.16.1 / 2.14.0+cu130 on aarch64. That is exactly the silent backend-delta class that has already voided conclusions on this project, so it was measured, not reasoned about: a full encode was run here into a throwaway output dir and the encoded corpus compared byte-for-byte against ana-ml2's.
ana-ml2 encoded-c16316f1c1bb21da.jsonl 197,360,233 B sha256 c08bb1fe2ecb0be3…
pfi-gx10 encoded-fd8fe1944fb316b2.jsonl 197,360,233 B sha256 c08bb1fe2ecb0be3…
Identical. Every aggregate matched too — 9,504 training vs 8,404 holdout ids with 0 overlap, 15 unfittable dropped, 9,662 records, ctx 18,600,057 / loss 13,310,930 tokens, and the five mix shares to four decimals.
⚠ The cache-key filenames differ and that is correct, not a mismatch.
base_model_path is part of the encode-cache key by design, so rehoming the base
changes the key while leaving the content identical. The key is an input hash;
the sha above is the output. Do not read the differing filenames as drift.
That verified artifact was then copied into run-03c/encode-cache/, so the run
trains on the exact bytes that were compared — it will report [encode] cache hit rather than re-encoding.
The harness runs on aarch64. Its own suite: 122 passed.
Throughput. ~79.4 s/it measured on this exact geometry (median of 10 timed steps, 0.19% spread), peak 75.1 of 121.6 GiB. 604 steps ≈ 13.3 h. Against ana-ml2's 10.8–15.8 s/it that is ~6× slower where raw compute predicts ~2.7×; the unconfirmed hypothesis is unified-LPDDR5X bandwidth, not a misconfiguration.
Launch
ssh infra-ops@10.100.50.60 '~/erp-tune/launch-run-03c.sh'
The script refuses rather than proceeds if the GPU is not clear, if
run-03c.pid names a live process, if the log already exists, if any input is
missing, or if there is under 40 GB free. It detaches with setsid nohup and
logs on the box, because a long remote job piped through a foreground SSH has
already been reaped once on this hardware — the work survived, the output did not.
Watching, and stopping:
tail -f ~/erp-tune/run-03c.log | tr '\r' '\n'
kill $(cat ~/erp-tune/run-03c.pid)
⚠ Never pkill -f erp_sft_harness over SSH. The pattern appears in your own
ssh argv, so it kills your shell as well as the run. This bit again during
staging. Kill by PID. The launcher's own guard uses the pidfile for the same
reason.
What to expect in the log
Order is preflight → encode (cache hit) → mix report → weight load (~2 min) → sampler → train. Checkpoints land every 50 steps at ~852 MB each, so ~10 GB over the run against 764 GB free. First checkpoint ≈ 66 minutes in.
Not verified
The harness's train loop has not been exercised end to end on this silicon — the throughput baseline used a synthetic replica of the geometry, and the staging encode was killed before the weight load. If something is going to break, it breaks in the first two minutes after the sampler line.