vh
dae77ee118
Rehome run 3c from ana-ml2 to pfi-gx10 unchanged: same corpus, base,
recipe and hyperparameters, different host. Slower (~13.3 h vs ~2.5 h)
and correct — an Anaheim breaker trip costs a 40-minute drive each way
and 13 hosts down, three of them SureFire client machines, while the
GX10 is a ~240 W appliance at NH3 that can take nothing else down.
Verified rather than assumed, because ana-ml2 ran transformers 5.15.1
on x86-64 and this box runs 5.16.1 on aarch64 — the silent
backend-delta class that has already voided conclusions here:
- both 49 GB base shards sha256-match ana-ml2's (size equality is a
weaker claim and was already true)
- a full encode was run into a throwaway dir and the encoded corpus
compared byte-for-byte: 197,360,233 B, sha256 c08bb1fe2ecb0be3,
identical. Every aggregate matched too. That verified artifact is
what the run will train on — it is seeded into run-03c/encode-cache
- the harness's own suite: 122 passed on aarch64
- the config generator asserts key-by-key that no non-path value
differs from run-03c.json
The encode-cache filename differs by design (base_model_path is part of
the key) — an input hash, not an output hash. Documented so it is not
misread as drift, or "fixed" by faking /tank on this box.
Corpus is copied to local NVMe; the box mounts no NFS. nh3-nas is now on
the same subnet, which makes mounting it tempting and still wrong under
a 13 h unattended run.
The launcher refuses on a live pidfile rather than a pgrep: `pgrep -f
erp_sft_harness` invoked over ssh matches the invoking shell's own argv.
That self-match cost a shell during staging.
Not launched. 13.3 h is the operator's call.
5.4 KiB
[2026-09-03] Run 3c STAGED on pfi-gx10 — verified end to end, deliberately NOT launched
The ERP-seat SFT LoRA that died on ana-ml2 at step 24 of 604 to an Anaheim breaker trip is now staged on pfi-gx10, unchanged. The launch is the operator's call and was not taken — he stood this port down once before, so a 13.3 h commitment is not an agent default.
Runbook docs/runbooks/gx10-run-03c.md; canonical config + launcher
scripts/erp-tune-gx10/; on the box /home/infra-ops/erp-tune/.
ssh infra-ops@10.100.50.60 '~/erp-tune/launch-run-03c.sh'
What is on the box
~/models/gemma4-26b-a4b-it-bf16 49 GB base, ALREADY THERE from the 09-01 probe
~/erp-tune/eitri-smithy harness, git 0a6bd2e, tracked tree clean
~/erp-tune/recipe-r3 recipe / survivors / loss-mask
~/erp-tune/datasets/{derived,holdout} 2.4 GB, COPIED (50 s at 49 MB/s from nh3-dev)
~/erp-tune/run-03c/encode-cache PRE-SEEDED with the verified encode
~/ml/.venv + protobuf, pytest (the only two gaps vs ana-ml2)
⚠ The corpus is copied and the box mounts NO NFS. /mnt/smithy lives on nh3-nas, now on
the same subnet as the racked GX10 — which makes mounting it tempting and still wrong. A
13 h unattended run is the worst place for a hard NFS dependency
(incident_esh_docker_nfs_boot_race). 2.4 GB copies in under a minute; there is nothing to
buy.
The verification that actually mattered — and it was NOT free reasoning
ana-ml2 ran transformers 5.15.1 / torch 2.13.0 on x86-64. The GX10 runs 5.16.1 / 2.14.0+cu130 on aarch64. That is precisely the silent backend-delta class CLAUDE.md records as having voided two frontier-panel conclusions. So it was measured: a full encode was run into a throwaway output dir and the encoded corpus compared byte-for-byte.
ana-ml2 encoded-c16316f1c1bb21da.jsonl 197,360,233 B sha256 c08bb1fe2ecb0be3...
pfi-gx10 encoded-fd8fe1944fb316b2.jsonl 197,360,233 B sha256 c08bb1fe2ecb0be3...
Byte-identical. Every aggregate matched too: 9,504 vs 8,404 ids / 0 overlap, 15 unfittable dropped, 9,662 records, ctx 18,600,057 / loss 13,310,930 tok, five mix shares to 4 dp.
⚠ The cache-key FILENAMES differ and that is correct, not drift. base_model_path is in
the encode-cache key by design (so a different base cannot silently reuse an encode), and
rehoming the base changes the key while leaving content identical. The key is an input hash;
the sha is the output. Do not read the differing filenames as a mismatch — and do not
"fix" it by symlinking /tank/aimodels onto this box to force a key match. That verified
artifact was then copied into run-03c/encode-cache/, so the run trains on the exact bytes
compared and will report [encode] cache hit.
Also verified rather than assumed: both 49 GB base shards sha256-match ana-ml2's (size equality was already true and is not the same claim), the harness's own suite is 122 passed on aarch64, and every one of the config's 8 path keys resolves to an existing local file.
The config is provably the same run
run-03c-gx10.json = ana-ml2's run-03c.json with 8 path keys rehomed and 2
substitute_controls entries appended (host move; library delta). A generator asserted
key-by-key that no non-path value differs rather than eyeballing a diff — lr 1e-05, rank 64,
alpha 128, seq 16384, batch 2 x accum 8, save_steps 50, seed 20260824 all intact, and the
existing 10 substitute_controls are a byte-identical prefix.
⚠ I TRIPPED THE pkill SELF-MATCH AGAIN, ~20 MINUTES AFTER READING THE MEMORY ABOUT IT
ssh gx10 'pkill -f "erp_sft_harness --config .../encode-check.json"' — the pattern is in the
remote shell's OWN argv, so it killed my shell alongside the target and the command returned
nothing. feedback_pkill_ssh_self_match describes this exactly. Reading the memory did not
prevent it; the guard has to be in the artifact, not in recall.
So the launcher's already-running guard is a pidfile, not a pgrep — pgrep -f erp_sft_harness in a script invoked over ssh matches the invoking shell and would refuse every
launch. Same root cause, and it would have presented as a mysterious always-refusing launcher.
The launcher's other guards, each bought with a past failure
GPU-clear assertion a stuck orphan held 80 GB while PyTorch reported 0 allocated;
every relaunch was doomed and blamed the NEW run
setsid nohup + on-box log a foreground ssh reaped the 09-01 probe: work survived, output did not
log-exists refusal two runs must not share a log
>=40 GB free 12 checkpoints x 852 MB (measured off run-03, not estimated)
Why the slow box is still the right box (unchanged, restated because it is the whole case)
~79.4 s/it here vs 10.8-15.8 on ana-ml2 -> 13.3 h vs ~2.5 h. An Anaheim breaker trip is not
priced in lost steps: it is a 40-minute drive each way on the operator's time, 13 hosts
down including pbs-ana and three SureFire client machines. Nothing is waiting on this run,
so the slowness is close to free.
NOT verified — the honest gap
The harness's train loop has not run end to end on sm_121. The 79.4 s/it baseline used a
synthetic replica of the geometry, and the staging encode was killed before the weight load.
If it breaks, it breaks in the first two minutes after the [sampler] line — roughly three
minutes after launch, well before the first checkpoint at ~66 min.