# `[2026-09-03]` Run 3c STAGED on pfi-gx10 — verified end to end, deliberately NOT launched

The ERP-seat SFT LoRA that died on ana-ml2 at step 24 of 604 to an Anaheim breaker trip is
now staged on pfi-gx10, unchanged. **The launch is the operator's call and was not taken** —
he stood this port down once before, so a 13.3 h commitment is not an agent default.

Runbook `docs/runbooks/gx10-run-03c.md`; canonical config + launcher
`scripts/erp-tune-gx10/`; on the box `/home/infra-ops/erp-tune/`.

    ssh infra-ops@10.100.50.60 '~/erp-tune/launch-run-03c.sh'

## What is on the box

    ~/models/gemma4-26b-a4b-it-bf16   49 GB base, ALREADY THERE from the 09-01 probe
    ~/erp-tune/eitri-smithy           harness, git 0a6bd2e, tracked tree clean
    ~/erp-tune/recipe-r3              recipe / survivors / loss-mask
    ~/erp-tune/datasets/{derived,holdout}   2.4 GB, COPIED (50 s at 49 MB/s from nh3-dev)
    ~/erp-tune/run-03c/encode-cache   PRE-SEEDED with the verified encode
    ~/ml/.venv                        + protobuf, pytest (the only two gaps vs ana-ml2)

⚠ **The corpus is copied and the box mounts NO NFS.** `/mnt/smithy` lives on nh3-nas, now on
the *same subnet* as the racked GX10 — which makes mounting it tempting and still wrong. A
13 h unattended run is the worst place for a hard NFS dependency
([[incident_esh_docker_nfs_boot_race]]). 2.4 GB copies in under a minute; there is nothing to
buy.

## The verification that actually mattered — and it was NOT free reasoning

ana-ml2 ran transformers 5.15.1 / torch 2.13.0 on x86-64. The GX10 runs 5.16.1 / 2.14.0+cu130
on aarch64. That is precisely the silent backend-delta class CLAUDE.md records as having voided
two frontier-panel conclusions. So it was **measured**: a full encode was run into a throwaway
output dir and the encoded corpus compared byte-for-byte.

    ana-ml2   encoded-c16316f1c1bb21da.jsonl   197,360,233 B   sha256 c08bb1fe2ecb0be3...
    pfi-gx10  encoded-fd8fe1944fb316b2.jsonl   197,360,233 B   sha256 c08bb1fe2ecb0be3...

**Byte-identical.** Every aggregate matched too: 9,504 vs 8,404 ids / 0 overlap, 15 unfittable
dropped, 9,662 records, ctx 18,600,057 / loss 13,310,930 tok, five mix shares to 4 dp.

⚠ **The cache-key FILENAMES differ and that is correct, not drift.** `base_model_path` is in
the encode-cache key *by design* (so a different base cannot silently reuse an encode), and
rehoming the base changes the key while leaving content identical. **The key is an input hash;
the sha is the output.** Do not read the differing filenames as a mismatch — and do not
"fix" it by symlinking `/tank/aimodels` onto this box to force a key match. That verified
artifact was then copied into `run-03c/encode-cache/`, so the run trains on the exact bytes
compared and will report `[encode] cache hit`.

Also verified rather than assumed: **both 49 GB base shards sha256-match ana-ml2's** (size
equality was already true and is not the same claim), the harness's own suite is **122 passed**
on aarch64, and every one of the config's 8 path keys resolves to an existing local file.

## The config is provably the same run

`run-03c-gx10.json` = ana-ml2's `run-03c.json` with 8 path keys rehomed and 2
`substitute_controls` entries appended (host move; library delta). A generator asserted
**key-by-key that no non-path value differs** rather than eyeballing a diff — lr 1e-05, rank 64,
alpha 128, seq 16384, batch 2 x accum 8, save_steps 50, seed 20260824 all intact, and the
existing 10 substitute_controls are a byte-identical prefix.

## ⚠ I TRIPPED THE pkill SELF-MATCH AGAIN, ~20 MINUTES AFTER READING THE MEMORY ABOUT IT

`ssh gx10 'pkill -f "erp_sft_harness --config .../encode-check.json"'` — the pattern is in the
remote shell's OWN argv, so it killed my shell alongside the target and the command returned
nothing. [[feedback_pkill_ssh_self_match]] describes this exactly. Reading the memory did not
prevent it; **the guard has to be in the artifact, not in recall.**

So the launcher's already-running guard is a **pidfile**, not a pgrep — `pgrep -f
erp_sft_harness` in a script invoked over ssh matches the invoking shell and would refuse every
launch. Same root cause, and it would have presented as a mysterious always-refusing launcher.

## The launcher's other guards, each bought with a past failure

    GPU-clear assertion      a stuck orphan held 80 GB while PyTorch reported 0 allocated;
                             every relaunch was doomed and blamed the NEW run
    setsid nohup + on-box log a foreground ssh reaped the 09-01 probe: work survived, output did not
    log-exists refusal       two runs must not share a log
    >=40 GB free             12 checkpoints x 852 MB (measured off run-03, not estimated)

## Why the slow box is still the right box (unchanged, restated because it is the whole case)

~79.4 s/it here vs 10.8-15.8 on ana-ml2 -> 13.3 h vs ~2.5 h. An Anaheim breaker trip is not
priced in lost steps: it is a 40-minute drive **each way** on the operator's time, 13 hosts
down including `pbs-ana` and **three SureFire client machines**. Nothing is waiting on this run,
so the slowness is close to free.

## NOT verified — the honest gap

The harness's **train loop** has not run end to end on sm_121. The 79.4 s/it baseline used a
synthetic replica of the geometry, and the staging encode was killed before the weight load.
If it breaks, it breaks in the first two minutes after the `[sampler]` line — roughly three
minutes after launch, well before the first checkpoint at ~66 min.