feat(pfi-gx10): stage ERP-seat SFT run 3c — verified, not launched

Rehome run 3c from ana-ml2 to pfi-gx10 unchanged: same corpus, base,
recipe and hyperparameters, different host. Slower (~13.3 h vs ~2.5 h)
and correct — an Anaheim breaker trip costs a 40-minute drive each way
and 13 hosts down, three of them SureFire client machines, while the
GX10 is a ~240 W appliance at NH3 that can take nothing else down.

Verified rather than assumed, because ana-ml2 ran transformers 5.15.1
on x86-64 and this box runs 5.16.1 on aarch64 — the silent
backend-delta class that has already voided conclusions here:

  - both 49 GB base shards sha256-match ana-ml2's (size equality is a
    weaker claim and was already true)
  - a full encode was run into a throwaway dir and the encoded corpus
    compared byte-for-byte: 197,360,233 B, sha256 c08bb1fe2ecb0be3,
    identical. Every aggregate matched too. That verified artifact is
    what the run will train on — it is seeded into run-03c/encode-cache
  - the harness's own suite: 122 passed on aarch64
  - the config generator asserts key-by-key that no non-path value
    differs from run-03c.json

The encode-cache filename differs by design (base_model_path is part of
the key) — an input hash, not an output hash. Documented so it is not
misread as drift, or "fixed" by faking /tank on this box.

Corpus is copied to local NVMe; the box mounts no NFS. nh3-nas is now on
the same subnet, which makes mounting it tempting and still wrong under
a 13 h unattended run.

The launcher refuses on a live pidfile rather than a pgrep: `pgrep -f
erp_sft_harness` invoked over ssh matches the invoking shell's own argv.
That self-match cost a shell during staging.

Not launched. 13.3 h is the operator's call.

docs/runbooks/gx10-run-03c.md

@@ -0,0 +1,115 @@
+# pfi-gx10 — ERP-seat SFT run 3c, staged and ready to launch
+
+Staged 2026-09-03. **Not launched** — the launch is the operator's call.
+
+Run 3c is the LoRA that died on ana-ml2 at step 24 of 604 when an Anaheim
+breaker tripped. It is rehomed to pfi-gx10 unchanged: same corpus, same base,
+same hyperparameters, different host.
+
+## Why this box and not the fast one
+
+ana-ml2 finishes this run in ~2.5 h and pfi-gx10 needs ~13.3 h, and the slow box
+is still correct. An Anaheim power event is not priced in lost training steps —
+it is a 40-minute drive each way on the operator's time, with 13 hosts down
+including `pbs-ana` (the fleet's primary backup server) and **three SureFire
+client machines** under a hosting agreement. The GX10 is a ~240 W appliance at
+NH3 that can take nothing else down, and nothing is waiting on this run.
+
+Power-capping ana-ml2 is not a third option: both RTX PRO 6000s floor at 250 W
+against a 300 W default, so the whole lever is 100 W on a ~1 kW box.
+
+## Layout on the box
+
+    /home/infra-ops/models/gemma4-26b-a4b-it-bf16   49 GB base (sha-verified, see below)
+    /home/infra-ops/erp-tune/
+      eitri-smithy/            harness, git 0a6bd2e, tracked tree clean
+      recipe-r3/               recipe, survivors, loss-mask
+      datasets/derived/        2.4 GB corpus  -- COPIED, see the NFS note
+      datasets/holdout/        holdout, for the disjointness assertion
+      run-03c/encode-cache/    pre-seeded with the VERIFIED encode (below)
+      run-03c-gx10.json        the config
+      launch-run-03c.sh        the launcher
+    /home/infra-ops/ml/.venv   torch 2.14.0+cu130, transformers 5.16.1, peft 0.20.0
+
+Canonical copies of the config and launcher live in this repo under
+`scripts/erp-tune-gx10/`.
+
+## ⚠ The corpus is COPIED, and the box mounts no NFS at all
+
+`/mnt/smithy` is deliberately **not** mounted here. A 13-hour unattended run is
+the worst possible place for a hard NFS dependency — the fleet has already lost
+a host to a hard mount wedging into unkillable D-state. The corpus is 2.4 GB and
+copied in 50 s from nh3-dev's mount; there is nothing to gain by mounting it.
+
+Verify before every launch: `mount | grep -E 'nfs|cifs'` returns nothing.
+
+## What was verified, rather than assumed
+
+**The base model is byte-identical to ana-ml2's.** Both shards sha256-matched
+across 51.6 GB, not just size-checked:
+
+    model-00001-of-00002.safetensors  1127684971bbca40465435a5cad69d67ad603bf5e61c6dfd5561fae4a3bcfdb3
+    model-00002-of-00002.safetensors  aab47033e1e8a492ef8e581efae1cf36478d0433567e7729b3c1728bc8970db7
+
+**The tokenizer did not drift across the library delta.** ana-ml2 ran
+transformers 5.15.1 / torch 2.13.0 on x86-64; this box runs 5.16.1 / 2.14.0+cu130
+on aarch64. That is exactly the silent backend-delta class that has already
+voided conclusions on this project, so it was measured, not reasoned about: a
+full encode was run here into a throwaway output dir and the encoded corpus
+compared byte-for-byte against ana-ml2's.
+
+    ana-ml2   encoded-c16316f1c1bb21da.jsonl   197,360,233 B   sha256 c08bb1fe2ecb0be3…
+    pfi-gx10  encoded-fd8fe1944fb316b2.jsonl   197,360,233 B   sha256 c08bb1fe2ecb0be3…
+
+Identical. Every aggregate matched too — 9,504 training vs 8,404 holdout ids with
+0 overlap, 15 unfittable dropped, 9,662 records, ctx 18,600,057 / loss 13,310,930
+tokens, and the five mix shares to four decimals.
+
+⚠ **The cache-key filenames differ and that is correct, not a mismatch.**
+`base_model_path` is part of the encode-cache key by design, so rehoming the base
+changes the key while leaving the content identical. The key is an input hash;
+the sha above is the output. Do not read the differing filenames as drift.
+
+That verified artifact was then copied into `run-03c/encode-cache/`, so the run
+trains on the exact bytes that were compared — it will report `[encode] cache
+hit` rather than re-encoding.
+
+**The harness runs on aarch64.** Its own suite: 122 passed.
+
+**Throughput.** ~79.4 s/it measured on this exact geometry (median of 10 timed
+steps, 0.19% spread), peak 75.1 of 121.6 GiB. 604 steps ≈ 13.3 h. Against
+ana-ml2's 10.8–15.8 s/it that is ~6× slower where raw compute predicts ~2.7×;
+the unconfirmed hypothesis is unified-LPDDR5X bandwidth, not a misconfiguration.
+
+## Launch
+
+    ssh infra-ops@10.100.50.60 '~/erp-tune/launch-run-03c.sh'
+
+The script refuses rather than proceeds if the GPU is not clear, if
+`run-03c.pid` names a live process, if the log already exists, if any input is
+missing, or if there is under 40 GB free. It detaches with `setsid nohup` and
+logs on the box, because a long remote job piped through a foreground SSH has
+already been reaped once on this hardware — the work survived, the output did not.
+
+Watching, and stopping:
+
+    tail -f ~/erp-tune/run-03c.log | tr '\r' '\n'
+    kill $(cat ~/erp-tune/run-03c.pid)
+
+⚠ **Never `pkill -f erp_sft_harness` over SSH.** The pattern appears in your own
+ssh argv, so it kills your shell as well as the run. This bit again during
+staging. Kill by PID. The launcher's own guard uses the pidfile for the same
+reason.
+
+## What to expect in the log
+
+Order is preflight → encode (cache hit) → mix report → weight load (~2 min) →
+sampler → train. Checkpoints land every 50 steps at ~852 MB each, so ~10 GB over
+the run against 764 GB free. First checkpoint ≈ 66 minutes in.
+
+## Not verified
+
+The harness's train loop has not been exercised end to end on this silicon — the
+throughput baseline used a synthetic replica of the geometry, and the staging
+encode was killed before the weight load. If something is going to break, it
+breaks in the first two minutes after the sampler line.