feat(pfi-gx10): stage ERP-seat SFT run 3c — verified, not launched
Rehome run 3c from ana-ml2 to pfi-gx10 unchanged: same corpus, base,
recipe and hyperparameters, different host. Slower (~13.3 h vs ~2.5 h)
and correct — an Anaheim breaker trip costs a 40-minute drive each way
and 13 hosts down, three of them SureFire client machines, while the
GX10 is a ~240 W appliance at NH3 that can take nothing else down.
Verified rather than assumed, because ana-ml2 ran transformers 5.15.1
on x86-64 and this box runs 5.16.1 on aarch64 — the silent
backend-delta class that has already voided conclusions here:
- both 49 GB base shards sha256-match ana-ml2's (size equality is a
weaker claim and was already true)
- a full encode was run into a throwaway dir and the encoded corpus
compared byte-for-byte: 197,360,233 B, sha256 c08bb1fe2ecb0be3,
identical. Every aggregate matched too. That verified artifact is
what the run will train on — it is seeded into run-03c/encode-cache
- the harness's own suite: 122 passed on aarch64
- the config generator asserts key-by-key that no non-path value
differs from run-03c.json
The encode-cache filename differs by design (base_model_path is part of
the key) — an input hash, not an output hash. Documented so it is not
misread as drift, or "fixed" by faking /tank on this box.
Corpus is copied to local NVMe; the box mounts no NFS. nh3-nas is now on
the same subnet, which makes mounting it tempting and still wrong under
a 13 h unattended run.
The launcher refuses on a live pidfile rather than a pgrep: `pgrep -f
erp_sft_harness` invoked over ssh matches the invoking shell's own argv.
That self-match cost a shell during staging.
Not launched. 13.3 h is the operator's call.
This commit is contained in:
@@ -0,0 +1,20 @@
|
||||
# erp-tune on pfi-gx10
|
||||
|
||||
Canonical copies of what is staged on pfi-gx10 for ERP-seat SFT **run 3c** —
|
||||
the LoRA that died on ana-ml2 at step 24 of 604 when an Anaheim breaker tripped.
|
||||
|
||||
| file | deployed to |
|
||||
|---|---|
|
||||
| `run-03c-gx10.json` | `/home/infra-ops/erp-tune/run-03c-gx10.json` |
|
||||
| `launch-run-03c.sh` | `/home/infra-ops/erp-tune/launch-run-03c.sh` |
|
||||
|
||||
`run-03c-gx10.json` is `ana-ml2:/tank/erp-tune/run-03c.json` with its eight path
|
||||
keys rehomed to local NVMe and two `substitute_controls` entries appended
|
||||
recording the host move and the library delta. **No hyperparameter, corpus,
|
||||
base, mask or recipe value differs** — verified key-by-key at generation.
|
||||
|
||||
Full context, what was verified and how to launch:
|
||||
[`docs/runbooks/gx10-run-03c.md`](../../docs/runbooks/gx10-run-03c.md).
|
||||
|
||||
The harness itself (`eitri-smithy`) is not vendored here; it lives on the box at
|
||||
`/home/infra-ops/erp-tune/eitri-smithy`, git `0a6bd2e`.
|
||||
Executable
+73
@@ -0,0 +1,73 @@
|
||||
#!/usr/bin/env bash
|
||||
# Launch ERP-seat SFT run 3c on pfi-gx10 (NVIDIA GB10, aarch64, sm_121).
|
||||
#
|
||||
# Run this ON pfi-gx10 as infra-ops. It detaches the job from the invoking
|
||||
# shell and logs to the box, so a reaped SSH session cannot take the run with
|
||||
# it -- the failure mode that lost the first probe launch on 2026-09-01.
|
||||
#
|
||||
# Expected: 604 optimizer steps at ~79.4 s/it => ~13.3 h.
|
||||
# Checkpoints every 50 steps, ~852 MB each (~10 GB total).
|
||||
set -euo pipefail
|
||||
|
||||
ROOT=/home/infra-ops/erp-tune
|
||||
HARNESS=$ROOT/eitri-smithy
|
||||
VENV=/home/infra-ops/ml/.venv/bin/python
|
||||
CONFIG=$ROOT/run-03c-gx10.json
|
||||
LOG=$ROOT/run-03c.log
|
||||
|
||||
# --- Preconditions, asserted rather than assumed -----------------------------
|
||||
|
||||
# A stuck orphan holding unified memory while PyTorch reports zero allocated
|
||||
# already doomed three relaunches on this box and got blamed on the new run
|
||||
# each time. Assert the GPU is clear.
|
||||
apps=$(nvidia-smi --query-compute-apps=pid --format=csv,noheader | tr -d '[:space:]')
|
||||
if [ -n "$apps" ]; then
|
||||
echo "REFUSING: GPU is not clear -- compute apps still resident:" >&2
|
||||
nvidia-smi --query-compute-apps=pid,process_name,used_memory --format=csv >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Deliberately NOT `pgrep -f erp_sft_harness`: run this over ssh and the
|
||||
# pattern appears in the invoking shell's own argv, so the guard matches
|
||||
# itself and refuses every launch. Same self-match that makes `pkill -f`
|
||||
# unsafe over ssh. The pidfile is exact and cannot self-match; the GPU
|
||||
# assertion above catches an orphan under any name.
|
||||
if [ -f "$ROOT/run-03c.pid" ] && kill -0 "$(cat "$ROOT/run-03c.pid")" 2>/dev/null; then
|
||||
echo "REFUSING: run-03c.pid names a live process $(cat "$ROOT/run-03c.pid"):" >&2
|
||||
ps -p "$(cat "$ROOT/run-03c.pid")" -o pid,etime,cmd >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if [ -e "$LOG" ]; then
|
||||
echo "REFUSING: $LOG exists. Move it aside first so two runs cannot share a log." >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
for p in "$HARNESS/erp_sft_harness/__main__.py" "$VENV" "$CONFIG"; do
|
||||
[ -e "$p" ] || { echo "REFUSING: missing $p" >&2; exit 1; }
|
||||
done
|
||||
|
||||
# Free space for checkpoints: 12 x 852 MB + final adapter, with headroom.
|
||||
avail=$(df --output=avail -BG "$ROOT" | tail -1 | tr -dc '0-9')
|
||||
if [ "$avail" -lt 40 ]; then
|
||||
echo "REFUSING: only ${avail}G free under $ROOT; want >=40G for checkpoints." >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# --- Launch ------------------------------------------------------------------
|
||||
|
||||
cd "$HARNESS"
|
||||
{
|
||||
echo "# launched $(date -Is) on $(hostname) by ${USER}"
|
||||
echo "# harness $(git rev-parse --short HEAD) config $CONFIG"
|
||||
} > "$LOG"
|
||||
|
||||
setsid nohup "$VENV" -m erp_sft_harness --config "$CONFIG" >> "$LOG" 2>&1 < /dev/null &
|
||||
pid=$!
|
||||
echo "$pid" > "$ROOT/run-03c.pid"
|
||||
|
||||
echo "launched pid $pid -> $LOG"
|
||||
echo
|
||||
echo "watch: tail -f $LOG | tr '\\r' '\\n'"
|
||||
echo "steps: grep -ao '[0-9]*/604 \[[^]]*\]' $LOG | tail -1"
|
||||
echo "stop: kill \$(cat $ROOT/run-03c.pid) # by PID -- never pkill -f over ssh"
|
||||
@@ -0,0 +1,47 @@
|
||||
{
|
||||
"output_dir": "/home/infra-ops/erp-tune/run-03c",
|
||||
"roots_dir": "/home/infra-ops/erp-tune/datasets/derived",
|
||||
"base_model_path": "/home/infra-ops/models/gemma4-26b-a4b-it-bf16",
|
||||
"base_model_revision": "google/gemma-4-26B-A4B-it (stock instruct, upstream) -- UNCHANGED from runs 2 and 3; run 3c varies the LEARNING RATE alone",
|
||||
"recipe": "/home/infra-ops/erp-tune/recipe-r3/recipe-erp-seat-sft-r3.json",
|
||||
"survivors": "/home/infra-ops/erp-tune/recipe-r3/survivors-r3.jsonl",
|
||||
"chat_template_path": "/home/infra-ops/models/gemma4-26b-a4b-it-bf16/chat_template.jinja",
|
||||
"impersonation_mask_path": "/home/infra-ops/erp-tune/recipe-r3/lossmask-r3.jsonl",
|
||||
"lora_rank": 64,
|
||||
"lora_alpha": 128,
|
||||
"lora_dropout": 0.0,
|
||||
"max_seq_len": 16384,
|
||||
"epochs": 1,
|
||||
"seed": 20260824,
|
||||
"per_device_batch_size": 2,
|
||||
"gradient_accumulation_steps": 8,
|
||||
"learning_rate": 1e-05,
|
||||
"warmup_ratio": 0.1,
|
||||
"lr_scheduler_type": "cosine",
|
||||
"weight_decay": 0.01,
|
||||
"load_in_4bit": false,
|
||||
"gradient_checkpointing": true,
|
||||
"loss_chunk_tokens": 1024,
|
||||
"training_eligibility_override": "operator-2026-08-26-run3-go",
|
||||
"overridden_blockers": [
|
||||
"contamination-scan-not-implemented",
|
||||
"stage-2-csam-detector-inert"
|
||||
],
|
||||
"substitute_controls": [
|
||||
"pre-training holdout (8,002 samples, work/card/session split)",
|
||||
"stage-A lexical quarantine, RP only (829 records held)",
|
||||
"F1 pippa root excision (target-list change, recipe erp-seat-sft-r3)",
|
||||
"F2/F3/F4 low-quality loss-mask, 5,867 turns (length floor 250w, register cap 20 asterisks/1kw, placeholder leak) -- MASKED not deleted",
|
||||
"impersonation loss-mask unioned into the same file, 16 turns not covered by F2/F3/F4; 5,883 masked turns total, sha-asserted",
|
||||
"kvasir subset to the recipe's stated 38.0% context share (3,498 of 12,003, seeded shuffle at run seed) -- the recipe names the share but stages no selection; see recipe-r3/RUN-03-BUILD-NOTE.md",
|
||||
"DEVIATION RECORDED: recipe token accounting is not the harness tokenizer; bluemoon is NOT the largest loss contributor as trained (32.9% vs c2-logs 35.6%) -- see RUN-03-BUILD-NOTE.md section 4",
|
||||
"SINGLE VARIABLE vs run 3: learning_rate 0.0002 -> 1e-05 (20x cut). Corpus BYTE-IDENTICAL (survivors-r3.jsonl, lossmask-r3.jsonl, same shas); base, max_seq_len 16384, rank 64, alpha 128, dropout 0.0, cosine, warmup 0.1, wd 0.01, batch 2, 1 epoch and target_modules ALL UNCHANGED. Rationale: the zerofata MeroMero v1 card, on this exact base, warns \"Google left the instruct model at the razor's edge of overfitting; finetune it at all and it feels like it'll rapidly lose intelligence\" -- and MeroMero v1 trains at 1e-5. Run 3 used 20x that.",
|
||||
"OPERATIONAL DEVIATION (not scientific): save_steps 100 -> 50. Checkpoint cadence only; it changes nothing about the optimisation, the corpus or the single-variable lr claim above. Recorded as a separate entry so the two deviations cannot be confused. Rationale: an Anaheim site power event on 2026-08-27 killed this run's first launch at step 80 of 604 with save_steps=100, so no checkpoint existed and it restarted from zero.",
|
||||
"relaunch_reason: Anaheim site power event 2026-08-27T01:2x UTC, confirmed by a 1-minute uptime on the host's return. First launch reached step 80 of 604 with save_steps=100, so NO checkpoint existed and this run starts from zero. Configured identically apart from save_steps; NOT a resumption.",
|
||||
"HOST MOVE (operational, not scientific): this run executes on pfi-gx10 (NVIDIA GB10, aarch64, sm_121, 121 GB unified) instead of ana-ml2. Paths are rehomed to local NVMe; NO hyperparameter, corpus, base, mask or recipe value differs from run-03c.json. Base model verified sha256-identical to ana-ml2's copy (both shards, 2026-09-03). Rationale: an Anaheim breaker trip already killed this run once and costs a 40-minute drive plus 13 hosts down, three of them SureFire client machines; the GX10 is a ~240 W appliance at NH3 that can take nothing else down. Measured cost: ~79.4 s/it vs ana-ml2's 10.8-15.8, so 604 steps is ~13.3 h instead of ~2.5 h.",
|
||||
"LIBRARY DELTA (recorded, not assumed harmless): torch 2.13.0 -> 2.14.0+cu130, transformers 5.15.1 -> 5.16.1, triton 3.7.1 -> 3.8.0; peft 0.20.0 and accelerate 1.14.0 unchanged. Attention backend flex_attention on both sides. Tokenisation equivalence across the transformers delta is asserted by comparing this host's encoded corpus against ana-ml2's byte-for-byte before launch, not inferred."
|
||||
],
|
||||
"unfittable": "drop",
|
||||
"holdout_dir": "/home/infra-ops/erp-tune/datasets/holdout",
|
||||
"save_steps": 50
|
||||
}
|
||||
Reference in New Issue
Block a user