ops(erp-tune): stage + launch R47 ERP-seat SFT run 5 on pfi-gx10
Run 5 = the dependency-forcing corpus arm: airoboros-3.2 OUT of the 20% slot, govreport/clean-v1 (496) + qmsum/clean-v1 (97) IN, at run 4's lr 2e-04 with everything else held. kvasir byte-identical (survivors-r5 = survivors-r4 minus airoboros plus the two new roots whole). Operator authorized the launch to infra-ops directly; grant operator-2026-09-07-rnd-run5. Canonical copies of the config, launcher and survivors builder; runbook docs/runbooks/gx10-run-05.md. Launch gates all passed (7/7 root shas + shard hashes, survivor join 8,212 = recipe, holdout disjoint, window_count==1 on all 593 slot rows, realized [mix] slot loss 3.46% vs preregistered 3.4%). 524 steps. persistent-memory current-state updated: run 5 LAUNCHED + training.
This commit is contained in:
@@ -1,20 +1,35 @@
|
||||
# erp-tune on pfi-gx10
|
||||
|
||||
Canonical copies of what is staged on pfi-gx10 for ERP-seat SFT **run 3c** —
|
||||
the LoRA that died on ana-ml2 at step 24 of 604 when an Anaheim breaker tripped.
|
||||
|
||||
| file | deployed to |
|
||||
|---|---|
|
||||
| `run-03c-gx10.json` | `/home/infra-ops/erp-tune/run-03c-gx10.json` |
|
||||
| `launch-run-03c.sh` | `/home/infra-ops/erp-tune/launch-run-03c.sh` |
|
||||
|
||||
`run-03c-gx10.json` is `ana-ml2:/tank/erp-tune/run-03c.json` with its eight path
|
||||
keys rehomed to local NVMe and two `substitute_controls` entries appended
|
||||
recording the host move and the library delta. **No hyperparameter, corpus,
|
||||
base, mask or recipe value differs** — verified key-by-key at generation.
|
||||
|
||||
Full context, what was verified and how to launch:
|
||||
[`docs/runbooks/gx10-run-03c.md`](../../docs/runbooks/gx10-run-03c.md).
|
||||
|
||||
Canonical copies of what is staged on pfi-gx10 for the R47 ERP-seat SFT runs.
|
||||
The harness itself (`eitri-smithy`) is not vendored here; it lives on the box at
|
||||
`/home/infra-ops/erp-tune/eitri-smithy`, git `0a6bd2e`.
|
||||
|
||||
| file | deployed to | run |
|
||||
|---|---|---|
|
||||
| `run-03c-gx10.json` | `/home/infra-ops/erp-tune/run-03c-gx10.json` | 3c |
|
||||
| `launch-run-03c.sh` | `/home/infra-ops/erp-tune/launch-run-03c.sh` | 3c |
|
||||
| `run-05-gx10.json` | `/home/infra-ops/erp-tune/run-05-gx10.json` | 5 |
|
||||
| `launch-run-05.sh` | `/home/infra-ops/erp-tune/launch-run-05.sh` | 5 |
|
||||
| `build_r5_survivors.py` | `/home/infra-ops/erp-tune/build_r5_survivors.py` | 5 |
|
||||
|
||||
Runbooks: [`docs/runbooks/gx10-run-03c.md`](../../docs/runbooks/gx10-run-03c.md),
|
||||
[`docs/runbooks/gx10-run-05.md`](../../docs/runbooks/gx10-run-05.md).
|
||||
|
||||
**Run 3c** — the LoRA that died on ana-ml2 at step 24 when an Anaheim breaker
|
||||
tripped, rehomed here unchanged (eight path keys rehomed to local NVMe, two host/
|
||||
library `substitute_controls` appended; no hyperparameter/corpus/base/mask/recipe
|
||||
value differs, verified key-by-key).
|
||||
|
||||
**Run 5** — third R47 corpus-gate run: the 20% slot's composition changes
|
||||
(airoboros OUT, dependency-forcing govreport + qmsum IN) at run 4's lr 2e-04,
|
||||
everything else held; kvasir byte-identical (survivors reused from run 4).
|
||||
`run-05-gx10.json` is run 4's config with recipe/survivors/override swapped.
|
||||
|
||||
> **Run 4 is not vendored here.** It ran on the box (config `run-04-gx10.json`,
|
||||
> gated STILL-COUPLED) but its canonical copies were never committed; run 5's
|
||||
> `build_r5_survivors.py` derives from `survivors-r4.jsonl` on the box, so run 4
|
||||
> is a live dependency of the run-5 build even though it is absent from this repo.
|
||||
|
||||
Each `run-*-gx10.json` carries its run-scoped `training_eligibility_override`,
|
||||
`overridden_blockers` and `substitute_controls` so the adapter's provenance
|
||||
manifest records the grant, the two open blockers and their substitute controls.
|
||||
|
||||
@@ -0,0 +1,47 @@
|
||||
import json, hashlib, collections
|
||||
|
||||
R4 = "/home/infra-ops/erp-tune/recipe-r4/survivors-r4.jsonl"
|
||||
GOV = "/home/infra-ops/erp-tune/datasets/derived/govreport/clean-v1/govreport-clean-v1.jsonl"
|
||||
QMS = "/home/infra-ops/erp-tune/datasets/derived/qmsum/clean-v1/qmsum-clean-v1.jsonl"
|
||||
OUT = "/home/infra-ops/erp-tune/recipe-r5/survivors-r5.jsonl"
|
||||
|
||||
# Start from run-4's survivors: they already carry the EXACT held dialogue
|
||||
# selection (c2-logs 301 / cwm 183 / bluemoon 126), fireball whole (5,396) and
|
||||
# kvasir's exact 1,613-sample prefix cut. Reusing them is what guarantees kvasir
|
||||
# is byte-identical to run 4 -- NOT re-cut (run-5 recipe held clause).
|
||||
r4 = [json.loads(l) for l in open(R4)]
|
||||
rows = [r for r in r4 if r["dataset_id"] != "airoboros-3.2"]
|
||||
|
||||
c = collections.Counter(r["dataset_id"] for r in rows)
|
||||
held = {"c2-logs-32k-llama3": 301, "creative-writing-multiturn": 183,
|
||||
"bluemoon": 126, "fireball": 5396, "kvasir": 1613}
|
||||
assert set(c) == set(held), "unexpected dataset_id set after airoboros removal: %s" % dict(c)
|
||||
for k, v in held.items():
|
||||
assert c[k] == v, "HELD ROOT CHANGED: %s %d != %d" % (k, c[k], v)
|
||||
print(" held (dialogue+fireball+kvasir) unchanged:", dict(c))
|
||||
|
||||
def append_root(path, dsid, expect):
|
||||
n = 0
|
||||
seen = set()
|
||||
for l in open(path):
|
||||
d = json.loads(l)
|
||||
assert d["id"] not in seen, "dup id in %s: %s" % (dsid, d["id"])
|
||||
seen.add(d["id"])
|
||||
rows.append({"dataset_id": dsid, "id": d["id"]})
|
||||
n += 1
|
||||
assert n == expect, "%s: %d != %d" % (dsid, n, expect)
|
||||
print(" %s appended: %d" % (dsid, n))
|
||||
|
||||
append_root(GOV, "govreport", 496)
|
||||
append_root(QMS, "qmsum", 97)
|
||||
|
||||
expected_total = 301 + 183 + 126 + 5396 + 1613 + 496 + 97 # 8212
|
||||
assert len(rows) == expected_total, "%d != %d" % (len(rows), expected_total)
|
||||
|
||||
with open(OUT, "w") as f:
|
||||
for r in rows:
|
||||
f.write(json.dumps(r, separators=(",", ":"), sort_keys=True) + "\n")
|
||||
sha = hashlib.sha256(open(OUT, "rb").read()).hexdigest()
|
||||
print(" wrote %s survivors -> %s" % (format(len(rows), ","), OUT))
|
||||
print(" final composition:", dict(collections.Counter(r["dataset_id"] for r in rows)))
|
||||
print(" sha256 %s" % sha)
|
||||
Executable
+72
@@ -0,0 +1,72 @@
|
||||
#!/usr/bin/env bash
|
||||
# Launch ERP-seat SFT run 5 on pfi-gx10 (NVIDIA GB10, aarch64, sm_121).
|
||||
#
|
||||
# Run this ON pfi-gx10 as infra-ops. It detaches the job from the invoking
|
||||
# shell and logs to the box, so a reaped SSH session cannot take the run with
|
||||
# it -- the failure mode that lost the first probe launch on 2026-09-01.
|
||||
#
|
||||
# Run 5 = the dependency-forcing slot: airoboros OUT, govreport + qmsum IN,
|
||||
# everything else held. ~8,212 survivors -> ~520-525 optimizer steps.
|
||||
# Checkpoints every 50 steps.
|
||||
set -euo pipefail
|
||||
|
||||
ROOT=/home/infra-ops/erp-tune
|
||||
HARNESS=$ROOT/eitri-smithy
|
||||
VENV=/home/infra-ops/ml/.venv/bin/python
|
||||
CONFIG=$ROOT/run-05-gx10.json
|
||||
LOG=$ROOT/run-05.log
|
||||
|
||||
# --- Preconditions, asserted rather than assumed -----------------------------
|
||||
|
||||
# A stuck orphan holding unified memory while PyTorch reports zero allocated
|
||||
# already doomed three relaunches on this box and got blamed on the new run
|
||||
# each time. Assert the GPU is clear.
|
||||
apps=$(nvidia-smi --query-compute-apps=pid --format=csv,noheader | tr -d '[:space:]')
|
||||
if [ -n "$apps" ]; then
|
||||
echo "REFUSING: GPU is not clear -- compute apps still resident:" >&2
|
||||
nvidia-smi --query-compute-apps=pid,process_name,used_memory --format=csv >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Deliberately NOT `pgrep -f erp_sft_harness`: run this over ssh and the
|
||||
# pattern appears in the invoking shell's own argv, so the guard matches
|
||||
# itself and refuses every launch. The pidfile is exact and cannot self-match;
|
||||
# the GPU assertion above catches an orphan under any name.
|
||||
if [ -f "$ROOT/run-05.pid" ] && kill -0 "$(cat "$ROOT/run-05.pid")" 2>/dev/null; then
|
||||
echo "REFUSING: run-05.pid names a live process $(cat "$ROOT/run-05.pid"):" >&2
|
||||
ps -p "$(cat "$ROOT/run-05.pid")" -o pid,etime,cmd >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if [ -e "$LOG" ]; then
|
||||
echo "REFUSING: $LOG exists. Move it aside first so two runs cannot share a log." >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
for p in "$HARNESS/erp_sft_harness/__main__.py" "$VENV" "$CONFIG"; do
|
||||
[ -e "$p" ] || { echo "REFUSING: missing $p" >&2; exit 1; }
|
||||
done
|
||||
|
||||
# Free space for checkpoints, with headroom.
|
||||
avail=$(df --output=avail -BG "$ROOT" | tail -1 | tr -dc '0-9')
|
||||
if [ "$avail" -lt 40 ]; then
|
||||
echo "REFUSING: only ${avail}G free under $ROOT; want >=40G for checkpoints." >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# --- Launch ------------------------------------------------------------------
|
||||
|
||||
cd "$HARNESS"
|
||||
{
|
||||
echo "# launched $(date -Is) on $(hostname) by ${USER}"
|
||||
echo "# harness $(git rev-parse --short HEAD) config $CONFIG"
|
||||
} > "$LOG"
|
||||
|
||||
setsid nohup "$VENV" -m erp_sft_harness --config "$CONFIG" >> "$LOG" 2>&1 < /dev/null &
|
||||
pid=$!
|
||||
echo "$pid" > "$ROOT/run-05.pid"
|
||||
|
||||
echo "launched pid $pid -> $LOG"
|
||||
echo
|
||||
echo "watch: tail -f $LOG | tr '\\r' '\\n'"
|
||||
echo "stop: kill \$(cat $ROOT/run-05.pid) # by PID -- never pkill -f over ssh"
|
||||
@@ -0,0 +1,45 @@
|
||||
{
|
||||
"output_dir": "/home/infra-ops/erp-tune/run-05",
|
||||
"roots_dir": "/home/infra-ops/erp-tune/datasets/derived",
|
||||
"base_model_path": "/home/infra-ops/models/gemma4-26b-a4b-it-bf16",
|
||||
"base_model_revision": "google/gemma-4-26B-A4B-it (stock instruct, upstream) -- UNCHANGED from runs 2, 3, 3c and 4. Run 5 varies the 20% slot's COMPOSITION only: airoboros-3.2 OUT, govreport + qmsum IN, at run 4's lr 2e-04. Base byte-identical to run 4 (index sha 907826a6...).",
|
||||
"recipe": "/home/infra-ops/erp-tune/recipe-r5/recipe-erp-seat-sft-r5.json",
|
||||
"survivors": "/home/infra-ops/erp-tune/recipe-r5/survivors-r5.jsonl",
|
||||
"chat_template_path": "/home/infra-ops/models/gemma4-26b-a4b-it-bf16/chat_template.jinja",
|
||||
"impersonation_mask_path": "/home/infra-ops/erp-tune/recipe-r3/lossmask-r3.jsonl",
|
||||
"lora_rank": 64,
|
||||
"lora_alpha": 128,
|
||||
"lora_dropout": 0.0,
|
||||
"max_seq_len": 16384,
|
||||
"epochs": 1,
|
||||
"seed": 20260824,
|
||||
"per_device_batch_size": 2,
|
||||
"gradient_accumulation_steps": 8,
|
||||
"learning_rate": 0.0002,
|
||||
"warmup_ratio": 0.1,
|
||||
"lr_scheduler_type": "cosine",
|
||||
"weight_decay": 0.01,
|
||||
"load_in_4bit": false,
|
||||
"gradient_checkpointing": true,
|
||||
"loss_chunk_tokens": 1024,
|
||||
"training_eligibility_override": "operator-2026-09-07-rnd-run5",
|
||||
"overridden_blockers": [
|
||||
"contamination-scan-not-implemented",
|
||||
"stage-2-csam-detector-inert"
|
||||
],
|
||||
"substitute_controls": [
|
||||
"pre-training holdout, run-1 (8,404 samples, work/card/session split)",
|
||||
"pre-training holdout, govreport/holdout-v1 (416 reports, sha256-ranked, never_trained_on)",
|
||||
"pre-training holdout, qmsum/holdout-v1 (5 transcripts, sha256-ranked, never_trained_on)",
|
||||
"stage-A lexical quarantine, RP (829 records held unread)",
|
||||
"stage-A lexical quarantine, run-5 slot (133 records held unread, /mnt/smithy/datasets/quarantine/r47-run5-longdep-screen/)",
|
||||
"SCROLLS-membership disclosure on both slot sources (avoidance, NOT a scan): govreport + qmsum are SCROLLS/ZeroSCROLLS members, in no hoard/default-benchmarks.yaml entry and used by no R47 instrument",
|
||||
"SINGLE VARIABLE vs run 4: the 20% slot's COMPOSITION changes -- airoboros-3.2 is REMOVED and govreport (496 reports) + qmsum (97 transcripts) are ADDED at run 4's realized slot ctx (3,723,090 vs 3,720,956, +0.06%). Dialogue survivors (survivors-r3 selection) and the impersonation loss-mask (lossmask-r3) are held BYTE-IDENTICAL to runs 3/4; fireball whole; kvasir at run 4's EXACT 1,613-sample prefix cut (survivors reused from survivors-r4, NOT re-cut); base, lr 2e-04, max_seq_len 16384, rank 64, alpha 128, dropout 0.0, cosine, warmup 0.1, wd 0.01, batch 2 x accum 8, 1 epoch all UNCHANGED.",
|
||||
"kvasir is HELD, not re-cut: the 1,613 kvasir survivors are reused verbatim from survivors-r4.jsonl (which cut run-3's seed-20260824 prefix at 3,347,622 ctx). survivors-r5.jsonl = survivors-r4 minus airoboros plus the govreport + qmsum roots whole; sha256 a25169a6258cd4abb0cb494a176a921c0e98eb73d65c53d033b6ee18293a43ae.",
|
||||
"window_count belt-and-suspenders (SFT-RECIPE-run5-SCOPE.md 7.1): every govreport + qmsum row renders <= 14,000 tokens (max 9,385 / 13,700) and the harness never packs across samples, so window_count MUST be 1 on every slot row; a chunked_into_2 or single_window_truncated on either new root in truncation-report.json is a BUILD DEFECT and the run is killed before training.",
|
||||
"HOST: pfi-gx10 (GB10, aarch64, sm_121, 121 GB unified). Base model sha256-verified identical to ana-ml2's copy; new-root shard sha256 verified against CLEANROOT after transfer (govreport d533a5cd, qmsum c0269d69); harness eitri-smithy 0a6bd2e; corpus COPIED, box mounts no NFS."
|
||||
],
|
||||
"unfittable": "drop",
|
||||
"holdout_dir": "/home/infra-ops/erp-tune/datasets/holdout",
|
||||
"save_steps": 50
|
||||
}
|
||||
Reference in New Issue
Block a user