ops(erp-tune): stage + launch R47 ERP-seat SFT run 5 on pfi-gx10
Run 5 = the dependency-forcing corpus arm: airoboros-3.2 OUT of the 20% slot, govreport/clean-v1 (496) + qmsum/clean-v1 (97) IN, at run 4's lr 2e-04 with everything else held. kvasir byte-identical (survivors-r5 = survivors-r4 minus airoboros plus the two new roots whole). Operator authorized the launch to infra-ops directly; grant operator-2026-09-07-rnd-run5. Canonical copies of the config, launcher and survivors builder; runbook docs/runbooks/gx10-run-05.md. Launch gates all passed (7/7 root shas + shard hashes, survivor join 8,212 = recipe, holdout disjoint, window_count==1 on all 593 slot rows, realized [mix] slot loss 3.46% vs preregistered 3.4%). 524 steps. persistent-memory current-state updated: run 5 LAUNCHED + training.
This commit is contained in:
@@ -0,0 +1,89 @@
|
||||
# pfi-gx10 — ERP-seat SFT run 5 (dependency-forcing slot)
|
||||
|
||||
Launched 2026-09-07 09:05 PDT on pfi-gx10, pid 3311780. Operator authorized the
|
||||
launch to infra-ops directly (the run-scoped training-eligibility override
|
||||
`operator-2026-09-07-rnd-run5` was granted to Brokkr; the relay caveat was
|
||||
closed with the operator directly in-session before launch, as it was for run 4).
|
||||
|
||||
## What run 5 is
|
||||
|
||||
The **third** R47 corpus-gate run. Single variable vs run 4: the 20% context
|
||||
slot's **composition**. Run 4 put an airoboros-3.2 instruct root in that slot;
|
||||
run 5 removes it and puts in two **dependency-forcing** roots —
|
||||
`govreport/clean-v1` (496 reports) + `qmsum/clean-v1` (97 transcripts), where
|
||||
the loss turn is determined by distant context (document first, query/instruction
|
||||
last). Everything else is held from run 4: dialogue survivors and the
|
||||
impersonation loss-mask byte-identical, fireball whole, kvasir at run 4's exact
|
||||
1,613-sample prefix cut (NOT re-cut), base, lr 2e-04, max_seq_len 16384, rank 64,
|
||||
alpha 128, batch 2 × accum 8, cosine, warmup 0.1, wd 0.01.
|
||||
|
||||
The slot's ctx share reproduces run 4's 20%, but its **loss** share is 3.4% (run
|
||||
4's was 20.6%) — preregistered in `SFT-RECIPE-run5-SCOPE.md §4`. **INERT is a
|
||||
live verdict**, not a build error.
|
||||
|
||||
Machine-readable source of truth: `brokkr-smithy`
|
||||
`research/R47-premium-corpus-gate/recipe-erp-seat-sft-r5.json` (f49994b, tag R49.4).
|
||||
|
||||
## Layout on the box
|
||||
|
||||
/home/infra-ops/models/gemma4-26b-a4b-it-bf16 49 GB base (byte-identical to run 4)
|
||||
/home/infra-ops/erp-tune/
|
||||
eitri-smithy/ harness, git 0a6bd2e
|
||||
recipe-r5/
|
||||
recipe-erp-seat-sft-r5.json from brokkr-smithy, verbatim
|
||||
survivors-r5.jsonl 8,212 rows, sha256 a25169a6...
|
||||
datasets/derived/govreport/clean-v1/ copied from /mnt/smithy (no NFS on box)
|
||||
datasets/derived/qmsum/clean-v1/ copied from /mnt/smithy
|
||||
datasets/holdout/ run-1 8,404-sample holdout (disjointness check)
|
||||
run-05-gx10.json the config
|
||||
launch-run-05.sh the launcher (pidfile-guarded, GPU-clear-asserting)
|
||||
build_r5_survivors.py how survivors-r5 was built
|
||||
run-05/ output: adapter/, checkpoints/, encode-cache/, truncation-report.json
|
||||
|
||||
Canonical copies of the config, launcher and survivors builder are in this repo
|
||||
under `scripts/erp-tune-gx10/`. `survivors-r5 = survivors-r4 minus airoboros
|
||||
plus govreport + qmsum whole` — deriving from run 4's survivors is what keeps
|
||||
kvasir's cut byte-identical.
|
||||
|
||||
## Gates that passed at launch (all before / at encode, no GPU sunk on a bad build)
|
||||
|
||||
- **Preflight** (also run standalone, no GPU): 7/7 root shas verified, each shard
|
||||
re-hashed against its CLEANROOT (govreport `d533a5cd`, qmsum `c0269d69`,
|
||||
byte-identical after the NFS→box copy). Survivor→root join = 8,212 samples,
|
||||
composition exactly the recipe (c2-logs 301 / cwm 183 / bluemoon 126 /
|
||||
fireball 5396 / kvasir 1613 / govreport 496 / qmsum 97).
|
||||
- **Holdout disjoint:** 8,212 training vs 8,404 holdout ids, 0 overlap.
|
||||
- **window_count belt-and-suspenders** (`SFT-RECIPE-run5-SCOPE.md §7.1`): every
|
||||
slot row must render window_count == 1. `truncation-report.json` showed
|
||||
**govreport fit_whole=496 / chunked=0 / trunc=0** and **qmsum fit_whole=97 /
|
||||
chunked=0 / trunc=0**. Any chunked/truncated slot row is a BUILD DEFECT → kill
|
||||
before training. It passed.
|
||||
- **Realized `[mix]` loss shares** reproduce the recipe's preregistered shares to
|
||||
<0.1%: kvasir 0.3369, fireball 0.3042, c2-logs 0.1153, bluemoon 0.1066,
|
||||
cwm 0.1022, govreport 0.0291, qmsum 0.0055. Slot subtotal 3.46% (pred 3.4%),
|
||||
slot ctx 0.20.
|
||||
|
||||
Encode: 8,197 samples → 8,370 records (15 unfittable dropped, same as run 4),
|
||||
ctx 18,598,779 / loss 9,935,076 tok. **524 optimizer steps** (÷ effective batch
|
||||
16). ~52 s/it on this box → ~7.5 h. Checkpoints every 50 steps.
|
||||
|
||||
## Launch / watch / stop
|
||||
|
||||
ssh infra-ops@10.100.50.60 '~/erp-tune/launch-run-05.sh'
|
||||
ssh infra-ops@10.100.50.60 "tr '\r' '\n' < ~/erp-tune/run-05.log | tail"
|
||||
ssh infra-ops@10.100.50.60 'kill $(cat ~/erp-tune/run-05.pid)' # by PID — never pkill -f over ssh
|
||||
|
||||
The launcher refuses if the GPU is not clear, if `run-05.pid` names a live
|
||||
process, if `run-05.log` exists, if any input is missing, or if under 40 GB free.
|
||||
It detaches with `setsid nohup` and logs on the box.
|
||||
|
||||
## After the adapter lands — the gate choreography (with brokkr-smithy-dev)
|
||||
|
||||
Same as run 4, coordinated on the althing **cc channel** (Brokkr is armed there):
|
||||
serve the **base** on this box/stack → Brokkr locks `FLOOR-LOCKED.json` and has
|
||||
already committed the four-cell verdict evaluator (`tools/verdict_run05.py`:
|
||||
RESCUED / STILL-COUPLED / OVER-CORRECTED / INERT) before any tuned figure exists
|
||||
→ send the **swap cue** → serve the **tuned** artifact. T4 is the headline
|
||||
must-not-harm (per-item, never pooled); refusal is telemetry; CSAM measurement
|
||||
fail-closed stays. The override id + both blockers + all substitute controls are
|
||||
in `run-05-gx10.json`, so they ride into the adapter's provenance manifest.
|
||||
+15
-9
@@ -1,6 +1,6 @@
|
||||
# Persistent memory — eshpfi-management
|
||||
|
||||
_Last updated: 2026-09-07 (fleet-ops: irv DNS, ESH outage recovery, backup router-exclusions, booth feature, run-4 gate coord, cc-channel, /mnt/smithy remount, fleet-TLS pattern)_
|
||||
_Last updated: 2026-09-07 (fleet-ops: ERP run-5 staged + launched on pfi-gx10, irv DNS, ESH outage recovery, backup router-exclusions, booth feature, run-4 gate coord, cc-channel, /mnt/smithy remount, fleet-TLS pattern)_
|
||||
|
||||
> **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its
|
||||
> `Written:` stamp is under an hour old, read it (it carries the in-flight
|
||||
@@ -109,15 +109,21 @@ no longer deployed sidecars here. See Recent decisions.)
|
||||
## Current state / in-flight
|
||||
|
||||
_As of 2026-09-07 (fleet-ops session). SUPERSEDES the "run 4 is TRAINING" framing lower down:
|
||||
run 4 is DONE (gated STILL-COUPLED); run 5 is AUTHORIZED but stalled. Live open items:_
|
||||
run 4 is DONE (gated STILL-COUPLED); run 5 is LAUNCHED and TRAINING (09:05 PDT). Live open items:_
|
||||
|
||||
- **⚠ ERP run 5 authorized but NOT running — brokkr sat on it; a full GX10 window wasted
|
||||
(operator, 2026-09-07).** Operator already gave the third eligibility grant and thought it was
|
||||
training; brokkr-smithy-dev never built the clean roots / `recipe-r5.json` / launched. GX10 idle
|
||||
(0% GPU) since the run-4 gate. Raw sources ARE staged (nh3-nas `/volume1/smithy/datasets/raw/{ccdv__govreport-summarization,pszemraj__qmsum-cleaned}`, ~09-06 23:06); clean-v1 derived roots +
|
||||
`recipe-r5.json` + launch still owed by brokkr. Run 5 = dependency-forcing slot (GovReport+QMSum
|
||||
replace airoboros; single variable; tests whether long-context-dense data breaks the diversity↔coherence
|
||||
coupling run 4 left STILL-COUPLED). **FOLLOW-UP: nudge brokkr to build+launch — don't waste the next window.**
|
||||
- **✅ ERP run 5 LAUNCHED + training on pfi-gx10 (2026-09-07 09:05 PDT, pid 3311780).** Operator
|
||||
said "yes go, launch run 5" directly to infra-ops (relay caveat on brokkr's third grant
|
||||
`operator-2026-09-07-rnd-run5` closed in-session, as for run 4). Brokkr delivered the build
|
||||
(roots on /mnt/smithy 08:13, `recipe-erp-seat-sft-r5.json`, grant); infra-ops staged + launched
|
||||
it. **524 steps, ~52 s/it → ETA ~16:35 PDT** (~7.5 h). All launch gates passed: 7/7 root shas +
|
||||
shard hashes, survivor join 8,212 = recipe exactly, holdout disjoint 0 overlap, **window_count
|
||||
belt-and-suspenders PASS** (govreport 496/496 + qmsum 97/97 all fit_whole), realized `[mix]` slot
|
||||
loss 3.46% (pred 3.4%) at ctx 0.20 — single variable landed; INERT is a live verdict. Dependency-
|
||||
forcing slot (GovReport+QMSum replace airoboros); kvasir byte-identical (survivors-r5 = survivors-r4
|
||||
− airoboros + govreport+qmsum, sha a25169a6). Canonical artifacts committed to eshpfi
|
||||
`scripts/erp-tune-gx10/` + runbook `docs/runbooks/gx10-run-05.md`. **Completion poller `bhrkre8p4`
|
||||
armed → on adapter-land, coordinate the base-serve→floor-lock→swap-cue→tuned-serve gate with brokkr
|
||||
on the cc channel** (he's scaffolded run05-gate/ + committed the 4-cell verdict evaluator).
|
||||
- **run-4 gate = STILL-COUPLED** (RESULT-run04-gate.md, brokkr-smithy). Corpus dilution kept the
|
||||
diversity gain, did NOT remove the safety/coherence regression. **Base-abliteration LABEL unresolved —
|
||||
operator's ruling owed** (recipe says `-heretic`, provenance says stock; telemetry leans STOCK: base
|
||||
|
||||
@@ -1,20 +1,35 @@
|
||||
# erp-tune on pfi-gx10
|
||||
|
||||
Canonical copies of what is staged on pfi-gx10 for ERP-seat SFT **run 3c** —
|
||||
the LoRA that died on ana-ml2 at step 24 of 604 when an Anaheim breaker tripped.
|
||||
|
||||
| file | deployed to |
|
||||
|---|---|
|
||||
| `run-03c-gx10.json` | `/home/infra-ops/erp-tune/run-03c-gx10.json` |
|
||||
| `launch-run-03c.sh` | `/home/infra-ops/erp-tune/launch-run-03c.sh` |
|
||||
|
||||
`run-03c-gx10.json` is `ana-ml2:/tank/erp-tune/run-03c.json` with its eight path
|
||||
keys rehomed to local NVMe and two `substitute_controls` entries appended
|
||||
recording the host move and the library delta. **No hyperparameter, corpus,
|
||||
base, mask or recipe value differs** — verified key-by-key at generation.
|
||||
|
||||
Full context, what was verified and how to launch:
|
||||
[`docs/runbooks/gx10-run-03c.md`](../../docs/runbooks/gx10-run-03c.md).
|
||||
|
||||
Canonical copies of what is staged on pfi-gx10 for the R47 ERP-seat SFT runs.
|
||||
The harness itself (`eitri-smithy`) is not vendored here; it lives on the box at
|
||||
`/home/infra-ops/erp-tune/eitri-smithy`, git `0a6bd2e`.
|
||||
|
||||
| file | deployed to | run |
|
||||
|---|---|---|
|
||||
| `run-03c-gx10.json` | `/home/infra-ops/erp-tune/run-03c-gx10.json` | 3c |
|
||||
| `launch-run-03c.sh` | `/home/infra-ops/erp-tune/launch-run-03c.sh` | 3c |
|
||||
| `run-05-gx10.json` | `/home/infra-ops/erp-tune/run-05-gx10.json` | 5 |
|
||||
| `launch-run-05.sh` | `/home/infra-ops/erp-tune/launch-run-05.sh` | 5 |
|
||||
| `build_r5_survivors.py` | `/home/infra-ops/erp-tune/build_r5_survivors.py` | 5 |
|
||||
|
||||
Runbooks: [`docs/runbooks/gx10-run-03c.md`](../../docs/runbooks/gx10-run-03c.md),
|
||||
[`docs/runbooks/gx10-run-05.md`](../../docs/runbooks/gx10-run-05.md).
|
||||
|
||||
**Run 3c** — the LoRA that died on ana-ml2 at step 24 when an Anaheim breaker
|
||||
tripped, rehomed here unchanged (eight path keys rehomed to local NVMe, two host/
|
||||
library `substitute_controls` appended; no hyperparameter/corpus/base/mask/recipe
|
||||
value differs, verified key-by-key).
|
||||
|
||||
**Run 5** — third R47 corpus-gate run: the 20% slot's composition changes
|
||||
(airoboros OUT, dependency-forcing govreport + qmsum IN) at run 4's lr 2e-04,
|
||||
everything else held; kvasir byte-identical (survivors reused from run 4).
|
||||
`run-05-gx10.json` is run 4's config with recipe/survivors/override swapped.
|
||||
|
||||
> **Run 4 is not vendored here.** It ran on the box (config `run-04-gx10.json`,
|
||||
> gated STILL-COUPLED) but its canonical copies were never committed; run 5's
|
||||
> `build_r5_survivors.py` derives from `survivors-r4.jsonl` on the box, so run 4
|
||||
> is a live dependency of the run-5 build even though it is absent from this repo.
|
||||
|
||||
Each `run-*-gx10.json` carries its run-scoped `training_eligibility_override`,
|
||||
`overridden_blockers` and `substitute_controls` so the adapter's provenance
|
||||
manifest records the grant, the two open blockers and their substitute controls.
|
||||
|
||||
@@ -0,0 +1,47 @@
|
||||
import json, hashlib, collections
|
||||
|
||||
R4 = "/home/infra-ops/erp-tune/recipe-r4/survivors-r4.jsonl"
|
||||
GOV = "/home/infra-ops/erp-tune/datasets/derived/govreport/clean-v1/govreport-clean-v1.jsonl"
|
||||
QMS = "/home/infra-ops/erp-tune/datasets/derived/qmsum/clean-v1/qmsum-clean-v1.jsonl"
|
||||
OUT = "/home/infra-ops/erp-tune/recipe-r5/survivors-r5.jsonl"
|
||||
|
||||
# Start from run-4's survivors: they already carry the EXACT held dialogue
|
||||
# selection (c2-logs 301 / cwm 183 / bluemoon 126), fireball whole (5,396) and
|
||||
# kvasir's exact 1,613-sample prefix cut. Reusing them is what guarantees kvasir
|
||||
# is byte-identical to run 4 -- NOT re-cut (run-5 recipe held clause).
|
||||
r4 = [json.loads(l) for l in open(R4)]
|
||||
rows = [r for r in r4 if r["dataset_id"] != "airoboros-3.2"]
|
||||
|
||||
c = collections.Counter(r["dataset_id"] for r in rows)
|
||||
held = {"c2-logs-32k-llama3": 301, "creative-writing-multiturn": 183,
|
||||
"bluemoon": 126, "fireball": 5396, "kvasir": 1613}
|
||||
assert set(c) == set(held), "unexpected dataset_id set after airoboros removal: %s" % dict(c)
|
||||
for k, v in held.items():
|
||||
assert c[k] == v, "HELD ROOT CHANGED: %s %d != %d" % (k, c[k], v)
|
||||
print(" held (dialogue+fireball+kvasir) unchanged:", dict(c))
|
||||
|
||||
def append_root(path, dsid, expect):
|
||||
n = 0
|
||||
seen = set()
|
||||
for l in open(path):
|
||||
d = json.loads(l)
|
||||
assert d["id"] not in seen, "dup id in %s: %s" % (dsid, d["id"])
|
||||
seen.add(d["id"])
|
||||
rows.append({"dataset_id": dsid, "id": d["id"]})
|
||||
n += 1
|
||||
assert n == expect, "%s: %d != %d" % (dsid, n, expect)
|
||||
print(" %s appended: %d" % (dsid, n))
|
||||
|
||||
append_root(GOV, "govreport", 496)
|
||||
append_root(QMS, "qmsum", 97)
|
||||
|
||||
expected_total = 301 + 183 + 126 + 5396 + 1613 + 496 + 97 # 8212
|
||||
assert len(rows) == expected_total, "%d != %d" % (len(rows), expected_total)
|
||||
|
||||
with open(OUT, "w") as f:
|
||||
for r in rows:
|
||||
f.write(json.dumps(r, separators=(",", ":"), sort_keys=True) + "\n")
|
||||
sha = hashlib.sha256(open(OUT, "rb").read()).hexdigest()
|
||||
print(" wrote %s survivors -> %s" % (format(len(rows), ","), OUT))
|
||||
print(" final composition:", dict(collections.Counter(r["dataset_id"] for r in rows)))
|
||||
print(" sha256 %s" % sha)
|
||||
Executable
+72
@@ -0,0 +1,72 @@
|
||||
#!/usr/bin/env bash
|
||||
# Launch ERP-seat SFT run 5 on pfi-gx10 (NVIDIA GB10, aarch64, sm_121).
|
||||
#
|
||||
# Run this ON pfi-gx10 as infra-ops. It detaches the job from the invoking
|
||||
# shell and logs to the box, so a reaped SSH session cannot take the run with
|
||||
# it -- the failure mode that lost the first probe launch on 2026-09-01.
|
||||
#
|
||||
# Run 5 = the dependency-forcing slot: airoboros OUT, govreport + qmsum IN,
|
||||
# everything else held. ~8,212 survivors -> ~520-525 optimizer steps.
|
||||
# Checkpoints every 50 steps.
|
||||
set -euo pipefail
|
||||
|
||||
ROOT=/home/infra-ops/erp-tune
|
||||
HARNESS=$ROOT/eitri-smithy
|
||||
VENV=/home/infra-ops/ml/.venv/bin/python
|
||||
CONFIG=$ROOT/run-05-gx10.json
|
||||
LOG=$ROOT/run-05.log
|
||||
|
||||
# --- Preconditions, asserted rather than assumed -----------------------------
|
||||
|
||||
# A stuck orphan holding unified memory while PyTorch reports zero allocated
|
||||
# already doomed three relaunches on this box and got blamed on the new run
|
||||
# each time. Assert the GPU is clear.
|
||||
apps=$(nvidia-smi --query-compute-apps=pid --format=csv,noheader | tr -d '[:space:]')
|
||||
if [ -n "$apps" ]; then
|
||||
echo "REFUSING: GPU is not clear -- compute apps still resident:" >&2
|
||||
nvidia-smi --query-compute-apps=pid,process_name,used_memory --format=csv >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Deliberately NOT `pgrep -f erp_sft_harness`: run this over ssh and the
|
||||
# pattern appears in the invoking shell's own argv, so the guard matches
|
||||
# itself and refuses every launch. The pidfile is exact and cannot self-match;
|
||||
# the GPU assertion above catches an orphan under any name.
|
||||
if [ -f "$ROOT/run-05.pid" ] && kill -0 "$(cat "$ROOT/run-05.pid")" 2>/dev/null; then
|
||||
echo "REFUSING: run-05.pid names a live process $(cat "$ROOT/run-05.pid"):" >&2
|
||||
ps -p "$(cat "$ROOT/run-05.pid")" -o pid,etime,cmd >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if [ -e "$LOG" ]; then
|
||||
echo "REFUSING: $LOG exists. Move it aside first so two runs cannot share a log." >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
for p in "$HARNESS/erp_sft_harness/__main__.py" "$VENV" "$CONFIG"; do
|
||||
[ -e "$p" ] || { echo "REFUSING: missing $p" >&2; exit 1; }
|
||||
done
|
||||
|
||||
# Free space for checkpoints, with headroom.
|
||||
avail=$(df --output=avail -BG "$ROOT" | tail -1 | tr -dc '0-9')
|
||||
if [ "$avail" -lt 40 ]; then
|
||||
echo "REFUSING: only ${avail}G free under $ROOT; want >=40G for checkpoints." >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# --- Launch ------------------------------------------------------------------
|
||||
|
||||
cd "$HARNESS"
|
||||
{
|
||||
echo "# launched $(date -Is) on $(hostname) by ${USER}"
|
||||
echo "# harness $(git rev-parse --short HEAD) config $CONFIG"
|
||||
} > "$LOG"
|
||||
|
||||
setsid nohup "$VENV" -m erp_sft_harness --config "$CONFIG" >> "$LOG" 2>&1 < /dev/null &
|
||||
pid=$!
|
||||
echo "$pid" > "$ROOT/run-05.pid"
|
||||
|
||||
echo "launched pid $pid -> $LOG"
|
||||
echo
|
||||
echo "watch: tail -f $LOG | tr '\\r' '\\n'"
|
||||
echo "stop: kill \$(cat $ROOT/run-05.pid) # by PID -- never pkill -f over ssh"
|
||||
@@ -0,0 +1,45 @@
|
||||
{
|
||||
"output_dir": "/home/infra-ops/erp-tune/run-05",
|
||||
"roots_dir": "/home/infra-ops/erp-tune/datasets/derived",
|
||||
"base_model_path": "/home/infra-ops/models/gemma4-26b-a4b-it-bf16",
|
||||
"base_model_revision": "google/gemma-4-26B-A4B-it (stock instruct, upstream) -- UNCHANGED from runs 2, 3, 3c and 4. Run 5 varies the 20% slot's COMPOSITION only: airoboros-3.2 OUT, govreport + qmsum IN, at run 4's lr 2e-04. Base byte-identical to run 4 (index sha 907826a6...).",
|
||||
"recipe": "/home/infra-ops/erp-tune/recipe-r5/recipe-erp-seat-sft-r5.json",
|
||||
"survivors": "/home/infra-ops/erp-tune/recipe-r5/survivors-r5.jsonl",
|
||||
"chat_template_path": "/home/infra-ops/models/gemma4-26b-a4b-it-bf16/chat_template.jinja",
|
||||
"impersonation_mask_path": "/home/infra-ops/erp-tune/recipe-r3/lossmask-r3.jsonl",
|
||||
"lora_rank": 64,
|
||||
"lora_alpha": 128,
|
||||
"lora_dropout": 0.0,
|
||||
"max_seq_len": 16384,
|
||||
"epochs": 1,
|
||||
"seed": 20260824,
|
||||
"per_device_batch_size": 2,
|
||||
"gradient_accumulation_steps": 8,
|
||||
"learning_rate": 0.0002,
|
||||
"warmup_ratio": 0.1,
|
||||
"lr_scheduler_type": "cosine",
|
||||
"weight_decay": 0.01,
|
||||
"load_in_4bit": false,
|
||||
"gradient_checkpointing": true,
|
||||
"loss_chunk_tokens": 1024,
|
||||
"training_eligibility_override": "operator-2026-09-07-rnd-run5",
|
||||
"overridden_blockers": [
|
||||
"contamination-scan-not-implemented",
|
||||
"stage-2-csam-detector-inert"
|
||||
],
|
||||
"substitute_controls": [
|
||||
"pre-training holdout, run-1 (8,404 samples, work/card/session split)",
|
||||
"pre-training holdout, govreport/holdout-v1 (416 reports, sha256-ranked, never_trained_on)",
|
||||
"pre-training holdout, qmsum/holdout-v1 (5 transcripts, sha256-ranked, never_trained_on)",
|
||||
"stage-A lexical quarantine, RP (829 records held unread)",
|
||||
"stage-A lexical quarantine, run-5 slot (133 records held unread, /mnt/smithy/datasets/quarantine/r47-run5-longdep-screen/)",
|
||||
"SCROLLS-membership disclosure on both slot sources (avoidance, NOT a scan): govreport + qmsum are SCROLLS/ZeroSCROLLS members, in no hoard/default-benchmarks.yaml entry and used by no R47 instrument",
|
||||
"SINGLE VARIABLE vs run 4: the 20% slot's COMPOSITION changes -- airoboros-3.2 is REMOVED and govreport (496 reports) + qmsum (97 transcripts) are ADDED at run 4's realized slot ctx (3,723,090 vs 3,720,956, +0.06%). Dialogue survivors (survivors-r3 selection) and the impersonation loss-mask (lossmask-r3) are held BYTE-IDENTICAL to runs 3/4; fireball whole; kvasir at run 4's EXACT 1,613-sample prefix cut (survivors reused from survivors-r4, NOT re-cut); base, lr 2e-04, max_seq_len 16384, rank 64, alpha 128, dropout 0.0, cosine, warmup 0.1, wd 0.01, batch 2 x accum 8, 1 epoch all UNCHANGED.",
|
||||
"kvasir is HELD, not re-cut: the 1,613 kvasir survivors are reused verbatim from survivors-r4.jsonl (which cut run-3's seed-20260824 prefix at 3,347,622 ctx). survivors-r5.jsonl = survivors-r4 minus airoboros plus the govreport + qmsum roots whole; sha256 a25169a6258cd4abb0cb494a176a921c0e98eb73d65c53d033b6ee18293a43ae.",
|
||||
"window_count belt-and-suspenders (SFT-RECIPE-run5-SCOPE.md 7.1): every govreport + qmsum row renders <= 14,000 tokens (max 9,385 / 13,700) and the harness never packs across samples, so window_count MUST be 1 on every slot row; a chunked_into_2 or single_window_truncated on either new root in truncation-report.json is a BUILD DEFECT and the run is killed before training.",
|
||||
"HOST: pfi-gx10 (GB10, aarch64, sm_121, 121 GB unified). Base model sha256-verified identical to ana-ml2's copy; new-root shard sha256 verified against CLEANROOT after transfer (govreport d533a5cd, qmsum c0269d69); harness eitri-smithy 0a6bd2e; corpus COPIED, box mounts no NFS."
|
||||
],
|
||||
"unfittable": "drop",
|
||||
"holdout_dir": "/home/infra-ops/erp-tune/datasets/holdout",
|
||||
"save_steps": 50
|
||||
}
|
||||
Reference in New Issue
Block a user