vh
265357efb2
Run 5 = the dependency-forcing corpus arm: airoboros-3.2 OUT of the 20% slot, govreport/clean-v1 (496) + qmsum/clean-v1 (97) IN, at run 4's lr 2e-04 with everything else held. kvasir byte-identical (survivors-r5 = survivors-r4 minus airoboros plus the two new roots whole). Operator authorized the launch to infra-ops directly; grant operator-2026-09-07-rnd-run5. Canonical copies of the config, launcher and survivors builder; runbook docs/runbooks/gx10-run-05.md. Launch gates all passed (7/7 root shas + shard hashes, survivor join 8,212 = recipe, holdout disjoint, window_count==1 on all 593 slot rows, realized [mix] slot loss 3.46% vs preregistered 3.4%). 524 steps. persistent-memory current-state updated: run 5 LAUNCHED + training.
90 lines
5.0 KiB
Markdown
90 lines
5.0 KiB
Markdown
# pfi-gx10 — ERP-seat SFT run 5 (dependency-forcing slot)
|
||
|
||
Launched 2026-09-07 09:05 PDT on pfi-gx10, pid 3311780. Operator authorized the
|
||
launch to infra-ops directly (the run-scoped training-eligibility override
|
||
`operator-2026-09-07-rnd-run5` was granted to Brokkr; the relay caveat was
|
||
closed with the operator directly in-session before launch, as it was for run 4).
|
||
|
||
## What run 5 is
|
||
|
||
The **third** R47 corpus-gate run. Single variable vs run 4: the 20% context
|
||
slot's **composition**. Run 4 put an airoboros-3.2 instruct root in that slot;
|
||
run 5 removes it and puts in two **dependency-forcing** roots —
|
||
`govreport/clean-v1` (496 reports) + `qmsum/clean-v1` (97 transcripts), where
|
||
the loss turn is determined by distant context (document first, query/instruction
|
||
last). Everything else is held from run 4: dialogue survivors and the
|
||
impersonation loss-mask byte-identical, fireball whole, kvasir at run 4's exact
|
||
1,613-sample prefix cut (NOT re-cut), base, lr 2e-04, max_seq_len 16384, rank 64,
|
||
alpha 128, batch 2 × accum 8, cosine, warmup 0.1, wd 0.01.
|
||
|
||
The slot's ctx share reproduces run 4's 20%, but its **loss** share is 3.4% (run
|
||
4's was 20.6%) — preregistered in `SFT-RECIPE-run5-SCOPE.md §4`. **INERT is a
|
||
live verdict**, not a build error.
|
||
|
||
Machine-readable source of truth: `brokkr-smithy`
|
||
`research/R47-premium-corpus-gate/recipe-erp-seat-sft-r5.json` (f49994b, tag R49.4).
|
||
|
||
## Layout on the box
|
||
|
||
/home/infra-ops/models/gemma4-26b-a4b-it-bf16 49 GB base (byte-identical to run 4)
|
||
/home/infra-ops/erp-tune/
|
||
eitri-smithy/ harness, git 0a6bd2e
|
||
recipe-r5/
|
||
recipe-erp-seat-sft-r5.json from brokkr-smithy, verbatim
|
||
survivors-r5.jsonl 8,212 rows, sha256 a25169a6...
|
||
datasets/derived/govreport/clean-v1/ copied from /mnt/smithy (no NFS on box)
|
||
datasets/derived/qmsum/clean-v1/ copied from /mnt/smithy
|
||
datasets/holdout/ run-1 8,404-sample holdout (disjointness check)
|
||
run-05-gx10.json the config
|
||
launch-run-05.sh the launcher (pidfile-guarded, GPU-clear-asserting)
|
||
build_r5_survivors.py how survivors-r5 was built
|
||
run-05/ output: adapter/, checkpoints/, encode-cache/, truncation-report.json
|
||
|
||
Canonical copies of the config, launcher and survivors builder are in this repo
|
||
under `scripts/erp-tune-gx10/`. `survivors-r5 = survivors-r4 minus airoboros
|
||
plus govreport + qmsum whole` — deriving from run 4's survivors is what keeps
|
||
kvasir's cut byte-identical.
|
||
|
||
## Gates that passed at launch (all before / at encode, no GPU sunk on a bad build)
|
||
|
||
- **Preflight** (also run standalone, no GPU): 7/7 root shas verified, each shard
|
||
re-hashed against its CLEANROOT (govreport `d533a5cd`, qmsum `c0269d69`,
|
||
byte-identical after the NFS→box copy). Survivor→root join = 8,212 samples,
|
||
composition exactly the recipe (c2-logs 301 / cwm 183 / bluemoon 126 /
|
||
fireball 5396 / kvasir 1613 / govreport 496 / qmsum 97).
|
||
- **Holdout disjoint:** 8,212 training vs 8,404 holdout ids, 0 overlap.
|
||
- **window_count belt-and-suspenders** (`SFT-RECIPE-run5-SCOPE.md §7.1`): every
|
||
slot row must render window_count == 1. `truncation-report.json` showed
|
||
**govreport fit_whole=496 / chunked=0 / trunc=0** and **qmsum fit_whole=97 /
|
||
chunked=0 / trunc=0**. Any chunked/truncated slot row is a BUILD DEFECT → kill
|
||
before training. It passed.
|
||
- **Realized `[mix]` loss shares** reproduce the recipe's preregistered shares to
|
||
<0.1%: kvasir 0.3369, fireball 0.3042, c2-logs 0.1153, bluemoon 0.1066,
|
||
cwm 0.1022, govreport 0.0291, qmsum 0.0055. Slot subtotal 3.46% (pred 3.4%),
|
||
slot ctx 0.20.
|
||
|
||
Encode: 8,197 samples → 8,370 records (15 unfittable dropped, same as run 4),
|
||
ctx 18,598,779 / loss 9,935,076 tok. **524 optimizer steps** (÷ effective batch
|
||
16). ~52 s/it on this box → ~7.5 h. Checkpoints every 50 steps.
|
||
|
||
## Launch / watch / stop
|
||
|
||
ssh infra-ops@10.100.50.60 '~/erp-tune/launch-run-05.sh'
|
||
ssh infra-ops@10.100.50.60 "tr '\r' '\n' < ~/erp-tune/run-05.log | tail"
|
||
ssh infra-ops@10.100.50.60 'kill $(cat ~/erp-tune/run-05.pid)' # by PID — never pkill -f over ssh
|
||
|
||
The launcher refuses if the GPU is not clear, if `run-05.pid` names a live
|
||
process, if `run-05.log` exists, if any input is missing, or if under 40 GB free.
|
||
It detaches with `setsid nohup` and logs on the box.
|
||
|
||
## After the adapter lands — the gate choreography (with brokkr-smithy-dev)
|
||
|
||
Same as run 4, coordinated on the althing **cc channel** (Brokkr is armed there):
|
||
serve the **base** on this box/stack → Brokkr locks `FLOOR-LOCKED.json` and has
|
||
already committed the four-cell verdict evaluator (`tools/verdict_run05.py`:
|
||
RESCUED / STILL-COUPLED / OVER-CORRECTED / INERT) before any tuned figure exists
|
||
→ send the **swap cue** → serve the **tuned** artifact. T4 is the headline
|
||
must-not-harm (per-item, never pooled); refusal is telemetry; CSAM measurement
|
||
fail-closed stays. The override id + both blockers + all substitute controls are
|
||
in `run-05-gx10.json`, so they ride into the adapter's provenance manifest.
|