Files

T

vh 265357efb2 ops(erp-tune): stage + launch R47 ERP-seat SFT run 5 on pfi-gx10

Run 5 = the dependency-forcing corpus arm: airoboros-3.2 OUT of the 20% slot,
govreport/clean-v1 (496) + qmsum/clean-v1 (97) IN, at run 4's lr 2e-04 with
everything else held. kvasir byte-identical (survivors-r5 = survivors-r4 minus
airoboros plus the two new roots whole). Operator authorized the launch to
infra-ops directly; grant operator-2026-09-07-rnd-run5.

Canonical copies of the config, launcher and survivors builder; runbook
docs/runbooks/gx10-run-05.md. Launch gates all passed (7/7 root shas + shard
hashes, survivor join 8,212 = recipe, holdout disjoint, window_count==1 on all
593 slot rows, realized [mix] slot loss 3.46% vs preregistered 3.4%). 524 steps.
persistent-memory current-state updated: run 5 LAUNCHED + training.

2026-09-07 09:33:03 -07:00

5.0 KiB

Raw Blame History

pfi-gx10 — ERP-seat SFT run 5 (dependency-forcing slot)

Launched 2026-09-07 09:05 PDT on pfi-gx10, pid 3311780. Operator authorized the launch to infra-ops directly (the run-scoped training-eligibility override operator-2026-09-07-rnd-run5 was granted to Brokkr; the relay caveat was closed with the operator directly in-session before launch, as it was for run 4).

What run 5 is

The third R47 corpus-gate run. Single variable vs run 4: the 20% context slot's composition. Run 4 put an airoboros-3.2 instruct root in that slot; run 5 removes it and puts in two dependency-forcing roots — govreport/clean-v1 (496 reports) + qmsum/clean-v1 (97 transcripts), where the loss turn is determined by distant context (document first, query/instruction last). Everything else is held from run 4: dialogue survivors and the impersonation loss-mask byte-identical, fireball whole, kvasir at run 4's exact 1,613-sample prefix cut (NOT re-cut), base, lr 2e-04, max_seq_len 16384, rank 64, alpha 128, batch 2 × accum 8, cosine, warmup 0.1, wd 0.01.

The slot's ctx share reproduces run 4's 20%, but its loss share is 3.4% (run 4's was 20.6%) — preregistered in SFT-RECIPE-run5-SCOPE.md §4. INERT is a live verdict, not a build error.

Machine-readable source of truth: brokkr-smithy research/R47-premium-corpus-gate/recipe-erp-seat-sft-r5.json (f49994b, tag R49.4).

Layout on the box

/home/infra-ops/models/gemma4-26b-a4b-it-bf16      49 GB base (byte-identical to run 4)
/home/infra-ops/erp-tune/
  eitri-smithy/                harness, git 0a6bd2e
  recipe-r5/
    recipe-erp-seat-sft-r5.json    from brokkr-smithy, verbatim
    survivors-r5.jsonl             8,212 rows, sha256 a25169a6...
  datasets/derived/govreport/clean-v1/   copied from /mnt/smithy (no NFS on box)
  datasets/derived/qmsum/clean-v1/       copied from /mnt/smithy
  datasets/holdout/            run-1 8,404-sample holdout (disjointness check)
  run-05-gx10.json             the config
  launch-run-05.sh             the launcher (pidfile-guarded, GPU-clear-asserting)
  build_r5_survivors.py        how survivors-r5 was built
  run-05/                      output: adapter/, checkpoints/, encode-cache/, truncation-report.json

Canonical copies of the config, launcher and survivors builder are in this repo under scripts/erp-tune-gx10/. survivors-r5 = survivors-r4 minus airoboros plus govreport + qmsum whole — deriving from run 4's survivors is what keeps kvasir's cut byte-identical.

Gates that passed at launch (all before / at encode, no GPU sunk on a bad build)

Preflight (also run standalone, no GPU): 7/7 root shas verified, each shard re-hashed against its CLEANROOT (govreport d533a5cd, qmsum c0269d69, byte-identical after the NFS→box copy). Survivor→root join = 8,212 samples, composition exactly the recipe (c2-logs 301 / cwm 183 / bluemoon 126 / fireball 5396 / kvasir 1613 / govreport 496 / qmsum 97).
Holdout disjoint: 8,212 training vs 8,404 holdout ids, 0 overlap.
window_count belt-and-suspenders (SFT-RECIPE-run5-SCOPE.md §7.1): every slot row must render window_count == 1. truncation-report.json showed govreport fit_whole=496 / chunked=0 / trunc=0 and qmsum fit_whole=97 / chunked=0 / trunc=0. Any chunked/truncated slot row is a BUILD DEFECT → kill before training. It passed.
Realized [mix] loss shares reproduce the recipe's preregistered shares to <0.1%: kvasir 0.3369, fireball 0.3042, c2-logs 0.1153, bluemoon 0.1066, cwm 0.1022, govreport 0.0291, qmsum 0.0055. Slot subtotal 3.46% (pred 3.4%), slot ctx 0.20.

Encode: 8,197 samples → 8,370 records (15 unfittable dropped, same as run 4), ctx 18,598,779 / loss 9,935,076 tok. 524 optimizer steps (÷ effective batch 16). ~52 s/it on this box → ~7.5 h. Checkpoints every 50 steps.

Launch / watch / stop

ssh infra-ops@10.100.50.60 '~/erp-tune/launch-run-05.sh'
ssh infra-ops@10.100.50.60 "tr '\r' '\n' < ~/erp-tune/run-05.log | tail"
ssh infra-ops@10.100.50.60 'kill $(cat ~/erp-tune/run-05.pid)'   # by PID — never pkill -f over ssh

The launcher refuses if the GPU is not clear, if run-05.pid names a live process, if run-05.log exists, if any input is missing, or if under 40 GB free. It detaches with setsid nohup and logs on the box.

After the adapter lands — the gate choreography (with brokkr-smithy-dev)

Same as run 4, coordinated on the althing cc channel (Brokkr is armed there): serve the base on this box/stack → Brokkr locks FLOOR-LOCKED.json and has already committed the four-cell verdict evaluator (tools/verdict_run05.py: RESCUED / STILL-COUPLED / OVER-CORRECTED / INERT) before any tuned figure exists → send the swap cue → serve the tuned artifact. T4 is the headline must-not-harm (per-item, never pooled); refusal is telemetry; CSAM measurement fail-closed stays. The override id + both blockers + all substitute controls are in run-05-gx10.json, so they ride into the adapter's provenance manifest.

5.0 KiB Raw Blame History Unescape Escape