vh
38eb40ace4
One arm is not a comparison. On the Brontë corpus Base won held-out loss (2.814 against the instruct arm's 2.908) while instruct won instruction-following, and that trade is the open question for Skaldsong. It has to be re-measured on this corpus rather than carried over: BabyYarros is 12% larger and contemporary. Two entry points because the right one depends on a measurement rather than an argument. The box has 98 GiB free against an 18.4 GiB training footprint, so VRAM is not the constraint -- but VRAM was never the binding constraint on a GB10 that onboarding measured at 6x slower than ana-ml2 where compute predicts 2.7x, and where batching was not a throughput lever. If the box is already at its bandwidth roofline, a second job splits the same bandwidth and buys nothing. So: launch-yarros-4b-base.sh takes an explicit --allow-shared-gpu that bypasses the GPU-clear guard, and chain-yarros-4b-base.sh waits on the instruct run and refuses if it produced no adapter. The bypass is an argument and never a default, because the guard's normal job -- stopping a chain from firing into a live run -- is what keeps one lost run from becoming two. A shared-GPU launch stamps its own log with a warning that its s/it is not comparable to a solo run, since the harness is part of the number. Both carry the same gate guard as the instruct arm: refuse to start unless the leak gate report on disk says PASSED.
65 lines
3.3 KiB
Bash
Executable File
65 lines
3.3 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# BabyYarros — Qwen3-4B INSTRUCT, 1 epoch, seed 4919, on the gated renamed corpus.
|
|
#
|
|
# Carrier is the instruct model on the operator's direction ("train the instruct on
|
|
# the yarros corpus -- babyyarros") and because the R49 instruct probe answered the
|
|
# question it was run to answer: voice and instruction-following COEXIST. On the
|
|
# Brontë corpus the instruct arm held curly quotes 16/18 -- identical to 4B-Base --
|
|
# took zero task-leak in 18 samples, and stayed on-beat 10/10 through the chat
|
|
# template, at a held-out 2.908 against Base's 2.814. The cost was length
|
|
# discipline (in-band 10/10 -> 6/10), not voice.
|
|
#
|
|
# ⚠ This still trains RAW CONTINUATION text into a model whose weights expect
|
|
# <|im_start|> framing. That is the known risk and it is the same one the Brontë
|
|
# probe measured; the product path (instruction-pair corpus, Skaldsong Option C)
|
|
# is separate and larger work.
|
|
#
|
|
# Everything else is held from the Brontë instruct arm so the CORPUS is the only
|
|
# variable: seed 4919, rank 32, lr 1e-4, seq 4096, batch 1 x accum 8, 1 epoch,
|
|
# eval + save every 25 so the loss minimum is LOCATED rather than assumed -- the
|
|
# 4B rung overfit inside one epoch and turned at step 75 of 159, and its shipped
|
|
# adapter/ was NOT the best weights.
|
|
#
|
|
# CORPUS PROVENANCE: 6 copies of 208 chapters, all five works, renamed under ONE
|
|
# corpus-wide map per copy. Leak gate PASSED -- 0 of 325 source entities and 0 of
|
|
# 91 audited phrases survive in any of the 30 copy files, both controls passing,
|
|
# sensitivity floor 3 occurrences for a name and 5 for a phrase.
|
|
set -euo pipefail
|
|
OUT=/home/infra-ops/r49-runs/yarros-4b-instruct-1ep
|
|
LOG=$OUT/train.log
|
|
CORPUS=/home/infra-ops/yarros-corpus-renamed
|
|
|
|
apps=$(nvidia-smi --query-compute-apps=pid --format=csv,noheader | tr -d '[:space:]')
|
|
[ -n "$apps" ] && { echo "REFUSING: GPU not clear" >&2; nvidia-smi --query-compute-apps=pid,process_name,used_memory --format=csv >&2; exit 1; }
|
|
[ -e "$LOG" ] && { echo "REFUSING: $LOG exists" >&2; exit 1; }
|
|
|
|
# ⚠ Refuse to train on a corpus whose gate has not passed. Training on ungated
|
|
# text is the one failure this whole pipeline exists to prevent, and a launcher
|
|
# that trusts a human to have remembered is not a guard.
|
|
GATE=$CORPUS/leak_gate_report.json
|
|
python3 - "$GATE" <<'PY'
|
|
import json, sys
|
|
r = json.load(open(sys.argv[1]))
|
|
c = r["controls"]
|
|
bad = (not c["positive_pass"] or not c["negative_pass"]
|
|
or r["surviving_renameable"] or r["surviving_sub_threshold"]
|
|
or r.get("surviving_phrases"))
|
|
print(f" gate report: {r['renameable_total']} surfaces, "
|
|
f"controls pos={c['positive_pass']} neg={c['negative_pass']}, "
|
|
f"surviving {len(r['surviving_renameable'])} entities / "
|
|
f"{len(r.get('surviving_phrases', {}))} phrases")
|
|
sys.exit(1 if bad else 0)
|
|
PY
|
|
|
|
mkdir -p "$OUT"
|
|
CSHA=$(cat "$CORPUS"/copies/*.jsonl | sha256sum | cut -c1-16)
|
|
echo "# launched $(date -Is) Qwen3-4B INSTRUCT, 1 epoch, seed 4919" > "$LOG"
|
|
echo "# corpus $CORPUS sha $CSHA (leak gate PASSED)" >> "$LOG"
|
|
setsid nohup /home/infra-ops/ml/.venv/bin/python /home/infra-ops/r49-prep/train_voice_lora.py \
|
|
--corpus "$CORPUS" \
|
|
--base /home/infra-ops/carriers/Qwen3-4B-Instruct \
|
|
--seed 4919 --epochs 1 --eval-steps 25 --save-steps 25 \
|
|
--out "$OUT" >> "$LOG" 2>&1 < /dev/null &
|
|
echo $! > "$OUT/run.pid"
|
|
echo "launched pid $(cat "$OUT/run.pid") -> $LOG (corpus sha $CSHA)"
|