vh
38eb40ace4
One arm is not a comparison. On the Brontë corpus Base won held-out loss (2.814 against the instruct arm's 2.908) while instruct won instruction-following, and that trade is the open question for Skaldsong. It has to be re-measured on this corpus rather than carried over: BabyYarros is 12% larger and contemporary. Two entry points because the right one depends on a measurement rather than an argument. The box has 98 GiB free against an 18.4 GiB training footprint, so VRAM is not the constraint -- but VRAM was never the binding constraint on a GB10 that onboarding measured at 6x slower than ana-ml2 where compute predicts 2.7x, and where batching was not a throughput lever. If the box is already at its bandwidth roofline, a second job splits the same bandwidth and buys nothing. So: launch-yarros-4b-base.sh takes an explicit --allow-shared-gpu that bypasses the GPU-clear guard, and chain-yarros-4b-base.sh waits on the instruct run and refuses if it produced no adapter. The bypass is an argument and never a default, because the guard's normal job -- stopping a chain from firing into a live run -- is what keeps one lost run from becoming two. A shared-GPU launch stamps its own log with a warning that its s/it is not comparable to a solo run, since the harness is part of the number. Both carry the same gate guard as the instruct arm: refuse to start unless the leak gate report on disk says PASSED.
60 lines
3.2 KiB
Bash
Executable File
60 lines
3.2 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# BabyYarros — Qwen3-4B-BASE, 1 epoch, seed 4919. The comparison partner to the
|
|
# instruct arm: same corpus, same seed, same everything, CARRIER is the only variable.
|
|
#
|
|
# Why it exists: one arm is not a comparison. On the Brontë corpus Base won held-out
|
|
# loss (2.814 vs the instruct arm's 2.908) while instruct won instruction-following
|
|
# (on-beat 10/10 through the chat template, task-leak 0/18). That trade is the whole
|
|
# open question for Skaldsong, and it has to be re-measured per corpus rather than
|
|
# carried over -- the Yarros corpus is 12% larger and contemporary rather than 1840s.
|
|
#
|
|
# ⚠ Expect the loss minimum EARLY. The Brontë 4B-Base rung overfit inside one epoch
|
|
# and turned at step 75 of 159; its shipped adapter/ was NOT the best weights. Hence
|
|
# eval + save every 25 so the minimum is LOCATED, and arms get cut from the checkpoint.
|
|
#
|
|
# --allow-shared-gpu DELIBERATELY bypasses the GPU-clear guard, for the measured case
|
|
# where a second run fits and the operator has asked for concurrency. It is an explicit
|
|
# argument and not a default, because the guard's normal job -- stopping a chain from
|
|
# firing into a live run -- is the thing that keeps one lost run from becoming two.
|
|
set -euo pipefail
|
|
SHARED=0
|
|
[ "${1:-}" = "--allow-shared-gpu" ] && SHARED=1
|
|
|
|
OUT=/home/infra-ops/r49-runs/yarros-4b-base-1ep
|
|
LOG=$OUT/train.log
|
|
CORPUS=/home/infra-ops/yarros-corpus-renamed
|
|
|
|
if [ "$SHARED" = "0" ]; then
|
|
apps=$(nvidia-smi --query-compute-apps=pid --format=csv,noheader | tr -d '[:space:]')
|
|
[ -n "$apps" ] && { echo "REFUSING: GPU not clear (pass --allow-shared-gpu to override)" >&2; \
|
|
nvidia-smi --query-compute-apps=pid,process_name,used_memory --format=csv >&2; exit 1; }
|
|
fi
|
|
[ -e "$LOG" ] && { echo "REFUSING: $LOG exists" >&2; exit 1; }
|
|
|
|
# ⚠ Same gate guard as the instruct arm. Training on ungated text is the one failure
|
|
# this pipeline exists to prevent, and a launcher that trusts a human to have
|
|
# remembered is not a guard.
|
|
python3 - "$CORPUS/leak_gate_report.json" <<'PY'
|
|
import json, sys
|
|
r = json.load(open(sys.argv[1])); c = r["controls"]
|
|
bad = (not c["positive_pass"] or not c["negative_pass"] or r["surviving_renameable"]
|
|
or r["surviving_sub_threshold"] or r.get("surviving_phrases"))
|
|
print(f" gate report: {r['renameable_total']} surfaces, controls pos={c['positive_pass']} "
|
|
f"neg={c['negative_pass']}, surviving {len(r['surviving_renameable'])} entities / "
|
|
f"{len(r.get('surviving_phrases', {}))} phrases")
|
|
sys.exit(1 if bad else 0)
|
|
PY
|
|
|
|
mkdir -p "$OUT"
|
|
CSHA=$(cat "$CORPUS"/copies/*.jsonl | sha256sum | cut -c1-16)
|
|
echo "# launched $(date -Is) Qwen3-4B-Base, 1 epoch, seed 4919, shared_gpu=$SHARED" > "$LOG"
|
|
echo "# corpus $CORPUS sha $CSHA (leak gate PASSED)" >> "$LOG"
|
|
[ "$SHARED" = "1" ] && echo "# ⚠ CONCURRENT with another training job -- s/it in this log is NOT comparable to a solo run" >> "$LOG"
|
|
setsid nohup /home/infra-ops/ml/.venv/bin/python /home/infra-ops/r49-prep/train_voice_lora.py \
|
|
--corpus "$CORPUS" \
|
|
--base /home/infra-ops/carriers/Qwen3-4B-Base \
|
|
--seed 4919 --epochs 1 --eval-steps 25 --save-steps 25 \
|
|
--out "$OUT" >> "$LOG" 2>&1 < /dev/null &
|
|
echo $! > "$OUT/run.pid"
|
|
echo "launched pid $(cat "$OUT/run.pid") -> $LOG (corpus sha $CSHA, shared_gpu=$SHARED)"
|