BabyYarros: stage the 4B-Base comparison arm, both concurrent and chained
One arm is not a comparison. On the Brontë corpus Base won held-out loss (2.814 against the instruct arm's 2.908) while instruct won instruction-following, and that trade is the open question for Skaldsong. It has to be re-measured on this corpus rather than carried over: BabyYarros is 12% larger and contemporary. Two entry points because the right one depends on a measurement rather than an argument. The box has 98 GiB free against an 18.4 GiB training footprint, so VRAM is not the constraint -- but VRAM was never the binding constraint on a GB10 that onboarding measured at 6x slower than ana-ml2 where compute predicts 2.7x, and where batching was not a throughput lever. If the box is already at its bandwidth roofline, a second job splits the same bandwidth and buys nothing. So: launch-yarros-4b-base.sh takes an explicit --allow-shared-gpu that bypasses the GPU-clear guard, and chain-yarros-4b-base.sh waits on the instruct run and refuses if it produced no adapter. The bypass is an argument and never a default, because the guard's normal job -- stopping a chain from firing into a live run -- is what keeps one lost run from becoming two. A shared-GPU launch stamps its own log with a warning that its s/it is not comparable to a solo run, since the harness is part of the number. Both carry the same gate guard as the instruct arm: refuse to start unless the leak gate report on disk says PASSED.
This commit is contained in:
+18
@@ -0,0 +1,18 @@
|
||||
#!/usr/bin/env bash
|
||||
# Chain the Qwen3-4B-Base arm behind the instruct arm.
|
||||
#
|
||||
# ⚠ Gated on the instruct run having actually produced an adapter. If that run died,
|
||||
# this must not quietly start and consume the box: a chain that fires on failure turns
|
||||
# one lost run into two.
|
||||
set -uo pipefail
|
||||
PREV=/home/infra-ops/r49-runs/yarros-4b-instruct-1ep
|
||||
CHAINLOG=/home/infra-ops/r49-runs/chain-yarros-4b-base.log
|
||||
exec >> "$CHAINLOG" 2>&1
|
||||
echo "=== $(date -Is) chain armed, waiting on $PREV"
|
||||
while [ -f "$PREV/run.pid" ] && kill -0 "$(cat "$PREV/run.pid")" 2>/dev/null; do sleep 60; done
|
||||
echo "=== $(date -Is) instruct run finished"
|
||||
if [ ! -f "$PREV/adapter/adapter_model.safetensors" ]; then
|
||||
echo "=== REFUSING to chain: $PREV produced no adapter -- the instruct run did not succeed"
|
||||
exit 1
|
||||
fi
|
||||
exec /home/infra-ops/r49-prep/launch-yarros-4b-base.sh
|
||||
+59
@@ -0,0 +1,59 @@
|
||||
#!/usr/bin/env bash
|
||||
# BabyYarros — Qwen3-4B-BASE, 1 epoch, seed 4919. The comparison partner to the
|
||||
# instruct arm: same corpus, same seed, same everything, CARRIER is the only variable.
|
||||
#
|
||||
# Why it exists: one arm is not a comparison. On the Brontë corpus Base won held-out
|
||||
# loss (2.814 vs the instruct arm's 2.908) while instruct won instruction-following
|
||||
# (on-beat 10/10 through the chat template, task-leak 0/18). That trade is the whole
|
||||
# open question for Skaldsong, and it has to be re-measured per corpus rather than
|
||||
# carried over -- the Yarros corpus is 12% larger and contemporary rather than 1840s.
|
||||
#
|
||||
# ⚠ Expect the loss minimum EARLY. The Brontë 4B-Base rung overfit inside one epoch
|
||||
# and turned at step 75 of 159; its shipped adapter/ was NOT the best weights. Hence
|
||||
# eval + save every 25 so the minimum is LOCATED, and arms get cut from the checkpoint.
|
||||
#
|
||||
# --allow-shared-gpu DELIBERATELY bypasses the GPU-clear guard, for the measured case
|
||||
# where a second run fits and the operator has asked for concurrency. It is an explicit
|
||||
# argument and not a default, because the guard's normal job -- stopping a chain from
|
||||
# firing into a live run -- is the thing that keeps one lost run from becoming two.
|
||||
set -euo pipefail
|
||||
SHARED=0
|
||||
[ "${1:-}" = "--allow-shared-gpu" ] && SHARED=1
|
||||
|
||||
OUT=/home/infra-ops/r49-runs/yarros-4b-base-1ep
|
||||
LOG=$OUT/train.log
|
||||
CORPUS=/home/infra-ops/yarros-corpus-renamed
|
||||
|
||||
if [ "$SHARED" = "0" ]; then
|
||||
apps=$(nvidia-smi --query-compute-apps=pid --format=csv,noheader | tr -d '[:space:]')
|
||||
[ -n "$apps" ] && { echo "REFUSING: GPU not clear (pass --allow-shared-gpu to override)" >&2; \
|
||||
nvidia-smi --query-compute-apps=pid,process_name,used_memory --format=csv >&2; exit 1; }
|
||||
fi
|
||||
[ -e "$LOG" ] && { echo "REFUSING: $LOG exists" >&2; exit 1; }
|
||||
|
||||
# ⚠ Same gate guard as the instruct arm. Training on ungated text is the one failure
|
||||
# this pipeline exists to prevent, and a launcher that trusts a human to have
|
||||
# remembered is not a guard.
|
||||
python3 - "$CORPUS/leak_gate_report.json" <<'PY'
|
||||
import json, sys
|
||||
r = json.load(open(sys.argv[1])); c = r["controls"]
|
||||
bad = (not c["positive_pass"] or not c["negative_pass"] or r["surviving_renameable"]
|
||||
or r["surviving_sub_threshold"] or r.get("surviving_phrases"))
|
||||
print(f" gate report: {r['renameable_total']} surfaces, controls pos={c['positive_pass']} "
|
||||
f"neg={c['negative_pass']}, surviving {len(r['surviving_renameable'])} entities / "
|
||||
f"{len(r.get('surviving_phrases', {}))} phrases")
|
||||
sys.exit(1 if bad else 0)
|
||||
PY
|
||||
|
||||
mkdir -p "$OUT"
|
||||
CSHA=$(cat "$CORPUS"/copies/*.jsonl | sha256sum | cut -c1-16)
|
||||
echo "# launched $(date -Is) Qwen3-4B-Base, 1 epoch, seed 4919, shared_gpu=$SHARED" > "$LOG"
|
||||
echo "# corpus $CORPUS sha $CSHA (leak gate PASSED)" >> "$LOG"
|
||||
[ "$SHARED" = "1" ] && echo "# ⚠ CONCURRENT with another training job -- s/it in this log is NOT comparable to a solo run" >> "$LOG"
|
||||
setsid nohup /home/infra-ops/ml/.venv/bin/python /home/infra-ops/r49-prep/train_voice_lora.py \
|
||||
--corpus "$CORPUS" \
|
||||
--base /home/infra-ops/carriers/Qwen3-4B-Base \
|
||||
--seed 4919 --epochs 1 --eval-steps 25 --save-steps 25 \
|
||||
--out "$OUT" >> "$LOG" 2>&1 < /dev/null &
|
||||
echo $! > "$OUT/run.pid"
|
||||
echo "launched pid $(cat "$OUT/run.pid") -> $LOG (corpus sha $CSHA, shared_gpu=$SHARED)"
|
||||
+64
@@ -0,0 +1,64 @@
|
||||
#!/usr/bin/env bash
|
||||
# BabyYarros — Qwen3-4B INSTRUCT, 1 epoch, seed 4919, on the gated renamed corpus.
|
||||
#
|
||||
# Carrier is the instruct model on the operator's direction ("train the instruct on
|
||||
# the yarros corpus -- babyyarros") and because the R49 instruct probe answered the
|
||||
# question it was run to answer: voice and instruction-following COEXIST. On the
|
||||
# Brontë corpus the instruct arm held curly quotes 16/18 -- identical to 4B-Base --
|
||||
# took zero task-leak in 18 samples, and stayed on-beat 10/10 through the chat
|
||||
# template, at a held-out 2.908 against Base's 2.814. The cost was length
|
||||
# discipline (in-band 10/10 -> 6/10), not voice.
|
||||
#
|
||||
# ⚠ This still trains RAW CONTINUATION text into a model whose weights expect
|
||||
# <|im_start|> framing. That is the known risk and it is the same one the Brontë
|
||||
# probe measured; the product path (instruction-pair corpus, Skaldsong Option C)
|
||||
# is separate and larger work.
|
||||
#
|
||||
# Everything else is held from the Brontë instruct arm so the CORPUS is the only
|
||||
# variable: seed 4919, rank 32, lr 1e-4, seq 4096, batch 1 x accum 8, 1 epoch,
|
||||
# eval + save every 25 so the loss minimum is LOCATED rather than assumed -- the
|
||||
# 4B rung overfit inside one epoch and turned at step 75 of 159, and its shipped
|
||||
# adapter/ was NOT the best weights.
|
||||
#
|
||||
# CORPUS PROVENANCE: 6 copies of 208 chapters, all five works, renamed under ONE
|
||||
# corpus-wide map per copy. Leak gate PASSED -- 0 of 325 source entities and 0 of
|
||||
# 91 audited phrases survive in any of the 30 copy files, both controls passing,
|
||||
# sensitivity floor 3 occurrences for a name and 5 for a phrase.
|
||||
set -euo pipefail
|
||||
OUT=/home/infra-ops/r49-runs/yarros-4b-instruct-1ep
|
||||
LOG=$OUT/train.log
|
||||
CORPUS=/home/infra-ops/yarros-corpus-renamed
|
||||
|
||||
apps=$(nvidia-smi --query-compute-apps=pid --format=csv,noheader | tr -d '[:space:]')
|
||||
[ -n "$apps" ] && { echo "REFUSING: GPU not clear" >&2; nvidia-smi --query-compute-apps=pid,process_name,used_memory --format=csv >&2; exit 1; }
|
||||
[ -e "$LOG" ] && { echo "REFUSING: $LOG exists" >&2; exit 1; }
|
||||
|
||||
# ⚠ Refuse to train on a corpus whose gate has not passed. Training on ungated
|
||||
# text is the one failure this whole pipeline exists to prevent, and a launcher
|
||||
# that trusts a human to have remembered is not a guard.
|
||||
GATE=$CORPUS/leak_gate_report.json
|
||||
python3 - "$GATE" <<'PY'
|
||||
import json, sys
|
||||
r = json.load(open(sys.argv[1]))
|
||||
c = r["controls"]
|
||||
bad = (not c["positive_pass"] or not c["negative_pass"]
|
||||
or r["surviving_renameable"] or r["surviving_sub_threshold"]
|
||||
or r.get("surviving_phrases"))
|
||||
print(f" gate report: {r['renameable_total']} surfaces, "
|
||||
f"controls pos={c['positive_pass']} neg={c['negative_pass']}, "
|
||||
f"surviving {len(r['surviving_renameable'])} entities / "
|
||||
f"{len(r.get('surviving_phrases', {}))} phrases")
|
||||
sys.exit(1 if bad else 0)
|
||||
PY
|
||||
|
||||
mkdir -p "$OUT"
|
||||
CSHA=$(cat "$CORPUS"/copies/*.jsonl | sha256sum | cut -c1-16)
|
||||
echo "# launched $(date -Is) Qwen3-4B INSTRUCT, 1 epoch, seed 4919" > "$LOG"
|
||||
echo "# corpus $CORPUS sha $CSHA (leak gate PASSED)" >> "$LOG"
|
||||
setsid nohup /home/infra-ops/ml/.venv/bin/python /home/infra-ops/r49-prep/train_voice_lora.py \
|
||||
--corpus "$CORPUS" \
|
||||
--base /home/infra-ops/carriers/Qwen3-4B-Instruct \
|
||||
--seed 4919 --epochs 1 --eval-steps 25 --save-steps 25 \
|
||||
--out "$OUT" >> "$LOG" 2>&1 < /dev/null &
|
||||
echo $! > "$OUT/run.pid"
|
||||
echo "launched pid $(cat "$OUT/run.pid") -> $LOG (corpus sha $CSHA)"
|
||||
Reference in New Issue
Block a user