From 38eb40ace463c6f75920a428bbfa3a90f82b72ef Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Fri, 11 Sep 2026 10:22:21 -0700 Subject: [PATCH] BabyYarros: stage the 4B-Base comparison arm, both concurrent and chained MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit One arm is not a comparison. On the Brontë corpus Base won held-out loss (2.814 against the instruct arm's 2.908) while instruct won instruction-following, and that trade is the open question for Skaldsong. It has to be re-measured on this corpus rather than carried over: BabyYarros is 12% larger and contemporary. Two entry points because the right one depends on a measurement rather than an argument. The box has 98 GiB free against an 18.4 GiB training footprint, so VRAM is not the constraint -- but VRAM was never the binding constraint on a GB10 that onboarding measured at 6x slower than ana-ml2 where compute predicts 2.7x, and where batching was not a throughput lever. If the box is already at its bandwidth roofline, a second job splits the same bandwidth and buys nothing. So: launch-yarros-4b-base.sh takes an explicit --allow-shared-gpu that bypasses the GPU-clear guard, and chain-yarros-4b-base.sh waits on the instruct run and refuses if it produced no adapter. The bypass is an argument and never a default, because the guard's normal job -- stopping a chain from firing into a live run -- is what keeps one lost run from becoming two. A shared-GPU launch stamps its own log with a warning that its s/it is not comparable to a solo run, since the harness is part of the number. Both carry the same gate guard as the instruct arm: refuse to start unless the leak gate report on disk says PASSED. --- scripts/yarros-corpus/chain-yarros-4b-base.sh | 18 ++++++ .../yarros-corpus/launch-yarros-4b-base.sh | 59 +++++++++++++++++ .../launch-yarros-4b-instruct.sh | 64 +++++++++++++++++++ 3 files changed, 141 insertions(+) create mode 100755 scripts/yarros-corpus/chain-yarros-4b-base.sh create mode 100755 scripts/yarros-corpus/launch-yarros-4b-base.sh create mode 100755 scripts/yarros-corpus/launch-yarros-4b-instruct.sh diff --git a/scripts/yarros-corpus/chain-yarros-4b-base.sh b/scripts/yarros-corpus/chain-yarros-4b-base.sh new file mode 100755 index 0000000..4c1acdb --- /dev/null +++ b/scripts/yarros-corpus/chain-yarros-4b-base.sh @@ -0,0 +1,18 @@ +#!/usr/bin/env bash +# Chain the Qwen3-4B-Base arm behind the instruct arm. +# +# ⚠ Gated on the instruct run having actually produced an adapter. If that run died, +# this must not quietly start and consume the box: a chain that fires on failure turns +# one lost run into two. +set -uo pipefail +PREV=/home/infra-ops/r49-runs/yarros-4b-instruct-1ep +CHAINLOG=/home/infra-ops/r49-runs/chain-yarros-4b-base.log +exec >> "$CHAINLOG" 2>&1 +echo "=== $(date -Is) chain armed, waiting on $PREV" +while [ -f "$PREV/run.pid" ] && kill -0 "$(cat "$PREV/run.pid")" 2>/dev/null; do sleep 60; done +echo "=== $(date -Is) instruct run finished" +if [ ! -f "$PREV/adapter/adapter_model.safetensors" ]; then + echo "=== REFUSING to chain: $PREV produced no adapter -- the instruct run did not succeed" + exit 1 +fi +exec /home/infra-ops/r49-prep/launch-yarros-4b-base.sh diff --git a/scripts/yarros-corpus/launch-yarros-4b-base.sh b/scripts/yarros-corpus/launch-yarros-4b-base.sh new file mode 100755 index 0000000..fdf959e --- /dev/null +++ b/scripts/yarros-corpus/launch-yarros-4b-base.sh @@ -0,0 +1,59 @@ +#!/usr/bin/env bash +# BabyYarros — Qwen3-4B-BASE, 1 epoch, seed 4919. The comparison partner to the +# instruct arm: same corpus, same seed, same everything, CARRIER is the only variable. +# +# Why it exists: one arm is not a comparison. On the Brontë corpus Base won held-out +# loss (2.814 vs the instruct arm's 2.908) while instruct won instruction-following +# (on-beat 10/10 through the chat template, task-leak 0/18). That trade is the whole +# open question for Skaldsong, and it has to be re-measured per corpus rather than +# carried over -- the Yarros corpus is 12% larger and contemporary rather than 1840s. +# +# ⚠ Expect the loss minimum EARLY. The Brontë 4B-Base rung overfit inside one epoch +# and turned at step 75 of 159; its shipped adapter/ was NOT the best weights. Hence +# eval + save every 25 so the minimum is LOCATED, and arms get cut from the checkpoint. +# +# --allow-shared-gpu DELIBERATELY bypasses the GPU-clear guard, for the measured case +# where a second run fits and the operator has asked for concurrency. It is an explicit +# argument and not a default, because the guard's normal job -- stopping a chain from +# firing into a live run -- is the thing that keeps one lost run from becoming two. +set -euo pipefail +SHARED=0 +[ "${1:-}" = "--allow-shared-gpu" ] && SHARED=1 + +OUT=/home/infra-ops/r49-runs/yarros-4b-base-1ep +LOG=$OUT/train.log +CORPUS=/home/infra-ops/yarros-corpus-renamed + +if [ "$SHARED" = "0" ]; then + apps=$(nvidia-smi --query-compute-apps=pid --format=csv,noheader | tr -d '[:space:]') + [ -n "$apps" ] && { echo "REFUSING: GPU not clear (pass --allow-shared-gpu to override)" >&2; \ + nvidia-smi --query-compute-apps=pid,process_name,used_memory --format=csv >&2; exit 1; } +fi +[ -e "$LOG" ] && { echo "REFUSING: $LOG exists" >&2; exit 1; } + +# ⚠ Same gate guard as the instruct arm. Training on ungated text is the one failure +# this pipeline exists to prevent, and a launcher that trusts a human to have +# remembered is not a guard. +python3 - "$CORPUS/leak_gate_report.json" <<'PY' +import json, sys +r = json.load(open(sys.argv[1])); c = r["controls"] +bad = (not c["positive_pass"] or not c["negative_pass"] or r["surviving_renameable"] + or r["surviving_sub_threshold"] or r.get("surviving_phrases")) +print(f" gate report: {r['renameable_total']} surfaces, controls pos={c['positive_pass']} " + f"neg={c['negative_pass']}, surviving {len(r['surviving_renameable'])} entities / " + f"{len(r.get('surviving_phrases', {}))} phrases") +sys.exit(1 if bad else 0) +PY + +mkdir -p "$OUT" +CSHA=$(cat "$CORPUS"/copies/*.jsonl | sha256sum | cut -c1-16) +echo "# launched $(date -Is) Qwen3-4B-Base, 1 epoch, seed 4919, shared_gpu=$SHARED" > "$LOG" +echo "# corpus $CORPUS sha $CSHA (leak gate PASSED)" >> "$LOG" +[ "$SHARED" = "1" ] && echo "# ⚠ CONCURRENT with another training job -- s/it in this log is NOT comparable to a solo run" >> "$LOG" +setsid nohup /home/infra-ops/ml/.venv/bin/python /home/infra-ops/r49-prep/train_voice_lora.py \ + --corpus "$CORPUS" \ + --base /home/infra-ops/carriers/Qwen3-4B-Base \ + --seed 4919 --epochs 1 --eval-steps 25 --save-steps 25 \ + --out "$OUT" >> "$LOG" 2>&1 < /dev/null & +echo $! > "$OUT/run.pid" +echo "launched pid $(cat "$OUT/run.pid") -> $LOG (corpus sha $CSHA, shared_gpu=$SHARED)" diff --git a/scripts/yarros-corpus/launch-yarros-4b-instruct.sh b/scripts/yarros-corpus/launch-yarros-4b-instruct.sh new file mode 100755 index 0000000..b02e421 --- /dev/null +++ b/scripts/yarros-corpus/launch-yarros-4b-instruct.sh @@ -0,0 +1,64 @@ +#!/usr/bin/env bash +# BabyYarros — Qwen3-4B INSTRUCT, 1 epoch, seed 4919, on the gated renamed corpus. +# +# Carrier is the instruct model on the operator's direction ("train the instruct on +# the yarros corpus -- babyyarros") and because the R49 instruct probe answered the +# question it was run to answer: voice and instruction-following COEXIST. On the +# Brontë corpus the instruct arm held curly quotes 16/18 -- identical to 4B-Base -- +# took zero task-leak in 18 samples, and stayed on-beat 10/10 through the chat +# template, at a held-out 2.908 against Base's 2.814. The cost was length +# discipline (in-band 10/10 -> 6/10), not voice. +# +# ⚠ This still trains RAW CONTINUATION text into a model whose weights expect +# <|im_start|> framing. That is the known risk and it is the same one the Brontë +# probe measured; the product path (instruction-pair corpus, Skaldsong Option C) +# is separate and larger work. +# +# Everything else is held from the Brontë instruct arm so the CORPUS is the only +# variable: seed 4919, rank 32, lr 1e-4, seq 4096, batch 1 x accum 8, 1 epoch, +# eval + save every 25 so the loss minimum is LOCATED rather than assumed -- the +# 4B rung overfit inside one epoch and turned at step 75 of 159, and its shipped +# adapter/ was NOT the best weights. +# +# CORPUS PROVENANCE: 6 copies of 208 chapters, all five works, renamed under ONE +# corpus-wide map per copy. Leak gate PASSED -- 0 of 325 source entities and 0 of +# 91 audited phrases survive in any of the 30 copy files, both controls passing, +# sensitivity floor 3 occurrences for a name and 5 for a phrase. +set -euo pipefail +OUT=/home/infra-ops/r49-runs/yarros-4b-instruct-1ep +LOG=$OUT/train.log +CORPUS=/home/infra-ops/yarros-corpus-renamed + +apps=$(nvidia-smi --query-compute-apps=pid --format=csv,noheader | tr -d '[:space:]') +[ -n "$apps" ] && { echo "REFUSING: GPU not clear" >&2; nvidia-smi --query-compute-apps=pid,process_name,used_memory --format=csv >&2; exit 1; } +[ -e "$LOG" ] && { echo "REFUSING: $LOG exists" >&2; exit 1; } + +# ⚠ Refuse to train on a corpus whose gate has not passed. Training on ungated +# text is the one failure this whole pipeline exists to prevent, and a launcher +# that trusts a human to have remembered is not a guard. +GATE=$CORPUS/leak_gate_report.json +python3 - "$GATE" <<'PY' +import json, sys +r = json.load(open(sys.argv[1])) +c = r["controls"] +bad = (not c["positive_pass"] or not c["negative_pass"] + or r["surviving_renameable"] or r["surviving_sub_threshold"] + or r.get("surviving_phrases")) +print(f" gate report: {r['renameable_total']} surfaces, " + f"controls pos={c['positive_pass']} neg={c['negative_pass']}, " + f"surviving {len(r['surviving_renameable'])} entities / " + f"{len(r.get('surviving_phrases', {}))} phrases") +sys.exit(1 if bad else 0) +PY + +mkdir -p "$OUT" +CSHA=$(cat "$CORPUS"/copies/*.jsonl | sha256sum | cut -c1-16) +echo "# launched $(date -Is) Qwen3-4B INSTRUCT, 1 epoch, seed 4919" > "$LOG" +echo "# corpus $CORPUS sha $CSHA (leak gate PASSED)" >> "$LOG" +setsid nohup /home/infra-ops/ml/.venv/bin/python /home/infra-ops/r49-prep/train_voice_lora.py \ + --corpus "$CORPUS" \ + --base /home/infra-ops/carriers/Qwen3-4B-Instruct \ + --seed 4919 --epochs 1 --eval-steps 25 --save-steps 25 \ + --out "$OUT" >> "$LOG" 2>&1 < /dev/null & +echo $! > "$OUT/run.pid" +echo "launched pid $(cat "$OUT/run.pid") -> $LOG (corpus sha $CSHA)"