Unwrap the Brontë corpus and launch the 1.7B rung
Operator: "start the 1.7b training." The 0.6B adapter learned the Gutenberg transcription's ~70-character line breaks along with the prose -- its output wrapped at a mid-length-line ratio of 0.85 against the base model's 0.00. That is typography rather than style, and every further rung would have inherited it, so the corpus is reflowed before rung 2 rather than after the sweep. The reflow joins 57,430 of 85,380 paragraph blocks and keeps 27,950. Verse is the hazard a blind join would destroy, so the decision is per block by median line length: blocks whose lines cluster near the wrap width are flowed prose, blocks of consistently short lines keep their breaks. Every kept multi-line block in the sample was genuinely verse with its lineation intact. No line ended in a lone hyphen, so the space-join could not split a word across lines. The acceptance check is content identity -- " ".join(text.split()) byte-identical before and after -- and it passed on all 852 records, proving only whitespace changed. Concrete cost of the old defect: 5.7% of the training budget was newline tokens. The same words pack to 5,210,112 tokens unwrapped against 5,525,504 wrapped. The 1.7B run is live at 159 steps and roughly 18.7 s/it. Everything but the carrier and the corpus is held from the 0.6B run: seed 4919, rank 32, lr 1e-4, seq 4096, batch 1 by accum 8, one epoch, eval and save every 25 steps so the minimum is located rather than assumed. That corpus change is a second variable and it is named as one. A 0.6B-vs-1.7B comparison is descriptive, not attributable, until the chained 0.6B rerun on the same unwrapped corpus lands behind it -- gated on the 1.7B actually producing an adapter, because a chain that fires on failure turns one lost run into two. "Did sense come back at 1.7B" is a within-arm reading and survives the confound; any between-rung delta does not. The original wrapped corpus is untouched, so the 0.6B run's pinned corpus sha 3959036cf851bf62 stays reproducible.
This commit is contained in:
Executable
+48
@@ -0,0 +1,48 @@
|
||||
#!/usr/bin/env bash
|
||||
# Chained after the 1.7B rung: re-run the 0.6B carrier on the SAME unwrapped corpus.
|
||||
#
|
||||
# The 1.7B run moved two variables at once -- carrier size AND corpus typography --
|
||||
# because the unwrap could not wait if every further rung was to avoid inheriting the
|
||||
# Gutenberg line breaks. That makes a 0.6B-vs-1.7B comparison descriptive rather than
|
||||
# attributable, which is fine for "did sense come back" (a within-arm reading) and not
|
||||
# fine for anything quantitative between rungs.
|
||||
#
|
||||
# This closes that hole for the price of ~36 minutes on an idle experimental box:
|
||||
# 0.6B on the unwrapped corpus, seed 4919, everything else held. Then carrier size is
|
||||
# the ONLY difference between this and h02-1p7b-1ep, and the sweep is single-variable
|
||||
# again.
|
||||
#
|
||||
# ⚠ Gated on the 1.7B run having actually produced an adapter. If that run died, this
|
||||
# must not quietly start and consume the box; a chain that fires on failure turns one
|
||||
# lost run into two.
|
||||
set -uo pipefail
|
||||
PREV=/home/infra-ops/r49-runs/h02-1p7b-1ep
|
||||
OUT=/home/infra-ops/r49-runs/h02-0p6b-1ep-unwrapped
|
||||
LOG=$OUT/train.log
|
||||
CHAINLOG=/home/infra-ops/r49-runs/chain-0p6b-unwrapped.log
|
||||
|
||||
exec >> "$CHAINLOG" 2>&1
|
||||
echo "=== $(date -Is) chain armed, waiting on $PREV"
|
||||
while [ -f "$PREV/run.pid" ] && kill -0 "$(cat "$PREV/run.pid")" 2>/dev/null; do sleep 60; done
|
||||
echo "=== $(date -Is) 1.7B run finished"
|
||||
|
||||
if [ ! -f "$PREV/adapter/adapter_model.safetensors" ]; then
|
||||
echo "=== REFUSING to chain: $PREV produced no adapter -- the 1.7B run did not succeed"
|
||||
exit 1
|
||||
fi
|
||||
apps=$(nvidia-smi --query-compute-apps=pid --format=csv,noheader | tr -d '[:space:]')
|
||||
if [ -n "$apps" ]; then
|
||||
echo "=== REFUSING to chain: GPU not clear"
|
||||
nvidia-smi --query-compute-apps=pid,process_name,used_memory --format=csv
|
||||
exit 1
|
||||
fi
|
||||
|
||||
mkdir -p "$OUT"
|
||||
echo "# launched $(date -Is) Qwen3-0.6B-Base, 1 epoch, seed 4919, UNWRAPPED corpus (single-variable partner to h02-1p7b-1ep)" > "$LOG"
|
||||
setsid nohup /home/infra-ops/ml/.venv/bin/python /home/infra-ops/r49-prep/train_voice_lora.py \
|
||||
--corpus /home/infra-ops/r49-corpus-renamed-unwrapped \
|
||||
--base /home/infra-ops/carriers/Qwen3-0.6B-Base \
|
||||
--seed 4919 --epochs 1 --eval-steps 25 --save-steps 25 \
|
||||
--out "$OUT" >> "$LOG" 2>&1 < /dev/null &
|
||||
echo $! > "$OUT/run.pid"
|
||||
echo "=== $(date -Is) chained 0.6B launched pid $(cat "$OUT/run.pid") -> $LOG"
|
||||
Reference in New Issue
Block a user