vh
935b35ac2e
Operator: "start the 1.7b training." The 0.6B adapter learned the Gutenberg transcription's ~70-character line breaks along with the prose -- its output wrapped at a mid-length-line ratio of 0.85 against the base model's 0.00. That is typography rather than style, and every further rung would have inherited it, so the corpus is reflowed before rung 2 rather than after the sweep. The reflow joins 57,430 of 85,380 paragraph blocks and keeps 27,950. Verse is the hazard a blind join would destroy, so the decision is per block by median line length: blocks whose lines cluster near the wrap width are flowed prose, blocks of consistently short lines keep their breaks. Every kept multi-line block in the sample was genuinely verse with its lineation intact. No line ended in a lone hyphen, so the space-join could not split a word across lines. The acceptance check is content identity -- " ".join(text.split()) byte-identical before and after -- and it passed on all 852 records, proving only whitespace changed. Concrete cost of the old defect: 5.7% of the training budget was newline tokens. The same words pack to 5,210,112 tokens unwrapped against 5,525,504 wrapped. The 1.7B run is live at 159 steps and roughly 18.7 s/it. Everything but the carrier and the corpus is held from the 0.6B run: seed 4919, rank 32, lr 1e-4, seq 4096, batch 1 by accum 8, one epoch, eval and save every 25 steps so the minimum is located rather than assumed. That corpus change is a second variable and it is named as one. A 0.6B-vs-1.7B comparison is descriptive, not attributable, until the chained 0.6B rerun on the same unwrapped corpus lands behind it -- gated on the 1.7B actually producing an adapter, because a chain that fires on failure turns one lost run into two. "Did sense come back at 1.7B" is a within-arm reading and survives the confound; any between-rung delta does not. The original wrapped corpus is untouched, so the 0.6B run's pinned corpus sha 3959036cf851bf62 stays reproducible.
49 lines
2.3 KiB
Bash
Executable File
49 lines
2.3 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# Chained after the 1.7B rung: re-run the 0.6B carrier on the SAME unwrapped corpus.
|
|
#
|
|
# The 1.7B run moved two variables at once -- carrier size AND corpus typography --
|
|
# because the unwrap could not wait if every further rung was to avoid inheriting the
|
|
# Gutenberg line breaks. That makes a 0.6B-vs-1.7B comparison descriptive rather than
|
|
# attributable, which is fine for "did sense come back" (a within-arm reading) and not
|
|
# fine for anything quantitative between rungs.
|
|
#
|
|
# This closes that hole for the price of ~36 minutes on an idle experimental box:
|
|
# 0.6B on the unwrapped corpus, seed 4919, everything else held. Then carrier size is
|
|
# the ONLY difference between this and h02-1p7b-1ep, and the sweep is single-variable
|
|
# again.
|
|
#
|
|
# ⚠ Gated on the 1.7B run having actually produced an adapter. If that run died, this
|
|
# must not quietly start and consume the box; a chain that fires on failure turns one
|
|
# lost run into two.
|
|
set -uo pipefail
|
|
PREV=/home/infra-ops/r49-runs/h02-1p7b-1ep
|
|
OUT=/home/infra-ops/r49-runs/h02-0p6b-1ep-unwrapped
|
|
LOG=$OUT/train.log
|
|
CHAINLOG=/home/infra-ops/r49-runs/chain-0p6b-unwrapped.log
|
|
|
|
exec >> "$CHAINLOG" 2>&1
|
|
echo "=== $(date -Is) chain armed, waiting on $PREV"
|
|
while [ -f "$PREV/run.pid" ] && kill -0 "$(cat "$PREV/run.pid")" 2>/dev/null; do sleep 60; done
|
|
echo "=== $(date -Is) 1.7B run finished"
|
|
|
|
if [ ! -f "$PREV/adapter/adapter_model.safetensors" ]; then
|
|
echo "=== REFUSING to chain: $PREV produced no adapter -- the 1.7B run did not succeed"
|
|
exit 1
|
|
fi
|
|
apps=$(nvidia-smi --query-compute-apps=pid --format=csv,noheader | tr -d '[:space:]')
|
|
if [ -n "$apps" ]; then
|
|
echo "=== REFUSING to chain: GPU not clear"
|
|
nvidia-smi --query-compute-apps=pid,process_name,used_memory --format=csv
|
|
exit 1
|
|
fi
|
|
|
|
mkdir -p "$OUT"
|
|
echo "# launched $(date -Is) Qwen3-0.6B-Base, 1 epoch, seed 4919, UNWRAPPED corpus (single-variable partner to h02-1p7b-1ep)" > "$LOG"
|
|
setsid nohup /home/infra-ops/ml/.venv/bin/python /home/infra-ops/r49-prep/train_voice_lora.py \
|
|
--corpus /home/infra-ops/r49-corpus-renamed-unwrapped \
|
|
--base /home/infra-ops/carriers/Qwen3-0.6B-Base \
|
|
--seed 4919 --epochs 1 --eval-steps 25 --save-steps 25 \
|
|
--out "$OUT" >> "$LOG" 2>&1 < /dev/null &
|
|
echo $! > "$OUT/run.pid"
|
|
echo "=== $(date -Is) chained 0.6B launched pid $(cat "$OUT/run.pid") -> $LOG"
|