vh
935b35ac2e
Operator: "start the 1.7b training." The 0.6B adapter learned the Gutenberg transcription's ~70-character line breaks along with the prose -- its output wrapped at a mid-length-line ratio of 0.85 against the base model's 0.00. That is typography rather than style, and every further rung would have inherited it, so the corpus is reflowed before rung 2 rather than after the sweep. The reflow joins 57,430 of 85,380 paragraph blocks and keeps 27,950. Verse is the hazard a blind join would destroy, so the decision is per block by median line length: blocks whose lines cluster near the wrap width are flowed prose, blocks of consistently short lines keep their breaks. Every kept multi-line block in the sample was genuinely verse with its lineation intact. No line ended in a lone hyphen, so the space-join could not split a word across lines. The acceptance check is content identity -- " ".join(text.split()) byte-identical before and after -- and it passed on all 852 records, proving only whitespace changed. Concrete cost of the old defect: 5.7% of the training budget was newline tokens. The same words pack to 5,210,112 tokens unwrapped against 5,525,504 wrapped. The 1.7B run is live at 159 steps and roughly 18.7 s/it. Everything but the carrier and the corpus is held from the 0.6B run: seed 4919, rank 32, lr 1e-4, seq 4096, batch 1 by accum 8, one epoch, eval and save every 25 steps so the minimum is located rather than assumed. That corpus change is a second variable and it is named as one. A 0.6B-vs-1.7B comparison is descriptive, not attributable, until the chained 0.6B rerun on the same unwrapped corpus lands behind it -- gated on the 1.7B actually producing an adapter, because a chain that fires on failure turns one lost run into two. "Did sense come back at 1.7B" is a within-arm reading and survives the confound; any between-rung delta does not. The original wrapped corpus is untouched, so the 0.6B run's pinned corpus sha 3959036cf851bf62 stays reproducible.
45 lines
2.8 KiB
Bash
Executable File
45 lines
2.8 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# R49 H02 — rung 2 of the carrier sweep: Qwen3-1.7B-Base, 1 epoch.
|
|
#
|
|
# WHY THIS RUN. The 0.6B rung answered the narrow question and the operator's read
|
|
# was the finding: "it's all nonsense, but it sounds like Brontë's nonsense."
|
|
# Voice transferred (curly quotes 1/18 -> 18/18, worksheet collapse 3/18 -> 0/18)
|
|
# while coherence did not. That separation is the premise the whole lightweight
|
|
# author-voice regime rests on, so the live question is which carrier size brings
|
|
# sense back while the voice stays. 1.7B is the next rung; 4B is after it.
|
|
#
|
|
# ⚠ THE CORPUS CHANGED, DELIBERATELY, AND IT IS A SECOND VARIABLE.
|
|
# The 0.6B adapter learned the Gutenberg transcription's ~70-char line breaks along
|
|
# with the prose -- its output wrapped at a mid-length-line ratio of 0.85 against
|
|
# the base model's 0.00. That is typography, not style, and every further rung
|
|
# would inherit it. So this trains on r49-corpus-renamed-unwrapped: same words,
|
|
# reflowed into paragraphs, verified whitespace-only on all 852 records, with verse
|
|
# blocks detected by median line length and their lineation preserved (0 lines
|
|
# ended in a lone hyphen, so the space-join could not split a word).
|
|
# CONSEQUENCE: a 0.6B-vs-1.7B comparison is now DESCRIPTIVE, not attributable --
|
|
# carrier size and corpus typography both moved. "Did sense come back at 1.7B" is a
|
|
# within-arm reading and survives that; any between-rung delta does not. The 0.6B
|
|
# rerun on this same corpus is chained after this run so the clean single-variable
|
|
# comparison exists too.
|
|
#
|
|
# Everything else is held from the 0.6B run: seed 4919, rank 32, lr 1e-4, seq 4096,
|
|
# batch 1 x accum 8, 1 epoch, eval+save every 25 steps so the minimum is LOCATED
|
|
# rather than assumed (the 3-epoch run overfit with save_strategy="no" and left
|
|
# nothing to fall back to).
|
|
set -euo pipefail
|
|
OUT=/home/infra-ops/r49-runs/h02-1p7b-1ep
|
|
LOG=$OUT/train.log
|
|
apps=$(nvidia-smi --query-compute-apps=pid --format=csv,noheader | tr -d '[:space:]')
|
|
[ -n "$apps" ] && { echo "REFUSING: GPU not clear" >&2; nvidia-smi --query-compute-apps=pid,process_name,used_memory --format=csv >&2; exit 1; }
|
|
[ -f "$OUT/run.pid" ] && kill -0 "$(cat "$OUT/run.pid")" 2>/dev/null && { echo "REFUSING: live pid" >&2; exit 1; }
|
|
[ -e "$LOG" ] && { echo "REFUSING: $LOG exists" >&2; exit 1; }
|
|
mkdir -p "$OUT"
|
|
echo "# launched $(date -Is) Qwen3-1.7B-Base, 1 epoch, seed 4919, UNWRAPPED corpus, eval+save every 25" > "$LOG"
|
|
setsid nohup /home/infra-ops/ml/.venv/bin/python /home/infra-ops/r49-prep/train_voice_lora.py \
|
|
--corpus /home/infra-ops/r49-corpus-renamed-unwrapped \
|
|
--base /home/infra-ops/carriers/Qwen3-1.7B-Base \
|
|
--seed 4919 --epochs 1 --eval-steps 25 --save-steps 25 \
|
|
--out "$OUT" >> "$LOG" 2>&1 < /dev/null &
|
|
echo $! > "$OUT/run.pid"
|
|
echo "launched pid $(cat "$OUT/run.pid") -> $LOG"
|