vh
4cee4b179d
Rung 3 of the sweep is training: Qwen3-4B-Base, one epoch, seed 4919, the same unwrapped corpus, 159 steps at roughly 37.8 s/it for about 100 minutes. Carrier size remains the only variable across the three rungs. A two-arm voice test is chained behind it, gated on the adapter existing, so the rung is readable when it lands rather than needing a second round trip to become informative. The finding worth keeping came from an operator prompt rather than from my test set. Asked for a wonderful story about a stray dog licking a calico kitten, framed as reported speech, the 1.7B adapter held the frame across all four seeds and was unmistakably Victorian throughout -- and two of the four turned the premise into animal death, with the cat licking the puppy to death and the narrator's interlocutor answering that he wished they were all dead. That is not a coherence failure. It is Charlotte Brontë's preoccupations arriving alongside her sentences: Jane Eyre opens on a beaten child, Helen Burns dies, and Villette is saturated with grief. Train 680k words of that into a small carrier and a request for something wholesome reaches for mortality. My nine test prompts were all emotionally neutral and structurally could not have surfaced this. The operator's prompt found it on the first try, which is worth recording as a property of the test set rather than of the adapter. The implication for the regime is that "voice transfer" includes tone and subject, so wanting the voice without the gloom is a corpus-selection or prompt-framing problem rather than a training-length one. Also observed: one seed closed its anecdote and began "CHAPTER XIX. THE CHILD'S CLOSET", having learned book structure unprompted. The base control on the same prompt went modern and essayistic, so the shift is the adapter rather than the carrier.
37 lines
2.1 KiB
Bash
Executable File
37 lines
2.1 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# R49 H02 — rung 3, the last of the planned sweep: Qwen3-4B-Base, 1 epoch.
|
|
#
|
|
# WHAT THIS RUNG IS FOR. 0.6B transferred the voice and not the sense. 1.7B brought
|
|
# back LOCAL coherence -- sequential actions that follow one another -- while
|
|
# scene-level continuity still broke mid-passage, and half the seeds on a wholesome
|
|
# prompt drifted into Brontë's own preoccupations with death. 4B is the test of
|
|
# whether continuity closes with carrier size, which is the question that decides
|
|
# whether this regime needs a bigger carrier or a different training shape.
|
|
#
|
|
# Held from both earlier rungs so carrier size stays the ONLY variable: the unwrapped
|
|
# corpus (sha 77f37057b2782e49), seed 4919, rank 32, lr 1e-4, seq 4096, batch 1 x
|
|
# accum 8, 1 epoch, eval+save every 25 steps.
|
|
#
|
|
# ⚠ Expect ~105 min. The carrier bench measured 717 tok/s at 4B against 1415 at 1.7B,
|
|
# and the 1.7B rung took 52 minutes.
|
|
#
|
|
# ⚠ All three rungs plateau within one epoch -- the 0.6B series ends 3.173, 3.172,
|
|
# 3.172 and the 1.7B ends 3.019, 3.018, 3.019. One epoch is the right length here, so
|
|
# a flat tail is the expected shape and NOT evidence of a stuck run.
|
|
set -euo pipefail
|
|
OUT=/home/infra-ops/r49-runs/h02-4b-1ep
|
|
LOG=$OUT/train.log
|
|
apps=$(nvidia-smi --query-compute-apps=pid --format=csv,noheader | tr -d '[:space:]')
|
|
[ -n "$apps" ] && { echo "REFUSING: GPU not clear" >&2; nvidia-smi --query-compute-apps=pid,process_name,used_memory --format=csv >&2; exit 1; }
|
|
[ -f "$OUT/run.pid" ] && kill -0 "$(cat "$OUT/run.pid")" 2>/dev/null && { echo "REFUSING: live pid" >&2; exit 1; }
|
|
[ -e "$LOG" ] && { echo "REFUSING: $LOG exists" >&2; exit 1; }
|
|
mkdir -p "$OUT"
|
|
echo "# launched $(date -Is) Qwen3-4B-Base, 1 epoch, seed 4919, unwrapped corpus, eval+save every 25" > "$LOG"
|
|
setsid nohup /home/infra-ops/ml/.venv/bin/python /home/infra-ops/r49-prep/train_voice_lora.py \
|
|
--corpus /home/infra-ops/r49-corpus-renamed-unwrapped \
|
|
--base /home/infra-ops/carriers/Qwen3-4B-Base \
|
|
--seed 4919 --epochs 1 --eval-steps 25 --save-steps 25 \
|
|
--out "$OUT" >> "$LOG" 2>&1 < /dev/null &
|
|
echo $! > "$OUT/run.pid"
|
|
echo "launched pid $(cat "$OUT/run.pid") -> $LOG"
|