vh
e15c5ee5ea
Trained the same corpus onto Qwen3-4B instruct rather than -Base, with seed, steps and token count held so the carrier is the only variable. The chain picked checkpoint-150 by loss automatically, applying the lesson that cost a re-cut on 4B-Base. The central risk did not materialise. The assistant prior did not block the voice: curly quotes land at 16 of 18, identical to the 4B-Base tuned arm, against 1 of 18 on the unadapted control, and task-leak is 0 of 18 where the base carrier leaked 4. Instruction-following also survived raw-text training -- 10 of 10 on-beat through the chat template, the same as the untuned control. The cost is length discipline rather than comprehension. In-band dropped from 10 of 10 to 6 of 10 and the median went from 124 to 140 words. Training on Victorian prose made it wordier, which is a soft degradation and not a break. Held-out sits at 2.908 against 4B-Base's 2.814, and it plateaus without turning where the base carrier overfit at step 75. The assistant prior competes for capacity, so the instruct carrier absorbs less rather than overfitting more. What raw-continuation training does not fix is the plot furniture. The tuned instruct arm renders the beat and then drags the referent -- "He licked her clean... my master thus, my husband thus", turning the dog into a man, because the corpus is about masters and husbands. Another beat ran to 247 words and gave the narrator a list of duties. That is precisely what instruction-pair training addresses, since pairs teach render-this-and-stop where continuation teaches keep-writing. The probe de-risks the instruction-pair path without substituting for it. One metric note against future misreading: ran_on reports 10 of 10 on both arms and is uninformative on this job, because a single paragraph contains no blank line for it to find.
51 lines
2.8 KiB
Bash
51 lines
2.8 KiB
Bash
#!/usr/bin/env bash
|
|
# Chained after the instruct probe: run the tests that decide whether it worked.
|
|
#
|
|
# ⚠ PICKS THE BEST CHECKPOINT, not the end-of-run adapter. 4B-Base turned at step 75
|
|
# and its shipped adapter was the WORSE weights; that cost a re-cut. Applying the
|
|
# lesson: read loss-series.json, take the argmin, and say which one was chosen.
|
|
#
|
|
# Four arms, and the CONTROL is the one that makes the rest readable:
|
|
# 1. tuned + chat template -- did instruction-following SURVIVE the training?
|
|
# 2. UNTUNED + chat template -- the control. Panel B used the 27B gen seat, so
|
|
# without a 4B control any degradation is unattributable between "the adapter
|
|
# broke it" and "a 4B is just weaker at this than a 27B".
|
|
# 3. tuned + raw continuation, the 9 voice prompts -- comparable to every rung.
|
|
# 4. tuned + the plot-furniture prompts -- the operator's point that a completion
|
|
# carrier reaches for Brontë's devices instead of Skaldsong's. If the instruct
|
|
# prior resists the gloom-override, that shows up here.
|
|
set -uo pipefail
|
|
PREV=/home/infra-ops/r49-runs/h02-4b-instruct-1ep
|
|
OUT=/home/infra-ops/r49-runs/instruct-probe
|
|
V=/home/infra-ops/ml/.venv/bin/python
|
|
BASE=/home/infra-ops/carriers/Qwen3-4B-Instruct
|
|
exec >> /home/infra-ops/r49-runs/chain-instruct-probe.log 2>&1
|
|
echo "=== $(date -Is) armed, waiting on $PREV"
|
|
while [ -f "$PREV/run.pid" ] && kill -0 "$(cat "$PREV/run.pid")" 2>/dev/null; do sleep 60; done
|
|
echo "=== $(date -Is) training finished"
|
|
[ -f "$PREV/adapter/adapter_model.safetensors" ] || { echo "=== REFUSING: no adapter"; exit 1; }
|
|
|
|
CK=$($V - <<'PY'
|
|
import json
|
|
d = json.load(open("/home/infra-ops/r49-runs/h02-4b-instruct-1ep/loss-series.json"))
|
|
rows = [(r["step"], r["eval_loss"]) for r in
|
|
(d if isinstance(d, list) else d.get("log_history", d.get("series", [])))
|
|
if isinstance(r, dict) and r.get("eval_loss") is not None]
|
|
best = min(rows, key=lambda r: r[1])
|
|
import os
|
|
p = f"/home/infra-ops/r49-runs/h02-4b-instruct-1ep/checkpoints/checkpoint-{best[0]}"
|
|
print(p if os.path.isdir(p) else "/home/infra-ops/r49-runs/h02-4b-instruct-1ep/adapter")
|
|
PY
|
|
)
|
|
echo "=== best checkpoint selected: $CK"
|
|
cd /home/infra-ops/r49-prep
|
|
$V gen_beats_chat.py --base "$BASE" --adapter "$CK" --beats beats.json --arm instruct-tuned-chat --out "$OUT/tuned-chat.jsonl"
|
|
$V gen_beats_chat.py --base "$BASE" --beats beats.json --arm instruct-untuned-chat --out "$OUT/untuned-chat.jsonl"
|
|
$V gen_voice_test.py --base "$BASE" --adapter "$CK" --arm instruct-tuned --prompts voice_prompts.json --out "$OUT/voice.jsonl"
|
|
for pf in abernathy.json letter.json; do
|
|
$V gen_voice_test.py --base "$BASE" --adapter "$CK" --arm instruct-tuned \
|
|
--prompts "$pf" --seeds 1234 5678 4242 --max-new-tokens 320 \
|
|
--out "$OUT/${pf%.json}.jsonl"
|
|
done
|
|
echo "=== $(date -Is) instruct probe arms written to $OUT"
|