diff --git a/persistent-memory.md b/persistent-memory.md index b4e2d05..0bc314e 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -206,6 +206,9 @@ _As of 2026-09-10 10:25 PT._ ## Recent decisions +- `[2026-09-10]` **R49 rung 3 LAUNCHED: Qwen3-4B-Base, 1 epoch, seed 4919, same unwrapped corpus** — `gx10:~/r49-runs/h02-4b-1ep/`, 159 steps at ~37.8 s/it (**~100 min**), 252 adapted modules (vs 196 at 0.6B/1.7B). Last rung of the planned sweep; it tests whether **scene-level continuity** closes with carrier size. A two-arm voice test (4B base + 4B tuned, the nine prompts plus the operator's Abernathy frame) is **chained behind it**, gated on the adapter existing. +- `[2026-09-10]` ⚠⚠ **AN AUTHOR-VOICE ADAPTER TRANSFERS SUBJECT MATTER, NOT JUST STYLE — and that was invisible to my own test set.** Operator prompt: *"Mr. Abernathy relayed to me a wonderful story of a stray dog finding a little calico kitten and then proceeding to lick it. He said ""*. At 1.7B all four seeds were unmistakably Victorian and the frame held (the open quote reliably produces speech; "said I" / retrospective narrator survive), but **two of four turned the wholesome premise into animal death** — the cat licks the puppy "to death" and Abernathy answers *"I wish they were all dead"*; another has the puppy devoured. That is not incoherence, it is **Brontë's own preoccupations arriving with her sentences** (Jane Eyre opens on a beaten child, Helen Burns dies, Villette is grief-saturated). ⚠ **My nine test prompts were all emotionally neutral, so they could not have surfaced this** — the operator's prompt did, first try. Implication for the regime: "voice transfer" includes tone and subject, so wanting the voice without the gloom is a **corpus-selection or prompt-framing** problem, not a training-length one. Also observed: one seed closed its anecdote and emitted `CHAPTER XIX. THE CHILD'S CLOSET.` — it learned book structure unprompted. Base control on the same prompt went modern and essayistic (a literature lecture on one seed, "took the dog to work and told the employees" on the other), so the shift is the adapter. + - `[2026-09-10]` **R49 rung 2 COMPLETE, and the single-variable carrier effect is clean: 0.6B held-out 3.329 vs 1.7B 3.018, Δ0.311 nats.** Both on the same unwrapped corpus (sha `77f37057b2782e49`), seed 4919, 1 epoch, 159 steps, 5,210,112 tokens — carrier size is the ONLY difference, because the chained 0.6B rerun closed the confound the unwrap opened. ⚠⚠ **DO NOT compare either against the original wrapped-corpus 0.6B run's 3.172 — that comparison is INVALID and reads backwards.** Different corpus means a different held-out set: the wrapped version's 5.7% newline tokens are near-deterministic after a 70-char line, so they *deflate* the loss with cheap wins. Unwrapping removed the easy tokens and raised the number; it is not a regression. ⚠ **Correction to my own earlier claim**: I twice described the 0.6B as "still descending, undertrained" at 3.172 — the series (3.176, 3.173, 3.172, 3.172) shows it FLATTENED. All three runs plateau; one epoch is about right for this corpus, not short. **Three-way eyeball booth** at `http://10.100.10.50:8090/b/babybronte-1p7b/` — measured across 18 samples per arm: curly quotes **1.7B base 0/18 → 1.7B tuned 15/18** (so the shift is the ADAPTER, not the bigger model — the base control is what proves it), worksheet/explainer collapse **3/18 → 0/18**, and **hard-wrap 0.85 → 0.18**, confirming the corpus unwrap carried through into the adapter. **Sense partially returned**: 1.7B produces locally coherent sequential Victorian prose where 0.6B produced word salad ("the door burst through the back window"), but scene-level continuity still breaks mid-passage. ⚠ Curly quotes are slightly LOWER at 1.7B (15/18) than 0.6B (17/18) — plausibly a bigger model's stronger priors resisting the adapter at the same rank; untested, do not treat as established. - `[2026-09-10]` **R49 rung 2 LAUNCHED: Qwen3-1.7B-Base, 1 epoch, seed 4919, on an UNWRAPPED corpus.** Operator: *"start the 1.7b training."* Live at `gx10:~/r49-runs/h02-1p7b-1ep/`, 159 steps at ~18.7 s/it (~50 min), corpus sha **`77f37057b2782e49`**. A 0.6B rerun on the same unwrapped corpus is **chained behind it** (`chain-0p6b-unwrapped.sh`, gated on the 1.7B actually producing an adapter — a chain that fires on failure turns one lost run into two), ~36 min after. ⚠⚠ **THE CORPUS CHANGED, SO 0.6B-vs-1.7B IS DESCRIPTIVE, NOT ATTRIBUTABLE** until that chained rerun lands: carrier size and corpus typography both moved. *"Did sense come back at 1.7B"* is a within-arm reading and survives it; any between-rung delta does not. **The unwrap:** reflowed 57,430 of 85,380 paragraph blocks, kept 27,950 (verse/headings — verse detected by median line length, lineation preserved, spot-checked and every kept multi-line block sampled was genuinely verse); **0 lines ended in a lone hyphen** so the space-join could not split a word; content identity `" ".join(text.split())` verified byte-identical on all **852 records**, i.e. whitespace-only. Mid-length-line ratio **0.94 → 0.25** (the residual is the preserved verse). ⚠ Concrete cost of the old defect: **5.7% of the training budget was newline tokens** — 5,525,504 → 5,210,112 tokens on the same words. Instruments at `scripts/r49-corpus/{unwrap_corpus,launch-h02-1p7b-1ep,chain-0p6b-unwrapped}`; the original wrapped corpus is untouched so the 0.6B run's pinned sha `3959036cf851bf62` stays reproducible. diff --git a/scripts/r49-corpus/abernathy.json b/scripts/r49-corpus/abernathy.json new file mode 100644 index 0000000..ca4d056 --- /dev/null +++ b/scripts/r49-corpus/abernathy.json @@ -0,0 +1 @@ +[{"id": "abernathy", "tier": "neutral", "prompt": "Mr. Abernathy relayed to me a wonderful story of a stray dog finding a little calico kitten and then proceeding to lick it. He said \""}] \ No newline at end of file diff --git a/scripts/r49-corpus/chain-4b-voicetest.sh b/scripts/r49-corpus/chain-4b-voicetest.sh new file mode 100755 index 0000000..5ec2d19 --- /dev/null +++ b/scripts/r49-corpus/chain-4b-voicetest.sh @@ -0,0 +1,30 @@ +#!/usr/bin/env bash +# Chained after the 4B rung: generate its voice arms so the rung is readable the +# moment it lands, rather than needing a second round trip to become informative. +# +# Two arms, because the base control is what earns the claim: at 1.7B the adapter +# took curly quotes from 0/18 to 15/18 and that number only means something because +# the 0/18 came from the SAME carrier. Same nine prompts, same two seeds, same +# sampler as both earlier rungs, plus the operator's own Abernathy frame prompt. +# +# ⚠ Gated on the 4B run having actually produced an adapter -- a chain that fires on +# failure turns one lost run into two. +set -uo pipefail +PREV=/home/infra-ops/r49-runs/h02-4b-1ep +OUT=/home/infra-ops/r49-runs/voice-test-4b +V=/home/infra-ops/ml/.venv/bin/python +exec >> /home/infra-ops/r49-runs/chain-4b-voicetest.log 2>&1 +echo "=== $(date -Is) armed, waiting on $PREV" +while [ -f "$PREV/run.pid" ] && kill -0 "$(cat "$PREV/run.pid")" 2>/dev/null; do sleep 60; done +echo "=== $(date -Is) 4B run finished" +[ -f "$PREV/adapter/adapter_model.safetensors" ] || { echo "=== REFUSING: no adapter, the 4B run did not succeed"; exit 1; } +cd /home/infra-ops/r49-prep +for spec in "4b-base::" "4b-tuned::--adapter $PREV/adapter"; do + arm="${spec%%::*}"; extra="${spec##*::}" + for pf in voice_prompts.json abernathy.json; do + tag=$([ "$pf" = "abernathy.json" ] && echo "-abernathy" || echo "") + $V gen_voice_test.py --base /home/infra-ops/carriers/Qwen3-4B-Base $extra \ + --arm "$arm" --prompts "$pf" --out "$OUT/${arm}${tag}.jsonl" + done +done +echo "=== $(date -Is) voice arms written to $OUT" diff --git a/scripts/r49-corpus/launch-h02-4b-1ep.sh b/scripts/r49-corpus/launch-h02-4b-1ep.sh new file mode 100755 index 0000000..a381352 --- /dev/null +++ b/scripts/r49-corpus/launch-h02-4b-1ep.sh @@ -0,0 +1,36 @@ +#!/usr/bin/env bash +# R49 H02 — rung 3, the last of the planned sweep: Qwen3-4B-Base, 1 epoch. +# +# WHAT THIS RUNG IS FOR. 0.6B transferred the voice and not the sense. 1.7B brought +# back LOCAL coherence -- sequential actions that follow one another -- while +# scene-level continuity still broke mid-passage, and half the seeds on a wholesome +# prompt drifted into Brontë's own preoccupations with death. 4B is the test of +# whether continuity closes with carrier size, which is the question that decides +# whether this regime needs a bigger carrier or a different training shape. +# +# Held from both earlier rungs so carrier size stays the ONLY variable: the unwrapped +# corpus (sha 77f37057b2782e49), seed 4919, rank 32, lr 1e-4, seq 4096, batch 1 x +# accum 8, 1 epoch, eval+save every 25 steps. +# +# ⚠ Expect ~105 min. The carrier bench measured 717 tok/s at 4B against 1415 at 1.7B, +# and the 1.7B rung took 52 minutes. +# +# ⚠ All three rungs plateau within one epoch -- the 0.6B series ends 3.173, 3.172, +# 3.172 and the 1.7B ends 3.019, 3.018, 3.019. One epoch is the right length here, so +# a flat tail is the expected shape and NOT evidence of a stuck run. +set -euo pipefail +OUT=/home/infra-ops/r49-runs/h02-4b-1ep +LOG=$OUT/train.log +apps=$(nvidia-smi --query-compute-apps=pid --format=csv,noheader | tr -d '[:space:]') +[ -n "$apps" ] && { echo "REFUSING: GPU not clear" >&2; nvidia-smi --query-compute-apps=pid,process_name,used_memory --format=csv >&2; exit 1; } +[ -f "$OUT/run.pid" ] && kill -0 "$(cat "$OUT/run.pid")" 2>/dev/null && { echo "REFUSING: live pid" >&2; exit 1; } +[ -e "$LOG" ] && { echo "REFUSING: $LOG exists" >&2; exit 1; } +mkdir -p "$OUT" +echo "# launched $(date -Is) Qwen3-4B-Base, 1 epoch, seed 4919, unwrapped corpus, eval+save every 25" > "$LOG" +setsid nohup /home/infra-ops/ml/.venv/bin/python /home/infra-ops/r49-prep/train_voice_lora.py \ + --corpus /home/infra-ops/r49-corpus-renamed-unwrapped \ + --base /home/infra-ops/carriers/Qwen3-4B-Base \ + --seed 4919 --epochs 1 --eval-steps 25 --save-steps 25 \ + --out "$OUT" >> "$LOG" 2>&1 < /dev/null & +echo $! > "$OUT/run.pid" +echo "launched pid $(cat "$OUT/run.pid") -> $LOG"