diff --git a/persistent-memory.md b/persistent-memory.md index e72cd46..b4e2d05 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -206,6 +206,8 @@ _As of 2026-09-10 10:25 PT._ ## Recent decisions +- `[2026-09-10]` **R49 rung 2 COMPLETE, and the single-variable carrier effect is clean: 0.6B held-out 3.329 vs 1.7B 3.018, Δ0.311 nats.** Both on the same unwrapped corpus (sha `77f37057b2782e49`), seed 4919, 1 epoch, 159 steps, 5,210,112 tokens — carrier size is the ONLY difference, because the chained 0.6B rerun closed the confound the unwrap opened. ⚠⚠ **DO NOT compare either against the original wrapped-corpus 0.6B run's 3.172 — that comparison is INVALID and reads backwards.** Different corpus means a different held-out set: the wrapped version's 5.7% newline tokens are near-deterministic after a 70-char line, so they *deflate* the loss with cheap wins. Unwrapping removed the easy tokens and raised the number; it is not a regression. ⚠ **Correction to my own earlier claim**: I twice described the 0.6B as "still descending, undertrained" at 3.172 — the series (3.176, 3.173, 3.172, 3.172) shows it FLATTENED. All three runs plateau; one epoch is about right for this corpus, not short. **Three-way eyeball booth** at `http://10.100.10.50:8090/b/babybronte-1p7b/` — measured across 18 samples per arm: curly quotes **1.7B base 0/18 → 1.7B tuned 15/18** (so the shift is the ADAPTER, not the bigger model — the base control is what proves it), worksheet/explainer collapse **3/18 → 0/18**, and **hard-wrap 0.85 → 0.18**, confirming the corpus unwrap carried through into the adapter. **Sense partially returned**: 1.7B produces locally coherent sequential Victorian prose where 0.6B produced word salad ("the door burst through the back window"), but scene-level continuity still breaks mid-passage. ⚠ Curly quotes are slightly LOWER at 1.7B (15/18) than 0.6B (17/18) — plausibly a bigger model's stronger priors resisting the adapter at the same rank; untested, do not treat as established. + - `[2026-09-10]` **R49 rung 2 LAUNCHED: Qwen3-1.7B-Base, 1 epoch, seed 4919, on an UNWRAPPED corpus.** Operator: *"start the 1.7b training."* Live at `gx10:~/r49-runs/h02-1p7b-1ep/`, 159 steps at ~18.7 s/it (~50 min), corpus sha **`77f37057b2782e49`**. A 0.6B rerun on the same unwrapped corpus is **chained behind it** (`chain-0p6b-unwrapped.sh`, gated on the 1.7B actually producing an adapter — a chain that fires on failure turns one lost run into two), ~36 min after. ⚠⚠ **THE CORPUS CHANGED, SO 0.6B-vs-1.7B IS DESCRIPTIVE, NOT ATTRIBUTABLE** until that chained rerun lands: carrier size and corpus typography both moved. *"Did sense come back at 1.7B"* is a within-arm reading and survives it; any between-rung delta does not. **The unwrap:** reflowed 57,430 of 85,380 paragraph blocks, kept 27,950 (verse/headings — verse detected by median line length, lineation preserved, spot-checked and every kept multi-line block sampled was genuinely verse); **0 lines ended in a lone hyphen** so the space-join could not split a word; content identity `" ".join(text.split())` verified byte-identical on all **852 records**, i.e. whitespace-only. Mid-length-line ratio **0.94 → 0.25** (the residual is the preserved verse). ⚠ Concrete cost of the old defect: **5.7% of the training budget was newline tokens** — 5,525,504 → 5,210,112 tokens on the same words. Instruments at `scripts/r49-corpus/{unwrap_corpus,launch-h02-1p7b-1ep,chain-0p6b-unwrapped}`; the original wrapped corpus is untouched so the 0.6B run's pinned sha `3959036cf851bf62` stays reproducible. - `[2026-09-10]` **BabyBronte H02 adapter: the VOICE transferred, the SENSE did not — operator's read, "it's all nonsense, but it sounds like Brontë's nonsense."** Eyeball A/B (NOT the adjudication; nothing here feeds the frozen rule), 9 arbitrary prompts on a deliberate difficulty gradient × 2 arms × 2 seeds, booth at `http://10.100.10.50:8090/b/babybronte-voice/`. Measured across the 18 pairs: **curly quotes 1/18 base → 18/18 tuned**, **math/worksheet collapse 3/18 base → 0/18 tuned**. Given *"The self-checkout machine refused her coupon"* the base 0.6B produced a **quadratic-formula worksheet**; the tuned arm wrote a clerk refusing a customer in Victorian retrospective first person. This is the expected and informative result for the smallest rung — **voice is separable from coherence at 0.6B**, which is the premise the whole lightweight-adapter regime rests on, and the 1.7B/4B rungs are where sense should return. The 1-epoch loss was still descending at step 169 (undertrained, not overfit), so the incoherence is carrier capacity, not training. ⚠ **Corpus-prep defect found: the tuned output is hard-wrapped at ~70 chars** (median mid-length-line ratio 0.85 vs base 0.00) — the Gutenberg source kept its original line breaks and the adapter learned the typography along with the voice. Unwrap to flowing paragraphs before any real use or the next rung learns it too. diff --git a/scripts/r49-corpus/build_booth3.py b/scripts/r49-corpus/build_booth3.py new file mode 100644 index 0000000..baff850 --- /dev/null +++ b/scripts/r49-corpus/build_booth3.py @@ -0,0 +1,133 @@ +"""Render the three-arm carrier comparison into a booth page. + +Three columns, chosen so the page answers two questions at once and neither answer +leans on the other: + + 1.7B base vs 1.7B tuned -- did the ADAPTER do anything at this carrier size, + or is any improvement just the bigger model? + 0.6B tuned vs 1.7B tuned -- did coherence come back as the carrier grew? + +Both tuned arms sit on the SAME unwrapped corpus (sha 77f37057b2782e49), same seed, +same sampler, so carrier size is the only difference between them. The 1.7B base arm +is generated fresh rather than reused, because a control from a different model would +control for nothing. + +Prompts run hardest-first: modern/mundane, then period-neutral, then Victorian- +adjacent. Both seeds of every arm sit in the same cell so within-arm sampling noise +is visible in the same glance as between-arm difference -- if two samples of one arm +differ as much as two arms differ, the page should make that obvious rather than hide +it. +""" +import html +import json +import sys +from collections import defaultdict +from pathlib import Path + +out_dir = Path(sys.argv[1]) +ARMS = [ + ("1p7b-base.jsonl", "1.7B base", "Qwen3-1.7B-Base, no adapter", ""), + ("1p7b-tuned.jsonl", "1.7B tuned", "+ H02 LoRA, 1 epoch, seed 4919", "tuned"), + ("0p6b-tuned.jsonl", "0.6B tuned", "+ H02 LoRA, same corpus & seed", "small"), +] +out_dir.mkdir(parents=True, exist_ok=True) + +def load(p): + d = defaultdict(dict) + for line in Path(p).read_text(encoding="utf-8").splitlines(): + if line.strip(): + r = json.loads(line) + d[r["id"]][r["seed"]] = r + return d + +data = [(lbl, sub, cls, load(out_dir / f)) for f, lbl, sub, cls in ARMS] +ids = sorted(set.intersection(*[set(d) for *_, d in data])) +TIER = {"modern": ("Tier A — modern / mundane", + "Nothing here invites Victorian prose. Brontë in this tier is the adapter's doing."), + "neutral": ("Tier B — period-neutral", + "Could be any century. A voice shift shows without the prompt supplying it."), + "period": ("Tier C — Victorian-adjacent, plainly worded", + "The setting leans period but the diction does not. Easiest tier; weakest evidence.")} +order = {"modern": 0, "neutral": 1, "period": 2} +tier_of = {i: data[0][3][i][list(data[0][3][i])[0]]["tier"] for i in ids} +ids.sort(key=lambda i: (order.get(tier_of[i], 9), i)) + +def cell(by_seed): + return "".join( + f'
seed {s}' + f'

{html.escape((by_seed[s]["continuation"] or "").strip()) or "(empty)"}

' + for s in sorted(by_seed)) + +rows, seen = [], set() +for i in ids: + if tier_of[i] not in seen: + seen.add(tier_of[i]) + title, sub = TIER.get(tier_of[i], (tier_of[i], "")) + rows.append(f'

{html.escape(title)}

{html.escape(sub)}

') + p = data[0][3][i][list(data[0][3][i])[0]]["prompt"] + cols = "".join( + f'

{lbl} {sub}

{cell(d[i])}
' + for lbl, sub, cls, d in data) + rows.append(f'
' + f'{html.escape(i)}{html.escape(p)}
' + f'
{cols}
') + +page = f"""BabyBronte — 1.7B rung + +
+

BabyBronte — rung 2: did the sense come back?

+

The 0.6B rung transferred the voice and not the coherence — "it's all nonsense, but it +sounds like Brontë's nonsense." This is the same nine prompts at 1.7B, with the 0.6B tuned arm beside +it for scale and the 1.7B base arm beside it for control.

+ +
Two questions, two columns each. +1.7B base vs 1.7B tuned asks whether the adapter did anything at this carrier size, or +whether any improvement is just the bigger model. +0.6B tuned vs 1.7B tuned asks whether coherence returned as the carrier grew — and those two +tuned arms sit on the same corpus, same seed, same sampler, so carrier size is the +only difference between them.
+ +
Still an eyeball test, not a result. Two samples per arm is enough +to see whether the gap between columns beats the gap between seeds inside one — and not enough for +anything else. No scoring. The frozen adjudication rule and the Burrows's-Delta instrument are +untouched and nothing here feeds them.

+All arms are base models doing continuation, not instruction-following, so each +prompt is an opening line carried on rather than an instruction to rewrite. +The corpus was unwrapped since the last booth, so the ~70-character hard wrapping +that disfigured the first 0.6B page should be gone from both tuned arms here.
+ +{''.join(rows)} + + +
""" +(out_dir / "index.html").write_text(page, encoding="utf-8") +print(f"wrote {out_dir/'index.html'} ({len(ids)} prompts x 3 arms x 2 seeds)")