From 25eb65e23b5c966474b20d7ebe7dc2ad2b2e5b7f Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Fri, 11 Sep 2026 05:36:11 -0700 Subject: [PATCH] Complete the R49 carrier sweep: 4B closes the continuity gap and overfits The three rungs now sit on the same unwrapped corpus, seed, step count and token count, so carrier size is the only variable. Held-out loss reads 3.329 at 0.6B, 3.018 at 1.7B and 2.814 at 4B -- deltas of 0.311 and then 0.204, diminishing but still real. 4B answers the question the rung existed for. Scene-level continuity holds: on the office-kitchen prompt it produces a named character with motivated dialogue, a spatial layout the narrator navigates, and a physical description, all in one passage, where 1.7B wrote pretty but eventless prose about opening doors and looking at stars. On the letter prompt it opens the letter, promises to quote it, and then quotes it across a paragraph break. Voice saturation is also the best of any rung: curly quotes 17 of 18 against its own base arm's 1 of 18, and collapse 0 of 18 against 4 of 18. Two findings that change earlier conclusions. 4B is the first rung to overfit inside one epoch. Its series runs 2.832, 2.816, 2.814, 2.820, 2.824, 2.825, 2.825 -- a minimum around step 75 and then a turn. Both smaller rungs plateaued without turning, so the optimal epoch count shrinks as the carrier grows and my earlier "one epoch is right for this corpus" holds only for the small end. The consequence is operational: the adapter directory holds the end-of-run weights at 2.825 rather than the step-75 best at 2.814, and it exists as a recoverable checkpoint only because save_steps was set. The voice arms were cut from the end-of-run adapter, so the booth understates 4B slightly. The tone-override also appears to close. On the operator's frame prompt asking for a wonderful story, 1.7B held the frame on every seed but killed the animals on two of four; 4B kept them alive on both seeds, and one of them had the narrator doubt the story he was told and supply a parallel childhood memory to explain the doubt. That is a narrator with an interior position on the tale. Two samples per arm, so directionally right rather than established. --- persistent-memory.md | 4 + scripts/r49-corpus/build_booth4b.py | 160 ++++++++++++++++++++++++++++ 2 files changed, 164 insertions(+) create mode 100644 scripts/r49-corpus/build_booth4b.py diff --git a/persistent-memory.md b/persistent-memory.md index 0bc314e..5f1d260 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -206,6 +206,10 @@ _As of 2026-09-10 10:25 PT._ ## Recent decisions +- `[2026-09-11]` **R49 SWEEP COMPLETE — 4B closes the continuity gap, and the carrier ladder is clean: 3.329 → 3.018 → 2.814 held-out** (0.6B / 1.7B / 4B, all on the same unwrapped corpus sha `77f37057b2782e49`, seed 4919, 159 steps, 5,210,112 tokens — carrier size the only variable). Deltas **0.311 then 0.204**: diminishing but still real. Booth: `http://10.100.10.50:8090/b/babybronte-4b/`. **4B tuned has the best voice saturation of any rung — curly quotes 17/18 against its own base arm's 1/18, collapse 0/18 against 4/18** — and, the thing the rung existed to test, **scene-level continuity HOLDS**: it produces a named character with motivated dialogue, a navigable spatial layout and a physical description in one passage, where 1.7B wrote pretty but eventless prose (opening doors, looking at stars). On the letter prompt it opens the letter, promises to quote it, and then actually quotes it across a paragraph break. +- `[2026-09-11]` ⚠⚠ **4B is the FIRST rung to OVERFIT inside one epoch, which inverts my earlier "one epoch is right for this corpus" call.** Series 2.832 · 2.816 · **2.814** · 2.820 · 2.824 · 2.825 · 2.825 — minimum at ~step 75, then it TURNS and settles worse. 0.6B and 1.7B both plateaued with no turn, so **the optimal epoch count shrinks as the carrier grows** — 4B wants roughly half an epoch. ⚠ **Consequence: the shipped `adapter/` at `h02-4b-1ep/` is NOT the best checkpoint** (it is the end-of-run 2.825); the step-75 checkpoint at 2.814 is, and it exists only because `save_steps=25` was set. The voice test used the end-of-run adapter, so the booth understates 4B by ~0.011 nats. Re-cut the arms off the step-75 checkpoint before any adjudication. +- `[2026-09-11]` **The tone-override appears to close at 4B too.** On the operator's Abernathy frame prompt ("a *wonderful* story"), 1.7B held the frame on every seed but **2 of 4 killed the animals anyway**; 4B kept them alive on **2 of 2** and one seed did something new — the narrator *doubts Abernathy's story* ("I felt sure the thing was a lie"), then supplies a parallel childhood memory of his own puppy and his sister's kitten to explain the doubt. That is a narrator with an interior position on the tale being told. ⚠ n=2 per arm; directionally right, not established. + - `[2026-09-10]` **R49 rung 3 LAUNCHED: Qwen3-4B-Base, 1 epoch, seed 4919, same unwrapped corpus** — `gx10:~/r49-runs/h02-4b-1ep/`, 159 steps at ~37.8 s/it (**~100 min**), 252 adapted modules (vs 196 at 0.6B/1.7B). Last rung of the planned sweep; it tests whether **scene-level continuity** closes with carrier size. A two-arm voice test (4B base + 4B tuned, the nine prompts plus the operator's Abernathy frame) is **chained behind it**, gated on the adapter existing. - `[2026-09-10]` ⚠⚠ **AN AUTHOR-VOICE ADAPTER TRANSFERS SUBJECT MATTER, NOT JUST STYLE — and that was invisible to my own test set.** Operator prompt: *"Mr. Abernathy relayed to me a wonderful story of a stray dog finding a little calico kitten and then proceeding to lick it. He said ""*. At 1.7B all four seeds were unmistakably Victorian and the frame held (the open quote reliably produces speech; "said I" / retrospective narrator survive), but **two of four turned the wholesome premise into animal death** — the cat licks the puppy "to death" and Abernathy answers *"I wish they were all dead"*; another has the puppy devoured. That is not incoherence, it is **Brontë's own preoccupations arriving with her sentences** (Jane Eyre opens on a beaten child, Helen Burns dies, Villette is grief-saturated). ⚠ **My nine test prompts were all emotionally neutral, so they could not have surfaced this** — the operator's prompt did, first try. Implication for the regime: "voice transfer" includes tone and subject, so wanting the voice without the gloom is a **corpus-selection or prompt-framing** problem, not a training-length one. Also observed: one seed closed its anecdote and emitted `CHAPTER XIX. THE CHILD'S CLOSET.` — it learned book structure unprompted. Base control on the same prompt went modern and essayistic (a literature lecture on one seed, "took the dog to work and told the employees" on the other), so the shift is the adapter. diff --git a/scripts/r49-corpus/build_booth4b.py b/scripts/r49-corpus/build_booth4b.py new file mode 100644 index 0000000..50b8621 --- /dev/null +++ b/scripts/r49-corpus/build_booth4b.py @@ -0,0 +1,160 @@ +"""Render rung 3 into a booth page: 4B base, 4B tuned, 1.7B tuned. + +Column choice, same logic as the 1.7B page. The 4B base arm is the control that +earns any claim about the adapter -- at 1.7B the shift read 0/18 to 15/18 on curly +quotes, and that number only meant something because the 0/18 came from the same +carrier. The 1.7B tuned arm is the rung below, on the identical corpus and seed, so +carrier size is the only difference between the two tuned columns. + +The operator's Abernathy prompt gets its own section at the bottom, across four +seeds. It belongs apart from the nine because it is doing something the nine cannot: +it carries an emotional frame ("a wonderful story"), and at 1.7B half the seeds +overrode that frame and killed the animals anyway -- Brontë's preoccupations +arriving with her sentences. Whether a bigger carrier holds the frame is the open +question that section exists to answer. +""" +import html +import json +import statistics +import sys +from collections import defaultdict +from pathlib import Path + +D = Path(sys.argv[1]) +ARMS = [("4b-base.jsonl", "4B base", "Qwen3-4B-Base, no adapter", ""), + ("4b-tuned.jsonl", "4B tuned", "+ H02 LoRA, 1 epoch, seed 4919", "tuned"), + ("1p7b-tuned.jsonl", "1.7B tuned", "the rung below, same corpus & seed", "small")] +AB = [("4b-base-abernathy.jsonl", "4B base", ""), ("4b-tuned-abernathy.jsonl", "4B tuned", "tuned")] + + +def load(p): + d = defaultdict(dict) + for line in Path(p).read_text(encoding="utf-8").splitlines(): + if line.strip(): + r = json.loads(line) + d[r["id"]][r["seed"]] = r + return d + + +def metrics(p): + rs = [json.loads(l) for l in Path(p).read_text(encoding="utf-8").splitlines() if l.strip()] + wrap, curly, junk = [], 0, 0 + for r in rs: + c = r["continuation"] + lines = [l for l in c.split("\n") if l.strip()] + wrap.append(sum(1 for l in lines if 20 < len(l) < 78) / max(1, len(lines))) + curly += ("“" in c or "’" in c) + junk += ("$" in c or "\\dfrac" in c or "Answer:" in c) + return len(rs), curly, junk, statistics.median(wrap) + + +data = [(lbl, sub, cls, load(D / f), metrics(D / f)) for f, lbl, sub, cls in ARMS] +ids = sorted(set.intersection(*[set(d) for *_, d, _ in data])) +TIER = {"modern": ("Tier A — modern / mundane", + "Nothing here invites Victorian prose. Brontë in this tier is the adapter's doing."), + "neutral": ("Tier B — period-neutral", + "Could be any century. A voice shift shows without the prompt supplying it."), + "period": ("Tier C — Victorian-adjacent, plainly worded", + "The setting leans period but the diction does not. Easiest tier; weakest evidence.")} +order = {"modern": 0, "neutral": 1, "period": 2} +tier_of = {i: data[0][3][i][list(data[0][3][i])[0]]["tier"] for i in ids} +ids.sort(key=lambda i: (order.get(tier_of[i], 9), i)) + + +def cell(by_seed): + return "".join( + f'
seed {s}

' + f'{html.escape((by_seed[s]["continuation"] or "").strip()) or "(empty)"}

' + for s in sorted(by_seed)) + + +tbl = "".join( + f"{lbl}{c}/{n}{j}/{n}{w:.2f}" + for lbl, _s, _c, _d, (n, c, j, w) in data) + +rows, seen = [], set() +for i in ids: + if tier_of[i] not in seen: + seen.add(tier_of[i]) + t, sb = TIER.get(tier_of[i], (tier_of[i], "")) + rows.append(f'

{html.escape(t)}

{html.escape(sb)}

') + pr = data[0][3][i][list(data[0][3][i])[0]]["prompt"] + cols = "".join(f'

{lbl} {sub}

{cell(d[i])}
' + for lbl, sub, cls, d, _ in data) + rows.append(f'
{html.escape(i)}' + f'{html.escape(pr)}
{cols}
') + +ab_html = "" +if all((D / f).exists() for f, _, _ in AB): + ab_data = [(lbl, cls, load(D / f)) for f, lbl, cls in AB] + pid = list(ab_data[0][2])[0] + pr = ab_data[0][2][pid][list(ab_data[0][2][pid])[0]]["prompt"] + cols = "".join(f'

{lbl}

{cell(d[pid])}
' + for lbl, cls, d in ab_data) + ab_html = (f'

The operator\'s frame prompt

Reported speech with an open ' + f'quotation mark, and an emotional frame the nine prompts do not carry. At 1.7B the ' + f'frame held on every seed and half of them killed the animals anyway.

' + f'
{html.escape(pr)}
' + f'
{cols}
') + +page = f"""BabyBronte — 4B rung + +
+

BabyBronte — rung 3: does the thread hold?

+

0.6B gave the voice and not the sense. 1.7B brought back sentence-to-sentence +coherence but still lost the thread inside a passage. 4B is the last rung of the planned sweep, and +the open question is scene-level continuity.

+ +{tbl}
armcurly quotesworksheet / explainer collapsehard-wrap ratio
+ +
The base column is the control that earns the claim. Any difference +between the two tuned columns is carrier size and nothing else — identical corpus +(sha 77f37057b2782e49), seed, sampler and step count. Any difference between 4B base and 4B tuned is +the adapter and nothing else.
+ +
Still an eyeball test. Two samples per arm is enough to see whether +the gap between columns beats the gap between seeds inside one, and not enough for anything else. No +scoring; the frozen adjudication rule and the Burrows's-Delta instrument are untouched and nothing +here feeds them. All arms are base models doing continuation, so each prompt is an +opening line carried on rather than an instruction to rewrite.
+ +{''.join(rows)} +{ab_html} + + +
""" +(D / "index.html").write_text(page, encoding="utf-8") +print(f"wrote {D/'index.html'} ({len(ids)} prompts x 3 arms" + (", + frame prompt" if ab_html else "") + ")")