Complete the R49 carrier sweep: 4B closes the continuity gap and overfits
The three rungs now sit on the same unwrapped corpus, seed, step count and token count, so carrier size is the only variable. Held-out loss reads 3.329 at 0.6B, 3.018 at 1.7B and 2.814 at 4B -- deltas of 0.311 and then 0.204, diminishing but still real. 4B answers the question the rung existed for. Scene-level continuity holds: on the office-kitchen prompt it produces a named character with motivated dialogue, a spatial layout the narrator navigates, and a physical description, all in one passage, where 1.7B wrote pretty but eventless prose about opening doors and looking at stars. On the letter prompt it opens the letter, promises to quote it, and then quotes it across a paragraph break. Voice saturation is also the best of any rung: curly quotes 17 of 18 against its own base arm's 1 of 18, and collapse 0 of 18 against 4 of 18. Two findings that change earlier conclusions. 4B is the first rung to overfit inside one epoch. Its series runs 2.832, 2.816, 2.814, 2.820, 2.824, 2.825, 2.825 -- a minimum around step 75 and then a turn. Both smaller rungs plateaued without turning, so the optimal epoch count shrinks as the carrier grows and my earlier "one epoch is right for this corpus" holds only for the small end. The consequence is operational: the adapter directory holds the end-of-run weights at 2.825 rather than the step-75 best at 2.814, and it exists as a recoverable checkpoint only because save_steps was set. The voice arms were cut from the end-of-run adapter, so the booth understates 4B slightly. The tone-override also appears to close. On the operator's frame prompt asking for a wonderful story, 1.7B held the frame on every seed but killed the animals on two of four; 4B kept them alive on both seeds, and one of them had the narrator doubt the story he was told and supply a parallel childhood memory to explain the doubt. That is a narrator with an interior position on the tale. Two samples per arm, so directionally right rather than established.
This commit is contained in:
@@ -206,6 +206,10 @@ _As of 2026-09-10 10:25 PT._
|
||||
|
||||
## Recent decisions
|
||||
|
||||
- `[2026-09-11]` **R49 SWEEP COMPLETE — 4B closes the continuity gap, and the carrier ladder is clean: 3.329 → 3.018 → 2.814 held-out** (0.6B / 1.7B / 4B, all on the same unwrapped corpus sha `77f37057b2782e49`, seed 4919, 159 steps, 5,210,112 tokens — carrier size the only variable). Deltas **0.311 then 0.204**: diminishing but still real. Booth: `http://10.100.10.50:8090/b/babybronte-4b/`. **4B tuned has the best voice saturation of any rung — curly quotes 17/18 against its own base arm's 1/18, collapse 0/18 against 4/18** — and, the thing the rung existed to test, **scene-level continuity HOLDS**: it produces a named character with motivated dialogue, a navigable spatial layout and a physical description in one passage, where 1.7B wrote pretty but eventless prose (opening doors, looking at stars). On the letter prompt it opens the letter, promises to quote it, and then actually quotes it across a paragraph break.
|
||||
- `[2026-09-11]` ⚠⚠ **4B is the FIRST rung to OVERFIT inside one epoch, which inverts my earlier "one epoch is right for this corpus" call.** Series 2.832 · 2.816 · **2.814** · 2.820 · 2.824 · 2.825 · 2.825 — minimum at ~step 75, then it TURNS and settles worse. 0.6B and 1.7B both plateaued with no turn, so **the optimal epoch count shrinks as the carrier grows** — 4B wants roughly half an epoch. ⚠ **Consequence: the shipped `adapter/` at `h02-4b-1ep/` is NOT the best checkpoint** (it is the end-of-run 2.825); the step-75 checkpoint at 2.814 is, and it exists only because `save_steps=25` was set. The voice test used the end-of-run adapter, so the booth understates 4B by ~0.011 nats. Re-cut the arms off the step-75 checkpoint before any adjudication.
|
||||
- `[2026-09-11]` **The tone-override appears to close at 4B too.** On the operator's Abernathy frame prompt ("a *wonderful* story"), 1.7B held the frame on every seed but **2 of 4 killed the animals anyway**; 4B kept them alive on **2 of 2** and one seed did something new — the narrator *doubts Abernathy's story* ("I felt sure the thing was a lie"), then supplies a parallel childhood memory of his own puppy and his sister's kitten to explain the doubt. That is a narrator with an interior position on the tale being told. ⚠ n=2 per arm; directionally right, not established.
|
||||
|
||||
- `[2026-09-10]` **R49 rung 3 LAUNCHED: Qwen3-4B-Base, 1 epoch, seed 4919, same unwrapped corpus** — `gx10:~/r49-runs/h02-4b-1ep/`, 159 steps at ~37.8 s/it (**~100 min**), 252 adapted modules (vs 196 at 0.6B/1.7B). Last rung of the planned sweep; it tests whether **scene-level continuity** closes with carrier size. A two-arm voice test (4B base + 4B tuned, the nine prompts plus the operator's Abernathy frame) is **chained behind it**, gated on the adapter existing.
|
||||
- `[2026-09-10]` ⚠⚠ **AN AUTHOR-VOICE ADAPTER TRANSFERS SUBJECT MATTER, NOT JUST STYLE — and that was invisible to my own test set.** Operator prompt: *"Mr. Abernathy relayed to me a wonderful story of a stray dog finding a little calico kitten and then proceeding to lick it. He said ""*. At 1.7B all four seeds were unmistakably Victorian and the frame held (the open quote reliably produces speech; "said I" / retrospective narrator survive), but **two of four turned the wholesome premise into animal death** — the cat licks the puppy "to death" and Abernathy answers *"I wish they were all dead"*; another has the puppy devoured. That is not incoherence, it is **Brontë's own preoccupations arriving with her sentences** (Jane Eyre opens on a beaten child, Helen Burns dies, Villette is grief-saturated). ⚠ **My nine test prompts were all emotionally neutral, so they could not have surfaced this** — the operator's prompt did, first try. Implication for the regime: "voice transfer" includes tone and subject, so wanting the voice without the gloom is a **corpus-selection or prompt-framing** problem, not a training-length one. Also observed: one seed closed its anecdote and emitted `CHAPTER XIX. THE CHILD'S CLOSET.` — it learned book structure unprompted. Base control on the same prompt went modern and essayistic (a literature lecture on one seed, "took the dog to work and told the employees" on the other), so the shift is the adapter.
|
||||
|
||||
|
||||
@@ -0,0 +1,160 @@
|
||||
"""Render rung 3 into a booth page: 4B base, 4B tuned, 1.7B tuned.
|
||||
|
||||
Column choice, same logic as the 1.7B page. The 4B base arm is the control that
|
||||
earns any claim about the adapter -- at 1.7B the shift read 0/18 to 15/18 on curly
|
||||
quotes, and that number only meant something because the 0/18 came from the same
|
||||
carrier. The 1.7B tuned arm is the rung below, on the identical corpus and seed, so
|
||||
carrier size is the only difference between the two tuned columns.
|
||||
|
||||
The operator's Abernathy prompt gets its own section at the bottom, across four
|
||||
seeds. It belongs apart from the nine because it is doing something the nine cannot:
|
||||
it carries an emotional frame ("a wonderful story"), and at 1.7B half the seeds
|
||||
overrode that frame and killed the animals anyway -- Brontë's preoccupations
|
||||
arriving with her sentences. Whether a bigger carrier holds the frame is the open
|
||||
question that section exists to answer.
|
||||
"""
|
||||
import html
|
||||
import json
|
||||
import statistics
|
||||
import sys
|
||||
from collections import defaultdict
|
||||
from pathlib import Path
|
||||
|
||||
D = Path(sys.argv[1])
|
||||
ARMS = [("4b-base.jsonl", "4B base", "Qwen3-4B-Base, no adapter", ""),
|
||||
("4b-tuned.jsonl", "4B tuned", "+ H02 LoRA, 1 epoch, seed 4919", "tuned"),
|
||||
("1p7b-tuned.jsonl", "1.7B tuned", "the rung below, same corpus & seed", "small")]
|
||||
AB = [("4b-base-abernathy.jsonl", "4B base", ""), ("4b-tuned-abernathy.jsonl", "4B tuned", "tuned")]
|
||||
|
||||
|
||||
def load(p):
|
||||
d = defaultdict(dict)
|
||||
for line in Path(p).read_text(encoding="utf-8").splitlines():
|
||||
if line.strip():
|
||||
r = json.loads(line)
|
||||
d[r["id"]][r["seed"]] = r
|
||||
return d
|
||||
|
||||
|
||||
def metrics(p):
|
||||
rs = [json.loads(l) for l in Path(p).read_text(encoding="utf-8").splitlines() if l.strip()]
|
||||
wrap, curly, junk = [], 0, 0
|
||||
for r in rs:
|
||||
c = r["continuation"]
|
||||
lines = [l for l in c.split("\n") if l.strip()]
|
||||
wrap.append(sum(1 for l in lines if 20 < len(l) < 78) / max(1, len(lines)))
|
||||
curly += ("“" in c or "’" in c)
|
||||
junk += ("$" in c or "\\dfrac" in c or "Answer:" in c)
|
||||
return len(rs), curly, junk, statistics.median(wrap)
|
||||
|
||||
|
||||
data = [(lbl, sub, cls, load(D / f), metrics(D / f)) for f, lbl, sub, cls in ARMS]
|
||||
ids = sorted(set.intersection(*[set(d) for *_, d, _ in data]))
|
||||
TIER = {"modern": ("Tier A — modern / mundane",
|
||||
"Nothing here invites Victorian prose. Brontë in this tier is the adapter's doing."),
|
||||
"neutral": ("Tier B — period-neutral",
|
||||
"Could be any century. A voice shift shows without the prompt supplying it."),
|
||||
"period": ("Tier C — Victorian-adjacent, plainly worded",
|
||||
"The setting leans period but the diction does not. Easiest tier; weakest evidence.")}
|
||||
order = {"modern": 0, "neutral": 1, "period": 2}
|
||||
tier_of = {i: data[0][3][i][list(data[0][3][i])[0]]["tier"] for i in ids}
|
||||
ids.sort(key=lambda i: (order.get(tier_of[i], 9), i))
|
||||
|
||||
|
||||
def cell(by_seed):
|
||||
return "".join(
|
||||
f'<div class="s"><span class="seed">seed {s}</span><p>'
|
||||
f'{html.escape((by_seed[s]["continuation"] or "").strip()) or "<em>(empty)</em>"}</p></div>'
|
||||
for s in sorted(by_seed))
|
||||
|
||||
|
||||
tbl = "".join(
|
||||
f"<tr><td>{lbl}</td><td>{c}/{n}</td><td>{j}/{n}</td><td>{w:.2f}</td></tr>"
|
||||
for lbl, _s, _c, _d, (n, c, j, w) in data)
|
||||
|
||||
rows, seen = [], set()
|
||||
for i in ids:
|
||||
if tier_of[i] not in seen:
|
||||
seen.add(tier_of[i])
|
||||
t, sb = TIER.get(tier_of[i], (tier_of[i], ""))
|
||||
rows.append(f'<h2>{html.escape(t)}</h2><p class="tsub">{html.escape(sb)}</p>')
|
||||
pr = data[0][3][i][list(data[0][3][i])[0]]["prompt"]
|
||||
cols = "".join(f'<div class="arm {cls}"><h3>{lbl} <small>{sub}</small></h3>{cell(d[i])}</div>'
|
||||
for lbl, sub, cls, d, _ in data)
|
||||
rows.append(f'<section class="row"><div class="prompt"><span class="pid">{html.escape(i)}'
|
||||
f'</span>{html.escape(pr)}</div><div class="arms">{cols}</div></section>')
|
||||
|
||||
ab_html = ""
|
||||
if all((D / f).exists() for f, _, _ in AB):
|
||||
ab_data = [(lbl, cls, load(D / f)) for f, lbl, cls in AB]
|
||||
pid = list(ab_data[0][2])[0]
|
||||
pr = ab_data[0][2][pid][list(ab_data[0][2][pid])[0]]["prompt"]
|
||||
cols = "".join(f'<div class="arm {cls}"><h3>{lbl}</h3>{cell(d[pid])}</div>'
|
||||
for lbl, cls, d in ab_data)
|
||||
ab_html = (f'<h2>The operator\'s frame prompt</h2><p class="tsub">Reported speech with an open '
|
||||
f'quotation mark, and an emotional frame the nine prompts do not carry. At 1.7B the '
|
||||
f'frame held on every seed and half of them killed the animals anyway.</p>'
|
||||
f'<section class="row"><div class="prompt">{html.escape(pr)}</div>'
|
||||
f'<div class="arms two">{cols}</div></section>')
|
||||
|
||||
page = f"""<!doctype html><meta charset="utf-8"><title>BabyBronte — 4B rung</title>
|
||||
<style>
|
||||
:root{{--bg:#faf8f5;--fg:#1c1a17;--mut:#6b6560;--line:#e0dad2;--acc:#7a3b2e;--tint:#fdfbf7;--cool:#f5f6f8}}
|
||||
*{{box-sizing:border-box}}
|
||||
body{{margin:0;background:var(--bg);color:var(--fg);font:16px/1.6 Georgia,"Iowan Old Style",serif;padding:2.5rem 1.5rem 5rem}}
|
||||
.wrap{{max-width:1500px;margin:0 auto}}
|
||||
h1{{font-size:1.9rem;margin:0 0 .3rem}}
|
||||
.lede{{color:var(--mut);max-width:74ch;margin:0 0 .9rem}}
|
||||
.warn{{border-left:3px solid var(--acc);background:#fff;padding:.8rem 1rem;margin:1.1rem 0;max-width:84ch;font-size:.93rem}}
|
||||
table.m{{border-collapse:collapse;margin:1.2rem 0;font-size:.9rem;background:#fff}}
|
||||
table.m th,table.m td{{border:1px solid var(--line);padding:.35rem .7rem;text-align:left}}
|
||||
table.m th{{background:var(--bg);font-weight:600}}
|
||||
h2{{font-size:1.15rem;margin:2.8rem 0 .2rem;padding-top:1rem;border-top:1px solid var(--line)}}
|
||||
.tsub{{color:var(--mut);font-size:.9rem;margin:0 0 1.2rem;font-style:italic;max-width:84ch}}
|
||||
.row{{margin:0 0 2.2rem}}
|
||||
.prompt{{background:#fff;border:1px solid var(--line);border-left:3px solid var(--acc);padding:.7rem .9rem;font-size:1.02rem;margin-bottom:.7rem}}
|
||||
.pid{{display:inline-block;font:600 .72rem/1 ui-monospace,monospace;color:var(--mut);background:var(--bg);border:1px solid var(--line);padding:.22rem .4rem;margin-right:.6rem;vertical-align:1px}}
|
||||
.arms{{display:grid;grid-template-columns:repeat(3,1fr);gap:.9rem}}
|
||||
.arms.two{{grid-template-columns:repeat(2,1fr)}}
|
||||
@media(max-width:1100px){{.arms,.arms.two{{grid-template-columns:1fr}}}}
|
||||
.arm{{background:var(--cool);border:1px solid var(--line);padding:.85rem .95rem}}
|
||||
.arm.tuned{{background:var(--tint);border-color:#d8ccbe}}
|
||||
.arm.small{{background:#fbf9fb;border-color:#ded6e0}}
|
||||
.arm h3{{margin:0 0 .6rem;font-size:.88rem;letter-spacing:.04em;text-transform:uppercase;color:var(--acc)}}
|
||||
.arm h3 small{{display:block;text-transform:none;letter-spacing:0;color:var(--mut);font-weight:400;font-size:.8rem;margin-top:.15rem}}
|
||||
.s{{border-top:1px dotted var(--line);padding-top:.6rem;margin-top:.6rem}}
|
||||
.arm .s:first-of-type{{border-top:0;padding-top:0;margin-top:0}}
|
||||
.seed{{display:block;font:600 .7rem/1 ui-monospace,monospace;color:var(--mut);margin-bottom:.25rem}}
|
||||
.s p{{margin:0;white-space:pre-wrap;font-size:.94rem}}
|
||||
footer{{margin-top:3rem;padding-top:1rem;border-top:1px solid var(--line);color:var(--mut);font-size:.85rem;max-width:84ch}}
|
||||
</style>
|
||||
<div class="wrap">
|
||||
<h1>BabyBronte — rung 3: does the thread hold?</h1>
|
||||
<p class="lede">0.6B gave the voice and not the sense. 1.7B brought back sentence-to-sentence
|
||||
coherence but still lost the thread inside a passage. 4B is the last rung of the planned sweep, and
|
||||
the open question is scene-level continuity.</p>
|
||||
|
||||
<table class="m"><tr><th>arm</th><th>curly quotes</th><th>worksheet / explainer collapse</th><th>hard-wrap ratio</th></tr>{tbl}</table>
|
||||
|
||||
<div class="warn"><strong>The base column is the control that earns the claim.</strong> Any difference
|
||||
between the two tuned columns is carrier size and nothing else — identical corpus
|
||||
(sha 77f37057b2782e49), seed, sampler and step count. Any difference between 4B base and 4B tuned is
|
||||
the adapter and nothing else.</div>
|
||||
|
||||
<div class="warn"><strong>Still an eyeball test.</strong> Two samples per arm is enough to see whether
|
||||
the gap between columns beats the gap between seeds inside one, and not enough for anything else. No
|
||||
scoring; the frozen adjudication rule and the Burrows's-Delta instrument are untouched and nothing
|
||||
here feeds them. All arms are <strong>base models doing continuation</strong>, so each prompt is an
|
||||
opening line carried on rather than an instruction to rewrite.</div>
|
||||
|
||||
{''.join(rows)}
|
||||
{ab_html}
|
||||
|
||||
<footer>Generated on pfi-gx10 (GB10), bf16, sdpa. Sampler identical across arms: temperature 0.9,
|
||||
top_p 0.95, 400 new tokens (300 on the frame prompt), seeds 1234 and 5678. All tuned arms: 1 epoch,
|
||||
seed 4919, corpus sha 77f37057b2782e49, 5,210,112 tokens, 159 steps. Held-out loss at plateau:
|
||||
0.6B 3.329 · 1.7B 3.018 · 4B see the run log. Adapter binding proven at generation time on every
|
||||
tuned arm (lora_B tensors non-zero).</footer>
|
||||
</div>"""
|
||||
(D / "index.html").write_text(page, encoding="utf-8")
|
||||
print(f"wrote {D/'index.html'} ({len(ids)} prompts x 3 arms" + (", + frame prompt" if ab_html else "") + ")")
|
||||
Reference in New Issue
Block a user