Complete the R49 carrier sweep: 4B closes the continuity gap and overfits
The three rungs now sit on the same unwrapped corpus, seed, step count and token count, so carrier size is the only variable. Held-out loss reads 3.329 at 0.6B, 3.018 at 1.7B and 2.814 at 4B -- deltas of 0.311 and then 0.204, diminishing but still real. 4B answers the question the rung existed for. Scene-level continuity holds: on the office-kitchen prompt it produces a named character with motivated dialogue, a spatial layout the narrator navigates, and a physical description, all in one passage, where 1.7B wrote pretty but eventless prose about opening doors and looking at stars. On the letter prompt it opens the letter, promises to quote it, and then quotes it across a paragraph break. Voice saturation is also the best of any rung: curly quotes 17 of 18 against its own base arm's 1 of 18, and collapse 0 of 18 against 4 of 18. Two findings that change earlier conclusions. 4B is the first rung to overfit inside one epoch. Its series runs 2.832, 2.816, 2.814, 2.820, 2.824, 2.825, 2.825 -- a minimum around step 75 and then a turn. Both smaller rungs plateaued without turning, so the optimal epoch count shrinks as the carrier grows and my earlier "one epoch is right for this corpus" holds only for the small end. The consequence is operational: the adapter directory holds the end-of-run weights at 2.825 rather than the step-75 best at 2.814, and it exists as a recoverable checkpoint only because save_steps was set. The voice arms were cut from the end-of-run adapter, so the booth understates 4B slightly. The tone-override also appears to close. On the operator's frame prompt asking for a wonderful story, 1.7B held the frame on every seed but killed the animals on two of four; 4B kept them alive on both seeds, and one of them had the narrator doubt the story he was told and supply a parallel childhood memory to explain the doubt. That is a narrator with an interior position on the tale. Two samples per arm, so directionally right rather than established.
This commit is contained in:
@@ -0,0 +1,160 @@
|
||||
"""Render rung 3 into a booth page: 4B base, 4B tuned, 1.7B tuned.
|
||||
|
||||
Column choice, same logic as the 1.7B page. The 4B base arm is the control that
|
||||
earns any claim about the adapter -- at 1.7B the shift read 0/18 to 15/18 on curly
|
||||
quotes, and that number only meant something because the 0/18 came from the same
|
||||
carrier. The 1.7B tuned arm is the rung below, on the identical corpus and seed, so
|
||||
carrier size is the only difference between the two tuned columns.
|
||||
|
||||
The operator's Abernathy prompt gets its own section at the bottom, across four
|
||||
seeds. It belongs apart from the nine because it is doing something the nine cannot:
|
||||
it carries an emotional frame ("a wonderful story"), and at 1.7B half the seeds
|
||||
overrode that frame and killed the animals anyway -- Brontë's preoccupations
|
||||
arriving with her sentences. Whether a bigger carrier holds the frame is the open
|
||||
question that section exists to answer.
|
||||
"""
|
||||
import html
|
||||
import json
|
||||
import statistics
|
||||
import sys
|
||||
from collections import defaultdict
|
||||
from pathlib import Path
|
||||
|
||||
D = Path(sys.argv[1])
|
||||
ARMS = [("4b-base.jsonl", "4B base", "Qwen3-4B-Base, no adapter", ""),
|
||||
("4b-tuned.jsonl", "4B tuned", "+ H02 LoRA, 1 epoch, seed 4919", "tuned"),
|
||||
("1p7b-tuned.jsonl", "1.7B tuned", "the rung below, same corpus & seed", "small")]
|
||||
AB = [("4b-base-abernathy.jsonl", "4B base", ""), ("4b-tuned-abernathy.jsonl", "4B tuned", "tuned")]
|
||||
|
||||
|
||||
def load(p):
|
||||
d = defaultdict(dict)
|
||||
for line in Path(p).read_text(encoding="utf-8").splitlines():
|
||||
if line.strip():
|
||||
r = json.loads(line)
|
||||
d[r["id"]][r["seed"]] = r
|
||||
return d
|
||||
|
||||
|
||||
def metrics(p):
|
||||
rs = [json.loads(l) for l in Path(p).read_text(encoding="utf-8").splitlines() if l.strip()]
|
||||
wrap, curly, junk = [], 0, 0
|
||||
for r in rs:
|
||||
c = r["continuation"]
|
||||
lines = [l for l in c.split("\n") if l.strip()]
|
||||
wrap.append(sum(1 for l in lines if 20 < len(l) < 78) / max(1, len(lines)))
|
||||
curly += ("“" in c or "’" in c)
|
||||
junk += ("$" in c or "\\dfrac" in c or "Answer:" in c)
|
||||
return len(rs), curly, junk, statistics.median(wrap)
|
||||
|
||||
|
||||
data = [(lbl, sub, cls, load(D / f), metrics(D / f)) for f, lbl, sub, cls in ARMS]
|
||||
ids = sorted(set.intersection(*[set(d) for *_, d, _ in data]))
|
||||
TIER = {"modern": ("Tier A — modern / mundane",
|
||||
"Nothing here invites Victorian prose. Brontë in this tier is the adapter's doing."),
|
||||
"neutral": ("Tier B — period-neutral",
|
||||
"Could be any century. A voice shift shows without the prompt supplying it."),
|
||||
"period": ("Tier C — Victorian-adjacent, plainly worded",
|
||||
"The setting leans period but the diction does not. Easiest tier; weakest evidence.")}
|
||||
order = {"modern": 0, "neutral": 1, "period": 2}
|
||||
tier_of = {i: data[0][3][i][list(data[0][3][i])[0]]["tier"] for i in ids}
|
||||
ids.sort(key=lambda i: (order.get(tier_of[i], 9), i))
|
||||
|
||||
|
||||
def cell(by_seed):
|
||||
return "".join(
|
||||
f'<div class="s"><span class="seed">seed {s}</span><p>'
|
||||
f'{html.escape((by_seed[s]["continuation"] or "").strip()) or "<em>(empty)</em>"}</p></div>'
|
||||
for s in sorted(by_seed))
|
||||
|
||||
|
||||
tbl = "".join(
|
||||
f"<tr><td>{lbl}</td><td>{c}/{n}</td><td>{j}/{n}</td><td>{w:.2f}</td></tr>"
|
||||
for lbl, _s, _c, _d, (n, c, j, w) in data)
|
||||
|
||||
rows, seen = [], set()
|
||||
for i in ids:
|
||||
if tier_of[i] not in seen:
|
||||
seen.add(tier_of[i])
|
||||
t, sb = TIER.get(tier_of[i], (tier_of[i], ""))
|
||||
rows.append(f'<h2>{html.escape(t)}</h2><p class="tsub">{html.escape(sb)}</p>')
|
||||
pr = data[0][3][i][list(data[0][3][i])[0]]["prompt"]
|
||||
cols = "".join(f'<div class="arm {cls}"><h3>{lbl} <small>{sub}</small></h3>{cell(d[i])}</div>'
|
||||
for lbl, sub, cls, d, _ in data)
|
||||
rows.append(f'<section class="row"><div class="prompt"><span class="pid">{html.escape(i)}'
|
||||
f'</span>{html.escape(pr)}</div><div class="arms">{cols}</div></section>')
|
||||
|
||||
ab_html = ""
|
||||
if all((D / f).exists() for f, _, _ in AB):
|
||||
ab_data = [(lbl, cls, load(D / f)) for f, lbl, cls in AB]
|
||||
pid = list(ab_data[0][2])[0]
|
||||
pr = ab_data[0][2][pid][list(ab_data[0][2][pid])[0]]["prompt"]
|
||||
cols = "".join(f'<div class="arm {cls}"><h3>{lbl}</h3>{cell(d[pid])}</div>'
|
||||
for lbl, cls, d in ab_data)
|
||||
ab_html = (f'<h2>The operator\'s frame prompt</h2><p class="tsub">Reported speech with an open '
|
||||
f'quotation mark, and an emotional frame the nine prompts do not carry. At 1.7B the '
|
||||
f'frame held on every seed and half of them killed the animals anyway.</p>'
|
||||
f'<section class="row"><div class="prompt">{html.escape(pr)}</div>'
|
||||
f'<div class="arms two">{cols}</div></section>')
|
||||
|
||||
page = f"""<!doctype html><meta charset="utf-8"><title>BabyBronte — 4B rung</title>
|
||||
<style>
|
||||
:root{{--bg:#faf8f5;--fg:#1c1a17;--mut:#6b6560;--line:#e0dad2;--acc:#7a3b2e;--tint:#fdfbf7;--cool:#f5f6f8}}
|
||||
*{{box-sizing:border-box}}
|
||||
body{{margin:0;background:var(--bg);color:var(--fg);font:16px/1.6 Georgia,"Iowan Old Style",serif;padding:2.5rem 1.5rem 5rem}}
|
||||
.wrap{{max-width:1500px;margin:0 auto}}
|
||||
h1{{font-size:1.9rem;margin:0 0 .3rem}}
|
||||
.lede{{color:var(--mut);max-width:74ch;margin:0 0 .9rem}}
|
||||
.warn{{border-left:3px solid var(--acc);background:#fff;padding:.8rem 1rem;margin:1.1rem 0;max-width:84ch;font-size:.93rem}}
|
||||
table.m{{border-collapse:collapse;margin:1.2rem 0;font-size:.9rem;background:#fff}}
|
||||
table.m th,table.m td{{border:1px solid var(--line);padding:.35rem .7rem;text-align:left}}
|
||||
table.m th{{background:var(--bg);font-weight:600}}
|
||||
h2{{font-size:1.15rem;margin:2.8rem 0 .2rem;padding-top:1rem;border-top:1px solid var(--line)}}
|
||||
.tsub{{color:var(--mut);font-size:.9rem;margin:0 0 1.2rem;font-style:italic;max-width:84ch}}
|
||||
.row{{margin:0 0 2.2rem}}
|
||||
.prompt{{background:#fff;border:1px solid var(--line);border-left:3px solid var(--acc);padding:.7rem .9rem;font-size:1.02rem;margin-bottom:.7rem}}
|
||||
.pid{{display:inline-block;font:600 .72rem/1 ui-monospace,monospace;color:var(--mut);background:var(--bg);border:1px solid var(--line);padding:.22rem .4rem;margin-right:.6rem;vertical-align:1px}}
|
||||
.arms{{display:grid;grid-template-columns:repeat(3,1fr);gap:.9rem}}
|
||||
.arms.two{{grid-template-columns:repeat(2,1fr)}}
|
||||
@media(max-width:1100px){{.arms,.arms.two{{grid-template-columns:1fr}}}}
|
||||
.arm{{background:var(--cool);border:1px solid var(--line);padding:.85rem .95rem}}
|
||||
.arm.tuned{{background:var(--tint);border-color:#d8ccbe}}
|
||||
.arm.small{{background:#fbf9fb;border-color:#ded6e0}}
|
||||
.arm h3{{margin:0 0 .6rem;font-size:.88rem;letter-spacing:.04em;text-transform:uppercase;color:var(--acc)}}
|
||||
.arm h3 small{{display:block;text-transform:none;letter-spacing:0;color:var(--mut);font-weight:400;font-size:.8rem;margin-top:.15rem}}
|
||||
.s{{border-top:1px dotted var(--line);padding-top:.6rem;margin-top:.6rem}}
|
||||
.arm .s:first-of-type{{border-top:0;padding-top:0;margin-top:0}}
|
||||
.seed{{display:block;font:600 .7rem/1 ui-monospace,monospace;color:var(--mut);margin-bottom:.25rem}}
|
||||
.s p{{margin:0;white-space:pre-wrap;font-size:.94rem}}
|
||||
footer{{margin-top:3rem;padding-top:1rem;border-top:1px solid var(--line);color:var(--mut);font-size:.85rem;max-width:84ch}}
|
||||
</style>
|
||||
<div class="wrap">
|
||||
<h1>BabyBronte — rung 3: does the thread hold?</h1>
|
||||
<p class="lede">0.6B gave the voice and not the sense. 1.7B brought back sentence-to-sentence
|
||||
coherence but still lost the thread inside a passage. 4B is the last rung of the planned sweep, and
|
||||
the open question is scene-level continuity.</p>
|
||||
|
||||
<table class="m"><tr><th>arm</th><th>curly quotes</th><th>worksheet / explainer collapse</th><th>hard-wrap ratio</th></tr>{tbl}</table>
|
||||
|
||||
<div class="warn"><strong>The base column is the control that earns the claim.</strong> Any difference
|
||||
between the two tuned columns is carrier size and nothing else — identical corpus
|
||||
(sha 77f37057b2782e49), seed, sampler and step count. Any difference between 4B base and 4B tuned is
|
||||
the adapter and nothing else.</div>
|
||||
|
||||
<div class="warn"><strong>Still an eyeball test.</strong> Two samples per arm is enough to see whether
|
||||
the gap between columns beats the gap between seeds inside one, and not enough for anything else. No
|
||||
scoring; the frozen adjudication rule and the Burrows's-Delta instrument are untouched and nothing
|
||||
here feeds them. All arms are <strong>base models doing continuation</strong>, so each prompt is an
|
||||
opening line carried on rather than an instruction to rewrite.</div>
|
||||
|
||||
{''.join(rows)}
|
||||
{ab_html}
|
||||
|
||||
<footer>Generated on pfi-gx10 (GB10), bf16, sdpa. Sampler identical across arms: temperature 0.9,
|
||||
top_p 0.95, 400 new tokens (300 on the frame prompt), seeds 1234 and 5678. All tuned arms: 1 epoch,
|
||||
seed 4919, corpus sha 77f37057b2782e49, 5,210,112 tokens, 159 steps. Held-out loss at plateau:
|
||||
0.6B 3.329 · 1.7B 3.018 · 4B see the run log. Adapter binding proven at generation time on every
|
||||
tuned arm (lora_B tensors non-zero).</footer>
|
||||
</div>"""
|
||||
(D / "index.html").write_text(page, encoding="utf-8")
|
||||
print(f"wrote {D/'index.html'} ({len(ids)} prompts x 3 arms" + (", + frame prompt" if ab_html else "") + ")")
|
||||
Reference in New Issue
Block a user