vh
9e18fd58ec
Both rungs now sit on the same unwrapped corpus with seed, steps and token count held, so carrier size is the only difference and the effect is attributable: held-out 3.329 at 0.6B against 3.018 at 1.7B, a gap of 0.311 nats. The chained 0.6B rerun closed the confound the unwrap opened. Two things in those numbers need stating or they will be misread. First, the original wrapped-corpus 0.6B reached 3.172, which looks better than the unwrapped 0.6B's 3.329 and is not. Different corpus means a different held-out set, and the wrapped version's 5.7% newline tokens are near-deterministic after a 70-character line, so they deflate the loss with cheap wins. Removing them removed the easy tokens. It is a measurement artifact, not a regression. Second, a correction to my own earlier claim: I twice described the 0.6B run as still descending and undertrained at 3.172. Its series reads 3.176, 3.173, 3.172, 3.172 -- it flattened. All three runs plateau, so one epoch is about right for this corpus rather than short. The three-way booth puts 1.7B base, 1.7B tuned and 0.6B tuned side by side on the same nine prompts and seeds. The base arm is the control that matters: curly quotes go 0 of 18 on 1.7B base to 15 of 18 on 1.7B tuned, and worksheet-or- explainer collapse goes 3 of 18 to 0 of 18, so the shift is the adapter rather than the larger carrier. Hard-wrapping fell from 0.85 to 0.18, confirming the corpus unwrap carried through into the adapter. Sense partially returned. The 1.7B arm produces locally coherent sequential Victorian prose where the 0.6B produced word salad, but scene-level continuity still breaks mid-passage. One observation held loosely: curly quotes are slightly lower at 1.7B than 0.6B, which would fit a bigger model's stronger priors resisting the adapter at the same rank. That is untested and is not offered as established.
134 lines
7.4 KiB
Python
134 lines
7.4 KiB
Python
"""Render the three-arm carrier comparison into a booth page.
|
|
|
|
Three columns, chosen so the page answers two questions at once and neither answer
|
|
leans on the other:
|
|
|
|
1.7B base vs 1.7B tuned -- did the ADAPTER do anything at this carrier size,
|
|
or is any improvement just the bigger model?
|
|
0.6B tuned vs 1.7B tuned -- did coherence come back as the carrier grew?
|
|
|
|
Both tuned arms sit on the SAME unwrapped corpus (sha 77f37057b2782e49), same seed,
|
|
same sampler, so carrier size is the only difference between them. The 1.7B base arm
|
|
is generated fresh rather than reused, because a control from a different model would
|
|
control for nothing.
|
|
|
|
Prompts run hardest-first: modern/mundane, then period-neutral, then Victorian-
|
|
adjacent. Both seeds of every arm sit in the same cell so within-arm sampling noise
|
|
is visible in the same glance as between-arm difference -- if two samples of one arm
|
|
differ as much as two arms differ, the page should make that obvious rather than hide
|
|
it.
|
|
"""
|
|
import html
|
|
import json
|
|
import sys
|
|
from collections import defaultdict
|
|
from pathlib import Path
|
|
|
|
out_dir = Path(sys.argv[1])
|
|
ARMS = [
|
|
("1p7b-base.jsonl", "1.7B base", "Qwen3-1.7B-Base, no adapter", ""),
|
|
("1p7b-tuned.jsonl", "1.7B tuned", "+ H02 LoRA, 1 epoch, seed 4919", "tuned"),
|
|
("0p6b-tuned.jsonl", "0.6B tuned", "+ H02 LoRA, same corpus & seed", "small"),
|
|
]
|
|
out_dir.mkdir(parents=True, exist_ok=True)
|
|
|
|
def load(p):
|
|
d = defaultdict(dict)
|
|
for line in Path(p).read_text(encoding="utf-8").splitlines():
|
|
if line.strip():
|
|
r = json.loads(line)
|
|
d[r["id"]][r["seed"]] = r
|
|
return d
|
|
|
|
data = [(lbl, sub, cls, load(out_dir / f)) for f, lbl, sub, cls in ARMS]
|
|
ids = sorted(set.intersection(*[set(d) for *_, d in data]))
|
|
TIER = {"modern": ("Tier A — modern / mundane",
|
|
"Nothing here invites Victorian prose. Brontë in this tier is the adapter's doing."),
|
|
"neutral": ("Tier B — period-neutral",
|
|
"Could be any century. A voice shift shows without the prompt supplying it."),
|
|
"period": ("Tier C — Victorian-adjacent, plainly worded",
|
|
"The setting leans period but the diction does not. Easiest tier; weakest evidence.")}
|
|
order = {"modern": 0, "neutral": 1, "period": 2}
|
|
tier_of = {i: data[0][3][i][list(data[0][3][i])[0]]["tier"] for i in ids}
|
|
ids.sort(key=lambda i: (order.get(tier_of[i], 9), i))
|
|
|
|
def cell(by_seed):
|
|
return "".join(
|
|
f'<div class="s"><span class="seed">seed {s}</span>'
|
|
f'<p>{html.escape((by_seed[s]["continuation"] or "").strip()) or "<em>(empty)</em>"}</p></div>'
|
|
for s in sorted(by_seed))
|
|
|
|
rows, seen = [], set()
|
|
for i in ids:
|
|
if tier_of[i] not in seen:
|
|
seen.add(tier_of[i])
|
|
title, sub = TIER.get(tier_of[i], (tier_of[i], ""))
|
|
rows.append(f'<h2>{html.escape(title)}</h2><p class="tsub">{html.escape(sub)}</p>')
|
|
p = data[0][3][i][list(data[0][3][i])[0]]["prompt"]
|
|
cols = "".join(
|
|
f'<div class="arm {cls}"><h3>{lbl} <small>{sub}</small></h3>{cell(d[i])}</div>'
|
|
for lbl, sub, cls, d in data)
|
|
rows.append(f'<section class="row"><div class="prompt"><span class="pid">'
|
|
f'{html.escape(i)}</span>{html.escape(p)}</div>'
|
|
f'<div class="arms">{cols}</div></section>')
|
|
|
|
page = f"""<!doctype html><meta charset="utf-8"><title>BabyBronte — 1.7B rung</title>
|
|
<style>
|
|
:root{{--bg:#faf8f5;--fg:#1c1a17;--mut:#6b6560;--line:#e0dad2;--acc:#7a3b2e;--tint:#fdfbf7;--cool:#f5f6f8}}
|
|
*{{box-sizing:border-box}}
|
|
body{{margin:0;background:var(--bg);color:var(--fg);font:16px/1.6 Georgia,"Iowan Old Style",serif;padding:2.5rem 1.5rem 5rem}}
|
|
.wrap{{max-width:1500px;margin:0 auto}}
|
|
h1{{font-size:1.9rem;margin:0 0 .3rem}}
|
|
.lede{{color:var(--mut);max-width:74ch;margin:0 0 .9rem}}
|
|
.warn{{border-left:3px solid var(--acc);background:#fff;padding:.8rem 1rem;margin:1.1rem 0;max-width:84ch;font-size:.93rem}}
|
|
h2{{font-size:1.15rem;margin:2.8rem 0 .2rem;padding-top:1rem;border-top:1px solid var(--line)}}
|
|
.tsub{{color:var(--mut);font-size:.9rem;margin:0 0 1.2rem;font-style:italic}}
|
|
.row{{margin:0 0 2.2rem}}
|
|
.prompt{{background:#fff;border:1px solid var(--line);border-left:3px solid var(--acc);padding:.7rem .9rem;font-size:1.02rem;margin-bottom:.7rem}}
|
|
.pid{{display:inline-block;font:600 .72rem/1 ui-monospace,monospace;color:var(--mut);background:var(--bg);border:1px solid var(--line);padding:.22rem .4rem;margin-right:.6rem;vertical-align:1px}}
|
|
.arms{{display:grid;grid-template-columns:repeat(3,1fr);gap:.9rem}}
|
|
@media(max-width:1100px){{.arms{{grid-template-columns:1fr}}}}
|
|
.arm{{background:var(--cool);border:1px solid var(--line);padding:.85rem .95rem}}
|
|
.arm.tuned{{background:var(--tint);border-color:#d8ccbe}}
|
|
.arm.small{{background:#fbf9fb;border-color:#ded6e0}}
|
|
.arm h3{{margin:0 0 .6rem;font-size:.88rem;letter-spacing:.04em;text-transform:uppercase;color:var(--acc)}}
|
|
.arm h3 small{{display:block;text-transform:none;letter-spacing:0;color:var(--mut);font-weight:400;font-size:.8rem;margin-top:.15rem}}
|
|
.s{{border-top:1px dotted var(--line);padding-top:.6rem;margin-top:.6rem}}
|
|
.arm .s:first-of-type{{border-top:0;padding-top:0;margin-top:0}}
|
|
.seed{{display:block;font:600 .7rem/1 ui-monospace,monospace;color:var(--mut);margin-bottom:.25rem}}
|
|
.s p{{margin:0;white-space:pre-wrap;font-size:.94rem}}
|
|
footer{{margin-top:3rem;padding-top:1rem;border-top:1px solid var(--line);color:var(--mut);font-size:.85rem;max-width:84ch}}
|
|
</style>
|
|
<div class="wrap">
|
|
<h1>BabyBronte — rung 2: did the sense come back?</h1>
|
|
<p class="lede">The 0.6B rung transferred the voice and not the coherence — "it's all nonsense, but it
|
|
sounds like Brontë's nonsense." This is the same nine prompts at 1.7B, with the 0.6B tuned arm beside
|
|
it for scale and the 1.7B base arm beside it for control.</p>
|
|
|
|
<div class="warn"><strong>Two questions, two columns each.</strong>
|
|
<em>1.7B base vs 1.7B tuned</em> asks whether the adapter did anything at this carrier size, or
|
|
whether any improvement is just the bigger model.
|
|
<em>0.6B tuned vs 1.7B tuned</em> asks whether coherence returned as the carrier grew — and those two
|
|
tuned arms sit on the <strong>same corpus, same seed, same sampler</strong>, so carrier size is the
|
|
only difference between them.</div>
|
|
|
|
<div class="warn"><strong>Still an eyeball test, not a result.</strong> Two samples per arm is enough
|
|
to see whether the gap between columns beats the gap between seeds inside one — and not enough for
|
|
anything else. No scoring. The frozen adjudication rule and the Burrows's-Delta instrument are
|
|
untouched and nothing here feeds them.<br><br>
|
|
All arms are base models doing <strong>continuation</strong>, not instruction-following, so each
|
|
prompt is an opening line carried on rather than an instruction to rewrite.
|
|
<strong>The corpus was unwrapped since the last booth</strong>, so the ~70-character hard wrapping
|
|
that disfigured the first 0.6B page should be gone from both tuned arms here.</div>
|
|
|
|
{''.join(rows)}
|
|
|
|
<footer>Generated on pfi-gx10 (GB10), bf16, sdpa. Sampler identical across all three arms:
|
|
temperature 0.9, top_p 0.95, 400 new tokens, seeds 1234 and 5678. Both tuned arms: 1 epoch,
|
|
seed 4919, corpus sha 77f37057b2782e49 (5,210,112 tokens, 159 steps). Held-out loss at the plateau:
|
|
0.6B 3.329, 1.7B 3.018. Adapter binding proven at generation time on both tuned arms
|
|
(196/196 lora_B tensors non-zero).</footer>
|
|
</div>"""
|
|
(out_dir / "index.html").write_text(page, encoding="utf-8")
|
|
print(f"wrote {out_dir/'index.html'} ({len(ids)} prompts x 3 arms x 2 seeds)")
|