vh
8787daf04f
Two operator prompts settled the question the handoff prompt opened. The winner labels the artifact and then begins it: "This is the letter I wrote verbatim, my two short paragraphs, detailing the time I saw the mangy gray dog meet and then lovingly and tenderly lick a calico kitten: Auntie, You'll never believe what I saw--". Two of three seeds delivered the actual event in first person, and one is the strongest output of the sweep -- the dog following her to the mill, the kitten caught in a hedge, the dog carrying it out and laying it in her lap before licking it. Coherent arc, correct subject, no gloom-override, no meta-frame. It also reproduced the Gutenberg underscore-italics convention. It works where the handoff failed for a specific reason. The handoff could be satisfied by narrating compliance, because the letter did not yet exist. Naming the artifact and starting it leaves nothing to narrate around, so the only continuation is the artifact. The chapter-heading prompt refuted a hypothesis of mine. I predicted that rendering the title in the corpus's own conventions would make it land harder than the operator's inline form. It did the opposite: both corpus-form seeds ignored the title and opened unrelated scenes, while the inline form at least finished the heading and wrote a chapter about the story. Corpus chapter titles are short and decorative, so a long descriptive one in that slot reads as decoration to skip. A label only instructs if the model treats that slot as load-bearing. That prompt also surfaced an unnoticed consequence of the D2/D3 rename pipeline: given "Alice", the adapter produced "Alexander the Alexander, as he was known in Little London". The corpus was entity-renamed from a French/English pool, so the adapter learned that character names come from that pool and rewrites outside names into it. Callers cannot reliably name their own characters at prompt time. That is not a defect of the rename, which exists to prevent memorising Brontë's cast, but it is a usability constraint worth stating.
223 lines
13 KiB
Python
223 lines
13 KiB
Python
"""Render rung 3 into a booth page: 4B base, 4B tuned, 1.7B tuned.
|
||
|
||
Column choice, same logic as the 1.7B page. The 4B base arm is the control that
|
||
earns any claim about the adapter -- at 1.7B the shift read 0/18 to 15/18 on curly
|
||
quotes, and that number only meant something because the 0/18 came from the same
|
||
carrier. The 1.7B tuned arm is the rung below, on the identical corpus and seed, so
|
||
carrier size is the only difference between the two tuned columns.
|
||
|
||
The operator's Abernathy prompt gets its own section at the bottom, across four
|
||
seeds. It belongs apart from the nine because it is doing something the nine cannot:
|
||
it carries an emotional frame ("a wonderful story"), and at 1.7B half the seeds
|
||
overrode that frame and killed the animals anyway -- Brontë's preoccupations
|
||
arriving with her sentences. Whether a bigger carrier holds the frame is the open
|
||
question that section exists to answer.
|
||
"""
|
||
import html
|
||
import json
|
||
import statistics
|
||
import sys
|
||
from collections import defaultdict
|
||
from pathlib import Path
|
||
|
||
D = Path(sys.argv[1])
|
||
ARMS = [("4b-base.jsonl", "4B base", "Qwen3-4B-Base, no adapter", ""),
|
||
("4b-tuned.jsonl", "4B tuned", "+ H02 LoRA, <b>checkpoint-75</b> (the loss minimum)", "tuned"),
|
||
("1p7b-tuned.jsonl", "1.7B tuned", "the rung below, same corpus & seed", "small")]
|
||
AB = [("4b-base-abernathy.jsonl", "4B base", ""), ("4b-tuned-abernathy.jsonl", "4B tuned", "tuned")]
|
||
|
||
|
||
def load(p):
|
||
d = defaultdict(dict)
|
||
for line in Path(p).read_text(encoding="utf-8").splitlines():
|
||
if line.strip():
|
||
r = json.loads(line)
|
||
d[r["id"]][r["seed"]] = r
|
||
return d
|
||
|
||
|
||
def metrics(p):
|
||
rs = [json.loads(l) for l in Path(p).read_text(encoding="utf-8").splitlines() if l.strip()]
|
||
wrap, curly, junk = [], 0, 0
|
||
for r in rs:
|
||
c = r["continuation"]
|
||
lines = [l for l in c.split("\n") if l.strip()]
|
||
wrap.append(sum(1 for l in lines if 20 < len(l) < 78) / max(1, len(lines)))
|
||
curly += ("“" in c or "’" in c)
|
||
junk += ("$" in c or "\\dfrac" in c or "Answer:" in c)
|
||
return len(rs), curly, junk, statistics.median(wrap)
|
||
|
||
|
||
data = [(lbl, sub, cls, load(D / f), metrics(D / f)) for f, lbl, sub, cls in ARMS]
|
||
ids = sorted(set.intersection(*[set(d) for *_, d, _ in data]))
|
||
TIER = {"modern": ("Tier A — modern / mundane",
|
||
"Nothing here invites Victorian prose. Brontë in this tier is the adapter's doing."),
|
||
"neutral": ("Tier B — period-neutral",
|
||
"Could be any century. A voice shift shows without the prompt supplying it."),
|
||
"period": ("Tier C — Victorian-adjacent, plainly worded",
|
||
"The setting leans period but the diction does not. Easiest tier; weakest evidence.")}
|
||
order = {"modern": 0, "neutral": 1, "period": 2}
|
||
tier_of = {i: data[0][3][i][list(data[0][3][i])[0]]["tier"] for i in ids}
|
||
ids.sort(key=lambda i: (order.get(tier_of[i], 9), i))
|
||
|
||
|
||
def cell(by_seed):
|
||
return "".join(
|
||
f'<div class="s"><span class="seed">seed {s}</span><p>'
|
||
f'{html.escape((by_seed[s]["continuation"] or "").strip()) or "<em>(empty)</em>"}</p></div>'
|
||
for s in sorted(by_seed))
|
||
|
||
|
||
tbl = "".join(
|
||
f"<tr><td>{lbl}</td><td>{c}/{n}</td><td>{j}/{n}</td><td>{w:.2f}</td></tr>"
|
||
for lbl, _s, _c, _d, (n, c, j, w) in data)
|
||
|
||
rows, seen = [], set()
|
||
for i in ids:
|
||
if tier_of[i] not in seen:
|
||
seen.add(tier_of[i])
|
||
t, sb = TIER.get(tier_of[i], (tier_of[i], ""))
|
||
rows.append(f'<h2>{html.escape(t)}</h2><p class="tsub">{html.escape(sb)}</p>')
|
||
pr = data[0][3][i][list(data[0][3][i])[0]]["prompt"]
|
||
cols = "".join(f'<div class="arm {cls}"><h3>{lbl} <small>{sub}</small></h3>{cell(d[i])}</div>'
|
||
for lbl, sub, cls, d, _ in data)
|
||
rows.append(f'<section class="row"><div class="prompt"><span class="pid">{html.escape(i)}'
|
||
f'</span>{html.escape(pr)}</div><div class="arms">{cols}</div></section>')
|
||
|
||
ab_html = ""
|
||
if all((D / f).exists() for f, _, _ in AB):
|
||
ab_data = [(lbl, cls, load(D / f)) for f, lbl, cls in AB]
|
||
pid = list(ab_data[0][2])[0]
|
||
pr = ab_data[0][2][pid][list(ab_data[0][2][pid])[0]]["prompt"]
|
||
cols = "".join(f'<div class="arm {cls}"><h3>{lbl}</h3>{cell(d[pid])}</div>'
|
||
for lbl, cls, d in ab_data)
|
||
ab_html = (f'<h2>The operator\'s frame prompt</h2><p class="tsub">Reported speech with an open '
|
||
f'quotation mark, and an emotional frame the nine prompts do not carry. At 1.7B the '
|
||
f'frame held on every seed and half of them killed the animals anyway.</p>'
|
||
f'<section class="row"><div class="prompt">{html.escape(pr)}</div>'
|
||
f'<div class="arms two">{cols}</div></section>')
|
||
|
||
hand_html = ""
|
||
hf = D / "handoff.jsonl"
|
||
if hf.exists():
|
||
hd = load(hf)
|
||
blocks = []
|
||
for pid, label in (("handoff", "as written"), ("handoff-break", "identical, plus a trailing paragraph break")):
|
||
if pid not in hd:
|
||
continue
|
||
pr = hd[pid][list(hd[pid])[0]]["prompt"]
|
||
blocks.append(f'<div class="prompt"><span class="pid">{html.escape(label)}</span>'
|
||
f'{html.escape(pr.strip())}</div>'
|
||
f'<div class="arms"><div class="arm tuned" style="grid-column:1/-1">'
|
||
f'{cell(hd[pid])}</div></div>')
|
||
hand_html = ('<h2>The embedded-instruction prompt</h2><p class="tsub">The instruction lives '
|
||
'INSIDE the fiction — Abernathy asks the narrator to retell the story — which is the '
|
||
'only way to hand an instruction to a completion model. Watch what it does with the '
|
||
'request: across every seed it narrates the retelling rather than performing it '
|
||
'("I told it, briefly", "So I wrote it out", "I will retell it, but I cannot '
|
||
'condense it"). In a novel, <em>she retold the story</em> is an ordinary sentence, '
|
||
'so the likeliest continuation of a request is narration of compliance — not '
|
||
'compliance. One seed even negotiates the word count in character and still never '
|
||
'tells it.</p><section class="row">' + "".join(blocks) + '</section>')
|
||
|
||
extra_html = ""
|
||
SPECIAL = [
|
||
("letter.jsonl", None, "The recipe that worked: LABEL the artifact, then BEGIN it",
|
||
"The handoff prompt could be satisfied by narrating compliance, because the letter did not yet "
|
||
"exist. Here it is named <em>and</em> already speaking — \"Auntie, You'll never believe what I "
|
||
"saw--\" — so there is nothing left to narrate around and the only continuation is the letter "
|
||
"itself. Two of three seeds deliver the actual event in first person; one drifts. Note the "
|
||
"<code>_underscore italics_</code>, learned from the Gutenberg source."),
|
||
("chapter.jsonl", "chapter-as-written", "A chapter heading, as written",
|
||
"A title is a label rather than a request, so this should have worked better than it did. It "
|
||
"finishes the heading, re-emits it in the corpus's own typography, and then writes a chapter in "
|
||
"which a gentleman <em>disputes the premise</em> — closer than the handoff, still the meta-frame. "
|
||
"⚠ It also renamed Alice to “Alexander the Alexander”: the corpus was entity-renamed "
|
||
"in D2/D3, so the adapter substitutes proper nouns it was never trained on."),
|
||
("chapter.jsonl", "chapter-corpus-form", "The same heading in the corpus's own typography",
|
||
"My hypothesis was that matching the source's heading conventions would make the title land "
|
||
"harder. It did the opposite — both seeds ignored the title entirely and opened generic scenes. "
|
||
"In the corpus, chapter titles are short and decorative, so a long descriptive one in that slot "
|
||
"reads as decoration to skip; inline, it reads as text to continue."),
|
||
]
|
||
blocks = []
|
||
for f, pid, title, note in SPECIAL:
|
||
fp = D / f
|
||
if not fp.exists():
|
||
continue
|
||
d = load(fp)
|
||
keys = [pid] if pid else list(d)
|
||
for k in keys:
|
||
if k not in d:
|
||
continue
|
||
pr = d[k][list(d[k])[0]]["prompt"]
|
||
blocks.append(f'<h2>{title}</h2><p class="tsub">{note}</p><section class="row">'
|
||
f'<div class="prompt">{html.escape(pr.strip())}</div>'
|
||
f'<div class="arms"><div class="arm tuned" style="grid-column:1/-1">'
|
||
f'{cell(d[k])}</div></div></section>')
|
||
extra_html = "".join(blocks)
|
||
|
||
page = f"""<!doctype html><meta charset="utf-8"><title>BabyBronte — 4B rung</title>
|
||
<style>
|
||
:root{{--bg:#faf8f5;--fg:#1c1a17;--mut:#6b6560;--line:#e0dad2;--acc:#7a3b2e;--tint:#fdfbf7;--cool:#f5f6f8}}
|
||
*{{box-sizing:border-box}}
|
||
body{{margin:0;background:var(--bg);color:var(--fg);font:16px/1.6 Georgia,"Iowan Old Style",serif;padding:2.5rem 1.5rem 5rem}}
|
||
.wrap{{max-width:1500px;margin:0 auto}}
|
||
h1{{font-size:1.9rem;margin:0 0 .3rem}}
|
||
.lede{{color:var(--mut);max-width:74ch;margin:0 0 .9rem}}
|
||
.warn{{border-left:3px solid var(--acc);background:#fff;padding:.8rem 1rem;margin:1.1rem 0;max-width:84ch;font-size:.93rem}}
|
||
table.m{{border-collapse:collapse;margin:1.2rem 0;font-size:.9rem;background:#fff}}
|
||
table.m th,table.m td{{border:1px solid var(--line);padding:.35rem .7rem;text-align:left}}
|
||
table.m th{{background:var(--bg);font-weight:600}}
|
||
h2{{font-size:1.15rem;margin:2.8rem 0 .2rem;padding-top:1rem;border-top:1px solid var(--line)}}
|
||
.tsub{{color:var(--mut);font-size:.9rem;margin:0 0 1.2rem;font-style:italic;max-width:84ch}}
|
||
.row{{margin:0 0 2.2rem}}
|
||
.prompt{{background:#fff;border:1px solid var(--line);border-left:3px solid var(--acc);padding:.7rem .9rem;font-size:1.02rem;margin-bottom:.7rem}}
|
||
.pid{{display:inline-block;font:600 .72rem/1 ui-monospace,monospace;color:var(--mut);background:var(--bg);border:1px solid var(--line);padding:.22rem .4rem;margin-right:.6rem;vertical-align:1px}}
|
||
.arms{{display:grid;grid-template-columns:repeat(3,1fr);gap:.9rem}}
|
||
.arms.two{{grid-template-columns:repeat(2,1fr)}}
|
||
@media(max-width:1100px){{.arms,.arms.two{{grid-template-columns:1fr}}}}
|
||
.arm{{background:var(--cool);border:1px solid var(--line);padding:.85rem .95rem}}
|
||
.arm.tuned{{background:var(--tint);border-color:#d8ccbe}}
|
||
.arm.small{{background:#fbf9fb;border-color:#ded6e0}}
|
||
.arm h3{{margin:0 0 .6rem;font-size:.88rem;letter-spacing:.04em;text-transform:uppercase;color:var(--acc)}}
|
||
.arm h3 small{{display:block;text-transform:none;letter-spacing:0;color:var(--mut);font-weight:400;font-size:.8rem;margin-top:.15rem}}
|
||
.s{{border-top:1px dotted var(--line);padding-top:.6rem;margin-top:.6rem}}
|
||
.arm .s:first-of-type{{border-top:0;padding-top:0;margin-top:0}}
|
||
.seed{{display:block;font:600 .7rem/1 ui-monospace,monospace;color:var(--mut);margin-bottom:.25rem}}
|
||
.s p{{margin:0;white-space:pre-wrap;font-size:.94rem}}
|
||
footer{{margin-top:3rem;padding-top:1rem;border-top:1px solid var(--line);color:var(--mut);font-size:.85rem;max-width:84ch}}
|
||
</style>
|
||
<div class="wrap">
|
||
<h1>BabyBronte — rung 3: does the thread hold?</h1>
|
||
<p class="lede">0.6B gave the voice and not the sense. 1.7B brought back sentence-to-sentence
|
||
coherence but still lost the thread inside a passage. 4B is the last rung of the planned sweep, and
|
||
the open question is scene-level continuity.</p>
|
||
|
||
<table class="m"><tr><th>arm</th><th>curly quotes</th><th>worksheet / explainer collapse</th><th>hard-wrap ratio</th></tr>{tbl}</table>
|
||
|
||
<div class="warn"><strong>The base column is the control that earns the claim.</strong> Any difference
|
||
between the two tuned columns is carrier size and nothing else — identical corpus
|
||
(sha 77f37057b2782e49), seed, sampler and step count. Any difference between 4B base and 4B tuned is
|
||
the adapter and nothing else.</div>
|
||
|
||
<div class="warn"><strong>Still an eyeball test.</strong> Two samples per arm is enough to see whether
|
||
the gap between columns beats the gap between seeds inside one, and not enough for anything else. No
|
||
scoring; the frozen adjudication rule and the Burrows's-Delta instrument are untouched and nothing
|
||
here feeds them. All arms are <strong>base models doing continuation</strong>, so each prompt is an
|
||
opening line carried on rather than an instruction to rewrite.</div>
|
||
|
||
{''.join(rows)}
|
||
{ab_html}
|
||
{hand_html}
|
||
{extra_html}
|
||
|
||
<footer>Generated on pfi-gx10 (GB10), bf16, sdpa. Sampler identical across arms: temperature 0.9,
|
||
top_p 0.95, 400 new tokens (300 on the frame prompt), seeds 1234 and 5678. All tuned arms: 1 epoch,
|
||
seed 4919, corpus sha 77f37057b2782e49, 5,210,112 tokens, 159 steps. Held-out loss at plateau:
|
||
0.6B 3.329 · 1.7B 3.018 · 4B see the run log. Adapter binding proven at generation time on every
|
||
tuned arm (lora_B tensors non-zero).</footer>
|
||
</div>"""
|
||
(D / "index.html").write_text(page, encoding="utf-8")
|
||
print(f"wrote {D/'index.html'} ({len(ids)} prompts x 3 arms" + (", + frame prompt" if ab_html else "") + ")")
|