Complete R49 rung 2 and booth the three-way voice comparison
Both rungs now sit on the same unwrapped corpus with seed, steps and token count held, so carrier size is the only difference and the effect is attributable: held-out 3.329 at 0.6B against 3.018 at 1.7B, a gap of 0.311 nats. The chained 0.6B rerun closed the confound the unwrap opened. Two things in those numbers need stating or they will be misread. First, the original wrapped-corpus 0.6B reached 3.172, which looks better than the unwrapped 0.6B's 3.329 and is not. Different corpus means a different held-out set, and the wrapped version's 5.7% newline tokens are near-deterministic after a 70-character line, so they deflate the loss with cheap wins. Removing them removed the easy tokens. It is a measurement artifact, not a regression. Second, a correction to my own earlier claim: I twice described the 0.6B run as still descending and undertrained at 3.172. Its series reads 3.176, 3.173, 3.172, 3.172 -- it flattened. All three runs plateau, so one epoch is about right for this corpus rather than short. The three-way booth puts 1.7B base, 1.7B tuned and 0.6B tuned side by side on the same nine prompts and seeds. The base arm is the control that matters: curly quotes go 0 of 18 on 1.7B base to 15 of 18 on 1.7B tuned, and worksheet-or- explainer collapse goes 3 of 18 to 0 of 18, so the shift is the adapter rather than the larger carrier. Hard-wrapping fell from 0.85 to 0.18, confirming the corpus unwrap carried through into the adapter. Sense partially returned. The 1.7B arm produces locally coherent sequential Victorian prose where the 0.6B produced word salad, but scene-level continuity still breaks mid-passage. One observation held loosely: curly quotes are slightly lower at 1.7B than 0.6B, which would fit a bigger model's stronger priors resisting the adapter at the same rank. That is untested and is not offered as established.
This commit is contained in:
@@ -206,6 +206,8 @@ _As of 2026-09-10 10:25 PT._
|
||||
|
||||
## Recent decisions
|
||||
|
||||
- `[2026-09-10]` **R49 rung 2 COMPLETE, and the single-variable carrier effect is clean: 0.6B held-out 3.329 vs 1.7B 3.018, Δ0.311 nats.** Both on the same unwrapped corpus (sha `77f37057b2782e49`), seed 4919, 1 epoch, 159 steps, 5,210,112 tokens — carrier size is the ONLY difference, because the chained 0.6B rerun closed the confound the unwrap opened. ⚠⚠ **DO NOT compare either against the original wrapped-corpus 0.6B run's 3.172 — that comparison is INVALID and reads backwards.** Different corpus means a different held-out set: the wrapped version's 5.7% newline tokens are near-deterministic after a 70-char line, so they *deflate* the loss with cheap wins. Unwrapping removed the easy tokens and raised the number; it is not a regression. ⚠ **Correction to my own earlier claim**: I twice described the 0.6B as "still descending, undertrained" at 3.172 — the series (3.176, 3.173, 3.172, 3.172) shows it FLATTENED. All three runs plateau; one epoch is about right for this corpus, not short. **Three-way eyeball booth** at `http://10.100.10.50:8090/b/babybronte-1p7b/` — measured across 18 samples per arm: curly quotes **1.7B base 0/18 → 1.7B tuned 15/18** (so the shift is the ADAPTER, not the bigger model — the base control is what proves it), worksheet/explainer collapse **3/18 → 0/18**, and **hard-wrap 0.85 → 0.18**, confirming the corpus unwrap carried through into the adapter. **Sense partially returned**: 1.7B produces locally coherent sequential Victorian prose where 0.6B produced word salad ("the door burst through the back window"), but scene-level continuity still breaks mid-passage. ⚠ Curly quotes are slightly LOWER at 1.7B (15/18) than 0.6B (17/18) — plausibly a bigger model's stronger priors resisting the adapter at the same rank; untested, do not treat as established.
|
||||
|
||||
- `[2026-09-10]` **R49 rung 2 LAUNCHED: Qwen3-1.7B-Base, 1 epoch, seed 4919, on an UNWRAPPED corpus.** Operator: *"start the 1.7b training."* Live at `gx10:~/r49-runs/h02-1p7b-1ep/`, 159 steps at ~18.7 s/it (~50 min), corpus sha **`77f37057b2782e49`**. A 0.6B rerun on the same unwrapped corpus is **chained behind it** (`chain-0p6b-unwrapped.sh`, gated on the 1.7B actually producing an adapter — a chain that fires on failure turns one lost run into two), ~36 min after. ⚠⚠ **THE CORPUS CHANGED, SO 0.6B-vs-1.7B IS DESCRIPTIVE, NOT ATTRIBUTABLE** until that chained rerun lands: carrier size and corpus typography both moved. *"Did sense come back at 1.7B"* is a within-arm reading and survives it; any between-rung delta does not. **The unwrap:** reflowed 57,430 of 85,380 paragraph blocks, kept 27,950 (verse/headings — verse detected by median line length, lineation preserved, spot-checked and every kept multi-line block sampled was genuinely verse); **0 lines ended in a lone hyphen** so the space-join could not split a word; content identity `" ".join(text.split())` verified byte-identical on all **852 records**, i.e. whitespace-only. Mid-length-line ratio **0.94 → 0.25** (the residual is the preserved verse). ⚠ Concrete cost of the old defect: **5.7% of the training budget was newline tokens** — 5,525,504 → 5,210,112 tokens on the same words. Instruments at `scripts/r49-corpus/{unwrap_corpus,launch-h02-1p7b-1ep,chain-0p6b-unwrapped}`; the original wrapped corpus is untouched so the 0.6B run's pinned sha `3959036cf851bf62` stays reproducible.
|
||||
|
||||
- `[2026-09-10]` **BabyBronte H02 adapter: the VOICE transferred, the SENSE did not — operator's read, "it's all nonsense, but it sounds like Brontë's nonsense."** Eyeball A/B (NOT the adjudication; nothing here feeds the frozen rule), 9 arbitrary prompts on a deliberate difficulty gradient × 2 arms × 2 seeds, booth at `http://10.100.10.50:8090/b/babybronte-voice/`. Measured across the 18 pairs: **curly quotes 1/18 base → 18/18 tuned**, **math/worksheet collapse 3/18 base → 0/18 tuned**. Given *"The self-checkout machine refused her coupon"* the base 0.6B produced a **quadratic-formula worksheet**; the tuned arm wrote a clerk refusing a customer in Victorian retrospective first person. This is the expected and informative result for the smallest rung — **voice is separable from coherence at 0.6B**, which is the premise the whole lightweight-adapter regime rests on, and the 1.7B/4B rungs are where sense should return. The 1-epoch loss was still descending at step 169 (undertrained, not overfit), so the incoherence is carrier capacity, not training. ⚠ **Corpus-prep defect found: the tuned output is hard-wrapped at ~70 chars** (median mid-length-line ratio 0.85 vs base 0.00) — the Gutenberg source kept its original line breaks and the adapter learned the typography along with the voice. Unwrap to flowing paragraphs before any real use or the next rung learns it too.
|
||||
|
||||
@@ -0,0 +1,133 @@
|
||||
"""Render the three-arm carrier comparison into a booth page.
|
||||
|
||||
Three columns, chosen so the page answers two questions at once and neither answer
|
||||
leans on the other:
|
||||
|
||||
1.7B base vs 1.7B tuned -- did the ADAPTER do anything at this carrier size,
|
||||
or is any improvement just the bigger model?
|
||||
0.6B tuned vs 1.7B tuned -- did coherence come back as the carrier grew?
|
||||
|
||||
Both tuned arms sit on the SAME unwrapped corpus (sha 77f37057b2782e49), same seed,
|
||||
same sampler, so carrier size is the only difference between them. The 1.7B base arm
|
||||
is generated fresh rather than reused, because a control from a different model would
|
||||
control for nothing.
|
||||
|
||||
Prompts run hardest-first: modern/mundane, then period-neutral, then Victorian-
|
||||
adjacent. Both seeds of every arm sit in the same cell so within-arm sampling noise
|
||||
is visible in the same glance as between-arm difference -- if two samples of one arm
|
||||
differ as much as two arms differ, the page should make that obvious rather than hide
|
||||
it.
|
||||
"""
|
||||
import html
|
||||
import json
|
||||
import sys
|
||||
from collections import defaultdict
|
||||
from pathlib import Path
|
||||
|
||||
out_dir = Path(sys.argv[1])
|
||||
ARMS = [
|
||||
("1p7b-base.jsonl", "1.7B base", "Qwen3-1.7B-Base, no adapter", ""),
|
||||
("1p7b-tuned.jsonl", "1.7B tuned", "+ H02 LoRA, 1 epoch, seed 4919", "tuned"),
|
||||
("0p6b-tuned.jsonl", "0.6B tuned", "+ H02 LoRA, same corpus & seed", "small"),
|
||||
]
|
||||
out_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
def load(p):
|
||||
d = defaultdict(dict)
|
||||
for line in Path(p).read_text(encoding="utf-8").splitlines():
|
||||
if line.strip():
|
||||
r = json.loads(line)
|
||||
d[r["id"]][r["seed"]] = r
|
||||
return d
|
||||
|
||||
data = [(lbl, sub, cls, load(out_dir / f)) for f, lbl, sub, cls in ARMS]
|
||||
ids = sorted(set.intersection(*[set(d) for *_, d in data]))
|
||||
TIER = {"modern": ("Tier A — modern / mundane",
|
||||
"Nothing here invites Victorian prose. Brontë in this tier is the adapter's doing."),
|
||||
"neutral": ("Tier B — period-neutral",
|
||||
"Could be any century. A voice shift shows without the prompt supplying it."),
|
||||
"period": ("Tier C — Victorian-adjacent, plainly worded",
|
||||
"The setting leans period but the diction does not. Easiest tier; weakest evidence.")}
|
||||
order = {"modern": 0, "neutral": 1, "period": 2}
|
||||
tier_of = {i: data[0][3][i][list(data[0][3][i])[0]]["tier"] for i in ids}
|
||||
ids.sort(key=lambda i: (order.get(tier_of[i], 9), i))
|
||||
|
||||
def cell(by_seed):
|
||||
return "".join(
|
||||
f'<div class="s"><span class="seed">seed {s}</span>'
|
||||
f'<p>{html.escape((by_seed[s]["continuation"] or "").strip()) or "<em>(empty)</em>"}</p></div>'
|
||||
for s in sorted(by_seed))
|
||||
|
||||
rows, seen = [], set()
|
||||
for i in ids:
|
||||
if tier_of[i] not in seen:
|
||||
seen.add(tier_of[i])
|
||||
title, sub = TIER.get(tier_of[i], (tier_of[i], ""))
|
||||
rows.append(f'<h2>{html.escape(title)}</h2><p class="tsub">{html.escape(sub)}</p>')
|
||||
p = data[0][3][i][list(data[0][3][i])[0]]["prompt"]
|
||||
cols = "".join(
|
||||
f'<div class="arm {cls}"><h3>{lbl} <small>{sub}</small></h3>{cell(d[i])}</div>'
|
||||
for lbl, sub, cls, d in data)
|
||||
rows.append(f'<section class="row"><div class="prompt"><span class="pid">'
|
||||
f'{html.escape(i)}</span>{html.escape(p)}</div>'
|
||||
f'<div class="arms">{cols}</div></section>')
|
||||
|
||||
page = f"""<!doctype html><meta charset="utf-8"><title>BabyBronte — 1.7B rung</title>
|
||||
<style>
|
||||
:root{{--bg:#faf8f5;--fg:#1c1a17;--mut:#6b6560;--line:#e0dad2;--acc:#7a3b2e;--tint:#fdfbf7;--cool:#f5f6f8}}
|
||||
*{{box-sizing:border-box}}
|
||||
body{{margin:0;background:var(--bg);color:var(--fg);font:16px/1.6 Georgia,"Iowan Old Style",serif;padding:2.5rem 1.5rem 5rem}}
|
||||
.wrap{{max-width:1500px;margin:0 auto}}
|
||||
h1{{font-size:1.9rem;margin:0 0 .3rem}}
|
||||
.lede{{color:var(--mut);max-width:74ch;margin:0 0 .9rem}}
|
||||
.warn{{border-left:3px solid var(--acc);background:#fff;padding:.8rem 1rem;margin:1.1rem 0;max-width:84ch;font-size:.93rem}}
|
||||
h2{{font-size:1.15rem;margin:2.8rem 0 .2rem;padding-top:1rem;border-top:1px solid var(--line)}}
|
||||
.tsub{{color:var(--mut);font-size:.9rem;margin:0 0 1.2rem;font-style:italic}}
|
||||
.row{{margin:0 0 2.2rem}}
|
||||
.prompt{{background:#fff;border:1px solid var(--line);border-left:3px solid var(--acc);padding:.7rem .9rem;font-size:1.02rem;margin-bottom:.7rem}}
|
||||
.pid{{display:inline-block;font:600 .72rem/1 ui-monospace,monospace;color:var(--mut);background:var(--bg);border:1px solid var(--line);padding:.22rem .4rem;margin-right:.6rem;vertical-align:1px}}
|
||||
.arms{{display:grid;grid-template-columns:repeat(3,1fr);gap:.9rem}}
|
||||
@media(max-width:1100px){{.arms{{grid-template-columns:1fr}}}}
|
||||
.arm{{background:var(--cool);border:1px solid var(--line);padding:.85rem .95rem}}
|
||||
.arm.tuned{{background:var(--tint);border-color:#d8ccbe}}
|
||||
.arm.small{{background:#fbf9fb;border-color:#ded6e0}}
|
||||
.arm h3{{margin:0 0 .6rem;font-size:.88rem;letter-spacing:.04em;text-transform:uppercase;color:var(--acc)}}
|
||||
.arm h3 small{{display:block;text-transform:none;letter-spacing:0;color:var(--mut);font-weight:400;font-size:.8rem;margin-top:.15rem}}
|
||||
.s{{border-top:1px dotted var(--line);padding-top:.6rem;margin-top:.6rem}}
|
||||
.arm .s:first-of-type{{border-top:0;padding-top:0;margin-top:0}}
|
||||
.seed{{display:block;font:600 .7rem/1 ui-monospace,monospace;color:var(--mut);margin-bottom:.25rem}}
|
||||
.s p{{margin:0;white-space:pre-wrap;font-size:.94rem}}
|
||||
footer{{margin-top:3rem;padding-top:1rem;border-top:1px solid var(--line);color:var(--mut);font-size:.85rem;max-width:84ch}}
|
||||
</style>
|
||||
<div class="wrap">
|
||||
<h1>BabyBronte — rung 2: did the sense come back?</h1>
|
||||
<p class="lede">The 0.6B rung transferred the voice and not the coherence — "it's all nonsense, but it
|
||||
sounds like Brontë's nonsense." This is the same nine prompts at 1.7B, with the 0.6B tuned arm beside
|
||||
it for scale and the 1.7B base arm beside it for control.</p>
|
||||
|
||||
<div class="warn"><strong>Two questions, two columns each.</strong>
|
||||
<em>1.7B base vs 1.7B tuned</em> asks whether the adapter did anything at this carrier size, or
|
||||
whether any improvement is just the bigger model.
|
||||
<em>0.6B tuned vs 1.7B tuned</em> asks whether coherence returned as the carrier grew — and those two
|
||||
tuned arms sit on the <strong>same corpus, same seed, same sampler</strong>, so carrier size is the
|
||||
only difference between them.</div>
|
||||
|
||||
<div class="warn"><strong>Still an eyeball test, not a result.</strong> Two samples per arm is enough
|
||||
to see whether the gap between columns beats the gap between seeds inside one — and not enough for
|
||||
anything else. No scoring. The frozen adjudication rule and the Burrows's-Delta instrument are
|
||||
untouched and nothing here feeds them.<br><br>
|
||||
All arms are base models doing <strong>continuation</strong>, not instruction-following, so each
|
||||
prompt is an opening line carried on rather than an instruction to rewrite.
|
||||
<strong>The corpus was unwrapped since the last booth</strong>, so the ~70-character hard wrapping
|
||||
that disfigured the first 0.6B page should be gone from both tuned arms here.</div>
|
||||
|
||||
{''.join(rows)}
|
||||
|
||||
<footer>Generated on pfi-gx10 (GB10), bf16, sdpa. Sampler identical across all three arms:
|
||||
temperature 0.9, top_p 0.95, 400 new tokens, seeds 1234 and 5678. Both tuned arms: 1 epoch,
|
||||
seed 4919, corpus sha 77f37057b2782e49 (5,210,112 tokens, 159 steps). Held-out loss at the plateau:
|
||||
0.6B 3.329, 1.7B 3.018. Adapter binding proven at generation time on both tuned arms
|
||||
(196/196 lora_B tensors non-zero).</footer>
|
||||
</div>"""
|
||||
(out_dir / "index.html").write_text(page, encoding="utf-8")
|
||||
print(f"wrote {out_dir/'index.html'} ({len(ids)} prompts x 3 arms x 2 seeds)")
|
||||
Reference in New Issue
Block a user