diff --git a/persistent-memory.md b/persistent-memory.md index 5f1d260..75f6cbe 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -206,6 +206,9 @@ _As of 2026-09-10 10:25 PT._ ## Recent decisions +- `[2026-09-11]` **4B arms RE-CUT from `checkpoint-75`, the true loss minimum (2.813826, confirmed from `loss-series.json` rather than my reading of the log); booth rebuilt.** Only the tuned arms needed it — the base arm never touches the adapter. ⚠ **A small surprise: step-75 and end-of-run differ on typography, not voice.** Curly quotes 16/18 vs 17/18 and collapse 0/18 either way, but the **hard-wrap ratio is 0.33 at step-75 against 0.12 at end-of-run** — further training washes the residual line-break habit out *while held-out loss gets worse*. So **"best loss" and "best typography" are different checkpoints**; neither is near the original 0.85 defect, and the corpus's own residual (preserved verse) is 0.25. +- `[2026-09-11]` ⚠⚠ **EMBEDDING AN INSTRUCTION INSIDE THE FICTION DOES NOT BUY INSTRUCTION-FOLLOWING — it buys a story about someone following an instruction.** Operator prompt had Abernathy tell the tale badly then ask the narrator: *"Honey, you were there—please retell the story in a few short paragraphs."* Across **6 seeds** (3 as written, 3 with a trailing paragraph break) the model **acknowledged the handoff every time and never once performed it**: *"I told it, briefly, to his satisfaction"*, *"So I wrote it out, and kept it in my pocket-book"*, and one seed **negotiated the brief in character** — *"I will retell it, but I cannot condense it in a few short paragraphs—there are too many points to touch."* Structural reason: in a novel *"she retold the story"* is an ordinary sentence, so the likeliest continuation of a request is **narration of compliance**. ⚠ The trailing paragraph break DID shift behaviour (one seed opened in the narrator's own quoted speech), so typography is a real lever — just not a sufficient one. **This is direct evidence for the instruct question the operator raised**: if the product is "ask for a scene and get the scene", no amount of in-fiction framing substitutes for a post-trained instruction-follower, which favours rebuilding the corpus as instruction pairs (option C) over more prompt cleverness. + - `[2026-09-11]` **R49 SWEEP COMPLETE — 4B closes the continuity gap, and the carrier ladder is clean: 3.329 → 3.018 → 2.814 held-out** (0.6B / 1.7B / 4B, all on the same unwrapped corpus sha `77f37057b2782e49`, seed 4919, 159 steps, 5,210,112 tokens — carrier size the only variable). Deltas **0.311 then 0.204**: diminishing but still real. Booth: `http://10.100.10.50:8090/b/babybronte-4b/`. **4B tuned has the best voice saturation of any rung — curly quotes 17/18 against its own base arm's 1/18, collapse 0/18 against 4/18** — and, the thing the rung existed to test, **scene-level continuity HOLDS**: it produces a named character with motivated dialogue, a navigable spatial layout and a physical description in one passage, where 1.7B wrote pretty but eventless prose (opening doors, looking at stars). On the letter prompt it opens the letter, promises to quote it, and then actually quotes it across a paragraph break. - `[2026-09-11]` ⚠⚠ **4B is the FIRST rung to OVERFIT inside one epoch, which inverts my earlier "one epoch is right for this corpus" call.** Series 2.832 · 2.816 · **2.814** · 2.820 · 2.824 · 2.825 · 2.825 — minimum at ~step 75, then it TURNS and settles worse. 0.6B and 1.7B both plateaued with no turn, so **the optimal epoch count shrinks as the carrier grows** — 4B wants roughly half an epoch. ⚠ **Consequence: the shipped `adapter/` at `h02-4b-1ep/` is NOT the best checkpoint** (it is the end-of-run 2.825); the step-75 checkpoint at 2.814 is, and it exists only because `save_steps=25` was set. The voice test used the end-of-run adapter, so the booth understates 4B by ~0.011 nats. Re-cut the arms off the step-75 checkpoint before any adjudication. - `[2026-09-11]` **The tone-override appears to close at 4B too.** On the operator's Abernathy frame prompt ("a *wonderful* story"), 1.7B held the frame on every seed but **2 of 4 killed the animals anyway**; 4B kept them alive on **2 of 2** and one seed did something new — the narrator *doubts Abernathy's story* ("I felt sure the thing was a lie"), then supplies a parallel childhood memory of his own puppy and his sister's kitten to explain the doubt. That is a narrator with an interior position on the tale being told. ⚠ n=2 per arm; directionally right, not established. diff --git a/scripts/r49-corpus/build_booth4b.py b/scripts/r49-corpus/build_booth4b.py index 50b8621..3de4f5c 100644 --- a/scripts/r49-corpus/build_booth4b.py +++ b/scripts/r49-corpus/build_booth4b.py @@ -22,7 +22,7 @@ from pathlib import Path D = Path(sys.argv[1]) ARMS = [("4b-base.jsonl", "4B base", "Qwen3-4B-Base, no adapter", ""), - ("4b-tuned.jsonl", "4B tuned", "+ H02 LoRA, 1 epoch, seed 4919", "tuned"), + ("4b-tuned.jsonl", "4B tuned", "+ H02 LoRA, checkpoint-75 (the loss minimum)", "tuned"), ("1p7b-tuned.jsonl", "1.7B tuned", "the rung below, same corpus & seed", "small")] AB = [("4b-base-abernathy.jsonl", "4B base", ""), ("4b-tuned-abernathy.jsonl", "4B tuned", "tuned")] @@ -97,6 +97,29 @@ if all((D / f).exists() for f, _, _ in AB): f'
{html.escape(pr)}
' f'
{cols}
') +hand_html = "" +hf = D / "handoff.jsonl" +if hf.exists(): + hd = load(hf) + blocks = [] + for pid, label in (("handoff", "as written"), ("handoff-break", "identical, plus a trailing paragraph break")): + if pid not in hd: + continue + pr = hd[pid][list(hd[pid])[0]]["prompt"] + blocks.append(f'
{html.escape(label)}' + f'{html.escape(pr.strip())}
' + f'
' + f'{cell(hd[pid])}
') + hand_html = ('

The embedded-instruction prompt

The instruction lives ' + 'INSIDE the fiction — Abernathy asks the narrator to retell the story — which is the ' + 'only way to hand an instruction to a completion model. Watch what it does with the ' + 'request: across every seed it narrates the retelling rather than performing it ' + '("I told it, briefly", "So I wrote it out", "I will retell it, but I cannot ' + 'condense it"). In a novel, she retold the story is an ordinary sentence, ' + 'so the likeliest continuation of a request is narration of compliance — not ' + 'compliance. One seed even negotiates the word count in character and still never ' + 'tells it.

' + "".join(blocks) + '
') + page = f"""BabyBronte — 4B rung