Re-cut the 4B arms from checkpoint-75 and test an in-fiction instruction

The 4B tuned arms now come from checkpoint-75, the actual loss minimum at 2.813826
confirmed from loss-series.json rather than from reading the training log. Only the
tuned arms needed re-cutting; the base arm never touches the adapter.

One surprise in the comparison: step-75 and end-of-run differ on typography rather
than voice. Curly quotes read 16 of 18 against 17, collapse is 0 either way, but
the hard-wrap ratio is 0.33 at step-75 against 0.12 at the end. Further training
washes the residual line-break habit out while held-out loss gets worse, so best
loss and best typography are different checkpoints. Neither is near the original
0.85 defect and the corpus's own residual from preserved verse is 0.25.

The operator's second prompt embeds the instruction inside the fiction: Abernathy
tells the tale badly and then asks the narrator to retell it in a few short
paragraphs. Across six seeds the model acknowledged the handoff every time and
never once performed it. It wrote "I told it, briefly, to his satisfaction", and
"So I wrote it out, and kept it in my pocket-book", and one seed negotiated the
brief in character, objecting that it could not condense the story into a few short
paragraphs because there were too many points to touch.

The reason is structural rather than a failure of the adapter. In a novel, "she
retold the story" is an ordinary sentence, so the likeliest continuation of a
request is narration of compliance rather than compliance.

A trailing paragraph break did shift the behaviour -- one seed opened in the
narrator's own quoted speech -- so typography is a real lever, just not a
sufficient one. That is direct evidence on the instruct question raised earlier: if
the product is asking for a scene and getting the scene, no amount of in-fiction
framing substitutes for a post-trained instruction-follower, which favours
rebuilding the corpus as instruction pairs over more prompt cleverness.
This commit is contained in:
vh
2026-09-11 05:55:15 -07:00
parent 25eb65e23b
commit e47295d2ce
3 changed files with 29 additions and 1 deletions
+25 -1
View File
@@ -22,7 +22,7 @@ from pathlib import Path
D = Path(sys.argv[1])
ARMS = [("4b-base.jsonl", "4B base", "Qwen3-4B-Base, no adapter", ""),
("4b-tuned.jsonl", "4B tuned", "+ H02 LoRA, 1 epoch, seed 4919", "tuned"),
("4b-tuned.jsonl", "4B tuned", "+ H02 LoRA, <b>checkpoint-75</b> (the loss minimum)", "tuned"),
("1p7b-tuned.jsonl", "1.7B tuned", "the rung below, same corpus &amp; seed", "small")]
AB = [("4b-base-abernathy.jsonl", "4B base", ""), ("4b-tuned-abernathy.jsonl", "4B tuned", "tuned")]
@@ -97,6 +97,29 @@ if all((D / f).exists() for f, _, _ in AB):
f'<section class="row"><div class="prompt">{html.escape(pr)}</div>'
f'<div class="arms two">{cols}</div></section>')
hand_html = ""
hf = D / "handoff.jsonl"
if hf.exists():
hd = load(hf)
blocks = []
for pid, label in (("handoff", "as written"), ("handoff-break", "identical, plus a trailing paragraph break")):
if pid not in hd:
continue
pr = hd[pid][list(hd[pid])[0]]["prompt"]
blocks.append(f'<div class="prompt"><span class="pid">{html.escape(label)}</span>'
f'{html.escape(pr.strip())}</div>'
f'<div class="arms"><div class="arm tuned" style="grid-column:1/-1">'
f'{cell(hd[pid])}</div></div>')
hand_html = ('<h2>The embedded-instruction prompt</h2><p class="tsub">The instruction lives '
'INSIDE the fiction — Abernathy asks the narrator to retell the story — which is the '
'only way to hand an instruction to a completion model. Watch what it does with the '
'request: across every seed it narrates the retelling rather than performing it '
'("I told it, briefly", "So I wrote it out", "I will retell it, but I cannot '
'condense it"). In a novel, <em>she retold the story</em> is an ordinary sentence, '
'so the likeliest continuation of a request is narration of compliance — not '
'compliance. One seed even negotiates the word count in character and still never '
'tells it.</p><section class="row">' + "".join(blocks) + '</section>')
page = f"""<!doctype html><meta charset="utf-8"><title>BabyBronte — 4B rung</title>
<style>
:root{{--bg:#faf8f5;--fg:#1c1a17;--mut:#6b6560;--line:#e0dad2;--acc:#7a3b2e;--tint:#fdfbf7;--cool:#f5f6f8}}
@@ -149,6 +172,7 @@ opening line carried on rather than an instruction to rewrite.</div>
{''.join(rows)}
{ab_html}
{hand_html}
<footer>Generated on pfi-gx10 (GB10), bf16, sdpa. Sampler identical across arms: temperature 0.9,
top_p 0.95, 400 new tokens (300 on the frame prompt), seeds 1234 and 5678. All tuned arms: 1 epoch,