vh
d9cf486004
Skaldsong's requirement is beat-sentence in, paragraph out, stitchable. That is narrower than free-form continuation and it fails differently, so it was tested directly rather than assumed from the earlier prompt work. The adapted 4B carrier cannot do it. Ten prompt formats across three seeds each -- bare, paragraph-break, labelled, epigraph, one-shot, one-shot unlabelled, three-shot, "it happened in this way", "I remember the whole of it", and the letter prompt's label-and-begin move applied to a beat -- produced thirty samples and not one that reliably renders the beat. They drift, they frame, or they truncate. The cause is structural rather than a shortfall of the adapter: "write a paragraph about this sentence" is an instruction, and a completion model has no mechanism for "about". It continues the text it is handed. Two formats failed worse than drifting and leaked pretraining task data. The paragraph-break format emitted an NLI multiple-choice item, and label-begin emitted a grammar-correction exercise. A standalone sentence followed by a blank line looks exactly like a dataset entry, and style adaptation does not remove base-model task artifacts. The same five beats through the gen seat with a style instruction and no adapter landed all ten samples inside the requested 90 to 140 word band, every one on its beat, with no drift into a following scene. The voice, however, is generic literary pastiche -- abstract-noun-heavy, fond of aphoristic openers, and over-written -- rather than Brontë. Voice without direction against direction without voice. The product needs both, which makes rebuilding the corpus as instruction-response pairs on an instruct carrier the evidence-backed option rather than more prompt cleverness. This applies to Yarros identically, because the carrier question is orthogonal to the author, so the next corpus should not re-run this experiment. Stitching also has its own failure mode, shown in the booth's third panel: the paragraphs were generated independently and the point of view slides between them, so by the fourth beat the narrator is both watching the girl carry the animals and carrying them herself. A real stitcher has to feed prior paragraphs back as context, which means the instruction-pair corpus wants multi-paragraph continuity examples and not only isolated beat-to-paragraph pairs.
165 lines
9.8 KiB
Python
165 lines
9.8 KiB
Python
"""The Skaldsong question, answered: can a beat sentence be expanded to a paragraph?
|
||
|
||
The page is built as an argument rather than a gallery, because the result is a
|
||
negative one on the adapted carrier and a negative result presented as a gallery
|
||
reads as "some of these look fine".
|
||
|
||
Panel A the adapted 4B across ten prompt formats -- what does not work, and why
|
||
Panel B the same beats through an instruct model with a style prompt -- what does
|
||
Panel C Panel B's paragraphs stitched, which is the deliverable Skaldsong wants
|
||
|
||
Two artifacts in Panel A are worth their own callout: two formats leaked *pretraining
|
||
task data* -- NLI multiple choice and a grammar-correction exercise -- which is a
|
||
base-model failure mode that no amount of style adaptation removes.
|
||
"""
|
||
import html, json, statistics, sys
|
||
from collections import defaultdict
|
||
from pathlib import Path
|
||
|
||
D = Path(sys.argv[1])
|
||
|
||
|
||
def load(f):
|
||
rows = [json.loads(l) for l in (D / f).read_text(encoding="utf-8").splitlines() if l.strip()]
|
||
d = defaultdict(list)
|
||
for r in rows:
|
||
d[r["format"]].append(r)
|
||
return d, rows
|
||
|
||
|
||
bake, rows1 = load("bakeoff.jsonl")
|
||
bake2, rows2 = load("bakeoff2.jsonl")
|
||
inst, rows3 = load("instruct.jsonl")
|
||
bake.update(bake2)
|
||
allrows = rows1 + rows2
|
||
|
||
FMT_NOTES = {
|
||
"bare": "The beat alone. Continues the situation rather than expanding it, and leaves the kitten out.",
|
||
"para-break": "⚠ Leaked pretraining task data — NLI multiple choice. A standalone sentence followed by a blank line looks exactly like a dataset entry.",
|
||
"labelled": "Named the artifact. Produced abstract moralising about punishment and husbands.",
|
||
"epigraph": "The beat in italics as an epigraph. Drifts immediately.",
|
||
"fewshot": "One worked example. Echoed the beat with pronouns flipped, then drifted to unrelated gossip.",
|
||
"fewshot-bare": "One example, no labels. Returned single lines of dialogue, one borrowing a character from the example itself.",
|
||
"fewshot3": "Three worked examples. Still drifts — into a woman and her husband, a child, a nurse.",
|
||
"elaborate": "Beat plus “It happened in this way.” Commits to elaborating and elaborates something else.",
|
||
"recount": "Beat plus “I remember the whole of it.” Same.",
|
||
"label-begin": "The letter prompt's winning move applied to a beat — label it and seed the opening words. Closest of the ten, and one seed leaked a grammar-correction exercise instead.",
|
||
}
|
||
|
||
|
||
def para_block(r):
|
||
tag = f"{r['words']}w" + (" · ran on" if r.get("ran_on") else "")
|
||
return (f'<div class="s"><span class="seed">seed {r["seed"]} · {tag}</span>'
|
||
f'<p>{html.escape(r["paragraph"].strip()) or "<em>(empty)</em>"}</p></div>')
|
||
|
||
|
||
beat_one = allrows[0]["beat"] if allrows else ""
|
||
panelA = "".join(
|
||
f'<section class="row"><div class="fmt"><code>{html.escape(f)}</code>'
|
||
f'<span class="note">{FMT_NOTES.get(f, "")}</span></div>'
|
||
f'<div class="arm">{"".join(para_block(r) for r in bake[f])}</div></section>'
|
||
for f in FMT_NOTES if f in bake)
|
||
|
||
by_beat = defaultdict(list)
|
||
for r in rows3:
|
||
by_beat[r["id"]].append(r)
|
||
panelB = "".join(
|
||
f'<section class="row"><div class="prompt"><span class="pid">{html.escape(bid)}</span>'
|
||
f'{html.escape(rs[0]["beat"])}</div><div class="arm tuned">'
|
||
f'{"".join(para_block(r) for r in rs)}</div></section>'
|
||
for bid, rs in sorted(by_beat.items()))
|
||
|
||
stitched = "\n\n".join(r["paragraph"].strip() for bid, rs in sorted(by_beat.items())
|
||
for r in rs if r["seed"] == 1234)
|
||
wl = [r["words"] for r in rows3]
|
||
|
||
page = f"""<!doctype html><meta charset="utf-8"><title>Beat to paragraph</title>
|
||
<style>
|
||
:root{{--bg:#faf8f5;--fg:#1c1a17;--mut:#6b6560;--line:#e0dad2;--acc:#7a3b2e;--tint:#fdfbf7;--bad:#8a3a3a}}
|
||
*{{box-sizing:border-box}}
|
||
body{{margin:0;background:var(--bg);color:var(--fg);font:16px/1.65 Georgia,"Iowan Old Style",serif;padding:2.5rem 1.5rem 5rem}}
|
||
.wrap{{max-width:1080px;margin:0 auto}}
|
||
h1{{font-size:1.9rem;margin:0 0 .3rem}}
|
||
.lede{{color:var(--mut);max-width:74ch;margin:0 0 1rem}}
|
||
.box{{border-left:3px solid var(--acc);background:#fff;padding:.85rem 1rem;margin:1.1rem 0;max-width:84ch;font-size:.94rem}}
|
||
.box.verdict{{border-left-color:var(--bad)}}
|
||
h2{{font-size:1.2rem;margin:3rem 0 .2rem;padding-top:1rem;border-top:2px solid var(--line)}}
|
||
.tsub{{color:var(--mut);font-size:.92rem;margin:0 0 1.3rem;font-style:italic;max-width:84ch}}
|
||
.row{{margin:0 0 1.5rem}}
|
||
.fmt{{background:#fff;border:1px solid var(--line);border-left:3px solid var(--acc);padding:.55rem .8rem;margin-bottom:.5rem}}
|
||
.fmt code{{font:600 .82rem/1 ui-monospace,monospace;color:var(--acc)}}
|
||
.fmt .note{{display:block;color:var(--mut);font-size:.88rem;margin-top:.2rem}}
|
||
.prompt{{background:#fff;border:1px solid var(--line);border-left:3px solid var(--acc);padding:.6rem .85rem;margin-bottom:.5rem}}
|
||
.pid{{display:inline-block;font:600 .72rem/1 ui-monospace,monospace;color:var(--mut);background:var(--bg);border:1px solid var(--line);padding:.22rem .4rem;margin-right:.55rem;vertical-align:1px}}
|
||
.arm{{background:#f5f6f8;border:1px solid var(--line);padding:.8rem .95rem}}
|
||
.arm.tuned{{background:var(--tint);border-color:#d8ccbe}}
|
||
.s{{border-top:1px dotted var(--line);padding-top:.55rem;margin-top:.55rem}}
|
||
.arm .s:first-of-type{{border-top:0;padding-top:0;margin-top:0}}
|
||
.seed{{display:block;font:600 .7rem/1 ui-monospace,monospace;color:var(--mut);margin-bottom:.22rem}}
|
||
.s p{{margin:0;white-space:pre-wrap;font-size:.94rem}}
|
||
.story{{background:#fff;border:1px solid var(--line);padding:1.4rem 1.6rem;font-size:1.02rem;white-space:pre-wrap}}
|
||
footer{{margin-top:3rem;padding-top:1rem;border-top:1px solid var(--line);color:var(--mut);font-size:.85rem;max-width:84ch}}
|
||
</style>
|
||
<div class="wrap">
|
||
<h1>Beat → paragraph: can the adapter do Skaldsong's job?</h1>
|
||
<p class="lede">Skaldsong wants to write story beats as single sentences, have a model expand each
|
||
into a paragraph, and stitch the paragraphs into a passable story. That is a narrower job than
|
||
free-form continuation, and it fails differently.</p>
|
||
|
||
<div class="box"><strong>Four ways this job breaks, all of which had to be measured rather than
|
||
eyeballed.</strong> <em>Drift</em> off the beat breaks the stitch, because the next paragraph no
|
||
longer follows. <em>Run-on</em> breaks it too — the deliverable is a paragraph, and the following
|
||
scene belongs to the next beat. <em>Framing</em> renders nothing at all ("I told it briefly").
|
||
<em>Renaming</em> is a live blocker: the entity-rename pool taught the adapter that character names
|
||
come from it, so a caller's own name can be rewritten mid-passage.</div>
|
||
|
||
<h2>Panel A — the adapted 4B, ten prompt formats</h2>
|
||
<p class="tsub">One beat, three seeds each, thirty samples. The beat is
|
||
<strong>"{html.escape(beat_one)}"</strong>. Read as many as you like; the finding is that none of
|
||
them render it.</p>
|
||
{panelA}
|
||
|
||
<div class="box verdict"><strong>Ten formats, thirty samples, none that reliably expand the beat.</strong>
|
||
The adapter writes Brontë well — that is settled elsewhere — but "write a paragraph <em>about</em>
|
||
this sentence" is an instruction, and a completion model has no mechanism for <em>about</em>. It
|
||
continues the text it is given. Two formats did something worse than drift and leaked
|
||
<strong>pretraining task data</strong>: an NLI multiple-choice item and a grammar-correction
|
||
exercise. That is a base-model artifact which no amount of style adaptation removes.</div>
|
||
|
||
<h2>Panel B — the same beats through an instruct model</h2>
|
||
<p class="tsub">The gen seat (Qwen3.8-27B, post-trained, no Brontë adapter) with a style instruction
|
||
asking for one paragraph of 90–140 words in her manner. Five beats, two seeds.</p>
|
||
{panelB}
|
||
|
||
<div class="box"><strong>It takes direction perfectly and has the wrong voice.</strong> All
|
||
{len(wl)} samples landed inside the requested band — {min(wl)}–{max(wl)} words, median
|
||
{statistics.median(wl):.0f} — every one stayed on its beat, and none drifted into a following scene.
|
||
But the prose is generic literary pastiche rather than Brontë: abstract-noun-heavy, fond of
|
||
aphoristic openers ("There is a peculiar, chilling stillness that attends the discovery of a life
|
||
nearly spent"), and it over-writes. Brontë is more concrete and more sharply observed than this.</div>
|
||
|
||
<h2>Panel C — Panel B's paragraphs, stitched</h2>
|
||
<p class="tsub">The deliverable shape, so the failure modes of stitching are visible too. Each
|
||
paragraph was generated independently, which is itself the next problem: watch the point of view
|
||
slide between beats — by the fourth the narrator is both watching the girl and carrying the animals.
|
||
A real stitcher has to feed prior paragraphs back as context.</p>
|
||
<div class="story">{html.escape(stitched)}</div>
|
||
|
||
<div class="box verdict"><strong>The conclusion, and it settles an architecture question.</strong>
|
||
The adapted completion carrier has the voice and cannot take direction. The instruct model takes
|
||
direction and has no voice. Skaldsong's job needs both, which means the corpus has to be rebuilt as
|
||
instruction→response pairs and trained onto an instruct carrier — not more prompt cleverness, which
|
||
is now ten formats deep with nothing to show. <strong>This applies to Yarros identically</strong>:
|
||
the carrier question is orthogonal to the author, so the next corpus does not need to re-run this
|
||
experiment.</div>
|
||
|
||
<footer>Adapted arm: Qwen3-4B-Base + H02 LoRA checkpoint-75 (held-out 2.8138, the loss minimum) on
|
||
pfi-gx10, bf16, sdpa, temperature 0.9 / top_p 0.95, 300 new tokens, truncated at the first paragraph
|
||
break for display with run-on reported. Instruct arm: `gen` on ana-ml2 via the gateway, temperature
|
||
0.9 / top_p 0.95, thinking off. Neither panel is scored; the frozen adjudication rule and the
|
||
Burrows's-Delta instrument are untouched and nothing here feeds them.</footer>
|
||
</div>"""
|
||
(D / "index.html").write_text(page, encoding="utf-8")
|
||
print(f"wrote {D/'index.html'} (panel A {sum(len(v) for v in bake.values())} samples, "
|
||
f"panel B {len(rows3)}, stitched {len(stitched.split())} words)")
|