Settle the Skaldsong architecture question with a ten-format negative result
Skaldsong's requirement is beat-sentence in, paragraph out, stitchable. That is narrower than free-form continuation and it fails differently, so it was tested directly rather than assumed from the earlier prompt work. The adapted 4B carrier cannot do it. Ten prompt formats across three seeds each -- bare, paragraph-break, labelled, epigraph, one-shot, one-shot unlabelled, three-shot, "it happened in this way", "I remember the whole of it", and the letter prompt's label-and-begin move applied to a beat -- produced thirty samples and not one that reliably renders the beat. They drift, they frame, or they truncate. The cause is structural rather than a shortfall of the adapter: "write a paragraph about this sentence" is an instruction, and a completion model has no mechanism for "about". It continues the text it is handed. Two formats failed worse than drifting and leaked pretraining task data. The paragraph-break format emitted an NLI multiple-choice item, and label-begin emitted a grammar-correction exercise. A standalone sentence followed by a blank line looks exactly like a dataset entry, and style adaptation does not remove base-model task artifacts. The same five beats through the gen seat with a style instruction and no adapter landed all ten samples inside the requested 90 to 140 word band, every one on its beat, with no drift into a following scene. The voice, however, is generic literary pastiche -- abstract-noun-heavy, fond of aphoristic openers, and over-written -- rather than Brontë. Voice without direction against direction without voice. The product needs both, which makes rebuilding the corpus as instruction-response pairs on an instruct carrier the evidence-backed option rather than more prompt cleverness. This applies to Yarros identically, because the carrier question is orthogonal to the author, so the next corpus should not re-run this experiment. Stitching also has its own failure mode, shown in the booth's third panel: the paragraphs were generated independently and the point of view slides between them, so by the fourth beat the narrator is both watching the girl carry the animals and carrying them herself. A real stitcher has to feed prior paragraphs back as context, which means the instruction-pair corpus wants multi-paragraph continuity examples and not only isolated beat-to-paragraph pairs.
This commit is contained in:
@@ -206,6 +206,9 @@ _As of 2026-09-10 10:25 PT._
|
|||||||
|
|
||||||
## Recent decisions
|
## Recent decisions
|
||||||
|
|
||||||
|
- `[2026-09-11]` ⭐⭐⭐ **SKALDSONG'S SHAPE SETTLES THE ARCHITECTURE: the adapted completion carrier CANNOT do beat→paragraph, and an instruct model can. Option C (instruct carrier + corpus rebuilt as instruction→response pairs) is now evidence-backed, not opinion.** Operator's requirement: *"skaldsong will want to write story beats which are a sentence, and have the LLM expound on that sentence to a paragraph and stitch it together."* Booth: `http://10.100.10.50:8090/b/skaldsong-beats/`. **Adapted 4B (checkpoint-75): TEN prompt formats × 3 seeds = 30 samples, ZERO that reliably render the beat** — bare, para-break, labelled, epigraph, fewshot(1), fewshot-bare, fewshot3, elaborate, recount, label-begin. Every one drifts, frames, or truncates. Root cause is structural: *"write a paragraph **about** this sentence"* is an instruction, and a completion model has no mechanism for *about* — it continues the text it is given. ⚠⚠ **Two formats leaked PRETRAINING TASK DATA**: `para-break` emitted an NLI multiple-choice item (*"Does it follow that... OPTIONS: (1). yes (2). it is not possible to tell"*) and `label-begin` a grammar-correction exercise (*"CORRECTION: ... The passage appears to be a sentence fragment"*). A standalone sentence plus a blank line looks exactly like a dataset entry; **style adaptation does not remove base-model task artifacts.** **Instruct arm (`gen` seat + style prompt, no adapter): 10/10 samples inside the requested 90–140 band (124–148w, median 130), every one on-beat, zero drift** — but the voice is generic literary pastiche, abstract-noun-heavy and over-written, not Brontë. **So: voice without direction vs direction without voice; the product needs both.** ⚠ **This applies to Yarros identically** — the carrier question is orthogonal to the author, so the next corpus must NOT re-run this experiment.
|
||||||
|
- `[2026-09-11]` ⚠ **Stitching has its own failure mode, visible in the booth's Panel C: independently-generated paragraphs drift in POINT OF VIEW.** By beat 4 of 5 the narrator is simultaneously watching the girl carry the animals and carrying them herself ("their weight a strange, heavy secret carried between my ribs"). Each paragraph was generated with no knowledge of the others. **A real stitcher must feed prior paragraphs back as context**, which also means the instruction-pair corpus should include multi-paragraph continuity examples, not just isolated beat→paragraph pairs.
|
||||||
|
|
||||||
- `[2026-09-11]` ⭐⭐ **THE RECIPE THAT WORKS ON A COMPLETION CARRIER: label the artifact AND begin it.** Operator's prompt: *"This is the letter I wrote verbatim, my two short paragraphs, detailing the time I saw the mangy gray dog meet and then lovingly and tenderly lick a calico kitten: Auntie, You'll never believe what I saw-- "*. **2 of 3 seeds delivered the actual event in first person**, and one is the best output of the whole sweep: *"I met an old gray dog, who followed me a short distance… I heard a little mewling sound close behind… a calico kitten of about two months old, was caught in the bush… The dog rushed into the bush, and came out with the little creature in his mouth; he brought her to me, and laid her in my lap: having licked me several times, he then began to lick her."* Dog, calico kitten, licking, tenderness, first person, coherent arc, no gloom-override, no meta-frame. **Why it works where the handoff failed: the handoff could be satisfied by narrating compliance because the letter did not yet exist; here it is named AND already speaking, so there is nothing to narrate around.** Also learned the Gutenberg `_underscore italics_` convention. 1 of 3 drifts.
|
- `[2026-09-11]` ⭐⭐ **THE RECIPE THAT WORKS ON A COMPLETION CARRIER: label the artifact AND begin it.** Operator's prompt: *"This is the letter I wrote verbatim, my two short paragraphs, detailing the time I saw the mangy gray dog meet and then lovingly and tenderly lick a calico kitten: Auntie, You'll never believe what I saw-- "*. **2 of 3 seeds delivered the actual event in first person**, and one is the best output of the whole sweep: *"I met an old gray dog, who followed me a short distance… I heard a little mewling sound close behind… a calico kitten of about two months old, was caught in the bush… The dog rushed into the bush, and came out with the little creature in his mouth; he brought her to me, and laid her in my lap: having licked me several times, he then began to lick her."* Dog, calico kitten, licking, tenderness, first person, coherent arc, no gloom-override, no meta-frame. **Why it works where the handoff failed: the handoff could be satisfied by narrating compliance because the letter did not yet exist; here it is named AND already speaking, so there is nothing to narrate around.** Also learned the Gutenberg `_underscore italics_` convention. 1 of 3 drifts.
|
||||||
- `[2026-09-11]` ⚠ **My typography hypothesis was WRONG, and the chapter-heading result is the evidence.** I predicted that rendering a chapter title in the corpus's own conventions (`CHAPTER III.` / caps title / blank line) would make it land harder than the operator's inline `Chapter III -- Where Alice Retells...`. **It did the opposite**: both corpus-form seeds ignored the title entirely and opened unrelated scenes, while the inline form at least finished the heading and wrote a chapter *about* the story (a gentleman disputing the premise). Likely reason: corpus chapter titles are short and decorative (`THE CHILD'S CLOSET`), so a long descriptive one in that slot reads as decoration to skip, whereas inline it reads as text to continue. **A label only instructs if the model treats that slot as load-bearing.**
|
- `[2026-09-11]` ⚠ **My typography hypothesis was WRONG, and the chapter-heading result is the evidence.** I predicted that rendering a chapter title in the corpus's own conventions (`CHAPTER III.` / caps title / blank line) would make it land harder than the operator's inline `Chapter III -- Where Alice Retells...`. **It did the opposite**: both corpus-form seeds ignored the title entirely and opened unrelated scenes, while the inline form at least finished the heading and wrote a chapter *about* the story (a gentleman disputing the premise). Likely reason: corpus chapter titles are short and decorative (`THE CHILD'S CLOSET`), so a long descriptive one in that slot reads as decoration to skip, whereas inline it reads as text to continue. **A label only instructs if the model treats that slot as load-bearing.**
|
||||||
- `[2026-09-11]` ⚠ **Unnoticed consequence of the D2/D3 rename pipeline: the adapter SUBSTITUTES proper nouns it was never trained on.** Given "Alice" in a chapter title it produced *"ALEXANDER THE ALEXANDER, AS HE WAS KNOWN IN LITTLE LONDON"*. The corpus was entity-renamed from a French/English pool, so the adapter learned that character names come from that pool and rewrites outside names into it. Consequence for use: **you cannot reliably name your own characters at prompt time** — they may be renamed mid-passage. Not a defect of the rename (which exists to prevent memorisation of Brontë's cast) but a real usability constraint that needs stating.
|
- `[2026-09-11]` ⚠ **Unnoticed consequence of the D2/D3 rename pipeline: the adapter SUBSTITUTES proper nouns it was never trained on.** Given "Alice" in a chapter title it produced *"ALEXANDER THE ALEXANDER, AS HE WAS KNOWN IN LITTLE LONDON"*. The corpus was entity-renamed from a French/English pool, so the adapter learned that character names come from that pool and rewrites outside names into it. Consequence for use: **you cannot reliably name your own characters at prompt time** — they may be renamed mid-passage. Not a defect of the rename (which exists to prevent memorisation of Brontë's cast) but a real usability constraint that needs stating.
|
||||||
|
|||||||
@@ -0,0 +1,7 @@
|
|||||||
|
[
|
||||||
|
{"id":"b1","beat":"The stray dog came down the lane in the rain, his ribs showing through his coat."},
|
||||||
|
{"id":"b2","beat":"He found the calico kitten under the mill gate, too weak to cry."},
|
||||||
|
{"id":"b3","beat":"He licked her clean, and would not be driven off."},
|
||||||
|
{"id":"b4","beat":"The girl carried them both home in her apron."},
|
||||||
|
{"id":"b5","beat":"By morning the kitten slept against the dog's flank as if she had never been alone."}
|
||||||
|
]
|
||||||
@@ -0,0 +1,164 @@
|
|||||||
|
"""The Skaldsong question, answered: can a beat sentence be expanded to a paragraph?
|
||||||
|
|
||||||
|
The page is built as an argument rather than a gallery, because the result is a
|
||||||
|
negative one on the adapted carrier and a negative result presented as a gallery
|
||||||
|
reads as "some of these look fine".
|
||||||
|
|
||||||
|
Panel A the adapted 4B across ten prompt formats -- what does not work, and why
|
||||||
|
Panel B the same beats through an instruct model with a style prompt -- what does
|
||||||
|
Panel C Panel B's paragraphs stitched, which is the deliverable Skaldsong wants
|
||||||
|
|
||||||
|
Two artifacts in Panel A are worth their own callout: two formats leaked *pretraining
|
||||||
|
task data* -- NLI multiple choice and a grammar-correction exercise -- which is a
|
||||||
|
base-model failure mode that no amount of style adaptation removes.
|
||||||
|
"""
|
||||||
|
import html, json, statistics, sys
|
||||||
|
from collections import defaultdict
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
D = Path(sys.argv[1])
|
||||||
|
|
||||||
|
|
||||||
|
def load(f):
|
||||||
|
rows = [json.loads(l) for l in (D / f).read_text(encoding="utf-8").splitlines() if l.strip()]
|
||||||
|
d = defaultdict(list)
|
||||||
|
for r in rows:
|
||||||
|
d[r["format"]].append(r)
|
||||||
|
return d, rows
|
||||||
|
|
||||||
|
|
||||||
|
bake, rows1 = load("bakeoff.jsonl")
|
||||||
|
bake2, rows2 = load("bakeoff2.jsonl")
|
||||||
|
inst, rows3 = load("instruct.jsonl")
|
||||||
|
bake.update(bake2)
|
||||||
|
allrows = rows1 + rows2
|
||||||
|
|
||||||
|
FMT_NOTES = {
|
||||||
|
"bare": "The beat alone. Continues the situation rather than expanding it, and leaves the kitten out.",
|
||||||
|
"para-break": "⚠ Leaked pretraining task data — NLI multiple choice. A standalone sentence followed by a blank line looks exactly like a dataset entry.",
|
||||||
|
"labelled": "Named the artifact. Produced abstract moralising about punishment and husbands.",
|
||||||
|
"epigraph": "The beat in italics as an epigraph. Drifts immediately.",
|
||||||
|
"fewshot": "One worked example. Echoed the beat with pronouns flipped, then drifted to unrelated gossip.",
|
||||||
|
"fewshot-bare": "One example, no labels. Returned single lines of dialogue, one borrowing a character from the example itself.",
|
||||||
|
"fewshot3": "Three worked examples. Still drifts — into a woman and her husband, a child, a nurse.",
|
||||||
|
"elaborate": "Beat plus “It happened in this way.” Commits to elaborating and elaborates something else.",
|
||||||
|
"recount": "Beat plus “I remember the whole of it.” Same.",
|
||||||
|
"label-begin": "The letter prompt's winning move applied to a beat — label it and seed the opening words. Closest of the ten, and one seed leaked a grammar-correction exercise instead.",
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def para_block(r):
|
||||||
|
tag = f"{r['words']}w" + (" · ran on" if r.get("ran_on") else "")
|
||||||
|
return (f'<div class="s"><span class="seed">seed {r["seed"]} · {tag}</span>'
|
||||||
|
f'<p>{html.escape(r["paragraph"].strip()) or "<em>(empty)</em>"}</p></div>')
|
||||||
|
|
||||||
|
|
||||||
|
beat_one = allrows[0]["beat"] if allrows else ""
|
||||||
|
panelA = "".join(
|
||||||
|
f'<section class="row"><div class="fmt"><code>{html.escape(f)}</code>'
|
||||||
|
f'<span class="note">{FMT_NOTES.get(f, "")}</span></div>'
|
||||||
|
f'<div class="arm">{"".join(para_block(r) for r in bake[f])}</div></section>'
|
||||||
|
for f in FMT_NOTES if f in bake)
|
||||||
|
|
||||||
|
by_beat = defaultdict(list)
|
||||||
|
for r in rows3:
|
||||||
|
by_beat[r["id"]].append(r)
|
||||||
|
panelB = "".join(
|
||||||
|
f'<section class="row"><div class="prompt"><span class="pid">{html.escape(bid)}</span>'
|
||||||
|
f'{html.escape(rs[0]["beat"])}</div><div class="arm tuned">'
|
||||||
|
f'{"".join(para_block(r) for r in rs)}</div></section>'
|
||||||
|
for bid, rs in sorted(by_beat.items()))
|
||||||
|
|
||||||
|
stitched = "\n\n".join(r["paragraph"].strip() for bid, rs in sorted(by_beat.items())
|
||||||
|
for r in rs if r["seed"] == 1234)
|
||||||
|
wl = [r["words"] for r in rows3]
|
||||||
|
|
||||||
|
page = f"""<!doctype html><meta charset="utf-8"><title>Beat to paragraph</title>
|
||||||
|
<style>
|
||||||
|
:root{{--bg:#faf8f5;--fg:#1c1a17;--mut:#6b6560;--line:#e0dad2;--acc:#7a3b2e;--tint:#fdfbf7;--bad:#8a3a3a}}
|
||||||
|
*{{box-sizing:border-box}}
|
||||||
|
body{{margin:0;background:var(--bg);color:var(--fg);font:16px/1.65 Georgia,"Iowan Old Style",serif;padding:2.5rem 1.5rem 5rem}}
|
||||||
|
.wrap{{max-width:1080px;margin:0 auto}}
|
||||||
|
h1{{font-size:1.9rem;margin:0 0 .3rem}}
|
||||||
|
.lede{{color:var(--mut);max-width:74ch;margin:0 0 1rem}}
|
||||||
|
.box{{border-left:3px solid var(--acc);background:#fff;padding:.85rem 1rem;margin:1.1rem 0;max-width:84ch;font-size:.94rem}}
|
||||||
|
.box.verdict{{border-left-color:var(--bad)}}
|
||||||
|
h2{{font-size:1.2rem;margin:3rem 0 .2rem;padding-top:1rem;border-top:2px solid var(--line)}}
|
||||||
|
.tsub{{color:var(--mut);font-size:.92rem;margin:0 0 1.3rem;font-style:italic;max-width:84ch}}
|
||||||
|
.row{{margin:0 0 1.5rem}}
|
||||||
|
.fmt{{background:#fff;border:1px solid var(--line);border-left:3px solid var(--acc);padding:.55rem .8rem;margin-bottom:.5rem}}
|
||||||
|
.fmt code{{font:600 .82rem/1 ui-monospace,monospace;color:var(--acc)}}
|
||||||
|
.fmt .note{{display:block;color:var(--mut);font-size:.88rem;margin-top:.2rem}}
|
||||||
|
.prompt{{background:#fff;border:1px solid var(--line);border-left:3px solid var(--acc);padding:.6rem .85rem;margin-bottom:.5rem}}
|
||||||
|
.pid{{display:inline-block;font:600 .72rem/1 ui-monospace,monospace;color:var(--mut);background:var(--bg);border:1px solid var(--line);padding:.22rem .4rem;margin-right:.55rem;vertical-align:1px}}
|
||||||
|
.arm{{background:#f5f6f8;border:1px solid var(--line);padding:.8rem .95rem}}
|
||||||
|
.arm.tuned{{background:var(--tint);border-color:#d8ccbe}}
|
||||||
|
.s{{border-top:1px dotted var(--line);padding-top:.55rem;margin-top:.55rem}}
|
||||||
|
.arm .s:first-of-type{{border-top:0;padding-top:0;margin-top:0}}
|
||||||
|
.seed{{display:block;font:600 .7rem/1 ui-monospace,monospace;color:var(--mut);margin-bottom:.22rem}}
|
||||||
|
.s p{{margin:0;white-space:pre-wrap;font-size:.94rem}}
|
||||||
|
.story{{background:#fff;border:1px solid var(--line);padding:1.4rem 1.6rem;font-size:1.02rem;white-space:pre-wrap}}
|
||||||
|
footer{{margin-top:3rem;padding-top:1rem;border-top:1px solid var(--line);color:var(--mut);font-size:.85rem;max-width:84ch}}
|
||||||
|
</style>
|
||||||
|
<div class="wrap">
|
||||||
|
<h1>Beat → paragraph: can the adapter do Skaldsong's job?</h1>
|
||||||
|
<p class="lede">Skaldsong wants to write story beats as single sentences, have a model expand each
|
||||||
|
into a paragraph, and stitch the paragraphs into a passable story. That is a narrower job than
|
||||||
|
free-form continuation, and it fails differently.</p>
|
||||||
|
|
||||||
|
<div class="box"><strong>Four ways this job breaks, all of which had to be measured rather than
|
||||||
|
eyeballed.</strong> <em>Drift</em> off the beat breaks the stitch, because the next paragraph no
|
||||||
|
longer follows. <em>Run-on</em> breaks it too — the deliverable is a paragraph, and the following
|
||||||
|
scene belongs to the next beat. <em>Framing</em> renders nothing at all ("I told it briefly").
|
||||||
|
<em>Renaming</em> is a live blocker: the entity-rename pool taught the adapter that character names
|
||||||
|
come from it, so a caller's own name can be rewritten mid-passage.</div>
|
||||||
|
|
||||||
|
<h2>Panel A — the adapted 4B, ten prompt formats</h2>
|
||||||
|
<p class="tsub">One beat, three seeds each, thirty samples. The beat is
|
||||||
|
<strong>"{html.escape(beat_one)}"</strong>. Read as many as you like; the finding is that none of
|
||||||
|
them render it.</p>
|
||||||
|
{panelA}
|
||||||
|
|
||||||
|
<div class="box verdict"><strong>Ten formats, thirty samples, none that reliably expand the beat.</strong>
|
||||||
|
The adapter writes Brontë well — that is settled elsewhere — but "write a paragraph <em>about</em>
|
||||||
|
this sentence" is an instruction, and a completion model has no mechanism for <em>about</em>. It
|
||||||
|
continues the text it is given. Two formats did something worse than drift and leaked
|
||||||
|
<strong>pretraining task data</strong>: an NLI multiple-choice item and a grammar-correction
|
||||||
|
exercise. That is a base-model artifact which no amount of style adaptation removes.</div>
|
||||||
|
|
||||||
|
<h2>Panel B — the same beats through an instruct model</h2>
|
||||||
|
<p class="tsub">The gen seat (Qwen3.8-27B, post-trained, no Brontë adapter) with a style instruction
|
||||||
|
asking for one paragraph of 90–140 words in her manner. Five beats, two seeds.</p>
|
||||||
|
{panelB}
|
||||||
|
|
||||||
|
<div class="box"><strong>It takes direction perfectly and has the wrong voice.</strong> All
|
||||||
|
{len(wl)} samples landed inside the requested band — {min(wl)}–{max(wl)} words, median
|
||||||
|
{statistics.median(wl):.0f} — every one stayed on its beat, and none drifted into a following scene.
|
||||||
|
But the prose is generic literary pastiche rather than Brontë: abstract-noun-heavy, fond of
|
||||||
|
aphoristic openers ("There is a peculiar, chilling stillness that attends the discovery of a life
|
||||||
|
nearly spent"), and it over-writes. Brontë is more concrete and more sharply observed than this.</div>
|
||||||
|
|
||||||
|
<h2>Panel C — Panel B's paragraphs, stitched</h2>
|
||||||
|
<p class="tsub">The deliverable shape, so the failure modes of stitching are visible too. Each
|
||||||
|
paragraph was generated independently, which is itself the next problem: watch the point of view
|
||||||
|
slide between beats — by the fourth the narrator is both watching the girl and carrying the animals.
|
||||||
|
A real stitcher has to feed prior paragraphs back as context.</p>
|
||||||
|
<div class="story">{html.escape(stitched)}</div>
|
||||||
|
|
||||||
|
<div class="box verdict"><strong>The conclusion, and it settles an architecture question.</strong>
|
||||||
|
The adapted completion carrier has the voice and cannot take direction. The instruct model takes
|
||||||
|
direction and has no voice. Skaldsong's job needs both, which means the corpus has to be rebuilt as
|
||||||
|
instruction→response pairs and trained onto an instruct carrier — not more prompt cleverness, which
|
||||||
|
is now ten formats deep with nothing to show. <strong>This applies to Yarros identically</strong>:
|
||||||
|
the carrier question is orthogonal to the author, so the next corpus does not need to re-run this
|
||||||
|
experiment.</div>
|
||||||
|
|
||||||
|
<footer>Adapted arm: Qwen3-4B-Base + H02 LoRA checkpoint-75 (held-out 2.8138, the loss minimum) on
|
||||||
|
pfi-gx10, bf16, sdpa, temperature 0.9 / top_p 0.95, 300 new tokens, truncated at the first paragraph
|
||||||
|
break for display with run-on reported. Instruct arm: `gen` on ana-ml2 via the gateway, temperature
|
||||||
|
0.9 / top_p 0.95, thinking off. Neither panel is scored; the frozen adjudication rule and the
|
||||||
|
Burrows's-Delta instrument are untouched and nothing here feeds them.</footer>
|
||||||
|
</div>"""
|
||||||
|
(D / "index.html").write_text(page, encoding="utf-8")
|
||||||
|
print(f"wrote {D/'index.html'} (panel A {sum(len(v) for v in bake.values())} samples, "
|
||||||
|
f"panel B {len(rows3)}, stitched {len(stitched.split())} words)")
|
||||||
@@ -0,0 +1,147 @@
|
|||||||
|
"""Skaldsong's actual shape: one beat sentence in, one paragraph out, stitchable.
|
||||||
|
|
||||||
|
That is a narrower job than anything tested so far, and it fails in ways free-form
|
||||||
|
continuation does not:
|
||||||
|
|
||||||
|
* DRIFT off the beat breaks the stitch -- the next paragraph no longer follows.
|
||||||
|
* RUN-ON breaks it too. The deliverable is a paragraph, not 400 tokens that wander
|
||||||
|
into the following scene, because the next beat owns that scene.
|
||||||
|
* FRAMING ("I told it briefly") renders nothing at all -- measured across 6 seeds
|
||||||
|
on the handoff prompt.
|
||||||
|
* RENAMING is now a product blocker rather than a curiosity: the D2/D3 rename pool
|
||||||
|
taught the adapter that character names come from it, so a caller's own name can
|
||||||
|
be rewritten mid-passage and the stitched story loses its protagonist.
|
||||||
|
|
||||||
|
So each format is scored on all four, not eyeballed. Run-on is measured by whether a
|
||||||
|
paragraph break arrived before the token budget ran out -- the text is truncated at
|
||||||
|
the first blank line for display, and whether truncation was NEEDED is the signal.
|
||||||
|
|
||||||
|
FORMATS, in ascending order of how much structure they impose. The few-shot one is
|
||||||
|
the interesting entry: a completion model's native instruction channel is a worked
|
||||||
|
example, and none of the earlier prompts gave it one.
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
import argparse, json, re, time
|
||||||
|
from pathlib import Path
|
||||||
|
import torch
|
||||||
|
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||||||
|
|
||||||
|
# A worked example for the few-shot formats. Written by hand in the target register,
|
||||||
|
# deliberately on a subject unrelated to dogs and kittens so it cannot leak content
|
||||||
|
# into the answer -- only shape.
|
||||||
|
EX_BEAT = "The carrier's cart broke its axle at the ford."
|
||||||
|
EX_PARA = ("The cart came to a standstill in the middle of the water, canted over like a "
|
||||||
|
"ship gone aground, and the carrier stood in the shallows with his hand on the "
|
||||||
|
"shaft, saying nothing at all. I watched from the bank. The river ran brown and "
|
||||||
|
"quick about his boots; a hamper had gone over and was turning slowly downstream, "
|
||||||
|
"and he let it go. It was not the loss that held him, I think, but the hour: he "
|
||||||
|
"had been due at the mill before noon, and it was past two.")
|
||||||
|
|
||||||
|
FORMATS = {
|
||||||
|
"bare": lambda b: b + " ",
|
||||||
|
"para-break": lambda b: b + "\n\n",
|
||||||
|
"labelled": lambda b: f"The passage I wrote from this beat:\n\nBEAT: {b}\n\nPASSAGE: ",
|
||||||
|
"epigraph": lambda b: f"_{b}_\n\n",
|
||||||
|
"fewshot": lambda b: (f"BEAT: {EX_BEAT}\nPASSAGE: {EX_PARA}\n\nBEAT: {b}\nPASSAGE: "),
|
||||||
|
"fewshot-bare": lambda b: (f"{EX_BEAT}\n\n{EX_PARA}\n\n{b}\n\n"),
|
||||||
|
# ---- round two. Round one failed everywhere, so before calling that a property
|
||||||
|
# of the adapter these four give the strongest untested patterns a fair run.
|
||||||
|
# THREE examples, not one: one-shot is thin, and a format dismissed on one
|
||||||
|
# example has not been tested, it has been under-fed.
|
||||||
|
"fewshot3": lambda b: ("".join(f"BEAT: {eb}\nPASSAGE: {ep}\n\n"
|
||||||
|
for eb, ep in EXTRA_EXAMPLES)
|
||||||
|
+ f"BEAT: {b}\nPASSAGE: "),
|
||||||
|
# The letter prompt's winning move was "label the artifact AND begin it". These
|
||||||
|
# apply it to a beat: state the beat, then open the paragraph with a phrase that
|
||||||
|
# COMMITS to elaborating what was just said, so moving on is off the table.
|
||||||
|
"elaborate": lambda b: f"{b} It happened in this way. ",
|
||||||
|
"recount": lambda b: f"{b} I remember the whole of it, and will set it down. ",
|
||||||
|
# Label + begin, with the paragraph seeded by the beat's own opening words so the
|
||||||
|
# first thing it writes is already inside the beat rather than after it.
|
||||||
|
"label-begin": lambda b: (f"BEAT: {b}\nPASSAGE: " + " ".join(b.split()[:3]) + " "),
|
||||||
|
}
|
||||||
|
EXTRA_EXAMPLES = [
|
||||||
|
(EX_BEAT, EX_PARA),
|
||||||
|
("The housekeeper refused to give up the key.",
|
||||||
|
"She stood with her hand closed over it and her chin down, and said that the room "
|
||||||
|
"had been shut since March and would stay shut. I asked her whose order it was. She "
|
||||||
|
"said it was nobody's order, it was sense; and then, seeing I meant to press her, she "
|
||||||
|
"put the key into her apron pocket and held the pocket. There was no arguing with the "
|
||||||
|
"gesture. I went back along the passage and heard her breathing behind me the whole way."),
|
||||||
|
("A letter came for the master and was burned unopened.",
|
||||||
|
"It lay on the salver a quarter of an hour, and I saw the hand on it -- a small, "
|
||||||
|
"sloped, foreign hand -- before he came in. He turned it over once, read the "
|
||||||
|
"postmark, and put it on the fire without breaking the seal. The wax ran first and "
|
||||||
|
"then the paper caught. He watched it to the end, which is what I remember: not the "
|
||||||
|
"burning, but that he stayed to see it finished."),
|
||||||
|
]
|
||||||
|
|
||||||
|
ap = argparse.ArgumentParser()
|
||||||
|
ap.add_argument("--base", required=True)
|
||||||
|
ap.add_argument("--adapter", default=None)
|
||||||
|
ap.add_argument("--beats", required=True, help="json list of {id, beat}")
|
||||||
|
ap.add_argument("--formats", nargs="+", default=list(FORMATS))
|
||||||
|
ap.add_argument("--out", required=True)
|
||||||
|
ap.add_argument("--seeds", type=int, nargs="+", default=[1234, 5678])
|
||||||
|
ap.add_argument("--max-new-tokens", type=int, default=300)
|
||||||
|
ap.add_argument("--temperature", type=float, default=0.9)
|
||||||
|
ap.add_argument("--top-p", type=float, default=0.95)
|
||||||
|
a = ap.parse_args()
|
||||||
|
|
||||||
|
beats = json.loads(Path(a.beats).read_text())
|
||||||
|
tok = AutoTokenizer.from_pretrained(a.base)
|
||||||
|
model = AutoModelForCausalLM.from_pretrained(a.base, dtype=torch.bfloat16,
|
||||||
|
attn_implementation="sdpa").to("cuda")
|
||||||
|
if a.adapter:
|
||||||
|
from peft import PeftModel
|
||||||
|
model = PeftModel.from_pretrained(model, a.adapter)
|
||||||
|
deltas = [float(m.lora_B["default"].weight.abs().sum())
|
||||||
|
for m in model.modules() if hasattr(m, "lora_B")]
|
||||||
|
nz = sum(1 for d in deltas if d > 0)
|
||||||
|
print(f"[gen] adapter bound: {nz}/{len(deltas)} lora_B tensors non-zero", flush=True)
|
||||||
|
if nz == 0:
|
||||||
|
raise SystemExit("REFUSING: adapter applied but every lora_B is zero")
|
||||||
|
model.eval()
|
||||||
|
|
||||||
|
STOP = re.compile(r"\n\s*\n")
|
||||||
|
|
||||||
|
|
||||||
|
def keywords(beat):
|
||||||
|
"""Content words worth checking for, to score staying ON the beat."""
|
||||||
|
drop = {"the", "a", "an", "and", "or", "but", "in", "on", "at", "to", "of", "his",
|
||||||
|
"her", "he", "she", "it", "was", "were", "had", "would", "not", "be",
|
||||||
|
"by", "as", "with", "for", "from", "that", "this", "up", "down", "she"}
|
||||||
|
return [w for w in re.findall(r"[a-z']+", beat.lower()) if w not in drop and len(w) > 3]
|
||||||
|
|
||||||
|
|
||||||
|
out = Path(a.out); out.parent.mkdir(parents=True, exist_ok=True)
|
||||||
|
t0 = time.time()
|
||||||
|
with out.open("w", encoding="utf-8") as fh:
|
||||||
|
for fmt in a.formats:
|
||||||
|
build = FORMATS[fmt]
|
||||||
|
for b in beats:
|
||||||
|
for seed in a.seeds:
|
||||||
|
torch.manual_seed(seed)
|
||||||
|
prompt = build(b["beat"])
|
||||||
|
ids = tok(prompt, return_tensors="pt").to("cuda")
|
||||||
|
with torch.no_grad():
|
||||||
|
g = model.generate(**ids, do_sample=True, temperature=a.temperature,
|
||||||
|
top_p=a.top_p, max_new_tokens=a.max_new_tokens,
|
||||||
|
pad_token_id=tok.eos_token_id)
|
||||||
|
raw = tok.decode(g[0][ids["input_ids"].shape[1]:], skip_special_tokens=True)
|
||||||
|
m = STOP.search(raw.strip())
|
||||||
|
para = (raw.strip()[:m.start()] if m else raw.strip()).strip()
|
||||||
|
kws = keywords(b["beat"])
|
||||||
|
hit = sum(1 for k in kws if k[:5] in para.lower())
|
||||||
|
fh.write(json.dumps({
|
||||||
|
"format": fmt, "id": b["id"], "beat": b["beat"], "seed": seed,
|
||||||
|
"prompt": prompt, "paragraph": para, "raw_tail": raw.strip()[m.end():][:200] if m else "",
|
||||||
|
# ran_on: the model never closed a paragraph inside the budget, so
|
||||||
|
# a stitcher would have to cut it mid-thought.
|
||||||
|
"ran_on": m is None,
|
||||||
|
"words": len(para.split()),
|
||||||
|
"beat_keywords": kws, "keyword_hits": hit,
|
||||||
|
}) + "\n")
|
||||||
|
print(f" {fmt:14} {b['id']:>8} seed={seed} {len(para.split()):>4}w "
|
||||||
|
f"kw {hit}/{len(kws)} {'RAN-ON' if m is None else ''}", flush=True)
|
||||||
|
print(f"[gen] -> {out} in {time.time()-t0:.0f}s", flush=True)
|
||||||
Reference in New Issue
Block a user