diff --git a/persistent-memory.md b/persistent-memory.md index 4f542e1..5cfc53e 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -206,6 +206,9 @@ _As of 2026-09-10 10:25 PT._ ## Recent decisions +- `[2026-09-11]` ⭐⭐⭐ **SKALDSONG'S SHAPE SETTLES THE ARCHITECTURE: the adapted completion carrier CANNOT do beat→paragraph, and an instruct model can. Option C (instruct carrier + corpus rebuilt as instruction→response pairs) is now evidence-backed, not opinion.** Operator's requirement: *"skaldsong will want to write story beats which are a sentence, and have the LLM expound on that sentence to a paragraph and stitch it together."* Booth: `http://10.100.10.50:8090/b/skaldsong-beats/`. **Adapted 4B (checkpoint-75): TEN prompt formats × 3 seeds = 30 samples, ZERO that reliably render the beat** — bare, para-break, labelled, epigraph, fewshot(1), fewshot-bare, fewshot3, elaborate, recount, label-begin. Every one drifts, frames, or truncates. Root cause is structural: *"write a paragraph **about** this sentence"* is an instruction, and a completion model has no mechanism for *about* — it continues the text it is given. ⚠⚠ **Two formats leaked PRETRAINING TASK DATA**: `para-break` emitted an NLI multiple-choice item (*"Does it follow that... OPTIONS: (1). yes (2). it is not possible to tell"*) and `label-begin` a grammar-correction exercise (*"CORRECTION: ... The passage appears to be a sentence fragment"*). A standalone sentence plus a blank line looks exactly like a dataset entry; **style adaptation does not remove base-model task artifacts.** **Instruct arm (`gen` seat + style prompt, no adapter): 10/10 samples inside the requested 90–140 band (124–148w, median 130), every one on-beat, zero drift** — but the voice is generic literary pastiche, abstract-noun-heavy and over-written, not Brontë. **So: voice without direction vs direction without voice; the product needs both.** ⚠ **This applies to Yarros identically** — the carrier question is orthogonal to the author, so the next corpus must NOT re-run this experiment. +- `[2026-09-11]` ⚠ **Stitching has its own failure mode, visible in the booth's Panel C: independently-generated paragraphs drift in POINT OF VIEW.** By beat 4 of 5 the narrator is simultaneously watching the girl carry the animals and carrying them herself ("their weight a strange, heavy secret carried between my ribs"). Each paragraph was generated with no knowledge of the others. **A real stitcher must feed prior paragraphs back as context**, which also means the instruction-pair corpus should include multi-paragraph continuity examples, not just isolated beat→paragraph pairs. + - `[2026-09-11]` ⭐⭐ **THE RECIPE THAT WORKS ON A COMPLETION CARRIER: label the artifact AND begin it.** Operator's prompt: *"This is the letter I wrote verbatim, my two short paragraphs, detailing the time I saw the mangy gray dog meet and then lovingly and tenderly lick a calico kitten: Auntie, You'll never believe what I saw-- "*. **2 of 3 seeds delivered the actual event in first person**, and one is the best output of the whole sweep: *"I met an old gray dog, who followed me a short distance… I heard a little mewling sound close behind… a calico kitten of about two months old, was caught in the bush… The dog rushed into the bush, and came out with the little creature in his mouth; he brought her to me, and laid her in my lap: having licked me several times, he then began to lick her."* Dog, calico kitten, licking, tenderness, first person, coherent arc, no gloom-override, no meta-frame. **Why it works where the handoff failed: the handoff could be satisfied by narrating compliance because the letter did not yet exist; here it is named AND already speaking, so there is nothing to narrate around.** Also learned the Gutenberg `_underscore italics_` convention. 1 of 3 drifts. - `[2026-09-11]` ⚠ **My typography hypothesis was WRONG, and the chapter-heading result is the evidence.** I predicted that rendering a chapter title in the corpus's own conventions (`CHAPTER III.` / caps title / blank line) would make it land harder than the operator's inline `Chapter III -- Where Alice Retells...`. **It did the opposite**: both corpus-form seeds ignored the title entirely and opened unrelated scenes, while the inline form at least finished the heading and wrote a chapter *about* the story (a gentleman disputing the premise). Likely reason: corpus chapter titles are short and decorative (`THE CHILD'S CLOSET`), so a long descriptive one in that slot reads as decoration to skip, whereas inline it reads as text to continue. **A label only instructs if the model treats that slot as load-bearing.** - `[2026-09-11]` ⚠ **Unnoticed consequence of the D2/D3 rename pipeline: the adapter SUBSTITUTES proper nouns it was never trained on.** Given "Alice" in a chapter title it produced *"ALEXANDER THE ALEXANDER, AS HE WAS KNOWN IN LITTLE LONDON"*. The corpus was entity-renamed from a French/English pool, so the adapter learned that character names come from that pool and rewrites outside names into it. Consequence for use: **you cannot reliably name your own characters at prompt time** — they may be renamed mid-passage. Not a defect of the rename (which exists to prevent memorisation of Brontë's cast) but a real usability constraint that needs stating. diff --git a/scripts/r49-corpus/beats.json b/scripts/r49-corpus/beats.json new file mode 100644 index 0000000..bb1c7b0 --- /dev/null +++ b/scripts/r49-corpus/beats.json @@ -0,0 +1,7 @@ +[ + {"id":"b1","beat":"The stray dog came down the lane in the rain, his ribs showing through his coat."}, + {"id":"b2","beat":"He found the calico kitten under the mill gate, too weak to cry."}, + {"id":"b3","beat":"He licked her clean, and would not be driven off."}, + {"id":"b4","beat":"The girl carried them both home in her apron."}, + {"id":"b5","beat":"By morning the kitten slept against the dog's flank as if she had never been alone."} +] diff --git a/scripts/r49-corpus/build_booth_beats.py b/scripts/r49-corpus/build_booth_beats.py new file mode 100644 index 0000000..c08e9c1 --- /dev/null +++ b/scripts/r49-corpus/build_booth_beats.py @@ -0,0 +1,164 @@ +"""The Skaldsong question, answered: can a beat sentence be expanded to a paragraph? + +The page is built as an argument rather than a gallery, because the result is a +negative one on the adapted carrier and a negative result presented as a gallery +reads as "some of these look fine". + + Panel A the adapted 4B across ten prompt formats -- what does not work, and why + Panel B the same beats through an instruct model with a style prompt -- what does + Panel C Panel B's paragraphs stitched, which is the deliverable Skaldsong wants + +Two artifacts in Panel A are worth their own callout: two formats leaked *pretraining +task data* -- NLI multiple choice and a grammar-correction exercise -- which is a +base-model failure mode that no amount of style adaptation removes. +""" +import html, json, statistics, sys +from collections import defaultdict +from pathlib import Path + +D = Path(sys.argv[1]) + + +def load(f): + rows = [json.loads(l) for l in (D / f).read_text(encoding="utf-8").splitlines() if l.strip()] + d = defaultdict(list) + for r in rows: + d[r["format"]].append(r) + return d, rows + + +bake, rows1 = load("bakeoff.jsonl") +bake2, rows2 = load("bakeoff2.jsonl") +inst, rows3 = load("instruct.jsonl") +bake.update(bake2) +allrows = rows1 + rows2 + +FMT_NOTES = { + "bare": "The beat alone. Continues the situation rather than expanding it, and leaves the kitten out.", + "para-break": "⚠ Leaked pretraining task data — NLI multiple choice. A standalone sentence followed by a blank line looks exactly like a dataset entry.", + "labelled": "Named the artifact. Produced abstract moralising about punishment and husbands.", + "epigraph": "The beat in italics as an epigraph. Drifts immediately.", + "fewshot": "One worked example. Echoed the beat with pronouns flipped, then drifted to unrelated gossip.", + "fewshot-bare": "One example, no labels. Returned single lines of dialogue, one borrowing a character from the example itself.", + "fewshot3": "Three worked examples. Still drifts — into a woman and her husband, a child, a nurse.", + "elaborate": "Beat plus “It happened in this way.” Commits to elaborating and elaborates something else.", + "recount": "Beat plus “I remember the whole of it.” Same.", + "label-begin": "The letter prompt's winning move applied to a beat — label it and seed the opening words. Closest of the ten, and one seed leaked a grammar-correction exercise instead.", +} + + +def para_block(r): + tag = f"{r['words']}w" + (" · ran on" if r.get("ran_on") else "") + return (f'
seed {r["seed"]} · {tag}' + f'

{html.escape(r["paragraph"].strip()) or "(empty)"}

') + + +beat_one = allrows[0]["beat"] if allrows else "" +panelA = "".join( + f'
{html.escape(f)}' + f'{FMT_NOTES.get(f, "")}
' + f'
{"".join(para_block(r) for r in bake[f])}
' + for f in FMT_NOTES if f in bake) + +by_beat = defaultdict(list) +for r in rows3: + by_beat[r["id"]].append(r) +panelB = "".join( + f'
{html.escape(bid)}' + f'{html.escape(rs[0]["beat"])}
' + f'{"".join(para_block(r) for r in rs)}
' + for bid, rs in sorted(by_beat.items())) + +stitched = "\n\n".join(r["paragraph"].strip() for bid, rs in sorted(by_beat.items()) + for r in rs if r["seed"] == 1234) +wl = [r["words"] for r in rows3] + +page = f"""Beat to paragraph + +
+

Beat → paragraph: can the adapter do Skaldsong's job?

+

Skaldsong wants to write story beats as single sentences, have a model expand each +into a paragraph, and stitch the paragraphs into a passable story. That is a narrower job than +free-form continuation, and it fails differently.

+ +
Four ways this job breaks, all of which had to be measured rather than +eyeballed. Drift off the beat breaks the stitch, because the next paragraph no +longer follows. Run-on breaks it too — the deliverable is a paragraph, and the following +scene belongs to the next beat. Framing renders nothing at all ("I told it briefly"). +Renaming is a live blocker: the entity-rename pool taught the adapter that character names +come from it, so a caller's own name can be rewritten mid-passage.
+ +

Panel A — the adapted 4B, ten prompt formats

+

One beat, three seeds each, thirty samples. The beat is +"{html.escape(beat_one)}". Read as many as you like; the finding is that none of +them render it.

+{panelA} + +
Ten formats, thirty samples, none that reliably expand the beat. +The adapter writes Brontë well — that is settled elsewhere — but "write a paragraph about +this sentence" is an instruction, and a completion model has no mechanism for about. It +continues the text it is given. Two formats did something worse than drift and leaked +pretraining task data: an NLI multiple-choice item and a grammar-correction +exercise. That is a base-model artifact which no amount of style adaptation removes.
+ +

Panel B — the same beats through an instruct model

+

The gen seat (Qwen3.8-27B, post-trained, no Brontë adapter) with a style instruction +asking for one paragraph of 90–140 words in her manner. Five beats, two seeds.

+{panelB} + +
It takes direction perfectly and has the wrong voice. All +{len(wl)} samples landed inside the requested band — {min(wl)}–{max(wl)} words, median +{statistics.median(wl):.0f} — every one stayed on its beat, and none drifted into a following scene. +But the prose is generic literary pastiche rather than Brontë: abstract-noun-heavy, fond of +aphoristic openers ("There is a peculiar, chilling stillness that attends the discovery of a life +nearly spent"), and it over-writes. Brontë is more concrete and more sharply observed than this.
+ +

Panel C — Panel B's paragraphs, stitched

+

The deliverable shape, so the failure modes of stitching are visible too. Each +paragraph was generated independently, which is itself the next problem: watch the point of view +slide between beats — by the fourth the narrator is both watching the girl and carrying the animals. +A real stitcher has to feed prior paragraphs back as context.

+
{html.escape(stitched)}
+ +
The conclusion, and it settles an architecture question. +The adapted completion carrier has the voice and cannot take direction. The instruct model takes +direction and has no voice. Skaldsong's job needs both, which means the corpus has to be rebuilt as +instruction→response pairs and trained onto an instruct carrier — not more prompt cleverness, which +is now ten formats deep with nothing to show. This applies to Yarros identically: +the carrier question is orthogonal to the author, so the next corpus does not need to re-run this +experiment.
+ + +
""" +(D / "index.html").write_text(page, encoding="utf-8") +print(f"wrote {D/'index.html'} (panel A {sum(len(v) for v in bake.values())} samples, " + f"panel B {len(rows3)}, stitched {len(stitched.split())} words)") diff --git a/scripts/r49-corpus/gen_beats.py b/scripts/r49-corpus/gen_beats.py new file mode 100644 index 0000000..73bdc06 --- /dev/null +++ b/scripts/r49-corpus/gen_beats.py @@ -0,0 +1,147 @@ +"""Skaldsong's actual shape: one beat sentence in, one paragraph out, stitchable. + +That is a narrower job than anything tested so far, and it fails in ways free-form +continuation does not: + + * DRIFT off the beat breaks the stitch -- the next paragraph no longer follows. + * RUN-ON breaks it too. The deliverable is a paragraph, not 400 tokens that wander + into the following scene, because the next beat owns that scene. + * FRAMING ("I told it briefly") renders nothing at all -- measured across 6 seeds + on the handoff prompt. + * RENAMING is now a product blocker rather than a curiosity: the D2/D3 rename pool + taught the adapter that character names come from it, so a caller's own name can + be rewritten mid-passage and the stitched story loses its protagonist. + +So each format is scored on all four, not eyeballed. Run-on is measured by whether a +paragraph break arrived before the token budget ran out -- the text is truncated at +the first blank line for display, and whether truncation was NEEDED is the signal. + +FORMATS, in ascending order of how much structure they impose. The few-shot one is +the interesting entry: a completion model's native instruction channel is a worked +example, and none of the earlier prompts gave it one. +""" +from __future__ import annotations +import argparse, json, re, time +from pathlib import Path +import torch +from transformers import AutoModelForCausalLM, AutoTokenizer + +# A worked example for the few-shot formats. Written by hand in the target register, +# deliberately on a subject unrelated to dogs and kittens so it cannot leak content +# into the answer -- only shape. +EX_BEAT = "The carrier's cart broke its axle at the ford." +EX_PARA = ("The cart came to a standstill in the middle of the water, canted over like a " + "ship gone aground, and the carrier stood in the shallows with his hand on the " + "shaft, saying nothing at all. I watched from the bank. The river ran brown and " + "quick about his boots; a hamper had gone over and was turning slowly downstream, " + "and he let it go. It was not the loss that held him, I think, but the hour: he " + "had been due at the mill before noon, and it was past two.") + +FORMATS = { + "bare": lambda b: b + " ", + "para-break": lambda b: b + "\n\n", + "labelled": lambda b: f"The passage I wrote from this beat:\n\nBEAT: {b}\n\nPASSAGE: ", + "epigraph": lambda b: f"_{b}_\n\n", + "fewshot": lambda b: (f"BEAT: {EX_BEAT}\nPASSAGE: {EX_PARA}\n\nBEAT: {b}\nPASSAGE: "), + "fewshot-bare": lambda b: (f"{EX_BEAT}\n\n{EX_PARA}\n\n{b}\n\n"), + # ---- round two. Round one failed everywhere, so before calling that a property + # of the adapter these four give the strongest untested patterns a fair run. + # THREE examples, not one: one-shot is thin, and a format dismissed on one + # example has not been tested, it has been under-fed. + "fewshot3": lambda b: ("".join(f"BEAT: {eb}\nPASSAGE: {ep}\n\n" + for eb, ep in EXTRA_EXAMPLES) + + f"BEAT: {b}\nPASSAGE: "), + # The letter prompt's winning move was "label the artifact AND begin it". These + # apply it to a beat: state the beat, then open the paragraph with a phrase that + # COMMITS to elaborating what was just said, so moving on is off the table. + "elaborate": lambda b: f"{b} It happened in this way. ", + "recount": lambda b: f"{b} I remember the whole of it, and will set it down. ", + # Label + begin, with the paragraph seeded by the beat's own opening words so the + # first thing it writes is already inside the beat rather than after it. + "label-begin": lambda b: (f"BEAT: {b}\nPASSAGE: " + " ".join(b.split()[:3]) + " "), +} +EXTRA_EXAMPLES = [ + (EX_BEAT, EX_PARA), + ("The housekeeper refused to give up the key.", + "She stood with her hand closed over it and her chin down, and said that the room " + "had been shut since March and would stay shut. I asked her whose order it was. She " + "said it was nobody's order, it was sense; and then, seeing I meant to press her, she " + "put the key into her apron pocket and held the pocket. There was no arguing with the " + "gesture. I went back along the passage and heard her breathing behind me the whole way."), + ("A letter came for the master and was burned unopened.", + "It lay on the salver a quarter of an hour, and I saw the hand on it -- a small, " + "sloped, foreign hand -- before he came in. He turned it over once, read the " + "postmark, and put it on the fire without breaking the seal. The wax ran first and " + "then the paper caught. He watched it to the end, which is what I remember: not the " + "burning, but that he stayed to see it finished."), +] + +ap = argparse.ArgumentParser() +ap.add_argument("--base", required=True) +ap.add_argument("--adapter", default=None) +ap.add_argument("--beats", required=True, help="json list of {id, beat}") +ap.add_argument("--formats", nargs="+", default=list(FORMATS)) +ap.add_argument("--out", required=True) +ap.add_argument("--seeds", type=int, nargs="+", default=[1234, 5678]) +ap.add_argument("--max-new-tokens", type=int, default=300) +ap.add_argument("--temperature", type=float, default=0.9) +ap.add_argument("--top-p", type=float, default=0.95) +a = ap.parse_args() + +beats = json.loads(Path(a.beats).read_text()) +tok = AutoTokenizer.from_pretrained(a.base) +model = AutoModelForCausalLM.from_pretrained(a.base, dtype=torch.bfloat16, + attn_implementation="sdpa").to("cuda") +if a.adapter: + from peft import PeftModel + model = PeftModel.from_pretrained(model, a.adapter) + deltas = [float(m.lora_B["default"].weight.abs().sum()) + for m in model.modules() if hasattr(m, "lora_B")] + nz = sum(1 for d in deltas if d > 0) + print(f"[gen] adapter bound: {nz}/{len(deltas)} lora_B tensors non-zero", flush=True) + if nz == 0: + raise SystemExit("REFUSING: adapter applied but every lora_B is zero") +model.eval() + +STOP = re.compile(r"\n\s*\n") + + +def keywords(beat): + """Content words worth checking for, to score staying ON the beat.""" + drop = {"the", "a", "an", "and", "or", "but", "in", "on", "at", "to", "of", "his", + "her", "he", "she", "it", "was", "were", "had", "would", "not", "be", + "by", "as", "with", "for", "from", "that", "this", "up", "down", "she"} + return [w for w in re.findall(r"[a-z']+", beat.lower()) if w not in drop and len(w) > 3] + + +out = Path(a.out); out.parent.mkdir(parents=True, exist_ok=True) +t0 = time.time() +with out.open("w", encoding="utf-8") as fh: + for fmt in a.formats: + build = FORMATS[fmt] + for b in beats: + for seed in a.seeds: + torch.manual_seed(seed) + prompt = build(b["beat"]) + ids = tok(prompt, return_tensors="pt").to("cuda") + with torch.no_grad(): + g = model.generate(**ids, do_sample=True, temperature=a.temperature, + top_p=a.top_p, max_new_tokens=a.max_new_tokens, + pad_token_id=tok.eos_token_id) + raw = tok.decode(g[0][ids["input_ids"].shape[1]:], skip_special_tokens=True) + m = STOP.search(raw.strip()) + para = (raw.strip()[:m.start()] if m else raw.strip()).strip() + kws = keywords(b["beat"]) + hit = sum(1 for k in kws if k[:5] in para.lower()) + fh.write(json.dumps({ + "format": fmt, "id": b["id"], "beat": b["beat"], "seed": seed, + "prompt": prompt, "paragraph": para, "raw_tail": raw.strip()[m.end():][:200] if m else "", + # ran_on: the model never closed a paragraph inside the budget, so + # a stitcher would have to cut it mid-thought. + "ran_on": m is None, + "words": len(para.split()), + "beat_keywords": kws, "keyword_hits": hit, + }) + "\n") + print(f" {fmt:14} {b['id']:>8} seed={seed} {len(para.split()):>4}w " + f"kw {hit}/{len(kws)} {'RAN-ON' if m is None else ''}", flush=True) +print(f"[gen] -> {out} in {time.time()-t0:.0f}s", flush=True)