a5ddfed81e89a44bed7b77e455dea7b0713f4d17
3
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 vh
|
8787daf04f |
Find the prompt recipe that works on a completion carrier
Two operator prompts settled the question the handoff prompt opened. The winner labels the artifact and then begins it: "This is the letter I wrote verbatim, my two short paragraphs, detailing the time I saw the mangy gray dog meet and then lovingly and tenderly lick a calico kitten: Auntie, You'll never believe what I saw--". Two of three seeds delivered the actual event in first person, and one is the strongest output of the sweep -- the dog following her to the mill, the kitten caught in a hedge, the dog carrying it out and laying it in her lap before licking it. Coherent arc, correct subject, no gloom-override, no meta-frame. It also reproduced the Gutenberg underscore-italics convention. It works where the handoff failed for a specific reason. The handoff could be satisfied by narrating compliance, because the letter did not yet exist. Naming the artifact and starting it leaves nothing to narrate around, so the only continuation is the artifact. The chapter-heading prompt refuted a hypothesis of mine. I predicted that rendering the title in the corpus's own conventions would make it land harder than the operator's inline form. It did the opposite: both corpus-form seeds ignored the title and opened unrelated scenes, while the inline form at least finished the heading and wrote a chapter about the story. Corpus chapter titles are short and decorative, so a long descriptive one in that slot reads as decoration to skip. A label only instructs if the model treats that slot as load-bearing. That prompt also surfaced an unnoticed consequence of the D2/D3 rename pipeline: given "Alice", the adapter produced "Alexander the Alexander, as he was known in Little London". The corpus was entity-renamed from a French/English pool, so the adapter learned that character names come from that pool and rewrites outside names into it. Callers cannot reliably name their own characters at prompt time. That is not a defect of the rename, which exists to prevent memorising Brontë's cast, but it is a usability constraint worth stating. |
||
|
vh
|
e47295d2ce |
Re-cut the 4B arms from checkpoint-75 and test an in-fiction instruction
The 4B tuned arms now come from checkpoint-75, the actual loss minimum at 2.813826 confirmed from loss-series.json rather than from reading the training log. Only the tuned arms needed re-cutting; the base arm never touches the adapter. One surprise in the comparison: step-75 and end-of-run differ on typography rather than voice. Curly quotes read 16 of 18 against 17, collapse is 0 either way, but the hard-wrap ratio is 0.33 at step-75 against 0.12 at the end. Further training washes the residual line-break habit out while held-out loss gets worse, so best loss and best typography are different checkpoints. Neither is near the original 0.85 defect and the corpus's own residual from preserved verse is 0.25. The operator's second prompt embeds the instruction inside the fiction: Abernathy tells the tale badly and then asks the narrator to retell it in a few short paragraphs. Across six seeds the model acknowledged the handoff every time and never once performed it. It wrote "I told it, briefly, to his satisfaction", and "So I wrote it out, and kept it in my pocket-book", and one seed negotiated the brief in character, objecting that it could not condense the story into a few short paragraphs because there were too many points to touch. The reason is structural rather than a failure of the adapter. In a novel, "she retold the story" is an ordinary sentence, so the likeliest continuation of a request is narration of compliance rather than compliance. A trailing paragraph break did shift the behaviour -- one seed opened in the narrator's own quoted speech -- so typography is a real lever, just not a sufficient one. That is direct evidence on the instruct question raised earlier: if the product is asking for a scene and getting the scene, no amount of in-fiction framing substitutes for a post-trained instruction-follower, which favours rebuilding the corpus as instruction pairs over more prompt cleverness. |
||
|
vh
|
25eb65e23b |
Complete the R49 carrier sweep: 4B closes the continuity gap and overfits
The three rungs now sit on the same unwrapped corpus, seed, step count and token count, so carrier size is the only variable. Held-out loss reads 3.329 at 0.6B, 3.018 at 1.7B and 2.814 at 4B -- deltas of 0.311 and then 0.204, diminishing but still real. 4B answers the question the rung existed for. Scene-level continuity holds: on the office-kitchen prompt it produces a named character with motivated dialogue, a spatial layout the narrator navigates, and a physical description, all in one passage, where 1.7B wrote pretty but eventless prose about opening doors and looking at stars. On the letter prompt it opens the letter, promises to quote it, and then quotes it across a paragraph break. Voice saturation is also the best of any rung: curly quotes 17 of 18 against its own base arm's 1 of 18, and collapse 0 of 18 against 4 of 18. Two findings that change earlier conclusions. 4B is the first rung to overfit inside one epoch. Its series runs 2.832, 2.816, 2.814, 2.820, 2.824, 2.825, 2.825 -- a minimum around step 75 and then a turn. Both smaller rungs plateaued without turning, so the optimal epoch count shrinks as the carrier grows and my earlier "one epoch is right for this corpus" holds only for the small end. The consequence is operational: the adapter directory holds the end-of-run weights at 2.825 rather than the step-75 best at 2.814, and it exists as a recoverable checkpoint only because save_steps was set. The voice arms were cut from the end-of-run adapter, so the booth understates 4B slightly. The tone-override also appears to close. On the operator's frame prompt asking for a wonderful story, 1.7B held the frame on every seed but killed the animals on two of four; 4B kept them alive on both seeds, and one of them had the narrator doubt the story he was told and supply a parallel childhood memory to explain the doubt. That is a narrator with an interior position on the tale. Two samples per arm, so directionally right rather than established. |