diff --git a/persistent-memory.d/2026-09-21-lv-mccarthy-gate.md b/persistent-memory.d/2026-09-21-lv-mccarthy-gate.md new file mode 100644 index 0000000..7a59c3e --- /dev/null +++ b/persistent-memory.d/2026-09-21-lv-mccarthy-gate.md @@ -0,0 +1,217 @@ +# `[2026-09-21]` lv-mccarthy: gate run, NOT SHIPPED — the voice works, the memorisation is the cleanest in the line, and the length discipline is gone + +**Status: NOT SHIPPED.** Both gated candidates are disqualified by axis C under the +pre-registration as frozen. The operator had pre-authorised "ship it if the gate passes"; +it did not pass, and their conditional preserved that branch. + +Gate design pre-registered before any generation existed: +`scripts/mccarthy-corpus/GATE-PREREG.md`, commit `9c8a4e9`, amended `b4ba731` and `0d80e49`. +Raw artifacts committed at `scripts/mccarthy-corpus/gate-results/`; arms and generations at +`gx10:~/r49-runs/mccarthy-eval/`. + +## First: the run WAS complete, and had been for three days + +`persistent-memory.md` carried "lv-mccarthy's run outcome STILL UNVERIFIED" as the top +in-flight item for two days. It finished **2026-09-18 00:49 PT** — 1,380/1,380 steps in +2h30m47s, `train_loss` 2.172. Three days of "unverified" was a reporting gap, not a failure. +⚠ `~/r49-runs/` does not exist on nh3-dev; the runs are on **pfi-gx10 = 10.100.50.60**, which +does not resolve by name from nh3-dev. + +## The result — 3 arms × 60 held-out beats × 4 seeds = 720 generations + +| axis | ckpt900 (ep 1.96, loss min) | ckpt450 (ep 0.98) | verdict | +|---|---|---|---| +| **A. VOICE** | +0.172, **1.2×** floor 0.148 | +0.152, **2.9×** floor 0.052 | ✅ both PASS, both reads | +| **B. NOT COPIED** | 0.22 = 1.8× the author | **0.12 = 1.0× the author** | ✅ ckpt450 clean, ckpt900 elevated | +| **C. NO DAMAGE** | fails all three criteria | fails two of three | ❌ **both FAIL** | + +``` +same-author target (held-out McCarthy vs itself) delta_cb 0.370 <- best achievable +ckpt900 0.490 +ckpt450 0.509 +base-unadapted 0.661 +``` + +Span 0.661 → 0.370 = 0.291. ckpt900 closed **59.1%**, ckpt450 **52.2%** — between lv-bronte +(48%) and lv-hemingway (73.8%), on an axis deliberately made harder (§5a hands the register's +punctuation tics to the control). + +## ⭐⭐ AXIS B: the corrected control turned a 12× red flag into a clean pass + +The pre-registration as first written inherited `memorization_check.py`'s **base-unadapted** +negative control, which the lv-hemingway record had already established is defective. Caught +while the base arm was still generating and amended append-only before any McCarthy number +was read (AMENDMENT 1). The correct innocent sample is the author himself. + +``` +sample n hit-rate mean-longest max +HELD-OUT McCARTHY (untrained) 364 0.12 1.0 12 <- the innocent rate +base-unadapted 240 0.00 0.0 8 +ckpt450 240 0.12 1.1 11 +ckpt900 240 0.22 1.9 11 +positive control (train vs train) 160 <- not blind +``` + +⭐ **ckpt450 is INDISTINGUISHABLE from real unseen McCarthy** — 0.12 against 0.12, and its +longest match (11 words) is **SHORTER than the author's own coincidental longest (12)**. That +is the best axis-B result in the line. Against the defective base control its 0.12-vs-0.00 +would have read as a **12× red flag**; against the correct one it is **1.0×**. The amendment +is the only reason this adapter is not on the record as memorising. + +⭐ **And the "register makes collisions inevitable" story — FALSE for Hemingway — is TRUE +here, measured rather than assumed.** Hemingway's held-out rate was 0.01; McCarthy's is 0.12. +His idiom really does self-collide at 8 grams (`and he looked at the`), so the same argument +that was a comfortable excuse there is a fact here. Authors are not interchangeable on this +axis and neither number transfers. + +**All 96 matched runs were READ** (`gate-results/memorisation-matches.txt`), not counted. +Longest 11 words, every one stock grammar in the commonest words: + +``` +he looked at the wolf and he looked at him +leaned back in his chair and looked at the boy +He reached into his pocket and took out the +I dont know What are you goin to do +``` + +The name-shaped hits (`Adam Caleb`, `Toribio`) are the **RENAMED invented names**, not +McCarthy's — the `swift tristan` precedent exactly. No plot, no imagery, nothing protectable. +⚠ McCarthy is in copyright with a living estate and this axis still comes out clean for +ckpt450. + +## ❌ AXIS C: the damage is real, and reading it confirms the criterion + +``` +arm n in-band on-beat ran-on words p90 max over-140 +base 240 0.89 0.71 0.01 110 126 149 1% +ckpt450 240 0.67 0.45 0.20 108 171 297 20% +ckpt900 240 0.67 0.42 0.28 114 190 279 28% + floor 0.200 +``` + +Not an artifact. **Base is GOOD here** (0.89 in-band against Hemingway's 0.05, because +McCarthy's register prompt is far more prescriptive and Qwen3-4B-Instruct follows it), and the +adapter measurably makes it worse — 22 points of in-band, 26–29 of on-beat, and 20–28× base's +overshoot rate. + +Read the worst case and it is **degenerate looping**, not a long McCarthy sentence: + +> *…and then he looked at the wolf and he looked at the road and he looked at the sun and he +> looked at the road again.* — ckpt900, b41, 279 words against a 90–140 ask + +## ⚠ THE COMPLICATION, AND WHY IT DID NOT BECOME A SHIP + +My pre-registration's §6 axis C transcribed `score_beats.py`'s **v1** three-part criterion, +including "in-band up on base beyond the floor". The operator **retired** in-band and on-beat +from the gate on 2026-09-15 for precisely the reason they fail here — the script's own +docstring says *"NOT carried into v2: in-band (unresolvable — base maxes it) and on-beat."* +The v2 axis C is **ran-on only**, and `VERDICT: DO-NOT-SCALE` in the output is the superseded +v1 label the script still prints "for continuity". + +``` +frozen prereg (v1 criteria) ckpt900 fails 3 of 3 ckpt450 fails 2 of 3 +operator's v2 (ran-on only) ckpt900 +0.27 FAILS ckpt450 +0.19 passes by 0.01 +``` + +So my own pre-registration contains a criterion **unsatisfiable on this corpus regardless of +adapter quality**. That is a defect in the pre-registration, not in the adapter — and it is +still not a reason to ship: + +1. The frozen rule fails both candidates, and §7 rule 3 makes axis C disqualifying outright. +2. The reading that rescues ckpt450 was found **after** seeing the numbers. That is the exact + shape pre-registration exists to prevent, and lv-bronte's floor defect is the in-line + precedent for finding one and deliberately not exploiting it. +3. Even under that reading the margin is **0.01 against a floor of 0.200** — noise-adjacent. +4. The damage survives reading, not just the criterion. 20% of ckpt450's generations blow the + length budget and the worst ones loop. + +**Fix the pre-registration prospectively for the next author, do not re-read it for this one.** + +## ⭐⭐⭐ THE TWO-EPOCH RECIPE IS NOW 0 FOR 3 — AND THIS TIME THE LOSS CURVE WAS CONFIDENTLY WRONG + +On Brontë and Hemingway the epoch-1 and epoch-2 checkpoints were **tied** on eval loss, so +preferring the earlier one cost nothing and could be dismissed as taste. Here the curve +**resolved** epoch 2 as better — ckpt900 at 2.38706 against ckpt450's 2.4063, **+4.9× the +0.00393 median neighbour jitter**, nowhere near tied — and it was wrong on every axis that +resolves: + +| | ckpt450 | ckpt900 | ratio | +|---|---|---|---| +| seed spread (voice) | **0.037** | 0.148 | **4.0× wider** | +| memorisation vs the author's 0.12 | **0.12 (1.0×)** | 0.22 (1.8×) | | +| ran-on | **0.20** | 0.28 | | +| on-beat | **0.45** | 0.42 | | +| epochs of overfit | **0.98** | 1.96 | | + +ckpt900's only advantage is a 0.019 better voice point estimate, which sits inside the floor. +And its spread is **one outlier seed** — 0.605 against 0.457 / 0.531 / 0.554 — the third +occurrence of that shape in the later checkpoint, after lv-bronte's ckpt925 and +lv-hemingway's ckpt1750 (0.491 / 0.449 / 0.468 / **0.562**). + +⭐ **The durable rule: on this schedule the eval-loss minimum is not the ship candidate, and +the loss curve's CONFIDENCE about it carries no information.** Read the resolving axes. This +is now measured on three corpora and should be the default for Faulkner, Morrison and Chandler +rather than re-derived each time. + +## ⚠ A bug in my own confound detector, and the order I found it in + +§5c pre-registered a trigger: base quote density over **100 per 10k** means the control did +not take the punctuation win it was handed, and the normalised secondary read is promoted to +load-bearing. The base arm finished first, so I evaluated it early, **saw it FIRE at 224.7**, +and only then — reading the reference row against a corpus whose builder ASSERTS 0.0 — found +that `_QUOTE_RE` contained `'` and `’`. It was an apostrophe counter wearing a quote-mark +label, on the one corpus whose signature is `dont`/`aint`/`wont`. + +``` + as implemented TRUE quotes all apostrophes +held-out McCarthy ref 121.1 0.0 121.1 +base-unadapted 224.7 19.9 204.8 +held-out Hemingway ref 1112.6 694.7 351.7 +``` + +Fixing a detector to measure the quantity the frozen rule names is not moving the rule — but +the fix un-fires the trigger, which is indistinguishable from shopping. **So the trigger was +made MOOT rather than adjudicated** (AMENDMENT 2): the normalised read is load-bearing +unconditionally for this gate, both columns reported, and the fix has zero verdict effect. +There is a better reason anyway — base's true density is 19.9 against the reference's 0.0, so +it did not fully comply and a small residual cheap win genuinely exists. + +⭐ The normalised read **HOLDS**: ckpt900 +0.124 at 1.2× and ckpt450 +0.114 at 2.5×, against +primaries of +0.172 and +0.152. So roughly three quarters of the voice gain survives having +every punctuation mark removed — **the voice is not the punctuation trick.** Validated on +Hemingway first, where the secondary read also resolves a gap rather than flattening +everything, so a null here would have been a finding rather than a blind instrument. + +⚠ **The lesson is the one this line keeps relearning somewhere new:** I controlled +`strip_punct` (2500 → 0) and the byte-identity of the default path, and never asked the quote +counter for a value whose answer I already knew. The corpus asserts 0.0. That check cost one +line and was available before the gate launched. + +## Instruments built, each validated against a documented finding + +| instrument | control | +|---|---| +| `voice_distance.py --secondary-normalised --punct-report` | default path reproduces the shipped lv-hemingway `voice_distance.txt` **byte for byte** | +| `memorization_check.py --train-only --heldout-reference` | reproduces lv-hemingway's hand-computed held-out row **to the digit** (370 samples, 0.01, 0.1, max 10, 101-word chunks) | +| `show_memorisation_matches.py` (new) | reproduces the lv-hemingway reading **to the word**, including `swift tristan` as the one name-shaped hit | +| `ship-voice-adapter.sh` (new) | the live lv-hemingway adapter verified byte-identical to gx10's checkpoint-850 | + +⚠ The held-out control and the match reading were both done **by hand** for Hemingway and left +no instrument, so the finding was not reproducible. Both are now committed. + +⚠ **The waiter I first armed was silently broken**: `pgrep -f "eval-mccarthy.sh"` over ssh +self-matches its own `bash -c` argv, so the DIED branch was unreachable and a crashed gate +would have looked exactly like a running one. Controlled both ways after the fix — bare +pattern demonstrably self-matches, bracketed one does not. +→ [[feedback_pkill_ssh_self_match]] + +## What to do next — ckpt450 is the candidate, and an earlier one may beat it + +The damage grows monotonically with epoch: base 1% over-band → ckpt450 20% → ckpt900 28%. +`--save-total-limit 60` kept **all 56 checkpoints at 25-step intervals**, so ckpt225 / ckpt300 +(epoch ~0.49 / 0.65) are on disk and untested. Voice at 0.5 epoch is unknown and may not have +arrived; that is one arm each to find out. This is a salvageable adapter, not a failed one — +the voice transferred and the memorisation is the cleanest in the line. + +Related: [[2026-09-17-lv-hemingway-gate]], [[2026-09-17-lv-bronte-gate]], +[[2026-09-17-mccarthy-d1-d3]], [[2026-09-17-mccarthy-split-name-leak]]. diff --git a/persistent-memory.md b/persistent-memory.md index ffb8fcb..d2356de 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -1,6 +1,6 @@ # Persistent memory — eshpfi-management -_Last updated: 2026-09-21 ~14:30 PT (⭐ **the ops log is BUILT** and then failed four ways in its first hours — every one recording something unfindable; the day's subject was instruments that report without looking. ⭐ Draupnir engine COMPLETE and acceptance-tested on irv-ml1. ⭐ Booth gained blur + a closed keep round trip after shipping two dead controls. ⭐ claude-bot is an org Owner; cicada+draupnir moved to `pfi`; vh-token use standing-authorized from the vault. ⭐ nh3-dev 84%→72%, and a LoRA adapter rescued from a 3-day-swept /tmp. ⚠ lv-mccarthy's run outcome STILL UNVERIFIED after two days.)_ +_Last updated: 2026-09-21 ~16:45 PT (⭐ **the ops log is BUILT** and then failed four ways in its first hours — every one recording something unfindable; the day's subject was instruments that report without looking. ⭐ Draupnir engine COMPLETE and acceptance-tested on irv-ml1. ⭐ Booth gained blur + a closed keep round trip after shipping two dead controls. ⭐ claude-bot is an org Owner; cicada+draupnir moved to `pfi`; vh-token use standing-authorized from the vault. ⭐ nh3-dev 84%→72%, and a LoRA adapter rescued from a 3-day-swept /tmp. ⭐⭐ lv-mccarthy GATED: voice passes and its memorisation is the cleanest in the line, but 20-28% of generations blow the length band — NOT shipped, and the two-epoch recipe is now 0 for 3 with the loss curve confidently wrong.)_ > **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its > `Written:` stamp is under **8 hours** old, read it (it carries the in-flight @@ -115,18 +115,36 @@ no longer deployed sidecars here. See Recent decisions.) ## Current state / in-flight -_As of 2026-09-21 ~14:30 PT._ +_As of 2026-09-21 ~16:40 PT._ -### ⚠ FIRST — lv-mccarthy's run outcome is STILL UNVERIFIED +### ⚠ FIRST — lv-mccarthy is GATED and NOT SHIPPED; the decision is the operator's -Carried unchanged from the 2026-09-19 handoff and untouched for two full days. -Look at `~/r49-runs/mccarthy-4b-pairs-3ep/` before anything else. ⚠ `pfi-gx10` -does not resolve from nh3-dev by that name. Assume nothing — it was never -checked, not checked-and-found-good. Then: checkpoint selection off the loss -curve → the v2 gate (`eval-*.sh`) → ship-or-park. ⚠ Before the gate, settle how -the VOICE axis is read: D1 pre-registered a punctuation-normalised secondary -read, and `--system-from` hands the register's tics to the BASE arm too, which -changes what the primary number means. +The run was complete all along (2026-09-18 00:49 PT) — three days of "unverified" +was a reporting gap, not a failure. The v2 gate has now RUN, 720 generations, and +both candidates are **disqualified by axis C**. Full record: +`persistent-memory.d/2026-09-21-lv-mccarthy-gate.md`. ⚠ `pfi-gx10` does not +resolve from nh3-dev by name — it is **10.100.50.60**. + +**The shape of it:** voice PASSES on both candidates and both reads (ckpt450 at +2.9× its floor, and ~3/4 of the gain survives having every punctuation mark +stripped, so it is not the cheap win). Memorisation on **ckpt450 is the cleanest +result in the line** — 0.12 against the author's own held-out 0.12, longest match +11 words against the author's coincidental 12, all 96 matches read and every one +stock grammar. **Damage is the blocker**: 20% (ckpt450) / 28% (ckpt900) of +generations overshoot the 90–140 band against base's 1%, worst case a degenerate +`he looked at X and he looked at Y` loop at 279 words. + +**Two things a next session must not re-litigate:** +1. **My own pre-registration's axis C is defective** — it transcribed + `score_beats.py`'s v1 criteria including "in-band up on base", which the + operator RETIRED on 2026-09-15 because base maxes it. Under the operator's v2 + (ran-on only) ckpt450 passes by **0.01 against a 0.200 floor**. That reading + was found AFTER the numbers, so it was not used. **Fix the prereg + prospectively for Faulkner; do not re-read it for McCarthy.** +2. **ckpt450 is the candidate, not ckpt900** (the loss minimum). It wins every + resolving axis. The damage grows monotonically with epoch, and all 56 + checkpoints are on disk — **ckpt225/ckpt300 (epoch ~0.5–0.65) are untested** + and are the obvious next probe, one arm each. ### Draupnir engine — COMPLETE on irv-ml1, acceptance passing @@ -160,6 +178,11 @@ the althing thread. ## Recent decisions +- `[2026-09-21]` ⭐⭐⭐ **lv-mccarthy GATED and NOT SHIPPED — voice passes, memorisation is the cleanest in the line, and the length discipline is gone.** 720 generations, 3 arms. Voice +0.152 at **2.9×** floor for ckpt450 (ckpt900 +0.172 at only 1.2×, its spread one outlier seed), and **~3/4 of the gain survives stripping every punctuation mark**, so it is not the cheap win. ⭐ Memorisation: **ckpt450 at 0.12 against the author's own held-out 0.12 — identical, longest match 11 words against the author's coincidental 12**, all 96 matches READ and every one stock grammar (`he looked at the wolf and he looked at him`); the name-shaped hits are the RENAMED inventions. Axis C is the blocker: **20% / 28% of generations overshoot the 90–140 band against base's 1%**, worst case a degenerate loop at 279 words. ⚠ My own prereg's axis C transcribed `score_beats.py`'s **v1** criteria including "in-band up on base", which the operator RETIRED 2026-09-15 because base maxes it — under the v2 (ran-on only) ckpt450 passes by **0.01 against a 0.200 floor**, but that reading was found AFTER the numbers and was not used. Fix the prereg prospectively. → `persistent-memory.d/2026-09-21-lv-mccarthy-gate.md` +- `[2026-09-21]` ⭐⭐⭐ **The two-epoch recipe is now 0 for 3, and this time the loss curve was CONFIDENTLY wrong.** On Brontë and Hemingway the epoch-1/epoch-2 checkpoints were TIED on eval loss, so preferring the earlier one cost nothing. Here the curve RESOLVED epoch 2 as better — ckpt900 +4.9× the 0.00393 median neighbour jitter above ckpt450, nowhere near tied — and epoch 2 lost every axis that resolves: **4.0× wider seed spread** (0.148 vs 0.037), **1.8× the author's memorisation rate vs 1.0×**, more ran-on (0.28 vs 0.20), worse on-beat. Its only win is a 0.019 voice point estimate, inside the floor, and its spread is ONE outlier seed (0.605 vs 0.457/0.531/0.554) — the third occurrence of that shape in the later checkpoint after lv-bronte's ckpt925 and lv-hemingway's ckpt1750. **Durable: on this schedule the eval-loss minimum is not the ship candidate, and the curve's CONFIDENCE about it carries no information.** Default this for Faulkner/Morrison/Chandler rather than re-deriving it. +- `[2026-09-21]` ⭐⭐ **The memorisation control that shipped broken is now committed, and it turned a 12× red flag into a clean pass.** `memorization_check.py` used **base-unadapted** as its negative control; base writes summary while the adapted arms write pastiche, so it cannot collide with a register it does not imitate and its zero is unearned. The lv-hemingway gate established this, computed the correct control (the author's own held-out text) **by hand**, and left no instrument — so the finding was not reproducible. Now `--train-only --heldout-reference`, refusing the unsafe combination, validated by reproducing Hemingway's hand-computed row **to the digit**. ⭐ On McCarthy it is the difference between reporting ckpt450 as memorising (0.12 vs base 0.00 = 12×) and clean (0.12 vs the author's 0.12 = 1.0×). ⭐ And the "the register makes collisions inevitable" story that was FALSE for Hemingway (0.01) is TRUE for McCarthy (0.12) — measured, not assumed; neither number transfers between authors. Also new: `show_memorisation_matches.py`, because rate and exposure are different questions and the reading was hand-done too. Commits `b4ba731` `7eadbd6`. +- `[2026-09-21]` ⚠⚠ **My confound detector was counting apostrophes as quote marks, and I saw it FIRE before I saw the bug.** `voice_distance.py`'s quote class shipped with `'` and `’` in it — on the one corpus whose signature is `dont`/`aint`/`wont`. It reported the held-out reference at 121.1 "quote marks" per 10k for a corpus whose builder **ASSERTS 0.0**, and fired the pre-registered trigger on a base arm whose true density is 19.9. Fixing a detector to measure the quantity the frozen rule names is not moving the rule, but the fix un-fires the trigger, which is indistinguishable from shopping — **so the trigger was made MOOT instead of adjudicated**: the normalised read is load-bearing unconditionally, both columns reported, zero verdict effect. ⚠ The lesson: I controlled `strip_punct` (2500→0) and the byte-identity of the default path, and never asked the quote counter for a value whose answer I already knew. Commit `0d80e49`. + - `[2026-09-21]` ⭐⭐⭐ **The ops log shipped — and then failed FOUR ways in its first hours, every one recording something unfindable.** Claim dropped by a sub-tool, hook behind graphify's eight `exit 0`s, no handle in the env, ssh-target written as a hostname. The general shape is **configured ≠ effective**; twelve instruments reported confidently and wrongly across three days, five of them mine. → `persistent-memory.d/2026-09-21-ops-log-and-the-instruments-that-lied.md` - `[2026-09-21]` ⭐⭐ **The Booth gained blur + a closed keep round trip, after shipping TWO controls that did nothing** — a reveal handler Jinja discarded for sitting after `{% endblock %}`, and a `×` a sibling form covered by 30×22 px. Both operator-found, both from reading templates instead of rendering them. `scripts/layout-probe.py` took four iterations to become trustworthy. ⚠ Blur is COSMETIC and a test asserts the 200 on purpose. → `persistent-memory.d/2026-09-21-booth-two-dead-controls.md` diff --git a/scripts/mccarthy-corpus/gate-results/memorisation-matches.txt b/scripts/mccarthy-corpus/gate-results/memorisation-matches.txt new file mode 100644 index 0000000..5a970a6 --- /dev/null +++ b/scripts/mccarthy-corpus/gate-results/memorisation-matches.txt @@ -0,0 +1,199 @@ +corpus (train split only): 548,202 words, 547,556 distinct 8-grams + [PASS] positive control: a corpus slice matches itself + +96 matched run(s), longest 11 words. 42 carry a mid-run capital. + + [ 11w] ckpt900 b31 seed=9012 ⚠ CAPS: Adam, Caleb, What + Toribio looked at Adam Caleb What do you want to do + [ 11w] ckpt900 b41 seed=5678 ⚠ CAPS: The + He looked at the old man The old man was watching + [ 10w] ckpt450 b9 seed=3456 + leaned back in his chair and looked at the boy + [ 10w] ckpt450 b21 seed=3456 + he sat with his elbows on his knees and looked + [ 10w] ckpt450 b29 seed=1234 + he looked at the wolf and he looked at him + [ 10w] ckpt900 b9 seed=3456 + The old man drank the last of his coffee and + [ 10w] ckpt900 b21 seed=5678 + sat leaning with his elbows on the table and his + [ 10w] ckpt900 b37 seed=1234 ⚠ CAPS: Toribio, I, Adam, Caleb, I + said Toribio I dont know said Adam Caleb I aint + [ 10w] ckpt900 b41 seed=3456 + he looked at the wolf and he looked at the + [ 10w] ckpt900 b46 seed=9012 + them in his hand and looked at them and then + [ 10w] ckpt900 b49 seed=3456 ⚠ CAPS: Adam, Caleb, Toribio + He looked at Adam Caleb and he looked at Toribio + [ 9w] ckpt450 b7 seed=1234 ⚠ CAPS: What + I dont know What are you goin to do + [ 9w] ckpt450 b7 seed=9012 + He reached into his pocket and took out the + [ 9w] ckpt450 b11 seed=1234 ⚠ CAPS: What + I dont know What are you goin to do + [ 9w] ckpt450 b22 seed=3456 + He got up and went into the bathroom and + [ 9w] ckpt450 b30 seed=5678 ⚠ CAPS: He + He turned and looked at the boy He was + [ 9w] ckpt450 b35 seed=1234 ⚠ CAPS: What + I dont know What are you goin to do + [ 9w] ckpt450 b47 seed=5678 ⚠ CAPS: The + He looked at the old man The old man + [ 9w] ckpt450 b47 seed=5678 ⚠ CAPS: The + He looked at the old man The old man + [ 9w] ckpt450 b48 seed=1234 ⚠ CAPS: What + I dont know What are you goin to do + [ 9w] ckpt450 b49 seed=3456 + he was sitting with his back to the wall + [ 9w] ckpt450 b53 seed=3456 + looked at the wolf and he looked at the + [ 9w] ckpt450 b55 seed=3456 + and took out a cigarette and lit it and + [ 9w] ckpt900 b4 seed=1234 + in one hand and the shotgun in the other + [ 9w] ckpt900 b6 seed=9012 + and held it out and the boy took it + [ 9w] ckpt900 b15 seed=5678 + on the far side of the road and he + [ 9w] ckpt900 b26 seed=5678 + and he stood with his hat in his hand + [ 9w] ckpt900 b28 seed=3456 ⚠ CAPS: I + I dont know what you re talkin about I + [ 9w] ckpt900 b31 seed=5678 + The sun had set and it was growing dark + [ 9w] ckpt900 b40 seed=1234 ⚠ CAPS: What + I dont know What are you goin to do + [ 9w] ckpt900 b42 seed=1234 ⚠ CAPS: The + He looked at the boy The boy was sitting + [ 9w] ckpt900 b47 seed=3456 ⚠ CAPS: He + man looked at him He looked at the horse + [ 9w] ckpt900 b48 seed=5678 ⚠ CAPS: I + I dont know he said I dont know how + [ 9w] ckpt900 b51 seed=9012 ⚠ CAPS: I, I + shook his head I dont know he said I + [ 9w] ckpt900 b59 seed=9012 ⚠ CAPS: He + The man looked at him He looked at the + [ 8w] base b53 seed=1234 ⚠ CAPS: The + looked at the old man The old man + [ 8w] ckpt450 b2 seed=9012 + He looked at the man and then he + [ 8w] ckpt450 b6 seed=3456 + He looked at the boy and he looked + [ 8w] ckpt450 b10 seed=5678 + and put the key in the door and + [ 8w] ckpt450 b10 seed=9012 + He put the key in the ignition and + [ 8w] ckpt450 b12 seed=3456 ⚠ CAPS: The + said The boy turned and looked at him + [ 8w] ckpt450 b22 seed=5678 + it in the palm of his hand and + [ 8w] ckpt450 b22 seed=5678 + it and put it in his pocket and + [ 8w] ckpt450 b25 seed=1234 ⚠ CAPS: They, What + he said They looked at each other What + [ 8w] ckpt450 b25 seed=3456 + and wiped his forehead with the back of + [ 8w] ckpt450 b28 seed=1234 + I dont know what you re talkin about + [ 8w] ckpt450 b30 seed=9012 + and unscrewed the cap and drank and screwed + [ 8w] ckpt450 b35 seed=5678 ⚠ CAPS: He + looked at him He shook his head and + [ 8w] ckpt450 b46 seed=3456 + He turned it over and looked at the + [ 8w] ckpt450 b47 seed=1234 + He sat with his elbows on his knees + [ 8w] ckpt450 b51 seed=5678 ⚠ CAPS: After, I + didnt answer After a while he said I + [ 8w] ckpt450 b52 seed=9012 ⚠ CAPS: He + He sat there for a long time He + [ 8w] ckpt450 b53 seed=3456 + He looked at the old man and he + [ 8w] ckpt450 b56 seed=3456 + he stood and put on his hat and + [ 8w] ckpt900 b2 seed=3456 ⚠ CAPS: He + He looked at them He looked at the + [ 8w] ckpt900 b4 seed=5678 ⚠ CAPS: He + there was no one in the room He + [ 8w] ckpt900 b9 seed=1234 + for him to say something but he didnt + [ 8w] ckpt900 b10 seed=3456 + and got out and walked down to the + [ 8w] ckpt900 b12 seed=3456 ⚠ CAPS: The + said The boy turned and looked at him + [ 8w] ckpt900 b13 seed=3456 + He stepped back and looked out the door + [ 8w] ckpt900 b17 seed=1234 + What are you goin to do with her + [ 8w] ckpt900 b17 seed=1234 + What are you goin to do with her + [ 8w] ckpt900 b17 seed=3456 + The boy sat holding the wolf by the + [ 8w] ckpt900 b23 seed=1234 ⚠ CAPS: The, He + said The boy looked at the wolf He + [ 8w] ckpt900 b23 seed=9012 ⚠ CAPS: He + man looked at him He looked at the + [ 8w] ckpt900 b27 seed=3456 ⚠ CAPS: He + man looked at him He looked at the + [ 8w] ckpt900 b27 seed=3456 + He put the cigarette in his mouth and + [ 8w] ckpt900 b28 seed=3456 + and he looked at it and he looked + [ 8w] ckpt900 b30 seed=5678 + at the boy and he looked at the + [ 8w] ckpt900 b31 seed=3456 ⚠ CAPS: What + he said What do you want to do + [ 8w] ckpt900 b34 seed=5678 + walked around the front of the truck and + [ 8w] ckpt900 b34 seed=9012 + He walked out into the road and stood + [ 8w] ckpt900 b34 seed=3456 + He walked out to the edge of the + [ 8w] ckpt900 b35 seed=3456 ⚠ CAPS: The + looked at the old man The old man + [ 8w] ckpt900 b35 seed=3456 ⚠ CAPS: She + her She s all right the boy said + [ 8w] ckpt900 b38 seed=9012 + with the heel of his hand and looked + [ 8w] ckpt900 b39 seed=1234 ⚠ CAPS: I + What are you doin he said I thought + [ 8w] ckpt900 b39 seed=1234 + and walked out and stood in the door + [ 8w] ckpt900 b39 seed=5678 ⚠ CAPS: Caleb, No + Adam Caleb shook his head No he said + [ 8w] ckpt900 b41 seed=1234 ⚠ CAPS: He + the road He turned and looked back at + [ 8w] ckpt900 b43 seed=1234 + They were already out of sight of the + [ 8w] ckpt900 b44 seed=5678 ⚠ CAPS: He + on the pommel of the saddle He looked + [ 8w] ckpt900 b46 seed=1234 + and put the key in his pocket and + [ 8w] ckpt900 b47 seed=3456 ⚠ CAPS: He + He leaned back in the chair He looked + [ 8w] ckpt900 b47 seed=3456 + at the boy and he looked at the + [ 8w] ckpt900 b48 seed=5678 ⚠ CAPS: I, I + said I dont know I dont know how + [ 8w] ckpt900 b48 seed=5678 + He looked at the boy and he looked + [ 8w] ckpt900 b49 seed=5678 + about a third of the way down the + [ 8w] ckpt900 b50 seed=3456 + and put it on and looked toward the + [ 8w] ckpt900 b51 seed=9012 + wiped his mouth with the back of his + [ 8w] ckpt900 b52 seed=9012 + He looked in the rearview mirror but he + [ 8w] ckpt900 b52 seed=3456 ⚠ CAPS: He + He looked out the window He looked at + [ 8w] ckpt900 b53 seed=1234 + What are you goin to do with her + [ 8w] ckpt900 b53 seed=9012 ⚠ CAPS: He + he turned and looked at the boy He + [ 8w] ckpt900 b53 seed=3456 ⚠ CAPS: What + he looked at the boy What do you + [ 8w] ckpt900 b54 seed=5678 + they came up out of the trees and + +⚠ READ THESE. The rate is not the finding; what is being reproduced is. diff --git a/scripts/mccarthy-corpus/gate-results/memorization.heldout-control.txt b/scripts/mccarthy-corpus/gate-results/memorization.heldout-control.txt new file mode 100644 index 0000000..67a12a7 --- /dev/null +++ b/scripts/mccarthy-corpus/gate-results/memorization.heldout-control.txt @@ -0,0 +1,14 @@ +corpus (train split only): 548,202 words, 547,556 distinct 8-grams + +arm gens hit-rate mean-longest max +------------------------------------------------------------ +base 240 0.00 0.0 8 +ckpt450 240 0.12 1.1 11 +ckpt900 240 0.22 1.9 11 +------------------------------------------------------------ +HELD-OUT (never trained) 364 0.12 1.0 12 + ^ the innocent-sample rate: real author text, same register, 112-word chunks. + THIS is what a candidate is compared against -- not base-unadapted, whose zero + partly measures `different register` rather than `did not memorise`. + +positive control (corpus slice vs corpus): longest = 160 (must be large, else the detector is blind) diff --git a/scripts/mccarthy-corpus/gate-results/memorization.txt b/scripts/mccarthy-corpus/gate-results/memorization.txt new file mode 100644 index 0000000..34eee2a --- /dev/null +++ b/scripts/mccarthy-corpus/gate-results/memorization.txt @@ -0,0 +1,9 @@ +corpus: 589,025 words, 588,286 distinct 8-grams + +arm gens hit-rate mean-longest max +------------------------------------------------------------ +base 240 0.02 0.1 10 +ckpt450 240 0.13 1.1 11 +ckpt900 240 0.23 2.0 11 + +positive control (corpus slice vs corpus): longest = 160 (must be large, else the detector is blind) diff --git a/scripts/mccarthy-corpus/gate-results/score.ckpt450.txt b/scripts/mccarthy-corpus/gate-results/score.ckpt450.txt new file mode 100644 index 0000000..6efab8e --- /dev/null +++ b/scripts/mccarthy-corpus/gate-results/score.ckpt450.txt @@ -0,0 +1,15 @@ +measurement surface: raw +arm n in-band on-beat cover ran-on words +------------------------------------------------------------------------- +base 240 0.89 0.71 0.63 0.01 110 +ckpt900 240 0.67 0.42 0.44 0.28 114 +ckpt450 240 0.67 0.45 0.44 0.20 108 + +noise floor (max within-arm spread across seeds): 0.200 +⚠ any between-arm gap at or under that is NOT a finding + +1. in-band 0.67 vs 0.89 delta -0.22 FAIL (needs > +0.200) +2. on-beat 0.45 vs 0.71 delta -0.26 FAIL (needs >= -0.200) +3. ran-on 0.20 vs 0.01 delta +0.19 PASS (needs <= +0.200) + +VERDICT: DO-NOT-SCALE diff --git a/scripts/mccarthy-corpus/gate-results/score.ckpt900.txt b/scripts/mccarthy-corpus/gate-results/score.ckpt900.txt new file mode 100644 index 0000000..85a59e2 --- /dev/null +++ b/scripts/mccarthy-corpus/gate-results/score.ckpt900.txt @@ -0,0 +1,15 @@ +measurement surface: raw +arm n in-band on-beat cover ran-on words +------------------------------------------------------------------------- +base 240 0.89 0.71 0.63 0.01 110 +ckpt900 240 0.67 0.42 0.44 0.28 114 +ckpt450 240 0.67 0.45 0.44 0.20 108 + +noise floor (max within-arm spread across seeds): 0.200 +⚠ any between-arm gap at or under that is NOT a finding + +1. in-band 0.67 vs 0.89 delta -0.22 FAIL (needs > +0.200) +2. on-beat 0.42 vs 0.71 delta -0.29 FAIL (needs >= -0.200) +3. ran-on 0.28 vs 0.01 delta +0.27 FAIL (needs <= +0.200) + +VERDICT: DO-NOT-SCALE diff --git a/scripts/mccarthy-corpus/gate-results/voice_distance.txt b/scripts/mccarthy-corpus/gate-results/voice_distance.txt new file mode 100644 index 0000000..c56c149 --- /dev/null +++ b/scripts/mccarthy-corpus/gate-results/voice_distance.txt @@ -0,0 +1,58 @@ +reference: held-out McCarthy, 198,486 words, 248 chunks, 400 char-bigram features +same-author target (held-out McCarthy vs itself): delta_cb = 0.370 + -> the floor of what any arm could reach; lower is more McCarthy-like, this is the best possible + + arm delta_cb per-seed [words] + ckpt900 0.490 (0.605 0.457 0.531 0.554) [30935] spread 0.148 + ckpt450 0.509 (0.562 0.530 0.563 0.567) [28494] spread 0.037 + base-unadapted 0.661 (0.711 0.667 0.673 0.659) [26078] spread 0.052 + + all-arms noise floor (largest within-arm seed spread, lv-bronte's rule): 0.148 + PAIRWISE floor is the verdict: max(spread(candidate), spread(base-unadapted) = 0.052) + + vs base-unadapted control (positive gap = moved toward McCarthy): + ckpt900 +0.172 (MOVED toward McCarthy (1.2x the pairwise floor 0.148)) + ckpt450 +0.152 (MOVED toward McCarthy (2.9x the pairwise floor 0.052)) + + ordering: ckpt900 < ckpt450 < base-unadapted (lower = more McCarthy-like) + ⚠ RELATIVE reading on one harness: 4 seed group(s) per arm, scored against this corpus's own held-out split. It is not an absolute-band claim and corroborates nothing on its own. + + PUNCTUATION DENSITY per 10k words -- the confound check, not an axis + (the eval harness drives EVERY arm with the same register prompt, tics included; + a compliant base control earns the adapter no delta_cb for them) + arm quote-marks all-apos contraction-apos dashes + held-out reference 0.0 121.1 117.6 7.6 + base-unadapted 19.9 204.8 203.6 1.9 + ckpt450 0.0 167.4 166.4 0.0 + ckpt900 0.0 160.7 160.7 0.0 + + [PASS] base control quote density 19.9 <= 100 per 10k: the control complied with the register, + so the punctuation win is handed to both sides and the primary read stands. + +============================================================================== +SECONDARY READ -- PUNCTUATION STRIPPED. Pre-registered, REPORTED, NOT THE VERDICT. +Every punctuation mark is removed from the reference and from every arm, so a +gap that survives here is carried by words rather than by marks. It is a LOWER +BOUND and not a better measurement: stripping terminal punctuation also strips +sentence-length signal the adapter legitimately learned. Read it as `at least +this much of the primary gap is not the punctuation trick`. +============================================================================== + +reference: held-out McCarthy, 200,682 words, 251 chunks, 400 char-bigram features +same-author target (held-out McCarthy vs itself): delta_cb = 0.363 + -> the floor of what any arm could reach; lower is more McCarthy-like, this is the best possible + + arm delta_cb per-seed [words] + ckpt900 0.464 (0.563 0.460 0.493 0.522) [31440] spread 0.104 + ckpt450 0.474 (0.518 0.506 0.528 0.526) [28975] spread 0.022 + base-unadapted 0.588 (0.634 0.611 0.594 0.588) [26645] spread 0.046 + + all-arms noise floor (largest within-arm seed spread, lv-bronte's rule): 0.104 + PAIRWISE floor is the verdict: max(spread(candidate), spread(base-unadapted) = 0.046) + + vs base-unadapted control (positive gap = moved toward McCarthy): + ckpt900 +0.124 (MOVED toward McCarthy (1.2x the pairwise floor 0.104)) + ckpt450 +0.114 (MOVED toward McCarthy (2.5x the pairwise floor 0.046)) + + ordering: ckpt900 < ckpt450 < base-unadapted (lower = more McCarthy-like) + ⚠ RELATIVE reading on one harness: 4 seed group(s) per arm, scored against this corpus's own held-out split. It is not an absolute-band claim and corroborates nothing on its own.