diff --git a/scripts/mccarthy-corpus/GATE-PREREG.md b/scripts/mccarthy-corpus/GATE-PREREG.md index f253331..58db172 100644 --- a/scripts/mccarthy-corpus/GATE-PREREG.md +++ b/scripts/mccarthy-corpus/GATE-PREREG.md @@ -215,3 +215,74 @@ gate here is a narrower claim than a clean leak gate there. # on pfi-gx10, ~/lv-mccarthy ./eval-mccarthy.sh # three arms, 60 beats, 4 seeds, all three axes ``` + +--- + +## AMENDMENT 1 — 2026-09-21, axis B's negative control + +**Written while the `base` arm was still generating and before a single McCarthy +delta_cb, memorisation rate or damage number had been read.** Recorded here rather +than folded into the text above, so the order of events stays legible. + +### What was wrong with §6 axis B as frozen + +§6 says axis B passes if hit-rate and max sit "at or near `base-unadapted`, which +never saw the corpus". **The lv-hemingway gate record already establishes that this +control is defective**, and I wrote the clause anyway: + +> `memorization_check.py` uses the base-unadapted arm as its negative control, but +> base writes 18,035 words of *summary* against the adapted arms' 27,413 of +> *pastiche* — text that does not imitate the register cannot collide with its +> n-grams, so a 0.00 there means "different register", not "did not memorise". +> — `persistent-memory.d/2026-09-17-lv-hemingway-gate.md` + +A negative control that differs from the candidate in a way **correlated with the +metric** is not a control. Base's zero is unearned, and an axis that passes by +comparing against an unearned zero is not measuring anything. + +There is a second, smaller fault in the same instrument: it builds its n-gram set +from **every** record in the renamed copies, val included. The adapter never trained +on val, so a collision with val text is counted as memorisation of training text +when it is not. Brontë's and Hemingway's numbers carry this overcount too. + +### What is added + +Two things, both **strictly harder to pass**. Neither can turn a failing candidate +into a passing one; both can turn a passing one into a failing one. + +1. **The correct negative control: the author himself.** Held-out (val) text — real + McCarthy no arm trained on, in the same register as the candidates by + construction — chunked to the arms' own median generation length and scored + against the training text. This is the innocent-sample rate. On Hemingway it was + **0.01 against the adapter's 0.07**, which refuted the comfortable "his plain + register makes collisions inevitable" story rather than assuming it. +2. **A train-only n-gram set**, so a collision with held-out text is not counted as + memorisation of the training text. The inherited all-records number is printed + alongside it, because Brontë's and Hemingway's records were written with it and a + reader comparing runs needs both. + +### The amended axis B, and the amended tiebreak + +- **Axis B passes** if the candidate's hit-rate and max are at or near the + **held-out-author reference**, not base. Base stays in the table as the + register-mismatch marker it actually is. The positive control (train vs train) + must still saturate or every zero in the table is blind. +- **§7 rule 2 is amended to match**: `ckpt450` displaces `ckpt900` only if it is not + resolvably worse on axis A **and** strictly better on axis B *measured against the + held-out-author reference*. +- **An elevated rate is not by itself a no-ship.** The lv-hemingway precedent is + explicit that rate and exposure are different questions: it shipped at 7× the + author's own rate because **every matched run was read** and every one was stock + dialogue in the commonest words in English, max 9 words, no proper noun. The same + reading is required here and is part of the axis, not a follow-up. ⚠ **McCarthy is + in copyright with a living estate**, so a matched run carrying distinctive imagery + or a proper noun is disqualifying in a way it was not for a rate number alone. + +### Why this is not threshold-shopping + +The rule that must not move is the one that decides a result. This amendment moves +the reference a candidate is compared against **from a known-broken one to a known- +correct one**, in the direction that makes passing harder, on an argument published +four days before this gate existed and independent of any McCarthy number — none of +which had been read when this was written. The honest alternative was to run an axis +whose control I already knew to be unearned. diff --git a/scripts/yarros-corpus/memorization_check.py b/scripts/yarros-corpus/memorization_check.py index 7d5fdf3..64150f9 100644 --- a/scripts/yarros-corpus/memorization_check.py +++ b/scripts/yarros-corpus/memorization_check.py @@ -9,8 +9,33 @@ pipeline and its gate exist to prevent. Char-bigram distance is blind to the dif Instrument: longest and mean maximal verbatim n-gram shared with the TRAINING corpus, per generation. Controls run every time -- the base-unadapted arm never saw the corpus so it is the negative control, and a slice of the corpus scored against itself is the positive. + +⚠⚠ THE BASE-UNADAPTED NEGATIVE CONTROL IS DEFECTIVE, and lv-hemingway is where it was +caught. Base writes SUMMARY (18,035 words) while the adapted arms write PASTICHE (27,413): +text that does not imitate the register cannot collide with that register's n-grams, so +base's 0.00 partly measures "different register" rather than "did not memorise". A negative +control that differs from the candidate in a way CORRELATED with the metric is not a control. + +The correct innocent sample is THE AUTHOR HIMSELF -- held-out text no arm trained on, in the +same register by construction. On Hemingway it read 0.01 against the shipped adapter's 0.07, +which REFUTED the comfortable "his plain register makes collisions inevitable" story instead +of assuming it. That control was computed by hand during the lv-hemingway gate and never +committed, so it was not reproducible; `--heldout-reference` is it, committed. + +⭐ Two opt-in flags added 2026-09-21 for lv-mccarthy, pre-registered in +`scripts/mccarthy-corpus/GATE-PREREG.md` AMENDMENT 1. Neither runs by default and neither +changes a byte of the default output, because the Yarros, Bronte and Hemingway records were +written by the default path and must stay reproducible. + + --train-only Build the n-gram set from split=="train" records ONLY. The default + builds it from EVERY record including val -- so a collision with + held-out text is counted as memorisation of training text, which it + is not. Mandatory with --heldout-reference, where the default would + score the val text against a gram set containing itself. + --heldout-reference Score the val-split text as an extra row, chunked to the arms' own + median generation length so the comparison is like for like. """ -import argparse, json, pathlib, re, sys +import argparse, json, pathlib, re, statistics, sys from collections import Counter # ⚠ These were hardcoded to Yarros. Pointed at a Brontë arm they would have compared @@ -24,8 +49,20 @@ _ap.add_argument("--corpus", default="/home/infra-ops/yarros-corpus-renamed/copi _ap.add_argument("--glob", default="beats5.*.jsonl", help="arm files inside --eval-dir") _ap.add_argument("--strip", default="beats5.", help="prefix trimmed to name the arm") _ap.add_argument("-n", type=int, default=8, help="n-gram length") +_ap.add_argument("--train-only", action="store_true", + help="build the gram set from split=='train' records only; see module docstring") +_ap.add_argument("--heldout-reference", action="store_true", + help="add the held-out-author innocent-sample row; requires --train-only") _a = _ap.parse_args() +if _a.heldout_reference and not _a.train_only: + # Without --train-only the gram set contains the val text, so the held-out row would + # score the author's own words against themselves and return a saturated number that + # means nothing. Refuse rather than print it. + print("== REFUSING: --heldout-reference requires --train-only, or the held-out text is") + print(" scored against a gram set that contains it and the row is meaningless.") + sys.exit(1) + EVAL = pathlib.Path(_a.eval_dir) CORP = pathlib.Path(_a.corpus) N = _a.n @@ -33,13 +70,20 @@ N = _a.n def norm(t): return re.findall(r"[a-z']+", t.lower()) corpus_words = [] +heldout_words = [] for f in sorted(CORP.glob("*.copy0.jsonl")): for l in f.read_text(encoding="utf-8").splitlines(): - corpus_words.extend(norm(json.loads(l)["text"])) + r = json.loads(l) + if _a.train_only and r.get("split") != "train": + if r.get("split") == "val": + heldout_words.append(norm(r["text"])) + continue + corpus_words.extend(norm(r["text"])) grams = set() for i in range(len(corpus_words) - N + 1): grams.add(" ".join(corpus_words[i:i + N])) -print(f"corpus: {len(corpus_words):,} words, {len(grams):,} distinct {N}-grams\n") +label = "corpus (train split only)" if _a.train_only else "corpus" +print(f"{label}: {len(corpus_words):,} words, {len(grams):,} distinct {N}-grams\n") def longest_match(words): best = 0 @@ -55,15 +99,41 @@ def longest_match(words): i += 1 return best +def row(name, samples): + longs = [longest_match(w) for w in samples] + hits = sum(1 for x in longs if x >= N) + print(f"{name:<22} {len(samples):>5} {hits/len(samples):>9.2f} " + f"{sum(longs)/len(longs):>13.1f} {max(longs):>5}") + return longs + print(f"{'arm':<22} {'gens':>5} {'hit-rate':>9} {'mean-longest':>13} {'max':>5}") print("-" * 60) +gen_lengths = [] for f in sorted(EVAL.glob(_a.glob)): arm = f.stem.replace(_a.strip, "") rows = [json.loads(l) for l in f.read_text(encoding="utf-8").splitlines()] - longs = [longest_match(norm(r["raw"])) for r in rows] - hits = sum(1 for x in longs if x >= N) - print(f"{arm:<22} {len(rows):>5} {hits/len(rows):>9.2f} " - f"{sum(longs)/len(longs):>13.1f} {max(longs):>5}") + samples = [norm(r["raw"]) for r in rows] + gen_lengths.extend(len(s) for s in samples) + row(arm, samples) + +if _a.heldout_reference: + # ⭐ THE CORRECT NEGATIVE CONTROL. Real author text, never trained on, in the same + # register as the candidates by construction -- which is exactly what base-unadapted + # is not. Chunked to the arms' own median generation length so a longer or shorter + # sample is not being compared with theirs. + if not heldout_words: + print("\n== no split=='val' records found; the held-out reference CANNOT be built.") + print(" This is not a pass -- the axis has no innocent sample.") + sys.exit(1) + chunk = int(statistics.median(gen_lengths)) if gen_lengths else 100 + flat = [w for unit in heldout_words for w in unit] + samples = [flat[i:i + chunk] for i in range(0, len(flat), chunk)] + samples = [s for s in samples if len(s) >= chunk // 2] + print("-" * 60) + row(f"HELD-OUT (never trained)", samples) + print(f" ^ the innocent-sample rate: real author text, same register, {chunk}-word chunks.") + print(" THIS is what a candidate is compared against -- not base-unadapted, whose zero") + print(" partly measures `different register` rather than `did not memorise`.") # positive control: corpus against itself must saturate slice_words = corpus_words[1000:1160]