vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity

fix(r49): commit the held-out-author memorisation control, and amend the mccarthy prereg to use it

Browse Source 
The lv-hemingway gate established that memorization_check.py's negative control is
defective: base-unadapted writes SUMMARY while the adapted arms write PASTICHE, and
text that does not imitate a register cannot collide with that register's n-grams.
Base's 0.00 partly measures "different register", not "did not memorise". A negative
control that differs from the candidate in a way correlated with the metric is not a
control.

The correct innocent sample is the author himself -- held-out text no arm trained on,
in the same register by construction. That control was computed BY HAND during the
lv-hemingway gate and never committed, so the finding it produced was not
reproducible. This commits it as --heldout-reference, with --train-only to build the
n-gram set from split=="train" records only (the default includes val, so a collision
with held-out text was being counted as memorisation of training text -- an overcount
Bronte's and Hemingway's numbers both carry).

--heldout-reference without --train-only REFUSES, because the held-out text would be
scored against a gram set containing itself.

Instrument controls:
  - default path reproduces the shipped lv-hemingway memorization.txt BYTE FOR BYTE
  - the new held-out row reproduces the lv-hemingway record's hand-computed numbers
    to the digit: 370 samples, hit-rate 0.01, mean-longest 0.1, max 10, at the same
    101-word median chunk length -- an independently written instrument reproducing a
    documented finding is what makes its zeroes mean absent rather than blind

GATE-PREREG.md AMENDMENT 1 adopts it for lv-mccarthy, appended rather than folded in
so the order of events stays legible. Written while the base arm was still generating
and before any McCarthy number had been read. It moves the reference a candidate is
compared against from a known-broken one to a known-correct one, in the direction
that makes the axis HARDER to pass, on an argument published four days before this
gate existed. Section 7's tiebreak is amended to match, and the amendment records
that an elevated rate is not by itself a no-ship -- rate and exposure are different
questions, every matched run gets READ, and McCarthy being in copyright with a living
estate makes a distinctive match disqualifying in a way a rate number alone is not.
This commit is contained in:
Vuong Hoang
2026-09-21 14:42:24 -07:00
parent 9c8a4e959c
commit b4ba731b26
2 changed files with 148 additions and 7 deletions
Download Patch File Download Diff File Expand all files Collapse all files
scripts/mccarthy-corpus/GATE-PREREG.md
+71
View File
@@ -215,3 +215,74 @@ gate here is a narrower claim than a clean leak gate there.
# on pfi-gx10, ~/lv-mccarthy
./eval-mccarthy.sh # three arms, 60 beats, 4 seeds, all three axes
```
---
## AMENDMENT 1 — 2026-09-21, axis B's negative control
**Written while the `base` arm was still generating and before a single McCarthy
delta_cb, memorisation rate or damage number had been read.** Recorded here rather
than folded into the text above, so the order of events stays legible.
### What was wrong with §6 axis B as frozen
§6 says axis B passes if hit-rate and max sit "at or near `base-unadapted`, which
never saw the corpus". **The lv-hemingway gate record already establishes that this
control is defective**, and I wrote the clause anyway:
> `memorization_check.py` uses the base-unadapted arm as its negative control, but
> base writes 18,035 words of *summary* against the adapted arms' 27,413 of
> *pastiche* — text that does not imitate the register cannot collide with its
> n-grams, so a 0.00 there means "different register", not "did not memorise".
> — `persistent-memory.d/2026-09-17-lv-hemingway-gate.md`
A negative control that differs from the candidate in a way **correlated with the
metric** is not a control. Base's zero is unearned, and an axis that passes by
comparing against an unearned zero is not measuring anything.
There is a second, smaller fault in the same instrument: it builds its n-gram set
from **every** record in the renamed copies, val included. The adapter never trained
on val, so a collision with val text is counted as memorisation of training text
when it is not. Brontë's and Hemingway's numbers carry this overcount too.
### What is added
Two things, both **strictly harder to pass**. Neither can turn a failing candidate
into a passing one; both can turn a passing one into a failing one.
1. **The correct negative control: the author himself.** Held-out (val) text — real
McCarthy no arm trained on, in the same register as the candidates by
construction — chunked to the arms' own median generation length and scored
against the training text. This is the innocent-sample rate. On Hemingway it was
**0.01 against the adapter's 0.07**, which refuted the comfortable "his plain
register makes collisions inevitable" story rather than assuming it.
2. **A train-only n-gram set**, so a collision with held-out text is not counted as
memorisation of the training text. The inherited all-records number is printed
alongside it, because Brontë's and Hemingway's records were written with it and a
reader comparing runs needs both.
### The amended axis B, and the amended tiebreak
- **Axis B passes** if the candidate's hit-rate and max are at or near the
**held-out-author reference**, not base. Base stays in the table as the
register-mismatch marker it actually is. The positive control (train vs train)
must still saturate or every zero in the table is blind.
- **§7 rule 2 is amended to match**: `ckpt450` displaces `ckpt900` only if it is not
resolvably worse on axis A **and** strictly better on axis B *measured against the
held-out-author reference*.
- **An elevated rate is not by itself a no-ship.** The lv-hemingway precedent is
explicit that rate and exposure are different questions: it shipped at 7× the
author's own rate because **every matched run was read** and every one was stock
dialogue in the commonest words in English, max 9 words, no proper noun. The same
reading is required here and is part of the axis, not a follow-up. ⚠ **McCarthy is
in copyright with a living estate**, so a matched run carrying distinctive imagery
or a proper noun is disqualifying in a way it was not for a rate number alone.
### Why this is not threshold-shopping
The rule that must not move is the one that decides a result. This amendment moves
the reference a candidate is compared against **from a known-broken one to a known-
correct one**, in the direction that makes passing harder, on an argument published
four days before this gate existed and independent of any McCarthy number — none of
which had been read when this was written. The honest alternative was to run an axis
whose control I already knew to be unearned.
scripts/yarros-corpus/memorization_check.py
+77 -7
View File
@@ -9,8 +9,33 @@ pipeline and its gate exist to prevent. Char-bigram distance is blind to the dif
Instrument: longest and mean maximal verbatim n-gram shared with the TRAINING corpus, per
generation. Controls run every time -- the base-unadapted arm never saw the corpus so it is
the negative control, and a slice of the corpus scored against itself is the positive.
⚠⚠ THE BASE-UNADAPTED NEGATIVE CONTROL IS DEFECTIVE, and lv-hemingway is where it was
caught. Base writes SUMMARY (18,035 words) while the adapted arms write PASTICHE (27,413):
text that does not imitate the register cannot collide with that register's n-grams, so
base's 0.00 partly measures "different register" rather than "did not memorise". A negative
control that differs from the candidate in a way CORRELATED with the metric is not a control.
The correct innocent sample is THE AUTHOR HIMSELF -- held-out text no arm trained on, in the
same register by construction. On Hemingway it read 0.01 against the shipped adapter's 0.07,
which REFUTED the comfortable "his plain register makes collisions inevitable" story instead
of assuming it. That control was computed by hand during the lv-hemingway gate and never
committed, so it was not reproducible; `--heldout-reference` is it, committed.
⭐ Two opt-in flags added 2026-09-21 for lv-mccarthy, pre-registered in
`scripts/mccarthy-corpus/GATE-PREREG.md` AMENDMENT 1. Neither runs by default and neither
changes a byte of the default output, because the Yarros, Bronte and Hemingway records were
written by the default path and must stay reproducible.
--train-only Build the n-gram set from split=="train" records ONLY. The default
builds it from EVERY record including val -- so a collision with
held-out text is counted as memorisation of training text, which it
is not. Mandatory with --heldout-reference, where the default would
score the val text against a gram set containing itself.
--heldout-reference Score the val-split text as an extra row, chunked to the arms' own
median generation length so the comparison is like for like.
"""
import argparse, json, pathlib, re, sys
import argparse, json, pathlib, re, statistics, sys
from collections import Counter
# ⚠ These were hardcoded to Yarros. Pointed at a Brontë arm they would have compared
@@ -24,8 +49,20 @@ _ap.add_argument("--corpus", default="/home/infra-ops/yarros-corpus-renamed/copi
_ap.add_argument("--glob", default="beats5.*.jsonl", help="arm files inside --eval-dir")
_ap.add_argument("--strip", default="beats5.", help="prefix trimmed to name the arm")
_ap.add_argument("-n", type=int, default=8, help="n-gram length")
_ap.add_argument("--train-only", action="store_true",
help="build the gram set from split=='train' records only; see module docstring")
_ap.add_argument("--heldout-reference", action="store_true",
help="add the held-out-author innocent-sample row; requires --train-only")
_a = _ap.parse_args()
if _a.heldout_reference and not _a.train_only:
# Without --train-only the gram set contains the val text, so the held-out row would
# score the author's own words against themselves and return a saturated number that
# means nothing. Refuse rather than print it.
print("== REFUSING: --heldout-reference requires --train-only, or the held-out text is")
print(" scored against a gram set that contains it and the row is meaningless.")
sys.exit(1)
EVAL = pathlib.Path(_a.eval_dir)
CORP = pathlib.Path(_a.corpus)
N = _a.n
@@ -33,13 +70,20 @@ N = _a.n
def norm(t): return re.findall(r"[a-z']+", t.lower())
corpus_words = []
heldout_words = []
for f in sorted(CORP.glob("*.copy0.jsonl")):
for l in f.read_text(encoding="utf-8").splitlines():
corpus_words.extend(norm(json.loads(l)["text"]))
r = json.loads(l)
if _a.train_only and r.get("split") != "train":
if r.get("split") == "val":
heldout_words.append(norm(r["text"]))
continue
corpus_words.extend(norm(r["text"]))
grams = set()
for i in range(len(corpus_words) - N + 1):
grams.add(" ".join(corpus_words[i:i + N]))
print(f"corpus: {len(corpus_words):,} words, {len(grams):,} distinct {N}-grams\n")
label = "corpus (train split only)" if _a.train_only else "corpus"
print(f"{label}: {len(corpus_words):,} words, {len(grams):,} distinct {N}-grams\n")
def longest_match(words):
best = 0
@@ -55,15 +99,41 @@ def longest_match(words):
i += 1
return best
def row(name, samples):
longs = [longest_match(w) for w in samples]
hits = sum(1 for x in longs if x >= N)
print(f"{name:<22} {len(samples):>5} {hits/len(samples):>9.2f} "
f"{sum(longs)/len(longs):>13.1f} {max(longs):>5}")
return longs
print(f"{'arm':<22} {'gens':>5} {'hit-rate':>9} {'mean-longest':>13} {'max':>5}")
print("-" * 60)
gen_lengths = []
for f in sorted(EVAL.glob(_a.glob)):
arm = f.stem.replace(_a.strip, "")
rows = [json.loads(l) for l in f.read_text(encoding="utf-8").splitlines()]
longs = [longest_match(norm(r["raw"])) for r in rows]
hits = sum(1 for x in longs if x >= N)
print(f"{arm:<22} {len(rows):>5} {hits/len(rows):>9.2f} "
f"{sum(longs)/len(longs):>13.1f} {max(longs):>5}")
samples = [norm(r["raw"]) for r in rows]
gen_lengths.extend(len(s) for s in samples)
row(arm, samples)
if _a.heldout_reference:
# ⭐ THE CORRECT NEGATIVE CONTROL. Real author text, never trained on, in the same
# register as the candidates by construction -- which is exactly what base-unadapted
# is not. Chunked to the arms' own median generation length so a longer or shorter
# sample is not being compared with theirs.
if not heldout_words:
print("\n== no split=='val' records found; the held-out reference CANNOT be built.")
print(" This is not a pass -- the axis has no innocent sample.")
sys.exit(1)
chunk = int(statistics.median(gen_lengths)) if gen_lengths else 100
flat = [w for unit in heldout_words for w in unit]
samples = [flat[i:i + chunk] for i in range(0, len(flat), chunk)]
samples = [s for s in samples if len(s) >= chunk // 2]
print("-" * 60)
row(f"HELD-OUT (never trained)", samples)
print(f" ^ the innocent-sample rate: real author text, same register, {chunk}-word chunks.")
print(" THIS is what a candidate is compared against -- not base-unadapted, whose zero")
print(" partly measures `different register` rather than `did not memorise`.")
# positive control: corpus against itself must saturate
slice_words = corpus_words[1000:1160]