fix(r49): commit the held-out-author memorisation control, and amend the mccarthy prereg to use it
The lv-hemingway gate established that memorization_check.py's negative control is
defective: base-unadapted writes SUMMARY while the adapted arms write PASTICHE, and
text that does not imitate a register cannot collide with that register's n-grams.
Base's 0.00 partly measures "different register", not "did not memorise". A negative
control that differs from the candidate in a way correlated with the metric is not a
control.
The correct innocent sample is the author himself -- held-out text no arm trained on,
in the same register by construction. That control was computed BY HAND during the
lv-hemingway gate and never committed, so the finding it produced was not
reproducible. This commits it as --heldout-reference, with --train-only to build the
n-gram set from split=="train" records only (the default includes val, so a collision
with held-out text was being counted as memorisation of training text -- an overcount
Bronte's and Hemingway's numbers both carry).
--heldout-reference without --train-only REFUSES, because the held-out text would be
scored against a gram set containing itself.
Instrument controls:
- default path reproduces the shipped lv-hemingway memorization.txt BYTE FOR BYTE
- the new held-out row reproduces the lv-hemingway record's hand-computed numbers
to the digit: 370 samples, hit-rate 0.01, mean-longest 0.1, max 10, at the same
101-word median chunk length -- an independently written instrument reproducing a
documented finding is what makes its zeroes mean absent rather than blind
GATE-PREREG.md AMENDMENT 1 adopts it for lv-mccarthy, appended rather than folded in
so the order of events stays legible. Written while the base arm was still generating
and before any McCarthy number had been read. It moves the reference a candidate is
compared against from a known-broken one to a known-correct one, in the direction
that makes the axis HARDER to pass, on an argument published four days before this
gate existed. Section 7's tiebreak is amended to match, and the amendment records
that an elevated rate is not by itself a no-ship -- rate and exposure are different
questions, every matched run gets READ, and McCarthy being in copyright with a living
estate makes a distinctive match disqualifying in a way a rate number alone is not.
This commit is contained in:
@@ -215,3 +215,74 @@ gate here is a narrower claim than a clean leak gate there.
|
||||
# on pfi-gx10, ~/lv-mccarthy
|
||||
./eval-mccarthy.sh # three arms, 60 beats, 4 seeds, all three axes
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## AMENDMENT 1 — 2026-09-21, axis B's negative control
|
||||
|
||||
**Written while the `base` arm was still generating and before a single McCarthy
|
||||
delta_cb, memorisation rate or damage number had been read.** Recorded here rather
|
||||
than folded into the text above, so the order of events stays legible.
|
||||
|
||||
### What was wrong with §6 axis B as frozen
|
||||
|
||||
§6 says axis B passes if hit-rate and max sit "at or near `base-unadapted`, which
|
||||
never saw the corpus". **The lv-hemingway gate record already establishes that this
|
||||
control is defective**, and I wrote the clause anyway:
|
||||
|
||||
> `memorization_check.py` uses the base-unadapted arm as its negative control, but
|
||||
> base writes 18,035 words of *summary* against the adapted arms' 27,413 of
|
||||
> *pastiche* — text that does not imitate the register cannot collide with its
|
||||
> n-grams, so a 0.00 there means "different register", not "did not memorise".
|
||||
> — `persistent-memory.d/2026-09-17-lv-hemingway-gate.md`
|
||||
|
||||
A negative control that differs from the candidate in a way **correlated with the
|
||||
metric** is not a control. Base's zero is unearned, and an axis that passes by
|
||||
comparing against an unearned zero is not measuring anything.
|
||||
|
||||
There is a second, smaller fault in the same instrument: it builds its n-gram set
|
||||
from **every** record in the renamed copies, val included. The adapter never trained
|
||||
on val, so a collision with val text is counted as memorisation of training text
|
||||
when it is not. Brontë's and Hemingway's numbers carry this overcount too.
|
||||
|
||||
### What is added
|
||||
|
||||
Two things, both **strictly harder to pass**. Neither can turn a failing candidate
|
||||
into a passing one; both can turn a passing one into a failing one.
|
||||
|
||||
1. **The correct negative control: the author himself.** Held-out (val) text — real
|
||||
McCarthy no arm trained on, in the same register as the candidates by
|
||||
construction — chunked to the arms' own median generation length and scored
|
||||
against the training text. This is the innocent-sample rate. On Hemingway it was
|
||||
**0.01 against the adapter's 0.07**, which refuted the comfortable "his plain
|
||||
register makes collisions inevitable" story rather than assuming it.
|
||||
2. **A train-only n-gram set**, so a collision with held-out text is not counted as
|
||||
memorisation of the training text. The inherited all-records number is printed
|
||||
alongside it, because Brontë's and Hemingway's records were written with it and a
|
||||
reader comparing runs needs both.
|
||||
|
||||
### The amended axis B, and the amended tiebreak
|
||||
|
||||
- **Axis B passes** if the candidate's hit-rate and max are at or near the
|
||||
**held-out-author reference**, not base. Base stays in the table as the
|
||||
register-mismatch marker it actually is. The positive control (train vs train)
|
||||
must still saturate or every zero in the table is blind.
|
||||
- **§7 rule 2 is amended to match**: `ckpt450` displaces `ckpt900` only if it is not
|
||||
resolvably worse on axis A **and** strictly better on axis B *measured against the
|
||||
held-out-author reference*.
|
||||
- **An elevated rate is not by itself a no-ship.** The lv-hemingway precedent is
|
||||
explicit that rate and exposure are different questions: it shipped at 7× the
|
||||
author's own rate because **every matched run was read** and every one was stock
|
||||
dialogue in the commonest words in English, max 9 words, no proper noun. The same
|
||||
reading is required here and is part of the axis, not a follow-up. ⚠ **McCarthy is
|
||||
in copyright with a living estate**, so a matched run carrying distinctive imagery
|
||||
or a proper noun is disqualifying in a way it was not for a rate number alone.
|
||||
|
||||
### Why this is not threshold-shopping
|
||||
|
||||
The rule that must not move is the one that decides a result. This amendment moves
|
||||
the reference a candidate is compared against **from a known-broken one to a known-
|
||||
correct one**, in the direction that makes passing harder, on an argument published
|
||||
four days before this gate existed and independent of any McCarthy number — none of
|
||||
which had been read when this was written. The honest alternative was to run an axis
|
||||
whose control I already knew to be unearned.
|
||||
|
||||
Reference in New Issue
Block a user