vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity

docs(lv-mccarthy): record the gate result -- voice passes, memorisation is the cleanest in the line, NOT shipped

Browse Source 
720 generations, 3 arms x 60 held-out beats x 4 seeds, against the design frozen
in GATE-PREREG.md before any arm existed.

AXIS A VOICE -- PASS, both candidates, both reads. Span 0.661 -> 0.370 = 0.291
achievable; ckpt900 closed 59.1% (+0.172, but only 1.2x its floor), ckpt450 52.2%
(+0.152 at 2.9x). The normalised secondary read HOLDS at +0.124 / +0.114, so about
three quarters of the gain survives stripping every punctuation mark -- the voice
is not the cheap win the register made available.

AXIS B NOT COPIED -- ckpt450 is the cleanest result in the line. 0.12 hit-rate
against the author's own held-out 0.12, and its longest match (11 words) is
SHORTER than the author's coincidental longest (12). All 96 matched runs were
read: stock grammar in the commonest words, the name-shaped hits are the RENAMED
inventions, nothing protectable. The amendment is why this reads as clean -- the
defective base control would have shown 0.12 vs 0.00 as a 12x red flag.
Separately measured: the "his register makes collisions inevitable" story that
was FALSE for Hemingway (0.01) is TRUE for McCarthy (0.12). Neither transfers.

AXIS C NO DAMAGE -- FAIL, both, and it survives reading. 20% (ckpt450) / 28%
(ckpt900) of generations overshoot the 90-140 band against base's 1%; p90 171/190
words, max 297/279. The worst case is degenerate looping, not a long McCarthy
sentence. Base is GOOD on this axis here (0.89 in-band vs Hemingway's 0.05), so
the adapter measurably makes instruction-following worse.

NOT SHIPPED. Section 7 rule 3 makes axis C disqualifying outright.

Recorded honestly: my own prereg's axis C transcribed score_beats.py's v1
criteria, including "in-band up on base", which the operator RETIRED on
2026-09-15 for exactly the reason it fails here -- base maxes it, so it is
unsatisfiable on this corpus regardless of adapter quality. Under the operator's
v2 (ran-on only) ckpt450 passes by 0.01 against a 0.200 floor. That reading was
found AFTER the numbers and was NOT used; lv-bronte's floor defect is the in-line
precedent for finding one and declining to exploit it. The prereg gets fixed
prospectively for the next author, not re-read for this one.

And the finding worth more than the adapter: the two-epoch recipe is now 0 for 3,
and this time the loss curve was CONFIDENTLY wrong. On Bronte and Hemingway the
epoch-1/epoch-2 checkpoints were tied, so preferring the earlier one cost nothing.
Here the curve resolved epoch 2 as better at 4.9x the median neighbour jitter --
and epoch 2 lost every axis that resolves: 4.0x wider seed spread, 1.8x the
author's memorisation rate against 1.0x, more ran-on, worse on-beat. Its only win
is a 0.019 voice point estimate, inside the floor, and its spread is one outlier
seed -- the third occurrence of that shape in the later checkpoint after
lv-bronte's ckpt925 and lv-hemingway's ckpt1750.

Raw artifacts committed at scripts/mccarthy-corpus/gate-results/ so the claims can
be re-read without gx10.
This commit is contained in:
Vuong Hoang
2026-09-21 16:31:41 -07:00
parent 0d80e493a8
commit 4c3f3896f1
8 changed files with 561 additions and 11 deletions
Download Patch File Download Diff File Expand all files Collapse all files
scripts/mccarthy-corpus/gate-results/memorisation-matches.txt
+199
View File
@@ -0,0 +1,199 @@
corpus (train split only): 548,202 words, 547,556 distinct 8-grams
[PASS] positive control: a corpus slice matches itself
96 matched run(s), longest 11 words. 42 carry a mid-run capital.
[ 11w] ckpt900 b31 seed=9012 ⚠ CAPS: Adam, Caleb, What
Toribio looked at Adam Caleb What do you want to do
[ 11w] ckpt900 b41 seed=5678 ⚠ CAPS: The
He looked at the old man The old man was watching
[ 10w] ckpt450 b9 seed=3456
leaned back in his chair and looked at the boy
[ 10w] ckpt450 b21 seed=3456
he sat with his elbows on his knees and looked
[ 10w] ckpt450 b29 seed=1234
he looked at the wolf and he looked at him
[ 10w] ckpt900 b9 seed=3456
The old man drank the last of his coffee and
[ 10w] ckpt900 b21 seed=5678
sat leaning with his elbows on the table and his
[ 10w] ckpt900 b37 seed=1234 ⚠ CAPS: Toribio, I, Adam, Caleb, I
said Toribio I dont know said Adam Caleb I aint
[ 10w] ckpt900 b41 seed=3456
he looked at the wolf and he looked at the
[ 10w] ckpt900 b46 seed=9012
them in his hand and looked at them and then
[ 10w] ckpt900 b49 seed=3456 ⚠ CAPS: Adam, Caleb, Toribio
He looked at Adam Caleb and he looked at Toribio
[ 9w] ckpt450 b7 seed=1234 ⚠ CAPS: What
I dont know What are you goin to do
[ 9w] ckpt450 b7 seed=9012
He reached into his pocket and took out the
[ 9w] ckpt450 b11 seed=1234 ⚠ CAPS: What
I dont know What are you goin to do
[ 9w] ckpt450 b22 seed=3456
He got up and went into the bathroom and
[ 9w] ckpt450 b30 seed=5678 ⚠ CAPS: He
He turned and looked at the boy He was
[ 9w] ckpt450 b35 seed=1234 ⚠ CAPS: What
I dont know What are you goin to do
[ 9w] ckpt450 b47 seed=5678 ⚠ CAPS: The
He looked at the old man The old man
[ 9w] ckpt450 b47 seed=5678 ⚠ CAPS: The
He looked at the old man The old man
[ 9w] ckpt450 b48 seed=1234 ⚠ CAPS: What
I dont know What are you goin to do
[ 9w] ckpt450 b49 seed=3456
he was sitting with his back to the wall
[ 9w] ckpt450 b53 seed=3456
looked at the wolf and he looked at the
[ 9w] ckpt450 b55 seed=3456
and took out a cigarette and lit it and
[ 9w] ckpt900 b4 seed=1234
in one hand and the shotgun in the other
[ 9w] ckpt900 b6 seed=9012
and held it out and the boy took it
[ 9w] ckpt900 b15 seed=5678
on the far side of the road and he
[ 9w] ckpt900 b26 seed=5678
and he stood with his hat in his hand
[ 9w] ckpt900 b28 seed=3456 ⚠ CAPS: I
I dont know what you re talkin about I
[ 9w] ckpt900 b31 seed=5678
The sun had set and it was growing dark
[ 9w] ckpt900 b40 seed=1234 ⚠ CAPS: What
I dont know What are you goin to do
[ 9w] ckpt900 b42 seed=1234 ⚠ CAPS: The
He looked at the boy The boy was sitting
[ 9w] ckpt900 b47 seed=3456 ⚠ CAPS: He
man looked at him He looked at the horse
[ 9w] ckpt900 b48 seed=5678 ⚠ CAPS: I
I dont know he said I dont know how
[ 9w] ckpt900 b51 seed=9012 ⚠ CAPS: I, I
shook his head I dont know he said I
[ 9w] ckpt900 b59 seed=9012 ⚠ CAPS: He
The man looked at him He looked at the
[ 8w] base b53 seed=1234 ⚠ CAPS: The
looked at the old man The old man
[ 8w] ckpt450 b2 seed=9012
He looked at the man and then he
[ 8w] ckpt450 b6 seed=3456
He looked at the boy and he looked
[ 8w] ckpt450 b10 seed=5678
and put the key in the door and
[ 8w] ckpt450 b10 seed=9012
He put the key in the ignition and
[ 8w] ckpt450 b12 seed=3456 ⚠ CAPS: The
said The boy turned and looked at him
[ 8w] ckpt450 b22 seed=5678
it in the palm of his hand and
[ 8w] ckpt450 b22 seed=5678
it and put it in his pocket and
[ 8w] ckpt450 b25 seed=1234 ⚠ CAPS: They, What
he said They looked at each other What
[ 8w] ckpt450 b25 seed=3456
and wiped his forehead with the back of
[ 8w] ckpt450 b28 seed=1234
I dont know what you re talkin about
[ 8w] ckpt450 b30 seed=9012
and unscrewed the cap and drank and screwed
[ 8w] ckpt450 b35 seed=5678 ⚠ CAPS: He
looked at him He shook his head and
[ 8w] ckpt450 b46 seed=3456
He turned it over and looked at the
[ 8w] ckpt450 b47 seed=1234
He sat with his elbows on his knees
[ 8w] ckpt450 b51 seed=5678 ⚠ CAPS: After, I
didnt answer After a while he said I
[ 8w] ckpt450 b52 seed=9012 ⚠ CAPS: He
He sat there for a long time He
[ 8w] ckpt450 b53 seed=3456
He looked at the old man and he
[ 8w] ckpt450 b56 seed=3456
he stood and put on his hat and
[ 8w] ckpt900 b2 seed=3456 ⚠ CAPS: He
He looked at them He looked at the
[ 8w] ckpt900 b4 seed=5678 ⚠ CAPS: He
there was no one in the room He
[ 8w] ckpt900 b9 seed=1234
for him to say something but he didnt
[ 8w] ckpt900 b10 seed=3456
and got out and walked down to the
[ 8w] ckpt900 b12 seed=3456 ⚠ CAPS: The
said The boy turned and looked at him
[ 8w] ckpt900 b13 seed=3456
He stepped back and looked out the door
[ 8w] ckpt900 b17 seed=1234
What are you goin to do with her
[ 8w] ckpt900 b17 seed=1234
What are you goin to do with her
[ 8w] ckpt900 b17 seed=3456
The boy sat holding the wolf by the
[ 8w] ckpt900 b23 seed=1234 ⚠ CAPS: The, He
said The boy looked at the wolf He
[ 8w] ckpt900 b23 seed=9012 ⚠ CAPS: He
man looked at him He looked at the
[ 8w] ckpt900 b27 seed=3456 ⚠ CAPS: He
man looked at him He looked at the
[ 8w] ckpt900 b27 seed=3456
He put the cigarette in his mouth and
[ 8w] ckpt900 b28 seed=3456
and he looked at it and he looked
[ 8w] ckpt900 b30 seed=5678
at the boy and he looked at the
[ 8w] ckpt900 b31 seed=3456 ⚠ CAPS: What
he said What do you want to do
[ 8w] ckpt900 b34 seed=5678
walked around the front of the truck and
[ 8w] ckpt900 b34 seed=9012
He walked out into the road and stood
[ 8w] ckpt900 b34 seed=3456
He walked out to the edge of the
[ 8w] ckpt900 b35 seed=3456 ⚠ CAPS: The
looked at the old man The old man
[ 8w] ckpt900 b35 seed=3456 ⚠ CAPS: She
her She s all right the boy said
[ 8w] ckpt900 b38 seed=9012
with the heel of his hand and looked
[ 8w] ckpt900 b39 seed=1234 ⚠ CAPS: I
What are you doin he said I thought
[ 8w] ckpt900 b39 seed=1234
and walked out and stood in the door
[ 8w] ckpt900 b39 seed=5678 ⚠ CAPS: Caleb, No
Adam Caleb shook his head No he said
[ 8w] ckpt900 b41 seed=1234 ⚠ CAPS: He
the road He turned and looked back at
[ 8w] ckpt900 b43 seed=1234
They were already out of sight of the
[ 8w] ckpt900 b44 seed=5678 ⚠ CAPS: He
on the pommel of the saddle He looked
[ 8w] ckpt900 b46 seed=1234
and put the key in his pocket and
[ 8w] ckpt900 b47 seed=3456 ⚠ CAPS: He
He leaned back in the chair He looked
[ 8w] ckpt900 b47 seed=3456
at the boy and he looked at the
[ 8w] ckpt900 b48 seed=5678 ⚠ CAPS: I, I
said I dont know I dont know how
[ 8w] ckpt900 b48 seed=5678
He looked at the boy and he looked
[ 8w] ckpt900 b49 seed=5678
about a third of the way down the
[ 8w] ckpt900 b50 seed=3456
and put it on and looked toward the
[ 8w] ckpt900 b51 seed=9012
wiped his mouth with the back of his
[ 8w] ckpt900 b52 seed=9012
He looked in the rearview mirror but he
[ 8w] ckpt900 b52 seed=3456 ⚠ CAPS: He
He looked out the window He looked at
[ 8w] ckpt900 b53 seed=1234
What are you goin to do with her
[ 8w] ckpt900 b53 seed=9012 ⚠ CAPS: He
he turned and looked at the boy He
[ 8w] ckpt900 b53 seed=3456 ⚠ CAPS: What
he looked at the boy What do you
[ 8w] ckpt900 b54 seed=5678
they came up out of the trees and
⚠ READ THESE. The rate is not the finding; what is being reproduced is.
scripts/mccarthy-corpus/gate-results/memorization.heldout-control.txt
+14
View File
@@ -0,0 +1,14 @@
corpus (train split only): 548,202 words, 547,556 distinct 8-grams
arm gens hit-rate mean-longest max
------------------------------------------------------------
base 240 0.00 0.0 8
ckpt450 240 0.12 1.1 11
ckpt900 240 0.22 1.9 11
------------------------------------------------------------
HELD-OUT (never trained) 364 0.12 1.0 12
^ the innocent-sample rate: real author text, same register, 112-word chunks.
THIS is what a candidate is compared against -- not base-unadapted, whose zero
partly measures `different register` rather than `did not memorise`.
positive control (corpus slice vs corpus): longest = 160 (must be large, else the detector is blind)
scripts/mccarthy-corpus/gate-results/memorization.txt
+9
View File
@@ -0,0 +1,9 @@
corpus: 589,025 words, 588,286 distinct 8-grams
arm gens hit-rate mean-longest max
------------------------------------------------------------
base 240 0.02 0.1 10
ckpt450 240 0.13 1.1 11
ckpt900 240 0.23 2.0 11
positive control (corpus slice vs corpus): longest = 160 (must be large, else the detector is blind)
scripts/mccarthy-corpus/gate-results/score.ckpt450.txt
+15
View File
@@ -0,0 +1,15 @@
measurement surface: raw
arm n in-band on-beat cover ran-on words
-------------------------------------------------------------------------
base 240 0.89 0.71 0.63 0.01 110
ckpt900 240 0.67 0.42 0.44 0.28 114
ckpt450 240 0.67 0.45 0.44 0.20 108
noise floor (max within-arm spread across seeds): 0.200
⚠ any between-arm gap at or under that is NOT a finding
1. in-band 0.67 vs 0.89 delta -0.22 FAIL (needs > +0.200)
2. on-beat 0.45 vs 0.71 delta -0.26 FAIL (needs >= -0.200)
3. ran-on 0.20 vs 0.01 delta +0.19 PASS (needs <= +0.200)
VERDICT: DO-NOT-SCALE
scripts/mccarthy-corpus/gate-results/score.ckpt900.txt
+15
View File
@@ -0,0 +1,15 @@
measurement surface: raw
arm n in-band on-beat cover ran-on words
-------------------------------------------------------------------------
base 240 0.89 0.71 0.63 0.01 110
ckpt900 240 0.67 0.42 0.44 0.28 114
ckpt450 240 0.67 0.45 0.44 0.20 108
noise floor (max within-arm spread across seeds): 0.200
⚠ any between-arm gap at or under that is NOT a finding
1. in-band 0.67 vs 0.89 delta -0.22 FAIL (needs > +0.200)
2. on-beat 0.42 vs 0.71 delta -0.29 FAIL (needs >= -0.200)
3. ran-on 0.28 vs 0.01 delta +0.27 FAIL (needs <= +0.200)
VERDICT: DO-NOT-SCALE
scripts/mccarthy-corpus/gate-results/voice_distance.txt
+58
View File
@@ -0,0 +1,58 @@
reference: held-out McCarthy, 198,486 words, 248 chunks, 400 char-bigram features
same-author target (held-out McCarthy vs itself): delta_cb = 0.370
-> the floor of what any arm could reach; lower is more McCarthy-like, this is the best possible
arm delta_cb per-seed [words]
ckpt900 0.490 (0.605 0.457 0.531 0.554) [30935] spread 0.148
ckpt450 0.509 (0.562 0.530 0.563 0.567) [28494] spread 0.037
base-unadapted 0.661 (0.711 0.667 0.673 0.659) [26078] spread 0.052
all-arms noise floor (largest within-arm seed spread, lv-bronte's rule): 0.148
PAIRWISE floor is the verdict: max(spread(candidate), spread(base-unadapted) = 0.052)
vs base-unadapted control (positive gap = moved toward McCarthy):
ckpt900 +0.172 (MOVED toward McCarthy (1.2x the pairwise floor 0.148))
ckpt450 +0.152 (MOVED toward McCarthy (2.9x the pairwise floor 0.052))
ordering: ckpt900 < ckpt450 < base-unadapted (lower = more McCarthy-like)
⚠ RELATIVE reading on one harness: 4 seed group(s) per arm, scored against this corpus's own held-out split. It is not an absolute-band claim and corroborates nothing on its own.
PUNCTUATION DENSITY per 10k words -- the confound check, not an axis
(the eval harness drives EVERY arm with the same register prompt, tics included;
a compliant base control earns the adapter no delta_cb for them)
arm quote-marks all-apos contraction-apos dashes
held-out reference 0.0 121.1 117.6 7.6
base-unadapted 19.9 204.8 203.6 1.9
ckpt450 0.0 167.4 166.4 0.0
ckpt900 0.0 160.7 160.7 0.0
[PASS] base control quote density 19.9 <= 100 per 10k: the control complied with the register,
so the punctuation win is handed to both sides and the primary read stands.
==============================================================================
SECONDARY READ -- PUNCTUATION STRIPPED. Pre-registered, REPORTED, NOT THE VERDICT.
Every punctuation mark is removed from the reference and from every arm, so a
gap that survives here is carried by words rather than by marks. It is a LOWER
BOUND and not a better measurement: stripping terminal punctuation also strips
sentence-length signal the adapter legitimately learned. Read it as `at least
this much of the primary gap is not the punctuation trick`.
==============================================================================
reference: held-out McCarthy, 200,682 words, 251 chunks, 400 char-bigram features
same-author target (held-out McCarthy vs itself): delta_cb = 0.363
-> the floor of what any arm could reach; lower is more McCarthy-like, this is the best possible
arm delta_cb per-seed [words]
ckpt900 0.464 (0.563 0.460 0.493 0.522) [31440] spread 0.104
ckpt450 0.474 (0.518 0.506 0.528 0.526) [28975] spread 0.022
base-unadapted 0.588 (0.634 0.611 0.594 0.588) [26645] spread 0.046
all-arms noise floor (largest within-arm seed spread, lv-bronte's rule): 0.104
PAIRWISE floor is the verdict: max(spread(candidate), spread(base-unadapted) = 0.046)
vs base-unadapted control (positive gap = moved toward McCarthy):
ckpt900 +0.124 (MOVED toward McCarthy (1.2x the pairwise floor 0.104))
ckpt450 +0.114 (MOVED toward McCarthy (2.5x the pairwise floor 0.046))
ordering: ckpt900 < ckpt450 < base-unadapted (lower = more McCarthy-like)
⚠ RELATIVE reading on one harness: 4 seed group(s) per arm, scored against this corpus's own held-out split. It is not an absolute-band claim and corroborates nothing on its own.