docs(lv-mccarthy): record the gate result -- voice passes, memorisation is the cleanest in the line, NOT shipped
720 generations, 3 arms x 60 held-out beats x 4 seeds, against the design frozen in GATE-PREREG.md before any arm existed. AXIS A VOICE -- PASS, both candidates, both reads. Span 0.661 -> 0.370 = 0.291 achievable; ckpt900 closed 59.1% (+0.172, but only 1.2x its floor), ckpt450 52.2% (+0.152 at 2.9x). The normalised secondary read HOLDS at +0.124 / +0.114, so about three quarters of the gain survives stripping every punctuation mark -- the voice is not the cheap win the register made available. AXIS B NOT COPIED -- ckpt450 is the cleanest result in the line. 0.12 hit-rate against the author's own held-out 0.12, and its longest match (11 words) is SHORTER than the author's coincidental longest (12). All 96 matched runs were read: stock grammar in the commonest words, the name-shaped hits are the RENAMED inventions, nothing protectable. The amendment is why this reads as clean -- the defective base control would have shown 0.12 vs 0.00 as a 12x red flag. Separately measured: the "his register makes collisions inevitable" story that was FALSE for Hemingway (0.01) is TRUE for McCarthy (0.12). Neither transfers. AXIS C NO DAMAGE -- FAIL, both, and it survives reading. 20% (ckpt450) / 28% (ckpt900) of generations overshoot the 90-140 band against base's 1%; p90 171/190 words, max 297/279. The worst case is degenerate looping, not a long McCarthy sentence. Base is GOOD on this axis here (0.89 in-band vs Hemingway's 0.05), so the adapter measurably makes instruction-following worse. NOT SHIPPED. Section 7 rule 3 makes axis C disqualifying outright. Recorded honestly: my own prereg's axis C transcribed score_beats.py's v1 criteria, including "in-band up on base", which the operator RETIRED on 2026-09-15 for exactly the reason it fails here -- base maxes it, so it is unsatisfiable on this corpus regardless of adapter quality. Under the operator's v2 (ran-on only) ckpt450 passes by 0.01 against a 0.200 floor. That reading was found AFTER the numbers and was NOT used; lv-bronte's floor defect is the in-line precedent for finding one and declining to exploit it. The prereg gets fixed prospectively for the next author, not re-read for this one. And the finding worth more than the adapter: the two-epoch recipe is now 0 for 3, and this time the loss curve was CONFIDENTLY wrong. On Bronte and Hemingway the epoch-1/epoch-2 checkpoints were tied, so preferring the earlier one cost nothing. Here the curve resolved epoch 2 as better at 4.9x the median neighbour jitter -- and epoch 2 lost every axis that resolves: 4.0x wider seed spread, 1.8x the author's memorisation rate against 1.0x, more ran-on, worse on-beat. Its only win is a 0.019 voice point estimate, inside the floor, and its spread is one outlier seed -- the third occurrence of that shape in the later checkpoint after lv-bronte's ckpt925 and lv-hemingway's ckpt1750. Raw artifacts committed at scripts/mccarthy-corpus/gate-results/ so the claims can be re-read without gx10.
This commit is contained in:
Vuong Hoang
@@ -0,0 +1,217 @@
|
||||
# `[2026-09-21]` lv-mccarthy: gate run, NOT SHIPPED — the voice works, the memorisation is the cleanest in the line, and the length discipline is gone
|
||||
|
||||
**Status: NOT SHIPPED.** Both gated candidates are disqualified by axis C under the
|
||||
pre-registration as frozen. The operator had pre-authorised "ship it if the gate passes";
|
||||
it did not pass, and their conditional preserved that branch.
|
||||
|
||||
Gate design pre-registered before any generation existed:
|
||||
`scripts/mccarthy-corpus/GATE-PREREG.md`, commit `9c8a4e9`, amended `b4ba731` and `0d80e49`.
|
||||
Raw artifacts committed at `scripts/mccarthy-corpus/gate-results/`; arms and generations at
|
||||
`gx10:~/r49-runs/mccarthy-eval/`.
|
||||
|
||||
## First: the run WAS complete, and had been for three days
|
||||
|
||||
`persistent-memory.md` carried "lv-mccarthy's run outcome STILL UNVERIFIED" as the top
|
||||
in-flight item for two days. It finished **2026-09-18 00:49 PT** — 1,380/1,380 steps in
|
||||
2h30m47s, `train_loss` 2.172. Three days of "unverified" was a reporting gap, not a failure.
|
||||
⚠ `~/r49-runs/` does not exist on nh3-dev; the runs are on **pfi-gx10 = 10.100.50.60**, which
|
||||
does not resolve by name from nh3-dev.
|
||||
|
||||
## The result — 3 arms × 60 held-out beats × 4 seeds = 720 generations
|
||||
|
||||
| axis | ckpt900 (ep 1.96, loss min) | ckpt450 (ep 0.98) | verdict |
|
||||
|---|---|---|---|
|
||||
| **A. VOICE** | +0.172, **1.2×** floor 0.148 | +0.152, **2.9×** floor 0.052 | ✅ both PASS, both reads |
|
||||
| **B. NOT COPIED** | 0.22 = 1.8× the author | **0.12 = 1.0× the author** | ✅ ckpt450 clean, ckpt900 elevated |
|
||||
| **C. NO DAMAGE** | fails all three criteria | fails two of three | ❌ **both FAIL** |
|
||||
|
||||
```
|
||||
same-author target (held-out McCarthy vs itself) delta_cb 0.370 <- best achievable
|
||||
ckpt900 0.490
|
||||
ckpt450 0.509
|
||||
base-unadapted 0.661
|
||||
```
|
||||
|
||||
Span 0.661 → 0.370 = 0.291. ckpt900 closed **59.1%**, ckpt450 **52.2%** — between lv-bronte
|
||||
(48%) and lv-hemingway (73.8%), on an axis deliberately made harder (§5a hands the register's
|
||||
punctuation tics to the control).
|
||||
|
||||
## ⭐⭐ AXIS B: the corrected control turned a 12× red flag into a clean pass
|
||||
|
||||
The pre-registration as first written inherited `memorization_check.py`'s **base-unadapted**
|
||||
negative control, which the lv-hemingway record had already established is defective. Caught
|
||||
while the base arm was still generating and amended append-only before any McCarthy number
|
||||
was read (AMENDMENT 1). The correct innocent sample is the author himself.
|
||||
|
||||
```
|
||||
sample n hit-rate mean-longest max
|
||||
HELD-OUT McCARTHY (untrained) 364 0.12 1.0 12 <- the innocent rate
|
||||
base-unadapted 240 0.00 0.0 8
|
||||
ckpt450 240 0.12 1.1 11
|
||||
ckpt900 240 0.22 1.9 11
|
||||
positive control (train vs train) 160 <- not blind
|
||||
```
|
||||
|
||||
⭐ **ckpt450 is INDISTINGUISHABLE from real unseen McCarthy** — 0.12 against 0.12, and its
|
||||
longest match (11 words) is **SHORTER than the author's own coincidental longest (12)**. That
|
||||
is the best axis-B result in the line. Against the defective base control its 0.12-vs-0.00
|
||||
would have read as a **12× red flag**; against the correct one it is **1.0×**. The amendment
|
||||
is the only reason this adapter is not on the record as memorising.
|
||||
|
||||
⭐ **And the "register makes collisions inevitable" story — FALSE for Hemingway — is TRUE
|
||||
here, measured rather than assumed.** Hemingway's held-out rate was 0.01; McCarthy's is 0.12.
|
||||
His idiom really does self-collide at 8 grams (`and he looked at the`), so the same argument
|
||||
that was a comfortable excuse there is a fact here. Authors are not interchangeable on this
|
||||
axis and neither number transfers.
|
||||
|
||||
**All 96 matched runs were READ** (`gate-results/memorisation-matches.txt`), not counted.
|
||||
Longest 11 words, every one stock grammar in the commonest words:
|
||||
|
||||
```
|
||||
he looked at the wolf and he looked at him
|
||||
leaned back in his chair and looked at the boy
|
||||
He reached into his pocket and took out the
|
||||
I dont know What are you goin to do
|
||||
```
|
||||
|
||||
The name-shaped hits (`Adam Caleb`, `Toribio`) are the **RENAMED invented names**, not
|
||||
McCarthy's — the `swift tristan` precedent exactly. No plot, no imagery, nothing protectable.
|
||||
⚠ McCarthy is in copyright with a living estate and this axis still comes out clean for
|
||||
ckpt450.
|
||||
|
||||
## ❌ AXIS C: the damage is real, and reading it confirms the criterion
|
||||
|
||||
```
|
||||
arm n in-band on-beat ran-on words p90 max over-140
|
||||
base 240 0.89 0.71 0.01 110 126 149 1%
|
||||
ckpt450 240 0.67 0.45 0.20 108 171 297 20%
|
||||
ckpt900 240 0.67 0.42 0.28 114 190 279 28%
|
||||
floor 0.200
|
||||
```
|
||||
|
||||
Not an artifact. **Base is GOOD here** (0.89 in-band against Hemingway's 0.05, because
|
||||
McCarthy's register prompt is far more prescriptive and Qwen3-4B-Instruct follows it), and the
|
||||
adapter measurably makes it worse — 22 points of in-band, 26–29 of on-beat, and 20–28× base's
|
||||
overshoot rate.
|
||||
|
||||
Read the worst case and it is **degenerate looping**, not a long McCarthy sentence:
|
||||
|
||||
> *…and then he looked at the wolf and he looked at the road and he looked at the sun and he
|
||||
> looked at the road again.* — ckpt900, b41, 279 words against a 90–140 ask
|
||||
|
||||
## ⚠ THE COMPLICATION, AND WHY IT DID NOT BECOME A SHIP
|
||||
|
||||
My pre-registration's §6 axis C transcribed `score_beats.py`'s **v1** three-part criterion,
|
||||
including "in-band up on base beyond the floor". The operator **retired** in-band and on-beat
|
||||
from the gate on 2026-09-15 for precisely the reason they fail here — the script's own
|
||||
docstring says *"NOT carried into v2: in-band (unresolvable — base maxes it) and on-beat."*
|
||||
The v2 axis C is **ran-on only**, and `VERDICT: DO-NOT-SCALE` in the output is the superseded
|
||||
v1 label the script still prints "for continuity".
|
||||
|
||||
```
|
||||
frozen prereg (v1 criteria) ckpt900 fails 3 of 3 ckpt450 fails 2 of 3
|
||||
operator's v2 (ran-on only) ckpt900 +0.27 FAILS ckpt450 +0.19 passes by 0.01
|
||||
```
|
||||
|
||||
So my own pre-registration contains a criterion **unsatisfiable on this corpus regardless of
|
||||
adapter quality**. That is a defect in the pre-registration, not in the adapter — and it is
|
||||
still not a reason to ship:
|
||||
|
||||
1. The frozen rule fails both candidates, and §7 rule 3 makes axis C disqualifying outright.
|
||||
2. The reading that rescues ckpt450 was found **after** seeing the numbers. That is the exact
|
||||
shape pre-registration exists to prevent, and lv-bronte's floor defect is the in-line
|
||||
precedent for finding one and deliberately not exploiting it.
|
||||
3. Even under that reading the margin is **0.01 against a floor of 0.200** — noise-adjacent.
|
||||
4. The damage survives reading, not just the criterion. 20% of ckpt450's generations blow the
|
||||
length budget and the worst ones loop.
|
||||
|
||||
**Fix the pre-registration prospectively for the next author, do not re-read it for this one.**
|
||||
|
||||
## ⭐⭐⭐ THE TWO-EPOCH RECIPE IS NOW 0 FOR 3 — AND THIS TIME THE LOSS CURVE WAS CONFIDENTLY WRONG
|
||||
|
||||
On Brontë and Hemingway the epoch-1 and epoch-2 checkpoints were **tied** on eval loss, so
|
||||
preferring the earlier one cost nothing and could be dismissed as taste. Here the curve
|
||||
**resolved** epoch 2 as better — ckpt900 at 2.38706 against ckpt450's 2.4063, **+4.9× the
|
||||
0.00393 median neighbour jitter**, nowhere near tied — and it was wrong on every axis that
|
||||
resolves:
|
||||
|
||||
| | ckpt450 | ckpt900 | ratio |
|
||||
|---|---|---|---|
|
||||
| seed spread (voice) | **0.037** | 0.148 | **4.0× wider** |
|
||||
| memorisation vs the author's 0.12 | **0.12 (1.0×)** | 0.22 (1.8×) | |
|
||||
| ran-on | **0.20** | 0.28 | |
|
||||
| on-beat | **0.45** | 0.42 | |
|
||||
| epochs of overfit | **0.98** | 1.96 | |
|
||||
|
||||
ckpt900's only advantage is a 0.019 better voice point estimate, which sits inside the floor.
|
||||
And its spread is **one outlier seed** — 0.605 against 0.457 / 0.531 / 0.554 — the third
|
||||
occurrence of that shape in the later checkpoint, after lv-bronte's ckpt925 and
|
||||
lv-hemingway's ckpt1750 (0.491 / 0.449 / 0.468 / **0.562**).
|
||||
|
||||
⭐ **The durable rule: on this schedule the eval-loss minimum is not the ship candidate, and
|
||||
the loss curve's CONFIDENCE about it carries no information.** Read the resolving axes. This
|
||||
is now measured on three corpora and should be the default for Faulkner, Morrison and Chandler
|
||||
rather than re-derived each time.
|
||||
|
||||
## ⚠ A bug in my own confound detector, and the order I found it in
|
||||
|
||||
§5c pre-registered a trigger: base quote density over **100 per 10k** means the control did
|
||||
not take the punctuation win it was handed, and the normalised secondary read is promoted to
|
||||
load-bearing. The base arm finished first, so I evaluated it early, **saw it FIRE at 224.7**,
|
||||
and only then — reading the reference row against a corpus whose builder ASSERTS 0.0 — found
|
||||
that `_QUOTE_RE` contained `'` and `’`. It was an apostrophe counter wearing a quote-mark
|
||||
label, on the one corpus whose signature is `dont`/`aint`/`wont`.
|
||||
|
||||
```
|
||||
as implemented TRUE quotes all apostrophes
|
||||
held-out McCarthy ref 121.1 0.0 121.1
|
||||
base-unadapted 224.7 19.9 204.8
|
||||
held-out Hemingway ref 1112.6 694.7 351.7
|
||||
```
|
||||
|
||||
Fixing a detector to measure the quantity the frozen rule names is not moving the rule — but
|
||||
the fix un-fires the trigger, which is indistinguishable from shopping. **So the trigger was
|
||||
made MOOT rather than adjudicated** (AMENDMENT 2): the normalised read is load-bearing
|
||||
unconditionally for this gate, both columns reported, and the fix has zero verdict effect.
|
||||
There is a better reason anyway — base's true density is 19.9 against the reference's 0.0, so
|
||||
it did not fully comply and a small residual cheap win genuinely exists.
|
||||
|
||||
⭐ The normalised read **HOLDS**: ckpt900 +0.124 at 1.2× and ckpt450 +0.114 at 2.5×, against
|
||||
primaries of +0.172 and +0.152. So roughly three quarters of the voice gain survives having
|
||||
every punctuation mark removed — **the voice is not the punctuation trick.** Validated on
|
||||
Hemingway first, where the secondary read also resolves a gap rather than flattening
|
||||
everything, so a null here would have been a finding rather than a blind instrument.
|
||||
|
||||
⚠ **The lesson is the one this line keeps relearning somewhere new:** I controlled
|
||||
`strip_punct` (2500 → 0) and the byte-identity of the default path, and never asked the quote
|
||||
counter for a value whose answer I already knew. The corpus asserts 0.0. That check cost one
|
||||
line and was available before the gate launched.
|
||||
|
||||
## Instruments built, each validated against a documented finding
|
||||
|
||||
| instrument | control |
|
||||
|---|---|
|
||||
| `voice_distance.py --secondary-normalised --punct-report` | default path reproduces the shipped lv-hemingway `voice_distance.txt` **byte for byte** |
|
||||
| `memorization_check.py --train-only --heldout-reference` | reproduces lv-hemingway's hand-computed held-out row **to the digit** (370 samples, 0.01, 0.1, max 10, 101-word chunks) |
|
||||
| `show_memorisation_matches.py` (new) | reproduces the lv-hemingway reading **to the word**, including `swift tristan` as the one name-shaped hit |
|
||||
| `ship-voice-adapter.sh` (new) | the live lv-hemingway adapter verified byte-identical to gx10's checkpoint-850 |
|
||||
|
||||
⚠ The held-out control and the match reading were both done **by hand** for Hemingway and left
|
||||
no instrument, so the finding was not reproducible. Both are now committed.
|
||||
|
||||
⚠ **The waiter I first armed was silently broken**: `pgrep -f "eval-mccarthy.sh"` over ssh
|
||||
self-matches its own `bash -c` argv, so the DIED branch was unreachable and a crashed gate
|
||||
would have looked exactly like a running one. Controlled both ways after the fix — bare
|
||||
pattern demonstrably self-matches, bracketed one does not.
|
||||
→ [[feedback_pkill_ssh_self_match]]
|
||||
|
||||
## What to do next — ckpt450 is the candidate, and an earlier one may beat it
|
||||
|
||||
The damage grows monotonically with epoch: base 1% over-band → ckpt450 20% → ckpt900 28%.
|
||||
`--save-total-limit 60` kept **all 56 checkpoints at 25-step intervals**, so ckpt225 / ckpt300
|
||||
(epoch ~0.49 / 0.65) are on disk and untested. Voice at 0.5 epoch is unknown and may not have
|
||||
arrived; that is one arm each to find out. This is a salvageable adapter, not a failed one —
|
||||
the voice transferred and the memorisation is the cleanest in the line.
|
||||
|
||||
Related: [[2026-09-17-lv-hemingway-gate]], [[2026-09-17-lv-bronte-gate]],
|
||||
[[2026-09-17-mccarthy-d1-d3]], [[2026-09-17-mccarthy-split-name-leak]].
|
||||
+34
-11
@@ -1,6 +1,6 @@
|
||||
# Persistent memory — eshpfi-management
|
||||
|
||||
_Last updated: 2026-09-21 ~14:30 PT (⭐ **the ops log is BUILT** and then failed four ways in its first hours — every one recording something unfindable; the day's subject was instruments that report without looking. ⭐ Draupnir engine COMPLETE and acceptance-tested on irv-ml1. ⭐ Booth gained blur + a closed keep round trip after shipping two dead controls. ⭐ claude-bot is an org Owner; cicada+draupnir moved to `pfi`; vh-token use standing-authorized from the vault. ⭐ nh3-dev 84%→72%, and a LoRA adapter rescued from a 3-day-swept /tmp. ⚠ lv-mccarthy's run outcome STILL UNVERIFIED after two days.)_
|
||||
_Last updated: 2026-09-21 ~16:45 PT (⭐ **the ops log is BUILT** and then failed four ways in its first hours — every one recording something unfindable; the day's subject was instruments that report without looking. ⭐ Draupnir engine COMPLETE and acceptance-tested on irv-ml1. ⭐ Booth gained blur + a closed keep round trip after shipping two dead controls. ⭐ claude-bot is an org Owner; cicada+draupnir moved to `pfi`; vh-token use standing-authorized from the vault. ⭐ nh3-dev 84%→72%, and a LoRA adapter rescued from a 3-day-swept /tmp. ⭐⭐ lv-mccarthy GATED: voice passes and its memorisation is the cleanest in the line, but 20-28% of generations blow the length band — NOT shipped, and the two-epoch recipe is now 0 for 3 with the loss curve confidently wrong.)_
|
||||
|
||||
> **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its
|
||||
> `Written:` stamp is under **8 hours** old, read it (it carries the in-flight
|
||||
@@ -115,18 +115,36 @@ no longer deployed sidecars here. See Recent decisions.)
|
||||
|
||||
## Current state / in-flight
|
||||
|
||||
_As of 2026-09-21 ~14:30 PT._
|
||||
_As of 2026-09-21 ~16:40 PT._
|
||||
|
||||
### ⚠ FIRST — lv-mccarthy's run outcome is STILL UNVERIFIED
|
||||
### ⚠ FIRST — lv-mccarthy is GATED and NOT SHIPPED; the decision is the operator's
|
||||
|
||||
Carried unchanged from the 2026-09-19 handoff and untouched for two full days.
|
||||
Look at `~/r49-runs/mccarthy-4b-pairs-3ep/` before anything else. ⚠ `pfi-gx10`
|
||||
does not resolve from nh3-dev by that name. Assume nothing — it was never
|
||||
checked, not checked-and-found-good. Then: checkpoint selection off the loss
|
||||
curve → the v2 gate (`eval-*.sh`) → ship-or-park. ⚠ Before the gate, settle how
|
||||
the VOICE axis is read: D1 pre-registered a punctuation-normalised secondary
|
||||
read, and `--system-from` hands the register's tics to the BASE arm too, which
|
||||
changes what the primary number means.
|
||||
The run was complete all along (2026-09-18 00:49 PT) — three days of "unverified"
|
||||
was a reporting gap, not a failure. The v2 gate has now RUN, 720 generations, and
|
||||
both candidates are **disqualified by axis C**. Full record:
|
||||
`persistent-memory.d/2026-09-21-lv-mccarthy-gate.md`. ⚠ `pfi-gx10` does not
|
||||
resolve from nh3-dev by name — it is **10.100.50.60**.
|
||||
|
||||
**The shape of it:** voice PASSES on both candidates and both reads (ckpt450 at
|
||||
2.9× its floor, and ~3/4 of the gain survives having every punctuation mark
|
||||
stripped, so it is not the cheap win). Memorisation on **ckpt450 is the cleanest
|
||||
result in the line** — 0.12 against the author's own held-out 0.12, longest match
|
||||
11 words against the author's coincidental 12, all 96 matches read and every one
|
||||
stock grammar. **Damage is the blocker**: 20% (ckpt450) / 28% (ckpt900) of
|
||||
generations overshoot the 90–140 band against base's 1%, worst case a degenerate
|
||||
`he looked at X and he looked at Y` loop at 279 words.
|
||||
|
||||
**Two things a next session must not re-litigate:**
|
||||
1. **My own pre-registration's axis C is defective** — it transcribed
|
||||
`score_beats.py`'s v1 criteria including "in-band up on base", which the
|
||||
operator RETIRED on 2026-09-15 because base maxes it. Under the operator's v2
|
||||
(ran-on only) ckpt450 passes by **0.01 against a 0.200 floor**. That reading
|
||||
was found AFTER the numbers, so it was not used. **Fix the prereg
|
||||
prospectively for Faulkner; do not re-read it for McCarthy.**
|
||||
2. **ckpt450 is the candidate, not ckpt900** (the loss minimum). It wins every
|
||||
resolving axis. The damage grows monotonically with epoch, and all 56
|
||||
checkpoints are on disk — **ckpt225/ckpt300 (epoch ~0.5–0.65) are untested**
|
||||
and are the obvious next probe, one arm each.
|
||||
|
||||
### Draupnir engine — COMPLETE on irv-ml1, acceptance passing
|
||||
|
||||
@@ -160,6 +178,11 @@ the althing thread.
|
||||
|
||||
## Recent decisions
|
||||
|
||||
- `[2026-09-21]` ⭐⭐⭐ **lv-mccarthy GATED and NOT SHIPPED — voice passes, memorisation is the cleanest in the line, and the length discipline is gone.** 720 generations, 3 arms. Voice +0.152 at **2.9×** floor for ckpt450 (ckpt900 +0.172 at only 1.2×, its spread one outlier seed), and **~3/4 of the gain survives stripping every punctuation mark**, so it is not the cheap win. ⭐ Memorisation: **ckpt450 at 0.12 against the author's own held-out 0.12 — identical, longest match 11 words against the author's coincidental 12**, all 96 matches READ and every one stock grammar (`he looked at the wolf and he looked at him`); the name-shaped hits are the RENAMED inventions. Axis C is the blocker: **20% / 28% of generations overshoot the 90–140 band against base's 1%**, worst case a degenerate loop at 279 words. ⚠ My own prereg's axis C transcribed `score_beats.py`'s **v1** criteria including "in-band up on base", which the operator RETIRED 2026-09-15 because base maxes it — under the v2 (ran-on only) ckpt450 passes by **0.01 against a 0.200 floor**, but that reading was found AFTER the numbers and was not used. Fix the prereg prospectively. → `persistent-memory.d/2026-09-21-lv-mccarthy-gate.md`
|
||||
- `[2026-09-21]` ⭐⭐⭐ **The two-epoch recipe is now 0 for 3, and this time the loss curve was CONFIDENTLY wrong.** On Brontë and Hemingway the epoch-1/epoch-2 checkpoints were TIED on eval loss, so preferring the earlier one cost nothing. Here the curve RESOLVED epoch 2 as better — ckpt900 +4.9× the 0.00393 median neighbour jitter above ckpt450, nowhere near tied — and epoch 2 lost every axis that resolves: **4.0× wider seed spread** (0.148 vs 0.037), **1.8× the author's memorisation rate vs 1.0×**, more ran-on (0.28 vs 0.20), worse on-beat. Its only win is a 0.019 voice point estimate, inside the floor, and its spread is ONE outlier seed (0.605 vs 0.457/0.531/0.554) — the third occurrence of that shape in the later checkpoint after lv-bronte's ckpt925 and lv-hemingway's ckpt1750. **Durable: on this schedule the eval-loss minimum is not the ship candidate, and the curve's CONFIDENCE about it carries no information.** Default this for Faulkner/Morrison/Chandler rather than re-deriving it.
|
||||
- `[2026-09-21]` ⭐⭐ **The memorisation control that shipped broken is now committed, and it turned a 12× red flag into a clean pass.** `memorization_check.py` used **base-unadapted** as its negative control; base writes summary while the adapted arms write pastiche, so it cannot collide with a register it does not imitate and its zero is unearned. The lv-hemingway gate established this, computed the correct control (the author's own held-out text) **by hand**, and left no instrument — so the finding was not reproducible. Now `--train-only --heldout-reference`, refusing the unsafe combination, validated by reproducing Hemingway's hand-computed row **to the digit**. ⭐ On McCarthy it is the difference between reporting ckpt450 as memorising (0.12 vs base 0.00 = 12×) and clean (0.12 vs the author's 0.12 = 1.0×). ⭐ And the "the register makes collisions inevitable" story that was FALSE for Hemingway (0.01) is TRUE for McCarthy (0.12) — measured, not assumed; neither number transfers between authors. Also new: `show_memorisation_matches.py`, because rate and exposure are different questions and the reading was hand-done too. Commits `b4ba731` `7eadbd6`.
|
||||
- `[2026-09-21]` ⚠⚠ **My confound detector was counting apostrophes as quote marks, and I saw it FIRE before I saw the bug.** `voice_distance.py`'s quote class shipped with `'` and `’` in it — on the one corpus whose signature is `dont`/`aint`/`wont`. It reported the held-out reference at 121.1 "quote marks" per 10k for a corpus whose builder **ASSERTS 0.0**, and fired the pre-registered trigger on a base arm whose true density is 19.9. Fixing a detector to measure the quantity the frozen rule names is not moving the rule, but the fix un-fires the trigger, which is indistinguishable from shopping — **so the trigger was made MOOT instead of adjudicated**: the normalised read is load-bearing unconditionally, both columns reported, zero verdict effect. ⚠ The lesson: I controlled `strip_punct` (2500→0) and the byte-identity of the default path, and never asked the quote counter for a value whose answer I already knew. Commit `0d80e49`.
|
||||
|
||||
- `[2026-09-21]` ⭐⭐⭐ **The ops log shipped — and then failed FOUR ways in its first hours, every one recording something unfindable.** Claim dropped by a sub-tool, hook behind graphify's eight `exit 0`s, no handle in the env, ssh-target written as a hostname. The general shape is **configured ≠ effective**; twelve instruments reported confidently and wrongly across three days, five of them mine. → `persistent-memory.d/2026-09-21-ops-log-and-the-instruments-that-lied.md`
|
||||
|
||||
- `[2026-09-21]` ⭐⭐ **The Booth gained blur + a closed keep round trip, after shipping TWO controls that did nothing** — a reveal handler Jinja discarded for sitting after `{% endblock %}`, and a `×` a sibling form covered by 30×22 px. Both operator-found, both from reading templates instead of rendering them. `scripts/layout-probe.py` took four iterations to become trustworthy. ⚠ Blur is COSMETIC and a test asserts the 200 on purpose. → `persistent-memory.d/2026-09-21-booth-two-dead-controls.md`
|
||||
|
||||
@@ -0,0 +1,199 @@
|
||||
corpus (train split only): 548,202 words, 547,556 distinct 8-grams
|
||||
[PASS] positive control: a corpus slice matches itself
|
||||
|
||||
96 matched run(s), longest 11 words. 42 carry a mid-run capital.
|
||||
|
||||
[ 11w] ckpt900 b31 seed=9012 ⚠ CAPS: Adam, Caleb, What
|
||||
Toribio looked at Adam Caleb What do you want to do
|
||||
[ 11w] ckpt900 b41 seed=5678 ⚠ CAPS: The
|
||||
He looked at the old man The old man was watching
|
||||
[ 10w] ckpt450 b9 seed=3456
|
||||
leaned back in his chair and looked at the boy
|
||||
[ 10w] ckpt450 b21 seed=3456
|
||||
he sat with his elbows on his knees and looked
|
||||
[ 10w] ckpt450 b29 seed=1234
|
||||
he looked at the wolf and he looked at him
|
||||
[ 10w] ckpt900 b9 seed=3456
|
||||
The old man drank the last of his coffee and
|
||||
[ 10w] ckpt900 b21 seed=5678
|
||||
sat leaning with his elbows on the table and his
|
||||
[ 10w] ckpt900 b37 seed=1234 ⚠ CAPS: Toribio, I, Adam, Caleb, I
|
||||
said Toribio I dont know said Adam Caleb I aint
|
||||
[ 10w] ckpt900 b41 seed=3456
|
||||
he looked at the wolf and he looked at the
|
||||
[ 10w] ckpt900 b46 seed=9012
|
||||
them in his hand and looked at them and then
|
||||
[ 10w] ckpt900 b49 seed=3456 ⚠ CAPS: Adam, Caleb, Toribio
|
||||
He looked at Adam Caleb and he looked at Toribio
|
||||
[ 9w] ckpt450 b7 seed=1234 ⚠ CAPS: What
|
||||
I dont know What are you goin to do
|
||||
[ 9w] ckpt450 b7 seed=9012
|
||||
He reached into his pocket and took out the
|
||||
[ 9w] ckpt450 b11 seed=1234 ⚠ CAPS: What
|
||||
I dont know What are you goin to do
|
||||
[ 9w] ckpt450 b22 seed=3456
|
||||
He got up and went into the bathroom and
|
||||
[ 9w] ckpt450 b30 seed=5678 ⚠ CAPS: He
|
||||
He turned and looked at the boy He was
|
||||
[ 9w] ckpt450 b35 seed=1234 ⚠ CAPS: What
|
||||
I dont know What are you goin to do
|
||||
[ 9w] ckpt450 b47 seed=5678 ⚠ CAPS: The
|
||||
He looked at the old man The old man
|
||||
[ 9w] ckpt450 b47 seed=5678 ⚠ CAPS: The
|
||||
He looked at the old man The old man
|
||||
[ 9w] ckpt450 b48 seed=1234 ⚠ CAPS: What
|
||||
I dont know What are you goin to do
|
||||
[ 9w] ckpt450 b49 seed=3456
|
||||
he was sitting with his back to the wall
|
||||
[ 9w] ckpt450 b53 seed=3456
|
||||
looked at the wolf and he looked at the
|
||||
[ 9w] ckpt450 b55 seed=3456
|
||||
and took out a cigarette and lit it and
|
||||
[ 9w] ckpt900 b4 seed=1234
|
||||
in one hand and the shotgun in the other
|
||||
[ 9w] ckpt900 b6 seed=9012
|
||||
and held it out and the boy took it
|
||||
[ 9w] ckpt900 b15 seed=5678
|
||||
on the far side of the road and he
|
||||
[ 9w] ckpt900 b26 seed=5678
|
||||
and he stood with his hat in his hand
|
||||
[ 9w] ckpt900 b28 seed=3456 ⚠ CAPS: I
|
||||
I dont know what you re talkin about I
|
||||
[ 9w] ckpt900 b31 seed=5678
|
||||
The sun had set and it was growing dark
|
||||
[ 9w] ckpt900 b40 seed=1234 ⚠ CAPS: What
|
||||
I dont know What are you goin to do
|
||||
[ 9w] ckpt900 b42 seed=1234 ⚠ CAPS: The
|
||||
He looked at the boy The boy was sitting
|
||||
[ 9w] ckpt900 b47 seed=3456 ⚠ CAPS: He
|
||||
man looked at him He looked at the horse
|
||||
[ 9w] ckpt900 b48 seed=5678 ⚠ CAPS: I
|
||||
I dont know he said I dont know how
|
||||
[ 9w] ckpt900 b51 seed=9012 ⚠ CAPS: I, I
|
||||
shook his head I dont know he said I
|
||||
[ 9w] ckpt900 b59 seed=9012 ⚠ CAPS: He
|
||||
The man looked at him He looked at the
|
||||
[ 8w] base b53 seed=1234 ⚠ CAPS: The
|
||||
looked at the old man The old man
|
||||
[ 8w] ckpt450 b2 seed=9012
|
||||
He looked at the man and then he
|
||||
[ 8w] ckpt450 b6 seed=3456
|
||||
He looked at the boy and he looked
|
||||
[ 8w] ckpt450 b10 seed=5678
|
||||
and put the key in the door and
|
||||
[ 8w] ckpt450 b10 seed=9012
|
||||
He put the key in the ignition and
|
||||
[ 8w] ckpt450 b12 seed=3456 ⚠ CAPS: The
|
||||
said The boy turned and looked at him
|
||||
[ 8w] ckpt450 b22 seed=5678
|
||||
it in the palm of his hand and
|
||||
[ 8w] ckpt450 b22 seed=5678
|
||||
it and put it in his pocket and
|
||||
[ 8w] ckpt450 b25 seed=1234 ⚠ CAPS: They, What
|
||||
he said They looked at each other What
|
||||
[ 8w] ckpt450 b25 seed=3456
|
||||
and wiped his forehead with the back of
|
||||
[ 8w] ckpt450 b28 seed=1234
|
||||
I dont know what you re talkin about
|
||||
[ 8w] ckpt450 b30 seed=9012
|
||||
and unscrewed the cap and drank and screwed
|
||||
[ 8w] ckpt450 b35 seed=5678 ⚠ CAPS: He
|
||||
looked at him He shook his head and
|
||||
[ 8w] ckpt450 b46 seed=3456
|
||||
He turned it over and looked at the
|
||||
[ 8w] ckpt450 b47 seed=1234
|
||||
He sat with his elbows on his knees
|
||||
[ 8w] ckpt450 b51 seed=5678 ⚠ CAPS: After, I
|
||||
didnt answer After a while he said I
|
||||
[ 8w] ckpt450 b52 seed=9012 ⚠ CAPS: He
|
||||
He sat there for a long time He
|
||||
[ 8w] ckpt450 b53 seed=3456
|
||||
He looked at the old man and he
|
||||
[ 8w] ckpt450 b56 seed=3456
|
||||
he stood and put on his hat and
|
||||
[ 8w] ckpt900 b2 seed=3456 ⚠ CAPS: He
|
||||
He looked at them He looked at the
|
||||
[ 8w] ckpt900 b4 seed=5678 ⚠ CAPS: He
|
||||
there was no one in the room He
|
||||
[ 8w] ckpt900 b9 seed=1234
|
||||
for him to say something but he didnt
|
||||
[ 8w] ckpt900 b10 seed=3456
|
||||
and got out and walked down to the
|
||||
[ 8w] ckpt900 b12 seed=3456 ⚠ CAPS: The
|
||||
said The boy turned and looked at him
|
||||
[ 8w] ckpt900 b13 seed=3456
|
||||
He stepped back and looked out the door
|
||||
[ 8w] ckpt900 b17 seed=1234
|
||||
What are you goin to do with her
|
||||
[ 8w] ckpt900 b17 seed=1234
|
||||
What are you goin to do with her
|
||||
[ 8w] ckpt900 b17 seed=3456
|
||||
The boy sat holding the wolf by the
|
||||
[ 8w] ckpt900 b23 seed=1234 ⚠ CAPS: The, He
|
||||
said The boy looked at the wolf He
|
||||
[ 8w] ckpt900 b23 seed=9012 ⚠ CAPS: He
|
||||
man looked at him He looked at the
|
||||
[ 8w] ckpt900 b27 seed=3456 ⚠ CAPS: He
|
||||
man looked at him He looked at the
|
||||
[ 8w] ckpt900 b27 seed=3456
|
||||
He put the cigarette in his mouth and
|
||||
[ 8w] ckpt900 b28 seed=3456
|
||||
and he looked at it and he looked
|
||||
[ 8w] ckpt900 b30 seed=5678
|
||||
at the boy and he looked at the
|
||||
[ 8w] ckpt900 b31 seed=3456 ⚠ CAPS: What
|
||||
he said What do you want to do
|
||||
[ 8w] ckpt900 b34 seed=5678
|
||||
walked around the front of the truck and
|
||||
[ 8w] ckpt900 b34 seed=9012
|
||||
He walked out into the road and stood
|
||||
[ 8w] ckpt900 b34 seed=3456
|
||||
He walked out to the edge of the
|
||||
[ 8w] ckpt900 b35 seed=3456 ⚠ CAPS: The
|
||||
looked at the old man The old man
|
||||
[ 8w] ckpt900 b35 seed=3456 ⚠ CAPS: She
|
||||
her She s all right the boy said
|
||||
[ 8w] ckpt900 b38 seed=9012
|
||||
with the heel of his hand and looked
|
||||
[ 8w] ckpt900 b39 seed=1234 ⚠ CAPS: I
|
||||
What are you doin he said I thought
|
||||
[ 8w] ckpt900 b39 seed=1234
|
||||
and walked out and stood in the door
|
||||
[ 8w] ckpt900 b39 seed=5678 ⚠ CAPS: Caleb, No
|
||||
Adam Caleb shook his head No he said
|
||||
[ 8w] ckpt900 b41 seed=1234 ⚠ CAPS: He
|
||||
the road He turned and looked back at
|
||||
[ 8w] ckpt900 b43 seed=1234
|
||||
They were already out of sight of the
|
||||
[ 8w] ckpt900 b44 seed=5678 ⚠ CAPS: He
|
||||
on the pommel of the saddle He looked
|
||||
[ 8w] ckpt900 b46 seed=1234
|
||||
and put the key in his pocket and
|
||||
[ 8w] ckpt900 b47 seed=3456 ⚠ CAPS: He
|
||||
He leaned back in the chair He looked
|
||||
[ 8w] ckpt900 b47 seed=3456
|
||||
at the boy and he looked at the
|
||||
[ 8w] ckpt900 b48 seed=5678 ⚠ CAPS: I, I
|
||||
said I dont know I dont know how
|
||||
[ 8w] ckpt900 b48 seed=5678
|
||||
He looked at the boy and he looked
|
||||
[ 8w] ckpt900 b49 seed=5678
|
||||
about a third of the way down the
|
||||
[ 8w] ckpt900 b50 seed=3456
|
||||
and put it on and looked toward the
|
||||
[ 8w] ckpt900 b51 seed=9012
|
||||
wiped his mouth with the back of his
|
||||
[ 8w] ckpt900 b52 seed=9012
|
||||
He looked in the rearview mirror but he
|
||||
[ 8w] ckpt900 b52 seed=3456 ⚠ CAPS: He
|
||||
He looked out the window He looked at
|
||||
[ 8w] ckpt900 b53 seed=1234
|
||||
What are you goin to do with her
|
||||
[ 8w] ckpt900 b53 seed=9012 ⚠ CAPS: He
|
||||
he turned and looked at the boy He
|
||||
[ 8w] ckpt900 b53 seed=3456 ⚠ CAPS: What
|
||||
he looked at the boy What do you
|
||||
[ 8w] ckpt900 b54 seed=5678
|
||||
they came up out of the trees and
|
||||
|
||||
⚠ READ THESE. The rate is not the finding; what is being reproduced is.
|
||||
@@ -0,0 +1,14 @@
|
||||
corpus (train split only): 548,202 words, 547,556 distinct 8-grams
|
||||
|
||||
arm gens hit-rate mean-longest max
|
||||
------------------------------------------------------------
|
||||
base 240 0.00 0.0 8
|
||||
ckpt450 240 0.12 1.1 11
|
||||
ckpt900 240 0.22 1.9 11
|
||||
------------------------------------------------------------
|
||||
HELD-OUT (never trained) 364 0.12 1.0 12
|
||||
^ the innocent-sample rate: real author text, same register, 112-word chunks.
|
||||
THIS is what a candidate is compared against -- not base-unadapted, whose zero
|
||||
partly measures `different register` rather than `did not memorise`.
|
||||
|
||||
positive control (corpus slice vs corpus): longest = 160 (must be large, else the detector is blind)
|
||||
@@ -0,0 +1,9 @@
|
||||
corpus: 589,025 words, 588,286 distinct 8-grams
|
||||
|
||||
arm gens hit-rate mean-longest max
|
||||
------------------------------------------------------------
|
||||
base 240 0.02 0.1 10
|
||||
ckpt450 240 0.13 1.1 11
|
||||
ckpt900 240 0.23 2.0 11
|
||||
|
||||
positive control (corpus slice vs corpus): longest = 160 (must be large, else the detector is blind)
|
||||
@@ -0,0 +1,15 @@
|
||||
measurement surface: raw
|
||||
arm n in-band on-beat cover ran-on words
|
||||
-------------------------------------------------------------------------
|
||||
base 240 0.89 0.71 0.63 0.01 110
|
||||
ckpt900 240 0.67 0.42 0.44 0.28 114
|
||||
ckpt450 240 0.67 0.45 0.44 0.20 108
|
||||
|
||||
noise floor (max within-arm spread across seeds): 0.200
|
||||
⚠ any between-arm gap at or under that is NOT a finding
|
||||
|
||||
1. in-band 0.67 vs 0.89 delta -0.22 FAIL (needs > +0.200)
|
||||
2. on-beat 0.45 vs 0.71 delta -0.26 FAIL (needs >= -0.200)
|
||||
3. ran-on 0.20 vs 0.01 delta +0.19 PASS (needs <= +0.200)
|
||||
|
||||
VERDICT: DO-NOT-SCALE
|
||||
@@ -0,0 +1,15 @@
|
||||
measurement surface: raw
|
||||
arm n in-band on-beat cover ran-on words
|
||||
-------------------------------------------------------------------------
|
||||
base 240 0.89 0.71 0.63 0.01 110
|
||||
ckpt900 240 0.67 0.42 0.44 0.28 114
|
||||
ckpt450 240 0.67 0.45 0.44 0.20 108
|
||||
|
||||
noise floor (max within-arm spread across seeds): 0.200
|
||||
⚠ any between-arm gap at or under that is NOT a finding
|
||||
|
||||
1. in-band 0.67 vs 0.89 delta -0.22 FAIL (needs > +0.200)
|
||||
2. on-beat 0.42 vs 0.71 delta -0.29 FAIL (needs >= -0.200)
|
||||
3. ran-on 0.28 vs 0.01 delta +0.27 FAIL (needs <= +0.200)
|
||||
|
||||
VERDICT: DO-NOT-SCALE
|
||||
@@ -0,0 +1,58 @@
|
||||
reference: held-out McCarthy, 198,486 words, 248 chunks, 400 char-bigram features
|
||||
same-author target (held-out McCarthy vs itself): delta_cb = 0.370
|
||||
-> the floor of what any arm could reach; lower is more McCarthy-like, this is the best possible
|
||||
|
||||
arm delta_cb per-seed [words]
|
||||
ckpt900 0.490 (0.605 0.457 0.531 0.554) [30935] spread 0.148
|
||||
ckpt450 0.509 (0.562 0.530 0.563 0.567) [28494] spread 0.037
|
||||
base-unadapted 0.661 (0.711 0.667 0.673 0.659) [26078] spread 0.052
|
||||
|
||||
all-arms noise floor (largest within-arm seed spread, lv-bronte's rule): 0.148
|
||||
PAIRWISE floor is the verdict: max(spread(candidate), spread(base-unadapted) = 0.052)
|
||||
|
||||
vs base-unadapted control (positive gap = moved toward McCarthy):
|
||||
ckpt900 +0.172 (MOVED toward McCarthy (1.2x the pairwise floor 0.148))
|
||||
ckpt450 +0.152 (MOVED toward McCarthy (2.9x the pairwise floor 0.052))
|
||||
|
||||
ordering: ckpt900 < ckpt450 < base-unadapted (lower = more McCarthy-like)
|
||||
⚠ RELATIVE reading on one harness: 4 seed group(s) per arm, scored against this corpus's own held-out split. It is not an absolute-band claim and corroborates nothing on its own.
|
||||
|
||||
PUNCTUATION DENSITY per 10k words -- the confound check, not an axis
|
||||
(the eval harness drives EVERY arm with the same register prompt, tics included;
|
||||
a compliant base control earns the adapter no delta_cb for them)
|
||||
arm quote-marks all-apos contraction-apos dashes
|
||||
held-out reference 0.0 121.1 117.6 7.6
|
||||
base-unadapted 19.9 204.8 203.6 1.9
|
||||
ckpt450 0.0 167.4 166.4 0.0
|
||||
ckpt900 0.0 160.7 160.7 0.0
|
||||
|
||||
[PASS] base control quote density 19.9 <= 100 per 10k: the control complied with the register,
|
||||
so the punctuation win is handed to both sides and the primary read stands.
|
||||
|
||||
==============================================================================
|
||||
SECONDARY READ -- PUNCTUATION STRIPPED. Pre-registered, REPORTED, NOT THE VERDICT.
|
||||
Every punctuation mark is removed from the reference and from every arm, so a
|
||||
gap that survives here is carried by words rather than by marks. It is a LOWER
|
||||
BOUND and not a better measurement: stripping terminal punctuation also strips
|
||||
sentence-length signal the adapter legitimately learned. Read it as `at least
|
||||
this much of the primary gap is not the punctuation trick`.
|
||||
==============================================================================
|
||||
|
||||
reference: held-out McCarthy, 200,682 words, 251 chunks, 400 char-bigram features
|
||||
same-author target (held-out McCarthy vs itself): delta_cb = 0.363
|
||||
-> the floor of what any arm could reach; lower is more McCarthy-like, this is the best possible
|
||||
|
||||
arm delta_cb per-seed [words]
|
||||
ckpt900 0.464 (0.563 0.460 0.493 0.522) [31440] spread 0.104
|
||||
ckpt450 0.474 (0.518 0.506 0.528 0.526) [28975] spread 0.022
|
||||
base-unadapted 0.588 (0.634 0.611 0.594 0.588) [26645] spread 0.046
|
||||
|
||||
all-arms noise floor (largest within-arm seed spread, lv-bronte's rule): 0.104
|
||||
PAIRWISE floor is the verdict: max(spread(candidate), spread(base-unadapted) = 0.046)
|
||||
|
||||
vs base-unadapted control (positive gap = moved toward McCarthy):
|
||||
ckpt900 +0.124 (MOVED toward McCarthy (1.2x the pairwise floor 0.104))
|
||||
ckpt450 +0.114 (MOVED toward McCarthy (2.5x the pairwise floor 0.046))
|
||||
|
||||
ordering: ckpt900 < ckpt450 < base-unadapted (lower = more McCarthy-like)
|
||||
⚠ RELATIVE reading on one harness: 4 seed group(s) per arm, scored against this corpus's own held-out split. It is not an absolute-band claim and corroborates nothing on its own.
|
||||
Reference in New Issue
Block a user