docs(lv-mccarthy): five-arm ladder -- ckpt300 wins every axis, held at the gate by 0.02

1,200 generations across five arms. ckpt300 (epoch 0.652) is the best arm in the
run on every axis that resolves:

  VOICE          +0.177 at 3.2x its pairwise floor (primary), +0.128 at 2.8x with
                 every punctuation mark stripped. Best point estimate AND best
                 margin of any arm, spread 0.055/0.038 with no outlier seed.
  MEMORISATION   0.12 against real unseen McCarthy's own 0.12 -- identical -- with
                 a longest match of 10 words against the author's coincidental 12.
                 All 31 matches read: stock grammar, names are the renamed
                 inventions, nothing protectable.
  DAMAGE         ran-on +0.12. Clears the operator's ratified v2 floor of 0.200 by
                 40%. FAILS AMENDMENT 3's self-imposed 0.100 bar by 0.02.

NOT SHIPPED, and the reason is the bar rather than the adapter. AMENDMENT 3 fixed
ran-on <= 0.100 before either new arm existed, precisely so a marginal number could
not be talked into a ship, and shipping at 0.12 would make that pre-registration
theatre. But the bar's stated rationale was written against ckpt450's pass by 0.01
-- 5% of the threshold -- and ckpt300 clears by 40%. The number excludes a candidate
the reasoning does not. That is an operator call.

ckpt325/350/375 are on disk and one may sit under 0.100. They were deliberately NOT
gated: searching the checkpoint space until something clears is candidate-shopping,
the same family as threshold-shopping approached from the other side.

THREE CLAIMS FROM EARLIER THIS SESSION ARE REFUTED and are corrected in the record:

  1. "The damage is flat across epochs and only rotates direction" -- FALSE. ran-on
     is non-monotonic (0.38 -> 0.13 -> 0.20 -> 0.28 across epochs 0.49/0.65/0.98/
     1.96) with a real minimum near 0.65, and ckpt225 is 48% out-of-band against
     ckpt300's 35%.
  2. "ckpt300 runs far too short, ckpt225 will clear ran-on by being short" -- FALSE
     on both. ckpt225 runs LONG (median 127, 38% over-band) and is the worst arm in
     the run. I generalised from SIX generations of one arm, which is the exact n=1
     violation the measurement-discipline rule names, committed in the same breath
     as a note about being careful.
  3. The original "gate an earlier checkpoint, the overshoot may not have arrived
     yet" recommendation was RIGHT. Retracting it an hour later on a three-arm read
     was the error, not the recommendation.

What is true and unresolved by any checkpoint choice: 35% of ckpt300's generations
miss the 90-140 band against base's 11%, and in-band is 0.65 against 0.89. An
adapter that buys a voice and costs a third of the length compliance is a trade, not
a defect -- but it is the operator's trade to accept.

Raw artifacts for all five arms at scripts/mccarthy-corpus/gate-results/.
This commit is contained in:
Vuong Hoang
2026-09-21 17:53:43 -07:00
parent 3de4d2dfaf
commit e8eb1594d9
13 changed files with 329 additions and 18 deletions
+8 -2
View File
@@ -62,7 +62,11 @@ else
log "fixture $BEATS already exists -- reusing"
fi
for arm in base:NONE ckpt900:$RUN/checkpoints/checkpoint-900 ckpt450:$RUN/checkpoints/checkpoint-450; do
# AMENDMENT 3 adds ckpt300 (ep 0.652) and ckpt225 (ep 0.489). Same fixture, same seeds,
# same rule -- adding candidates does not move a bar. Arms with generations already on
# disk are skipped, so re-running this script only fills in the new ones.
for arm in base:NONE ckpt900:$RUN/checkpoints/checkpoint-900 ckpt450:$RUN/checkpoints/checkpoint-450 \
ckpt300:$RUN/checkpoints/checkpoint-300 ckpt225:$RUN/checkpoints/checkpoint-225; do
name=${arm%%:*}; path=${arm#*:}
if [ "$name" != "base" ] && [ ! -d "$path" ]; then log "MISSING $path"; exit 1; fi
if [ -s "$OUT/beats5.$name.jsonl" ]; then
@@ -92,12 +96,14 @@ log "AXIS B -- MEMORISATION (corpus = the renamed copies the adapter trained on)
--eval-dir "$OUT" --corpus corpus-renamed/copies --glob 'beats5.*.jsonl' --strip 'beats5.' -n 8 \
2>&1 | tee "$OUT/memorization.txt"
for cand in ckpt900 ckpt450; do
for cand in ckpt900 ckpt450 ckpt300 ckpt225; do
log "AXIS C -- DAMAGE (ran-on / out-of-band), $cand vs base"
"$PY" scripts/yarros-corpus/score_beats.py \
--arm base="$OUT/beats5.base.jsonl" \
--arm ckpt900="$OUT/beats5.ckpt900.jsonl" \
--arm ckpt450="$OUT/beats5.ckpt450.jsonl" \
--arm ckpt300="$OUT/beats5.ckpt300.jsonl" \
--arm ckpt225="$OUT/beats5.ckpt225.jsonl" \
--baseline base --candidate "$cand" --metric-source raw \
--out "$OUT/score.$cand.json" 2>&1 | tee "$OUT/score.$cand.txt"
done