esh-pfi-infrastructure/scripts/hemingway-corpus/GATE-PREREG.md

lv-hemingway v2 gate — pre-registration

Written 2026-09-17, before a single generation existed. Everything below is fixed. The point of writing it down is that a threshold chosen after seeing the numbers is not a threshold, and this line has already been burned once: lv-bronte's verdict turned on which arms happened to be in the run (see § 4).

Nothing here may be amended after the first arm is generated. If the design turns out to be wrong, the run is wrong and gets re-run — the rule does not move.

---

1. What is being gated

gx10:~/r49-runs/hemingway-4b-pairs-3ep/ — Qwen3-4B-Instruct + instruction-pair LoRA on the renamed Hemingway corpus (994,760 words, 318 units, 6 renamed copies; leak gate 0 of 941 entities with both controls green). 3-epoch schedule, seed 4919, 2,661 steps, finished 2026-09-16 18:27.

2. Arms — three, and why each one is present

arm	what it is	role
base-unadapted	~/carriers/Qwen3-4B-Instruct, no adapter	negative control for memorisation (never saw the corpus) and the voice baseline
ckpt1750	epoch 1.973, eval_loss 2.2783 — the curve's minimum	ship candidate
ckpt850	epoch 0.959, eval_loss 2.2823	tied candidate, see below

⭐ ckpt850 is in the run because the loss curve cannot separate it from ckpt1750. Measured on this run's own 54 eval points: median neighbour jitter 0.0044, and step 850 sits +0.0040 above the minimum. Three checkpoints are inside one jitter of the best (850, 1700, 1750). "The minimum is at 1750" is true and is not a reason to believe 1750 is the better adapter — picking it by staring at the fourth decimal is reading noise, which is exactly what lv-bronte recorded about its own 475-vs-925 pair.

There is also a prior worth testing rather than assuming: on Brontë the EARLIER, epoch-1 checkpoint won the tiebreak on the axes that resolve (identical-to-control memorisation, 2.7× tighter seed variance). Epoch 2 bought Brontë nothing. If that repeats here, ckpt850 ships and half an epoch of overfit is avoided.

adapter/ (end of run, epoch 3.0, 2.3546) is NOT gated. It is +0.0762 worse than the minimum — 17.4× the median jitter — so unlike the 850/1750 pair this one is resolved by the loss curve alone. The epoch-3 collapse is the robust finding on this schedule; the epoch-2-vs-epoch-1 question is the open one.

3. Fixture and seeds — fixed now

• 60 beats, sampled by build_beat_fixture.py from ~/hemingway-pairs/pairs-val.jsonl (held-out split=val, 200 pairs, 173 inside the 90–150 word band), --seed 4919, spread across works round-robin.
• 4 seeds: 1234 5678 9012 3456. Same as lv-bronte.
• → 240 generations per arm, 720 total.

Why 60 and not 30. The floor is a per-seed range and per-seed variance scales as 1/√N, so doubling lv-bronte's 30-beat fixture buys a ~1.41× tighter floor. 173 in-band val pairs make 60 available; Brontë had 44 in-band and could not have run this. More seeds would not help — the floor is a range statistic and ranges widen with more samples.

4. ⚠ THE FLOOR RULE — changed, and this is the change that matters

The noise floor for a candidate is the larger of the within-arm seed spreads of the TWO arms being compared: max(spread(candidate), spread(base-unadapted)).

This is a deliberate amendment to the rule lv-bronte ran under, made before any Hemingway number exists. Under the old "largest spread across ALL arms" reading:

base-unadapted   spread 0.062
ckpt475          spread 0.092   <- shipped
ckpt925          spread 0.251   <- set the floor, on ONE outlier seed

ckpt475's +0.193 was failed by a floor contributed entirely by a third arm nobody was shipping. Scored pairwise the floor is 0.092 and the same gap clears at 2.1×. A candidate's verdict must not depend on which other arms you happened to generate — and with a three-arm design that defect is live again, which is why the rule is settled here rather than after the table prints.

voice_distance.py prints both floors and flags any case where the two rules disagree. The pairwise number is the verdict; the all-arms number is kept so the lv-bronte record stays comparable.

5. The three axes, and what passes

axis	instrument	passes if
A. VOICE	voice_distance.py --author Hemingway (Burrows's Delta over char bigrams vs the held-out val split)	delta_cb(base) − delta_cb(candidate) exceeds the pairwise floor
B. NOT COPIED	memorization_check.py (longest verbatim n-gram vs the renamed copies, n=8)	hit-rate and max at or near base-unadapted, which never saw the corpus. Positive control (corpus vs itself) must saturate, or the zeroes are blind
C. NO DAMAGE	score_beats.py --metric-source raw	ran-on / out-of-band rate not worse than base beyond that metric's own floor

A FAIL on A with B and C clean is the lv-bronte outcome and is NOT automatically a no-ship — it was shipped there as additive, reversible and safety-clean. That call is the operator's, and the difference here is that Hemingway is not public domain the way Brontë is, so axis B carries more weight in that conversation, not less.

6. Pre-flight already run — the beat-contamination audit

audit_pairs_sourcenames.py over all 7,294 built pairs, controls green (941/941 surfaces found in the unrenamed source; nonce absent from both trees; 6/6 planted names detected in a positive control):

• val pairs — 0 of 200. The fixture is drawn from these, so the eval beats are clean and the gate is unconfounded. This is why the gate runs at all.
• train pairs — 70 of 7,094 (0.96%) name a character the rename removed (Santiago ×16, Catherine ×7, Rinaldi ×3, Brett, Harry, Jake …), with 0 of 7,294 responses affected — the lv-bronte beat-only signature exactly. The pairs predate build_sft_pairs.py --source-entities.

This is a property of the trained adapter and it is on the record before the gate runs, not discovered afterwards. It does not invalidate any axis: the instruction half is contaminated, the target half is not, and the fixture is clean. Whether to retrain on filtered pairs is a separate decision with its own cost, and it belongs to the operator.

7. Reproduce

# on pfi-gx10, ~/lv-hemingway
./eval-hemingway.sh          # three arms, 60 beats, 4 seeds
python voice-prep.py         # beats5.* -> voice.*, control renamed to *unadapted*
scripts/r49-corpus/voice_distance.py corpus-renamed ~/r49-runs/hemingway-eval --author Hemingway