vh
0bb4938518
The gate design is written before any generation exists, because lv-bronte's
verdict turned on a choice that was only visible after the numbers printed.
THE FLOOR RULE IS NOW PAIRWISE. lv-bronte computed the noise floor as the largest
within-arm seed spread across ALL arms present. Its ckpt475 shipped at +0.193
against a 0.251 floor set entirely by ckpt925 -- a third arm nobody was shipping,
on one outlier seed. Scored against the arm it was actually compared to, the floor
is 0.092 and the same gap clears at 2.1x. A candidate's verdict must not depend on
which other arms happened to be generated. voice_distance.py now prints both floors
and flags any disagreement, so the lv-bronte record stays comparable.
audit_pairs_sourcenames.py closes the blind spot leak_gate.py has by construction:
it reads the corpus and the renamed copies, never the generated beats, so it cannot
see a beat-writing model restoring the author's real character names. Run over the
Hemingway pairs, which predate build_sft_pairs.py --source-entities:
val 0 of 200 -- the eval fixture is clean, the gate is unconfounded
train 70 of 7,094 (0.96%) -- Santiago x16, Catherine x7, Rinaldi x3, Brett,
Harry, Jake, Pablo, Nick, Maria ...
responses 0 of 7,294 -- the lv-bronte beat-only signature exactly
A matched surface is only counted when the rename actually removed it, verified
against the renamed copies, so a beat naming a held real-world place is not a leak.
Controls run every time: 941/941 surfaces found in the unrenamed source, nonce
absent from both trees, and 6 planted canonical names detected 6/6.
voice_distance.py --author is now REQUIRED. It was hardcoded "Yarros" and printed
"reference: held-out Yarros" over Brontë's numbers into a committed artifact. A
default would have moved the silent-wrong-label failure rather than removed it. The
stale "one seed-pair per arm / corroborates Base < Instruct" footer is replaced with
what the run actually carries.
Gate design: three arms (base-unadapted, ckpt1750, ckpt850), 60 beats, 4 seeds.
ckpt850 is present because the loss curve cannot separate it from ckpt1750 -- +0.0040
against a 0.0044 median neighbour jitter, with three checkpoints inside one jitter of
the minimum. adapter/ is excluded: +0.0762 is 17.4x the jitter and is resolved without
a gate.
125 lines
6.5 KiB
Markdown
125 lines
6.5 KiB
Markdown
# lv-hemingway v2 gate — pre-registration
|
||
|
||
**Written 2026-09-17, before a single generation existed.** Everything below is
|
||
fixed. The point of writing it down is that a threshold chosen after seeing the
|
||
numbers is not a threshold, and this line has already been burned once: lv-bronte's
|
||
verdict turned on which arms happened to be in the run (see § 4).
|
||
|
||
Nothing here may be amended after the first arm is generated. If the design turns
|
||
out to be wrong, the run is wrong and gets re-run — the rule does not move.
|
||
|
||
---
|
||
|
||
## 1. What is being gated
|
||
|
||
`gx10:~/r49-runs/hemingway-4b-pairs-3ep/` — Qwen3-4B-Instruct + instruction-pair
|
||
LoRA on the renamed Hemingway corpus (994,760 words, 318 units, 6 renamed copies;
|
||
leak gate 0 of 941 entities with both controls green). 3-epoch schedule, seed 4919,
|
||
2,661 steps, finished 2026-09-16 18:27.
|
||
|
||
## 2. Arms — three, and why each one is present
|
||
|
||
| arm | what it is | role |
|
||
|---|---|---|
|
||
| `base-unadapted` | `~/carriers/Qwen3-4B-Instruct`, no adapter | negative control for memorisation (never saw the corpus) **and** the voice baseline |
|
||
| `ckpt1750` | epoch 1.973, eval_loss **2.2783** — the curve's minimum | ship candidate |
|
||
| `ckpt850` | epoch 0.959, eval_loss **2.2823** | **tied** candidate, see below |
|
||
|
||
⭐ **`ckpt850` is in the run because the loss curve cannot separate it from
|
||
`ckpt1750`.** Measured on this run's own 54 eval points: median neighbour jitter
|
||
**0.0044**, and step 850 sits **+0.0040** above the minimum. Three checkpoints are
|
||
inside one jitter of the best (850, 1700, 1750). "The minimum is at 1750" is true
|
||
and is *not* a reason to believe 1750 is the better adapter — picking it by staring
|
||
at the fourth decimal is reading noise, which is exactly what lv-bronte recorded
|
||
about its own 475-vs-925 pair.
|
||
|
||
There is also a prior worth testing rather than assuming: on Brontë the EARLIER,
|
||
epoch-1 checkpoint won the tiebreak on the axes that resolve (identical-to-control
|
||
memorisation, 2.7× tighter seed variance). Epoch 2 bought Brontë nothing. If that
|
||
repeats here, `ckpt850` ships and half an epoch of overfit is avoided.
|
||
|
||
**`adapter/` (end of run, epoch 3.0, 2.3546) is NOT gated.** It is +0.0762 worse
|
||
than the minimum — **17.4× the median jitter** — so unlike the 850/1750 pair this
|
||
one is resolved by the loss curve alone. The epoch-3 collapse is the robust finding
|
||
on this schedule; the epoch-2-vs-epoch-1 question is the open one.
|
||
|
||
## 3. Fixture and seeds — fixed now
|
||
|
||
- **60 beats**, sampled by `build_beat_fixture.py` from `~/hemingway-pairs/pairs-val.jsonl`
|
||
(held-out `split=val`, 200 pairs, **173 inside the 90–150 word band**), `--seed 4919`,
|
||
spread across works round-robin.
|
||
- **4 seeds: 1234 5678 9012 3456.** Same as lv-bronte.
|
||
- → 240 generations per arm, 720 total.
|
||
|
||
**Why 60 and not 30.** The floor is a per-seed range and per-seed variance scales
|
||
as 1/√N, so doubling lv-bronte's 30-beat fixture buys a ~1.41× tighter floor. 173
|
||
in-band val pairs make 60 available; Brontë had 44 in-band and *could not* have run
|
||
this. **More seeds would not help** — the floor is a range statistic and ranges
|
||
widen with more samples.
|
||
|
||
## 4. ⚠ THE FLOOR RULE — changed, and this is the change that matters
|
||
|
||
**The noise floor for a candidate is the larger of the within-arm seed spreads of
|
||
the TWO arms being compared: `max(spread(candidate), spread(base-unadapted))`.**
|
||
|
||
This is a deliberate amendment to the rule lv-bronte ran under, made before any
|
||
Hemingway number exists. Under the old "largest spread across ALL arms" reading:
|
||
|
||
```
|
||
base-unadapted spread 0.062
|
||
ckpt475 spread 0.092 <- shipped
|
||
ckpt925 spread 0.251 <- set the floor, on ONE outlier seed
|
||
```
|
||
|
||
`ckpt475`'s **+0.193 was failed by a floor contributed entirely by a third arm
|
||
nobody was shipping.** Scored pairwise the floor is 0.092 and the same gap clears
|
||
at 2.1×. A candidate's verdict must not depend on which *other* arms you happened
|
||
to generate — and with a three-arm design that defect is live again, which is why
|
||
the rule is settled here rather than after the table prints.
|
||
|
||
`voice_distance.py` prints **both** floors and flags any case where the two rules
|
||
disagree. The pairwise number is the verdict; the all-arms number is kept so the
|
||
lv-bronte record stays comparable.
|
||
|
||
## 5. The three axes, and what passes
|
||
|
||
| axis | instrument | passes if |
|
||
|---|---|---|
|
||
| **A. VOICE** | `voice_distance.py --author Hemingway` (Burrows's Delta over char bigrams vs the held-out val split) | `delta_cb(base) − delta_cb(candidate)` **exceeds the pairwise floor** |
|
||
| **B. NOT COPIED** | `memorization_check.py` (longest verbatim n-gram vs the renamed copies, n=8) | hit-rate and max at or near `base-unadapted`, which never saw the corpus. Positive control (corpus vs itself) must saturate, or the zeroes are blind |
|
||
| **C. NO DAMAGE** | `score_beats.py --metric-source raw` | ran-on / out-of-band rate not worse than base beyond that metric's own floor |
|
||
|
||
A **FAIL on A with B and C clean** is the lv-bronte outcome and is NOT
|
||
automatically a no-ship — it was shipped there as additive, reversible and
|
||
safety-clean. That call is the operator's, and the difference here is that
|
||
**Hemingway is not public domain the way Brontë is**, so axis B carries more weight
|
||
in that conversation, not less.
|
||
|
||
## 6. Pre-flight already run — the beat-contamination audit
|
||
|
||
`audit_pairs_sourcenames.py` over all 7,294 built pairs, controls green
|
||
(941/941 surfaces found in the unrenamed source; nonce absent from both trees;
|
||
6/6 planted names detected in a positive control):
|
||
|
||
- **val pairs — 0 of 200.** The fixture is drawn from these, so the eval beats are
|
||
clean and the gate is unconfounded. This is why the gate runs at all.
|
||
- **train pairs — 70 of 7,094 (0.96%)** name a character the rename removed
|
||
(`Santiago` ×16, `Catherine` ×7, `Rinaldi` ×3, `Brett`, `Harry`, `Jake` …),
|
||
with **0 of 7,294 responses** affected — the lv-bronte beat-only signature
|
||
exactly. The pairs predate `build_sft_pairs.py --source-entities`.
|
||
|
||
**This is a property of the trained adapter and it is on the record before the
|
||
gate runs, not discovered afterwards.** It does not invalidate any axis: the
|
||
instruction half is contaminated, the target half is not, and the fixture is clean.
|
||
Whether to retrain on filtered pairs is a separate decision with its own cost, and
|
||
it belongs to the operator.
|
||
|
||
## 7. Reproduce
|
||
|
||
```bash
|
||
# on pfi-gx10, ~/lv-hemingway
|
||
./eval-hemingway.sh # three arms, 60 beats, 4 seeds
|
||
python voice-prep.py # beats5.* -> voice.*, control renamed to *unadapted*
|
||
scripts/r49-corpus/voice_distance.py corpus-renamed ~/r49-runs/hemingway-eval --author Hemingway
|
||
```
|