Vuong Hoang
9c8a4e959c
Freezes the lv-mccarthy gate design while no McCarthy generation exists, per the
lv-hemingway precedent: a threshold chosen after seeing the numbers is not a
threshold. Three arms (base-unadapted, ckpt900, ckpt450), 60 beats, 4 seeds.
Settles the voice-axis question D1 deferred. McCarthy's corpus measures 0.0 quote
marks per 10k against Hemingway's 838, so "emit no quotation marks" is the cheapest
way to move a char-bigram Delta without learning a sentence. Three parts:
PRIMARY unchanged -- the mccarthy register names the punctuation and
--system-from drives the base control with the same prompt, so the
cheap win is handed to both sides.
SECONDARY voice_distance.py --secondary-normalised re-runs the whole analysis
with punctuation stripped from the reference and every arm. A
conservative lower bound; reported, never the verdict.
TRIGGER voice_distance.py --punct-report evaluates a pre-registered line --
base-unadapted quote density above 100 per 10k means the control did
not take the win it was handed, and the normalised read is promoted
to load-bearing. 100 is the order-of-magnitude line between this
corpus's 0.0 and Hemingway's 838, chosen now rather than after the
table prints.
ckpt450 is NOT tied with the minimum (+4.9x the 0.00393 median neighbour jitter)
and the pre-registration says so. It is generated to test a stated prior and to
price the memorisation headroom an earlier checkpoint buys on an in-copyright
author, with a decision rule that says exactly what result would let it displace
ckpt900.
Instrument controls, run before this landed:
- the voice_distance.py refactor reproduces the shipped lv-hemingway
voice_distance.txt BYTE FOR BYTE on the default path
- strip_punct drops a quote-bearing sample from 2500.0 to 0.0 marks per 10k
- the secondary read still resolves a gap on the Hemingway arms (+0.410 at
7.3x floor), so a null on McCarthy would be a finding, not a blind detector
Pre-flight re-run at gate time rather than quoted from 2026-09-17:
- leak gate: 0 of 75 renameable, 0 of 37 sub-threshold, 0 separator-split,
four controls green
- beat-contamination audit: 0 of 3942 beats AND 0 of 3942 responses, against
Hemingway's 70 of 7094 -- build_sft_pairs.py --source-entities earned its
mandatory flag
Also records three provenance defects found on first read of the run and their
disposition: the hardcoded "r49-babyyarros-pairs-pilot" run label (cosmetic, same
literal on all three runs), the empty harness_commit (all three runs), and a
pairs_sha256_16 that is not a sha256sum of the file (consistent across runs, so a
cache key rather than a fault). The run is bound to McCarthy's pairs by record
count, not by the label.
218 lines
12 KiB
Markdown
218 lines
12 KiB
Markdown
# lv-mccarthy v2 gate — pre-registration
|
||
|
||
**Written 2026-09-21, before a single generation existed.** Everything below is
|
||
fixed. The point of writing it down is that a threshold chosen after seeing the
|
||
numbers is not a threshold. This is the third pre-registration in the line and it
|
||
inherits the lv-hemingway design deliberately, so the two runs stay comparable;
|
||
every place it departs is marked and argued.
|
||
|
||
Nothing here may be amended after the first arm is generated. If the design turns
|
||
out to be wrong, the run is wrong and gets re-run — the rule does not move.
|
||
|
||
---
|
||
|
||
## 1. What is being gated
|
||
|
||
`gx10:~/r49-runs/mccarthy-4b-pairs-3ep/` — Qwen3-4B-Instruct + instruction-pair
|
||
LoRA on the renamed McCarthy corpus (584,684 words, 167 units, 6 works, 6 renamed
|
||
copies). 3-epoch schedule, seed 4919, rank 32 / alpha 64, lr 1e-4, seq 1536,
|
||
1,380 steps in 2h30m47s, finished **2026-09-18 00:49 PT**. `train_loss` 2.172,
|
||
end-of-run `eval_loss` 2.4594.
|
||
|
||
**The run outcome was unverified for three days** and this gate is the first read
|
||
of it. That is recorded here rather than quietly fixed, because "the run finished"
|
||
and "the run did what was configured" are different claims and only the second one
|
||
is worth anything.
|
||
|
||
### 1a. Provenance defects found on first read, and their disposition
|
||
|
||
| found | disposition |
|
||
|---|---|
|
||
| `provenance.json` `"run": "r49-babyyarros-pairs-pilot"` | **Cosmetic, not a mis-copy.** The same literal appears in the yarros AND hemingway provenance — it is a hardcoded label in `train_pairs_lora.py`, not a stale field for this run. Fix the harness; it invalidates nothing. |
|
||
| `harness_commit: ""` | **Empty on all three runs.** The training harness never recorded its own commit. No claim in this gate rests on it. Named here so it is not re-discovered as news. |
|
||
| `pairs_sha256_16` does not match `sha256sum` of the pairs file | **Not a defect.** It mismatches identically on yarros and hemingway too, so the harness computes it over the loaded records rather than the raw bytes. It is a consistent, per-corpus-unique cache key. |
|
||
| `pairs` recorded as the relative `pairs/pairs-full.jsonl` | Resolved against the launch CWD `~/lv-mccarthy`. **Bound by count instead of by path**: `train_pairs` 3673 and `val_pairs_n` 269 match `lv-mccarthy/pairs/` exactly and match no other pair set on the box. |
|
||
|
||
The run trained on McCarthy's pairs. That is established by the record counts and
|
||
by `system_prompt_source: "pairs provenance (mccarthy)"`, not by the `run` label.
|
||
|
||
## 2. Arms — three, and the third is here for a different reason than Hemingway's was
|
||
|
||
| arm | what it is | role |
|
||
|---|---|---|
|
||
| `base-unadapted` | `~/carriers/Qwen3-4B-Instruct`, no adapter | negative control for memorisation (never saw the corpus) **and** the voice baseline |
|
||
| `ckpt900` | epoch 1.958, eval_loss **2.38706** — the curve's minimum | ship candidate |
|
||
| `ckpt450` | epoch 0.980, eval_loss **2.4063** | **prior test**, see below |
|
||
|
||
Measured on this run's own 56 eval points: **median neighbour jitter 0.00393**.
|
||
|
||
⚠ **`ckpt450` IS NOT TIED, and saying so is the point.** On Hemingway the second
|
||
arm was in the run because the loss curve could not separate it (+0.9× jitter).
|
||
Here it can: `ckpt450` sits **+0.0193 = 4.9× jitter** above the minimum. Three
|
||
checkpoints *are* inside one jitter of the best — 875 (+0.4×), 850 (+1.1×), 925
|
||
(+1.5×) — and the choice of 900 among them is arbitrary and declared arbitrary;
|
||
they are the same point on the curve. `ckpt450` is not one of them.
|
||
|
||
So it is generated for a stated, falsifiable reason rather than as a coin-flip:
|
||
|
||
- **The prior it tests.** On Brontë the earlier epoch-1 checkpoint won the tiebreak
|
||
on the axes that resolve. On Hemingway the pair was tied and the question stayed
|
||
open. **McCarthy is the first run in the line whose loss curve says epoch 2 is
|
||
genuinely better than epoch 1.** That is a claim the axes can check, and checking
|
||
it costs one arm.
|
||
- **The thing it would buy.** Axis B (memorisation) carries more weight here than on
|
||
any previous corpus — Brontë is public domain, Hemingway is not, and **McCarthy is
|
||
in copyright with a living estate.** An earlier checkpoint is the cheap way to buy
|
||
a lower verbatim-overlap rate. Whether it actually does is measured below, not
|
||
assumed.
|
||
|
||
**`adapter/` (end of run, epoch 3.0, 2.4594) is NOT gated.** At **+18.4× jitter**
|
||
it is resolved by the loss curve alone — and note the curve does not drift there,
|
||
it *steps*: 2.393 at step 925, 2.457 at step 950, and flat for the remaining 430
|
||
steps. The epoch-3 collapse reproduces the Hemingway finding on a second corpus.
|
||
|
||
⚠ The shipped `adapter/` directory is the epoch-3 weights. **Whatever ships is a
|
||
checkpoint, not `adapter/`.**
|
||
|
||
## 3. Fixture and seeds — fixed now
|
||
|
||
- **60 beats**, sampled by `build_beat_fixture.py` from
|
||
`~/lv-mccarthy/pairs/pairs-val.jsonl` (held-out `split=val`, 269 pairs), `--seed 4919`,
|
||
spread across works round-robin.
|
||
- **`--max-words 140`, not the script's default 150.** McCarthy's register asks for
|
||
90–140 words and `score_beats.py` scores the in-band rate at 90–140. A fixture built
|
||
at 90–150 would put reference passages outside the band the product asks for and the
|
||
scorer measures. **207 of 269 val pairs are in-band**, across all six works
|
||
(blood-meridian 51, all-the-pretty-horses 40, the-road 38, cities-of-the-plain 33,
|
||
no-country-for-old-men 28, the-crossing 17).
|
||
- **4 seeds: 1234 5678 9012 3456.** Same as lv-bronte and lv-hemingway.
|
||
- → **240 generations per arm, 720 total.**
|
||
|
||
60 is carried over from Hemingway unchanged so the two floors are comparable; 207
|
||
in-band pairs make it available with room to spare (Hemingway had 173, Brontë 44).
|
||
|
||
## 4. The floor rule — inherited, unchanged
|
||
|
||
**The noise floor for a candidate is the larger of the within-arm seed spreads of
|
||
the TWO arms being compared: `max(spread(candidate), spread(base-unadapted))`.**
|
||
|
||
Settled at lv-hemingway and not reopened. `voice_distance.py` prints both this and
|
||
lv-bronte's all-arms floor and flags any case where the two rules disagree; the
|
||
pairwise number is the verdict.
|
||
|
||
## 5. ⭐ THE VOICE AXIS — the open question, settled here before any number exists
|
||
|
||
McCarthy's corpus measures **0.0 quote marks per 10k words** against Hemingway's
|
||
838, and `voice_distance.py` is Burrows's Delta over **character bigrams**. "Emit no
|
||
quotation marks" is therefore the single cheapest available way to move this metric
|
||
a long way without having learned a sentence. lv-mccarthy D1 flagged this and
|
||
deferred it; it is settled now, in three parts.
|
||
|
||
**5a. PRIMARY read — unchanged, and already hardened.** The `mccarthy` register
|
||
names the punctuation on purpose (`build_sft_pairs.py` REGISTERS, commit `f3bf3ca`),
|
||
and `eval-mccarthy.sh` drives **every** arm including `base-unadapted` with that same
|
||
system prompt via `--system-from`. The control is therefore *told* to drop quote marks
|
||
and write `dont`/`aint`/`wont`. The cheap win is handed to both sides and the adapter
|
||
earns no delta for it. This was decided when the register was written, before any
|
||
McCarthy number existed, and is not reopened here. **Stated cost: the voice axis is
|
||
harder for this corpus than for Hemingway's, deliberately.**
|
||
|
||
**5b. SECONDARY read — pre-registered, REPORTED, never the verdict.**
|
||
`voice_distance.py --secondary-normalised` re-runs the entire analysis with all
|
||
punctuation stripped from the reference and from every arm. A gap that survives is
|
||
carried by words rather than by marks.
|
||
|
||
⚠ It is a **conservative lower bound, not a better measurement**: stripping terminal
|
||
punctuation also strips sentence-length signal the adapter legitimately learned, and
|
||
sentence length is a real McCarthy trait. It reads as *"at least this much of the
|
||
primary gap is not the punctuation trick."* **It cannot overturn the primary verdict.**
|
||
|
||
**5c. THE CONFOUND TRIGGER — the part that makes 5a falsifiable.** 5a's whole
|
||
argument is that the base control *complies* with the instruction. That is an
|
||
assumption about a model's behaviour, and it is checkable. `--punct-report` prints
|
||
per-arm quote-mark, contraction-apostrophe and dash density against the reference,
|
||
and evaluates this line:
|
||
|
||
> **If `base-unadapted` quote density exceeds 100 marks per 10k words, the control
|
||
> did not take the win it was handed, part of the primary gap IS the punctuation
|
||
> trick, and the normalised secondary read (5b) is promoted to load-bearing for the
|
||
> verdict.**
|
||
|
||
100 is the order-of-magnitude line between this corpus's 0.0 and Hemingway's 838,
|
||
chosen now rather than after the table prints.
|
||
|
||
**Instrument controls, run before this was committed.** The `voice_distance.py`
|
||
refactor that added 5b and 5c reproduces the shipped lv-hemingway
|
||
`voice_distance.txt` **byte for byte** on the default path (positive control: the
|
||
edit did not move an existing record). `strip_punct` drops a quote-bearing sample's
|
||
density from 2500.0 to **0.0** per 10k (negative control: the normalisation is not
|
||
a no-op). On the Hemingway arms the secondary read still resolves a gap
|
||
(ckpt1750 +0.410 at 7.3× floor) rather than flattening everything — so a null
|
||
result from 5b on McCarthy would be a finding about McCarthy, not a blind detector.
|
||
|
||
## 6. The three axes, and what passes
|
||
|
||
| axis | instrument | passes if |
|
||
|---|---|---|
|
||
| **A. VOICE** | `voice_distance.py --author McCarthy --punct-report --secondary-normalised` | `delta_cb(base) − delta_cb(candidate)` **exceeds the pairwise floor** on the PRIMARY read, unless 5c fires, in which case on the NORMALISED read |
|
||
| **B. NOT COPIED** | `memorization_check.py` (longest verbatim n-gram vs the renamed copies, n=8) | hit-rate and max at or near `base-unadapted`, which never saw the corpus. Positive control (corpus vs itself) must saturate, or the zeroes are blind |
|
||
| **C. NO DAMAGE** | `score_beats.py --metric-source raw` | in-band up on base beyond the metric's own floor; on-beat and ran-on not worse beyond it |
|
||
|
||
## 7. The decision rule — frozen
|
||
|
||
1. **`ckpt900` is the default ship candidate.** The loss curve resolves it above
|
||
`ckpt450` at 4.9× jitter.
|
||
2. **`ckpt450` displaces it only if BOTH hold:** it is *not resolvably worse* on
|
||
axis A (`gap_450 ≥ gap_900 − pairwise_floor`) **AND** it is *strictly better* on
|
||
axis B by more than the base-to-candidate separation on that axis. Buying
|
||
memorisation headroom is worth it; buying it by giving up the voice the adapter
|
||
exists for is not.
|
||
3. **Axis C failure disqualifies that candidate outright**, whatever axis A says.
|
||
4. **If both candidates fail axis A** (gap inside the pairwise floor), the result is
|
||
a **negative reported with its sensitivity floor**, and the ship-or-park call goes
|
||
to the operator. The lv-bronte precedent — fail on A, clean on B and C, shipped
|
||
anyway as additive and reversible — **does not transfer automatically**: Brontë is
|
||
public domain and McCarthy is not, so axis B carries more weight in that
|
||
conversation, not less.
|
||
5. **Nothing ships on the strength of the epoch-3 `adapter/`.** It is not gated.
|
||
|
||
## 8. Pre-flight — already run, and clean
|
||
|
||
Both re-run at gate time rather than quoted from the 2026-09-17 session, because
|
||
"it passed three days ago" is a claim about a tree that may have moved.
|
||
|
||
**Leak gate** (`leak_gate.py --min-cap 8`, report `leak_gate_report.gate-time.json`):
|
||
|
||
```
|
||
0 of 75 renameable · 0 of 37 sub-threshold · 0 separator-split survivors
|
||
positive control 108/108 surfaces found in the unrenamed source
|
||
negative control nonce absent · split-scan controls: planted `W olfenbarger` detected
|
||
```
|
||
|
||
**Beat-contamination audit** (`audit_pairs_sourcenames.py --min-cap 5`, mirroring
|
||
the floor `rename.py` actually ran at), over **all 3,942 built pairs**:
|
||
|
||
```
|
||
BEAT naming a removed surface: 0 of 3942 (0.00%)
|
||
RESPONSE naming a removed surface: 0 of 3942 (0.00%)
|
||
controls green: 108/108 found in source · nonce absent from both trees
|
||
```
|
||
|
||
⭐ **Zero, where Hemingway carried 70 of 7,094 contaminated beats (0.96%).** That is
|
||
`build_sft_pairs.py --source-entities`, which was mandatory for this corpus's D4 and
|
||
which Hemingway's pairs predate. The gate is unconfounded on both halves of the pair,
|
||
not just the fixture half.
|
||
|
||
**Sensitivity floor, stated as part of the pre-flight rather than as a footnote:** a
|
||
name appearing fewer than 8 times per work is never detected and a phrase recurring
|
||
fewer than 5 times is never audited — and **no phrase map exists for this corpus, so
|
||
the phrase audit does not run at all.** Yarros and Brontë both had one. A clean leak
|
||
gate here is a narrower claim than a clean leak gate there.
|
||
|
||
## 9. Reproduce
|
||
|
||
```bash
|
||
# on pfi-gx10, ~/lv-mccarthy
|
||
./eval-mccarthy.sh # three arms, 60 beats, 4 seeds, all three axes
|
||
```
|