esh-pfi-infrastructure/scripts/mccarthy-corpus/RUNBOOK.md

# lv-mccarthy — corpus → leak gate → pairs → train → v2 gate

Six works, 167 units, 584,684 words. Built on **pfi-gx10** under `~/lv-mccarthy/`
as `infra-ops`, except **D1, which must run on nh3-dev**: the builder reads the
kvasir catalogue at `/home/lkraven/development/kvasir/data/library/catalog.sqlite`
and that path exists only there.

⚠ **This file exists because the 2026-09-17 D1→D3 session recorded nothing.** There
was no runbook, the commands were issued over non-interactive ssh so no shell history
survived, and a later session had to recover the whole chain by rebuilding candidates
and matching sha256 against the artifacts on disk. Every deviation below is now pinned
by a reproducibility control; keep it that way.

## The chain

```bash
R=~/development/eshpfi-management/scripts        # nh3-dev for D1, ~/lv-mccarthy/scripts on gx10

# D1 — build. ON nh3-dev (needs the kvasir catalogue), then rsync corpus-clean/ to gx10.
python3 $R/mccarthy-corpus/build_corpus_mccarthy.py --out corpus-clean

# D2 — entity map. --min-mid-ratio is what keeps `Yeah`/`Buenas`/`Shh` out; see below.
python3 $R/r49-corpus/entities.py corpus-clean --out corpus-clean/entities.json \
  --stoplist $R/mccarthy-corpus/stoplist_mccarthy.json \
  --min-count 5 --fold-clitics --drop-acronyms --min-mid-ratio 0.2 --min-mid 2

# D2c — the four hand-verified genders.
python3 $R/r49-corpus/apply_gender_overrides.py --entities corpus-clean/entities.json \
  --overrides $R/mccarthy-corpus/gender_overrides_mccarthy.json \
  --out corpus-clean/entities-final.json

# D3 — rename. --scope corpus, --preset mccarthy and --min-cap 5 are all deviations.
python3 $R/r49-corpus/rename.py corpus-clean --entities corpus-clean/entities-final.json \
  --dictionary ~/r49-prep/name_dictionary.json --out corpus-renamed \
  --preset mccarthy --scope corpus --min-cap 5 --copies 6 --seed 4919 \
  --holdout-chapter 7 17

# GATE — must pass before anything is trained. --min-cap 8 here is deliberate; see below.
python3 $R/r49-corpus/leak_gate.py corpus-clean --entities corpus-clean/entities-final.json \
  --renamed corpus-renamed --min-cap 8 --report corpus-renamed/leak_gate_report.json

# D4 — instruction pairs. ON gx10; ~24 min against the `gen` seat.
./run-pairs.sh
```

Gate result, 2026-09-17 (after the separator fix): **0 of 75 renameable, 0 of 37
sub-threshold, 0 separator-split surfaces**, all four controls green. Sensitivity
floor: a name under 8 capitals per work is never detected, a phrase under 5
recurrences never audited, and **no phrase map exists for this corpus, so the phrase
audit does not run at all** — the Yarros and Brontë runs both had one.

## The deviations, and what forced each

| deviation | why |
|---|---|
| **D1 runs on nh3-dev** | The builder reads the kvasir catalogue by absolute path. gx10 has no copy. |
| **`--min-count 5`** (was 8) | The entity map's floor has to match rename's, or entities between the two floors sit in the map, are never renamed, and count as leaks. The first D3 gate run failed with 45 survivors for exactly this. |
| **`--min-mid-ratio 0.2 --min-mid 2`** | Without it the map admits `Yeah`, `Buenas`, `Shh`, `Mande`, `Sí`, `Git` — words that are only ever capitalised at a sentence start. ⚠ The map is INSENSITIVE to the exact value: any ratio in **[0.05, 0.3]** with `--min-mid` 1 or 2 reproduces it byte-for-byte. The original run's values are unrecoverable and it does not matter. `--min-mid 3` does NOT reproduce it. |
| **`--fold-clitics --drop-acronyms`** | Both are needed to reproduce the map; `--rescue-honorific` is inert here (0 rescued). |
| **`--scope corpus`** (was `work`) | Nine surfaces appear in more than one work — Parham, Grady, Cole, Socorro, Héctor. A per-work map gives John Grady a different invented name in each Border Trilogy novel. |
| **`--preset mccarthy`** | Hemingway's romance pool carries `it_IT`/`fr_FR` for his Italian and French casts; McCarthy writes neither. `en_US` + `es_MX`/`es_ES` at an even share. |
| **`--holdout-chapter 7 17`** (a LIST) | The val split is one chapter per work, so its size scales with WORK COUNT, not corpus size. Six works would have given a Brontë-class ~18,000-word val reference; two indices give 11 units and 40,653 words per copy, larger than Hemingway's, for 7% of the corpus. |
| **gate at `--min-cap 8`, rename at `--min-cap 5`** | Not a mistake. The gate reports the 37 entities rename never touched *separately*, which is strictly more informative than running both at 5 (where the sub-threshold bucket is empty). |
| **D4 `--reflow-hard-wraps`** | The Road is hard-wrapped at ~76 characters and the other five works are not. See DEFECT 4 in `build_sft_pairs.py`. |
| **D4 `--source-entities`** | Mandatory. The beat generator reads the passage and will supply canonical names from its own memory of the book; the corpus gate never reads the generated beats. |

## The defect the gate could not see, and now can

⚠⚠ **The 2026-09-17 tree passed this gate at "0 of 75 renameable and 0 of 37
sub-threshold" while carrying 13 occurrences of five protagonist names in all six
copies.** `leak_gate.py` scanned `\b(Surface)\b`, and a character inserted inside a
name defeats that pattern outright — so the occurrence was unrenameable by `rename.py`
*and* unreportable by the gate. Two mechanisms, both in the source extraction:

```
B ell  C higurh  M oss  T oadvine     a small-caps drop cap survived as its own token
Toad-vine  Glan-ton                   a print line-break hyphen survived the extraction
```

Every *visible* occurrence had been renamed correctly, which is what made the residue
invisible to any spot-read: the names are gone everywhere you look.

Fixed in three places, and all three must stay:

1. **`build_corpus_mccarthy.py` rules 4 and 5** repair the text at its source — 32
   split initials with a lowercase remainder, 5 hyphen-split names, both with expected
   counts so a master change fails the build. ⚠ Rule 4's letter class is **consonants
   only**: `I` opens 1,966 paragraphs, `A` opens 143 and `Y` opens 32 (Spanish *y*), and
   folding any of them would corrupt 2,141 lines to fix 32.
2. **`leak_gate.py --split-frag-max`** runs a separator-tolerant scan every time, with
   its own positive and negative controls, and **fails the gate**. Verified against the
   pre-fix tree: it reports all five surfaces and exits 1.
3. The exact-match passes are unchanged, so the old verdict is reproduced alongside.

Cross-checked on the two shipped corpora: **lv-bronte is clean** of this class;
**lv-hemingway carries 3** (`Primi tivo`, `Pasionar ia`, `Chi cote`) and is live.

## Corpus properties worth knowing before you change anything

- **`repair_typography.py` MUST NOT be run on this corpus.** It normalises "toward what
  the text does" and would put quotation marks back into a corpus that measures 0.0 per
  10k against Hemingway's 838. The builder runs no normalisation and asserts the density.
- **Back matter is stripped BEFORE the split**, inverting the Hemingway order: Blood
  Meridian and The Crossing end with a dumped table of contents made of bare roman
  numerals, which is the exact shape of a chapter marker.
- **Blood Meridian sets a dash-separated chapter argument under each roman numeral.**
  131 paragraphs; `--drop-leading-heading` now consumes them.
- Two truncated catalogue rows are excluded in favour of complete siblings.

## D5 — train the adapter

```bash
# ON gx10, ~/lv-mccarthy. 1,380 steps, 2h30m47s on the GB10.
./launch-train.sh          # scripts/mccarthy-corpus/launch-train.sh
```

Qwen3-4B-Instruct, rank 32 / alpha 64, lr 1e-4, seq 1536, batch 1 × accum 8, 3 epochs,
seed 4919 → `~/r49-runs/mccarthy-4b-pairs-3ep/`. The two deliberate deviations from the
script defaults (`--save-total-limit 60`, `--eval-steps/--save-steps 25`) are argued in the
launch script's own header and exist to keep the epoch-1 and epoch-2 checkpoints alive for
the tiebreak; the default limit of 12 would have deleted both.

**Result, 2026-09-18 00:49 PT:** `train_loss` 2.172, end-of-run `eval_loss` 2.4594, 56 eval
points, **median neighbour jitter 0.00393**.

```
ckpt900   2.38706  ep1.958   the minimum          875 +0.4x · 850 +1.1x · 925 +1.5x = tied
ckpt450   2.4063   ep0.980   +4.9x jitter         the epoch-1 local minimum, NOT tied
adapter/  2.4594   ep3.000   +18.4x jitter        the epoch-3 collapse, resolved by the curve
```

⚠ **The curve does not drift into the epoch-3 collapse, it STEPS**: 2.393 at step 925,
2.457 at 950, then flat for the remaining 430 steps. Same shape as Hemingway's.

⚠ **`adapter/` is the epoch-3 weights.** Whatever ships is a checkpoint, not `adapter/`.

### Provenance fields that look wrong and are not

Checked when the run was first read, three days after it finished. All three reproduce on
the yarros and hemingway runs, so none is specific to this one:

| field | reading |
|---|---|
| `"run": "r49-babyyarros-pairs-pilot"` | a hardcoded label in `train_pairs_lora.py`, not a stale copy. Cosmetic. |
| `harness_commit: ""` | the trainer never recorded its own commit, on any run. No claim rests on it. |
| `pairs_sha256_16` ≠ `sha256sum` of the pairs file | computed over the loaded records, not the raw bytes. Consistent and per-corpus-unique, so a valid cache key. |
| `pairs` recorded as the relative `pairs/pairs-full.jsonl` | resolves against the launch CWD. **Bind it by COUNT, not path**: `train_pairs` 3673 / `val_pairs_n` 269 match `lv-mccarthy/pairs/` exactly and match no other pair set on the box. |

## D6 — the gate

```bash
# ON gx10, ~/lv-mccarthy. 3 arms x 60 beats x 4 seeds = 720 generations, ~2h.
./eval-mccarthy.sh
```

**The design is FROZEN in `scripts/mccarthy-corpus/GATE-PREREG.md`, written and committed
before a single generation existed** (`9c8a4e9`, amended `b4ba731`). Do not edit the arms,
the fixture size or the seeds to chase a result — re-run, do not re-tune.

Three arms: `base` (negative control + voice baseline), `ckpt900` (ship candidate),
`ckpt450` (prior test — it is **not** tied, and the pre-registration says so). Axis A voice
via `voice_distance.py`, axis B memorisation, axis C damage via `score_beats.py`.

| deviation from the lv-hemingway gate | why |
|---|---|
| **fixture `--max-words 140`** (script default 150) | the mccarthy register asks for 90–140 and `score_beats.py` scores in-band at 90–140. 207 of 269 val pairs are in-band, across all six works. |
| **the run/pairs system-prompt equality is ASSERTED**, not claimed in a comment | Hemingway's script said "verified" in prose, which no reader can check. Here a mismatch exits 1. |
| **`--punct-report --secondary-normalised`** | McCarthy-specific; see below. |
| **axis B uses `--train-only --heldout-reference`** | the base-unadapted control is defective; see below. |

### ⚠ The voice axis needed settling before it could run

This corpus measures **0.0 quote marks per 10k** against Hemingway's 838, and `delta_cb` is
Burrows's Delta over CHARACTER BIGRAMS — so "emit no quotation marks" moves the metric a long
way without having learned a sentence. Settled in three parts, all pre-registered:

1. **PRIMARY** — the `mccarthy` register NAMES the punctuation, and `--system-from` drives
   the base control with the same prompt. The cheap win is handed to both sides.
2. **SECONDARY** — `--secondary-normalised` re-runs everything with punctuation stripped. A
   conservative LOWER BOUND (it also strips sentence-length signal); reported, never the verdict.
3. **TRIGGER** — `--punct-report` checks whether base actually complied. Base quote density
   above **100 per 10k** means it did not, the primary gap is partly the cheap win, and the
   normalised read is promoted to load-bearing. The line was fixed before the table printed.

### ⚠ The memorisation control shipped broken and is now fixed

`memorization_check.py`'s negative control was `base-unadapted`, and the lv-hemingway gate
established that this is defective: base writes *summary*, the adapted arms write *pastiche*,
and text that does not imitate a register cannot collide with its n-grams. The correct
innocent sample is **the author himself** — held-out text, same register by construction.
That control was computed by hand for Hemingway and never committed. It is now
`--heldout-reference` (requiring `--train-only`, or the val text is scored against a gram set
containing itself, and the script refuses).

Validated by reproducing Hemingway's hand-computed numbers to the digit — 370 samples,
hit-rate 0.01, mean-longest 0.1, max 10, 101-word median chunks.

⚠ **An elevated rate is not by itself a no-ship — rate and exposure are different questions.**
Every matched run gets READ. Hemingway shipped at 7× the author's own rate because every match
was stock dialogue, max 9 words, no proper noun. **McCarthy is in copyright with a living
estate**, so a match carrying distinctive imagery or a proper noun is disqualifying in a way a
rate number alone is not.

### A log tag that reads backwards on this corpus

`gen_beats_chat_yarros.py` prints `RAN-ON` when its `STOP` regex finds no paragraph break in
the raw output. That heuristic was written for the Yarros register. **McCarthy's register asks
for continuous scene prose, so a single unbroken block is the target, not a defect** — expect
the tag on most generations and do not read it as damage. The axis-C metric is
`score_beats.py --metric-source raw`, where ran-on means `words > 140`, and it is unaffected.