# lv-mccarthy — corpus → gate → pairs

Six works, 167 units, 584,684 words. Built on **pfi-gx10** under `~/lv-mccarthy/`
as `infra-ops`, except **D1, which must run on nh3-dev**: the builder reads the
kvasir catalogue at `/home/lkraven/development/kvasir/data/library/catalog.sqlite`
and that path exists only there.

⚠ **This file exists because the 2026-09-17 D1→D3 session recorded nothing.** There
was no runbook, the commands were issued over non-interactive ssh so no shell history
survived, and a later session had to recover the whole chain by rebuilding candidates
and matching sha256 against the artifacts on disk. Every deviation below is now pinned
by a reproducibility control; keep it that way.

## The chain

```bash
R=~/development/eshpfi-management/scripts        # nh3-dev for D1, ~/lv-mccarthy/scripts on gx10

# D1 — build. ON nh3-dev (needs the kvasir catalogue), then rsync corpus-clean/ to gx10.
python3 $R/mccarthy-corpus/build_corpus_mccarthy.py --out corpus-clean

# D2 — entity map. --min-mid-ratio is what keeps `Yeah`/`Buenas`/`Shh` out; see below.
python3 $R/r49-corpus/entities.py corpus-clean --out corpus-clean/entities.json \
  --stoplist $R/mccarthy-corpus/stoplist_mccarthy.json \
  --min-count 5 --fold-clitics --drop-acronyms --min-mid-ratio 0.2 --min-mid 2

# D2c — the four hand-verified genders.
python3 $R/r49-corpus/apply_gender_overrides.py --entities corpus-clean/entities.json \
  --overrides $R/mccarthy-corpus/gender_overrides_mccarthy.json \
  --out corpus-clean/entities-final.json

# D3 — rename. --scope corpus, --preset mccarthy and --min-cap 5 are all deviations.
python3 $R/r49-corpus/rename.py corpus-clean --entities corpus-clean/entities-final.json \
  --dictionary ~/r49-prep/name_dictionary.json --out corpus-renamed \
  --preset mccarthy --scope corpus --min-cap 5 --copies 6 --seed 4919 \
  --holdout-chapter 7 17

# GATE — must pass before anything is trained. --min-cap 8 here is deliberate; see below.
python3 $R/r49-corpus/leak_gate.py corpus-clean --entities corpus-clean/entities-final.json \
  --renamed corpus-renamed --min-cap 8 --report corpus-renamed/leak_gate_report.json

# D4 — instruction pairs. ON gx10; ~24 min against the `gen` seat.
./run-pairs.sh
```

Gate result, 2026-09-17 (after the separator fix): **0 of 75 renameable, 0 of 37
sub-threshold, 0 separator-split surfaces**, all four controls green. Sensitivity
floor: a name under 8 capitals per work is never detected, a phrase under 5
recurrences never audited, and **no phrase map exists for this corpus, so the phrase
audit does not run at all** — the Yarros and Brontë runs both had one.

## The deviations, and what forced each

| deviation | why |
|---|---|
| **D1 runs on nh3-dev** | The builder reads the kvasir catalogue by absolute path. gx10 has no copy. |
| **`--min-count 5`** (was 8) | The entity map's floor has to match rename's, or entities between the two floors sit in the map, are never renamed, and count as leaks. The first D3 gate run failed with 45 survivors for exactly this. |
| **`--min-mid-ratio 0.2 --min-mid 2`** | Without it the map admits `Yeah`, `Buenas`, `Shh`, `Mande`, `Sí`, `Git` — words that are only ever capitalised at a sentence start. ⚠ The map is INSENSITIVE to the exact value: any ratio in **[0.05, 0.3]** with `--min-mid` 1 or 2 reproduces it byte-for-byte. The original run's values are unrecoverable and it does not matter. `--min-mid 3` does NOT reproduce it. |
| **`--fold-clitics --drop-acronyms`** | Both are needed to reproduce the map; `--rescue-honorific` is inert here (0 rescued). |
| **`--scope corpus`** (was `work`) | Nine surfaces appear in more than one work — Parham, Grady, Cole, Socorro, Héctor. A per-work map gives John Grady a different invented name in each Border Trilogy novel. |
| **`--preset mccarthy`** | Hemingway's romance pool carries `it_IT`/`fr_FR` for his Italian and French casts; McCarthy writes neither. `en_US` + `es_MX`/`es_ES` at an even share. |
| **`--holdout-chapter 7 17`** (a LIST) | The val split is one chapter per work, so its size scales with WORK COUNT, not corpus size. Six works would have given a Brontë-class ~18,000-word val reference; two indices give 11 units and 40,653 words per copy, larger than Hemingway's, for 7% of the corpus. |
| **gate at `--min-cap 8`, rename at `--min-cap 5`** | Not a mistake. The gate reports the 37 entities rename never touched *separately*, which is strictly more informative than running both at 5 (where the sub-threshold bucket is empty). |
| **D4 `--reflow-hard-wraps`** | The Road is hard-wrapped at ~76 characters and the other five works are not. See DEFECT 4 in `build_sft_pairs.py`. |
| **D4 `--source-entities`** | Mandatory. The beat generator reads the passage and will supply canonical names from its own memory of the book; the corpus gate never reads the generated beats. |

## The defect the gate could not see, and now can

⚠⚠ **The 2026-09-17 tree passed this gate at "0 of 75 renameable and 0 of 37
sub-threshold" while carrying 13 occurrences of five protagonist names in all six
copies.** `leak_gate.py` scanned `\b(Surface)\b`, and a character inserted inside a
name defeats that pattern outright — so the occurrence was unrenameable by `rename.py`
*and* unreportable by the gate. Two mechanisms, both in the source extraction:

```
B ell  C higurh  M oss  T oadvine     a small-caps drop cap survived as its own token
Toad-vine  Glan-ton                   a print line-break hyphen survived the extraction
```

Every *visible* occurrence had been renamed correctly, which is what made the residue
invisible to any spot-read: the names are gone everywhere you look.

Fixed in three places, and all three must stay:

1. **`build_corpus_mccarthy.py` rules 4 and 5** repair the text at its source — 32
   split initials with a lowercase remainder, 5 hyphen-split names, both with expected
   counts so a master change fails the build. ⚠ Rule 4's letter class is **consonants
   only**: `I` opens 1,966 paragraphs, `A` opens 143 and `Y` opens 32 (Spanish *y*), and
   folding any of them would corrupt 2,141 lines to fix 32.
2. **`leak_gate.py --split-frag-max`** runs a separator-tolerant scan every time, with
   its own positive and negative controls, and **fails the gate**. Verified against the
   pre-fix tree: it reports all five surfaces and exits 1.
3. The exact-match passes are unchanged, so the old verdict is reproduced alongside.

Cross-checked on the two shipped corpora: **lv-bronte is clean** of this class;
**lv-hemingway carries 3** (`Primi tivo`, `Pasionar ia`, `Chi cote`) and is live.

## Corpus properties worth knowing before you change anything

- **`repair_typography.py` MUST NOT be run on this corpus.** It normalises "toward what
  the text does" and would put quotation marks back into a corpus that measures 0.0 per
  10k against Hemingway's 838. The builder runs no normalisation and asserts the density.
- **Back matter is stripped BEFORE the split**, inverting the Hemingway order: Blood
  Meridian and The Crossing end with a dumped table of contents made of bare roman
  numerals, which is the exact shape of a chapter marker.
- **Blood Meridian sets a dash-separated chapter argument under each roman numeral.**
  131 paragraphs; `--drop-leading-heading` now consumes them.
- Two truncated catalogue rows are excluded in favour of complete siblings.