# lv-bronte — corpus pipeline, end to end

Charlotte Brontë: *Jane Eyre*, *Villette*, *Shirley*, *The Professor*. Four
unrelated novels, 680,291 words as built, **678,367 after repair**. Source is
Project Gutenberg, so unlike the Yarros and Hemingway lines there is no licensed
master and no distribution question.

`$R` is this repo's `scripts/`. Everything below runs on **pfi-gx10**
(`10.100.50.60`), which also holds the carrier and does the training.

Every stage refuses to write if its own controls fail, so a silent bad stage is
not a failure mode here. Two stages actually did refuse during the first run, and
both refusals were correct — see *What the controls caught*.

```bash
# D1b — repair the Gutenberg source. NOT optional; see below.
python3 $R/bronte-corpus/repair_corpus_bronte.py ~/r49-corpus --out corpus-clean

# D2 — entity map.
python3 $R/r49-corpus/entities.py corpus-clean --out entities.json \
  --stoplist $R/bronte-corpus/stoplist_bronte.json \
  --min-count 3 --min-mid-ratio 0.35 --fold-clitics --drop-acronyms \
  --control "Jane,Rochester,Thornfield,Lowood,Helen,Bessie,Adèle,Diana,Bertha,Lucy,Ginevra,Graham,Shirley,Caroline,Robert,Louis,Hortense,Yorke,Crimsworth,Frances,Hunsden,Pelet,Reed,Brocklehurst,Fairfax,Moore,Keeldar,Helstone,Malone,Sympson,Villette,Paulina,Terrasse,Vashti,Labassecour,Ferndean" \
  --negative-control "Indeed,Perhaps,However,Nevertheless,Certainly,Chapter,Yes,Well,Nothing,Something,Truly,Presently,Meantime,Again,Once,Besides,Moreover,Whether,Mdlle,Rue,Shakespeare,India,Dieu,Jesuit"

# D2b — gender by proximity, scored against the CORPUS BASE RATE.
python3 $R/hemingway-corpus/gender_by_proximity.py corpus-clean \
  --entities entities.json --out entities-gender.json --control "Jane=f,Rochester=m,…"

# D2c — pin hand-verified ground truth. Brontë-specific; see below.
python3 $R/bronte-corpus/pin_known_gender.py \
  --entities entities-gender.json --out entities-final.json --pin "Jane=f,Lucy=f,…,Moore=n,Helstone=n"

# D3 — rename. NOTE --scope corpus and --min-cap 3, both deviations.
python3 $R/r49-corpus/rename.py corpus-clean --entities entities-final.json \
  --dictionary ~/r49-prep/name_dictionary.json --out corpus-renamed \
  --preset bronte --scope corpus --min-cap 3 --copies 6 --seed 4919 \
  --phrase-map $R/bronte-corpus/phrase_map_bronte.json

# GATE — must pass before anything is trained.
python3 $R/r49-corpus/leak_gate.py corpus-clean --entities entities-final.json \
  --renamed corpus-renamed --min-cap 3 \
  --phrase-map $R/bronte-corpus/phrase_map_bronte.json \
  --report corpus-renamed/leak_gate_report.json

# D4 — instruction pairs. --source-entities is MANDATORY here; see below.
python3 $R/yarros-corpus/build_sft_pairs.py --corpus corpus-renamed/copies \
  --out pairs/pairs-full.jsonl --register bronte --model gen \
  --source-entities entities-final.json --n 4093 --split train --seed 4919
#   …and the same with --split val --n 200 --out pairs/pairs-val.jsonl
```

Gate result, 2026-09-16: **0 of 365 renameable surfaces survive**, positive
control 365/365, negative control clean, phrase audit 0 of 82. Sensitivity floor:
a name under 3 capitals per work is never detected, a phrase under 5 recurrences
never audited.

## Why this corpus needed its own pipeline

Five deviations from the Yarros/Hemingway chain, each forced by a measurement.

| deviation | why |
|---|---|
| **D1b repair exists at all** | 1,922 words of publisher back matter in Shirley's last unit and 1,368 Gutenberg `_italic_` spans. See below. |
| **`--min-count 3` / `--min-cap 3`** (was 8) | At 8, `Bertha`, `Ferndean`, `Rochesters` and `Creemsvort` sat below the floor — detected by nothing, renamed by nothing, reported by nothing. |
| **`--scope corpus`** (was `work`) | `Jane` is above threshold in *Jane Eyre* and *The Professor* but appears 5 times in *Shirley*. Work-scope renames it in two novels and prints it verbatim in the third. The gate is a whole-corpus scan and would have caught it; corpus scope means it never happens. |
| **D2c `pin_known_gender.py`** | The resolver puts `Jane` MALE across 336 occurrences. Structural, not tuning — see below. |
| **`--source-entities` on the pair build** | A leak class the gate cannot see — see below. |

## The two source defects

**Publisher back matter, Shirley only, 1,922 words.** A T. Nelson & Sons printer's
block, a `THE NELSON CLASSICS` catalogue advertising Ainsworth, Marryat, Verne,
Kingsley and Dickens, and a Gutenberg transcriber's punctuation list — all inside
the final chapter record. Not Brontë, and the origin of the entity `CHARLES` (12
occurrences of another author's given name, in the training text). Structurally
identical to the Hemingway back-matter defect and it arises the same way: a
splitter cuts on headings and nothing follows the final one. The other three works
are clean, which the repair script's per-work report shows as an implicit control.

**Gutenberg `_italic_` markup, 1,368 spans.** Two harms. It teaches the adapter to
emit underscores as prose. And `_` is a **word character**, so the gate's
`\b(Name)\b` scan cannot match inside `_Antigua_` — an entity that appears only in
italics is invisible to the gate, never renamed and never reported, which is the
same failure shape as Yarros's possessive-only `Afendra`. Measured blast radius
here is exactly one entity, but that is a fact about this corpus, not a property of
the defect. ⚠ 24 of the spans wrap across a single newline (`_sotto\nvoce_`,
`_ignis\nfatuus_`); a newline-free pattern silently leaves exactly those behind,
and the script's read-back guard is what caught it.

## Why gender needs a separate pinning stage

The inherited honorific/window resolver scores 19 correct / 5 held / **5 wrong**.
Hemingway's base-rate resolver is strictly better on the axis that matters — 17
correct / 11 held / **1 wrong** — because a wrong gender actively corrupts pronoun
agreement while a held one merely yields a neutral name. Brontë's corpus is nearly
pronoun-balanced (15,588 male / 16,144 female, base_m 0.491), unlike Hemingway's
near-4:1, so the base-rate correction does less work here but still helps.

The one it still gets wrong is **`Jane`, 341 occurrences, the title character and
narrator**. This is not a tuning miss. Brontë's three narrators are first-person,
so their names appear almost only in *dialogue*, spoken by other characters,
surrounded by those characters' pronouns — proximity inference is structurally
blind to exactly the characters the adapter is being trained on. Yarros solved its
version with the POV chapter header; Brontë's editions have none, so that fix does
not transfer any more than it transferred to Hemingway.

⚠ **Keep the resolver's evaluation and the shipped map separate.** `pin_known_gender.py`
does not improve the resolver's score and must never be quoted as if it had. It
applies ground truth a human read out of the novels. It also fails loudly on a pin
that matches no entity, which caught a stale `Hall=n` pin on the first run.

Shared surnames are pinned **neutral** on purpose: `Moore` is Robert, Louis *and*
Hortense; `Helstone` is Matthewson, James and Caroline; `Yorke` is Hiram, Mrs
Yorke, Jessy, Rose and Martin. A gendered rename of a surname shared across sexes
is wrong in one direction no matter which you pick.

## ⭐ The leak the gate cannot see

**The corpus gate reads the corpus and the renamed copies. It never reads the
generated beats.** The beat is written by an LLM that just read the passage, and
if it recognises the book it supplies the canonical names from its own training.

Measured on the first 714 Brontë pairs, before the filter existed: **13 beats
(1.8%) named source characters** — `Rochester` ×6, `Jane` ×3, `Brocklehurst` ×2,
`Beck`, `Fairfax`, `Helen`, `Burns`, `Eyre`, `Reed`, `Rivers` — while **0 of 714
responses did**. The rename was perfect and the instruction side was not. One beat
read *"Saoirse confirms Rochester's flaws, then agrees in English to marry him"*,
mixing a renamed name and a canonical one in a single sentence.

The beat is the INSTRUCTION half of the pair, so training on it re-teaches exactly
the inventions the rename exists to remove.

⚠ **Exposure scales with how well the generator knows the book**, so it is worst
for public-domain classics and mildest for recent work. That is why Yarros and
Hemingway came up clean and Brontë did not — **their clean runs are not evidence
this cannot happen to them.** Pass `--source-entities` on every corpus, and rebuild
those two with it if they are ever regenerated.

## What the controls caught

Worth recording, because in three of four cases the control was wrong and the
detector was right — the opposite of the reflex.

- **`Adele` NOT DETECTED.** The corpus spells it `Adèle` (135×). My control string
  was ASCII. Detector right.
- **`Hollow` NOT DETECTED.** 85 capitalised against 20 genuinely lowercase
  (`rushing down the hollow`), ratio 0.235. Detector right; `Hollow's Mill` belongs
  in the phrase map, which is where it went.
- **`Grace` NOT DETECTED.** 0.224 lowercase ratio — `grace` is a common noun.
  Detector right. But sampling all 21 bare capitalised `Grace` found 20 are the
  character in direct address and one is theological, so it is token-mapped, with
  that single collateral substitution accepted and recorded.
- **`Blanche` NOT DETECTED — and here the detector was wrong.** 19 capitalised
  against **one** lowercase gives 0.0526, over the 0.05 bar by a single token. A
  named character with 19 mentions, dropped by a hair. Token-mapped.
- **`Antigua` missing from the positive control at gate time.** This is what
  exposed the italics defect above.