# BabyYarros — corpus pipeline, end to end

Every stage is reproducible from the Kvasir masters. Run them in order; each one
refuses to write if its own controls fail, so a silent bad stage is not a
failure mode here.

`$R` is this repo's `scripts/` directory. Work happens on **nh3-dev** (it has the
Kvasir library); training happens on **pfi-gx10**.

```bash
# D1 — build from the licensed masters
python3 $R/yarros-corpus/build_corpus_yarros.py --out ~/yarros-corpus

# D1b — repair EPUB typography, strip back matter
#   small-caps epigraphs, drop caps, acknowledgments/newsletter tails
python3 $R/yarros-corpus/repair_typography.py ~/yarros-corpus --out ~/yarros-corpus-r

# D2 — entity map. Every flag below is OPT-IN and off by default so the Brontë
#      run stays byte-reproducible; all five are needed for this corpus.
python3 $R/r49-corpus/entities.py ~/yarros-corpus-r \
  --out ~/yarros-corpus-r/entities.json \
  --min-count 3 --min-mid-ratio 0.35 --drop-acronyms --fold-clitics \
  --rescue-honorific 2 --stoplist $R/yarros-corpus/stoplist_yarros.json \
  --control "Violet,Xaden,Basgiath,Tairn,Leah,Landon,Brennan,Mira,Rhiannon,Dain,Jesinia,Bodhi,Garrick,Imogen,Sorrengail,Riorson,Delgado,Schur,Rhee,Messina,Masen,Violence,Montserrat,Barrens,Originals,Lilith,Nyra,Naolin,Afendra,Baxter" \
  --negative-control "Hey,Holy,Hopefully,Yep,Hi,Whoa,Nope,Ugh,Ouch,Okay,Yeah,Thankfully,Honestly,Seriously,Hmm,Jesus,Logically,Chapter,Article,TV,VIP,ATV,BMX,Google,Nepal,American,Pacific,Harvard,Sergeant,Majesty,YouTube,Colorado"

# D2b — gender, resolved from the chapters a name does NOT narrate
python3 $R/yarros-corpus/pov_gender.py ~/yarros-corpus-r \
  --entities ~/yarros-corpus-r/entities.json --out ~/yarros-corpus-r/entities-pov.json \
  --control "Violet=f,Xaden=m,Leah=f,Landon=m,Rhiannon=f,Dain=m,Mira=f,Brennan=m,Imogen=f,Paxton=m,Rachel=f,Penna=f,Nick=m,Liam=m,Sloane=f,Bodhi=m,Garrick=m,Jesinia=f"

# D3 — rename, ONE map per copy across the whole corpus
python3 $R/r49-corpus/rename.py ~/yarros-corpus-r \
  --entities ~/yarros-corpus-r/entities-pov.json \
  --dictionary ~/r49-prep/name_dictionary.json \
  --out ~/yarros-corpus-renamed-v2 --preset yarros --scope corpus --min-cap 3 \
  --copies 6 --seed 4919 --phrase-map $R/yarros-corpus/phrase_map_yarros.json

# GATE — must pass before anything is trained
python3 $R/r49-corpus/leak_gate.py ~/yarros-corpus-r \
  --entities ~/yarros-corpus-r/entities-pov.json --renamed ~/yarros-corpus-renamed-v2 \
  --min-cap 3 --phrase-map $R/yarros-corpus/phrase_map_yarros.json \
  --report ~/yarros-corpus-renamed-v2/leak_gate_report.json
```

## Why each opt-in flag exists

Each one was added because the gate caught something, and each is measured, not
assumed. All five default to OFF, and the Brontë entity map was re-derived after
every change and confirmed identical in keys, surfaces and every field.

| flag | the defect it fixes | evidence |
|---|---|---|
| `--min-mid-ratio 0.35` | `Hey`, `Holy`, `Hopefully`, `Yep`, `Whoa`, `Nope`, `Ugh` were entities | 33 verified names sit at 0.567–0.985 mid-sentence, 19 verified interjections at 0.000–0.222 |
| `--drop-acronyms` | `TV`, `VIP`, `ATV`, `BMX`, `RSC` renamed to surnames | tested on the DOMINANT surface, so `Braevick` inside an all-caps dispatch is not lost |
| `--fold-clitics` | `Afendra` never appears unpossessed, so it keyed as `Afendra’s` — which rename AND the gate both skip | it survived every copy while the gate read 0 of 314 |
| `--rescue-honorific 2` | `Baxter` rejected at ratio 0.13 because an in-book news article is set all-lowercase | `ms. baxter` ×14; the wide honorific list matched case-insensitively readmitted 143 junk tokens, so the rescue list is 5 abbreviations and the lowercase arm requires the period |
| `--stoplist` | real-world referents renamed (`Google`, `Nepal`, `American`) | every surface read in context first — `Violence` is Xaden's nickname for Violet and would have been wrongly excluded by a guess |

`--scope corpus` and `--phrase-map` are rename-side, same shape:

- **`--scope corpus`** — per-work maps leak across works (`Rebel` renamed in `rebel`,
  printed verbatim in the other two Renegades books) and give one character two
  names inside a single copy. Yarros is two *series*; Brontë was four unrelated novels.
- **`--phrase-map`** — the unigram pass cannot reach `Riders Quadrant`, `Flame Section`
  or `Fourth Wing`, the book's own title, because every component is an ordinary word.

## Sensitivity floor

⚠ The gate resolves leak down to **3 capitalised occurrences per work** for names
and **5 recurrences** for phrases. Below those it does not detect, does not
rename, and does not report. "0 survive" means zero above that floor.