vh
7b0580dcbe
The gate is new. There was no committed instrument for "does any of the author's own proper nouns survive the rename" -- the Brontë number was produced by hand -- so leak_gate.py is now that instrument, and it runs both directions every time: the same scan over the unrenamed source as a positive control, and a nonce string as a negative one. A detector that only ever sees renamed text cannot distinguish absent from blind. Run against BabyYarros as built it reported 212 surviving entities, not the 86 recorded earlier, because it scans the whole corpus rather than each work separately and it counts the sub-threshold entities rename never looked at. Three findings came out of closing that. The corpus had a typography defect of its own. The D1 notes correctly say no unwrap was needed; a different defect was there instead. The Empyrean books set their chapter epigraphs in small caps and the extractor rendered the run as uppercase while leaving the large initial as a separate token, so the corpus carried "M AJOR A FENDRA'S G UIDE TO THE R IDERS Q UADRANT" -- 106 lines, ~700 splits -- plus 52 drop caps like "T he flight field". That is where the entities called IDERS, UADRANT, NAUTHORIZED and seventeen bare single letters came from. A split initial next to an uppercased run is enough to recover the original mixed case, so the restore is exact rather than approximate: a word with a split initial was capitalised, an all-caps word without one was lowercase. Back matter was inside the prose. The builder splits on chapter headings and nothing follows the last one, so every work carried its acknowledgments, newsletter pitches and cover-artist credits -- 4,555 words naming the author's agent, editors and children, in a corpus whose entire purpose is that no identifiable name survives. And the gate passed at 0 of 314 while Afendra was still in every copy. The name never appears unpossessed, so it keyed as an apostrophe form, and rename and the gate both skip those as contractions -- unrenamed and unreported at once, which is the worst failure shape available. Baxter escaped a different way: wilder renders an in-book news article entirely in lowercase, putting the cap/lowercase ratio at 0.13 against a 0.05 bar. Then a second class the unigram scan structurally cannot see. Riders Quadrant, Flame Section, War Games and Fourth Wing -- the book's own title -- are built from ordinary words the detector correctly refuses to call names. The gate now audits recurring capitalised 2-3grams against an explicit allow list, and rename applies a phrase map after the entity pass. Every new detector flag is opt-in and off by default, and the Brontë entity map was re-derived after each change and confirmed identical in keys, surfaces and every field. The stoplist was built by reading each surface in context, which is why it is short: Violence is Xaden's nickname for Violet, and Continent, Presentation, Barrens, Originals, Montserrat, Athena, Aura, Curator and Sage are all in-world. A plausible-looking guess would have excluded most of them. Final: 0 of 325 entities and 0 of 91 audited phrases survive in any of 30 copy files, both controls passing. The sensitivity floor is stated in the gate's own output -- 3 occurrences for a name, 5 for a phrase -- because a negative without one is unfalsifiable.
4.6 KiB
4.6 KiB
BabyYarros — corpus pipeline, end to end
Every stage is reproducible from the Kvasir masters. Run them in order; each one refuses to write if its own controls fail, so a silent bad stage is not a failure mode here.
$R is this repo's scripts/ directory. Work happens on nh3-dev (it has the
Kvasir library); training happens on pfi-gx10.
# D1 — build from the licensed masters
python3 $R/yarros-corpus/build_corpus_yarros.py --out ~/yarros-corpus
# D1b — repair EPUB typography, strip back matter
# small-caps epigraphs, drop caps, acknowledgments/newsletter tails
python3 $R/yarros-corpus/repair_typography.py ~/yarros-corpus --out ~/yarros-corpus-r
# D2 — entity map. Every flag below is OPT-IN and off by default so the Brontë
# run stays byte-reproducible; all five are needed for this corpus.
python3 $R/r49-corpus/entities.py ~/yarros-corpus-r \
--out ~/yarros-corpus-r/entities.json \
--min-count 3 --min-mid-ratio 0.35 --drop-acronyms --fold-clitics \
--rescue-honorific 2 --stoplist $R/yarros-corpus/stoplist_yarros.json \
--control "Violet,Xaden,Basgiath,Tairn,Leah,Landon,Brennan,Mira,Rhiannon,Dain,Jesinia,Bodhi,Garrick,Imogen,Sorrengail,Riorson,Delgado,Schur,Rhee,Messina,Masen,Violence,Montserrat,Barrens,Originals,Lilith,Nyra,Naolin,Afendra,Baxter" \
--negative-control "Hey,Holy,Hopefully,Yep,Hi,Whoa,Nope,Ugh,Ouch,Okay,Yeah,Thankfully,Honestly,Seriously,Hmm,Jesus,Logically,Chapter,Article,TV,VIP,ATV,BMX,Google,Nepal,American,Pacific,Harvard,Sergeant,Majesty,YouTube,Colorado"
# D2b — gender, resolved from the chapters a name does NOT narrate
python3 $R/yarros-corpus/pov_gender.py ~/yarros-corpus-r \
--entities ~/yarros-corpus-r/entities.json --out ~/yarros-corpus-r/entities-pov.json \
--control "Violet=f,Xaden=m,Leah=f,Landon=m,Rhiannon=f,Dain=m,Mira=f,Brennan=m,Imogen=f,Paxton=m,Rachel=f,Penna=f,Nick=m,Liam=m,Sloane=f,Bodhi=m,Garrick=m,Jesinia=f"
# D3 — rename, ONE map per copy across the whole corpus
python3 $R/r49-corpus/rename.py ~/yarros-corpus-r \
--entities ~/yarros-corpus-r/entities-pov.json \
--dictionary ~/r49-prep/name_dictionary.json \
--out ~/yarros-corpus-renamed-v2 --preset yarros --scope corpus --min-cap 3 \
--copies 6 --seed 4919 --phrase-map $R/yarros-corpus/phrase_map_yarros.json
# GATE — must pass before anything is trained
python3 $R/r49-corpus/leak_gate.py ~/yarros-corpus-r \
--entities ~/yarros-corpus-r/entities-pov.json --renamed ~/yarros-corpus-renamed-v2 \
--min-cap 3 --phrase-map $R/yarros-corpus/phrase_map_yarros.json \
--report ~/yarros-corpus-renamed-v2/leak_gate_report.json
Why each opt-in flag exists
Each one was added because the gate caught something, and each is measured, not assumed. All five default to OFF, and the Brontë entity map was re-derived after every change and confirmed identical in keys, surfaces and every field.
| flag | the defect it fixes | evidence |
|---|---|---|
--min-mid-ratio 0.35 |
Hey, Holy, Hopefully, Yep, Whoa, Nope, Ugh were entities |
33 verified names sit at 0.567–0.985 mid-sentence, 19 verified interjections at 0.000–0.222 |
--drop-acronyms |
TV, VIP, ATV, BMX, RSC renamed to surnames |
tested on the DOMINANT surface, so Braevick inside an all-caps dispatch is not lost |
--fold-clitics |
Afendra never appears unpossessed, so it keyed as Afendra’s — which rename AND the gate both skip |
it survived every copy while the gate read 0 of 314 |
--rescue-honorific 2 |
Baxter rejected at ratio 0.13 because an in-book news article is set all-lowercase |
ms. baxter ×14; the wide honorific list matched case-insensitively readmitted 143 junk tokens, so the rescue list is 5 abbreviations and the lowercase arm requires the period |
--stoplist |
real-world referents renamed (Google, Nepal, American) |
every surface read in context first — Violence is Xaden's nickname for Violet and would have been wrongly excluded by a guess |
--scope corpus and --phrase-map are rename-side, same shape:
--scope corpus— per-work maps leak across works (Rebelrenamed inrebel, printed verbatim in the other two Renegades books) and give one character two names inside a single copy. Yarros is two series; Brontë was four unrelated novels.--phrase-map— the unigram pass cannot reachRiders Quadrant,Flame SectionorFourth Wing, the book's own title, because every component is an ordinary word.
Sensitivity floor
⚠ The gate resolves leak down to 3 capitalised occurrences per work for names and 5 recurrences for phrases. Below those it does not detect, does not rename, and does not report. "0 survive" means zero above that floor.