vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity

BabyYarros: corpus built, gender resolution fixed, rename blocked on leak gate

Browse Source 
Located the source: five Rebecca Yarros works in the Kvasir licensed library, with
rights recorded as gated. Built D1 at 208 chapters and 780,744 words, which is 15%
larger than the Brontë corpus. No unwrap step was needed because Kvasir's cleaner
already emits flowing paragraphs, so the hard-wrap defect that cost a re-cut on
Brontë does not exist here. The alphabet was re-derived rather than inherited: 23
non-ASCII letters across three forms, against F02's 4 on a smaller sample. Same
ASCII-fold conclusion from a different measurement, which is the reason to re-derive
per corpus.

The interesting finding is a new pathology. In a rotating first-person POV corpus,
every book's narrator gets the wrong gender. Measured against six names verified in
the text, the pronoun resolver called Violet male, Leah male and Landon female --
three of eighteen wrong, and all three are the narrator of the book where they were
misgendered. A narrator is "I" in her own book, so her name appears mostly inside
the other lead's dialogue surrounded by his pronouns. This is Brontë's "Jane called
male" amplified by rotating POV. Title-first resolution, which fixed it for Brontë,
is nearly blind here because contemporary romance uses given names rather than
honorifics. What works is the POV header: resolve each name from the chapters it
does not narrate. Validated at 9 correct, 9 held, 0 wrong against the previous 7, 8
and 3 wrong, and the instrument refuses to write unless it beats what it replaces.

Re-pointing rename.py surfaced three bugs, two of which would have silently
corrupted the corpus. Gender came only from honorifics and the entities file's
gender field was ignored, so the POV fix had no effect until wired through; that
took wilder from 1 gendered entity to 13. The pool labels were hardcoded in a print
statement, so any non-Brontë preset crashed. And the collision-filter log claimed
it dropped names colliding with Brontë entities regardless of which corpus it
filtered against -- the logic was right but the message named the wrong corpus,
which is how a reader later concludes the filter ran on the wrong thing.

D3 is blocked and nothing has been trained. The leak gate shows 86 of 232
renameable source entities surviving where the Brontë run reached 0 of 203. It
decomposes into detector false positives that need a stopword filter rather than
renaming, genuine misses among worldbuilding proper nouns, and a third class whose
cause is not yet established. Training before the gate passes means fitting
in-copyright text with 86 identifiable source entities intact, in a corpus F02
already flagged as small enough for leak to be a real concern.
This commit is contained in:
vh
2026-09-11 08:46:45 -07:00
parent e15c5ee5ea
commit 6dba912324
6 changed files with 408 additions and 10 deletions
Download Patch File Download Diff File Expand all files Collapse all files
scripts/r49-corpus/rename.py
+32 -10
View File
@@ -36,6 +36,19 @@ ENGLISH_LOCALES = ["en_GB", "en_IE"]
#: orthography -- a Yorkshire mill town full of Parisian surnames reads wrong.
FRENCH_SHARE = {"villette": 0.60, "the-professor": 0.60, "jane-eyre": 0.25, "shirley": 0.25}
#: The pool is now per-corpus rather than per-author-hardcoded, because the same
#: register argument points somewhere else for every corpus. Brontë EXCLUDES en_US
#: (modern surnames read wrong for the 1840s); contemporary American romance wants
#: exactly those, with the European admixture F02 found matches Yarros's register.
#: Defaults reproduce the Brontë run byte-for-byte, so this is additive.
PRESETS = {
"bronte": {"a": ("fr", FRENCH_LOCALES), "b": ("en", ENGLISH_LOCALES),
"share": FRENCH_SHARE, "default_share": 0.25},
"yarros": {"a": ("us", ["en_US", "en_CA"]),
"b": ("eu", ["es_ES", "es_MX", "it_IT", "de_DE", "fr_FR"]),
"share": {}, "default_share": 0.62},
}
def title_gender(text: str) -> dict[str, str]:
mt = collections.Counter(m.group(1).lower() for m in
@@ -52,10 +65,11 @@ def title_gender(text: str) -> dict[str, str]:
return out
def build_pool(dict_path: Path, alphabet: set[str]) -> dict:
def build_pool(dict_path: Path, alphabet: set[str], preset: str = "bronte") -> dict:
d = json.loads(dict_path.read_text())
pool = {}
for label, locales in (("fr", FRENCH_LOCALES), ("en", ENGLISH_LOCALES)):
cfg = PRESETS[preset]
for label, locales in (cfg["a"], cfg["b"]):
m, f, s = set(), set(), set()
for loc in locales:
v = d["by_locale"].get(loc, {})
@@ -82,13 +96,17 @@ def main() -> int:
ap.add_argument("--copies", type=int, default=6)
ap.add_argument("--seed", type=int, default=4919)
ap.add_argument("--holdout-chapter", type=int, default=10)
ap.add_argument("--preset", default="bronte", choices=sorted(PRESETS),
help="which corpus's name-pool register to draw from")
a = ap.parse_args()
corpus = Path(a.corpus)
man = json.loads((corpus / "manifest.json").read_text())
alphabet = set(json.loads((corpus / "corpus_alphabet.json").read_text())["letters"])
ents_all = json.loads(Path(a.entities).read_text())
pool = build_pool(Path(a.dictionary), alphabet)
pool = build_pool(Path(a.dictionary), alphabet, a.preset)
cfg = PRESETS[a.preset]
label_a, label_b = cfg["a"][0], cfg["b"][0]
# ⚠ Collision filter, against THIS corpus. F02 dropped 35 names for colliding
# with the Yarros source so a rename could never map one of the author's
# entities onto another; that filter is corpus-specific and does not carry.
@@ -113,10 +131,11 @@ def main() -> int:
if n not in source_names
and not (set(re.split(r"[-\s’']", n)) & source_names)]
dropped += before - len(pool[lang][bucket])
print(f" collision filter: dropped {dropped} pool names that are Bronte entities")
print(f" pool (alphabet-filtered): "
f"fr {len(pool['fr']['male'])}m/{len(pool['fr']['female'])}f/{len(pool['fr']['surname'])}s "
f"en {len(pool['en']['male'])}m/{len(pool['en']['female'])}f/{len(pool['en']['surname'])}s")
print(f" collision filter: dropped {dropped} pool names that collide with "
f"{len(source_names)} source entities in THIS corpus ({a.preset})")
print(" pool (alphabet-filtered): " + " ".join(
f"{lab} {len(pool[lab]['male'])}m/{len(pool[lab]['female'])}f/{len(pool[lab]['surname'])}s"
for lab in (label_a, label_b)))
out = Path(a.out); (out / "copies").mkdir(parents=True, exist_ok=True)
stats = {"copies": a.copies, "seed": a.seed, "works": {}, "renamed": 0, "held": 0}
@@ -137,7 +156,7 @@ def main() -> int:
for key, e in ents.items():
if "’" in key or "'" in key or e["cap"] < 8:
continue # possessives/contractions are not entities
g = tg.get(key)
g = tg.get(key) or e.get("gender")
if g:
renameable[key] = {"surface": e["surface"], "kind": "given", "gender": g}
else:
@@ -162,11 +181,14 @@ def main() -> int:
for c in range(a.copies):
rng = random.Random(a.seed + c * 1000)
for slug, rows in works.items():
fr_share = FRENCH_SHARE[slug]
# Share of pool A for this work. Brontë sets it per novel (Brussels
# vs Yorkshire); Yarros uses one default, because the register does
# not split by book the way hers does.
share_a = cfg["share"].get(slug, cfg["default_share"])
used = set()
def draw(kind: str, gender: str | None) -> str:
lang = "fr" if rng.random() < fr_share else "en"
lang = label_a if rng.random() < share_a else label_b
bucket = {"m": "male", "f": "female"}.get(gender or "", "surname")
for _ in range(200):
n = rng.choice(pool[lang][bucket])