bronte-corpus: gate lv-bronte for real — 0 of 365 with both controls green
The Brontë corpus's "0 of 203" was a HAND COUNT made before leak_gate.py
existed. On Yarros the automated instrument read 212 surviving where a hand
count said 86, so the hand count was never evidence. This runs the real gate,
and getting it to pass required fixing four defects the hand count could not
have seen.
CORPUS DEFECTS (repair_corpus_bronte.py, both measured):
- 1,922 words of publisher back matter inside Shirley's last unit — a
T. Nelson & Sons catalogue advertising Ainsworth, Marryat, Verne, Kingsley
and Dickens, plus a Gutenberg transcriber's punctuation list. Not Brontë,
and the source of the entity CHARLES. Same structural cause as the
Hemingway run: a splitter cuts on headings, nothing follows the final one.
- 1,368 Gutenberg italic spans. Two harms: they teach the adapter to emit
underscores, and the underscore is a word character, so the gate's
word-boundary scan cannot match inside an italicised name. An entity in
italics is invisible to the gate — the same never-renamed-AND-never-
reported shape as Yarros's possessive-only Afendra.
DETECTOR GAPS (phrase_map_bronte.json):
- Blanche is 19 capitalised against ONE lowercase — ratio 0.0526, over the
0.05 bar by a single token, so a named character with 19 mentions is
dropped by a hair.
- Grace (0.224) and Hollow (0.235) are refused correctly — both are common
nouns — but Grace Poole and Hollow's Mill are Brontë's. Sampling all 21
bare capitalised Grace found 20 are the character in direct address and
exactly one is the theological noun.
- Five compounds whose every component is non-renameable survive verbatim:
Moor House, Marsh End, Vale Hall, Bigben Close, Royd Lane. The other 77
audited phrases do not, because each has a renameable component.
GENDER (pin_known_gender.py): the inherited resolver put Jane MALE across 336
occurrences. Hemingway's base-rate resolver is strictly better here (1 wrong vs
4) but still fails on Jane, and the failure is structural, not tuning — Brontë's
three narrators are first-person, so their names appear almost only in dialogue
surrounded by other characters' pronouns. Ground truth is pinned separately from
the resolver's evaluation so the two are never conflated.
Also: min-cap lowered 8 to 3, which pulled Bertha, Ferndean, Rochesters and
Creemsvort in from below the old floor; corpus-scope rename so a name below
threshold in one novel is not printed verbatim there while renamed in another.
Gate: 0 of 365 surviving, positive control 365/365, negative control clean,
phrase audit 0 of 82. Floor stated: 3 capitals per work, 5 recurrences.
This commit is contained in:
@@ -0,0 +1,29 @@
|
||||
{
|
||||
"corpus": "lv-bronte (Charlotte Brontë — Jane Eyre, Villette, Shirley, The Professor)",
|
||||
"why": "The unigram rename cannot reach an in-world compound built out of ordinary words, and it cannot reach a character whose given name is also an English noun. Both classes survive a gate that reads 0 of 203, for the same reason `Riders Quadrant` and `Fourth Wing` survived Yarros's: the cap/lowercase detector correctly refuses to call `Moor`, `Marsh`, `Vale` or `Grace` a name, so rename never sees them.",
|
||||
"how_derived": "Measured, not guessed. Every capitalised 2-3gram recurring >= 5 times in the source (88 of them) was checked component-by-component against the renameable entity set; only the 5 whose EVERY component is non-renameable survive the rename verbatim, and those are the `phrases` below. The `tokens` are the two characters the ratio test drops and one given name the compound pass leaves bare. Every replacement was collision-checked against the whole corpus and occurs 0 times in it.",
|
||||
"phrases": {
|
||||
"Moor House": "Ashdene House",
|
||||
"Marsh End": "Thurlby End",
|
||||
"Vale Hall": "Kelmar Hall",
|
||||
"Bigben Close": "Denholm Close",
|
||||
"Royd Lane": "Crowmoor Lane"
|
||||
},
|
||||
"tokens": {
|
||||
"Blanche": "Rosalind",
|
||||
"Grace": "Millicent",
|
||||
"Matthewson": "Everard",
|
||||
"Hollow": "Sowerby"
|
||||
},
|
||||
"tokens_why": "`Blanche` (Blanche Ingram) has 19 capitalised occurrences against ONE lowercase, giving ratio 0.0526 — over the 0.05 bar by a single token, so a named character with 19 mentions is dropped by a hair. `Grace` (Grace Poole) is rejected honestly: `grace` the common noun runs at a 0.224 lowercase ratio in Jane Eyre. Sampling all 21 bare capitalised `Grace` found 20 are the character in direct address (`Too much noise, Grace`, `Good morning, Grace`) and exactly one is the theological noun (`children of Grace`), which this map will also substitute — a known and accepted cost of one occurrence against a character leaking 37 times. `Matthewson` is Rev. Helstone's given name; the surname renames, the given name does not. `Hollow` (Hollow's Mill, and `the Hollow` for the valley) is 85 capitalised against 20 genuinely lowercase (`rushing down the hollow`), ratio 0.235 — the detector refuses it correctly. The token map is capitalised-and-whole-word, so it reaches `Hollow’s Mill` and `Hollow’s Miln` without touching the common noun. Caught only because a positive control asserted `Hollow` should be an entity and the gate disagreed; the control was wrong, not the detector — the same way `Adèle` failed as `Adele`.",
|
||||
"allow": [
|
||||
"Good God",
|
||||
"Thank God",
|
||||
"Lord Wellington",
|
||||
"Human Justice",
|
||||
"Miss Smith",
|
||||
"And Mrs"
|
||||
],
|
||||
"allow_why": "Real-world referents and generic English any novelist could write. `Lord Wellington` is the historical commander (Shirley is set during the Napoleonic wars). `Human Justice` is an allegorical personification in Villette built from two ordinary nouns. `Miss Smith` is the most generic surname in English and belongs to a minor Lowood teacher. `Good God` and `Thank God` are exclamations. `And Mrs` is a regex artefact of the audit's own 2-3gram pattern catching a sentence-initial `And Mrs.`, not a name.",
|
||||
"deliberately_NOT_mapped": "`Rue Fossette`, `Thornfield Hall`, `Crimsworth Hall`, `Walden Hall`, `Sympson Grove`, `Nunnely Common`, `Grace Poole`, `Jean Baptiste`, `Meess Lucie`, `Cyril Hall` and `Margaret Hall` all LOOK like leaks in the audit listing and are not: each has at least one renameable component, so the phrase never survives verbatim into a copy. Mapping them would be double-substitution. `Moor` is mapped only inside `Moor House` — bare `Moor` is the Yorkshire moorland and renaming it would damage the prose for no leak benefit."
|
||||
}
|
||||
@@ -0,0 +1,73 @@
|
||||
"""D2c for lv-bronte — pin hand-verified gender onto the resolved entity map.
|
||||
|
||||
WHY THIS EXISTS, AND WHY IT IS NOT CIRCULAR. `gender_by_proximity.py` scores each
|
||||
name's local pronoun mix against the corpus base rate, and on this corpus it beats
|
||||
the inherited resolver where it counts: 1 wrong against 5. But it still resolves
|
||||
`Jane` MALE (m=182 f=97), and that is not a tuning miss — it is structural. Brontë's
|
||||
three narrators are first-person, so their names appear almost only in DIALOGUE,
|
||||
spoken by other characters, surrounded by those characters' pronouns. Proximity
|
||||
inference is therefore blind to exactly the characters the adapter is being trained
|
||||
on. Hemingway's notes record the same blindness from the other side, and record that
|
||||
Yarros's fix (the POV chapter header) does not transfer. Brontë's editions have no
|
||||
such header either.
|
||||
|
||||
So the resolver's EVALUATION (how many did it get right, unaided) and the map we
|
||||
actually TRAIN on are two different artefacts, and conflating them is the error to
|
||||
avoid. This script does not improve the resolver's score and must never be quoted as
|
||||
if it had. It applies ground truth that a human read out of the novels to the map
|
||||
that goes downstream, and it reports every field it changed so the delta is visible.
|
||||
|
||||
Renaming `Jane` to a male name would invert the pronoun agreement of the single most
|
||||
important text in the corpus, 336 times. That is the cost this step buys off.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
import argparse, json
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
def main() -> int:
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("--entities", required=True, help="entities json from gender_by_proximity.py")
|
||||
ap.add_argument("--out", required=True)
|
||||
ap.add_argument("--pin", required=True,
|
||||
help="comma-separated Name=f|m|n ground truth, read from the novels. "
|
||||
"`n` forces NEUTRAL where a surname is genuinely shared across sexes.")
|
||||
a = ap.parse_args()
|
||||
|
||||
pins: dict[str, str | None] = {}
|
||||
for item in (s.strip() for s in a.pin.split(",") if s.strip()):
|
||||
name, _, g = item.partition("=")
|
||||
if g not in ("f", "m", "n"):
|
||||
print(f"== refusing: {item!r} — gender must be f, m or n")
|
||||
return 2
|
||||
pins[name.strip().lower()] = None if g == "n" else g
|
||||
|
||||
ents = json.loads(Path(a.entities).read_text())
|
||||
changed, already, unseen = [], [], sorted(pins)
|
||||
for slug, w in ents.items():
|
||||
for key, e in w["entities"].items():
|
||||
if key in pins:
|
||||
if key in unseen:
|
||||
unseen.remove(key)
|
||||
was = e.get("gender")
|
||||
if was != pins[key]:
|
||||
e["gender"] = pins[key]
|
||||
changed.append((slug, key, was, pins[key], e["cap"]))
|
||||
else:
|
||||
already.append((slug, key, was))
|
||||
|
||||
Path(a.out).write_text(json.dumps(ents, ensure_ascii=False, indent=2), encoding="utf-8")
|
||||
|
||||
print(f" pinned {len(pins)} names · changed {len(changed)} · already correct {len(already)}")
|
||||
for slug, key, was, now, cap in sorted(changed, key=lambda r: -r[4]):
|
||||
print(f" {key:<16} {slug:<14} {str(was):>4} -> {str(now):<4} ({cap} occurrences)")
|
||||
if unseen:
|
||||
# A pin that matches nothing is a typo or a stale name, and silently
|
||||
# ignoring it would let the operator believe a character was fixed.
|
||||
print(f" ⚠ {len(unseen)} pins matched NO entity: {unseen}")
|
||||
return 1
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,101 @@
|
||||
"""D1b for lv-bronte — repair the Gutenberg source before anything reads it.
|
||||
|
||||
Two defects, both measured on this corpus, both of which survive silently into a
|
||||
trained adapter if nothing removes them.
|
||||
|
||||
1. PUBLISHER BACK MATTER, 1,922 words, Shirley only. The final unit carries a
|
||||
T. Nelson & Sons printer's block, a `THE NELSON CLASSICS` catalogue advertising
|
||||
Ainsworth, Marryat, Verne, Kingsley and Dickens, and a Project Gutenberg
|
||||
transcriber's note listing punctuation corrections. None of it is Brontë. It is
|
||||
there for the structural reason the Hemingway run recorded: a splitter cuts on
|
||||
headings, and nothing follows the final one, so whatever the edition appends
|
||||
rides inside the last chapter. It is also where the entity `CHARLES` (12
|
||||
occurrences) came from — another author's given name, in the training text.
|
||||
|
||||
2. GUTENBERG ITALIC MARKUP `_like this_`, 1,344 spans across three works (The
|
||||
Professor's edition has none). Two separate harms. It teaches the adapter to
|
||||
emit underscores as prose. And `_` is a WORD CHARACTER in regex, so the leak
|
||||
gate's `\\b(Name)\\b` scan cannot match inside `_Antigua_` — an entity wrapped
|
||||
in italics is invisible to the gate, which is the same failure shape as Yarros's
|
||||
possessive-only `Afendra`: never renamed AND never reported. Measured blast
|
||||
radius here is exactly one entity, but "one" is a fact about this corpus, not a
|
||||
property of the defect.
|
||||
|
||||
Writes a repaired copy; never mutates the source tree.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
import argparse, json, re, shutil
|
||||
from pathlib import Path
|
||||
|
||||
#: The printed end-of-novel marker, alone on its line.
|
||||
END = re.compile(r"\nTHE END\.?\s*\n")
|
||||
#: Fingerprints that distinguish a real apparatus block from a stray phrase.
|
||||
APPARATUS = re.compile(r"NELSON|PUBLISHERS|CLASSICS|Transcriber|LIBRARY|Illustration", re.I)
|
||||
#: `_word_` / `_several words_`. A span MAY wrap across a single newline — 24 of
|
||||
#: them do here (`_sotto\nvoce_`, `_ignis\nfatuus_`), and a newline-free pattern
|
||||
#: silently leaves exactly those behind — but never across a blank line, which
|
||||
#: would let an unbalanced underscore swallow whole paragraphs.
|
||||
ITALIC = re.compile(r"_((?:[^_\n]|\n(?!\n)){1,200})_")
|
||||
|
||||
|
||||
def main() -> int:
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("corpus")
|
||||
ap.add_argument("--out", required=True)
|
||||
a = ap.parse_args()
|
||||
|
||||
src, dst = Path(a.corpus), Path(a.out)
|
||||
if dst.exists():
|
||||
shutil.rmtree(dst)
|
||||
shutil.copytree(src, dst)
|
||||
man = json.loads((dst / "manifest.json").read_text())
|
||||
|
||||
print(f" {'work':<16}{'words in':>10}{'words out':>11}{'back matter':>13}{'italic spans':>14}")
|
||||
total_back = total_ital = 0
|
||||
for w in man["works"]:
|
||||
p = dst / w["path"]
|
||||
rows = [json.loads(l) for l in p.read_text(encoding="utf-8").splitlines() if l.strip()]
|
||||
before = sum(len(r["text"].split()) for r in rows)
|
||||
|
||||
# (1) back matter — only the LAST unit can carry it
|
||||
stripped = 0
|
||||
last = rows[-1]
|
||||
m = END.search(last["text"])
|
||||
if m and APPARATUS.search(last["text"][m.end():]):
|
||||
stripped = len(last["text"][m.end():].split())
|
||||
last["text"] = last["text"][: m.start()].rstrip() + "\n"
|
||||
|
||||
# (2) italic markup, every unit
|
||||
spans = 0
|
||||
for r in rows:
|
||||
r["text"], n = ITALIC.subn(r"\1", r["text"])
|
||||
spans += n
|
||||
|
||||
after = sum(len(r["text"].split()) for r in rows)
|
||||
p.write_text("\n".join(json.dumps(r, ensure_ascii=False) for r in rows) + "\n", encoding="utf-8")
|
||||
w["words"] = after
|
||||
total_back += stripped
|
||||
total_ital += spans
|
||||
print(f" {w['slug']:<16}{before:>10}{after:>11}{stripped:>13}{spans:>14}")
|
||||
|
||||
man["total_words"] = sum(w["words"] for w in man["works"])
|
||||
man["normalisation"] = (man.get("normalisation", "") +
|
||||
" | back-matter stripped + Gutenberg italic markup removed").strip(" |")
|
||||
(dst / "manifest.json").write_text(json.dumps(man, ensure_ascii=False, indent=2), encoding="utf-8")
|
||||
|
||||
# Read back what we claim to have done, rather than asserting it.
|
||||
full = "\n".join((dst / w["path"]).read_text(encoding="utf-8") for w in man["works"])
|
||||
residue = {p: full.count(p) for p in
|
||||
("CHARLES KINGSLEY", "CHARLES DICKENS", "NELSON", "Transcriber", "JULES VERNE")}
|
||||
ital_left = len(ITALIC.findall(full))
|
||||
print(f"\n stripped {total_back} words of back matter · unwrapped {total_ital} italic spans")
|
||||
print(f" read-back: apparatus residue {residue} · italic spans remaining {ital_left}")
|
||||
if any(residue.values()) or ital_left:
|
||||
print("== REPAIR INCOMPLETE")
|
||||
return 1
|
||||
print(f" total_words -> {man['total_words']}")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,97 @@
|
||||
{
|
||||
"corpus": "lv-bronte (Charlotte Brontë — Jane Eyre, Villette, Shirley, The Professor)",
|
||||
"why": "The rename exists to keep the author's INVENTIONS out of the adapter. A real-world referent any novelist could use is not hers, and renaming it damages prose for no leak benefit. entities.py already carries a built-in STOP_COMMON that was tuned on this very corpus (London/Paris/Brussels/Yorkshire/God/Lord/Madam/Sir/Monsieur/Mademoiselle). This file is the DELTA that pass missed — overwhelmingly French honorific abbreviations and generic French place-nouns, because three of the four novels are set in or shadowed by Belgium.",
|
||||
"how_derived": "Every surface here was read IN CONTEXT before it landed, and the reading changed the answer twice. `India` is never the country — it is only ever the compound `India-rubber`, so renaming it yields `Braevick-rubber`. `Terrasse` looked like a generic French noun and is in fact `La Terrasse`, the Brettons' house, so it was PUT BACK as renameable. The ambiguous direction is deliberately biased toward renaming: leaving is the leaking direction.",
|
||||
"honorific_abbreviations": [
|
||||
"Mdlle",
|
||||
"Mdlles",
|
||||
"Mdme",
|
||||
"Mons",
|
||||
"Messieurs",
|
||||
"Mesdames",
|
||||
"Misses",
|
||||
"Esq",
|
||||
"Père",
|
||||
"Mère"
|
||||
],
|
||||
"generic_french_nouns_used_as_place_components": [
|
||||
"Rue",
|
||||
"Hôtel",
|
||||
"Athénée",
|
||||
"Basse-Ville",
|
||||
"Faubourg",
|
||||
"Porte",
|
||||
"Boulevard"
|
||||
],
|
||||
"real_world_geography": [
|
||||
"India",
|
||||
"Madeira",
|
||||
"Antwerp",
|
||||
"Ostend",
|
||||
"Calais",
|
||||
"Birmingham",
|
||||
"Eton",
|
||||
"Louvain",
|
||||
"Jamaica",
|
||||
"Antigua",
|
||||
"Guadaloupe",
|
||||
"Egypt"
|
||||
],
|
||||
"real_world_people_and_works": [
|
||||
"Shakespeare",
|
||||
"Wellington",
|
||||
"Cowper",
|
||||
"Coriolanus",
|
||||
"Cleopatra",
|
||||
"Moses",
|
||||
"Bonaparte",
|
||||
"Napoleon",
|
||||
"Rousseau",
|
||||
"Chénier",
|
||||
"Timon",
|
||||
"Pierrot",
|
||||
"Gytrash"
|
||||
],
|
||||
"political_and_period_terms": [
|
||||
"Jacobin",
|
||||
"Tory"
|
||||
],
|
||||
"typographic_devices": [
|
||||
"X----",
|
||||
"J",
|
||||
"S",
|
||||
"Bah",
|
||||
"Royale"
|
||||
],
|
||||
"deliberately_NOT_here": "Read and KEPT renameable, each for a stated reason: `Terrasse` (La Terrasse, the Brettons' house), `Walden` (Walden Hall, an in-world estate), `Crécy` (Boulevard/Porte de Crécy inside the invented city), `Hiram` (Hiram Yorke's given name), `Vashti` (the actress's stage name in Villette, a character), `Labassecour` and `Villette` (Brontë's invented country and city), `Temple` `Cave` (character surnames), `Pilot` `Tartar` (the dogs in Jane Eyre and Shirley — proper names).",
|
||||
"known_gap": "`Moses` is stoplisted for the rhetoric (`France is Israel, and Napoleon is Moses`), which means the given name of the character Moses Barraclough is NOT renamed. His surname is. `Grace` and `Blanche` are below the detector's floor for the opposite reason — `grace` the common noun runs at a 0.224 lowercase ratio in Jane Eyre — so `Grace Poole` and `Blanche Ingram` are handled in phrase_map_bronte.json, not here.",
|
||||
"real_world_nationalities_and_languages": [
|
||||
"Indian",
|
||||
"Flemish",
|
||||
"Anglais",
|
||||
"Anglaise",
|
||||
"Englishman",
|
||||
"Englishwoman",
|
||||
"Parisienne",
|
||||
"Belgians",
|
||||
"Flamand",
|
||||
"Cossack",
|
||||
"Hindostanee"
|
||||
],
|
||||
"religion_and_scripture": [
|
||||
"Jesuit",
|
||||
"Protestantism",
|
||||
"Protestants",
|
||||
"Romanism",
|
||||
"Dissenters",
|
||||
"Maker",
|
||||
"Dieu",
|
||||
"Notre",
|
||||
"Adam",
|
||||
"Saul",
|
||||
"Daniel",
|
||||
"Samuel",
|
||||
"Leaven"
|
||||
],
|
||||
"sub_threshold_triage_note": "Everything added in the 2026-09-16 second pass came from the gate's own sub-threshold report — the 84 surfaces below the rename floor that the gate lists separately because rename never saw them. Each was read before landing here. Deliberately NOT stoplisted, because renaming is the safe direction and these may be characters rather than scripture: David, Joseph, Thomas. `Gytrash` is Yorkshire folklore Brontë did not invent; `Pierrot` and `Timon` are stock figures."
|
||||
}
|
||||
Reference in New Issue
Block a user