vh
fc834a8a23
The Brontë corpus's "0 of 203" was a HAND COUNT made before leak_gate.py
existed. On Yarros the automated instrument read 212 surviving where a hand
count said 86, so the hand count was never evidence. This runs the real gate,
and getting it to pass required fixing four defects the hand count could not
have seen.
CORPUS DEFECTS (repair_corpus_bronte.py, both measured):
- 1,922 words of publisher back matter inside Shirley's last unit — a
T. Nelson & Sons catalogue advertising Ainsworth, Marryat, Verne, Kingsley
and Dickens, plus a Gutenberg transcriber's punctuation list. Not Brontë,
and the source of the entity CHARLES. Same structural cause as the
Hemingway run: a splitter cuts on headings, nothing follows the final one.
- 1,368 Gutenberg italic spans. Two harms: they teach the adapter to emit
underscores, and the underscore is a word character, so the gate's
word-boundary scan cannot match inside an italicised name. An entity in
italics is invisible to the gate — the same never-renamed-AND-never-
reported shape as Yarros's possessive-only Afendra.
DETECTOR GAPS (phrase_map_bronte.json):
- Blanche is 19 capitalised against ONE lowercase — ratio 0.0526, over the
0.05 bar by a single token, so a named character with 19 mentions is
dropped by a hair.
- Grace (0.224) and Hollow (0.235) are refused correctly — both are common
nouns — but Grace Poole and Hollow's Mill are Brontë's. Sampling all 21
bare capitalised Grace found 20 are the character in direct address and
exactly one is the theological noun.
- Five compounds whose every component is non-renameable survive verbatim:
Moor House, Marsh End, Vale Hall, Bigben Close, Royd Lane. The other 77
audited phrases do not, because each has a renameable component.
GENDER (pin_known_gender.py): the inherited resolver put Jane MALE across 336
occurrences. Hemingway's base-rate resolver is strictly better here (1 wrong vs
4) but still fails on Jane, and the failure is structural, not tuning — Brontë's
three narrators are first-person, so their names appear almost only in dialogue
surrounded by other characters' pronouns. Ground truth is pinned separately from
the resolver's evaluation so the two are never conflated.
Also: min-cap lowered 8 to 3, which pulled Bertha, Ferndean, Rochesters and
Creemsvort in from below the old floor; corpus-scope rename so a name below
threshold in one novel is not printed verbatim there while renamed in another.
Gate: 0 of 365 surviving, positive control 365/365, negative control clean,
phrase audit 0 of 82. Floor stated: 3 capitals per work, 5 recurrences.
74 lines
3.5 KiB
Python
74 lines
3.5 KiB
Python
"""D2c for lv-bronte — pin hand-verified gender onto the resolved entity map.
|
|
|
|
WHY THIS EXISTS, AND WHY IT IS NOT CIRCULAR. `gender_by_proximity.py` scores each
|
|
name's local pronoun mix against the corpus base rate, and on this corpus it beats
|
|
the inherited resolver where it counts: 1 wrong against 5. But it still resolves
|
|
`Jane` MALE (m=182 f=97), and that is not a tuning miss — it is structural. Brontë's
|
|
three narrators are first-person, so their names appear almost only in DIALOGUE,
|
|
spoken by other characters, surrounded by those characters' pronouns. Proximity
|
|
inference is therefore blind to exactly the characters the adapter is being trained
|
|
on. Hemingway's notes record the same blindness from the other side, and record that
|
|
Yarros's fix (the POV chapter header) does not transfer. Brontë's editions have no
|
|
such header either.
|
|
|
|
So the resolver's EVALUATION (how many did it get right, unaided) and the map we
|
|
actually TRAIN on are two different artefacts, and conflating them is the error to
|
|
avoid. This script does not improve the resolver's score and must never be quoted as
|
|
if it had. It applies ground truth that a human read out of the novels to the map
|
|
that goes downstream, and it reports every field it changed so the delta is visible.
|
|
|
|
Renaming `Jane` to a male name would invert the pronoun agreement of the single most
|
|
important text in the corpus, 336 times. That is the cost this step buys off.
|
|
"""
|
|
from __future__ import annotations
|
|
import argparse, json
|
|
from pathlib import Path
|
|
|
|
|
|
def main() -> int:
|
|
ap = argparse.ArgumentParser()
|
|
ap.add_argument("--entities", required=True, help="entities json from gender_by_proximity.py")
|
|
ap.add_argument("--out", required=True)
|
|
ap.add_argument("--pin", required=True,
|
|
help="comma-separated Name=f|m|n ground truth, read from the novels. "
|
|
"`n` forces NEUTRAL where a surname is genuinely shared across sexes.")
|
|
a = ap.parse_args()
|
|
|
|
pins: dict[str, str | None] = {}
|
|
for item in (s.strip() for s in a.pin.split(",") if s.strip()):
|
|
name, _, g = item.partition("=")
|
|
if g not in ("f", "m", "n"):
|
|
print(f"== refusing: {item!r} — gender must be f, m or n")
|
|
return 2
|
|
pins[name.strip().lower()] = None if g == "n" else g
|
|
|
|
ents = json.loads(Path(a.entities).read_text())
|
|
changed, already, unseen = [], [], sorted(pins)
|
|
for slug, w in ents.items():
|
|
for key, e in w["entities"].items():
|
|
if key in pins:
|
|
if key in unseen:
|
|
unseen.remove(key)
|
|
was = e.get("gender")
|
|
if was != pins[key]:
|
|
e["gender"] = pins[key]
|
|
changed.append((slug, key, was, pins[key], e["cap"]))
|
|
else:
|
|
already.append((slug, key, was))
|
|
|
|
Path(a.out).write_text(json.dumps(ents, ensure_ascii=False, indent=2), encoding="utf-8")
|
|
|
|
print(f" pinned {len(pins)} names · changed {len(changed)} · already correct {len(already)}")
|
|
for slug, key, was, now, cap in sorted(changed, key=lambda r: -r[4]):
|
|
print(f" {key:<16} {slug:<14} {str(was):>4} -> {str(now):<4} ({cap} occurrences)")
|
|
if unseen:
|
|
# A pin that matches nothing is a typo or a stale name, and silently
|
|
# ignoring it would let the operator believe a character was fixed.
|
|
print(f" ⚠ {len(unseen)} pins matched NO entity: {unseen}")
|
|
return 1
|
|
return 0
|
|
|
|
|
|
if __name__ == "__main__":
|
|
raise SystemExit(main())
|