vh
fc834a8a23
The Brontë corpus's "0 of 203" was a HAND COUNT made before leak_gate.py
existed. On Yarros the automated instrument read 212 surviving where a hand
count said 86, so the hand count was never evidence. This runs the real gate,
and getting it to pass required fixing four defects the hand count could not
have seen.
CORPUS DEFECTS (repair_corpus_bronte.py, both measured):
- 1,922 words of publisher back matter inside Shirley's last unit — a
T. Nelson & Sons catalogue advertising Ainsworth, Marryat, Verne, Kingsley
and Dickens, plus a Gutenberg transcriber's punctuation list. Not Brontë,
and the source of the entity CHARLES. Same structural cause as the
Hemingway run: a splitter cuts on headings, nothing follows the final one.
- 1,368 Gutenberg italic spans. Two harms: they teach the adapter to emit
underscores, and the underscore is a word character, so the gate's
word-boundary scan cannot match inside an italicised name. An entity in
italics is invisible to the gate — the same never-renamed-AND-never-
reported shape as Yarros's possessive-only Afendra.
DETECTOR GAPS (phrase_map_bronte.json):
- Blanche is 19 capitalised against ONE lowercase — ratio 0.0526, over the
0.05 bar by a single token, so a named character with 19 mentions is
dropped by a hair.
- Grace (0.224) and Hollow (0.235) are refused correctly — both are common
nouns — but Grace Poole and Hollow's Mill are Brontë's. Sampling all 21
bare capitalised Grace found 20 are the character in direct address and
exactly one is the theological noun.
- Five compounds whose every component is non-renameable survive verbatim:
Moor House, Marsh End, Vale Hall, Bigben Close, Royd Lane. The other 77
audited phrases do not, because each has a renameable component.
GENDER (pin_known_gender.py): the inherited resolver put Jane MALE across 336
occurrences. Hemingway's base-rate resolver is strictly better here (1 wrong vs
4) but still fails on Jane, and the failure is structural, not tuning — Brontë's
three narrators are first-person, so their names appear almost only in dialogue
surrounded by other characters' pronouns. Ground truth is pinned separately from
the resolver's evaluation so the two are never conflated.
Also: min-cap lowered 8 to 3, which pulled Bertha, Ferndean, Rochesters and
Creemsvort in from below the old floor; corpus-scope rename so a name below
threshold in one novel is not printed verbatim there while renamed in another.
Gate: 0 of 365 surviving, positive control 365/365, negative control clean,
phrase audit 0 of 82. Floor stated: 3 capitals per work, 5 recurrences.
102 lines
4.7 KiB
Python
102 lines
4.7 KiB
Python
"""D1b for lv-bronte — repair the Gutenberg source before anything reads it.
|
|
|
|
Two defects, both measured on this corpus, both of which survive silently into a
|
|
trained adapter if nothing removes them.
|
|
|
|
1. PUBLISHER BACK MATTER, 1,922 words, Shirley only. The final unit carries a
|
|
T. Nelson & Sons printer's block, a `THE NELSON CLASSICS` catalogue advertising
|
|
Ainsworth, Marryat, Verne, Kingsley and Dickens, and a Project Gutenberg
|
|
transcriber's note listing punctuation corrections. None of it is Brontë. It is
|
|
there for the structural reason the Hemingway run recorded: a splitter cuts on
|
|
headings, and nothing follows the final one, so whatever the edition appends
|
|
rides inside the last chapter. It is also where the entity `CHARLES` (12
|
|
occurrences) came from — another author's given name, in the training text.
|
|
|
|
2. GUTENBERG ITALIC MARKUP `_like this_`, 1,344 spans across three works (The
|
|
Professor's edition has none). Two separate harms. It teaches the adapter to
|
|
emit underscores as prose. And `_` is a WORD CHARACTER in regex, so the leak
|
|
gate's `\\b(Name)\\b` scan cannot match inside `_Antigua_` — an entity wrapped
|
|
in italics is invisible to the gate, which is the same failure shape as Yarros's
|
|
possessive-only `Afendra`: never renamed AND never reported. Measured blast
|
|
radius here is exactly one entity, but "one" is a fact about this corpus, not a
|
|
property of the defect.
|
|
|
|
Writes a repaired copy; never mutates the source tree.
|
|
"""
|
|
from __future__ import annotations
|
|
import argparse, json, re, shutil
|
|
from pathlib import Path
|
|
|
|
#: The printed end-of-novel marker, alone on its line.
|
|
END = re.compile(r"\nTHE END\.?\s*\n")
|
|
#: Fingerprints that distinguish a real apparatus block from a stray phrase.
|
|
APPARATUS = re.compile(r"NELSON|PUBLISHERS|CLASSICS|Transcriber|LIBRARY|Illustration", re.I)
|
|
#: `_word_` / `_several words_`. A span MAY wrap across a single newline — 24 of
|
|
#: them do here (`_sotto\nvoce_`, `_ignis\nfatuus_`), and a newline-free pattern
|
|
#: silently leaves exactly those behind — but never across a blank line, which
|
|
#: would let an unbalanced underscore swallow whole paragraphs.
|
|
ITALIC = re.compile(r"_((?:[^_\n]|\n(?!\n)){1,200})_")
|
|
|
|
|
|
def main() -> int:
|
|
ap = argparse.ArgumentParser()
|
|
ap.add_argument("corpus")
|
|
ap.add_argument("--out", required=True)
|
|
a = ap.parse_args()
|
|
|
|
src, dst = Path(a.corpus), Path(a.out)
|
|
if dst.exists():
|
|
shutil.rmtree(dst)
|
|
shutil.copytree(src, dst)
|
|
man = json.loads((dst / "manifest.json").read_text())
|
|
|
|
print(f" {'work':<16}{'words in':>10}{'words out':>11}{'back matter':>13}{'italic spans':>14}")
|
|
total_back = total_ital = 0
|
|
for w in man["works"]:
|
|
p = dst / w["path"]
|
|
rows = [json.loads(l) for l in p.read_text(encoding="utf-8").splitlines() if l.strip()]
|
|
before = sum(len(r["text"].split()) for r in rows)
|
|
|
|
# (1) back matter — only the LAST unit can carry it
|
|
stripped = 0
|
|
last = rows[-1]
|
|
m = END.search(last["text"])
|
|
if m and APPARATUS.search(last["text"][m.end():]):
|
|
stripped = len(last["text"][m.end():].split())
|
|
last["text"] = last["text"][: m.start()].rstrip() + "\n"
|
|
|
|
# (2) italic markup, every unit
|
|
spans = 0
|
|
for r in rows:
|
|
r["text"], n = ITALIC.subn(r"\1", r["text"])
|
|
spans += n
|
|
|
|
after = sum(len(r["text"].split()) for r in rows)
|
|
p.write_text("\n".join(json.dumps(r, ensure_ascii=False) for r in rows) + "\n", encoding="utf-8")
|
|
w["words"] = after
|
|
total_back += stripped
|
|
total_ital += spans
|
|
print(f" {w['slug']:<16}{before:>10}{after:>11}{stripped:>13}{spans:>14}")
|
|
|
|
man["total_words"] = sum(w["words"] for w in man["works"])
|
|
man["normalisation"] = (man.get("normalisation", "") +
|
|
" | back-matter stripped + Gutenberg italic markup removed").strip(" |")
|
|
(dst / "manifest.json").write_text(json.dumps(man, ensure_ascii=False, indent=2), encoding="utf-8")
|
|
|
|
# Read back what we claim to have done, rather than asserting it.
|
|
full = "\n".join((dst / w["path"]).read_text(encoding="utf-8") for w in man["works"])
|
|
residue = {p: full.count(p) for p in
|
|
("CHARLES KINGSLEY", "CHARLES DICKENS", "NELSON", "Transcriber", "JULES VERNE")}
|
|
ital_left = len(ITALIC.findall(full))
|
|
print(f"\n stripped {total_back} words of back matter · unwrapped {total_ital} italic spans")
|
|
print(f" read-back: apparatus residue {residue} · italic spans remaining {ital_left}")
|
|
if any(residue.values()) or ital_left:
|
|
print("== REPAIR INCOMPLETE")
|
|
return 1
|
|
print(f" total_words -> {man['total_words']}")
|
|
return 0
|
|
|
|
|
|
if __name__ == "__main__":
|
|
raise SystemExit(main())
|