feat(r49): D2/D3 complete and the H02 pilot is training on gx10
Entity resolution, deterministic rename augmentation, packing and the pilot trainer. Qwen3-0.6B-Base is training now: 507 steps, 11.2 s/it, ~1h35m. D2 -- gender resolution is TITLE-FIRST, and that is a change from F02's method rather than a port of it. F02 used pronoun proximity and recorded that it is structurally blind to the first-person narrator, whose name appears mainly in dialogue surrounded by other people's pronouns. Measured here, proximity called JANE MALE -- the narrator of Jane Eyre and the single worst entity to get wrong. Titles have no such blind spot: Miss Eyre, Mrs. Fairfax, Mr. Rochester, Madame Beck, M. Paul, and a 19th-century novel is saturated with them. Measured: 16 entities resolved, zero wrong, every ambiguous case landing on HELD -- shared family surnames like Helstone and Pelet genuinely belong to both a man and a woman and hold as they should. Held means ungendered, not unrenamed. A HELD entity is still renamed, from the gender-neutral surname pool, because the operator's Yarros directive was "rename all proper nouns" and holding a place leaks it -- Thornfield appears 100 times in Jane Eyre and is as author-specific as Riders Quadrant was. Substituting a neutral token makes no gender claim, so no gender claim can be wrong. D3 -- pool is French + English per the operator, weighted per work by setting: Brussels novels 60% French, Yorkshire novels 25%. Locales restricted to fr_FR/fr_BE/en_GB/en_IE; en_US and en_AU carry modern surnames that are wrong register for the 1840s. The pool is filtered against Brontë's own 75-letter alphabet, so French accents stay and Czech/Latvian marks do not. Two collision defects found by running the leak gate rather than trusting it: `Burns` and `Marie` were drawn as replacements while being Brontë characters -- F02's collision filter was built against Yarros and does not carry -- and then `Pierre-Yves` passed a whole-string filter while `Pierre` (Mademoiselle St. Pierre) is a Villette character. The filter now compares by COMPONENT. Final gate: 0 of 203 source entities survive in any of 24 copy-files. Trainer records what the run RESOLVED to rather than what it requested -- attention implementation, dtype, device, corpus sha and harness cleanliness are read back off the live objects. transformers 5.x has dropped warmup_ratio, caught by reading the signature after the first launch failed on it; the 3% warmup is computed into warmup_steps instead.
This commit is contained in:
@@ -0,0 +1,202 @@
|
||||
"""R49 Stage D2 — build the per-work entity map, deterministically.
|
||||
|
||||
Technique is F02's, which took three generations to get right and whose lesson is
|
||||
one of level rather than of cleverness: **the entity map is built once per work,
|
||||
so the detector must see the work, not the paragraph.**
|
||||
|
||||
v1 position-based -> MISSES names that start sentences (characters do, constantly)
|
||||
v2 dictionary-based -> MISSES names that are words (fiction names people after flowers)
|
||||
v3 corpus cap-ratio -> works. No wordlist, no position rule, no LLM.
|
||||
|
||||
A token's capitalised count against its lowercase count across the WHOLE work
|
||||
separates `Jane` (only ever capitalised) from `Door` (capitalised only when it
|
||||
starts a sentence). Identity linking then joins adjacent capitalised pairs that
|
||||
recur, which is also what recovers the first-person narrator's gender -- her name
|
||||
appears mainly in dialogue, surrounded by other people's pronouns, so proximity
|
||||
inference is structurally blind to exactly the character the adapter is being
|
||||
trained on.
|
||||
|
||||
Nothing here guesses. Unresolved entities block corpus emission and go to a human
|
||||
pass: held is cheap, wrong is poison -- a silently mis-gendered entity scrambles
|
||||
pronoun agreement through every renamed copy and nothing downstream would catch it.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
import argparse, collections, json, re, sys
|
||||
from pathlib import Path
|
||||
|
||||
WORD = re.compile(r"\b[A-Za-zÀ-ÿŒœÆæ][a-zà-ÿœæ'’\-]*\b")
|
||||
TOKEN = re.compile(r"[A-Za-zÀ-ÿŒœÆæ][A-Za-zà-ÿœæ'’\-]*")
|
||||
|
||||
#: Ranks, honorifics and address forms are not names. F02 lost `Colonel Aetos`
|
||||
#: and `Professor Kaori` to this -- without the stoplist the rename replaces the
|
||||
#: rank. Kinship terms likewise: `Mom` renamed to `Ingrid` was a v2 defect.
|
||||
STOP_TITLES = {
|
||||
"Mr", "Mrs", "Miss", "Ms", "Dr", "Sir", "Lady", "Lord", "Madam", "Madame",
|
||||
"Mademoiselle", "Monsieur", "Master", "Captain", "Colonel", "Major", "General",
|
||||
"Professor", "Reverend", "Rev", "Doctor", "Saint", "St", "Aunt", "Uncle",
|
||||
"Mother", "Father", "Papa", "Mamma", "Mama", "Brother", "Sister", "Cousin",
|
||||
"Grandmother", "Grandfather", "Nurse", "King", "Queen", "Prince", "Princess",
|
||||
"Duke", "Duchess", "Earl", "Count", "Countess", "Baron", "Squire", "Parson",
|
||||
"Monseigneur", "Mlle", "Mme", "M", "Messrs",
|
||||
}
|
||||
#: Days, months, and the language/nation adjectives a 19th-century novel is full
|
||||
#: of. All are always-capitalised and would otherwise pass the ratio test.
|
||||
STOP_COMMON = {
|
||||
"Monday","Tuesday","Wednesday","Thursday","Friday","Saturday","Sunday",
|
||||
"January","February","March","April","May","June","July","August",
|
||||
"September","October","November","December",
|
||||
"English","France","French","England","Britain","British","Europe","European",
|
||||
"German","Germany","Belgian","Belgium","Scotch","Scottish","Scotland","Irish",
|
||||
"Ireland","Welsh","Wales","Latin","Greek","Italian","Italy","Spanish","Spain",
|
||||
"Swiss","Switzerland","Dutch","Holland","Roman","Rome","Catholic","Protestant",
|
||||
"Christian","Christ","God","Lord","Heaven","Providence","Bible","Sabbath",
|
||||
"Christmas","Easter","London","Paris","Brussels","Yorkshire","I","O","Oh","Ah",
|
||||
"Yes","No","Well","Now","Then","But","And","The","A","An","He","She","It","They",
|
||||
"You","We","His","Her","My","Your","Their","This","That","There","Here","What",
|
||||
"Who","When","Where","Why","How","If","So","As","At","In","On","To","For","Of",
|
||||
"Nay","Alas","Madam","Sir","Mademoiselle","Monsieur",
|
||||
}
|
||||
STOP = STOP_TITLES | STOP_COMMON
|
||||
|
||||
MALE_PRON = {"he", "him", "his", "himself"}
|
||||
FEM_PRON = {"she", "her", "hers", "herself"}
|
||||
|
||||
|
||||
def load(corpus: Path) -> dict[str, str]:
|
||||
man = json.loads((corpus / "manifest.json").read_text())
|
||||
out = {}
|
||||
for w in man["works"]:
|
||||
rows = [json.loads(l) for l in (corpus / w["path"]).read_text(encoding="utf-8").splitlines()]
|
||||
out[w["slug"]] = "\n\n".join(r["text"] for r in rows)
|
||||
return out
|
||||
|
||||
|
||||
def detect(text: str, min_count: int, max_ratio: float) -> dict[str, dict]:
|
||||
"""Corpus-level capitalised-vs-lowercase ratio. See module docstring."""
|
||||
cap, low = collections.Counter(), collections.Counter()
|
||||
for m in TOKEN.finditer(text):
|
||||
t = m.group(0)
|
||||
(cap if t[:1].isupper() else low)[t.lower()] += 1
|
||||
ents = {}
|
||||
for key, c in cap.items():
|
||||
if c < min_count:
|
||||
continue
|
||||
l = low[key]
|
||||
ratio = l / c
|
||||
if ratio > max_ratio:
|
||||
continue
|
||||
# recover the dominant surface spelling
|
||||
ents[key] = {"cap": c, "lower": l, "ratio": round(ratio, 4)}
|
||||
return ents
|
||||
|
||||
|
||||
def surface_forms(text: str, keys: set[str]) -> dict[str, str]:
|
||||
best = collections.defaultdict(collections.Counter)
|
||||
for m in TOKEN.finditer(text):
|
||||
t = m.group(0)
|
||||
if t[:1].isupper() and t.lower() in keys:
|
||||
best[t.lower()][t] += 1
|
||||
return {k: c.most_common(1)[0][0] for k, c in best.items()}
|
||||
|
||||
|
||||
def link_identities(text: str, names: set[str], min_pairs: int) -> list[tuple[str, str]]:
|
||||
"""Adjacent capitalised pairs that recur are one person.
|
||||
|
||||
This is what makes `Xaden Riorson` a single identity so the bare given name
|
||||
maps to the given part and the surname to the surname part, keeping the
|
||||
honorific form working. It is also what recovers the POV character's gender.
|
||||
"""
|
||||
pairs = collections.Counter()
|
||||
toks = [(m.group(0), m.start()) for m in TOKEN.finditer(text)]
|
||||
for i in range(len(toks) - 1):
|
||||
a, b = toks[i][0], toks[i + 1][0]
|
||||
if toks[i + 1][1] - toks[i][1] > len(a) + 2:
|
||||
continue # not actually adjacent
|
||||
if a[:1].isupper() and b[:1].isupper() and a not in STOP and b not in STOP:
|
||||
if a.lower() in names and b.lower() in names:
|
||||
pairs[(a, b)] += 1
|
||||
return [p for p, n in pairs.items() if n >= min_pairs]
|
||||
|
||||
|
||||
def resolve_gender(text: str, names: set[str]) -> dict[str, str]:
|
||||
"""Same-sentence pronoun co-occurrence. Never guesses; unresolved stays unresolved.
|
||||
|
||||
F02: tightening from a +/-200-char window to same-sentence converted a WRONG
|
||||
to a HELD while keeping every correct call. Held is cheap; wrong is poison.
|
||||
"""
|
||||
score = collections.defaultdict(lambda: [0, 0])
|
||||
for sent in re.split(r"(?<=[.!?])\s+", text):
|
||||
low = {w.lower() for w in TOKEN.findall(sent)}
|
||||
m, f = bool(low & MALE_PRON), bool(low & FEM_PRON)
|
||||
if m == f:
|
||||
continue # both or neither -> no signal
|
||||
for t in TOKEN.findall(sent):
|
||||
if t[:1].isupper() and t.lower() in names:
|
||||
score[t.lower()][0 if m else 1] += 1
|
||||
out = {}
|
||||
for k, (mm, ff) in score.items():
|
||||
tot = mm + ff
|
||||
if tot < 3:
|
||||
continue
|
||||
if mm / tot >= 0.75:
|
||||
out[k] = "m"
|
||||
elif ff / tot >= 0.75:
|
||||
out[k] = "f"
|
||||
return out
|
||||
|
||||
|
||||
def main() -> int:
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("corpus")
|
||||
ap.add_argument("--out", default=None)
|
||||
ap.add_argument("--min-count", type=int, default=5)
|
||||
ap.add_argument("--max-ratio", type=float, default=0.05)
|
||||
ap.add_argument("--min-pairs", type=int, default=2)
|
||||
ap.add_argument("--control", default="", help="comma-separated known-true names (positive control)")
|
||||
a = ap.parse_args()
|
||||
corpus = Path(a.corpus)
|
||||
works = load(corpus)
|
||||
|
||||
controls = [c.strip() for c in a.control.split(",") if c.strip()]
|
||||
report, failed_control = {}, []
|
||||
for slug, text in works.items():
|
||||
ents = detect(text, a.min_count, a.max_ratio)
|
||||
keys = {k for k in ents if k.capitalize() not in STOP and k.title() not in STOP}
|
||||
keys = {k for k in keys if k not in {s.lower() for s in STOP}}
|
||||
forms = surface_forms(text, keys)
|
||||
links = link_identities(text, keys, a.min_pairs)
|
||||
gender = resolve_gender(text, keys)
|
||||
# identity linking propagates gender: a bare surname inherits from its given name
|
||||
for g, s in links:
|
||||
gl, sl = g.lower(), s.lower()
|
||||
if gl in gender and sl not in gender:
|
||||
gender[sl] = gender[gl]
|
||||
elif sl in gender and gl not in gender:
|
||||
gender[gl] = gender[sl]
|
||||
report[slug] = {"entities": {k: {**ents[k], "surface": forms.get(k, k),
|
||||
"gender": gender.get(k)} for k in sorted(keys)},
|
||||
"identity_links": [list(p) for p in links]}
|
||||
print(f" {slug:<14} {len(keys):>4} entities {len(links):>3} identity links "
|
||||
f"{sum(1 for k in keys if gender.get(k)):>3} gendered "
|
||||
f"{sum(1 for k in keys if not gender.get(k)):>4} ungendered")
|
||||
|
||||
if controls:
|
||||
print("\n positive control -- names known to be real must be FOUND:")
|
||||
for name in controls:
|
||||
hits = [s for s, r in report.items() if name.lower() in r["entities"]]
|
||||
ok = bool(hits)
|
||||
print(f" [{'PASS' if ok else 'FAIL'}] {name:<14} {', '.join(hits) if hits else 'NOT DETECTED'}")
|
||||
if not ok:
|
||||
failed_control.append(name)
|
||||
|
||||
if a.out:
|
||||
Path(a.out).write_text(json.dumps(report, ensure_ascii=False, indent=2), encoding="utf-8")
|
||||
print(f"\n wrote {a.out}")
|
||||
if failed_control:
|
||||
print(f"\n== POSITIVE CONTROL FAILED for {failed_control} -- the detector's negatives are worthless")
|
||||
return 1
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
Reference in New Issue
Block a user