feat(r49): D2/D3 complete and the H02 pilot is training on gx10
Entity resolution, deterministic rename augmentation, packing and the pilot trainer. Qwen3-0.6B-Base is training now: 507 steps, 11.2 s/it, ~1h35m. D2 -- gender resolution is TITLE-FIRST, and that is a change from F02's method rather than a port of it. F02 used pronoun proximity and recorded that it is structurally blind to the first-person narrator, whose name appears mainly in dialogue surrounded by other people's pronouns. Measured here, proximity called JANE MALE -- the narrator of Jane Eyre and the single worst entity to get wrong. Titles have no such blind spot: Miss Eyre, Mrs. Fairfax, Mr. Rochester, Madame Beck, M. Paul, and a 19th-century novel is saturated with them. Measured: 16 entities resolved, zero wrong, every ambiguous case landing on HELD -- shared family surnames like Helstone and Pelet genuinely belong to both a man and a woman and hold as they should. Held means ungendered, not unrenamed. A HELD entity is still renamed, from the gender-neutral surname pool, because the operator's Yarros directive was "rename all proper nouns" and holding a place leaks it -- Thornfield appears 100 times in Jane Eyre and is as author-specific as Riders Quadrant was. Substituting a neutral token makes no gender claim, so no gender claim can be wrong. D3 -- pool is French + English per the operator, weighted per work by setting: Brussels novels 60% French, Yorkshire novels 25%. Locales restricted to fr_FR/fr_BE/en_GB/en_IE; en_US and en_AU carry modern surnames that are wrong register for the 1840s. The pool is filtered against Brontë's own 75-letter alphabet, so French accents stay and Czech/Latvian marks do not. Two collision defects found by running the leak gate rather than trusting it: `Burns` and `Marie` were drawn as replacements while being Brontë characters -- F02's collision filter was built against Yarros and does not carry -- and then `Pierre-Yves` passed a whole-string filter while `Pierre` (Mademoiselle St. Pierre) is a Villette character. The filter now compares by COMPONENT. Final gate: 0 of 203 source entities survive in any of 24 copy-files. Trainer records what the run RESOLVED to rather than what it requested -- attention implementation, dtype, device, corpus sha and harness cleanliness are read back off the live objects. transformers 5.x has dropped warmup_ratio, caught by reading the signature after the first launch failed on it; the 3% warmup is computed into warmup_steps instead.
This commit is contained in:
@@ -0,0 +1,202 @@
|
|||||||
|
"""R49 Stage D2 — build the per-work entity map, deterministically.
|
||||||
|
|
||||||
|
Technique is F02's, which took three generations to get right and whose lesson is
|
||||||
|
one of level rather than of cleverness: **the entity map is built once per work,
|
||||||
|
so the detector must see the work, not the paragraph.**
|
||||||
|
|
||||||
|
v1 position-based -> MISSES names that start sentences (characters do, constantly)
|
||||||
|
v2 dictionary-based -> MISSES names that are words (fiction names people after flowers)
|
||||||
|
v3 corpus cap-ratio -> works. No wordlist, no position rule, no LLM.
|
||||||
|
|
||||||
|
A token's capitalised count against its lowercase count across the WHOLE work
|
||||||
|
separates `Jane` (only ever capitalised) from `Door` (capitalised only when it
|
||||||
|
starts a sentence). Identity linking then joins adjacent capitalised pairs that
|
||||||
|
recur, which is also what recovers the first-person narrator's gender -- her name
|
||||||
|
appears mainly in dialogue, surrounded by other people's pronouns, so proximity
|
||||||
|
inference is structurally blind to exactly the character the adapter is being
|
||||||
|
trained on.
|
||||||
|
|
||||||
|
Nothing here guesses. Unresolved entities block corpus emission and go to a human
|
||||||
|
pass: held is cheap, wrong is poison -- a silently mis-gendered entity scrambles
|
||||||
|
pronoun agreement through every renamed copy and nothing downstream would catch it.
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
import argparse, collections, json, re, sys
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
WORD = re.compile(r"\b[A-Za-zÀ-ÿŒœÆæ][a-zà-ÿœæ'’\-]*\b")
|
||||||
|
TOKEN = re.compile(r"[A-Za-zÀ-ÿŒœÆæ][A-Za-zà-ÿœæ'’\-]*")
|
||||||
|
|
||||||
|
#: Ranks, honorifics and address forms are not names. F02 lost `Colonel Aetos`
|
||||||
|
#: and `Professor Kaori` to this -- without the stoplist the rename replaces the
|
||||||
|
#: rank. Kinship terms likewise: `Mom` renamed to `Ingrid` was a v2 defect.
|
||||||
|
STOP_TITLES = {
|
||||||
|
"Mr", "Mrs", "Miss", "Ms", "Dr", "Sir", "Lady", "Lord", "Madam", "Madame",
|
||||||
|
"Mademoiselle", "Monsieur", "Master", "Captain", "Colonel", "Major", "General",
|
||||||
|
"Professor", "Reverend", "Rev", "Doctor", "Saint", "St", "Aunt", "Uncle",
|
||||||
|
"Mother", "Father", "Papa", "Mamma", "Mama", "Brother", "Sister", "Cousin",
|
||||||
|
"Grandmother", "Grandfather", "Nurse", "King", "Queen", "Prince", "Princess",
|
||||||
|
"Duke", "Duchess", "Earl", "Count", "Countess", "Baron", "Squire", "Parson",
|
||||||
|
"Monseigneur", "Mlle", "Mme", "M", "Messrs",
|
||||||
|
}
|
||||||
|
#: Days, months, and the language/nation adjectives a 19th-century novel is full
|
||||||
|
#: of. All are always-capitalised and would otherwise pass the ratio test.
|
||||||
|
STOP_COMMON = {
|
||||||
|
"Monday","Tuesday","Wednesday","Thursday","Friday","Saturday","Sunday",
|
||||||
|
"January","February","March","April","May","June","July","August",
|
||||||
|
"September","October","November","December",
|
||||||
|
"English","France","French","England","Britain","British","Europe","European",
|
||||||
|
"German","Germany","Belgian","Belgium","Scotch","Scottish","Scotland","Irish",
|
||||||
|
"Ireland","Welsh","Wales","Latin","Greek","Italian","Italy","Spanish","Spain",
|
||||||
|
"Swiss","Switzerland","Dutch","Holland","Roman","Rome","Catholic","Protestant",
|
||||||
|
"Christian","Christ","God","Lord","Heaven","Providence","Bible","Sabbath",
|
||||||
|
"Christmas","Easter","London","Paris","Brussels","Yorkshire","I","O","Oh","Ah",
|
||||||
|
"Yes","No","Well","Now","Then","But","And","The","A","An","He","She","It","They",
|
||||||
|
"You","We","His","Her","My","Your","Their","This","That","There","Here","What",
|
||||||
|
"Who","When","Where","Why","How","If","So","As","At","In","On","To","For","Of",
|
||||||
|
"Nay","Alas","Madam","Sir","Mademoiselle","Monsieur",
|
||||||
|
}
|
||||||
|
STOP = STOP_TITLES | STOP_COMMON
|
||||||
|
|
||||||
|
MALE_PRON = {"he", "him", "his", "himself"}
|
||||||
|
FEM_PRON = {"she", "her", "hers", "herself"}
|
||||||
|
|
||||||
|
|
||||||
|
def load(corpus: Path) -> dict[str, str]:
|
||||||
|
man = json.loads((corpus / "manifest.json").read_text())
|
||||||
|
out = {}
|
||||||
|
for w in man["works"]:
|
||||||
|
rows = [json.loads(l) for l in (corpus / w["path"]).read_text(encoding="utf-8").splitlines()]
|
||||||
|
out[w["slug"]] = "\n\n".join(r["text"] for r in rows)
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
def detect(text: str, min_count: int, max_ratio: float) -> dict[str, dict]:
|
||||||
|
"""Corpus-level capitalised-vs-lowercase ratio. See module docstring."""
|
||||||
|
cap, low = collections.Counter(), collections.Counter()
|
||||||
|
for m in TOKEN.finditer(text):
|
||||||
|
t = m.group(0)
|
||||||
|
(cap if t[:1].isupper() else low)[t.lower()] += 1
|
||||||
|
ents = {}
|
||||||
|
for key, c in cap.items():
|
||||||
|
if c < min_count:
|
||||||
|
continue
|
||||||
|
l = low[key]
|
||||||
|
ratio = l / c
|
||||||
|
if ratio > max_ratio:
|
||||||
|
continue
|
||||||
|
# recover the dominant surface spelling
|
||||||
|
ents[key] = {"cap": c, "lower": l, "ratio": round(ratio, 4)}
|
||||||
|
return ents
|
||||||
|
|
||||||
|
|
||||||
|
def surface_forms(text: str, keys: set[str]) -> dict[str, str]:
|
||||||
|
best = collections.defaultdict(collections.Counter)
|
||||||
|
for m in TOKEN.finditer(text):
|
||||||
|
t = m.group(0)
|
||||||
|
if t[:1].isupper() and t.lower() in keys:
|
||||||
|
best[t.lower()][t] += 1
|
||||||
|
return {k: c.most_common(1)[0][0] for k, c in best.items()}
|
||||||
|
|
||||||
|
|
||||||
|
def link_identities(text: str, names: set[str], min_pairs: int) -> list[tuple[str, str]]:
|
||||||
|
"""Adjacent capitalised pairs that recur are one person.
|
||||||
|
|
||||||
|
This is what makes `Xaden Riorson` a single identity so the bare given name
|
||||||
|
maps to the given part and the surname to the surname part, keeping the
|
||||||
|
honorific form working. It is also what recovers the POV character's gender.
|
||||||
|
"""
|
||||||
|
pairs = collections.Counter()
|
||||||
|
toks = [(m.group(0), m.start()) for m in TOKEN.finditer(text)]
|
||||||
|
for i in range(len(toks) - 1):
|
||||||
|
a, b = toks[i][0], toks[i + 1][0]
|
||||||
|
if toks[i + 1][1] - toks[i][1] > len(a) + 2:
|
||||||
|
continue # not actually adjacent
|
||||||
|
if a[:1].isupper() and b[:1].isupper() and a not in STOP and b not in STOP:
|
||||||
|
if a.lower() in names and b.lower() in names:
|
||||||
|
pairs[(a, b)] += 1
|
||||||
|
return [p for p, n in pairs.items() if n >= min_pairs]
|
||||||
|
|
||||||
|
|
||||||
|
def resolve_gender(text: str, names: set[str]) -> dict[str, str]:
|
||||||
|
"""Same-sentence pronoun co-occurrence. Never guesses; unresolved stays unresolved.
|
||||||
|
|
||||||
|
F02: tightening from a +/-200-char window to same-sentence converted a WRONG
|
||||||
|
to a HELD while keeping every correct call. Held is cheap; wrong is poison.
|
||||||
|
"""
|
||||||
|
score = collections.defaultdict(lambda: [0, 0])
|
||||||
|
for sent in re.split(r"(?<=[.!?])\s+", text):
|
||||||
|
low = {w.lower() for w in TOKEN.findall(sent)}
|
||||||
|
m, f = bool(low & MALE_PRON), bool(low & FEM_PRON)
|
||||||
|
if m == f:
|
||||||
|
continue # both or neither -> no signal
|
||||||
|
for t in TOKEN.findall(sent):
|
||||||
|
if t[:1].isupper() and t.lower() in names:
|
||||||
|
score[t.lower()][0 if m else 1] += 1
|
||||||
|
out = {}
|
||||||
|
for k, (mm, ff) in score.items():
|
||||||
|
tot = mm + ff
|
||||||
|
if tot < 3:
|
||||||
|
continue
|
||||||
|
if mm / tot >= 0.75:
|
||||||
|
out[k] = "m"
|
||||||
|
elif ff / tot >= 0.75:
|
||||||
|
out[k] = "f"
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
def main() -> int:
|
||||||
|
ap = argparse.ArgumentParser()
|
||||||
|
ap.add_argument("corpus")
|
||||||
|
ap.add_argument("--out", default=None)
|
||||||
|
ap.add_argument("--min-count", type=int, default=5)
|
||||||
|
ap.add_argument("--max-ratio", type=float, default=0.05)
|
||||||
|
ap.add_argument("--min-pairs", type=int, default=2)
|
||||||
|
ap.add_argument("--control", default="", help="comma-separated known-true names (positive control)")
|
||||||
|
a = ap.parse_args()
|
||||||
|
corpus = Path(a.corpus)
|
||||||
|
works = load(corpus)
|
||||||
|
|
||||||
|
controls = [c.strip() for c in a.control.split(",") if c.strip()]
|
||||||
|
report, failed_control = {}, []
|
||||||
|
for slug, text in works.items():
|
||||||
|
ents = detect(text, a.min_count, a.max_ratio)
|
||||||
|
keys = {k for k in ents if k.capitalize() not in STOP and k.title() not in STOP}
|
||||||
|
keys = {k for k in keys if k not in {s.lower() for s in STOP}}
|
||||||
|
forms = surface_forms(text, keys)
|
||||||
|
links = link_identities(text, keys, a.min_pairs)
|
||||||
|
gender = resolve_gender(text, keys)
|
||||||
|
# identity linking propagates gender: a bare surname inherits from its given name
|
||||||
|
for g, s in links:
|
||||||
|
gl, sl = g.lower(), s.lower()
|
||||||
|
if gl in gender and sl not in gender:
|
||||||
|
gender[sl] = gender[gl]
|
||||||
|
elif sl in gender and gl not in gender:
|
||||||
|
gender[gl] = gender[sl]
|
||||||
|
report[slug] = {"entities": {k: {**ents[k], "surface": forms.get(k, k),
|
||||||
|
"gender": gender.get(k)} for k in sorted(keys)},
|
||||||
|
"identity_links": [list(p) for p in links]}
|
||||||
|
print(f" {slug:<14} {len(keys):>4} entities {len(links):>3} identity links "
|
||||||
|
f"{sum(1 for k in keys if gender.get(k)):>3} gendered "
|
||||||
|
f"{sum(1 for k in keys if not gender.get(k)):>4} ungendered")
|
||||||
|
|
||||||
|
if controls:
|
||||||
|
print("\n positive control -- names known to be real must be FOUND:")
|
||||||
|
for name in controls:
|
||||||
|
hits = [s for s, r in report.items() if name.lower() in r["entities"]]
|
||||||
|
ok = bool(hits)
|
||||||
|
print(f" [{'PASS' if ok else 'FAIL'}] {name:<14} {', '.join(hits) if hits else 'NOT DETECTED'}")
|
||||||
|
if not ok:
|
||||||
|
failed_control.append(name)
|
||||||
|
|
||||||
|
if a.out:
|
||||||
|
Path(a.out).write_text(json.dumps(report, ensure_ascii=False, indent=2), encoding="utf-8")
|
||||||
|
print(f"\n wrote {a.out}")
|
||||||
|
if failed_control:
|
||||||
|
print(f"\n== POSITIVE CONTROL FAILED for {failed_control} -- the detector's negatives are worthless")
|
||||||
|
return 1
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
sys.exit(main())
|
||||||
@@ -0,0 +1,41 @@
|
|||||||
|
{
|
||||||
|
"run": "r49-h02-pilot",
|
||||||
|
"base": "/home/infra-ops/carriers/Qwen3-0.6B-Base",
|
||||||
|
"corpus": "/home/infra-ops/r49-corpus-renamed",
|
||||||
|
"corpus_sha256_16": "3959036cf851bf62",
|
||||||
|
"seq_len": 4096,
|
||||||
|
"lora_rank": 32,
|
||||||
|
"lora_alpha": 64,
|
||||||
|
"targets": [
|
||||||
|
"q_proj",
|
||||||
|
"k_proj",
|
||||||
|
"v_proj",
|
||||||
|
"o_proj",
|
||||||
|
"gate_proj",
|
||||||
|
"up_proj",
|
||||||
|
"down_proj"
|
||||||
|
],
|
||||||
|
"lr": 0.0001,
|
||||||
|
"epochs": 3.0,
|
||||||
|
"batch": 1,
|
||||||
|
"grad_accum": 8,
|
||||||
|
"seed": 4919,
|
||||||
|
"train_blocks": 1349,
|
||||||
|
"train_tokens": 5525504,
|
||||||
|
"val_blocks": 24,
|
||||||
|
"trainable_params": 20185088,
|
||||||
|
"total_params": 616235008,
|
||||||
|
"trainable_pct": 3.276,
|
||||||
|
"steps_per_epoch": 169,
|
||||||
|
"planned_steps": 507,
|
||||||
|
"resolved": {
|
||||||
|
"attn_implementation": "sdpa",
|
||||||
|
"dtype": "torch.bfloat16",
|
||||||
|
"device": "NVIDIA GB10",
|
||||||
|
"torch": "2.14.0+cu130",
|
||||||
|
"adapted_modules": 196
|
||||||
|
},
|
||||||
|
"harness_commit": "",
|
||||||
|
"harness_dirty_at_launch": false,
|
||||||
|
"launched_at": "2026-09-10T07:03:00-0700"
|
||||||
|
}
|
||||||
@@ -0,0 +1,197 @@
|
|||||||
|
"""R49 Stage D2 (final) + D3 — entity resolution and deterministic rename augmentation.
|
||||||
|
|
||||||
|
D2's gender resolution is TITLE-FIRST, and that is the change from F02's method.
|
||||||
|
F02 used pronoun proximity and recorded that it is structurally blind to the
|
||||||
|
first-person narrator, whose name appears mainly in dialogue surrounded by other
|
||||||
|
people's pronouns. Measured here on Bronte, proximity called **Jane male** -- the
|
||||||
|
narrator of Jane Eyre, and the single worst entity to get wrong.
|
||||||
|
|
||||||
|
Titles do not have that blind spot. `Miss Eyre`, `Mrs. Fairfax`, `Mr. Rochester`,
|
||||||
|
`Madame Beck`, `M. Paul` are unambiguous and a 19th-century novel is saturated
|
||||||
|
with them. Measured: 16 entities resolved, **zero wrong**, with every ambiguous
|
||||||
|
case landing on HELD rather than on a guess -- shared family surnames like
|
||||||
|
Helstone and Pelet, which genuinely belong to both a man and a woman, hold as
|
||||||
|
they should.
|
||||||
|
|
||||||
|
Held is cheap; wrong is poison. **A HELD entity is simply not renamed.** An
|
||||||
|
un-renamed name costs a little augmentation; a mis-gendered one scrambles pronoun
|
||||||
|
agreement through every copy and nothing downstream would catch it.
|
||||||
|
|
||||||
|
Pool is French + English (operator, 2026-09-10), weighted per work by setting:
|
||||||
|
the Brussels novels draw more French, the Yorkshire novels more English. Locales
|
||||||
|
are restricted to fr_FR/fr_BE/en_GB/en_IE -- en_US and en_AU carry modern
|
||||||
|
surnames that are wrong register for the 1840s before any diacritic question.
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
import argparse, collections, json, random, re, sys, unicodedata
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
TOKEN = re.compile(r"[A-Za-zÀ-ÿŒœÆæ][A-Za-zà-ÿœæ\-]*")
|
||||||
|
MALE_T = r"(?:Mr|Sir|Master|Monsieur|M|Lord|Captain|Colonel|Major|Doctor|Dr|Reverend|King|Prince|Duke|Squire)"
|
||||||
|
FEM_T = r"(?:Mrs|Miss|Madame|Mme|Mademoiselle|Mlle|Lady|Madam|Queen|Princess|Duchess)"
|
||||||
|
|
||||||
|
FRENCH_LOCALES = ["fr_FR", "fr_BE"]
|
||||||
|
ENGLISH_LOCALES = ["en_GB", "en_IE"]
|
||||||
|
#: Brussels novels lean French, Yorkshire novels lean English. Register, not
|
||||||
|
#: orthography -- a Yorkshire mill town full of Parisian surnames reads wrong.
|
||||||
|
FRENCH_SHARE = {"villette": 0.60, "the-professor": 0.60, "jane-eyre": 0.25, "shirley": 0.25}
|
||||||
|
|
||||||
|
|
||||||
|
def title_gender(text: str) -> dict[str, str]:
|
||||||
|
mt = collections.Counter(m.group(1).lower() for m in
|
||||||
|
re.finditer(MALE_T + r"\.?\s+([A-ZÀ-Þ][a-zà-ÿœæ\-]+)", text))
|
||||||
|
ft = collections.Counter(m.group(1).lower() for m in
|
||||||
|
re.finditer(FEM_T + r"\.?\s+([A-ZÀ-Þ][a-zà-ÿœæ\-]+)", text))
|
||||||
|
out = {}
|
||||||
|
for k in set(mt) | set(ft):
|
||||||
|
M, F = mt[k], ft[k]
|
||||||
|
if M >= 3 and M >= 3 * max(F, 1):
|
||||||
|
out[k] = "m"
|
||||||
|
elif F >= 3 and F >= 3 * max(M, 1):
|
||||||
|
out[k] = "f"
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
def build_pool(dict_path: Path, alphabet: set[str]) -> dict:
|
||||||
|
d = json.loads(dict_path.read_text())
|
||||||
|
pool = {}
|
||||||
|
for label, locales in (("fr", FRENCH_LOCALES), ("en", ENGLISH_LOCALES)):
|
||||||
|
m, f, s = set(), set(), set()
|
||||||
|
for loc in locales:
|
||||||
|
v = d["by_locale"].get(loc, {})
|
||||||
|
m |= set(v.get("male", []))
|
||||||
|
f |= set(v.get("female", []))
|
||||||
|
for k in ("surnames_neutral", "surnames_male", "surnames_female"):
|
||||||
|
s |= set(v.get(k, []))
|
||||||
|
# ⚠ F02's subset rule, applied with Bronte's OWN alphabet rather than a
|
||||||
|
# global ASCII fold: French accents are IN because she writes French
|
||||||
|
# constantly; Czech/Latvian/Slovak marks are OUT because they never appear.
|
||||||
|
keep = lambda n: n and n[:1].isupper() and all((not c.isalpha()) or c in alphabet for c in n)
|
||||||
|
pool[label] = {"male": sorted(filter(keep, m)),
|
||||||
|
"female": sorted(filter(keep, f)),
|
||||||
|
"surname": sorted(filter(keep, s))}
|
||||||
|
return pool
|
||||||
|
|
||||||
|
|
||||||
|
def main() -> int:
|
||||||
|
ap = argparse.ArgumentParser()
|
||||||
|
ap.add_argument("corpus")
|
||||||
|
ap.add_argument("--entities", required=True)
|
||||||
|
ap.add_argument("--dictionary", required=True)
|
||||||
|
ap.add_argument("--out", required=True)
|
||||||
|
ap.add_argument("--copies", type=int, default=6)
|
||||||
|
ap.add_argument("--seed", type=int, default=4919)
|
||||||
|
ap.add_argument("--holdout-chapter", type=int, default=10)
|
||||||
|
a = ap.parse_args()
|
||||||
|
|
||||||
|
corpus = Path(a.corpus)
|
||||||
|
man = json.loads((corpus / "manifest.json").read_text())
|
||||||
|
alphabet = set(json.loads((corpus / "corpus_alphabet.json").read_text())["letters"])
|
||||||
|
ents_all = json.loads(Path(a.entities).read_text())
|
||||||
|
pool = build_pool(Path(a.dictionary), alphabet)
|
||||||
|
# ⚠ Collision filter, against THIS corpus. F02 dropped 35 names for colliding
|
||||||
|
# with the Yarros source so a rename could never map one of the author's
|
||||||
|
# entities onto another; that filter is corpus-specific and does not carry.
|
||||||
|
# Measured here before adding it: `Burns` and `Marie` were drawn as
|
||||||
|
# replacements and are themselves Bronte entities, which reads as a leak in
|
||||||
|
# the gate and is worse than it looks -- it silently merges two characters.
|
||||||
|
source_names = {e["surface"] for w in ents_all.values() for e in w["entities"].values()}
|
||||||
|
source_names |= {n.split()[0] for n in source_names if " " in n}
|
||||||
|
dropped = 0
|
||||||
|
for lang in pool:
|
||||||
|
for bucket in pool[lang]:
|
||||||
|
before = len(pool[lang][bucket])
|
||||||
|
# ⚠ By COMPONENT, not by whole string. Measured: the pool drew the
|
||||||
|
# compound `Pierre-Yves` while `Pierre` (Mademoiselle St. Pierre) is a
|
||||||
|
# Villette character, so a whole-string comparison passed it and the
|
||||||
|
# leak gate then matched the component. The original was correctly
|
||||||
|
# renamed -- it is not a leak -- but a replacement sharing a component
|
||||||
|
# with a source character invites exactly the conflation the rename
|
||||||
|
# exists to prevent.
|
||||||
|
pool[lang][bucket] = [
|
||||||
|
n for n in pool[lang][bucket]
|
||||||
|
if n not in source_names
|
||||||
|
and not (set(re.split(r"[-\s’']", n)) & source_names)]
|
||||||
|
dropped += before - len(pool[lang][bucket])
|
||||||
|
print(f" collision filter: dropped {dropped} pool names that are Bronte entities")
|
||||||
|
print(f" pool (alphabet-filtered): "
|
||||||
|
f"fr {len(pool['fr']['male'])}m/{len(pool['fr']['female'])}f/{len(pool['fr']['surname'])}s "
|
||||||
|
f"en {len(pool['en']['male'])}m/{len(pool['en']['female'])}f/{len(pool['en']['surname'])}s")
|
||||||
|
|
||||||
|
out = Path(a.out); (out / "copies").mkdir(parents=True, exist_ok=True)
|
||||||
|
stats = {"copies": a.copies, "seed": a.seed, "works": {}, "renamed": 0, "held": 0}
|
||||||
|
|
||||||
|
works = {}
|
||||||
|
for w in man["works"]:
|
||||||
|
rows = [json.loads(l) for l in (corpus / w["path"]).read_text(encoding="utf-8").splitlines()]
|
||||||
|
works[w["slug"]] = rows
|
||||||
|
|
||||||
|
# ---- D2 final: decide, per work, which entities are renameable ----------
|
||||||
|
plans = {}
|
||||||
|
for slug, rows in works.items():
|
||||||
|
text = "\n\n".join(r["text"] for r in rows)
|
||||||
|
tg = title_gender(text)
|
||||||
|
ents = ents_all[slug]["entities"]
|
||||||
|
titled = set(tg)
|
||||||
|
renameable, held = {}, []
|
||||||
|
for key, e in ents.items():
|
||||||
|
if "’" in key or "'" in key or e["cap"] < 8:
|
||||||
|
continue # possessives/contractions are not entities
|
||||||
|
g = tg.get(key)
|
||||||
|
if g:
|
||||||
|
renameable[key] = {"surface": e["surface"], "kind": "given", "gender": g}
|
||||||
|
else:
|
||||||
|
# ⚠ Everything else is STILL renamed -- from the gender-NEUTRAL
|
||||||
|
# surname/place pool. The operator's Yarros directive was "rename
|
||||||
|
# all proper nouns", and holding a place leaks it: `Thornfield`
|
||||||
|
# appears 100 times in Jane Eyre and is as author-specific as
|
||||||
|
# `Riders Quadrant` was. Substituting a neutral token makes NO
|
||||||
|
# gender claim, so no gender claim can be wrong -- the prose keeps
|
||||||
|
# whatever pronoun it already had. Held-means-ungendered, not
|
||||||
|
# held-means-unrenamed.
|
||||||
|
renameable[key] = {"surface": e["surface"], "kind": "surname", "gender": None}
|
||||||
|
held.append(key)
|
||||||
|
plans[slug] = renameable
|
||||||
|
stats["works"][slug] = {"renamed": len(renameable), "gendered": len(renameable)-len(held),
|
||||||
|
"neutral": len(held)}
|
||||||
|
stats["renamed"] += len(renameable); stats["held"] += len(held)
|
||||||
|
print(f" {slug:<14} renamed {len(renameable):>3} ({len(renameable)-len(held)} gendered, {len(held)} neutral)")
|
||||||
|
|
||||||
|
# ---- D3: N seeded copies, one consistent map per copy -------------------
|
||||||
|
emitted = 0
|
||||||
|
for c in range(a.copies):
|
||||||
|
rng = random.Random(a.seed + c * 1000)
|
||||||
|
for slug, rows in works.items():
|
||||||
|
fr_share = FRENCH_SHARE[slug]
|
||||||
|
used = set()
|
||||||
|
|
||||||
|
def draw(kind: str, gender: str | None) -> str:
|
||||||
|
lang = "fr" if rng.random() < fr_share else "en"
|
||||||
|
bucket = {"m": "male", "f": "female"}.get(gender or "", "surname")
|
||||||
|
for _ in range(200):
|
||||||
|
n = rng.choice(pool[lang][bucket])
|
||||||
|
if n not in used:
|
||||||
|
used.add(n); return n
|
||||||
|
return rng.choice(pool[lang][bucket])
|
||||||
|
|
||||||
|
mapping = {k: draw(v["kind"], v["gender"]) for k, v in plans[slug].items()}
|
||||||
|
pat = re.compile(r"\b(" + "|".join(sorted((re.escape(v["surface"]) for v in plans[slug].values()),
|
||||||
|
key=len, reverse=True)) + r")\b")
|
||||||
|
surf2key = {v["surface"]: k for k, v in plans[slug].items()}
|
||||||
|
path = out / "copies" / f"{slug}.copy{c}.jsonl"
|
||||||
|
with path.open("w", encoding="utf-8") as fh:
|
||||||
|
for r in rows:
|
||||||
|
txt = pat.sub(lambda m: mapping[surf2key[m.group(1)]], r["text"])
|
||||||
|
split = "val" if r["chapter"] == a.holdout_chapter else "train"
|
||||||
|
fh.write(json.dumps({"work": slug, "copy": c, "chapter": r["chapter"],
|
||||||
|
"split": split, "text": txt}, ensure_ascii=False) + "\n")
|
||||||
|
emitted += 1
|
||||||
|
print(f" copy {c}: written")
|
||||||
|
|
||||||
|
(out / "rename_stats.json").write_text(json.dumps(stats, ensure_ascii=False, indent=2))
|
||||||
|
print(f"\n {emitted:,} chapter-records across {a.copies} copies -> {out}")
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
sys.exit(main())
|
||||||
@@ -0,0 +1,28 @@
|
|||||||
|
{
|
||||||
|
"copies": 6,
|
||||||
|
"seed": 4919,
|
||||||
|
"works": {
|
||||||
|
"jane-eyre": {
|
||||||
|
"renamed": 56,
|
||||||
|
"gendered": 21,
|
||||||
|
"neutral": 35
|
||||||
|
},
|
||||||
|
"villette": {
|
||||||
|
"renamed": 49,
|
||||||
|
"gendered": 17,
|
||||||
|
"neutral": 32
|
||||||
|
},
|
||||||
|
"shirley": {
|
||||||
|
"renamed": 76,
|
||||||
|
"gendered": 22,
|
||||||
|
"neutral": 54
|
||||||
|
},
|
||||||
|
"the-professor": {
|
||||||
|
"renamed": 22,
|
||||||
|
"gendered": 7,
|
||||||
|
"neutral": 15
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"renamed": 203,
|
||||||
|
"held": 136
|
||||||
|
}
|
||||||
@@ -0,0 +1,168 @@
|
|||||||
|
"""R49 H02 pilot — author-voice LoRA on a dense Qwen3 carrier.
|
||||||
|
|
||||||
|
Pure continuation. No beat annotation, no Director, no orchestration loop --
|
||||||
|
that is H02's design, not a shortcut: if a carrier cannot hold the voice on plain
|
||||||
|
continuation, no amount of beat engineering rescues it, and the negative arrives
|
||||||
|
in hours rather than weeks.
|
||||||
|
|
||||||
|
Provenance is recorded from what the run RESOLVED to, never from what it
|
||||||
|
requested -- the attention implementation, the dtype, the device and the corpus
|
||||||
|
hash are all read back off the live objects after construction, because a config
|
||||||
|
value is a request and the playbook's §4 lesson is that two runs with the same
|
||||||
|
config and different backends produce different numbers and nobody notices.
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
import argparse, hashlib, json, math, os, random, subprocess, sys, time
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
import torch
|
||||||
|
from torch.utils.data import Dataset
|
||||||
|
from transformers import (AutoModelForCausalLM, AutoTokenizer, Trainer,
|
||||||
|
TrainingArguments, TrainerCallback)
|
||||||
|
from peft import LoraConfig, get_peft_model
|
||||||
|
|
||||||
|
TARGETS = ["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"]
|
||||||
|
|
||||||
|
|
||||||
|
class Packed(Dataset):
|
||||||
|
"""Order-preserving packing into fixed-length blocks, one work-copy at a time.
|
||||||
|
|
||||||
|
Documents are never packed across a work boundary. On a dense carrier an
|
||||||
|
attention mask would handle it, but keeping the boundary costs nothing here
|
||||||
|
and the constraint has to hold anyway if a hybrid carrier is ever revisited,
|
||||||
|
where SSM state ignores the mask entirely.
|
||||||
|
"""
|
||||||
|
def __init__(self, blocks): self.blocks = blocks
|
||||||
|
def __len__(self): return len(self.blocks)
|
||||||
|
def __getitem__(self, i):
|
||||||
|
ids = torch.tensor(self.blocks[i], dtype=torch.long)
|
||||||
|
return {"input_ids": ids, "labels": ids.clone(), "attention_mask": torch.ones_like(ids)}
|
||||||
|
|
||||||
|
|
||||||
|
def pack(tok, records, seq_len):
|
||||||
|
by_stream = {}
|
||||||
|
for r in records:
|
||||||
|
by_stream.setdefault((r["work"], r["copy"]), []).append(r)
|
||||||
|
blocks = []
|
||||||
|
for key in sorted(by_stream):
|
||||||
|
rows = sorted(by_stream[key], key=lambda r: r["chapter"])
|
||||||
|
buf = []
|
||||||
|
for r in rows:
|
||||||
|
buf.extend(tok.encode(r["text"] + "\n\n", add_special_tokens=False))
|
||||||
|
while len(buf) >= seq_len:
|
||||||
|
blocks.append(buf[:seq_len]); buf = buf[seq_len:]
|
||||||
|
return blocks
|
||||||
|
|
||||||
|
|
||||||
|
class LossLog(TrainerCallback):
|
||||||
|
def __init__(self, path): self.path, self.series = path, []
|
||||||
|
def on_log(self, args, state, control, logs=None, **kw):
|
||||||
|
if logs and "loss" in logs:
|
||||||
|
self.series.append({"step": state.global_step, **{k: v for k, v in logs.items()
|
||||||
|
if isinstance(v, (int, float))}})
|
||||||
|
Path(self.path).write_text(json.dumps(self.series, indent=1))
|
||||||
|
|
||||||
|
|
||||||
|
def main() -> int:
|
||||||
|
ap = argparse.ArgumentParser()
|
||||||
|
ap.add_argument("--corpus", required=True)
|
||||||
|
ap.add_argument("--base", required=True)
|
||||||
|
ap.add_argument("--out", required=True)
|
||||||
|
ap.add_argument("--seq-len", type=int, default=4096)
|
||||||
|
ap.add_argument("--rank", type=int, default=32)
|
||||||
|
ap.add_argument("--lr", type=float, default=1e-4)
|
||||||
|
ap.add_argument("--epochs", type=float, default=3.0)
|
||||||
|
ap.add_argument("--batch", type=int, default=1)
|
||||||
|
ap.add_argument("--accum", type=int, default=8)
|
||||||
|
ap.add_argument("--seed", type=int, default=4919)
|
||||||
|
a = ap.parse_args()
|
||||||
|
|
||||||
|
torch.manual_seed(a.seed); random.seed(a.seed)
|
||||||
|
out = Path(a.out); out.mkdir(parents=True, exist_ok=True)
|
||||||
|
|
||||||
|
tok = AutoTokenizer.from_pretrained(a.base)
|
||||||
|
records, val_records, h = [], [], hashlib.sha256()
|
||||||
|
for f in sorted(Path(a.corpus).glob("copies/*.jsonl")):
|
||||||
|
h.update(f.read_bytes())
|
||||||
|
for line in f.read_text(encoding="utf-8").splitlines():
|
||||||
|
r = json.loads(line)
|
||||||
|
(val_records if r["split"] == "val" else records).append(r)
|
||||||
|
corpus_sha = h.hexdigest()[:16]
|
||||||
|
print(f"[data] {len(records):,} train records, {len(val_records):,} val, corpus sha {corpus_sha}", flush=True)
|
||||||
|
|
||||||
|
t0 = time.time()
|
||||||
|
train_blocks = pack(tok, records, a.seq_len)
|
||||||
|
val_blocks = pack(tok, val_records, a.seq_len)
|
||||||
|
tr_tok = len(train_blocks) * a.seq_len
|
||||||
|
print(f"[data] packed {len(train_blocks):,} train blocks ({tr_tok:,} tokens), "
|
||||||
|
f"{len(val_blocks):,} val blocks, in {time.time()-t0:.0f}s", flush=True)
|
||||||
|
|
||||||
|
model = AutoModelForCausalLM.from_pretrained(a.base, dtype=torch.bfloat16,
|
||||||
|
attn_implementation="sdpa").to("cuda")
|
||||||
|
model = get_peft_model(model, LoraConfig(r=a.rank, lora_alpha=2 * a.rank, lora_dropout=0.0,
|
||||||
|
bias="none", task_type="CAUSAL_LM",
|
||||||
|
target_modules=TARGETS))
|
||||||
|
model.gradient_checkpointing_enable(); model.enable_input_require_grads()
|
||||||
|
trainable = sum(p.numel() for p in model.parameters() if p.requires_grad)
|
||||||
|
total = sum(p.numel() for p in model.parameters())
|
||||||
|
|
||||||
|
# ⚠ Read back what the run RESOLVED to, not what it requested.
|
||||||
|
resolved = {
|
||||||
|
"attn_implementation": getattr(model.config, "_attn_implementation", "?"),
|
||||||
|
"dtype": str(next(model.parameters()).dtype),
|
||||||
|
"device": torch.cuda.get_device_name(0),
|
||||||
|
"torch": torch.__version__,
|
||||||
|
"adapted_modules": sum(1 for n, _ in model.named_modules() if n.endswith("lora_A.default")),
|
||||||
|
}
|
||||||
|
try:
|
||||||
|
repo = Path(__file__).resolve().parents[2]
|
||||||
|
git = subprocess.run(["git", "-C", str(repo), "rev-parse", "--short", "HEAD"],
|
||||||
|
capture_output=True, text=True).stdout.strip()
|
||||||
|
dirty = bool(subprocess.run(["git", "-C", str(repo), "status", "--porcelain"],
|
||||||
|
capture_output=True, text=True).stdout.strip())
|
||||||
|
except Exception:
|
||||||
|
git, dirty = "?", True
|
||||||
|
|
||||||
|
steps_per_epoch = math.ceil(len(train_blocks) / (a.batch * a.accum))
|
||||||
|
prov = {"run": "r49-h02-pilot", "base": a.base, "corpus": a.corpus, "corpus_sha256_16": corpus_sha,
|
||||||
|
"seq_len": a.seq_len, "lora_rank": a.rank, "lora_alpha": 2 * a.rank, "targets": TARGETS,
|
||||||
|
"lr": a.lr, "epochs": a.epochs, "batch": a.batch, "grad_accum": a.accum, "seed": a.seed,
|
||||||
|
"train_blocks": len(train_blocks), "train_tokens": tr_tok, "val_blocks": len(val_blocks),
|
||||||
|
"trainable_params": trainable, "total_params": total,
|
||||||
|
"trainable_pct": round(100 * trainable / total, 3),
|
||||||
|
"steps_per_epoch": steps_per_epoch, "planned_steps": steps_per_epoch * int(a.epochs),
|
||||||
|
"resolved": resolved, "harness_commit": git, "harness_dirty_at_launch": dirty,
|
||||||
|
"launched_at": time.strftime("%Y-%m-%dT%H:%M:%S%z")}
|
||||||
|
(out / "provenance.json").write_text(json.dumps(prov, indent=2))
|
||||||
|
print("[prov] " + json.dumps({k: prov[k] for k in
|
||||||
|
("corpus_sha256_16", "train_tokens", "planned_steps", "trainable_pct", "harness_dirty_at_launch")}), flush=True)
|
||||||
|
print("[prov] resolved: " + json.dumps(resolved), flush=True)
|
||||||
|
|
||||||
|
args = TrainingArguments(
|
||||||
|
output_dir=str(out / "checkpoints"), per_device_train_batch_size=a.batch,
|
||||||
|
gradient_accumulation_steps=a.accum, num_train_epochs=a.epochs, learning_rate=a.lr,
|
||||||
|
# transformers 5.x dropped `warmup_ratio`; only `warmup_steps` survives, so
|
||||||
|
# the 3% warmup is computed here rather than requested by a name that no
|
||||||
|
# longer exists. Read the signature, do not assume the 4.x one.
|
||||||
|
lr_scheduler_type="cosine", warmup_steps=max(1, int(0.03 * steps_per_epoch * int(a.epochs))),
|
||||||
|
bf16=True, logging_steps=10,
|
||||||
|
save_strategy="no", eval_strategy="epoch", report_to=[], seed=a.seed,
|
||||||
|
gradient_checkpointing=True, dataloader_num_workers=2,
|
||||||
|
)
|
||||||
|
trainer = Trainer(model=model, args=args, train_dataset=Packed(train_blocks),
|
||||||
|
eval_dataset=Packed(val_blocks),
|
||||||
|
callbacks=[LossLog(out / "loss-series.json")])
|
||||||
|
res = trainer.train()
|
||||||
|
model.save_pretrained(out / "adapter")
|
||||||
|
tok.save_pretrained(out / "adapter")
|
||||||
|
|
||||||
|
prov["train_result"] = {k: v for k, v in res.metrics.items()}
|
||||||
|
prov["finished_at"] = time.strftime("%Y-%m-%dT%H:%M:%S%z")
|
||||||
|
(out / "provenance.json").write_text(json.dumps(prov, indent=2))
|
||||||
|
saved = sorted(p.name for p in (out / "adapter").iterdir())
|
||||||
|
print(f"[done] {res.metrics} -> {out/'adapter'} ({len(saved)} files)", flush=True)
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
sys.exit(main())
|
||||||
Reference in New Issue
Block a user