vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity

feat(r49): D1 corpus built and green — Charlotte Brontë, 680k words, 951k tokens

Browse Source 
scripts/r49-corpus/{build_corpus,verify_corpus}.py; corpus staged at
gx10:~/r49-corpus/. Catalogue ids verified against gutenberg.org's own search
rather than recalled. Charlotte only -- the Bell poems are co-authored and the
Gaskell biography is a different hand, so neither belongs in a single-voice corpus.

  Jane Eyre 1260 · Villette 9182 · Shirley 30486 · The Professor 1028
  680,291 words · 142 chapters · 950,974 Qwen3 tokens (1.40 tok/word)
  alphabet 75 letters, 23 non-ASCII · round-trip lossless · 0 byte-fallback

All 11 acceptance checks pass, including both tokenizer legs run against the pilot
carrier itself. With a real denominator the projections tighten: at 6 rename copies
x 3 epochs = 17.1M tokens, the 0.6B pilot is 1.98 h.

THE ALPHABET INVERTS THE YARROS RESULT. Brontë writes French constantly -- Villette
is set in a French-speaking city, Jane Eyre has Adèle, The Professor is set in
Brussels -- so the corpus carries é 432, è 237, à 93, ê 79, ô 48 plus œ and æ. F02
measured Yarros at 0.0002% non-ASCII and derived an ASCII-fold for the name pool.
Under F02's own subset rule the Brontë pool may keep FRENCH accents and must still
exclude the Czech/Latvian/Slovak/Hungarian marks that never appear here. The fold is
per-work, and this is the first corpus where deriving it changes the answer.

Typography was inconsistent across works and it was the transcriber, not the author:
Shirley uses straight quotes and `--` with zero em-dashes while Jane Eyre and
Villette use curly and em-dash. Normalised toward what the text means.

Three defects, each found by running something rather than reasoning about it:
`Produced by` matched Brontë's own prose four times, which is the adjective-"minor"
shape again and is fixed by anchoring boilerplate patterns to line start; asserting
open/close quote counts must be equal is wrong, because 19th-century multi-paragraph
speech legitimately runs a surplus of opens, so the real error signature is that no
paragraph may begin with a closing quote; and The Professor's table of contents puts
two chapter names per line, so a bare regex returns 38 headings for a 25-chapter
novel and a minimum-gap filter still leaks its tail -- the rule that works is that
the body's "CHAPTER I" is the last one in the file.

Records the operator's pilot ruling: trial on Qwen3-0.6B-Base first, move up only if
it produces something useful.
This commit is contained in:
vh
2026-09-10 06:51:32 -07:00
parent 6cbc9c7a2c
commit ba8dac2c80
6 changed files with 644 additions and 4 deletions
Download Patch File Download Diff File Expand all files Collapse all files
scripts/r49-corpus/build_corpus.py
+269
View File
@@ -0,0 +1,269 @@
"""R49 Stage D1 — acquire and clean a public-domain author corpus.
Charlotte Brontë's four novels from Project Gutenberg, stripped of boilerplate,
chapter-segmented, typography-normalised, with the corpus's own character
inventory derived from the result.
The alphabet is not cosmetic. R49 F02's rule is that the rename pool's character
inventory must be a SUBSET of the source corpus's -- substituting a 26%-diacritic
name pool into prose the author wrote in plain ASCII teaches the adapter a false
orthographic habit, landing directly on the axis being trained. So the corpus
derives the constraint and the pool obeys it, per work.
Two stages on purpose. `--survey` reports what is actually in the text before any
normalisation is chosen; normalisation decided from a guess rather than from the
survey is how a cleanup silently deletes something. Run the survey, read it, then
run the build.
python build_corpus.py --survey # measure, change nothing
python build_corpus.py --build --out DIR # emit the cleaned corpus
"""
from __future__ import annotations
import argparse, collections, json, re, sys, unicodedata, urllib.request
from pathlib import Path
# Catalogue ids verified against gutenberg.org's own search 2026-09-10, not
# recalled. Charlotte only -- the Bell poems are co-authored and the Gaskell
# biography is a different hand, so neither belongs in a single-voice corpus.
WORKS = [
{"id": 1260, "slug": "jane-eyre", "title": "Jane Eyre: An Autobiography"},
{"id": 9182, "slug": "villette", "title": "Villette"},
{"id": 30486, "slug": "shirley", "title": "Shirley"},
{"id": 1028, "slug": "the-professor", "title": "The Professor"},
]
URLS = ["https://www.gutenberg.org/cache/epub/{id}/pg{id}.txt",
"https://www.gutenberg.org/files/{id}/{id}-0.txt",
"https://www.gutenberg.org/files/{id}/{id}.txt"]
START = re.compile(r"^\*\*\*\s*START OF (?:THE|THIS) PROJECT GUTENBERG EBOOK.*?\*\*\*\s*$", re.M | re.I)
END = re.compile(r"^\*\*\*\s*END OF (?:THE|THIS) PROJECT GUTENBERG EBOOK.*?\*\*\*\s*$", re.M | re.I)
CHAPTER = re.compile(r"^\s*(CHAPTER\s+[IVXLCDM]+|CHAPTER\s+\d+)\.?\s*(.*)$", re.M)
def fetch(work, cache: Path) -> str:
cache.mkdir(parents=True, exist_ok=True)
raw = cache / f"{work['slug']}.raw.txt"
if raw.exists():
return raw.read_text(encoding="utf-8")
for tmpl in URLS:
url = tmpl.format(id=work["id"])
try:
with urllib.request.urlopen(url, timeout=60) as r:
if r.status != 200:
continue
text = r.read().decode("utf-8-sig")
raw.write_text(text, encoding="utf-8")
print(f" fetched {work['slug']:<14} {url} {len(text):,} bytes")
return text
except Exception as e:
print(f" .. {url} -> {type(e).__name__}")
raise SystemExit(f"REFUSING: could not fetch {work['slug']} (id {work['id']})")
def strip_boilerplate(text: str, slug: str) -> str:
"""Keep only what lies between Gutenberg's own START/END markers.
Anchoring on the markers rather than on a line count is what makes this
safe across editions -- the front matter length differs per work.
"""
m1, m2 = START.search(text), END.search(text)
if not m1 or not m2:
raise SystemExit(f"REFUSING: {slug} has no START/END markers; refusing to guess where the text begins")
body = text[m1.end():m2.start()]
# A transcriber credit block sometimes sits just inside the START marker.
body = re.sub(r"\A\s*(?:Produced by|E-text prepared by|Transcribed from).*?\n\s*\n", "", body, flags=re.S | re.I)
return body.strip("\n")
ROMAN = {"I":1,"V":5,"X":10,"L":50,"C":100,"D":500,"M":1000}
def roman_to_int(r: str) -> int:
total, prev = 0, 0
for ch in reversed(r.upper()):
v = ROMAN.get(ch, 0)
total = total - v if v < prev else total + v
prev = max(prev, v)
return total
def find_chapters(body: str) -> list[tuple[int, str, int]]:
"""Body chapter headings only, with any table of contents discarded.
Measured 2026-09-10: The Professor ships a TOC that puts TWO chapter names
on one line, so a bare regex returns 38 headings for a 25-chapter novel and
a naive minimum-gap filter still leaks the TOC's tail. The rule that works
is structural rather than cosmetic -- the body's "CHAPTER I" is the LAST one
in the file, because a TOC always precedes the text it indexes. From there,
keep only headings that continue the sequence and are separated by prose.
"""
hits = []
for m in CHAPTER.finditer(body):
num = m.group(1).split()[-1].rstrip(".")
n = int(num) if num.isdigit() else roman_to_int(num)
hits.append((m.start(), m.group(1).strip(), n))
if not hits:
return []
ones = [i for i, h in enumerate(hits) if h[2] == 1]
start = ones[-1] if ones else 0
kept, expect, last_pos = [], 1, -10**9
for pos, label, n in hits[start:]:
if n == expect and pos - last_pos > 500:
kept.append((pos, label, n))
expect, last_pos = expect + 1, pos
return kept
#: Normalisation is decided from the survey, not from a guess. Measured across
#: the four works: Jane Eyre and Villette use curly quotes and em-dashes;
#: SHIRLEY uses straight quotes and `--` with zero em-dashes; The Professor
#: mixes curly quotes with `--`. That split is a transcriber artefact, not
#: Charlotte Bronte's punctuation, and leaving it would teach the adapter that
#: this author "sometimes" writes each form -- a false habit on the exact axis
#: being trained. Normalise toward what the text MEANS: `--` is a transcription
#: of an em-dash, so it becomes one.
def normalise_quotes(text: str) -> str:
"""Straight quotes -> curly, paired by alternation within each paragraph."""
out = []
for para in text.split("\n\n"):
buf, open_d = [], True
for ch in para:
if ch == '"':
buf.append("\u201c" if open_d else "\u201d")
open_d = not open_d
else:
buf.append(ch)
para = "".join(buf)
# single quotes: apostrophe if flanked by letters, else a quote mark
para = re.sub(r"(?<=[A-Za-z])'(?=[A-Za-z])", "\u2019", para)
buf, open_s = [], True
for ch in para:
if ch == "'":
buf.append("\u2018" if open_s else "\u2019")
open_s = not open_s
else:
buf.append(ch)
out.append("".join(buf))
return "\n\n".join(out)
def clean(text: str) -> str:
text = text.replace("\u00a0", " ")
text = re.sub(r"(?<!-)--(?!-)", "\u2014", text)
text = normalise_quotes(text)
text = re.sub(r"[ \t]+\n", "\n", text)
text = re.sub(r"\n{3,}", "\n\n", text)
return text.strip("\n")
def survey(bodies: dict[str, str]) -> None:
print("\n== character inventory, BEFORE any normalisation")
allchars = collections.Counter()
for slug, b in bodies.items():
allchars.update(b)
letters = {c for c in allchars if c.isalpha()}
ascii_letters = {c for c in letters if ord(c) < 128}
non_ascii = sorted(c for c in allchars if ord(c) > 127)
print(f" distinct characters : {len(allchars)}")
print(f" distinct letters : {len(letters)} (ascii {len(ascii_letters)}, non-ascii {len(letters - ascii_letters)})")
print(f" distinct non-ascii chars : {len(non_ascii)}")
print(" non-ascii, by frequency:")
for c in sorted(non_ascii, key=lambda c: -allchars[c]):
name = unicodedata.name(c, "?")
print(f" U+{ord(c):04X} {c!r:<8} {allchars[c]:>7} {name}")
print("\n== structure")
for slug, b in bodies.items():
heads = find_chapters(b)
words = len(b.split())
print(f" {slug:<14} {words:>8,} words {len(heads):>3} chapters last: {heads[-1][1] if heads else '-'}")
print(f" {'TOTAL':<14} {sum(len(b.split()) for b in bodies.values()):>8,} words")
def main() -> int:
ap = argparse.ArgumentParser()
ap.add_argument("--survey", action="store_true")
ap.add_argument("--build", action="store_true")
ap.add_argument("--out", default="corpus")
ap.add_argument("--cache", default="raw")
a = ap.parse_args()
if not (a.survey or a.build):
ap.error("pick --survey or --build")
cache = Path(a.cache)
print("== fetch")
bodies = {}
for w in WORKS:
bodies[w["slug"]] = strip_boilerplate(fetch(w, cache), w["slug"])
assert "PROJECT GUTENBERG" not in bodies[w["slug"]][:2000].upper(), f"{w['slug']}: boilerplate survived"
if a.survey:
survey(bodies)
return 0
out = Path(a.out)
(out / "works").mkdir(parents=True, exist_ok=True)
manifest, alphabet = [], set()
for w in WORKS:
slug = w["slug"]
body = clean(bodies[slug])
chaps = find_chapters(body)
if not chaps:
raise SystemExit(f"REFUSING: no chapters found in {slug}")
# Self-consistency: the count must equal the last heading's numeral, or
# the segmentation has silently over- or under-matched.
if len(chaps) != chaps[-1][2]:
raise SystemExit(
f"REFUSING: {slug} segmented into {len(chaps)} chapters but the last "
f"heading is {chaps[-1][1]} (= {chaps[-1][2]}). Segmentation is wrong.")
records = []
for i, (pos, label, n) in enumerate(chaps):
end = chaps[i + 1][0] if i + 1 < len(chaps) else len(body)
text = body[pos:end].strip("\n")
records.append({"work": slug, "chapter": n, "heading": label,
"words": len(text.split()), "text": text})
path = out / "works" / f"{slug}.jsonl"
with path.open("w", encoding="utf-8") as fh:
for r in records:
fh.write(json.dumps(r, ensure_ascii=False) + "\n")
alphabet |= {c for c in body if c.isalpha()}
# Relative to the corpus root, never absolute: the corpus is built on one
# box and trained on another, and an absolute build path makes the
# manifest unreadable the moment it moves.
manifest.append({"slug": slug, "gutenberg_id": w["id"], "title": w["title"],
"chapters": len(records),
"words": sum(r["words"] for r in records),
"chars": len(body), "path": f"works/{slug}.jsonl"})
print(f" wrote {slug:<14} {len(records):>3} chapters {sum(r['words'] for r in records):>8,} words")
alpha = sorted(alphabet)
(out / "corpus_alphabet.json").write_text(json.dumps({
"derived_from": "Charlotte Bronte, 4 novels, Project Gutenberg",
"derived_at": "2026-09-10",
"note": ("R49 F02 rule: a rename pool's character inventory must be a SUBSET of "
"this. Bronte writes French constantly (Villette, Adele, Brussels), so "
"unlike the Yarros corpus this alphabet legitimately carries accents -- "
"but only FRENCH ones. Czech/Latvian/Slovak/Hungarian marks never appear "
"and must not enter the pool."),
"count": len(alpha), "letters": alpha,
"non_ascii": [c for c in alpha if ord(c) > 127],
}, ensure_ascii=False, indent=2), encoding="utf-8")
(out / "manifest.json").write_text(json.dumps({
"corpus": "bronte-charlotte-v1", "built_at": "2026-09-10",
"source": "Project Gutenberg (public domain)",
"normalisation": ("no-break space -> space; `--` -> em dash; straight quotes -> "
"curly, paired per paragraph. Decided from the survey: Shirley "
"was transcribed with straight quotes and zero em-dashes while "
"Jane Eyre and Villette use curly and em-dash, a transcriber "
"split rather than the author's punctuation."),
"works": manifest,
"total_words": sum(m["words"] for m in manifest),
"total_chapters": sum(m["chapters"] for m in manifest),
}, ensure_ascii=False, indent=2), encoding="utf-8")
print(f"\n alphabet: {len(alpha)} letters ({len([c for c in alpha if ord(c)>127])} non-ascii)")
print(f" TOTAL : {sum(m['words'] for m in manifest):,} words in "
f"{sum(m['chapters'] for m in manifest)} chapters -> {out}")
return 0
if __name__ == "__main__":
sys.exit(main())
scripts/r49-corpus/corpus_alphabet.json
+108
View File
@@ -0,0 +1,108 @@
{
"derived_from": "Charlotte Bronte, 4 novels, Project Gutenberg",
"derived_at": "2026-09-10",
"note": "R49 F02 rule: a rename pool's character inventory must be a SUBSET of this. Bronte writes French constantly (Villette, Adele, Brussels), so unlike the Yarros corpus this alphabet legitimately carries accents -- but only FRENCH ones. Czech/Latvian/Slovak/Hungarian marks never appear and must not enter the pool.",
"count": 75,
"letters": [
"A",
"B",
"C",
"D",
"E",
"F",
"G",
"H",
"I",
"J",
"K",
"L",
"M",
"N",
"O",
"P",
"Q",
"R",
"S",
"T",
"U",
"V",
"W",
"X",
"Y",
"Z",
"a",
"b",
"c",
"d",
"e",
"f",
"g",
"h",
"i",
"j",
"k",
"l",
"m",
"n",
"o",
"p",
"q",
"r",
"s",
"t",
"u",
"v",
"w",
"x",
"y",
"z",
"Æ",
"É",
"Ê",
"Ë",
"Ô",
"à",
"â",
"ä",
"æ",
"ç",
"è",
"é",
"ê",
"ë",
"î",
"ï",
"ô",
"ö",
"ù",
"û",
"ü",
"Œ",
"œ"
],
"non_ascii": [
"Æ",
"É",
"Ê",
"Ë",
"Ô",
"à",
"â",
"ä",
"æ",
"ç",
"è",
"é",
"ê",
"ë",
"î",
"ï",
"ô",
"ö",
"ù",
"û",
"ü",
"Œ",
"œ"
]
}
scripts/r49-corpus/manifest.json
+46
View File
@@ -0,0 +1,46 @@
{
"corpus": "bronte-charlotte-v1",
"built_at": "2026-09-10",
"source": "Project Gutenberg (public domain)",
"normalisation": "no-break space -> space; `--` -> em dash; straight quotes -> curly, paired per paragraph. Decided from the survey: Shirley was transcribed with straight quotes and zero em-dashes while Jane Eyre and Villette use curly and em-dash, a transcriber split rather than the author's punctuation.",
"works": [
{
"slug": "jane-eyre",
"gutenberg_id": 1260,
"title": "Jane Eyre: An Autobiography",
"chapters": 38,
"words": 184452,
"chars": 1022193,
"path": "works/jane-eyre.jsonl"
},
{
"slug": "villette",
"gutenberg_id": 9182,
"title": "Villette",
"chapters": 42,
"words": 192411,
"chars": 1092741,
"path": "works/villette.jsonl"
},
{
"slug": "shirley",
"gutenberg_id": 30486,
"title": "Shirley",
"chapters": 37,
"words": 216016,
"chars": 1226278,
"path": "works/shirley.jsonl"
},
{
"slug": "the-professor",
"gutenberg_id": 1028,
"title": "The Professor",
"chapters": 25,
"words": 87412,
"chars": 500054,
"path": "works/the-professor.jsonl"
}
],
"total_words": 680291,
"total_chapters": 142
}
scripts/r49-corpus/verify_corpus.py
+108
View File
@@ -0,0 +1,108 @@
"""R49 D1 acceptance gate for a built corpus.
The design doc's D1 acceptance is "clean UTF-8, chapter-segmented, zero
boilerplate lines, stable tokenization". Each is checked here as something that
can actually go RED -- a gate that cannot fail is the third failure mode this
target has already recorded, and it is not repeated here.
python verify_corpus.py <corpus-dir> [--tokenizer PATH]
"""
from __future__ import annotations
import argparse, collections, json, re, sys, unicodedata
from pathlib import Path
#: ⚠ Anchored to line start, and that is not cosmetic. The first draft matched
#: `Produced by` anywhere and went RED on four hits that were all Charlotte
#: Bronte's own prose -- "a chilling effect produced by his steady announcement",
#: "how such a result was produced by such means". A hard rule on a phrase with a
#: common non-boilerplate sense manufactures failures; same shape as the drift
#: detector that fired on the adjective "minor" and stopped work three times.
#: Gutenberg credits always begin a line, so require that.
BOILER = [r"^.*PROJECT GUTENBERG.*$", r"^.*gutenberg\.org.*$", r"^\s*Produced by\b",
r"^\s*E-text prepared by\b", r"^\s*Transcribed from\b",
r"^\s*Distributed Proofread", r"^\*\*\*\s*(?:START|END) OF"]
ap = argparse.ArgumentParser()
ap.add_argument("corpus")
ap.add_argument("--tokenizer", default=None)
a = ap.parse_args()
root = Path(a.corpus)
man = json.loads((root / "manifest.json").read_text())
alpha = json.loads((root / "corpus_alphabet.json").read_text())
records = []
for w in man["works"]:
for line in (root / w["path"]).read_text(encoding="utf-8").splitlines():
records.append(json.loads(line))
text = "\n\n".join(r["text"] for r in records)
fails = []
def check(name, ok, detail=""):
print(f" [{'PASS' if ok else 'FAIL'}] {name}{(' -- ' + detail) if detail else ''}")
if not ok:
fails.append(name)
print(f"== {len(records)} chapters, {sum(r['words'] for r in records):,} words, {len(text):,} chars\n")
# 1. boilerplate
hits = {p: len(re.findall(p, text, re.I | re.M)) for p in BOILER}
bad = {p: n for p, n in hits.items() if n}
check("zero Gutenberg boilerplate", not bad, f"found {bad}" if bad else "7 patterns, 0 hits")
# 2. structure
per_work = collections.Counter(r["work"] for r in records)
seq_ok = all(
[r["chapter"] for r in records if r["work"] == w] == list(range(1, per_work[w] + 1))
for w in per_work)
check("chapters number 1..N with no gaps", seq_ok, ", ".join(f"{w}:{n}" for w, n in per_work.items()))
check("no empty chapters", all(r["words"] > 100 for r in records),
f"min {min(r['words'] for r in records)} words")
# 3. typography consistency AFTER normalisation -- the reason normalisation exists
counts = collections.Counter(text)
straight = counts['"'] + counts["'"]
dbl_hyphen = len(re.findall(r"(?<!-)--(?!-)", text))
check("no straight quotes survive", straight == 0, f'" {counts[chr(34)]}, \' {counts[chr(39)]}')
check("no `--` survives", dbl_hyphen == 0, f"{dbl_hyphen} occurrences")
#: An open/close COUNT mismatch is not an error here and asserting equality was
#: a bad gate. Nineteenth-century convention runs a speech across paragraphs by
#: opening each one and closing only the last, so every work carries a surplus of
#: opens -- measured +46 / +49 / +51 on the three works whose quotes were never
#: touched. The real error signature is a paragraph that BEGINS with a closing
#: quote, which convention never produces and a bad conversion does.
paras = [p.strip() for p in text.split("\n\n") if p.strip()]
lead_close = [p[:60] for p in paras if p.lstrip()[:1] == chr(0x201d)]
check("no paragraph opens with a closing quote", not lead_close,
f"{len(lead_close)} of {len(paras):,} paragraphs" + (f" e.g. {lead_close[0]!r}" if lead_close else ""))
surplus = counts[chr(0x201c)] - counts[chr(0x201d)]
print(f" open-quote surplus {surplus:+} of {counts[chr(0x201c)]:,} "
f"(multi-paragraph speech; expected, not a failure)")
# 4. alphabet is the real inventory
observed = {c for c in text if c.isalpha()}
check("alphabet matches the text exactly", observed == set(alpha["letters"]),
f"declared {len(alpha['letters'])}, observed {len(observed)}, "
f"diff {sorted(observed ^ set(alpha['letters']))}")
# 5. no control / exotic codepoints
weird = {c for c in text if unicodedata.category(c) in ("Cc", "Cf", "Co", "Cs") and c != "\n"}
check("no control or private-use codepoints", not weird, repr(sorted(weird)))
# 6. tokenizer stability -- F02's byte-fallback lesson, on the real carrier
if a.tokenizer:
from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained(a.tokenizer)
sample = text[:400000]
ids = tok.encode(sample, add_special_tokens=False)
back = tok.decode(ids)
check("tokenizer round-trip is lossless", back == sample,
f"{len(ids):,} tokens from {len(sample):,} chars")
pieces = tok.convert_ids_to_tokens(ids)
fallback = [p for p in pieces if "�" in p]
check("no byte-fallback pieces", not fallback,
f"{len(fallback)} of {len(pieces):,} pieces")
total = len(tok.encode(text, add_special_tokens=False))
print(f"\n full corpus = {total:,} tokens ({total/sum(r['words'] for r in records):.2f} tok/word)")
print(f"\n== {'ALL CHECKS PASSED' if not fails else 'FAILED: ' + ', '.join(fails)}")
sys.exit(1 if fails else 0)