diff --git a/docs/pfi/author-voice-lora-regime.md b/docs/pfi/author-voice-lora-regime.md index f1243d5..6aaa9d2 100644 --- a/docs/pfi/author-voice-lora-regime.md +++ b/docs/pfi/author-voice-lora-regime.md @@ -174,9 +174,7 @@ first real corpus. ## 5. Prep remaining, in order 1. ~~Carrier family decision~~ — **settled: dense `Qwen3`** (§6a). -2. **Corpus D1** — Gutenberg Brontë (Jane Eyre, Villette, Shirley, The - Professor), boilerplate stripped, chapter-segmented, typography normalised, - character inventory recorded. Public domain, clean under any disposition. +2. ~~Corpus D1~~ — **BUILT AND GREEN 2026-09-10** (§8). 3. **Re-point the R49 deterministic machinery at Brontë.** The entity detector (corpus-level capitalised-vs-lowercase ratio), identity linking, gender resolution and the 23,398-name dictionary were all built and hardened against @@ -310,3 +308,89 @@ upstream cannot hand back, so `configs/restic/ana-ml2/profiles.yaml` now carries a single documented carve-out, `/tank/erp-tune/run-*/adapter`, verified by `resticprofile --dry-run` to expand to exactly those eight paths and nothing else. The nightly 01:00 run picks them up. + +--- + +## 8. D1 is built — the corpus, and three things it taught + +`scripts/r49-corpus/build_corpus.py --build` and `verify_corpus.py`, corpus staged +at `gx10:~/r49-corpus/`. **All 11 acceptance checks pass**, including the two +tokenizer legs run against the pilot carrier itself. + +| | | +|---|---| +| works | Jane Eyre (1260) · Villette (9182) · Shirley (30486) · The Professor (1028) | +| size | **680,291 words · 142 chapters · 950,974 tokens** under the Qwen3 tokenizer (1.40 tok/word) | +| alphabet | 75 letters, **23 of them non-ASCII** | +| tokenizer | round-trip lossless, **0 byte-fallback pieces** of 98,860 | + +Catalogue ids were verified against gutenberg.org's own search, not recalled. +Charlotte only — the Bell poems are co-authored and the Gaskell biography is a +different hand, so neither belongs in a single-voice corpus. + +**Real denominator, so the projections tighten.** At 6 rename copies × 3 epochs = +17.1M tokens: **0.6B → 1.98 h**, 1.7B → 3.36 h, 4B → 6.63 h per voice. The pilot +is a two-hour run. + +### ⚠ The alphabet finding inverts the Yarros result, and brokkr's re-point depends on it + + ÆÉÊËÔàâäæçèéêëîïôöùûüŒœ + +F02 measured the Yarros corpus at **4 non-ASCII characters in 1.8M letters** +(0.0002%) and derived an ASCII-fold rule for the name pool. **Charlotte Brontë is +the opposite case**: she writes French constantly — Villette is set in a +French-speaking city, Jane Eyre has Adèle, The Professor is set in Brussels — and +the corpus carries `é` 432 times, `è` 237, `à` 93, `ê` 79, `ô` 48, plus the `œ` +and `æ` ligatures. + +Under F02's own rule (*the pool's character inventory must be a subset of the +corpus's*) the Brontë pool **may keep French accents** and **must still exclude** +the Czech/Latvian/Slovak/Hungarian marks that never appear here. So the fold is +not global — it is derived per work, which is exactly what the rule said, and +this is the first corpus where the derivation changes the answer. + +### Typography was inconsistent, and it was the transcriber, not the author + +| work | quotes | dashes | +|---|---|---| +| Jane Eyre | curly | em-dash 2,058 | +| Villette | curly | em-dash 2,272 | +| **Shirley** | **straight** 9,115 | **`--` 2,228, zero em-dashes** | +| The Professor | curly | `--` 964, zero em-dashes | + +Left alone the adapter would learn that this author "sometimes" writes each form +— a false habit on the exact axis being trained. Normalised **toward what the text +means**: `--` is a transcription of an em-dash, so it becomes one; straight quotes +are paired into curly per paragraph. + +### Two gate defects, both found by running the gate + +1. **`Produced by` matched Brontë's own prose** — *"a chilling effect produced by + his steady announcement"*, three more like it. A hard rule on a phrase with a + common innocent sense, manufacturing failures: the same shape as the drift + detector that fired on the adjective "minor". Fixed by anchoring the + boilerplate patterns to line start, where Gutenberg credits actually live. +2. **Asserting open/close quote counts must be equal was wrong.** Nineteenth-century + convention runs a speech across paragraphs by opening each and closing only the + last, so a surplus of opens is correct — measured **+46 / +49 / +51** on the + three works whose quotes were never touched. Replaced with the real error + signature: *no paragraph may begin with a closing quote*, which convention never + produces and a bad conversion does. 0 of 14,230 paragraphs. + +A third, mine: **the manifest baked absolute build-machine paths**, so the corpus +was unreadable the moment it moved to gx10. Paths are relative to the corpus root +now. It failed loudly rather than silently reading nothing, which is why it was +cheap. + +### Next + +D2/D3 — re-point F02's entity detector, identity linking and gender resolution off +the Yarros sample onto Brontë, with the alphabet above constraining the pool. Then +D4 annotation, then the pilot. + +**Pilot ruling (operator, 2026-09-10): trial on `Qwen3-0.6B-Base` first and only +move up if it produces something useful.** So the sweep is not three arms up front +— it is one ~2 h run, judged, and then a decision. That is the cheap ordering and +it front-loads the kill signal: if voice does not transfer at 0.6B the question +becomes *how far up* rather than *whether at all*, and if it does transfer the +larger arms are a refinement rather than a gamble. diff --git a/persistent-memory.md b/persistent-memory.md index 771ba7e..247d384 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -1,6 +1,6 @@ # Persistent memory — eshpfi-management -_Last updated: 2026-09-10 06:30 PT (**Pfish-6** = run-6 NVFP4 is the standing seat, ana-ml2 :8021 ONLY; run 7 PURGED ~139 GiB; pfi-gx10 is an experimental/TRAINING box and carries no serving seat; all five ERP adapters now MIRRORED to ana-ml2 and inside restic; **BabyBronte / R49 author-voice regime is in PREP on gx10, carrier SETTLED = dense Qwen3**; checkpoints AND superseded merges PURGED both boxes, **~573 GB total**; only merged-run06 + the v6 quant survive)_ +_Last updated: 2026-09-10 07:05 PT (**Pfish-6** = run-6 NVFP4 is the standing seat, ana-ml2 :8021 ONLY; run 7 PURGED ~139 GiB; pfi-gx10 is an experimental/TRAINING box and carries no serving seat; all five ERP adapters now MIRRORED to ana-ml2 and inside restic; **BabyBronte / R49 author-voice regime is in PREP on gx10, carrier SETTLED = dense Qwen3**; checkpoints AND superseded merges PURGED both boxes, **~573 GB total**; only merged-run06 + the v6 quant survive)_ > **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its > `Written:` stamp is under an hour old, read it (it carries the in-flight @@ -191,6 +191,31 @@ preserved verbatim in `archival-memory.md` § Superseded in-flight snapshots._ (`finish_reason: stop`, correct text), container `Up 4 hours (healthy)`. ⚠ `gx10:~/erp-tune/relaunch-trial-seat.sh` names a now-deleted model; NOT removed (its flags carry the FlashInfer JIT/PATH trap + gpu-clear/never-pkill notes) but banner-marked RETIRED. +- **✅ R49 D1 CORPUS BUILT AND GREEN 2026-09-10.** `gx10:~/r49-corpus/`, instruments at + `scripts/r49-corpus/{build_corpus,verify_corpus}.py`. Charlotte only (ids verified against + gutenberg.org's own search): Jane Eyre 1260, Villette 9182, Shirley 30486, The Professor 1028 — + **680,291 words · 142 chapters · 950,974 Qwen3 tokens** (1.40 tok/word). All 11 acceptance checks + pass incl. lossless round-trip and **0 byte-fallback** on the pilot carrier's tokenizer. + Real projection at 6 copies × 3 epochs = 17.1M tokens: **0.6B 1.98 h**, 1.7B 3.36 h, 4B 6.63 h. + ⚠⚠ **THE ALPHABET INVERTS THE YARROS RESULT — tell anyone re-pointing the name pool.** Brontë's + inventory is 75 letters, **23 non-ASCII**: `ÆÉÊËÔàâäæçèéêëîïôöùûüŒœ` (é 432, è 237, à 93, ê 79, + ô 48, + œ/æ). She writes French constantly — Villette, Adèle, Brussels. F02 measured Yarros at + 0.0002% non-ASCII and derived an ASCII-fold; under F02's OWN subset rule the **Brontë pool may keep + FRENCH accents and must still exclude Czech/Latvian/Slovak/Hungarian marks.** The fold is per-work, + and this is the first corpus where deriving it changes the answer. + ⚠ **Typography was inconsistent and it was the TRANSCRIBER, not the author**: Shirley = straight + quotes + `--` + ZERO em-dashes; Jane Eyre/Villette = curly + em-dash; The Professor = curly + `--`. + Normalised toward meaning (`--` → em dash, straight → curly paired per paragraph). + ⚠ **Three defects, all found by running things rather than reasoning:** (a) `Produced by` matched + Brontë's OWN PROSE 4× ("a chilling effect produced by his steady announcement") — the adjective- + `minor` shape again, fixed by anchoring boilerplate patterns to line start; (b) asserting + open/close quote counts must be EQUAL is wrong — 19th-c multi-paragraph speech legitimately runs a + surplus of opens (+46/+49/+51 on untouched works), so the real signature is *no paragraph begins + with a closing quote* (0 of 14,230); (c) The Professor's TOC puts TWO chapter names per line, so a + bare regex returns 38 headings for a 25-chapter novel AND a min-gap filter still leaks the tail — + the rule that works is that the BODY's "CHAPTER I" is the LAST one in the file. +- **⭐ PILOT RULING (operator, 2026-09-10): trial on `Qwen3-0.6B-Base` FIRST, move up only if useful.** + Not a three-arm sweep up front — one ~2 h run, judged, then a decision. - **🖋 BabyBronte / R49 author-voice LoRA regime — IN PREP on pfi-gx10, nothing training.** Plan + every measured number: [`docs/pfi/author-voice-lora-regime.md`](docs/pfi/author-voice-lora-regime.md). Research target is **brokkr-smithy R49** (`research/R49-author-voice-adapters/`) — brokkr owns diff --git a/scripts/r49-corpus/build_corpus.py b/scripts/r49-corpus/build_corpus.py new file mode 100644 index 0000000..af9d2eb --- /dev/null +++ b/scripts/r49-corpus/build_corpus.py @@ -0,0 +1,269 @@ +"""R49 Stage D1 — acquire and clean a public-domain author corpus. + +Charlotte Brontë's four novels from Project Gutenberg, stripped of boilerplate, +chapter-segmented, typography-normalised, with the corpus's own character +inventory derived from the result. + +The alphabet is not cosmetic. R49 F02's rule is that the rename pool's character +inventory must be a SUBSET of the source corpus's -- substituting a 26%-diacritic +name pool into prose the author wrote in plain ASCII teaches the adapter a false +orthographic habit, landing directly on the axis being trained. So the corpus +derives the constraint and the pool obeys it, per work. + +Two stages on purpose. `--survey` reports what is actually in the text before any +normalisation is chosen; normalisation decided from a guess rather than from the +survey is how a cleanup silently deletes something. Run the survey, read it, then +run the build. + + python build_corpus.py --survey # measure, change nothing + python build_corpus.py --build --out DIR # emit the cleaned corpus +""" +from __future__ import annotations +import argparse, collections, json, re, sys, unicodedata, urllib.request +from pathlib import Path + +# Catalogue ids verified against gutenberg.org's own search 2026-09-10, not +# recalled. Charlotte only -- the Bell poems are co-authored and the Gaskell +# biography is a different hand, so neither belongs in a single-voice corpus. +WORKS = [ + {"id": 1260, "slug": "jane-eyre", "title": "Jane Eyre: An Autobiography"}, + {"id": 9182, "slug": "villette", "title": "Villette"}, + {"id": 30486, "slug": "shirley", "title": "Shirley"}, + {"id": 1028, "slug": "the-professor", "title": "The Professor"}, +] +URLS = ["https://www.gutenberg.org/cache/epub/{id}/pg{id}.txt", + "https://www.gutenberg.org/files/{id}/{id}-0.txt", + "https://www.gutenberg.org/files/{id}/{id}.txt"] + +START = re.compile(r"^\*\*\*\s*START OF (?:THE|THIS) PROJECT GUTENBERG EBOOK.*?\*\*\*\s*$", re.M | re.I) +END = re.compile(r"^\*\*\*\s*END OF (?:THE|THIS) PROJECT GUTENBERG EBOOK.*?\*\*\*\s*$", re.M | re.I) +CHAPTER = re.compile(r"^\s*(CHAPTER\s+[IVXLCDM]+|CHAPTER\s+\d+)\.?\s*(.*)$", re.M) + + +def fetch(work, cache: Path) -> str: + cache.mkdir(parents=True, exist_ok=True) + raw = cache / f"{work['slug']}.raw.txt" + if raw.exists(): + return raw.read_text(encoding="utf-8") + for tmpl in URLS: + url = tmpl.format(id=work["id"]) + try: + with urllib.request.urlopen(url, timeout=60) as r: + if r.status != 200: + continue + text = r.read().decode("utf-8-sig") + raw.write_text(text, encoding="utf-8") + print(f" fetched {work['slug']:<14} {url} {len(text):,} bytes") + return text + except Exception as e: + print(f" .. {url} -> {type(e).__name__}") + raise SystemExit(f"REFUSING: could not fetch {work['slug']} (id {work['id']})") + + +def strip_boilerplate(text: str, slug: str) -> str: + """Keep only what lies between Gutenberg's own START/END markers. + + Anchoring on the markers rather than on a line count is what makes this + safe across editions -- the front matter length differs per work. + """ + m1, m2 = START.search(text), END.search(text) + if not m1 or not m2: + raise SystemExit(f"REFUSING: {slug} has no START/END markers; refusing to guess where the text begins") + body = text[m1.end():m2.start()] + # A transcriber credit block sometimes sits just inside the START marker. + body = re.sub(r"\A\s*(?:Produced by|E-text prepared by|Transcribed from).*?\n\s*\n", "", body, flags=re.S | re.I) + return body.strip("\n") + + +ROMAN = {"I":1,"V":5,"X":10,"L":50,"C":100,"D":500,"M":1000} + + +def roman_to_int(r: str) -> int: + total, prev = 0, 0 + for ch in reversed(r.upper()): + v = ROMAN.get(ch, 0) + total = total - v if v < prev else total + v + prev = max(prev, v) + return total + + +def find_chapters(body: str) -> list[tuple[int, str, int]]: + """Body chapter headings only, with any table of contents discarded. + + Measured 2026-09-10: The Professor ships a TOC that puts TWO chapter names + on one line, so a bare regex returns 38 headings for a 25-chapter novel and + a naive minimum-gap filter still leaks the TOC's tail. The rule that works + is structural rather than cosmetic -- the body's "CHAPTER I" is the LAST one + in the file, because a TOC always precedes the text it indexes. From there, + keep only headings that continue the sequence and are separated by prose. + """ + hits = [] + for m in CHAPTER.finditer(body): + num = m.group(1).split()[-1].rstrip(".") + n = int(num) if num.isdigit() else roman_to_int(num) + hits.append((m.start(), m.group(1).strip(), n)) + if not hits: + return [] + ones = [i for i, h in enumerate(hits) if h[2] == 1] + start = ones[-1] if ones else 0 + kept, expect, last_pos = [], 1, -10**9 + for pos, label, n in hits[start:]: + if n == expect and pos - last_pos > 500: + kept.append((pos, label, n)) + expect, last_pos = expect + 1, pos + return kept + + +#: Normalisation is decided from the survey, not from a guess. Measured across +#: the four works: Jane Eyre and Villette use curly quotes and em-dashes; +#: SHIRLEY uses straight quotes and `--` with zero em-dashes; The Professor +#: mixes curly quotes with `--`. That split is a transcriber artefact, not +#: Charlotte Bronte's punctuation, and leaving it would teach the adapter that +#: this author "sometimes" writes each form -- a false habit on the exact axis +#: being trained. Normalise toward what the text MEANS: `--` is a transcription +#: of an em-dash, so it becomes one. +def normalise_quotes(text: str) -> str: + """Straight quotes -> curly, paired by alternation within each paragraph.""" + out = [] + for para in text.split("\n\n"): + buf, open_d = [], True + for ch in para: + if ch == '"': + buf.append("\u201c" if open_d else "\u201d") + open_d = not open_d + else: + buf.append(ch) + para = "".join(buf) + # single quotes: apostrophe if flanked by letters, else a quote mark + para = re.sub(r"(?<=[A-Za-z])'(?=[A-Za-z])", "\u2019", para) + buf, open_s = [], True + for ch in para: + if ch == "'": + buf.append("\u2018" if open_s else "\u2019") + open_s = not open_s + else: + buf.append(ch) + out.append("".join(buf)) + return "\n\n".join(out) + + +def clean(text: str) -> str: + text = text.replace("\u00a0", " ") + text = re.sub(r"(? None: + print("\n== character inventory, BEFORE any normalisation") + allchars = collections.Counter() + for slug, b in bodies.items(): + allchars.update(b) + letters = {c for c in allchars if c.isalpha()} + ascii_letters = {c for c in letters if ord(c) < 128} + non_ascii = sorted(c for c in allchars if ord(c) > 127) + print(f" distinct characters : {len(allchars)}") + print(f" distinct letters : {len(letters)} (ascii {len(ascii_letters)}, non-ascii {len(letters - ascii_letters)})") + print(f" distinct non-ascii chars : {len(non_ascii)}") + print(" non-ascii, by frequency:") + for c in sorted(non_ascii, key=lambda c: -allchars[c]): + name = unicodedata.name(c, "?") + print(f" U+{ord(c):04X} {c!r:<8} {allchars[c]:>7} {name}") + print("\n== structure") + for slug, b in bodies.items(): + heads = find_chapters(b) + words = len(b.split()) + print(f" {slug:<14} {words:>8,} words {len(heads):>3} chapters last: {heads[-1][1] if heads else '-'}") + print(f" {'TOTAL':<14} {sum(len(b.split()) for b in bodies.values()):>8,} words") + + +def main() -> int: + ap = argparse.ArgumentParser() + ap.add_argument("--survey", action="store_true") + ap.add_argument("--build", action="store_true") + ap.add_argument("--out", default="corpus") + ap.add_argument("--cache", default="raw") + a = ap.parse_args() + if not (a.survey or a.build): + ap.error("pick --survey or --build") + + cache = Path(a.cache) + print("== fetch") + bodies = {} + for w in WORKS: + bodies[w["slug"]] = strip_boilerplate(fetch(w, cache), w["slug"]) + assert "PROJECT GUTENBERG" not in bodies[w["slug"]][:2000].upper(), f"{w['slug']}: boilerplate survived" + + if a.survey: + survey(bodies) + return 0 + + out = Path(a.out) + (out / "works").mkdir(parents=True, exist_ok=True) + manifest, alphabet = [], set() + for w in WORKS: + slug = w["slug"] + body = clean(bodies[slug]) + chaps = find_chapters(body) + if not chaps: + raise SystemExit(f"REFUSING: no chapters found in {slug}") + # Self-consistency: the count must equal the last heading's numeral, or + # the segmentation has silently over- or under-matched. + if len(chaps) != chaps[-1][2]: + raise SystemExit( + f"REFUSING: {slug} segmented into {len(chaps)} chapters but the last " + f"heading is {chaps[-1][1]} (= {chaps[-1][2]}). Segmentation is wrong.") + records = [] + for i, (pos, label, n) in enumerate(chaps): + end = chaps[i + 1][0] if i + 1 < len(chaps) else len(body) + text = body[pos:end].strip("\n") + records.append({"work": slug, "chapter": n, "heading": label, + "words": len(text.split()), "text": text}) + path = out / "works" / f"{slug}.jsonl" + with path.open("w", encoding="utf-8") as fh: + for r in records: + fh.write(json.dumps(r, ensure_ascii=False) + "\n") + alphabet |= {c for c in body if c.isalpha()} + # Relative to the corpus root, never absolute: the corpus is built on one + # box and trained on another, and an absolute build path makes the + # manifest unreadable the moment it moves. + manifest.append({"slug": slug, "gutenberg_id": w["id"], "title": w["title"], + "chapters": len(records), + "words": sum(r["words"] for r in records), + "chars": len(body), "path": f"works/{slug}.jsonl"}) + print(f" wrote {slug:<14} {len(records):>3} chapters {sum(r['words'] for r in records):>8,} words") + + alpha = sorted(alphabet) + (out / "corpus_alphabet.json").write_text(json.dumps({ + "derived_from": "Charlotte Bronte, 4 novels, Project Gutenberg", + "derived_at": "2026-09-10", + "note": ("R49 F02 rule: a rename pool's character inventory must be a SUBSET of " + "this. Bronte writes French constantly (Villette, Adele, Brussels), so " + "unlike the Yarros corpus this alphabet legitimately carries accents -- " + "but only FRENCH ones. Czech/Latvian/Slovak/Hungarian marks never appear " + "and must not enter the pool."), + "count": len(alpha), "letters": alpha, + "non_ascii": [c for c in alpha if ord(c) > 127], + }, ensure_ascii=False, indent=2), encoding="utf-8") + (out / "manifest.json").write_text(json.dumps({ + "corpus": "bronte-charlotte-v1", "built_at": "2026-09-10", + "source": "Project Gutenberg (public domain)", + "normalisation": ("no-break space -> space; `--` -> em dash; straight quotes -> " + "curly, paired per paragraph. Decided from the survey: Shirley " + "was transcribed with straight quotes and zero em-dashes while " + "Jane Eyre and Villette use curly and em-dash, a transcriber " + "split rather than the author's punctuation."), + "works": manifest, + "total_words": sum(m["words"] for m in manifest), + "total_chapters": sum(m["chapters"] for m in manifest), + }, ensure_ascii=False, indent=2), encoding="utf-8") + print(f"\n alphabet: {len(alpha)} letters ({len([c for c in alpha if ord(c)>127])} non-ascii)") + print(f" TOTAL : {sum(m['words'] for m in manifest):,} words in " + f"{sum(m['chapters'] for m in manifest)} chapters -> {out}") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/scripts/r49-corpus/corpus_alphabet.json b/scripts/r49-corpus/corpus_alphabet.json new file mode 100644 index 0000000..a0eefa7 --- /dev/null +++ b/scripts/r49-corpus/corpus_alphabet.json @@ -0,0 +1,108 @@ +{ + "derived_from": "Charlotte Bronte, 4 novels, Project Gutenberg", + "derived_at": "2026-09-10", + "note": "R49 F02 rule: a rename pool's character inventory must be a SUBSET of this. Bronte writes French constantly (Villette, Adele, Brussels), so unlike the Yarros corpus this alphabet legitimately carries accents -- but only FRENCH ones. Czech/Latvian/Slovak/Hungarian marks never appear and must not enter the pool.", + "count": 75, + "letters": [ + "A", + "B", + "C", + "D", + "E", + "F", + "G", + "H", + "I", + "J", + "K", + "L", + "M", + "N", + "O", + "P", + "Q", + "R", + "S", + "T", + "U", + "V", + "W", + "X", + "Y", + "Z", + "a", + "b", + "c", + "d", + "e", + "f", + "g", + "h", + "i", + "j", + "k", + "l", + "m", + "n", + "o", + "p", + "q", + "r", + "s", + "t", + "u", + "v", + "w", + "x", + "y", + "z", + "Æ", + "É", + "Ê", + "Ë", + "Ô", + "à", + "â", + "ä", + "æ", + "ç", + "è", + "é", + "ê", + "ë", + "î", + "ï", + "ô", + "ö", + "ù", + "û", + "ü", + "Œ", + "œ" + ], + "non_ascii": [ + "Æ", + "É", + "Ê", + "Ë", + "Ô", + "à", + "â", + "ä", + "æ", + "ç", + "è", + "é", + "ê", + "ë", + "î", + "ï", + "ô", + "ö", + "ù", + "û", + "ü", + "Œ", + "œ" + ] +} \ No newline at end of file diff --git a/scripts/r49-corpus/manifest.json b/scripts/r49-corpus/manifest.json new file mode 100644 index 0000000..10b5b54 --- /dev/null +++ b/scripts/r49-corpus/manifest.json @@ -0,0 +1,46 @@ +{ + "corpus": "bronte-charlotte-v1", + "built_at": "2026-09-10", + "source": "Project Gutenberg (public domain)", + "normalisation": "no-break space -> space; `--` -> em dash; straight quotes -> curly, paired per paragraph. Decided from the survey: Shirley was transcribed with straight quotes and zero em-dashes while Jane Eyre and Villette use curly and em-dash, a transcriber split rather than the author's punctuation.", + "works": [ + { + "slug": "jane-eyre", + "gutenberg_id": 1260, + "title": "Jane Eyre: An Autobiography", + "chapters": 38, + "words": 184452, + "chars": 1022193, + "path": "works/jane-eyre.jsonl" + }, + { + "slug": "villette", + "gutenberg_id": 9182, + "title": "Villette", + "chapters": 42, + "words": 192411, + "chars": 1092741, + "path": "works/villette.jsonl" + }, + { + "slug": "shirley", + "gutenberg_id": 30486, + "title": "Shirley", + "chapters": 37, + "words": 216016, + "chars": 1226278, + "path": "works/shirley.jsonl" + }, + { + "slug": "the-professor", + "gutenberg_id": 1028, + "title": "The Professor", + "chapters": 25, + "words": 87412, + "chars": 500054, + "path": "works/the-professor.jsonl" + } + ], + "total_words": 680291, + "total_chapters": 142 +} \ No newline at end of file diff --git a/scripts/r49-corpus/verify_corpus.py b/scripts/r49-corpus/verify_corpus.py new file mode 100644 index 0000000..ee86c0c --- /dev/null +++ b/scripts/r49-corpus/verify_corpus.py @@ -0,0 +1,108 @@ +"""R49 D1 acceptance gate for a built corpus. + +The design doc's D1 acceptance is "clean UTF-8, chapter-segmented, zero +boilerplate lines, stable tokenization". Each is checked here as something that +can actually go RED -- a gate that cannot fail is the third failure mode this +target has already recorded, and it is not repeated here. + + python verify_corpus.py [--tokenizer PATH] +""" +from __future__ import annotations +import argparse, collections, json, re, sys, unicodedata +from pathlib import Path + +#: ⚠ Anchored to line start, and that is not cosmetic. The first draft matched +#: `Produced by` anywhere and went RED on four hits that were all Charlotte +#: Bronte's own prose -- "a chilling effect produced by his steady announcement", +#: "how such a result was produced by such means". A hard rule on a phrase with a +#: common non-boilerplate sense manufactures failures; same shape as the drift +#: detector that fired on the adjective "minor" and stopped work three times. +#: Gutenberg credits always begin a line, so require that. +BOILER = [r"^.*PROJECT GUTENBERG.*$", r"^.*gutenberg\.org.*$", r"^\s*Produced by\b", + r"^\s*E-text prepared by\b", r"^\s*Transcribed from\b", + r"^\s*Distributed Proofread", r"^\*\*\*\s*(?:START|END) OF"] + +ap = argparse.ArgumentParser() +ap.add_argument("corpus") +ap.add_argument("--tokenizer", default=None) +a = ap.parse_args() +root = Path(a.corpus) +man = json.loads((root / "manifest.json").read_text()) +alpha = json.loads((root / "corpus_alphabet.json").read_text()) + +records = [] +for w in man["works"]: + for line in (root / w["path"]).read_text(encoding="utf-8").splitlines(): + records.append(json.loads(line)) +text = "\n\n".join(r["text"] for r in records) +fails = [] + +def check(name, ok, detail=""): + print(f" [{'PASS' if ok else 'FAIL'}] {name}{(' -- ' + detail) if detail else ''}") + if not ok: + fails.append(name) + +print(f"== {len(records)} chapters, {sum(r['words'] for r in records):,} words, {len(text):,} chars\n") + +# 1. boilerplate +hits = {p: len(re.findall(p, text, re.I | re.M)) for p in BOILER} +bad = {p: n for p, n in hits.items() if n} +check("zero Gutenberg boilerplate", not bad, f"found {bad}" if bad else "7 patterns, 0 hits") + +# 2. structure +per_work = collections.Counter(r["work"] for r in records) +seq_ok = all( + [r["chapter"] for r in records if r["work"] == w] == list(range(1, per_work[w] + 1)) + for w in per_work) +check("chapters number 1..N with no gaps", seq_ok, ", ".join(f"{w}:{n}" for w, n in per_work.items())) +check("no empty chapters", all(r["words"] > 100 for r in records), + f"min {min(r['words'] for r in records)} words") + +# 3. typography consistency AFTER normalisation -- the reason normalisation exists +counts = collections.Counter(text) +straight = counts['"'] + counts["'"] +dbl_hyphen = len(re.findall(r"(?