feat(r49): D1 corpus built and green — Charlotte Brontë, 680k words, 951k tokens
scripts/r49-corpus/{build_corpus,verify_corpus}.py; corpus staged at
gx10:~/r49-corpus/. Catalogue ids verified against gutenberg.org's own search
rather than recalled. Charlotte only -- the Bell poems are co-authored and the
Gaskell biography is a different hand, so neither belongs in a single-voice corpus.
Jane Eyre 1260 · Villette 9182 · Shirley 30486 · The Professor 1028
680,291 words · 142 chapters · 950,974 Qwen3 tokens (1.40 tok/word)
alphabet 75 letters, 23 non-ASCII · round-trip lossless · 0 byte-fallback
All 11 acceptance checks pass, including both tokenizer legs run against the pilot
carrier itself. With a real denominator the projections tighten: at 6 rename copies
x 3 epochs = 17.1M tokens, the 0.6B pilot is 1.98 h.
THE ALPHABET INVERTS THE YARROS RESULT. Brontë writes French constantly -- Villette
is set in a French-speaking city, Jane Eyre has Adèle, The Professor is set in
Brussels -- so the corpus carries é 432, è 237, à 93, ê 79, ô 48 plus œ and æ. F02
measured Yarros at 0.0002% non-ASCII and derived an ASCII-fold for the name pool.
Under F02's own subset rule the Brontë pool may keep FRENCH accents and must still
exclude the Czech/Latvian/Slovak/Hungarian marks that never appear here. The fold is
per-work, and this is the first corpus where deriving it changes the answer.
Typography was inconsistent across works and it was the transcriber, not the author:
Shirley uses straight quotes and `--` with zero em-dashes while Jane Eyre and
Villette use curly and em-dash. Normalised toward what the text means.
Three defects, each found by running something rather than reasoning about it:
`Produced by` matched Brontë's own prose four times, which is the adjective-"minor"
shape again and is fixed by anchoring boilerplate patterns to line start; asserting
open/close quote counts must be equal is wrong, because 19th-century multi-paragraph
speech legitimately runs a surplus of opens, so the real error signature is that no
paragraph may begin with a closing quote; and The Professor's table of contents puts
two chapter names per line, so a bare regex returns 38 headings for a 25-chapter
novel and a minimum-gap filter still leaks its tail -- the rule that works is that
the body's "CHAPTER I" is the last one in the file.
Records the operator's pilot ruling: trial on Qwen3-0.6B-Base first, move up only if
it produces something useful.
This commit is contained in:
@@ -174,9 +174,7 @@ first real corpus.
|
||||
## 5. Prep remaining, in order
|
||||
|
||||
1. ~~Carrier family decision~~ — **settled: dense `Qwen3`** (§6a).
|
||||
2. **Corpus D1** — Gutenberg Brontë (Jane Eyre, Villette, Shirley, The
|
||||
Professor), boilerplate stripped, chapter-segmented, typography normalised,
|
||||
character inventory recorded. Public domain, clean under any disposition.
|
||||
2. ~~Corpus D1~~ — **BUILT AND GREEN 2026-09-10** (§8).
|
||||
3. **Re-point the R49 deterministic machinery at Brontë.** The entity detector
|
||||
(corpus-level capitalised-vs-lowercase ratio), identity linking, gender
|
||||
resolution and the 23,398-name dictionary were all built and hardened against
|
||||
@@ -310,3 +308,89 @@ upstream cannot hand back, so `configs/restic/ana-ml2/profiles.yaml` now carries
|
||||
a single documented carve-out, `/tank/erp-tune/run-*/adapter`, verified by
|
||||
`resticprofile --dry-run` to expand to exactly those eight paths and nothing
|
||||
else. The nightly 01:00 run picks them up.
|
||||
|
||||
---
|
||||
|
||||
## 8. D1 is built — the corpus, and three things it taught
|
||||
|
||||
`scripts/r49-corpus/build_corpus.py --build` and `verify_corpus.py`, corpus staged
|
||||
at `gx10:~/r49-corpus/`. **All 11 acceptance checks pass**, including the two
|
||||
tokenizer legs run against the pilot carrier itself.
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| works | Jane Eyre (1260) · Villette (9182) · Shirley (30486) · The Professor (1028) |
|
||||
| size | **680,291 words · 142 chapters · 950,974 tokens** under the Qwen3 tokenizer (1.40 tok/word) |
|
||||
| alphabet | 75 letters, **23 of them non-ASCII** |
|
||||
| tokenizer | round-trip lossless, **0 byte-fallback pieces** of 98,860 |
|
||||
|
||||
Catalogue ids were verified against gutenberg.org's own search, not recalled.
|
||||
Charlotte only — the Bell poems are co-authored and the Gaskell biography is a
|
||||
different hand, so neither belongs in a single-voice corpus.
|
||||
|
||||
**Real denominator, so the projections tighten.** At 6 rename copies × 3 epochs =
|
||||
17.1M tokens: **0.6B → 1.98 h**, 1.7B → 3.36 h, 4B → 6.63 h per voice. The pilot
|
||||
is a two-hour run.
|
||||
|
||||
### ⚠ The alphabet finding inverts the Yarros result, and brokkr's re-point depends on it
|
||||
|
||||
ÆÉÊËÔàâäæçèéêëîïôöùûüŒœ
|
||||
|
||||
F02 measured the Yarros corpus at **4 non-ASCII characters in 1.8M letters**
|
||||
(0.0002%) and derived an ASCII-fold rule for the name pool. **Charlotte Brontë is
|
||||
the opposite case**: she writes French constantly — Villette is set in a
|
||||
French-speaking city, Jane Eyre has Adèle, The Professor is set in Brussels — and
|
||||
the corpus carries `é` 432 times, `è` 237, `à` 93, `ê` 79, `ô` 48, plus the `œ`
|
||||
and `æ` ligatures.
|
||||
|
||||
Under F02's own rule (*the pool's character inventory must be a subset of the
|
||||
corpus's*) the Brontë pool **may keep French accents** and **must still exclude**
|
||||
the Czech/Latvian/Slovak/Hungarian marks that never appear here. So the fold is
|
||||
not global — it is derived per work, which is exactly what the rule said, and
|
||||
this is the first corpus where the derivation changes the answer.
|
||||
|
||||
### Typography was inconsistent, and it was the transcriber, not the author
|
||||
|
||||
| work | quotes | dashes |
|
||||
|---|---|---|
|
||||
| Jane Eyre | curly | em-dash 2,058 |
|
||||
| Villette | curly | em-dash 2,272 |
|
||||
| **Shirley** | **straight** 9,115 | **`--` 2,228, zero em-dashes** |
|
||||
| The Professor | curly | `--` 964, zero em-dashes |
|
||||
|
||||
Left alone the adapter would learn that this author "sometimes" writes each form
|
||||
— a false habit on the exact axis being trained. Normalised **toward what the text
|
||||
means**: `--` is a transcription of an em-dash, so it becomes one; straight quotes
|
||||
are paired into curly per paragraph.
|
||||
|
||||
### Two gate defects, both found by running the gate
|
||||
|
||||
1. **`Produced by` matched Brontë's own prose** — *"a chilling effect produced by
|
||||
his steady announcement"*, three more like it. A hard rule on a phrase with a
|
||||
common innocent sense, manufacturing failures: the same shape as the drift
|
||||
detector that fired on the adjective "minor". Fixed by anchoring the
|
||||
boilerplate patterns to line start, where Gutenberg credits actually live.
|
||||
2. **Asserting open/close quote counts must be equal was wrong.** Nineteenth-century
|
||||
convention runs a speech across paragraphs by opening each and closing only the
|
||||
last, so a surplus of opens is correct — measured **+46 / +49 / +51** on the
|
||||
three works whose quotes were never touched. Replaced with the real error
|
||||
signature: *no paragraph may begin with a closing quote*, which convention never
|
||||
produces and a bad conversion does. 0 of 14,230 paragraphs.
|
||||
|
||||
A third, mine: **the manifest baked absolute build-machine paths**, so the corpus
|
||||
was unreadable the moment it moved to gx10. Paths are relative to the corpus root
|
||||
now. It failed loudly rather than silently reading nothing, which is why it was
|
||||
cheap.
|
||||
|
||||
### Next
|
||||
|
||||
D2/D3 — re-point F02's entity detector, identity linking and gender resolution off
|
||||
the Yarros sample onto Brontë, with the alphabet above constraining the pool. Then
|
||||
D4 annotation, then the pilot.
|
||||
|
||||
**Pilot ruling (operator, 2026-09-10): trial on `Qwen3-0.6B-Base` first and only
|
||||
move up if it produces something useful.** So the sweep is not three arms up front
|
||||
— it is one ~2 h run, judged, and then a decision. That is the cheap ordering and
|
||||
it front-loads the kill signal: if voice does not transfer at 0.6B the question
|
||||
becomes *how far up* rather than *whether at all*, and if it does transfer the
|
||||
larger arms are a refinement rather than a gamble.
|
||||
|
||||
+26
-1
@@ -1,6 +1,6 @@
|
||||
# Persistent memory — eshpfi-management
|
||||
|
||||
_Last updated: 2026-09-10 06:30 PT (**Pfish-6** = run-6 NVFP4 is the standing seat, ana-ml2 :8021 ONLY; run 7 PURGED ~139 GiB; pfi-gx10 is an experimental/TRAINING box and carries no serving seat; all five ERP adapters now MIRRORED to ana-ml2 and inside restic; **BabyBronte / R49 author-voice regime is in PREP on gx10, carrier SETTLED = dense Qwen3**; checkpoints AND superseded merges PURGED both boxes, **~573 GB total**; only merged-run06 + the v6 quant survive)_
|
||||
_Last updated: 2026-09-10 07:05 PT (**Pfish-6** = run-6 NVFP4 is the standing seat, ana-ml2 :8021 ONLY; run 7 PURGED ~139 GiB; pfi-gx10 is an experimental/TRAINING box and carries no serving seat; all five ERP adapters now MIRRORED to ana-ml2 and inside restic; **BabyBronte / R49 author-voice regime is in PREP on gx10, carrier SETTLED = dense Qwen3**; checkpoints AND superseded merges PURGED both boxes, **~573 GB total**; only merged-run06 + the v6 quant survive)_
|
||||
|
||||
> **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its
|
||||
> `Written:` stamp is under an hour old, read it (it carries the in-flight
|
||||
@@ -191,6 +191,31 @@ preserved verbatim in `archival-memory.md` § Superseded in-flight snapshots._
|
||||
(`finish_reason: stop`, correct text), container `Up 4 hours (healthy)`.
|
||||
⚠ `gx10:~/erp-tune/relaunch-trial-seat.sh` names a now-deleted model; NOT removed (its flags carry
|
||||
the FlashInfer JIT/PATH trap + gpu-clear/never-pkill notes) but banner-marked RETIRED.
|
||||
- **✅ R49 D1 CORPUS BUILT AND GREEN 2026-09-10.** `gx10:~/r49-corpus/`, instruments at
|
||||
`scripts/r49-corpus/{build_corpus,verify_corpus}.py`. Charlotte only (ids verified against
|
||||
gutenberg.org's own search): Jane Eyre 1260, Villette 9182, Shirley 30486, The Professor 1028 —
|
||||
**680,291 words · 142 chapters · 950,974 Qwen3 tokens** (1.40 tok/word). All 11 acceptance checks
|
||||
pass incl. lossless round-trip and **0 byte-fallback** on the pilot carrier's tokenizer.
|
||||
Real projection at 6 copies × 3 epochs = 17.1M tokens: **0.6B 1.98 h**, 1.7B 3.36 h, 4B 6.63 h.
|
||||
⚠⚠ **THE ALPHABET INVERTS THE YARROS RESULT — tell anyone re-pointing the name pool.** Brontë's
|
||||
inventory is 75 letters, **23 non-ASCII**: `ÆÉÊËÔàâäæçèéêëîïôöùûüŒœ` (é 432, è 237, à 93, ê 79,
|
||||
ô 48, + œ/æ). She writes French constantly — Villette, Adèle, Brussels. F02 measured Yarros at
|
||||
0.0002% non-ASCII and derived an ASCII-fold; under F02's OWN subset rule the **Brontë pool may keep
|
||||
FRENCH accents and must still exclude Czech/Latvian/Slovak/Hungarian marks.** The fold is per-work,
|
||||
and this is the first corpus where deriving it changes the answer.
|
||||
⚠ **Typography was inconsistent and it was the TRANSCRIBER, not the author**: Shirley = straight
|
||||
quotes + `--` + ZERO em-dashes; Jane Eyre/Villette = curly + em-dash; The Professor = curly + `--`.
|
||||
Normalised toward meaning (`--` → em dash, straight → curly paired per paragraph).
|
||||
⚠ **Three defects, all found by running things rather than reasoning:** (a) `Produced by` matched
|
||||
Brontë's OWN PROSE 4× ("a chilling effect produced by his steady announcement") — the adjective-
|
||||
`minor` shape again, fixed by anchoring boilerplate patterns to line start; (b) asserting
|
||||
open/close quote counts must be EQUAL is wrong — 19th-c multi-paragraph speech legitimately runs a
|
||||
surplus of opens (+46/+49/+51 on untouched works), so the real signature is *no paragraph begins
|
||||
with a closing quote* (0 of 14,230); (c) The Professor's TOC puts TWO chapter names per line, so a
|
||||
bare regex returns 38 headings for a 25-chapter novel AND a min-gap filter still leaks the tail —
|
||||
the rule that works is that the BODY's "CHAPTER I" is the LAST one in the file.
|
||||
- **⭐ PILOT RULING (operator, 2026-09-10): trial on `Qwen3-0.6B-Base` FIRST, move up only if useful.**
|
||||
Not a three-arm sweep up front — one ~2 h run, judged, then a decision.
|
||||
- **🖋 BabyBronte / R49 author-voice LoRA regime — IN PREP on pfi-gx10, nothing training.** Plan +
|
||||
every measured number: [`docs/pfi/author-voice-lora-regime.md`](docs/pfi/author-voice-lora-regime.md).
|
||||
Research target is **brokkr-smithy R49** (`research/R49-author-voice-adapters/`) — brokkr owns
|
||||
|
||||
@@ -0,0 +1,269 @@
|
||||
"""R49 Stage D1 — acquire and clean a public-domain author corpus.
|
||||
|
||||
Charlotte Brontë's four novels from Project Gutenberg, stripped of boilerplate,
|
||||
chapter-segmented, typography-normalised, with the corpus's own character
|
||||
inventory derived from the result.
|
||||
|
||||
The alphabet is not cosmetic. R49 F02's rule is that the rename pool's character
|
||||
inventory must be a SUBSET of the source corpus's -- substituting a 26%-diacritic
|
||||
name pool into prose the author wrote in plain ASCII teaches the adapter a false
|
||||
orthographic habit, landing directly on the axis being trained. So the corpus
|
||||
derives the constraint and the pool obeys it, per work.
|
||||
|
||||
Two stages on purpose. `--survey` reports what is actually in the text before any
|
||||
normalisation is chosen; normalisation decided from a guess rather than from the
|
||||
survey is how a cleanup silently deletes something. Run the survey, read it, then
|
||||
run the build.
|
||||
|
||||
python build_corpus.py --survey # measure, change nothing
|
||||
python build_corpus.py --build --out DIR # emit the cleaned corpus
|
||||
"""
|
||||
from __future__ import annotations
|
||||
import argparse, collections, json, re, sys, unicodedata, urllib.request
|
||||
from pathlib import Path
|
||||
|
||||
# Catalogue ids verified against gutenberg.org's own search 2026-09-10, not
|
||||
# recalled. Charlotte only -- the Bell poems are co-authored and the Gaskell
|
||||
# biography is a different hand, so neither belongs in a single-voice corpus.
|
||||
WORKS = [
|
||||
{"id": 1260, "slug": "jane-eyre", "title": "Jane Eyre: An Autobiography"},
|
||||
{"id": 9182, "slug": "villette", "title": "Villette"},
|
||||
{"id": 30486, "slug": "shirley", "title": "Shirley"},
|
||||
{"id": 1028, "slug": "the-professor", "title": "The Professor"},
|
||||
]
|
||||
URLS = ["https://www.gutenberg.org/cache/epub/{id}/pg{id}.txt",
|
||||
"https://www.gutenberg.org/files/{id}/{id}-0.txt",
|
||||
"https://www.gutenberg.org/files/{id}/{id}.txt"]
|
||||
|
||||
START = re.compile(r"^\*\*\*\s*START OF (?:THE|THIS) PROJECT GUTENBERG EBOOK.*?\*\*\*\s*$", re.M | re.I)
|
||||
END = re.compile(r"^\*\*\*\s*END OF (?:THE|THIS) PROJECT GUTENBERG EBOOK.*?\*\*\*\s*$", re.M | re.I)
|
||||
CHAPTER = re.compile(r"^\s*(CHAPTER\s+[IVXLCDM]+|CHAPTER\s+\d+)\.?\s*(.*)$", re.M)
|
||||
|
||||
|
||||
def fetch(work, cache: Path) -> str:
|
||||
cache.mkdir(parents=True, exist_ok=True)
|
||||
raw = cache / f"{work['slug']}.raw.txt"
|
||||
if raw.exists():
|
||||
return raw.read_text(encoding="utf-8")
|
||||
for tmpl in URLS:
|
||||
url = tmpl.format(id=work["id"])
|
||||
try:
|
||||
with urllib.request.urlopen(url, timeout=60) as r:
|
||||
if r.status != 200:
|
||||
continue
|
||||
text = r.read().decode("utf-8-sig")
|
||||
raw.write_text(text, encoding="utf-8")
|
||||
print(f" fetched {work['slug']:<14} {url} {len(text):,} bytes")
|
||||
return text
|
||||
except Exception as e:
|
||||
print(f" .. {url} -> {type(e).__name__}")
|
||||
raise SystemExit(f"REFUSING: could not fetch {work['slug']} (id {work['id']})")
|
||||
|
||||
|
||||
def strip_boilerplate(text: str, slug: str) -> str:
|
||||
"""Keep only what lies between Gutenberg's own START/END markers.
|
||||
|
||||
Anchoring on the markers rather than on a line count is what makes this
|
||||
safe across editions -- the front matter length differs per work.
|
||||
"""
|
||||
m1, m2 = START.search(text), END.search(text)
|
||||
if not m1 or not m2:
|
||||
raise SystemExit(f"REFUSING: {slug} has no START/END markers; refusing to guess where the text begins")
|
||||
body = text[m1.end():m2.start()]
|
||||
# A transcriber credit block sometimes sits just inside the START marker.
|
||||
body = re.sub(r"\A\s*(?:Produced by|E-text prepared by|Transcribed from).*?\n\s*\n", "", body, flags=re.S | re.I)
|
||||
return body.strip("\n")
|
||||
|
||||
|
||||
ROMAN = {"I":1,"V":5,"X":10,"L":50,"C":100,"D":500,"M":1000}
|
||||
|
||||
|
||||
def roman_to_int(r: str) -> int:
|
||||
total, prev = 0, 0
|
||||
for ch in reversed(r.upper()):
|
||||
v = ROMAN.get(ch, 0)
|
||||
total = total - v if v < prev else total + v
|
||||
prev = max(prev, v)
|
||||
return total
|
||||
|
||||
|
||||
def find_chapters(body: str) -> list[tuple[int, str, int]]:
|
||||
"""Body chapter headings only, with any table of contents discarded.
|
||||
|
||||
Measured 2026-09-10: The Professor ships a TOC that puts TWO chapter names
|
||||
on one line, so a bare regex returns 38 headings for a 25-chapter novel and
|
||||
a naive minimum-gap filter still leaks the TOC's tail. The rule that works
|
||||
is structural rather than cosmetic -- the body's "CHAPTER I" is the LAST one
|
||||
in the file, because a TOC always precedes the text it indexes. From there,
|
||||
keep only headings that continue the sequence and are separated by prose.
|
||||
"""
|
||||
hits = []
|
||||
for m in CHAPTER.finditer(body):
|
||||
num = m.group(1).split()[-1].rstrip(".")
|
||||
n = int(num) if num.isdigit() else roman_to_int(num)
|
||||
hits.append((m.start(), m.group(1).strip(), n))
|
||||
if not hits:
|
||||
return []
|
||||
ones = [i for i, h in enumerate(hits) if h[2] == 1]
|
||||
start = ones[-1] if ones else 0
|
||||
kept, expect, last_pos = [], 1, -10**9
|
||||
for pos, label, n in hits[start:]:
|
||||
if n == expect and pos - last_pos > 500:
|
||||
kept.append((pos, label, n))
|
||||
expect, last_pos = expect + 1, pos
|
||||
return kept
|
||||
|
||||
|
||||
#: Normalisation is decided from the survey, not from a guess. Measured across
|
||||
#: the four works: Jane Eyre and Villette use curly quotes and em-dashes;
|
||||
#: SHIRLEY uses straight quotes and `--` with zero em-dashes; The Professor
|
||||
#: mixes curly quotes with `--`. That split is a transcriber artefact, not
|
||||
#: Charlotte Bronte's punctuation, and leaving it would teach the adapter that
|
||||
#: this author "sometimes" writes each form -- a false habit on the exact axis
|
||||
#: being trained. Normalise toward what the text MEANS: `--` is a transcription
|
||||
#: of an em-dash, so it becomes one.
|
||||
def normalise_quotes(text: str) -> str:
|
||||
"""Straight quotes -> curly, paired by alternation within each paragraph."""
|
||||
out = []
|
||||
for para in text.split("\n\n"):
|
||||
buf, open_d = [], True
|
||||
for ch in para:
|
||||
if ch == '"':
|
||||
buf.append("\u201c" if open_d else "\u201d")
|
||||
open_d = not open_d
|
||||
else:
|
||||
buf.append(ch)
|
||||
para = "".join(buf)
|
||||
# single quotes: apostrophe if flanked by letters, else a quote mark
|
||||
para = re.sub(r"(?<=[A-Za-z])'(?=[A-Za-z])", "\u2019", para)
|
||||
buf, open_s = [], True
|
||||
for ch in para:
|
||||
if ch == "'":
|
||||
buf.append("\u2018" if open_s else "\u2019")
|
||||
open_s = not open_s
|
||||
else:
|
||||
buf.append(ch)
|
||||
out.append("".join(buf))
|
||||
return "\n\n".join(out)
|
||||
|
||||
|
||||
def clean(text: str) -> str:
|
||||
text = text.replace("\u00a0", " ")
|
||||
text = re.sub(r"(?<!-)--(?!-)", "\u2014", text)
|
||||
text = normalise_quotes(text)
|
||||
text = re.sub(r"[ \t]+\n", "\n", text)
|
||||
text = re.sub(r"\n{3,}", "\n\n", text)
|
||||
return text.strip("\n")
|
||||
|
||||
|
||||
def survey(bodies: dict[str, str]) -> None:
|
||||
print("\n== character inventory, BEFORE any normalisation")
|
||||
allchars = collections.Counter()
|
||||
for slug, b in bodies.items():
|
||||
allchars.update(b)
|
||||
letters = {c for c in allchars if c.isalpha()}
|
||||
ascii_letters = {c for c in letters if ord(c) < 128}
|
||||
non_ascii = sorted(c for c in allchars if ord(c) > 127)
|
||||
print(f" distinct characters : {len(allchars)}")
|
||||
print(f" distinct letters : {len(letters)} (ascii {len(ascii_letters)}, non-ascii {len(letters - ascii_letters)})")
|
||||
print(f" distinct non-ascii chars : {len(non_ascii)}")
|
||||
print(" non-ascii, by frequency:")
|
||||
for c in sorted(non_ascii, key=lambda c: -allchars[c]):
|
||||
name = unicodedata.name(c, "?")
|
||||
print(f" U+{ord(c):04X} {c!r:<8} {allchars[c]:>7} {name}")
|
||||
print("\n== structure")
|
||||
for slug, b in bodies.items():
|
||||
heads = find_chapters(b)
|
||||
words = len(b.split())
|
||||
print(f" {slug:<14} {words:>8,} words {len(heads):>3} chapters last: {heads[-1][1] if heads else '-'}")
|
||||
print(f" {'TOTAL':<14} {sum(len(b.split()) for b in bodies.values()):>8,} words")
|
||||
|
||||
|
||||
def main() -> int:
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("--survey", action="store_true")
|
||||
ap.add_argument("--build", action="store_true")
|
||||
ap.add_argument("--out", default="corpus")
|
||||
ap.add_argument("--cache", default="raw")
|
||||
a = ap.parse_args()
|
||||
if not (a.survey or a.build):
|
||||
ap.error("pick --survey or --build")
|
||||
|
||||
cache = Path(a.cache)
|
||||
print("== fetch")
|
||||
bodies = {}
|
||||
for w in WORKS:
|
||||
bodies[w["slug"]] = strip_boilerplate(fetch(w, cache), w["slug"])
|
||||
assert "PROJECT GUTENBERG" not in bodies[w["slug"]][:2000].upper(), f"{w['slug']}: boilerplate survived"
|
||||
|
||||
if a.survey:
|
||||
survey(bodies)
|
||||
return 0
|
||||
|
||||
out = Path(a.out)
|
||||
(out / "works").mkdir(parents=True, exist_ok=True)
|
||||
manifest, alphabet = [], set()
|
||||
for w in WORKS:
|
||||
slug = w["slug"]
|
||||
body = clean(bodies[slug])
|
||||
chaps = find_chapters(body)
|
||||
if not chaps:
|
||||
raise SystemExit(f"REFUSING: no chapters found in {slug}")
|
||||
# Self-consistency: the count must equal the last heading's numeral, or
|
||||
# the segmentation has silently over- or under-matched.
|
||||
if len(chaps) != chaps[-1][2]:
|
||||
raise SystemExit(
|
||||
f"REFUSING: {slug} segmented into {len(chaps)} chapters but the last "
|
||||
f"heading is {chaps[-1][1]} (= {chaps[-1][2]}). Segmentation is wrong.")
|
||||
records = []
|
||||
for i, (pos, label, n) in enumerate(chaps):
|
||||
end = chaps[i + 1][0] if i + 1 < len(chaps) else len(body)
|
||||
text = body[pos:end].strip("\n")
|
||||
records.append({"work": slug, "chapter": n, "heading": label,
|
||||
"words": len(text.split()), "text": text})
|
||||
path = out / "works" / f"{slug}.jsonl"
|
||||
with path.open("w", encoding="utf-8") as fh:
|
||||
for r in records:
|
||||
fh.write(json.dumps(r, ensure_ascii=False) + "\n")
|
||||
alphabet |= {c for c in body if c.isalpha()}
|
||||
# Relative to the corpus root, never absolute: the corpus is built on one
|
||||
# box and trained on another, and an absolute build path makes the
|
||||
# manifest unreadable the moment it moves.
|
||||
manifest.append({"slug": slug, "gutenberg_id": w["id"], "title": w["title"],
|
||||
"chapters": len(records),
|
||||
"words": sum(r["words"] for r in records),
|
||||
"chars": len(body), "path": f"works/{slug}.jsonl"})
|
||||
print(f" wrote {slug:<14} {len(records):>3} chapters {sum(r['words'] for r in records):>8,} words")
|
||||
|
||||
alpha = sorted(alphabet)
|
||||
(out / "corpus_alphabet.json").write_text(json.dumps({
|
||||
"derived_from": "Charlotte Bronte, 4 novels, Project Gutenberg",
|
||||
"derived_at": "2026-09-10",
|
||||
"note": ("R49 F02 rule: a rename pool's character inventory must be a SUBSET of "
|
||||
"this. Bronte writes French constantly (Villette, Adele, Brussels), so "
|
||||
"unlike the Yarros corpus this alphabet legitimately carries accents -- "
|
||||
"but only FRENCH ones. Czech/Latvian/Slovak/Hungarian marks never appear "
|
||||
"and must not enter the pool."),
|
||||
"count": len(alpha), "letters": alpha,
|
||||
"non_ascii": [c for c in alpha if ord(c) > 127],
|
||||
}, ensure_ascii=False, indent=2), encoding="utf-8")
|
||||
(out / "manifest.json").write_text(json.dumps({
|
||||
"corpus": "bronte-charlotte-v1", "built_at": "2026-09-10",
|
||||
"source": "Project Gutenberg (public domain)",
|
||||
"normalisation": ("no-break space -> space; `--` -> em dash; straight quotes -> "
|
||||
"curly, paired per paragraph. Decided from the survey: Shirley "
|
||||
"was transcribed with straight quotes and zero em-dashes while "
|
||||
"Jane Eyre and Villette use curly and em-dash, a transcriber "
|
||||
"split rather than the author's punctuation."),
|
||||
"works": manifest,
|
||||
"total_words": sum(m["words"] for m in manifest),
|
||||
"total_chapters": sum(m["chapters"] for m in manifest),
|
||||
}, ensure_ascii=False, indent=2), encoding="utf-8")
|
||||
print(f"\n alphabet: {len(alpha)} letters ({len([c for c in alpha if ord(c)>127])} non-ascii)")
|
||||
print(f" TOTAL : {sum(m['words'] for m in manifest):,} words in "
|
||||
f"{sum(m['chapters'] for m in manifest)} chapters -> {out}")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,108 @@
|
||||
{
|
||||
"derived_from": "Charlotte Bronte, 4 novels, Project Gutenberg",
|
||||
"derived_at": "2026-09-10",
|
||||
"note": "R49 F02 rule: a rename pool's character inventory must be a SUBSET of this. Bronte writes French constantly (Villette, Adele, Brussels), so unlike the Yarros corpus this alphabet legitimately carries accents -- but only FRENCH ones. Czech/Latvian/Slovak/Hungarian marks never appear and must not enter the pool.",
|
||||
"count": 75,
|
||||
"letters": [
|
||||
"A",
|
||||
"B",
|
||||
"C",
|
||||
"D",
|
||||
"E",
|
||||
"F",
|
||||
"G",
|
||||
"H",
|
||||
"I",
|
||||
"J",
|
||||
"K",
|
||||
"L",
|
||||
"M",
|
||||
"N",
|
||||
"O",
|
||||
"P",
|
||||
"Q",
|
||||
"R",
|
||||
"S",
|
||||
"T",
|
||||
"U",
|
||||
"V",
|
||||
"W",
|
||||
"X",
|
||||
"Y",
|
||||
"Z",
|
||||
"a",
|
||||
"b",
|
||||
"c",
|
||||
"d",
|
||||
"e",
|
||||
"f",
|
||||
"g",
|
||||
"h",
|
||||
"i",
|
||||
"j",
|
||||
"k",
|
||||
"l",
|
||||
"m",
|
||||
"n",
|
||||
"o",
|
||||
"p",
|
||||
"q",
|
||||
"r",
|
||||
"s",
|
||||
"t",
|
||||
"u",
|
||||
"v",
|
||||
"w",
|
||||
"x",
|
||||
"y",
|
||||
"z",
|
||||
"Æ",
|
||||
"É",
|
||||
"Ê",
|
||||
"Ë",
|
||||
"Ô",
|
||||
"à",
|
||||
"â",
|
||||
"ä",
|
||||
"æ",
|
||||
"ç",
|
||||
"è",
|
||||
"é",
|
||||
"ê",
|
||||
"ë",
|
||||
"î",
|
||||
"ï",
|
||||
"ô",
|
||||
"ö",
|
||||
"ù",
|
||||
"û",
|
||||
"ü",
|
||||
"Œ",
|
||||
"œ"
|
||||
],
|
||||
"non_ascii": [
|
||||
"Æ",
|
||||
"É",
|
||||
"Ê",
|
||||
"Ë",
|
||||
"Ô",
|
||||
"à",
|
||||
"â",
|
||||
"ä",
|
||||
"æ",
|
||||
"ç",
|
||||
"è",
|
||||
"é",
|
||||
"ê",
|
||||
"ë",
|
||||
"î",
|
||||
"ï",
|
||||
"ô",
|
||||
"ö",
|
||||
"ù",
|
||||
"û",
|
||||
"ü",
|
||||
"Œ",
|
||||
"œ"
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,46 @@
|
||||
{
|
||||
"corpus": "bronte-charlotte-v1",
|
||||
"built_at": "2026-09-10",
|
||||
"source": "Project Gutenberg (public domain)",
|
||||
"normalisation": "no-break space -> space; `--` -> em dash; straight quotes -> curly, paired per paragraph. Decided from the survey: Shirley was transcribed with straight quotes and zero em-dashes while Jane Eyre and Villette use curly and em-dash, a transcriber split rather than the author's punctuation.",
|
||||
"works": [
|
||||
{
|
||||
"slug": "jane-eyre",
|
||||
"gutenberg_id": 1260,
|
||||
"title": "Jane Eyre: An Autobiography",
|
||||
"chapters": 38,
|
||||
"words": 184452,
|
||||
"chars": 1022193,
|
||||
"path": "works/jane-eyre.jsonl"
|
||||
},
|
||||
{
|
||||
"slug": "villette",
|
||||
"gutenberg_id": 9182,
|
||||
"title": "Villette",
|
||||
"chapters": 42,
|
||||
"words": 192411,
|
||||
"chars": 1092741,
|
||||
"path": "works/villette.jsonl"
|
||||
},
|
||||
{
|
||||
"slug": "shirley",
|
||||
"gutenberg_id": 30486,
|
||||
"title": "Shirley",
|
||||
"chapters": 37,
|
||||
"words": 216016,
|
||||
"chars": 1226278,
|
||||
"path": "works/shirley.jsonl"
|
||||
},
|
||||
{
|
||||
"slug": "the-professor",
|
||||
"gutenberg_id": 1028,
|
||||
"title": "The Professor",
|
||||
"chapters": 25,
|
||||
"words": 87412,
|
||||
"chars": 500054,
|
||||
"path": "works/the-professor.jsonl"
|
||||
}
|
||||
],
|
||||
"total_words": 680291,
|
||||
"total_chapters": 142
|
||||
}
|
||||
@@ -0,0 +1,108 @@
|
||||
"""R49 D1 acceptance gate for a built corpus.
|
||||
|
||||
The design doc's D1 acceptance is "clean UTF-8, chapter-segmented, zero
|
||||
boilerplate lines, stable tokenization". Each is checked here as something that
|
||||
can actually go RED -- a gate that cannot fail is the third failure mode this
|
||||
target has already recorded, and it is not repeated here.
|
||||
|
||||
python verify_corpus.py <corpus-dir> [--tokenizer PATH]
|
||||
"""
|
||||
from __future__ import annotations
|
||||
import argparse, collections, json, re, sys, unicodedata
|
||||
from pathlib import Path
|
||||
|
||||
#: ⚠ Anchored to line start, and that is not cosmetic. The first draft matched
|
||||
#: `Produced by` anywhere and went RED on four hits that were all Charlotte
|
||||
#: Bronte's own prose -- "a chilling effect produced by his steady announcement",
|
||||
#: "how such a result was produced by such means". A hard rule on a phrase with a
|
||||
#: common non-boilerplate sense manufactures failures; same shape as the drift
|
||||
#: detector that fired on the adjective "minor" and stopped work three times.
|
||||
#: Gutenberg credits always begin a line, so require that.
|
||||
BOILER = [r"^.*PROJECT GUTENBERG.*$", r"^.*gutenberg\.org.*$", r"^\s*Produced by\b",
|
||||
r"^\s*E-text prepared by\b", r"^\s*Transcribed from\b",
|
||||
r"^\s*Distributed Proofread", r"^\*\*\*\s*(?:START|END) OF"]
|
||||
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("corpus")
|
||||
ap.add_argument("--tokenizer", default=None)
|
||||
a = ap.parse_args()
|
||||
root = Path(a.corpus)
|
||||
man = json.loads((root / "manifest.json").read_text())
|
||||
alpha = json.loads((root / "corpus_alphabet.json").read_text())
|
||||
|
||||
records = []
|
||||
for w in man["works"]:
|
||||
for line in (root / w["path"]).read_text(encoding="utf-8").splitlines():
|
||||
records.append(json.loads(line))
|
||||
text = "\n\n".join(r["text"] for r in records)
|
||||
fails = []
|
||||
|
||||
def check(name, ok, detail=""):
|
||||
print(f" [{'PASS' if ok else 'FAIL'}] {name}{(' -- ' + detail) if detail else ''}")
|
||||
if not ok:
|
||||
fails.append(name)
|
||||
|
||||
print(f"== {len(records)} chapters, {sum(r['words'] for r in records):,} words, {len(text):,} chars\n")
|
||||
|
||||
# 1. boilerplate
|
||||
hits = {p: len(re.findall(p, text, re.I | re.M)) for p in BOILER}
|
||||
bad = {p: n for p, n in hits.items() if n}
|
||||
check("zero Gutenberg boilerplate", not bad, f"found {bad}" if bad else "7 patterns, 0 hits")
|
||||
|
||||
# 2. structure
|
||||
per_work = collections.Counter(r["work"] for r in records)
|
||||
seq_ok = all(
|
||||
[r["chapter"] for r in records if r["work"] == w] == list(range(1, per_work[w] + 1))
|
||||
for w in per_work)
|
||||
check("chapters number 1..N with no gaps", seq_ok, ", ".join(f"{w}:{n}" for w, n in per_work.items()))
|
||||
check("no empty chapters", all(r["words"] > 100 for r in records),
|
||||
f"min {min(r['words'] for r in records)} words")
|
||||
|
||||
# 3. typography consistency AFTER normalisation -- the reason normalisation exists
|
||||
counts = collections.Counter(text)
|
||||
straight = counts['"'] + counts["'"]
|
||||
dbl_hyphen = len(re.findall(r"(?<!-)--(?!-)", text))
|
||||
check("no straight quotes survive", straight == 0, f'" {counts[chr(34)]}, \' {counts[chr(39)]}')
|
||||
check("no `--` survives", dbl_hyphen == 0, f"{dbl_hyphen} occurrences")
|
||||
#: An open/close COUNT mismatch is not an error here and asserting equality was
|
||||
#: a bad gate. Nineteenth-century convention runs a speech across paragraphs by
|
||||
#: opening each one and closing only the last, so every work carries a surplus of
|
||||
#: opens -- measured +46 / +49 / +51 on the three works whose quotes were never
|
||||
#: touched. The real error signature is a paragraph that BEGINS with a closing
|
||||
#: quote, which convention never produces and a bad conversion does.
|
||||
paras = [p.strip() for p in text.split("\n\n") if p.strip()]
|
||||
lead_close = [p[:60] for p in paras if p.lstrip()[:1] == chr(0x201d)]
|
||||
check("no paragraph opens with a closing quote", not lead_close,
|
||||
f"{len(lead_close)} of {len(paras):,} paragraphs" + (f" e.g. {lead_close[0]!r}" if lead_close else ""))
|
||||
surplus = counts[chr(0x201c)] - counts[chr(0x201d)]
|
||||
print(f" open-quote surplus {surplus:+} of {counts[chr(0x201c)]:,} "
|
||||
f"(multi-paragraph speech; expected, not a failure)")
|
||||
|
||||
# 4. alphabet is the real inventory
|
||||
observed = {c for c in text if c.isalpha()}
|
||||
check("alphabet matches the text exactly", observed == set(alpha["letters"]),
|
||||
f"declared {len(alpha['letters'])}, observed {len(observed)}, "
|
||||
f"diff {sorted(observed ^ set(alpha['letters']))}")
|
||||
|
||||
# 5. no control / exotic codepoints
|
||||
weird = {c for c in text if unicodedata.category(c) in ("Cc", "Cf", "Co", "Cs") and c != "\n"}
|
||||
check("no control or private-use codepoints", not weird, repr(sorted(weird)))
|
||||
|
||||
# 6. tokenizer stability -- F02's byte-fallback lesson, on the real carrier
|
||||
if a.tokenizer:
|
||||
from transformers import AutoTokenizer
|
||||
tok = AutoTokenizer.from_pretrained(a.tokenizer)
|
||||
sample = text[:400000]
|
||||
ids = tok.encode(sample, add_special_tokens=False)
|
||||
back = tok.decode(ids)
|
||||
check("tokenizer round-trip is lossless", back == sample,
|
||||
f"{len(ids):,} tokens from {len(sample):,} chars")
|
||||
pieces = tok.convert_ids_to_tokens(ids)
|
||||
fallback = [p for p in pieces if "�" in p]
|
||||
check("no byte-fallback pieces", not fallback,
|
||||
f"{len(fallback)} of {len(pieces):,} pieces")
|
||||
total = len(tok.encode(text, add_special_tokens=False))
|
||||
print(f"\n full corpus = {total:,} tokens ({total/sum(r['words'] for r in records):.2f} tok/word)")
|
||||
|
||||
print(f"\n== {'ALL CHECKS PASSED' if not fails else 'FAILED: ' + ', '.join(fails)}")
|
||||
sys.exit(1 if fails else 0)
|
||||
Reference in New Issue
Block a user