vh
ba8dac2c80
scripts/r49-corpus/{build_corpus,verify_corpus}.py; corpus staged at
gx10:~/r49-corpus/. Catalogue ids verified against gutenberg.org's own search
rather than recalled. Charlotte only -- the Bell poems are co-authored and the
Gaskell biography is a different hand, so neither belongs in a single-voice corpus.
Jane Eyre 1260 · Villette 9182 · Shirley 30486 · The Professor 1028
680,291 words · 142 chapters · 950,974 Qwen3 tokens (1.40 tok/word)
alphabet 75 letters, 23 non-ASCII · round-trip lossless · 0 byte-fallback
All 11 acceptance checks pass, including both tokenizer legs run against the pilot
carrier itself. With a real denominator the projections tighten: at 6 rename copies
x 3 epochs = 17.1M tokens, the 0.6B pilot is 1.98 h.
THE ALPHABET INVERTS THE YARROS RESULT. Brontë writes French constantly -- Villette
is set in a French-speaking city, Jane Eyre has Adèle, The Professor is set in
Brussels -- so the corpus carries é 432, è 237, à 93, ê 79, ô 48 plus œ and æ. F02
measured Yarros at 0.0002% non-ASCII and derived an ASCII-fold for the name pool.
Under F02's own subset rule the Brontë pool may keep FRENCH accents and must still
exclude the Czech/Latvian/Slovak/Hungarian marks that never appear here. The fold is
per-work, and this is the first corpus where deriving it changes the answer.
Typography was inconsistent across works and it was the transcriber, not the author:
Shirley uses straight quotes and `--` with zero em-dashes while Jane Eyre and
Villette use curly and em-dash. Normalised toward what the text means.
Three defects, each found by running something rather than reasoning about it:
`Produced by` matched Brontë's own prose four times, which is the adjective-"minor"
shape again and is fixed by anchoring boilerplate patterns to line start; asserting
open/close quote counts must be equal is wrong, because 19th-century multi-paragraph
speech legitimately runs a surplus of opens, so the real error signature is that no
paragraph may begin with a closing quote; and The Professor's table of contents puts
two chapter names per line, so a bare regex returns 38 headings for a 25-chapter
novel and a minimum-gap filter still leaks its tail -- the rule that works is that
the body's "CHAPTER I" is the last one in the file.
Records the operator's pilot ruling: trial on Qwen3-0.6B-Base first, move up only if
it produces something useful.
46 lines
1.3 KiB
JSON
46 lines
1.3 KiB
JSON
{
|
|
"corpus": "bronte-charlotte-v1",
|
|
"built_at": "2026-09-10",
|
|
"source": "Project Gutenberg (public domain)",
|
|
"normalisation": "no-break space -> space; `--` -> em dash; straight quotes -> curly, paired per paragraph. Decided from the survey: Shirley was transcribed with straight quotes and zero em-dashes while Jane Eyre and Villette use curly and em-dash, a transcriber split rather than the author's punctuation.",
|
|
"works": [
|
|
{
|
|
"slug": "jane-eyre",
|
|
"gutenberg_id": 1260,
|
|
"title": "Jane Eyre: An Autobiography",
|
|
"chapters": 38,
|
|
"words": 184452,
|
|
"chars": 1022193,
|
|
"path": "works/jane-eyre.jsonl"
|
|
},
|
|
{
|
|
"slug": "villette",
|
|
"gutenberg_id": 9182,
|
|
"title": "Villette",
|
|
"chapters": 42,
|
|
"words": 192411,
|
|
"chars": 1092741,
|
|
"path": "works/villette.jsonl"
|
|
},
|
|
{
|
|
"slug": "shirley",
|
|
"gutenberg_id": 30486,
|
|
"title": "Shirley",
|
|
"chapters": 37,
|
|
"words": 216016,
|
|
"chars": 1226278,
|
|
"path": "works/shirley.jsonl"
|
|
},
|
|
{
|
|
"slug": "the-professor",
|
|
"gutenberg_id": 1028,
|
|
"title": "The Professor",
|
|
"chapters": 25,
|
|
"words": 87412,
|
|
"chars": 500054,
|
|
"path": "works/the-professor.jsonl"
|
|
}
|
|
],
|
|
"total_words": 680291,
|
|
"total_chapters": 142
|
|
} |