vh
ba8dac2c80
scripts/r49-corpus/{build_corpus,verify_corpus}.py; corpus staged at
gx10:~/r49-corpus/. Catalogue ids verified against gutenberg.org's own search
rather than recalled. Charlotte only -- the Bell poems are co-authored and the
Gaskell biography is a different hand, so neither belongs in a single-voice corpus.
Jane Eyre 1260 · Villette 9182 · Shirley 30486 · The Professor 1028
680,291 words · 142 chapters · 950,974 Qwen3 tokens (1.40 tok/word)
alphabet 75 letters, 23 non-ASCII · round-trip lossless · 0 byte-fallback
All 11 acceptance checks pass, including both tokenizer legs run against the pilot
carrier itself. With a real denominator the projections tighten: at 6 rename copies
x 3 epochs = 17.1M tokens, the 0.6B pilot is 1.98 h.
THE ALPHABET INVERTS THE YARROS RESULT. Brontë writes French constantly -- Villette
is set in a French-speaking city, Jane Eyre has Adèle, The Professor is set in
Brussels -- so the corpus carries é 432, è 237, à 93, ê 79, ô 48 plus œ and æ. F02
measured Yarros at 0.0002% non-ASCII and derived an ASCII-fold for the name pool.
Under F02's own subset rule the Brontë pool may keep FRENCH accents and must still
exclude the Czech/Latvian/Slovak/Hungarian marks that never appear here. The fold is
per-work, and this is the first corpus where deriving it changes the answer.
Typography was inconsistent across works and it was the transcriber, not the author:
Shirley uses straight quotes and `--` with zero em-dashes while Jane Eyre and
Villette use curly and em-dash. Normalised toward what the text means.
Three defects, each found by running something rather than reasoning about it:
`Produced by` matched Brontë's own prose four times, which is the adjective-"minor"
shape again and is fixed by anchoring boilerplate patterns to line start; asserting
open/close quote counts must be equal is wrong, because 19th-century multi-paragraph
speech legitimately runs a surplus of opens, so the real error signature is that no
paragraph may begin with a closing quote; and The Professor's table of contents puts
two chapter names per line, so a bare regex returns 38 headings for a 25-chapter
novel and a minimum-gap filter still leaks its tail -- the rule that works is that
the body's "CHAPTER I" is the last one in the file.
Records the operator's pilot ruling: trial on Qwen3-0.6B-Base first, move up only if
it produces something useful.
108 lines
1.4 KiB
JSON
108 lines
1.4 KiB
JSON
{
|
|
"derived_from": "Charlotte Bronte, 4 novels, Project Gutenberg",
|
|
"derived_at": "2026-09-10",
|
|
"note": "R49 F02 rule: a rename pool's character inventory must be a SUBSET of this. Bronte writes French constantly (Villette, Adele, Brussels), so unlike the Yarros corpus this alphabet legitimately carries accents -- but only FRENCH ones. Czech/Latvian/Slovak/Hungarian marks never appear and must not enter the pool.",
|
|
"count": 75,
|
|
"letters": [
|
|
"A",
|
|
"B",
|
|
"C",
|
|
"D",
|
|
"E",
|
|
"F",
|
|
"G",
|
|
"H",
|
|
"I",
|
|
"J",
|
|
"K",
|
|
"L",
|
|
"M",
|
|
"N",
|
|
"O",
|
|
"P",
|
|
"Q",
|
|
"R",
|
|
"S",
|
|
"T",
|
|
"U",
|
|
"V",
|
|
"W",
|
|
"X",
|
|
"Y",
|
|
"Z",
|
|
"a",
|
|
"b",
|
|
"c",
|
|
"d",
|
|
"e",
|
|
"f",
|
|
"g",
|
|
"h",
|
|
"i",
|
|
"j",
|
|
"k",
|
|
"l",
|
|
"m",
|
|
"n",
|
|
"o",
|
|
"p",
|
|
"q",
|
|
"r",
|
|
"s",
|
|
"t",
|
|
"u",
|
|
"v",
|
|
"w",
|
|
"x",
|
|
"y",
|
|
"z",
|
|
"Æ",
|
|
"É",
|
|
"Ê",
|
|
"Ë",
|
|
"Ô",
|
|
"à",
|
|
"â",
|
|
"ä",
|
|
"æ",
|
|
"ç",
|
|
"è",
|
|
"é",
|
|
"ê",
|
|
"ë",
|
|
"î",
|
|
"ï",
|
|
"ô",
|
|
"ö",
|
|
"ù",
|
|
"û",
|
|
"ü",
|
|
"Œ",
|
|
"œ"
|
|
],
|
|
"non_ascii": [
|
|
"Æ",
|
|
"É",
|
|
"Ê",
|
|
"Ë",
|
|
"Ô",
|
|
"à",
|
|
"â",
|
|
"ä",
|
|
"æ",
|
|
"ç",
|
|
"è",
|
|
"é",
|
|
"ê",
|
|
"ë",
|
|
"î",
|
|
"ï",
|
|
"ô",
|
|
"ö",
|
|
"ù",
|
|
"û",
|
|
"ü",
|
|
"Œ",
|
|
"œ"
|
|
]
|
|
} |