feat(r49): D1 corpus built and green — Charlotte Brontë, 680k words, 951k tokens

scripts/r49-corpus/{build_corpus,verify_corpus}.py; corpus staged at
gx10:~/r49-corpus/. Catalogue ids verified against gutenberg.org's own search
rather than recalled. Charlotte only -- the Bell poems are co-authored and the
Gaskell biography is a different hand, so neither belongs in a single-voice corpus.

  Jane Eyre 1260 · Villette 9182 · Shirley 30486 · The Professor 1028
  680,291 words · 142 chapters · 950,974 Qwen3 tokens (1.40 tok/word)
  alphabet 75 letters, 23 non-ASCII · round-trip lossless · 0 byte-fallback

All 11 acceptance checks pass, including both tokenizer legs run against the pilot
carrier itself. With a real denominator the projections tighten: at 6 rename copies
x 3 epochs = 17.1M tokens, the 0.6B pilot is 1.98 h.

THE ALPHABET INVERTS THE YARROS RESULT. Brontë writes French constantly -- Villette
is set in a French-speaking city, Jane Eyre has Adèle, The Professor is set in
Brussels -- so the corpus carries é 432, è 237, à 93, ê 79, ô 48 plus œ and æ. F02
measured Yarros at 0.0002% non-ASCII and derived an ASCII-fold for the name pool.
Under F02's own subset rule the Brontë pool may keep FRENCH accents and must still
exclude the Czech/Latvian/Slovak/Hungarian marks that never appear here. The fold is
per-work, and this is the first corpus where deriving it changes the answer.

Typography was inconsistent across works and it was the transcriber, not the author:
Shirley uses straight quotes and `--` with zero em-dashes while Jane Eyre and
Villette use curly and em-dash. Normalised toward what the text means.

Three defects, each found by running something rather than reasoning about it:
`Produced by` matched Brontë's own prose four times, which is the adjective-"minor"
shape again and is fixed by anchoring boilerplate patterns to line start; asserting
open/close quote counts must be equal is wrong, because 19th-century multi-paragraph
speech legitimately runs a surplus of opens, so the real error signature is that no
paragraph may begin with a closing quote; and The Professor's table of contents puts
two chapter names per line, so a bare regex returns 38 headings for a 25-chapter
novel and a minimum-gap filter still leaks its tail -- the rule that works is that
the body's "CHAPTER I" is the last one in the file.

Records the operator's pilot ruling: trial on Qwen3-0.6B-Base first, move up only if
it produces something useful.
This commit is contained in:
vh
2026-09-10 06:51:32 -07:00
parent 6cbc9c7a2c
commit ba8dac2c80
6 changed files with 644 additions and 4 deletions
+87 -3
View File
@@ -174,9 +174,7 @@ first real corpus.
## 5. Prep remaining, in order
1. ~~Carrier family decision~~ — **settled: dense `Qwen3`** (§6a).
2. **Corpus D1** — Gutenberg Brontë (Jane Eyre, Villette, Shirley, The
Professor), boilerplate stripped, chapter-segmented, typography normalised,
character inventory recorded. Public domain, clean under any disposition.
2. ~~Corpus D1~~ — **BUILT AND GREEN 2026-09-10** (§8).
3. **Re-point the R49 deterministic machinery at Brontë.** The entity detector
(corpus-level capitalised-vs-lowercase ratio), identity linking, gender
resolution and the 23,398-name dictionary were all built and hardened against
@@ -310,3 +308,89 @@ upstream cannot hand back, so `configs/restic/ana-ml2/profiles.yaml` now carries
a single documented carve-out, `/tank/erp-tune/run-*/adapter`, verified by
`resticprofile --dry-run` to expand to exactly those eight paths and nothing
else. The nightly 01:00 run picks them up.
---
## 8. D1 is built — the corpus, and three things it taught
`scripts/r49-corpus/build_corpus.py --build` and `verify_corpus.py`, corpus staged
at `gx10:~/r49-corpus/`. **All 11 acceptance checks pass**, including the two
tokenizer legs run against the pilot carrier itself.
| | |
|---|---|
| works | Jane Eyre (1260) · Villette (9182) · Shirley (30486) · The Professor (1028) |
| size | **680,291 words · 142 chapters · 950,974 tokens** under the Qwen3 tokenizer (1.40 tok/word) |
| alphabet | 75 letters, **23 of them non-ASCII** |
| tokenizer | round-trip lossless, **0 byte-fallback pieces** of 98,860 |
Catalogue ids were verified against gutenberg.org's own search, not recalled.
Charlotte only — the Bell poems are co-authored and the Gaskell biography is a
different hand, so neither belongs in a single-voice corpus.
**Real denominator, so the projections tighten.** At 6 rename copies × 3 epochs =
17.1M tokens: **0.6B → 1.98 h**, 1.7B → 3.36 h, 4B → 6.63 h per voice. The pilot
is a two-hour run.
### ⚠ The alphabet finding inverts the Yarros result, and brokkr's re-point depends on it
ÆÉÊËÔàâäæçèéêëîïôöùûüŒœ
F02 measured the Yarros corpus at **4 non-ASCII characters in 1.8M letters**
(0.0002%) and derived an ASCII-fold rule for the name pool. **Charlotte Brontë is
the opposite case**: she writes French constantly — Villette is set in a
French-speaking city, Jane Eyre has Adèle, The Professor is set in Brussels — and
the corpus carries `é` 432 times, `è` 237, `à` 93, `ê` 79, `ô` 48, plus the `œ`
and `æ` ligatures.
Under F02's own rule (*the pool's character inventory must be a subset of the
corpus's*) the Brontë pool **may keep French accents** and **must still exclude**
the Czech/Latvian/Slovak/Hungarian marks that never appear here. So the fold is
not global — it is derived per work, which is exactly what the rule said, and
this is the first corpus where the derivation changes the answer.
### Typography was inconsistent, and it was the transcriber, not the author
| work | quotes | dashes |
|---|---|---|
| Jane Eyre | curly | em-dash 2,058 |
| Villette | curly | em-dash 2,272 |
| **Shirley** | **straight** 9,115 | **`--` 2,228, zero em-dashes** |
| The Professor | curly | `--` 964, zero em-dashes |
Left alone the adapter would learn that this author "sometimes" writes each form
— a false habit on the exact axis being trained. Normalised **toward what the text
means**: `--` is a transcription of an em-dash, so it becomes one; straight quotes
are paired into curly per paragraph.
### Two gate defects, both found by running the gate
1. **`Produced by` matched Brontë's own prose** — *"a chilling effect produced by
his steady announcement"*, three more like it. A hard rule on a phrase with a
common innocent sense, manufacturing failures: the same shape as the drift
detector that fired on the adjective "minor". Fixed by anchoring the
boilerplate patterns to line start, where Gutenberg credits actually live.
2. **Asserting open/close quote counts must be equal was wrong.** Nineteenth-century
convention runs a speech across paragraphs by opening each and closing only the
last, so a surplus of opens is correct — measured **+46 / +49 / +51** on the
three works whose quotes were never touched. Replaced with the real error
signature: *no paragraph may begin with a closing quote*, which convention never
produces and a bad conversion does. 0 of 14,230 paragraphs.
A third, mine: **the manifest baked absolute build-machine paths**, so the corpus
was unreadable the moment it moved to gx10. Paths are relative to the corpus root
now. It failed loudly rather than silently reading nothing, which is why it was
cheap.
### Next
D2/D3 — re-point F02's entity detector, identity linking and gender resolution off
the Yarros sample onto Brontë, with the alphabet above constraining the pool. Then
D4 annotation, then the pilot.
**Pilot ruling (operator, 2026-09-10): trial on `Qwen3-0.6B-Base` first and only
move up if it produces something useful.** So the sweep is not three arms up front
— it is one ~2 h run, judged, and then a decision. That is the cheap ordering and
it front-loads the kill signal: if voice does not transfer at 0.6B the question
becomes *how far up* rather than *whether at all*, and if it does transfer the
larger arms are a refinement rather than a gamble.