feat(r49): D1 corpus built and green — Charlotte Brontë, 680k words, 951k tokens
scripts/r49-corpus/{build_corpus,verify_corpus}.py; corpus staged at
gx10:~/r49-corpus/. Catalogue ids verified against gutenberg.org's own search
rather than recalled. Charlotte only -- the Bell poems are co-authored and the
Gaskell biography is a different hand, so neither belongs in a single-voice corpus.
Jane Eyre 1260 · Villette 9182 · Shirley 30486 · The Professor 1028
680,291 words · 142 chapters · 950,974 Qwen3 tokens (1.40 tok/word)
alphabet 75 letters, 23 non-ASCII · round-trip lossless · 0 byte-fallback
All 11 acceptance checks pass, including both tokenizer legs run against the pilot
carrier itself. With a real denominator the projections tighten: at 6 rename copies
x 3 epochs = 17.1M tokens, the 0.6B pilot is 1.98 h.
THE ALPHABET INVERTS THE YARROS RESULT. Brontë writes French constantly -- Villette
is set in a French-speaking city, Jane Eyre has Adèle, The Professor is set in
Brussels -- so the corpus carries é 432, è 237, à 93, ê 79, ô 48 plus œ and æ. F02
measured Yarros at 0.0002% non-ASCII and derived an ASCII-fold for the name pool.
Under F02's own subset rule the Brontë pool may keep FRENCH accents and must still
exclude the Czech/Latvian/Slovak/Hungarian marks that never appear here. The fold is
per-work, and this is the first corpus where deriving it changes the answer.
Typography was inconsistent across works and it was the transcriber, not the author:
Shirley uses straight quotes and `--` with zero em-dashes while Jane Eyre and
Villette use curly and em-dash. Normalised toward what the text means.
Three defects, each found by running something rather than reasoning about it:
`Produced by` matched Brontë's own prose four times, which is the adjective-"minor"
shape again and is fixed by anchoring boilerplate patterns to line start; asserting
open/close quote counts must be equal is wrong, because 19th-century multi-paragraph
speech legitimately runs a surplus of opens, so the real error signature is that no
paragraph may begin with a closing quote; and The Professor's table of contents puts
two chapter names per line, so a bare regex returns 38 headings for a 25-chapter
novel and a minimum-gap filter still leaks its tail -- the rule that works is that
the body's "CHAPTER I" is the last one in the file.
Records the operator's pilot ruling: trial on Qwen3-0.6B-Base first, move up only if
it produces something useful.
This commit is contained in:
@@ -174,9 +174,7 @@ first real corpus.
|
||||
## 5. Prep remaining, in order
|
||||
|
||||
1. ~~Carrier family decision~~ — **settled: dense `Qwen3`** (§6a).
|
||||
2. **Corpus D1** — Gutenberg Brontë (Jane Eyre, Villette, Shirley, The
|
||||
Professor), boilerplate stripped, chapter-segmented, typography normalised,
|
||||
character inventory recorded. Public domain, clean under any disposition.
|
||||
2. ~~Corpus D1~~ — **BUILT AND GREEN 2026-09-10** (§8).
|
||||
3. **Re-point the R49 deterministic machinery at Brontë.** The entity detector
|
||||
(corpus-level capitalised-vs-lowercase ratio), identity linking, gender
|
||||
resolution and the 23,398-name dictionary were all built and hardened against
|
||||
@@ -310,3 +308,89 @@ upstream cannot hand back, so `configs/restic/ana-ml2/profiles.yaml` now carries
|
||||
a single documented carve-out, `/tank/erp-tune/run-*/adapter`, verified by
|
||||
`resticprofile --dry-run` to expand to exactly those eight paths and nothing
|
||||
else. The nightly 01:00 run picks them up.
|
||||
|
||||
---
|
||||
|
||||
## 8. D1 is built — the corpus, and three things it taught
|
||||
|
||||
`scripts/r49-corpus/build_corpus.py --build` and `verify_corpus.py`, corpus staged
|
||||
at `gx10:~/r49-corpus/`. **All 11 acceptance checks pass**, including the two
|
||||
tokenizer legs run against the pilot carrier itself.
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| works | Jane Eyre (1260) · Villette (9182) · Shirley (30486) · The Professor (1028) |
|
||||
| size | **680,291 words · 142 chapters · 950,974 tokens** under the Qwen3 tokenizer (1.40 tok/word) |
|
||||
| alphabet | 75 letters, **23 of them non-ASCII** |
|
||||
| tokenizer | round-trip lossless, **0 byte-fallback pieces** of 98,860 |
|
||||
|
||||
Catalogue ids were verified against gutenberg.org's own search, not recalled.
|
||||
Charlotte only — the Bell poems are co-authored and the Gaskell biography is a
|
||||
different hand, so neither belongs in a single-voice corpus.
|
||||
|
||||
**Real denominator, so the projections tighten.** At 6 rename copies × 3 epochs =
|
||||
17.1M tokens: **0.6B → 1.98 h**, 1.7B → 3.36 h, 4B → 6.63 h per voice. The pilot
|
||||
is a two-hour run.
|
||||
|
||||
### ⚠ The alphabet finding inverts the Yarros result, and brokkr's re-point depends on it
|
||||
|
||||
ÆÉÊËÔàâäæçèéêëîïôöùûüŒœ
|
||||
|
||||
F02 measured the Yarros corpus at **4 non-ASCII characters in 1.8M letters**
|
||||
(0.0002%) and derived an ASCII-fold rule for the name pool. **Charlotte Brontë is
|
||||
the opposite case**: she writes French constantly — Villette is set in a
|
||||
French-speaking city, Jane Eyre has Adèle, The Professor is set in Brussels — and
|
||||
the corpus carries `é` 432 times, `è` 237, `à` 93, `ê` 79, `ô` 48, plus the `œ`
|
||||
and `æ` ligatures.
|
||||
|
||||
Under F02's own rule (*the pool's character inventory must be a subset of the
|
||||
corpus's*) the Brontë pool **may keep French accents** and **must still exclude**
|
||||
the Czech/Latvian/Slovak/Hungarian marks that never appear here. So the fold is
|
||||
not global — it is derived per work, which is exactly what the rule said, and
|
||||
this is the first corpus where the derivation changes the answer.
|
||||
|
||||
### Typography was inconsistent, and it was the transcriber, not the author
|
||||
|
||||
| work | quotes | dashes |
|
||||
|---|---|---|
|
||||
| Jane Eyre | curly | em-dash 2,058 |
|
||||
| Villette | curly | em-dash 2,272 |
|
||||
| **Shirley** | **straight** 9,115 | **`--` 2,228, zero em-dashes** |
|
||||
| The Professor | curly | `--` 964, zero em-dashes |
|
||||
|
||||
Left alone the adapter would learn that this author "sometimes" writes each form
|
||||
— a false habit on the exact axis being trained. Normalised **toward what the text
|
||||
means**: `--` is a transcription of an em-dash, so it becomes one; straight quotes
|
||||
are paired into curly per paragraph.
|
||||
|
||||
### Two gate defects, both found by running the gate
|
||||
|
||||
1. **`Produced by` matched Brontë's own prose** — *"a chilling effect produced by
|
||||
his steady announcement"*, three more like it. A hard rule on a phrase with a
|
||||
common innocent sense, manufacturing failures: the same shape as the drift
|
||||
detector that fired on the adjective "minor". Fixed by anchoring the
|
||||
boilerplate patterns to line start, where Gutenberg credits actually live.
|
||||
2. **Asserting open/close quote counts must be equal was wrong.** Nineteenth-century
|
||||
convention runs a speech across paragraphs by opening each and closing only the
|
||||
last, so a surplus of opens is correct — measured **+46 / +49 / +51** on the
|
||||
three works whose quotes were never touched. Replaced with the real error
|
||||
signature: *no paragraph may begin with a closing quote*, which convention never
|
||||
produces and a bad conversion does. 0 of 14,230 paragraphs.
|
||||
|
||||
A third, mine: **the manifest baked absolute build-machine paths**, so the corpus
|
||||
was unreadable the moment it moved to gx10. Paths are relative to the corpus root
|
||||
now. It failed loudly rather than silently reading nothing, which is why it was
|
||||
cheap.
|
||||
|
||||
### Next
|
||||
|
||||
D2/D3 — re-point F02's entity detector, identity linking and gender resolution off
|
||||
the Yarros sample onto Brontë, with the alphabet above constraining the pool. Then
|
||||
D4 annotation, then the pilot.
|
||||
|
||||
**Pilot ruling (operator, 2026-09-10): trial on `Qwen3-0.6B-Base` first and only
|
||||
move up if it produces something useful.** So the sweep is not three arms up front
|
||||
— it is one ~2 h run, judged, and then a decision. That is the cheap ordering and
|
||||
it front-loads the kill signal: if voice does not transfer at 0.6B the question
|
||||
becomes *how far up* rather than *whether at all*, and if it does transfer the
|
||||
larger arms are a refinement rather than a gamble.
|
||||
|
||||
Reference in New Issue
Block a user