121 lines
6.9 KiB
Markdown
121 lines
6.9 KiB
Markdown
# `[2026-09-17]` McCarthy and Krakauer D1 — both corpora built, and one of them should not be trained yet
|
||
|
||
Operator picked **both** after the library survey. McCarthy first, because the splitter work
|
||
is shared and he is the cleaner author.
|
||
|
||
| corpus | units | words | where | guards |
|
||
|---|---|---|---|---|
|
||
| **lv-mccarthy** | 167 | **584,756** | `~/mccarthy-corpus` | style ✅ name ✅ |
|
||
| **lv-krakauer** | 126 | **422,880** | `~/krakauer-corpus` | name ✅ (2 allowed) · quotation ⚠ UNRESOLVED |
|
||
|
||
Commits `705fa3a` (splitter) `f3bf3ca` (McCarthy) `4be0630` (Krakauer).
|
||
|
||
## The shared blocker: a splitter that chooses by SIZE
|
||
|
||
`scripts/r49-corpus/split_units.py`. The inherited rule was "most units above a floor", and
|
||
it is wrong for any book whose markers are PARTS:
|
||
|
||
```
|
||
Cities of the Plain 4 roman marks -> 4 units, median 22,312w
|
||
The Crossing 4 roman marks -> 4 units, median 37,310w
|
||
```
|
||
|
||
"Most units" scores 4 over the 1 that finding-nothing gives, so it wins, and the existing
|
||
guard only fires at exactly one unit. Now: a mode qualifies only if its median unit is inside
|
||
**[600, 12000]** AND no single unit holds half the work; among qualifying modes **priority**
|
||
breaks the tie (contents > chapter-word > roman > bare-numeral > caps-title). Works with no
|
||
divisions fall back to **paragraph-block sections** at ~3,000 words, cut on block boundaries.
|
||
|
||
⭐⭐ **BOTH RULES EXIST BECAUSE A CONTROL RUN CAUGHT THEM, and the first version was worse
|
||
than what it replaced.** Scoring eligible modes by "median closest to target" chose
|
||
`caps-title` over the real chapters of Hemingway's *True at First Light*:
|
||
|
||
```
|
||
bare-numeral 20 units median 5,337w max 11,155 <- the book's own chapters
|
||
caps-title 6 units median 777w max 113,886 <- median looked BETTER
|
||
```
|
||
|
||
Five stray all-caps lines gave five tiny units beside **one holding 97% of the book**. A
|
||
median cannot see that distribution; a max bound can. Controls now green both ways: 8/10
|
||
Hemingway works reproduce the shipped mode and unit count exactly (the two differences are
|
||
`contents`, which the harness does not supply, and The Old Man and the Sea, deliberately kept
|
||
whole), and 40,000 words with no blank lines returns **1 unit** rather than fabricating
|
||
sections.
|
||
|
||
⚠ The Hemingway builder is deliberately NOT repointed at this module. Its corpus is shipped
|
||
and its sha is pinned by a live adapter.
|
||
|
||
## lv-mccarthy — the job was protecting a style that reads as damage
|
||
|
||
```
|
||
quote marks 0.0 per 10k (Hemingway 838)
|
||
apostrophes 123 per 10k (Hemingway 241) `dont` `aint` `wont` `didnt`
|
||
semicolons 0 per 10k
|
||
```
|
||
|
||
⚠⚠ **`repair_typography.py` MUST NOT be run on this corpus.** It normalises "toward what the
|
||
text does" and would put the quotation marks back, deleting the single most identifiable
|
||
thing about the author before training starts. The builder runs no normalisation and then
|
||
**asserts** the quote density, so a future well-meaning change fails the build.
|
||
|
||
⚠ **AND IT MAKES THE VOICE GATE EASY TO PASS FOR THE WRONG REASON.** `voice_distance.py` is
|
||
Burrows's Delta over CHARACTER BIGRAMS. An adapter that learns only "emit no quotation marks"
|
||
moves delta_cb a long way without having learned a sentence. **Pre-register a
|
||
punctuation-normalised secondary read before gating lv-mccarthy**, the same way the pairwise
|
||
floor rule was pre-registered for Hemingway.
|
||
|
||
Other measured work: two truncated catalogue rows dropped for complete mobi siblings; all 15
|
||
cross-work containment pairs measured (worst **0.10%**, no subsumption); back matter in 4 of
|
||
6 works, different marker each time, carrying the author's own name 26 times → **0**; one
|
||
lost drop cap (`HE CANDLEFLAME`) patched by name with an assertion rather than a heuristic;
|
||
alphabet re-derived at 1,411 non-ASCII letters across 14 forms (the Border Trilogy is half
|
||
set in Mexico).
|
||
|
||
⚠ **The back-matter strip runs BEFORE the split for McCarthy**, inverting the Hemingway
|
||
order. Blood Meridian and The Crossing end with a dumped table of contents made of bare roman
|
||
numerals on their own lines — the exact shape of a chapter marker. Only the 150-word floor
|
||
saves the current build by accident.
|
||
|
||
## lv-krakauer — builds clean, and the open question is not technical
|
||
|
||
⭐ **THREE DEFECTS THE NAME GUARD CAUGHT**, none of which the build would otherwise report:
|
||
|
||
1. **Back matter searched only the LAST unit.** Where Men Win Glory's ACKNOWLEDGMENTS sits at
|
||
94.8% and the splitter made 41 units, so the apparatus landed in unit **37**, with NOTES
|
||
and BIBLIOGRAPHY after it — all past a strip that only looked at unit 41. Into the Wild
|
||
kept its acknowledgments *and* a full-page advertisement for another of his books.
|
||
2. **Relying on the splitter to drop front matter did not work.** Units begin at the first
|
||
heading mark, and in two works the ebook's table of contents sits above the author's note,
|
||
handing the splitter a `Chapter Thirty-Two` to start on — so unit 1 swallowed the whole
|
||
apparatus including its signed `Jon Krakauer , February 2015`.
|
||
3. ⭐ **Zero was the wrong bar.** 21 survivors became 2, and both were read: `Lewis Krakauer
|
||
loved his five children deeply` is Krakauer writing about his own father in Into the
|
||
Wild's two autobiographical chapters, and the other is a reader's letter he quotes calling
|
||
him a kook. **Hemingway's own name in his corpus was always publisher apparatus, so 0 was
|
||
right there; this author writes about himself.** Pinned at 2, every survivor printed.
|
||
|
||
⚠ Both strips are **windowed in the opposite direction from McCarthy's** — front matter cut
|
||
at the author's signature inside the first 10%, back matter at the earliest apparatus marker
|
||
inside the last 25% — because Krakauer's `ALSO BY` / `Copyright` / `About the Author` sit at
|
||
0.0–0.6% of the file and an earliest-marker rule would cut 99.9% of the book.
|
||
|
||
## ⚠⚠ THE OPEN CALL: an unmeasured fraction of Krakauer is not his prose
|
||
|
||
He quotes constantly and at length — McCandless's journals and letters, Tillman's diaries,
|
||
court transcripts, depositions, nineteenth-century Mormon documents, newspaper reports, and
|
||
whole paragraphs of Jack London, Wallace Stegner and Thoreau at the chapter heads. In print
|
||
those are indented or italic. **The extraction lost both**, so in the master they are
|
||
ordinary paragraphs, indistinguishable from his own sentences by any signal available.
|
||
|
||
Only **52 words** were removable — chapter-head epigraphs whose all-caps attribution line
|
||
survived. That is **0.01%, and it is not the answer**: the method would report 0.0% for a
|
||
book made entirely of undated block quotes. The floor is the finding, not the number.
|
||
|
||
This is the same error as excluding *The Torrents of Spring* from Hemingway — another
|
||
author's style under the target's name — **distributed rather than concentrated**, and the
|
||
fraction is unknown. Scope is the operator's call exactly as fiction-only was. D2 for
|
||
Krakauer is the only thing waiting on it.
|
||
|
||
Related: [[2026-09-17-lv-hemingway-gate]], [[2026-09-16-lv-hemingway-corpus]],
|
||
[[2026-09-17-lv-bronte-gate]].
|