memory: snapshot — McCarthy and Krakauer D1 built, Krakauer's quotation scope is an open operator call
This commit is contained in:
@@ -0,0 +1,120 @@
|
||||
# `[2026-09-17]` McCarthy and Krakauer D1 — both corpora built, and one of them should not be trained yet
|
||||
|
||||
Operator picked **both** after the library survey. McCarthy first, because the splitter work
|
||||
is shared and he is the cleaner author.
|
||||
|
||||
| corpus | units | words | where | guards |
|
||||
|---|---|---|---|---|
|
||||
| **lv-mccarthy** | 167 | **584,756** | `~/mccarthy-corpus` | style ✅ name ✅ |
|
||||
| **lv-krakauer** | 126 | **422,880** | `~/krakauer-corpus` | name ✅ (2 allowed) · quotation ⚠ UNRESOLVED |
|
||||
|
||||
Commits `705fa3a` (splitter) `f3bf3ca` (McCarthy) `4be0630` (Krakauer).
|
||||
|
||||
## The shared blocker: a splitter that chooses by SIZE
|
||||
|
||||
`scripts/r49-corpus/split_units.py`. The inherited rule was "most units above a floor", and
|
||||
it is wrong for any book whose markers are PARTS:
|
||||
|
||||
```
|
||||
Cities of the Plain 4 roman marks -> 4 units, median 22,312w
|
||||
The Crossing 4 roman marks -> 4 units, median 37,310w
|
||||
```
|
||||
|
||||
"Most units" scores 4 over the 1 that finding-nothing gives, so it wins, and the existing
|
||||
guard only fires at exactly one unit. Now: a mode qualifies only if its median unit is inside
|
||||
**[600, 12000]** AND no single unit holds half the work; among qualifying modes **priority**
|
||||
breaks the tie (contents > chapter-word > roman > bare-numeral > caps-title). Works with no
|
||||
divisions fall back to **paragraph-block sections** at ~3,000 words, cut on block boundaries.
|
||||
|
||||
⭐⭐ **BOTH RULES EXIST BECAUSE A CONTROL RUN CAUGHT THEM, and the first version was worse
|
||||
than what it replaced.** Scoring eligible modes by "median closest to target" chose
|
||||
`caps-title` over the real chapters of Hemingway's *True at First Light*:
|
||||
|
||||
```
|
||||
bare-numeral 20 units median 5,337w max 11,155 <- the book's own chapters
|
||||
caps-title 6 units median 777w max 113,886 <- median looked BETTER
|
||||
```
|
||||
|
||||
Five stray all-caps lines gave five tiny units beside **one holding 97% of the book**. A
|
||||
median cannot see that distribution; a max bound can. Controls now green both ways: 8/10
|
||||
Hemingway works reproduce the shipped mode and unit count exactly (the two differences are
|
||||
`contents`, which the harness does not supply, and The Old Man and the Sea, deliberately kept
|
||||
whole), and 40,000 words with no blank lines returns **1 unit** rather than fabricating
|
||||
sections.
|
||||
|
||||
⚠ The Hemingway builder is deliberately NOT repointed at this module. Its corpus is shipped
|
||||
and its sha is pinned by a live adapter.
|
||||
|
||||
## lv-mccarthy — the job was protecting a style that reads as damage
|
||||
|
||||
```
|
||||
quote marks 0.0 per 10k (Hemingway 838)
|
||||
apostrophes 123 per 10k (Hemingway 241) `dont` `aint` `wont` `didnt`
|
||||
semicolons 0 per 10k
|
||||
```
|
||||
|
||||
⚠⚠ **`repair_typography.py` MUST NOT be run on this corpus.** It normalises "toward what the
|
||||
text does" and would put the quotation marks back, deleting the single most identifiable
|
||||
thing about the author before training starts. The builder runs no normalisation and then
|
||||
**asserts** the quote density, so a future well-meaning change fails the build.
|
||||
|
||||
⚠ **AND IT MAKES THE VOICE GATE EASY TO PASS FOR THE WRONG REASON.** `voice_distance.py` is
|
||||
Burrows's Delta over CHARACTER BIGRAMS. An adapter that learns only "emit no quotation marks"
|
||||
moves delta_cb a long way without having learned a sentence. **Pre-register a
|
||||
punctuation-normalised secondary read before gating lv-mccarthy**, the same way the pairwise
|
||||
floor rule was pre-registered for Hemingway.
|
||||
|
||||
Other measured work: two truncated catalogue rows dropped for complete mobi siblings; all 15
|
||||
cross-work containment pairs measured (worst **0.10%**, no subsumption); back matter in 4 of
|
||||
6 works, different marker each time, carrying the author's own name 26 times → **0**; one
|
||||
lost drop cap (`HE CANDLEFLAME`) patched by name with an assertion rather than a heuristic;
|
||||
alphabet re-derived at 1,411 non-ASCII letters across 14 forms (the Border Trilogy is half
|
||||
set in Mexico).
|
||||
|
||||
⚠ **The back-matter strip runs BEFORE the split for McCarthy**, inverting the Hemingway
|
||||
order. Blood Meridian and The Crossing end with a dumped table of contents made of bare roman
|
||||
numerals on their own lines — the exact shape of a chapter marker. Only the 150-word floor
|
||||
saves the current build by accident.
|
||||
|
||||
## lv-krakauer — builds clean, and the open question is not technical
|
||||
|
||||
⭐ **THREE DEFECTS THE NAME GUARD CAUGHT**, none of which the build would otherwise report:
|
||||
|
||||
1. **Back matter searched only the LAST unit.** Where Men Win Glory's ACKNOWLEDGMENTS sits at
|
||||
94.8% and the splitter made 41 units, so the apparatus landed in unit **37**, with NOTES
|
||||
and BIBLIOGRAPHY after it — all past a strip that only looked at unit 41. Into the Wild
|
||||
kept its acknowledgments *and* a full-page advertisement for another of his books.
|
||||
2. **Relying on the splitter to drop front matter did not work.** Units begin at the first
|
||||
heading mark, and in two works the ebook's table of contents sits above the author's note,
|
||||
handing the splitter a `Chapter Thirty-Two` to start on — so unit 1 swallowed the whole
|
||||
apparatus including its signed `Jon Krakauer , February 2015`.
|
||||
3. ⭐ **Zero was the wrong bar.** 21 survivors became 2, and both were read: `Lewis Krakauer
|
||||
loved his five children deeply` is Krakauer writing about his own father in Into the
|
||||
Wild's two autobiographical chapters, and the other is a reader's letter he quotes calling
|
||||
him a kook. **Hemingway's own name in his corpus was always publisher apparatus, so 0 was
|
||||
right there; this author writes about himself.** Pinned at 2, every survivor printed.
|
||||
|
||||
⚠ Both strips are **windowed in the opposite direction from McCarthy's** — front matter cut
|
||||
at the author's signature inside the first 10%, back matter at the earliest apparatus marker
|
||||
inside the last 25% — because Krakauer's `ALSO BY` / `Copyright` / `About the Author` sit at
|
||||
0.0–0.6% of the file and an earliest-marker rule would cut 99.9% of the book.
|
||||
|
||||
## ⚠⚠ THE OPEN CALL: an unmeasured fraction of Krakauer is not his prose
|
||||
|
||||
He quotes constantly and at length — McCandless's journals and letters, Tillman's diaries,
|
||||
court transcripts, depositions, nineteenth-century Mormon documents, newspaper reports, and
|
||||
whole paragraphs of Jack London, Wallace Stegner and Thoreau at the chapter heads. In print
|
||||
those are indented or italic. **The extraction lost both**, so in the master they are
|
||||
ordinary paragraphs, indistinguishable from his own sentences by any signal available.
|
||||
|
||||
Only **52 words** were removable — chapter-head epigraphs whose all-caps attribution line
|
||||
survived. That is **0.01%, and it is not the answer**: the method would report 0.0% for a
|
||||
book made entirely of undated block quotes. The floor is the finding, not the number.
|
||||
|
||||
This is the same error as excluding *The Torrents of Spring* from Hemingway — another
|
||||
author's style under the target's name — **distributed rather than concentrated**, and the
|
||||
fraction is unknown. Scope is the operator's call exactly as fiction-only was. D2 for
|
||||
Krakauer is the only thing waiting on it.
|
||||
|
||||
Related: [[2026-09-17-lv-hemingway-gate]], [[2026-09-16-lv-hemingway-corpus]],
|
||||
[[2026-09-17-lv-bronte-gate]].
|
||||
Reference in New Issue
Block a user