67 lines
5.0 KiB
Markdown
67 lines
5.0 KiB
Markdown
# `[2026-09-16]` lv-hemingway corpus: half the work was EXCLUSION, and the gate found what a hand count would not
|
|
|
|
**994,760 words · 318 units · 6 renamed copies · leak gate PASSED 0 of 941 entities and 0 of 117
|
|
audited phrases, both controls green.** `~/hemingway-corpus{,-renamed}`, builder
|
|
`scripts/hemingway-corpus/build_corpus_hemingway.py`, commits `9598d0b` `03b4a3f`.
|
|
|
|
⭐ **THE CATALOGUE HOLDS 2,105,679 WORDS AND ROUGHLY HALF MUST NOT BE TRAINED ON.** Operator
|
|
scoped it to fiction only. Three exclusion passes, each measured or voice-specific:
|
|
|
|
1. **Non-fiction, 8 works ~911k words** — By-Line, Dateline: Toronto, Death in the Afternoon,
|
|
Green Hills of Africa, The Dangerous Summer, the three posthumous "Hemingway on X" anthologies.
|
|
2. ⭐ **Four story collections, 169,759 words — MEASURED, not assumed.** `Short Stories` is the
|
|
First Forty-Nine and CONTAINS the others. 8-gram containment of the smaller work: Winner Take
|
|
Nothing **96.0%**, Snows of Kilimanjaro **95.2%**, Men Without Women **92.9%**, In Our Time
|
|
**90.6%**. ⚠⚠ **The catalogue's own `near_dup_pairs` table is BLIND to this** — it holds
|
|
whole-document simhashes (ONE row in the entire 1,284-work library) and this is PARTIAL
|
|
containment. Whole-document dedup cannot see a collection inside a larger collection.
|
|
3. ⭐ **`The Torrents of Spring` — excluded for a reason no word count could justify.** It is a
|
|
deliberate PARODY of Sherwood Anderson: the target author's name on a different author's
|
|
style, i.e. mislabelled data for a voice adapter.
|
|
|
|
⚠ **THE AUTHOR'S OWN NAME WAS IN THE TRAINING TEXT 95 TIMES ACROSS 7 WORKS** — publisher back
|
|
matter ("Ernest Hemingway was one of America's foremost journalists… died in 1961") riding inside
|
|
the last unit, because a splitter cuts on headings and nothing follows the final one. **Identical
|
|
to the Yarros defect; nothing about the source changed to cause it.** Stripping the publisher
|
|
block left 18, all in `true-at-first-light`, inside a **CAST OF CHARACTERS and SWAHILI GLOSSARY
|
|
written by Patrick Hemingway** — an editor describing the author's real household. Markers are
|
|
matched in file order, earliest wins. Now 0.
|
|
|
|
⚠ **`G` WAS ABOUT TO BE RENAMED TO A SURNAME, 248 TIMES.** Not a name: the fragment left by
|
|
`B.G.`, `G.M.`, `G2`, `G3`. Caught by reading surfaces IN CONTEXT, which is the Yarros lesson
|
|
repeating. Also read in context: `Gran` (fragment of `Gran Sasso`/`Gran Italia`/`Gran Hotel`),
|
|
`Shamba` (Swahili common noun), and `Inglés` — **kept renameable deliberately**, the gypsies'
|
|
in-world nickname for Robert Jordan, exactly parallel to Yarros's `Violence`.
|
|
|
|
⭐⭐ **THE GENDER RESOLVER HAD TO BE REBUILT AND ITS OWN GATE CAUGHT THE FIRST ATTEMPT.** The
|
|
inherited one returned **397 male / 20 female** across 1,102 records with Catherine Barkley,
|
|
Brett Ashley, Pilar, Maria and Mary all held neutral. A plain majority vote over nearby pronouns
|
|
scored 18 correct but **5 WRONG** against the incumbent's 1 — and **every error was
|
|
female-read-as-male** (Pilar m=426 f=243, Brett m=249 f=137). The refuse-unless-better guard
|
|
rejected it, correctly. ⭐ **Cause, measured: the corpus base rate is 34,315 male pronouns to
|
|
8,699 female, nearly 4:1.** Pilar's "male-dominated" 426:243 is strongly FEMALE against that
|
|
background. Scoring each name's local mix against the corpus base rate instead of 50:50 gives
|
|
**18 correct / 11 held / 0 WRONG**, distribution 275m / 120f.
|
|
`scripts/hemingway-corpus/gender_by_proximity.py`. ⚠ Yarros solved its version with the POV
|
|
chapter header; Hemingway's editions have none, so that fix does NOT transfer.
|
|
|
|
⚠ **THE ALPHABET DOES NOT TRANSFER EITHER: 1,496 non-ASCII letters across 23 forms** against
|
|
Yarros's 2. Hemingway writes Spanish, French and Italian constantly, so the rename pool needs
|
|
accents (new `hemingway` preset in `rename.py`). The Yarros ASCII-only conclusion would have
|
|
stranded every Spanish and Italian name in the cast — which is why F02 says re-derive per corpus.
|
|
|
|
**Three source defects the splitter surfaced.** `Islands in the Stream` came out as ONE
|
|
143k-word record (roman numerals, unhandled). `Short Stories` came out as 5 units then 27,
|
|
because the edition carries a **SECOND contents listing** and first-occurrence matching resolved
|
|
31 of 58 titles to an index entry — keeping every occurrence and letting the word floor decide is
|
|
self-correcting; now 57. ⚠⚠ **And the drop-cap defect is in the HEADINGS here** (`T HE O LD M AN
|
|
AND THE S EA`), which **INVERTS the Yarros pipeline order: repair must run BEFORE the split**, or
|
|
the splitter cannot see the headings it needs.
|
|
|
|
⭐ **Hemingway needed THREE mapped phrases where Yarros needed 48** (`Gran Maestro`,
|
|
`Unknown Tongue`, `Sin House`) and a 45-entry allow list — the whole difference being that Yarros
|
|
invented a world and Hemingway named the real one. Several allow entries were non-obvious and
|
|
required reading: `Royal Game` is a real colonial-Kenyan legal category, `White Heather` a Scotch
|
|
brand, `Bwana Game` a job title, `Roman Soldier`/`Wine Seller` stage-direction labels from the
|
|
one-act play `Today is Friday`.
|