esh-pfi-infrastructure/persistent-memory.d/2026-09-16-lv-hemingway-corpus.md

[2026-09-16] lv-hemingway corpus: half the work was EXCLUSION, and the gate found what a hand count would not

994,760 words · 318 units · 6 renamed copies · leak gate PASSED 0 of 941 entities and 0 of 117 audited phrases, both controls green. ~/hemingway-corpus{,-renamed}, builder scripts/hemingway-corpus/build_corpus_hemingway.py, commits 9598d0b 03b4a3f.

⭐ THE CATALOGUE HOLDS 2,105,679 WORDS AND ROUGHLY HALF MUST NOT BE TRAINED ON. Operator scoped it to fiction only. Three exclusion passes, each measured or voice-specific:

1. Non-fiction, 8 works ~911k words — By-Line, Dateline: Toronto, Death in the Afternoon, Green Hills of Africa, The Dangerous Summer, the three posthumous "Hemingway on X" anthologies.
2. ⭐ Four story collections, 169,759 words — MEASURED, not assumed. Short Stories is the First Forty-Nine and CONTAINS the others. 8-gram containment of the smaller work: Winner Take Nothing 96.0%, Snows of Kilimanjaro 95.2%, Men Without Women 92.9%, In Our Time 90.6%. ⚠⚠ The catalogue's own near_dup_pairs table is BLIND to this — it holds whole-document simhashes (ONE row in the entire 1,284-work library) and this is PARTIAL containment. Whole-document dedup cannot see a collection inside a larger collection.
3. ⭐ The Torrents of Spring — excluded for a reason no word count could justify. It is a deliberate PARODY of Sherwood Anderson: the target author's name on a different author's style, i.e. mislabelled data for a voice adapter.

⚠ THE AUTHOR'S OWN NAME WAS IN THE TRAINING TEXT 95 TIMES ACROSS 7 WORKS — publisher back matter ("Ernest Hemingway was one of America's foremost journalists… died in 1961") riding inside the last unit, because a splitter cuts on headings and nothing follows the final one. Identical to the Yarros defect; nothing about the source changed to cause it. Stripping the publisher block left 18, all in true-at-first-light, inside a CAST OF CHARACTERS and SWAHILI GLOSSARY written by Patrick Hemingway — an editor describing the author's real household. Markers are matched in file order, earliest wins. Now 0.

⚠ G WAS ABOUT TO BE RENAMED TO A SURNAME, 248 TIMES. Not a name: the fragment left by B.G., G.M., G2, G3. Caught by reading surfaces IN CONTEXT, which is the Yarros lesson repeating. Also read in context: Gran (fragment of Gran Sasso/Gran Italia/Gran Hotel), Shamba (Swahili common noun), and Inglés — kept renameable deliberately, the gypsies' in-world nickname for Robert Jordan, exactly parallel to Yarros's Violence.

⭐⭐ THE GENDER RESOLVER HAD TO BE REBUILT AND ITS OWN GATE CAUGHT THE FIRST ATTEMPT. The inherited one returned 397 male / 20 female across 1,102 records with Catherine Barkley, Brett Ashley, Pilar, Maria and Mary all held neutral. A plain majority vote over nearby pronouns scored 18 correct but 5 WRONG against the incumbent's 1 — and every error was female-read-as-male (Pilar m=426 f=243, Brett m=249 f=137). The refuse-unless-better guard rejected it, correctly. ⭐ Cause, measured: the corpus base rate is 34,315 male pronouns to 8,699 female, nearly 4:1. Pilar's "male-dominated" 426:243 is strongly FEMALE against that background. Scoring each name's local mix against the corpus base rate instead of 50:50 gives 18 correct / 11 held / 0 WRONG, distribution 275m / 120f. scripts/hemingway-corpus/gender_by_proximity.py. ⚠ Yarros solved its version with the POV chapter header; Hemingway's editions have none, so that fix does NOT transfer.

⚠ THE ALPHABET DOES NOT TRANSFER EITHER: 1,496 non-ASCII letters across 23 forms against Yarros's 2. Hemingway writes Spanish, French and Italian constantly, so the rename pool needs accents (new hemingway preset in rename.py). The Yarros ASCII-only conclusion would have stranded every Spanish and Italian name in the cast — which is why F02 says re-derive per corpus.

Three source defects the splitter surfaced. Islands in the Stream came out as ONE 143k-word record (roman numerals, unhandled). Short Stories came out as 5 units then 27, because the edition carries a SECOND contents listing and first-occurrence matching resolved 31 of 58 titles to an index entry — keeping every occurrence and letting the word floor decide is self-correcting; now 57. ⚠⚠ And the drop-cap defect is in the HEADINGS here (T HE O LD M AN AND THE S EA), which INVERTS the Yarros pipeline order: repair must run BEFORE the split, or the splitter cannot see the headings it needs.

⭐ Hemingway needed THREE mapped phrases where Yarros needed 48 (Gran Maestro, Unknown Tongue, Sin House) and a 45-entry allow list — the whole difference being that Yarros invented a world and Hemingway named the real one. Several allow entries were non-obvious and required reading: Royal Game is a real colonial-Kenyan legal category, White Heather a Scotch brand, Bwana Game a job title, Roman Soldier/Wine Seller stage-direction labels from the one-act play Today is Friday.