"""D1 for lv-mccarthy: build the corpus from the licensed Kvasir masters. Emits the same record schema as the Brontë, Yarros and Hemingway builders ({work, chapter, heading, words, text} per work file, plus corpus_alphabet.json and manifest.json) so entities.py, rename.py, leak_gate.py and the trainers run unchanged. ⭐ THE WORK THIS BUILDER DOES THAT THE OTHERS DID NOT IS PROTECTING A STYLE THAT LOOKS LIKE DAMAGE. McCarthy uses **no quotation marks at all** and drops the apostrophe from most contractions — `dont`, `aint`, `wont`, `didnt`. Measured across the six works: quotation marks 1 per 10,000 words (Hemingway: 838) apostrophes 123 per 10,000 words (Hemingway: 241) semicolons 0 per 10,000 words Every one of those numbers is the voice. ⚠⚠ `repair_typography.py` normalises "toward what the text does" and MUST NOT be run on this corpus — it would read an unquoted line of dialogue as a defect and put the quotes back, deleting the single most identifiable thing about the author before training ever starts. This builder therefore runs NO typography normalisation, and asserts the quote density afterwards so a future well-meaning change cannot quietly undo it. ⚠ AND IT MAKES THE VOICE GATE HARDER TO READ, WHICH IS A SEPARATE PROBLEM FOR LATER. `voice_distance.py` is Burrows's Delta over CHARACTER BIGRAMS. An adapter that learns only "emit no quotation marks" will move delta_cb a long way without having learned a sentence. Pre-register a punctuation-normalised secondary read before gating this one, or the axis will pass for the least interesting reason available. WHAT WAS EXCLUDED, MEASURED RATHER THAN ASSUMED. 1. TWO TRUNCATED CATALOGUE ROWS. The catalogue holds eight McCarthy rows; two are fragments that carry a real title, a real author and `triage_disposition = accepted`: Blood Meridian [epub] 1,167 words (the book is ~117,000) The Crossing [epub] 222 words (the book is ~150,000) Both hold genuine McCarthy prose, so nothing about the text says "broken" — the epub extraction simply stopped. Each has a complete **mobi** sibling, which is what this builder takes. ⚠ The catalogue's `near_dup_pairs` cannot see this: it holds ONE row in the entire 1,284-work library, and a whole-document simhash cannot match a 222-word fragment to the book it came from. 2. NOTHING ELSE. Cross-work 8-gram containment was measured over all fifteen pairs, on the Hemingway precedent where a collection turned out to hold four other works at 90-96%. Worst pair here is **0.10%** (No Country in All the Pretty Horses). Six independent works; no subsumption, no parody, no non-fiction. BACK MATTER RIDES INSIDE THE LAST UNIT IN FOUR OF SIX WORKS, the same defect Yarros and Hemingway hit, and the marker differs every time: The Crossing `THE END` then a dumped `Table of Contents` Blood Meridian a dumped `Table of Contents` No Country the author's name on its own line, an About-the-Author, then blurbs All the Pretty.. a `Reader's Guide`, promo copy for The Road, then the CIP page ⚠ IT CARRIES THE AUTHOR'S OWN NAME — 19 times in All the Pretty Horses, 7 in No Country — which is exactly the leak the rename pipeline exists to prevent, sitting in the training text before the pipeline runs. Markers are matched in FILE ORDER and the earliest wins. ⚠⚠ AND THE STRIP RUNS **BEFORE** THE SPLIT HERE, INVERTING THE HEMINGWAY ORDER. Blood Meridian and The Crossing end with a dumped table of contents made of bare roman numerals on their own lines — which is the exact shape of a chapter marker. Splitting first would feed the TOC to the splitter as two dozen extra chapters; only the 150-word floor accidentally saves it today. """ from __future__ import annotations import argparse, collections, datetime, json, os, re, sqlite3, sys from pathlib import Path sys.path.insert(0, str(Path(__file__).resolve().parent.parent / "r49-corpus")) from split_units import choose_units, format_report # noqa: E402 CATALOG = "/home/lkraven/development/kvasir/data/library/catalog.sqlite" KVASIR = "/home/lkraven/development/kvasir" # title -> (slug, the format whose extraction is COMPLETE) WORKS = { "All the Pretty Horses": ("all-the-pretty-horses", "epub"), "Blood Meridian Or the Evening Redness in": ("blood-meridian", "mobi"), "Cities of the Plain": ("cities-of-the-plain", "epub"), "No country for old men": ("no-country-for-old-men", "epub"), "The Crossing": ("the-crossing", "mobi"), "The Road": ("the-road", "epub"), } TRUNCATED = { ("Blood Meridian Or the Evening Redness in the West", "epub"): "1,167 words of a ~117,000-word book", ("The Crossing", "epub"): "222 words of a ~150,000-word book", } BACKMATTER = re.compile( r"^[ \t]*(THE END|Table of Contents|TABLE OF CONTENTS|Reader.s Guide|READER.S GUIDE|" r"Reading Group Guide|READING GROUP GUIDE|Questions? (?:for|and) Discussion|" r"Cormac McCarthy|About the Author|ABOUT THE AUTHOR|A NOTE ABOUT THE AUTHOR|" r"Acclaim for|ACCLAIM FOR|Praise for|PRAISE FOR|Also by|ALSO BY|Books by|BOOKS BY|" r"VINTAGE INTERNATIONAL|FIRST VINTAGE|Copyright ©|COPYRIGHT|Library of Congress)" r"[ \t]*.{0,60}$", re.M) # ⚠⚠ THESE EDITIONS SET SECTION OPENINGS IN SMALL CAPS AND THE EXTRACTOR MANGLED THEM THREE # DIFFERENT WAYS. The entity map is what surfaced it: `E`, `H`, `T` and `K` came back as # renameable entities with 17-33 capitalised occurrences each, which is the `G` class from # the Hemingway build — a bare initial is never a name. # # 1. SPLIT INITIAL `T HE HOUSE was built` -> `The house was built` 14 cases # The drop cap survived as its own token. Hemingway's restore_smallcaps only fires on # TWO or more split initials in a line, so it does not see these single ones. # 2. UNMARKED RUN `THEY STOOD in the doorway` -> `They stood in the ...` 88 cases # The small-caps run came through as ordinary capitals with no split at all. Concentrated # in Cities of the Plain (49) and The Crossing (37). # 3. LOST INITIAL `HE CANDLEFLAME` -> `THE CANDLEFLAME` 1 case # The decorative letter was a separate image and is simply gone. # # ⚠⚠ RULE 3 IS ANCHORED TO A BLOCK START, AND THAT IS NOT DECORATION. A second entry was # nearly added here for `HEY RODE` -> `THEY RODE` in Cities of the Plain, because the # opening showed up in a survey of the BUILT corpus. The raw master has `THEY RODE` intact, # twice — `HEY RODE` was matching as a SUBSTRING of the correct text, and an unanchored # replace turned both into `TTHEY RODE`, which rule 2 then lowercased to `Tthey rode`. The # count assertion caught it (expected 1, found 2) and reading the master settled it. The # entry is gone and the survivor is anchored so a substring can never fire it. # # ⚠ THE `[a-z]` ANCHOR ON RULE 2 IS WHAT MAKES IT SAFE. Lowercasing every all-caps run at a # block start would eat a genuine shout or a sign; requiring the run to be followed # immediately by a lowercase word means it is a sentence continuing, which a sign is not. # Checked before shipping: all 23 distinct first words of the 88 are real words (HE, WHEN, # THE, THEY, QUINQUAGESIMA ...) except the one `HEY`, which is case 3. # # ⚠ AND RULE 1 REQUIRES A FOLLOWING ALL-CAPS WORD, because `A TV was playing` and # `A Mexican was changing` are an article plus a capitalised word, not a drop cap. Verified # against all four such probes: `A TV`, `A Mexican`, `A God`, `A Tennessean` are untouched. SPLIT_INITIAL = re.compile(r"\b([A-Z]) ([A-Z]{2,})\b") DROPCAP = re.compile(r"(?m)^([A-Z]) ([A-Z]+(?:[ ][A-Z']{2,})+)") SMALLCAPS_OPENING = re.compile(r"(?m)^([A-Z]{2,}(?:[ ][A-Z']{2,})+)(?=[ ][a-z])") # Case 3 is patched by name with an expected count, never by heuristic — restoring a letter # the extractor deleted means guessing it, and a guess that fires silently is worse than a # missing capital. A master change makes the count wrong and fails the build. LOST_INITIALS = { "all-the-pretty-horses": [(re.compile(r"(?m)^HE CANDLEFLAME"), "THE CANDLEFLAME", 1)], } # ⚠⚠⚠ RULES 4 AND 5 ARE A LEAK FIX, NOT A TYPOGRAPHY FIX, AND THE GATE PASSED WITHOUT THEM. # # `leak_gate.py` scans `\b(Surface)\b`. ANY character inserted inside a name defeats that # pattern outright, so a mangled occurrence is not merely unrepaired -- it is UNRENAMEABLE and # UNREPORTABLE. Measured on the gated 2026-09-17 tree, which had reported "0 of 75 renameable # and 0 of 37 sub-threshold survive in any copy": # # B ell x2 Chigurh cap 119 in the map no-country-for-old-men # C higurh x3 Bell cap 147 no-country-for-old-men # M oss x2 Moss cap 126 no-country-for-old-men # T oadvine x1 Toadvine cap 111 blood-meridian # Toad-vine x3 Glanton cap 365 blood-meridian # Glan-ton x2 # # 13 occurrences of FIVE protagonist names, present in ALL SIX renamed copies, with 0 visible # survivors -- so the gate's verdict was true for the forms it can see and false overall. Every # unmangled occurrence WAS renamed (exact-match survivors: 0), which is what makes the residue # so easy to miss: the names are gone everywhere you look. # # Positive control on the probe that found it: the same separator-tolerant scan over the # UNRENAMED source returns 115-365 hits per name, so it detects these names when present. # Cross-check on the two SHIPPED corpora: lv-bronte is clean of this class; lv-hemingway carries # 3 (`Primi tivo`, `Pasionar ia`, `Chi cote`). `leak_gate.py` now runs the scan itself. # # 4. SPLIT INITIAL, LOWERCASE REMAINDER `S ee the child` -> `See the child` 32 cases # The drop cap survived as its own token and the small-caps remainder came through # LOWERCASE, so rule 1 cannot see it -- rule 1 requires a following ALL-CAPS word, and # DROPCAP requires two. This is the class both of them leave behind. Confined to # blood-meridian (16) and no-country-for-old-men (16); the other four works have none. # # ⚠ THE VOWELS AND `Y` ARE EXCLUDED BECAUSE THEY ARE REAL WORDS AT A SENTENCE START, and # this is the `I`/`A` trap that already bit audit_stoplist.py once. Measured in the built # corpus: `I` opens 1,966 paragraphs (the pronoun), `A` opens 143 (the article, e.g. # `A blivet is ten pounds of shit in a five pound sack.`), and `Y` opens 32 (Spanish `y`, # e.g. `Y de los hombres?`). Folding any of those would corrupt 2,141 lines to fix 32. # The letter class is consonants only, and all 32 survivors were read: the second # fragments are `ell heir he hen hey higurh ith ive oadvine or oss ow e ee hen`. # # ⚠ Runs LAST, after DROPCAP. DROPCAP rewrites `T HE HOUSE was built` to `The house was # built` -- the space between its groups is consumed, not captured -- so its output is # never a rule-4 target. Verified: line-anchored and paragraph-anchored counts are both 32, # i.e. no hit sits mid-paragraph on one of The Road's hard-wrapped lines. # # 5. HYPHEN-SPLIT NAME `Toad-vine` -> `Toadvine` 5 cases # A print edition hyphenated the name across a line break and the extraction kept the # hyphen while dropping the break. Patched BY NAME with an expected count, never by # heuristic: McCarthy writes real hyphenated compounds and a rule broad enough to catch # these would eat them. A master change makes the count wrong and says so. SPLIT_INITIAL_LOWER = re.compile(r"(?m)^([B-DF-HJ-NP-TV-XZ]) ([a-z][a-z']*)") HYPHEN_SPLIT_NAMES = { "blood-meridian": [(re.compile(r"\bToad-vine\b"), "Toadvine", 3), (re.compile(r"\bGlan-ton\b"), "Glanton", 2)], } def restore_smallcaps(line: str) -> str: if len(SPLIT_INITIAL.findall(line)) < 2: return line return SPLIT_INITIAL.sub(lambda m: m.group(1) + m.group(2).lower(), line) def repair_smallcaps(text: str, slug: str) -> tuple[str, dict]: """Run the five repairs in order; the lost initials MUST go first, rule 4 MUST go last.""" stats = {"lost_initial": 0, "split_initial": 0, "unmarked_run": 0, "split_initial_lower": 0, "hyphen_split_name": 0} for pat, good, expect in LOST_INITIALS.get(slug, []): text, n = pat.subn(good, text) if n != expect: print(f" ⚠ {slug}: expected {expect} block-anchored occurrence(s) of " f"{pat.pattern!r} and replaced {n} — the master changed; re-check before " f"trusting this build", file=sys.stderr) stats["lost_initial"] += n text = "\n".join(restore_smallcaps(ln) for ln in text.split("\n")) text, stats["split_initial"] = DROPCAP.subn( lambda m: m.group(1) + m.group(2).lower(), text) text, stats["unmarked_run"] = SMALLCAPS_OPENING.subn( lambda m: m.group(1)[0] + m.group(1)[1:].lower(), text) for pat, good, expect in HYPHEN_SPLIT_NAMES.get(slug, []): text, n = pat.subn(good, text) if n != expect: print(f" ⚠ {slug}: expected {expect} occurrence(s) of {pat.pattern!r} and " f"replaced {n} — the master changed; re-check before trusting this build", file=sys.stderr) stats["hyphen_split_name"] += n text, stats["split_initial_lower"] = SPLIT_INITIAL_LOWER.subn( lambda m: m.group(1) + m.group(2), text) return text, stats def masters(): c = sqlite3.connect(CATALOG) rows = c.execute( "select w.title, w.source_format, m.master_path, m.rights, m.normalized_text_sha256 " "from masters m join works w on w.current_normalized_content_id = " "m.normalized_content_id where w.author = 'Cormac McCarthy'").fetchall() out, seen = [], set() for title, fmt, path, rights, sha in rows: if (title, fmt) in TRUNCATED: print(f" excluded {title[:44]:<46} [{fmt}] — {TRUNCATED[(title, fmt)]}") continue if title not in WORKS or WORKS[title][1] != fmt: continue if title in seen: print(f" ⚠ duplicate row for {title}, keeping the first", file=sys.stderr) continue p = Path(path if os.path.isabs(path) else os.path.join(KVASIR, path)) if not p.exists(): print(f" ⚠ MISSING master for {title}: {p}", file=sys.stderr) continue seen.add(title) out.append({"title": title, "slug": WORKS[title][0], "path": p, "fmt": fmt, "rights": rights, "sha256": sha}) return sorted(out, key=lambda w: w["slug"]) def strip_backmatter(text: str, title: str) -> tuple[str, int]: """Cut at the earliest back-matter marker. Bounded: refuse a strip over 5% of the work.""" m = BACKMATTER.search(text) if not m: return text, 0 cut = text[:m.start()].rstrip() removed = len(text.split()) - len(cut.split()) if removed > 0.05 * len(text.split()): print(f" ⚠ REFUSING back-matter strip on {title}: would remove {removed:,} words " f"({removed/len(text.split()):.0%}) — marker {m.group(1)!r} is probably a false " f"positive inside the novel", file=sys.stderr) return text, 0 return cut, removed def main() -> int: ap = argparse.ArgumentParser() ap.add_argument("--out", required=True) ap.add_argument("--survey", action="store_true", help="report and write nothing") a = ap.parse_args() works = masters() if len(works) != len(WORKS): print(f" ⚠ resolved {len(works)} of {len(WORKS)} works", file=sys.stderr) if not works: raise SystemExit("REFUSING: no McCarthy masters resolved from the catalog") out = Path(a.out) alphabet = collections.Counter() total_words = total_units = 0 own_name_before = own_name_after = 0 manifest = {"corpus": "lv-mccarthy", "author": "Cormac McCarthy", "scope": "all six complete novels in the licensed library", "source": "kvasir data/library masters (licensed, rights=gated)", "built_at": datetime.date.today().isoformat(), "excluded_truncated_rows": {f"{t} [{f}]": why for (t, f), why in TRUNCATED.items()}, "containment_checked": "all 15 pairs, worst 0.10% — no work subsumes another", "typography": ("NO normalisation applied. McCarthy's missing quotation marks " "and apostrophes are the voice, not damage. Do not run " "repair_typography.py on this corpus."), "works": []} for w in works: text = w["path"].read_text(encoding="utf-8", errors="replace") own_name_before += len(re.findall(r"McCarthy", text)) text, back_removed = strip_backmatter(text, w["title"]) text, caps = repair_smallcaps(text, w["slug"]) mode, units, report = choose_units(text) own_name_after += sum(len(re.findall(r"McCarthy", b)) for _, b in units) alphabet.update(ch for ch in text if ch.isalpha()) words = sum(len(b.split()) for _, b in units) total_words += words total_units += len(units) synth = " (synthetic sections — the source has no usable divisions)" \ if mode == "paragraph-blocks" else "" capnote = "".join(f" [{k.replace('_', ' ')} {v}]" for k, v in caps.items() if v) print(f"\n {w['slug']:<24} {len(units):>4} units {words:>8,} words via {mode}{synth}" + (f" [back -{back_removed}w]" if back_removed else "") + capnote) print(format_report(report, mode)) manifest["works"].append({"slug": w["slug"], "title": w["title"], "rights": w["rights"], "source_format": w["fmt"], "master_sha256": w["sha256"], "units": len(units), "words": words, "heading_pattern": mode, "synthetic_sections": mode == "paragraph-blocks", "back_matter_words_removed": back_removed, "smallcaps_repairs": caps, "mode_report": report, "path": f"works/{w['slug']}.jsonl"}) if not a.survey: (out / "works").mkdir(parents=True, exist_ok=True) with (out / "works" / f"{w['slug']}.jsonl").open("w", encoding="utf-8") as fh: for i, (head, body) in enumerate(units, 1): fh.write(json.dumps({"work": w["slug"], "chapter": i, "heading": head, "words": len(body.split()), "text": body}, ensure_ascii=False) + "\n") # ---- GUARDS: assert the things this corpus is unusual about ---------- body = "\n".join(b for w in works for _, b in choose_units(repair_smallcaps( strip_backmatter(w["path"].read_text(encoding="utf-8", errors="replace"), w["title"])[0], w["slug"])[0])[1]) per10k = len(body.split()) / 10000 or 1 quotes = len(re.findall(r'["“”]', body)) / per10k apos = len(re.findall(r"['’]", body)) / per10k print(f"\n STYLE GUARD quotation marks {quotes:.1f}/10k · apostrophes {apos:.0f}/10k") ok_style = quotes < 10 print(f" [{'PASS' if ok_style else 'FAIL'}] the unquoted-dialogue style survived the build " f"(expect <10 quote marks per 10k; Hemingway's corpus reads 838)") print(f" NAME GUARD 'McCarthy' in the raw masters {own_name_before} -> " f"in the built corpus {own_name_after}") ok_name = own_name_after == 0 print(f" [{'PASS' if ok_name else 'FAIL'}] the author's own name is out of the training text") non_ascii = {c: n for c, n in alphabet.items() if ord(c) > 127} print(f"\n TOTAL {total_units} units · {total_words:,} words · {len(alphabet)} distinct letters") print(f" non-ASCII letters: {sum(non_ascii.values())} across {len(non_ascii)} forms") print(f" (Brontë 680,291 · Yarros 780,744 · Hemingway 994,760 for scale)") manifest["totals"] = {"units": total_units, "words": total_words, "distinct_letters": len(alphabet), "non_ascii_letters": sum(non_ascii.values()), "quote_marks_per_10k": round(quotes, 2), "apostrophes_per_10k": round(apos, 1), "author_name_occurrences": own_name_after} manifest["total_words"] = total_words manifest["total_chapters"] = total_units if not a.survey: (out / "manifest.json").write_text(json.dumps(manifest, indent=2), encoding="utf-8") (out / "corpus_alphabet.json").write_text(json.dumps({ "derived_from": "Cormac McCarthy, 6 novels, Kvasir licensed library", "derived_at": manifest["built_at"], "note": ("R49 F02 rule: a rename pool's character inventory must be a SUBSET of " "this. RE-DERIVED on this corpus rather than inherited — measured " f"{sum(non_ascii.values())} non-ASCII letters across {len(non_ascii)} " "forms. McCarthy writes Spanish constantly (the Border Trilogy is half " "set in Mexico), so the Yarros ASCII-only conclusion does NOT transfer."), "letters": sorted(alphabet), "non_ascii": {c: n for c, n in sorted(non_ascii.items())}, }, indent=2, ensure_ascii=False), encoding="utf-8") print(f" wrote {out}") return 0 if (ok_style and ok_name) else 1 if __name__ == "__main__": raise SystemExit(main())