vh
5aa10bf138
Entity map at ~/mccarthy-corpus/entities.json. 123 surfaces after a 107-surface stoplist.
entities.py 27/27 controls -- 19 positive (Glanton, Toadvine, Rawlins, Blevins,
Alejandra, Chigurh, Moss, Bell, Boyd, Holden, Tobin, Magdalena,
Eduardo, Parham, Socorro, Webster, Redbo, Niño, Franklin) and 8 negative
audit_stoplist PASS -- no stoplisted surface is ever addressed as a person
audit_entity_map PASS -- positive `boy` 0.89, negative band tops out at Riddle 0.17,
all 5 remaining flags on the read-and-cleared list
⚠⚠ A DEFECT IN audit_stoplist.py ITSELF, latent for every corpus before this one. It built
its surface set from every list value in the stoplist JSON -- including `_why`, which by
convention is a LIST OF PROSE LINES. Every sentence of the rationale went into the matcher,
and the empty separator line matched the honorific pattern 139 times, printing a flag with no
surface name at the top of the report, above the one real catch. It now skips `_`-prefixed
metadata keys and empty strings.
THE ONE REAL CATCH WAS A CONTRADICTION INSIDE MY OWN FILE. `Franklin` sat in the geography
list because it is the old name for El Paso, while the same file's context note recorded
'I'm here to see Mr Franklin' -- a lawyer in All the Pretty Horses. The honorific audit found
the contradiction between the two halves of the file. Franklin is now renameable.
A SECOND SELF-INFLICTED ONE: the fragments list was a speculative A-Z, which stoplisted `I`
and `A` -- ordinary English words -- and `Sir I dont think I can do that` duly tripped the
honorific audit. It is now the four letters actually MEASURED as entities (E, H, T, K).
Stoplist what the entity map produced, not the alphabet.
Everything ambiguous was read in context before placement, and the reasoning is in the file:
Socorro is the ranch COOK in Cities of the Plain, not the New Mexico town -- renameable
Webster, Jackson, Harlan, Lamar are Glanton's men and lawmen, not places -- renameable
Niño, Keno, Redbo are HORSES, the author's inventions -- renameable, the `Inglés` precedent
Mangas, Travis, Venada, Moderno are genuinely dual-use -- renamed, the safe direction
Santa, Varas, Griffin, Eagle, Avenue, Calle, Terrell are real geography -- stoplisted
Yaqui and Gilenos are real peoples; Ford and Hashknives are a brand and a real outfit
Ed (Ed Tom Bell) and JC are short but are names, read and kept renameable
Sensitivity floor, stated because it is part of the result: the top 170 of 199 surfaces were
classified. The bottom 29 were not individually read, so a rare real-world referent may be
renamed -- the safe direction, an accepted cost, not an oversight.
97 lines
4.6 KiB
Python
97 lines
4.6 KiB
Python
"""Audit a stoplist for entries that are actually CHARACTERS.
|
|
|
|
⭐ WHY THIS IS A SEPARATE INSTRUMENT. A stoplist entry is an assertion the leak gate
|
|
can no longer check. Stoplisting a surface removes it from the entity map, so rename
|
|
never touches it and the gate never scans for it — which is exactly what a stoplist is
|
|
FOR when the surface is a real-world referent, and exactly how a wrongly stoplisted
|
|
CHARACTER becomes an undetectable leak. The gate will report 0 of N surviving and be
|
|
telling the truth about the set it was given.
|
|
|
|
Found on lv-bronte 2026-09-16, and found by luck: a generated beat said "Mrs. Leaven",
|
|
and `Leaven` had been filed under scripture (the bread noun). Reading it back:
|
|
"Robert Leaven, the coachman" — Bessie's married surname. Running this audit instead of
|
|
trusting the luck then caught two more, `Pierrot` ("Madame Pierrot: she comes from
|
|
Lisle") and `Samuel` ("Mr. Samuel Wynne"), and correctly cleared `Wellington`
|
|
("that Baal of a Lord Wellington" — the real Duke).
|
|
|
|
THE SIGNAL: an honorific in front of it. Real-world referents are not addressed as
|
|
Mr/Mrs/Miss/Madame/Lord. It is a heuristic, not a proof — `Lord Wellington` is a real
|
|
person and `Rev. Moses Barraclough` is a genuine dual-use — so every hit is REPORTED
|
|
FOR READING, never auto-removed. Deciding what a surface is requires reading it in
|
|
context, which is the lesson this whole line keeps relearning.
|
|
|
|
Exit 1 when anything is flagged, so it can gate a pipeline: a clean run is silence.
|
|
"""
|
|
from __future__ import annotations
|
|
import argparse, json, re
|
|
from pathlib import Path
|
|
|
|
HONORIFIC = (r"(?:Mr|Mrs|Miss|Misses|Madame|Mdlle|Mademoiselle|Monsieur|Lord|Lady|Sir|Dr"
|
|
r"|Doctor|Captain|Colonel|Major|General|Aunt|Uncle|Rev|Reverend|Professor"
|
|
r"|Master|Saint|St)\.?\s+")
|
|
|
|
|
|
def load_text(corpus: Path) -> str:
|
|
man = json.loads((corpus / "manifest.json").read_text())
|
|
parts = []
|
|
for w in man["works"]:
|
|
for line in (corpus / w["path"]).read_text(encoding="utf-8").splitlines():
|
|
if line.strip():
|
|
parts.append(json.loads(line)["text"])
|
|
return "\n\n".join(parts)
|
|
|
|
|
|
def main() -> int:
|
|
ap = argparse.ArgumentParser()
|
|
ap.add_argument("corpus")
|
|
ap.add_argument("--stoplist", required=True)
|
|
ap.add_argument("--allow", default="",
|
|
help="comma-separated surfaces already READ and confirmed real-world "
|
|
"or accepted dual-use; they are still shown, but do not fail the run")
|
|
ap.add_argument("--context", type=int, default=46)
|
|
a = ap.parse_args()
|
|
|
|
text = load_text(Path(a.corpus))
|
|
blob = json.loads(Path(a.stoplist).read_text())
|
|
# ⚠ SKIP `_`-PREFIXED KEYS. By convention a stoplist carries its rationale in `_why` and
|
|
# `_caught_by_reading_in_context`, and `_why` is a LIST OF PROSE LINES. Scanning it as
|
|
# surfaces put every sentence of the rationale into the matcher, and its empty separator
|
|
# line matched the honorific pattern 139 times on McCarthy -- a flag with no surface name
|
|
# attached, at the top of the report, above the one real catch. A metadata key is not a
|
|
# surface set; found 2026-09-17 and latent for every corpus before it.
|
|
surfaces = sorted({s for k, v in blob.items() if not k.startswith("_")
|
|
and isinstance(v, list) for s in v if s})
|
|
cleared = {s.strip() for s in a.allow.split(",") if s.strip()}
|
|
|
|
flagged = []
|
|
for s in surfaces:
|
|
pat = re.compile(HONORIFIC + re.escape(s) + r"\b")
|
|
hits = list(pat.finditer(text))
|
|
if hits:
|
|
flagged.append((len(hits), s, hits[0]))
|
|
|
|
print(f" {len(surfaces)} stoplisted surfaces scanned for a preceding honorific")
|
|
if not flagged:
|
|
print(" [PASS] none of them is ever addressed as a person")
|
|
return 0
|
|
|
|
unresolved = 0
|
|
for n, s, m in sorted(flagged, reverse=True):
|
|
mark = "cleared" if s in cleared else "⚠ READ THIS"
|
|
if s not in cleared:
|
|
unresolved += 1
|
|
ctx = text[max(0, m.start() - a.context): m.end() + a.context].replace("\n", " ")
|
|
print(f" {mark:<12} {s:<16} {n:>4} honorific hits …{ctx}…")
|
|
|
|
if unresolved:
|
|
print(f"\n== {unresolved} stoplisted surface(s) look like PEOPLE and are not on --allow.")
|
|
print(" Read each in context. If it is a character, take it OUT of the stoplist and")
|
|
print(" re-run the rename — the leak gate cannot see it while it is stoplisted.")
|
|
return 1
|
|
print("\n [PASS] every honorific hit is on the read-and-cleared list")
|
|
return 0
|
|
|
|
|
|
if __name__ == "__main__":
|
|
raise SystemExit(main())
|