From 0d80e493a8f6cb3dda4838644689445ad35ab397 Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Mon, 21 Sep 2026 15:19:33 -0700 Subject: [PATCH] fix(r49): the quote-mark counter was counting apostrophes, and I saw it fire before I saw the bug MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit voice_distance.py's quote class shipped this morning as "'‘’“”«»‹›‚„` -- with the apostrophe characters in it. On a corpus whose defining tic is dont/aint/wont and whose possessives are everywhere, that made it an apostrophe counter wearing a quote-mark label. as implemented TRUE quotes all apostrophes held-out McCarthy ref 121.1 0.0 121.1 base-unadapted 224.7 19.9 204.8 held-out Hemingway ref 1112.6 694.7 351.7 The corrected column is the one the pre-registration names: 0.0 for McCarthy, which is exactly what build_corpus_mccarthy.py ASSERTS, and 694.7 for Hemingway, the documented ~838 scale the 100-per-10k trigger line was anchored to. The as-implemented column matched neither. ORDER OF EVENTS, because it is the material fact: the base arm finished first, so the trigger became evaluable while the adapted arms were still generating. I evaluated it, saw it FIRE at 224.7, and only then -- reading the reference row of my own table against a corpus I knew asserts 0.0 -- found the bug. No delta_cb, memorisation rate or damage number had been read at any point. Fixing a detector to measure the quantity the frozen rule names is not moving the rule, but the fix un-fires the trigger and no reader should have to take my word about my motives. So GATE-PREREG.md AMENDMENT 2 makes the trigger MOOT instead of adjudicating it: the normalised secondary read is load-bearing UNCONDITIONALLY for this gate, whichever reading you accept, both columns reported. The fix therefore has no effect on the verdict. There is a better reason than the bug anyway: base's true quote density is 19.9 against the reference's 0.0, so it did not fully comply. A small residual cheap win IS available to the adapter, and the normalised read is what prices it. A threshold is a blunt instrument for a residual that size. Apostrophes now get their own column and are never folded into quotes again. Default path stays byte-identical to the shipped lv-hemingway artifact. The durable lesson is the one this line keeps relearning in new places: I controlled strip_punct (2500 -> 0) and the byte-identity of the default path, but never asked the quote counter for a value whose answer I already knew. The corpus asserts 0.0. That check cost one line and was available before the gate ever launched. --- scripts/mccarthy-corpus/GATE-PREREG.md | 66 ++++++++++++++++++++++++++ scripts/r49-corpus/voice_distance.py | 20 ++++++-- 2 files changed, 82 insertions(+), 4 deletions(-) diff --git a/scripts/mccarthy-corpus/GATE-PREREG.md b/scripts/mccarthy-corpus/GATE-PREREG.md index 58db172..10f94ac 100644 --- a/scripts/mccarthy-corpus/GATE-PREREG.md +++ b/scripts/mccarthy-corpus/GATE-PREREG.md @@ -286,3 +286,69 @@ correct one**, in the direction that makes passing harder, on an argument publis four days before this gate existed and independent of any McCarthy number — none of which had been read when this was written. The honest alternative was to run an axis whose control I already knew to be unearned. + +--- + +## AMENDMENT 2 — 2026-09-21, the confound trigger's own instrument was broken + +**Read the order of events before the numbers, because the order is the point.** The base +arm finished first, so §5c's trigger became evaluable while the two adapted arms were still +generating. I evaluated it, **saw it FIRE**, and only then — reading the reference row of my +own table — found that the detector was measuring the wrong thing. No delta_cb, memorisation +rate or damage number had been read at any point. + +### The bug + +`voice_distance.py`'s quote-mark class shipped as ``"'‘’“”«»‹›‚„` `` — it included the +apostrophe characters. So it was an **apostrophe counter wearing a quote-mark label**, on a +corpus whose defining tic is `dont`/`aint`/`wont` and whose possessives are everywhere. + +``` + as implemented TRUE quote marks all apostrophes +held-out McCarthy ref 121.1 0.0 121.1 +base-unadapted 224.7 19.9 204.8 +held-out Hemingway ref 1112.6 694.7 351.7 +``` + +The corrected column is the one the pre-registration names: **0.0 for this corpus**, which is +exactly what `build_corpus_mccarthy.py` asserts, and **694.7 for Hemingway's val split**, +which is the documented ~838 scale the 100-per-10k line was anchored to. The as-implemented +column matched neither, so it was never measuring the quantity §5c specified. + +### The two readings, and why I am not choosing between them + +``` +as implemented (apostrophe-inclusive) base 224.7 > 100 -> TRIGGERED +as specified (quote marks only) base 19.9 < 100 -> does NOT trigger +``` + +Fixing a detector so it measures the quantity the frozen rule names is not moving the rule. +**But I saw the firing value first and the fix un-fires it, which is precisely the shape of +threshold-shopping**, and no reader should have to take my word about my own motives. + +So the trigger is made moot instead of adjudicated: + +> **For this gate the NORMALISED SECONDARY READ (§5b) is load-bearing UNCONDITIONALLY, +> whichever reading of §5c you accept.** Both quote-mark columns are reported. The bug fix +> therefore has no effect on the verdict, and the conservative branch is taken by default. + +There is an argument for that independent of the bug, and it is the better reason: **base's +true quote density is 19.9 against the reference's 0.0, so it did not fully comply.** A small +residual cheap win is genuinely available to the adapter, and the normalised read is what +prices it. A trigger threshold is a blunt instrument for a residual this size; running the +conservative read always is simply better than deciding when to. + +### What changed in the code + +`_QUOTE_RE` is quote marks only. Apostrophes get their own column and are never folded back +in. `--punct-report` now prints quote-marks, all-apostrophes, contraction-apostrophes and +dashes, so a future reader can see the thing that fooled this one. + +### The durable lesson + +This is the same failure the r49 line keeps finding, in a new place: **a detector validated +only against the present case cannot tell a wrong measurement from a right one.** I controlled +`strip_punct` (2500 → 0, it works) and the byte-identity of the default path (it matches), but +I never asked the quote counter for a value I already knew the answer to. The corpus asserts +0.0. Reading 121.1 against a known 0.0 is what exposed it — and that check cost one line and +was available before the gate ever launched. diff --git a/scripts/r49-corpus/voice_distance.py b/scripts/r49-corpus/voice_distance.py index 1e7ac16..8d398c4 100644 --- a/scripts/r49-corpus/voice_distance.py +++ b/scripts/r49-corpus/voice_distance.py @@ -56,9 +56,19 @@ from pathlib import Path # and Hemingway's measures 838; 100 is the order-of-magnitude line between them. PUNCT_CONFOUND_PER_10K = 100.0 -QUOTE_CHARS = "\"'‘’“”«»‹›‚„`" +# ⚠⚠ QUOTE MARKS ONLY -- NO APOSTROPHE CHARACTERS. This class shipped 2026-09-21 with +# `'` and `’` in it, which made it an APOSTROPHE counter wearing a quote-mark label. On +# lv-mccarthy that reported the held-out reference at 121.1 "quote marks" per 10k for a +# corpus whose builder ASSERTS 0.0, and it fired the confound trigger on a base arm whose +# true quote density is 19.9. The pre-registered trigger names quote marks and anchors its +# line to this corpus's 0.0 against Hemingway's 838 -- both quotation-mark counts -- so an +# apostrophe-inclusive class does not measure the quantity the rule names. +# Verified after the fix: McCarthy held-out ref 0.0 (matches the builder's assertion), +# Hemingway held-out ref 694.7 (the documented ~838 scale). See GATE-PREREG.md AMENDMENT 2. +_QUOTE_RE = re.compile('["“”«»‹›„]') _PUNCT_RE = re.compile(r"[^\w\s]|_", re.UNICODE) -_QUOTE_RE = re.compile("[" + re.escape(QUOTE_CHARS) + "]") +# apostrophes are counted SEPARATELY and never folded into the quote column again. +_APOS_RE = re.compile(r"['’‘`]") # a contraction apostrophe is one sitting BETWEEN letters -- `dont` vs `don't` is the tic # the register names, and a possessive or a quote mark is not the same measurement. _CONTRACTION_APOS_RE = re.compile(r"(?<=[A-Za-z])['’](?=[A-Za-z])") @@ -113,13 +123,15 @@ def punct_report(ref_text: str, arms: list[tuple[str, list[dict]]]) -> None: print("\n PUNCTUATION DENSITY per 10k words -- the confound check, not an axis") print(" (the eval harness drives EVERY arm with the same register prompt, tics included;") print(" a compliant base control earns the adapter no delta_cb for them)") - print(f" {'arm':22s} {'quote-marks':>12s} {'contraction-apos':>18s} {'dashes':>9s}") + print(f" {'arm':22s} {'quote-marks':>12s} {'all-apos':>10s} " + f"{'contraction-apos':>18s} {'dashes':>9s}") rows = [("held-out reference", ref_text)] rows += [(a, "\n".join(r["continuation"] for r in recs)) for a, recs in arms] base_q = None for name, txt in rows: q = density(txt, _QUOTE_RE) - print(f" {name:22s} {q:12.1f} {density(txt, _CONTRACTION_APOS_RE):18.1f} " + print(f" {name:22s} {q:12.1f} {density(txt, _APOS_RE):10.1f} " + f"{density(txt, _CONTRACTION_APOS_RE):18.1f} " f"{density(txt, _DASH_RE):9.1f}") if "unadapted" in name: base_q = q