From 9c8a4e959cbb2799718a54b4481b774f5d16e7c2 Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Mon, 21 Sep 2026 14:39:10 -0700 Subject: [PATCH] feat(lv-mccarthy): pre-register the v2 gate before any arm is generated Freezes the lv-mccarthy gate design while no McCarthy generation exists, per the lv-hemingway precedent: a threshold chosen after seeing the numbers is not a threshold. Three arms (base-unadapted, ckpt900, ckpt450), 60 beats, 4 seeds. Settles the voice-axis question D1 deferred. McCarthy's corpus measures 0.0 quote marks per 10k against Hemingway's 838, so "emit no quotation marks" is the cheapest way to move a char-bigram Delta without learning a sentence. Three parts: PRIMARY unchanged -- the mccarthy register names the punctuation and --system-from drives the base control with the same prompt, so the cheap win is handed to both sides. SECONDARY voice_distance.py --secondary-normalised re-runs the whole analysis with punctuation stripped from the reference and every arm. A conservative lower bound; reported, never the verdict. TRIGGER voice_distance.py --punct-report evaluates a pre-registered line -- base-unadapted quote density above 100 per 10k means the control did not take the win it was handed, and the normalised read is promoted to load-bearing. 100 is the order-of-magnitude line between this corpus's 0.0 and Hemingway's 838, chosen now rather than after the table prints. ckpt450 is NOT tied with the minimum (+4.9x the 0.00393 median neighbour jitter) and the pre-registration says so. It is generated to test a stated prior and to price the memorisation headroom an earlier checkpoint buys on an in-copyright author, with a decision rule that says exactly what result would let it displace ckpt900. Instrument controls, run before this landed: - the voice_distance.py refactor reproduces the shipped lv-hemingway voice_distance.txt BYTE FOR BYTE on the default path - strip_punct drops a quote-bearing sample from 2500.0 to 0.0 marks per 10k - the secondary read still resolves a gap on the Hemingway arms (+0.410 at 7.3x floor), so a null on McCarthy would be a finding, not a blind detector Pre-flight re-run at gate time rather than quoted from 2026-09-17: - leak gate: 0 of 75 renameable, 0 of 37 sub-threshold, 0 separator-split, four controls green - beat-contamination audit: 0 of 3942 beats AND 0 of 3942 responses, against Hemingway's 70 of 7094 -- build_sft_pairs.py --source-entities earned its mandatory flag Also records three provenance defects found on first read of the run and their disposition: the hardcoded "r49-babyyarros-pairs-pilot" run label (cosmetic, same literal on all three runs), the empty harness_commit (all three runs), and a pairs_sha256_16 that is not a sha256sum of the file (consistent across runs, so a cache key rather than a fault). The run is bound to McCarthy's pairs by record count, not by the label. --- scripts/mccarthy-corpus/GATE-PREREG.md | 217 +++++++++++++++++++++++ scripts/mccarthy-corpus/eval-mccarthy.sh | 112 ++++++++++++ scripts/mccarthy-corpus/voice-prep.py | 49 +++++ scripts/r49-corpus/voice_distance.py | 195 ++++++++++++++++---- 4 files changed, 538 insertions(+), 35 deletions(-) create mode 100644 scripts/mccarthy-corpus/GATE-PREREG.md create mode 100755 scripts/mccarthy-corpus/eval-mccarthy.sh create mode 100644 scripts/mccarthy-corpus/voice-prep.py diff --git a/scripts/mccarthy-corpus/GATE-PREREG.md b/scripts/mccarthy-corpus/GATE-PREREG.md new file mode 100644 index 0000000..f253331 --- /dev/null +++ b/scripts/mccarthy-corpus/GATE-PREREG.md @@ -0,0 +1,217 @@ +# lv-mccarthy v2 gate — pre-registration + +**Written 2026-09-21, before a single generation existed.** Everything below is +fixed. The point of writing it down is that a threshold chosen after seeing the +numbers is not a threshold. This is the third pre-registration in the line and it +inherits the lv-hemingway design deliberately, so the two runs stay comparable; +every place it departs is marked and argued. + +Nothing here may be amended after the first arm is generated. If the design turns +out to be wrong, the run is wrong and gets re-run — the rule does not move. + +--- + +## 1. What is being gated + +`gx10:~/r49-runs/mccarthy-4b-pairs-3ep/` — Qwen3-4B-Instruct + instruction-pair +LoRA on the renamed McCarthy corpus (584,684 words, 167 units, 6 works, 6 renamed +copies). 3-epoch schedule, seed 4919, rank 32 / alpha 64, lr 1e-4, seq 1536, +1,380 steps in 2h30m47s, finished **2026-09-18 00:49 PT**. `train_loss` 2.172, +end-of-run `eval_loss` 2.4594. + +**The run outcome was unverified for three days** and this gate is the first read +of it. That is recorded here rather than quietly fixed, because "the run finished" +and "the run did what was configured" are different claims and only the second one +is worth anything. + +### 1a. Provenance defects found on first read, and their disposition + +| found | disposition | +|---|---| +| `provenance.json` `"run": "r49-babyyarros-pairs-pilot"` | **Cosmetic, not a mis-copy.** The same literal appears in the yarros AND hemingway provenance — it is a hardcoded label in `train_pairs_lora.py`, not a stale field for this run. Fix the harness; it invalidates nothing. | +| `harness_commit: ""` | **Empty on all three runs.** The training harness never recorded its own commit. No claim in this gate rests on it. Named here so it is not re-discovered as news. | +| `pairs_sha256_16` does not match `sha256sum` of the pairs file | **Not a defect.** It mismatches identically on yarros and hemingway too, so the harness computes it over the loaded records rather than the raw bytes. It is a consistent, per-corpus-unique cache key. | +| `pairs` recorded as the relative `pairs/pairs-full.jsonl` | Resolved against the launch CWD `~/lv-mccarthy`. **Bound by count instead of by path**: `train_pairs` 3673 and `val_pairs_n` 269 match `lv-mccarthy/pairs/` exactly and match no other pair set on the box. | + +The run trained on McCarthy's pairs. That is established by the record counts and +by `system_prompt_source: "pairs provenance (mccarthy)"`, not by the `run` label. + +## 2. Arms — three, and the third is here for a different reason than Hemingway's was + +| arm | what it is | role | +|---|---|---| +| `base-unadapted` | `~/carriers/Qwen3-4B-Instruct`, no adapter | negative control for memorisation (never saw the corpus) **and** the voice baseline | +| `ckpt900` | epoch 1.958, eval_loss **2.38706** — the curve's minimum | ship candidate | +| `ckpt450` | epoch 0.980, eval_loss **2.4063** | **prior test**, see below | + +Measured on this run's own 56 eval points: **median neighbour jitter 0.00393**. + +⚠ **`ckpt450` IS NOT TIED, and saying so is the point.** On Hemingway the second +arm was in the run because the loss curve could not separate it (+0.9× jitter). +Here it can: `ckpt450` sits **+0.0193 = 4.9× jitter** above the minimum. Three +checkpoints *are* inside one jitter of the best — 875 (+0.4×), 850 (+1.1×), 925 +(+1.5×) — and the choice of 900 among them is arbitrary and declared arbitrary; +they are the same point on the curve. `ckpt450` is not one of them. + +So it is generated for a stated, falsifiable reason rather than as a coin-flip: + +- **The prior it tests.** On Brontë the earlier epoch-1 checkpoint won the tiebreak + on the axes that resolve. On Hemingway the pair was tied and the question stayed + open. **McCarthy is the first run in the line whose loss curve says epoch 2 is + genuinely better than epoch 1.** That is a claim the axes can check, and checking + it costs one arm. +- **The thing it would buy.** Axis B (memorisation) carries more weight here than on + any previous corpus — Brontë is public domain, Hemingway is not, and **McCarthy is + in copyright with a living estate.** An earlier checkpoint is the cheap way to buy + a lower verbatim-overlap rate. Whether it actually does is measured below, not + assumed. + +**`adapter/` (end of run, epoch 3.0, 2.4594) is NOT gated.** At **+18.4× jitter** +it is resolved by the loss curve alone — and note the curve does not drift there, +it *steps*: 2.393 at step 925, 2.457 at step 950, and flat for the remaining 430 +steps. The epoch-3 collapse reproduces the Hemingway finding on a second corpus. + +⚠ The shipped `adapter/` directory is the epoch-3 weights. **Whatever ships is a +checkpoint, not `adapter/`.** + +## 3. Fixture and seeds — fixed now + +- **60 beats**, sampled by `build_beat_fixture.py` from + `~/lv-mccarthy/pairs/pairs-val.jsonl` (held-out `split=val`, 269 pairs), `--seed 4919`, + spread across works round-robin. +- **`--max-words 140`, not the script's default 150.** McCarthy's register asks for + 90–140 words and `score_beats.py` scores the in-band rate at 90–140. A fixture built + at 90–150 would put reference passages outside the band the product asks for and the + scorer measures. **207 of 269 val pairs are in-band**, across all six works + (blood-meridian 51, all-the-pretty-horses 40, the-road 38, cities-of-the-plain 33, + no-country-for-old-men 28, the-crossing 17). +- **4 seeds: 1234 5678 9012 3456.** Same as lv-bronte and lv-hemingway. +- → **240 generations per arm, 720 total.** + +60 is carried over from Hemingway unchanged so the two floors are comparable; 207 +in-band pairs make it available with room to spare (Hemingway had 173, Brontë 44). + +## 4. The floor rule — inherited, unchanged + +**The noise floor for a candidate is the larger of the within-arm seed spreads of +the TWO arms being compared: `max(spread(candidate), spread(base-unadapted))`.** + +Settled at lv-hemingway and not reopened. `voice_distance.py` prints both this and +lv-bronte's all-arms floor and flags any case where the two rules disagree; the +pairwise number is the verdict. + +## 5. ⭐ THE VOICE AXIS — the open question, settled here before any number exists + +McCarthy's corpus measures **0.0 quote marks per 10k words** against Hemingway's +838, and `voice_distance.py` is Burrows's Delta over **character bigrams**. "Emit no +quotation marks" is therefore the single cheapest available way to move this metric +a long way without having learned a sentence. lv-mccarthy D1 flagged this and +deferred it; it is settled now, in three parts. + +**5a. PRIMARY read — unchanged, and already hardened.** The `mccarthy` register +names the punctuation on purpose (`build_sft_pairs.py` REGISTERS, commit `f3bf3ca`), +and `eval-mccarthy.sh` drives **every** arm including `base-unadapted` with that same +system prompt via `--system-from`. The control is therefore *told* to drop quote marks +and write `dont`/`aint`/`wont`. The cheap win is handed to both sides and the adapter +earns no delta for it. This was decided when the register was written, before any +McCarthy number existed, and is not reopened here. **Stated cost: the voice axis is +harder for this corpus than for Hemingway's, deliberately.** + +**5b. SECONDARY read — pre-registered, REPORTED, never the verdict.** +`voice_distance.py --secondary-normalised` re-runs the entire analysis with all +punctuation stripped from the reference and from every arm. A gap that survives is +carried by words rather than by marks. + +⚠ It is a **conservative lower bound, not a better measurement**: stripping terminal +punctuation also strips sentence-length signal the adapter legitimately learned, and +sentence length is a real McCarthy trait. It reads as *"at least this much of the +primary gap is not the punctuation trick."* **It cannot overturn the primary verdict.** + +**5c. THE CONFOUND TRIGGER — the part that makes 5a falsifiable.** 5a's whole +argument is that the base control *complies* with the instruction. That is an +assumption about a model's behaviour, and it is checkable. `--punct-report` prints +per-arm quote-mark, contraction-apostrophe and dash density against the reference, +and evaluates this line: + +> **If `base-unadapted` quote density exceeds 100 marks per 10k words, the control +> did not take the win it was handed, part of the primary gap IS the punctuation +> trick, and the normalised secondary read (5b) is promoted to load-bearing for the +> verdict.** + +100 is the order-of-magnitude line between this corpus's 0.0 and Hemingway's 838, +chosen now rather than after the table prints. + +**Instrument controls, run before this was committed.** The `voice_distance.py` +refactor that added 5b and 5c reproduces the shipped lv-hemingway +`voice_distance.txt` **byte for byte** on the default path (positive control: the +edit did not move an existing record). `strip_punct` drops a quote-bearing sample's +density from 2500.0 to **0.0** per 10k (negative control: the normalisation is not +a no-op). On the Hemingway arms the secondary read still resolves a gap +(ckpt1750 +0.410 at 7.3× floor) rather than flattening everything — so a null +result from 5b on McCarthy would be a finding about McCarthy, not a blind detector. + +## 6. The three axes, and what passes + +| axis | instrument | passes if | +|---|---|---| +| **A. VOICE** | `voice_distance.py --author McCarthy --punct-report --secondary-normalised` | `delta_cb(base) − delta_cb(candidate)` **exceeds the pairwise floor** on the PRIMARY read, unless 5c fires, in which case on the NORMALISED read | +| **B. NOT COPIED** | `memorization_check.py` (longest verbatim n-gram vs the renamed copies, n=8) | hit-rate and max at or near `base-unadapted`, which never saw the corpus. Positive control (corpus vs itself) must saturate, or the zeroes are blind | +| **C. NO DAMAGE** | `score_beats.py --metric-source raw` | in-band up on base beyond the metric's own floor; on-beat and ran-on not worse beyond it | + +## 7. The decision rule — frozen + +1. **`ckpt900` is the default ship candidate.** The loss curve resolves it above + `ckpt450` at 4.9× jitter. +2. **`ckpt450` displaces it only if BOTH hold:** it is *not resolvably worse* on + axis A (`gap_450 ≥ gap_900 − pairwise_floor`) **AND** it is *strictly better* on + axis B by more than the base-to-candidate separation on that axis. Buying + memorisation headroom is worth it; buying it by giving up the voice the adapter + exists for is not. +3. **Axis C failure disqualifies that candidate outright**, whatever axis A says. +4. **If both candidates fail axis A** (gap inside the pairwise floor), the result is + a **negative reported with its sensitivity floor**, and the ship-or-park call goes + to the operator. The lv-bronte precedent — fail on A, clean on B and C, shipped + anyway as additive and reversible — **does not transfer automatically**: Brontë is + public domain and McCarthy is not, so axis B carries more weight in that + conversation, not less. +5. **Nothing ships on the strength of the epoch-3 `adapter/`.** It is not gated. + +## 8. Pre-flight — already run, and clean + +Both re-run at gate time rather than quoted from the 2026-09-17 session, because +"it passed three days ago" is a claim about a tree that may have moved. + +**Leak gate** (`leak_gate.py --min-cap 8`, report `leak_gate_report.gate-time.json`): + +``` +0 of 75 renameable · 0 of 37 sub-threshold · 0 separator-split survivors +positive control 108/108 surfaces found in the unrenamed source +negative control nonce absent · split-scan controls: planted `W olfenbarger` detected +``` + +**Beat-contamination audit** (`audit_pairs_sourcenames.py --min-cap 5`, mirroring +the floor `rename.py` actually ran at), over **all 3,942 built pairs**: + +``` +BEAT naming a removed surface: 0 of 3942 (0.00%) +RESPONSE naming a removed surface: 0 of 3942 (0.00%) +controls green: 108/108 found in source · nonce absent from both trees +``` + +⭐ **Zero, where Hemingway carried 70 of 7,094 contaminated beats (0.96%).** That is +`build_sft_pairs.py --source-entities`, which was mandatory for this corpus's D4 and +which Hemingway's pairs predate. The gate is unconfounded on both halves of the pair, +not just the fixture half. + +**Sensitivity floor, stated as part of the pre-flight rather than as a footnote:** a +name appearing fewer than 8 times per work is never detected and a phrase recurring +fewer than 5 times is never audited — and **no phrase map exists for this corpus, so +the phrase audit does not run at all.** Yarros and Brontë both had one. A clean leak +gate here is a narrower claim than a clean leak gate there. + +## 9. Reproduce + +```bash +# on pfi-gx10, ~/lv-mccarthy +./eval-mccarthy.sh # three arms, 60 beats, 4 seeds, all three axes +``` diff --git a/scripts/mccarthy-corpus/eval-mccarthy.sh b/scripts/mccarthy-corpus/eval-mccarthy.sh new file mode 100755 index 0000000..085b1ca --- /dev/null +++ b/scripts/mccarthy-corpus/eval-mccarthy.sh @@ -0,0 +1,112 @@ +#!/bin/bash +# lv-mccarthy v2 gate. Design is FROZEN in scripts/mccarthy-corpus/GATE-PREREG.md +# and was written before this script ever ran. Do not edit the arms, the fixture +# size or the seeds to chase a result -- re-run, do not re-tune. +# +# THREE ARMS. ckpt900 is the eval-loss minimum (2.38706, epoch 1.958). ckpt450 +# (2.4063, epoch 0.980) is +0.0193 against a 0.00393 median neighbour jitter = +# 4.9x, i.e. NOT tied -- and that is the difference from the Hemingway gate, where +# the second arm was a coin-flip. It is here to test a stated prior (on Bronte the +# earlier epoch-1 checkpoint won on the axes that resolve) and to price the +# memorisation headroom an earlier checkpoint would buy on an IN-COPYRIGHT author. +# `base` is the negative control for memorisation and the voice baseline. +# adapter/ (epoch 3.0, +18.4x jitter) is NOT gated: the loss curve settles that one. +set -o pipefail +cd ~/lv-mccarthy || exit 1 +PY=/home/infra-ops/ml/.venv/bin/python +RUN=~/r49-runs/mccarthy-4b-pairs-3ep +OUT=~/r49-runs/mccarthy-eval +BEATS=beats-mccarthy-60.json +SIDE=beats-mccarthy-60.sidecar.json +PROV=pairs/pairs-full.jsonl.provenance.json +SEEDS="1234 5678 9012 3456" +mkdir -p "$OUT" +log(){ echo "[eval $(date +%H:%M:%S)] $*"; } + +# --system-from is MANDATORY and it is load-bearing for the VOICE axis here, not +# merely hygienic. The mccarthy register NAMES the punctuation (no quote marks, +# `dont`/`aint`/`wont`), so driving the base control with the same prompt hands the +# cheap char-bigram win to BOTH sides and the adapter earns no delta for it. See +# GATE-PREREG.md section 5. The harness's built-in SYS is Yarros's; using it would +# confound the adapter change with a prompt change AND hand the adapter the win. +[ -f "$PROV" ] || { log "MISSING $PROV"; exit 1; } + +# The run must have trained under the same system prompt the eval drives. Asserted, +# not assumed -- the Hemingway script claimed this was "verified" in a comment, which +# is a claim no reader can check. Here it fails the run. +"$PY" - "$PROV" "$RUN/provenance.json" <<'PYCHK' || exit 1 +import json, sys +pairs = json.load(open(sys.argv[1])) +run = json.load(open(sys.argv[2])) +ps = pairs.get("system_prompt") or (pairs.get("register") or {}).get("system_prompt") +rs = run.get("system_prompt") +if ps is None: + print(f"== CANNOT VERIFY: no system_prompt in {sys.argv[1]}"); sys.exit(1) +if ps != rs: + print("== REFUSING: the run trained under a DIFFERENT system prompt than the eval would drive.") + print(f" pairs provenance: {ps[:120]!r}") + print(f" run provenance : {(rs or '')[:120]!r}") + sys.exit(1) +print(f" [PASS] run system_prompt == pairs system_prompt ({len(ps)} chars)") +PYCHK + +# 60 beats, max-words 140: the mccarthy register asks for 90-140 and score_beats.py +# scores the in-band rate at 90-140. Building the fixture at the script's default 150 +# would put reference passages outside the band the product asks for. +if [ ! -s "$BEATS" ]; then + log "building fixture" + "$PY" scripts/r49-corpus/build_beat_fixture.py \ + --pairs pairs/pairs-val.jsonl --out "$BEATS" --sidecar "$SIDE" \ + -n 60 --seed 4919 --min-words 90 --max-words 140 || exit 1 +else + log "fixture $BEATS already exists -- reusing" +fi + +for arm in base:NONE ckpt900:$RUN/checkpoints/checkpoint-900 ckpt450:$RUN/checkpoints/checkpoint-450; do + name=${arm%%:*}; path=${arm#*:} + if [ "$name" != "base" ] && [ ! -d "$path" ]; then log "MISSING $path"; exit 1; fi + if [ -s "$OUT/beats5.$name.jsonl" ]; then + log "arm $name already has $(wc -l < "$OUT/beats5.$name.jsonl") generations -- skipping" + continue + fi + log "arm $name" + if [ "$name" = "base" ]; then + "$PY" scripts/r49-corpus/gen_beats_chat_yarros.py \ + --base ~/carriers/Qwen3-4B-Instruct --beats "$BEATS" \ + --out "$OUT/beats5.$name.jsonl" --arm "$name" --seeds $SEEDS \ + --system-from "$PROV" || exit 1 + else + "$PY" scripts/r49-corpus/gen_beats_chat_yarros.py \ + --base ~/carriers/Qwen3-4B-Instruct --adapter "$path" --beats "$BEATS" \ + --out "$OUT/beats5.$name.jsonl" --arm "$name" --seeds $SEEDS \ + --system-from "$PROV" || exit 1 + fi + log " $(wc -l < "$OUT/beats5.$name.jsonl") generations" +done + +log "AXIS B -- MEMORISATION (corpus = the renamed copies the adapter trained on)" +# --corpus and --eval-dir are passed explicitly: the script's Yarros defaults would +# compare a McCarthy arm against the YARROS corpus and report a clean zero that +# means "different book", not "did not memorise". +"$PY" scripts/yarros-corpus/memorization_check.py \ + --eval-dir "$OUT" --corpus corpus-renamed/copies --glob 'beats5.*.jsonl' --strip 'beats5.' -n 8 \ + 2>&1 | tee "$OUT/memorization.txt" + +for cand in ckpt900 ckpt450; do + log "AXIS C -- DAMAGE (ran-on / out-of-band), $cand vs base" + "$PY" scripts/yarros-corpus/score_beats.py \ + --arm base="$OUT/beats5.base.jsonl" \ + --arm ckpt900="$OUT/beats5.ckpt900.jsonl" \ + --arm ckpt450="$OUT/beats5.ckpt450.jsonl" \ + --baseline base --candidate "$cand" --metric-source raw \ + --out "$OUT/score.$cand.json" 2>&1 | tee "$OUT/score.$cand.txt" +done + +log "AXIS A -- VOICE (delta_cb vs held-out McCarthy), + the pre-registered punct reads" +"$PY" scripts/mccarthy-corpus/voice-prep.py || exit 1 +"$PY" scripts/r49-corpus/voice_distance.py corpus-renamed "$OUT" --author McCarthy \ + --punct-report --secondary-normalised \ + 2>&1 | tee "$OUT/voice_distance.txt" + +echo "rc=0" > ~/lv-mccarthy/.eval-complete +log "done" diff --git a/scripts/mccarthy-corpus/voice-prep.py b/scripts/mccarthy-corpus/voice-prep.py new file mode 100644 index 0000000..2d1e12d --- /dev/null +++ b/scripts/mccarthy-corpus/voice-prep.py @@ -0,0 +1,49 @@ +"""Adapt the lv-mccarthy arms for voice_distance.py, and refuse if its reference is empty. + +Same contract as the lv-hemingway sibling — the arm NAMES are the only real difference, +and the control must carry the substring `unadapted` or voice_distance.py cannot identify +it and prints an empty vs-control table instead of an error. + +voice_distance.py expects: + - files matching voice..jsonl in the eval dir + - a `continuation` field per record (gen_beats_chat writes `raw`) + - a `seed` field (present) + - the control arm's NAME to contain the substring "unadapted" + - a reference built from corpus records whose split == "val" +""" +import json, sys +from collections import Counter +from pathlib import Path + +CORP = Path("/home/infra-ops/lv-mccarthy/corpus-renamed/copies") +EVAL = Path("/home/infra-ops/r49-runs/mccarthy-eval") + +c = Counter() +n = 0 +for p in sorted(CORP.glob("*.jsonl")): + for line in p.read_text(encoding="utf-8").splitlines(): + if not line.strip(): + continue + r = json.loads(line) + n += 1 + c[r.get("split")] += 1 +print(f"corpus records: {n} split values: {dict(c)}") +if c.get("val", 0) == 0: + print("== REFUSING: no split=val records; voice_distance would build an EMPTY reference") + print(" and every delta_cb would be meaningless rather than absent.") + sys.exit(1) + +NAMES = {"base": "base-unadapted", "ckpt900": "ckpt900", "ckpt450": "ckpt450"} +for src_arm, out_arm in NAMES.items(): + src = EVAL / f"beats5.{src_arm}.jsonl" + if not src.exists(): + print(f"== missing {src}") + sys.exit(1) + rows = [json.loads(l) for l in src.read_text(encoding="utf-8").splitlines() if l.strip()] + out = EVAL / f"voice.{out_arm}.jsonl" + with out.open("w", encoding="utf-8") as fh: + for r in rows: + fh.write(json.dumps({"id": r["id"], "seed": r["seed"], + "continuation": r["raw"]}, ensure_ascii=False) + "\n") + print(f" {src.name} -> {out.name} ({len(rows)} records)") +print("ready") diff --git a/scripts/r49-corpus/voice_distance.py b/scripts/r49-corpus/voice_distance.py index e31a4fa..1e7ac16 100644 --- a/scripts/r49-corpus/voice_distance.py +++ b/scripts/r49-corpus/voice_distance.py @@ -19,12 +19,65 @@ absolute-band claim. The A-vs-A floor below is the only thing that makes a betwe gap meaningful — half-vs-half of the held-out reference gives the distance the metric returns for two samples of the SAME author, so a between-arm gap smaller than that floor is not a finding. + +⭐ TWO OPT-IN READS ADDED FOR lv-mccarthy (2026-09-21), both pre-registered in +`scripts/mccarthy-corpus/GATE-PREREG.md` before any McCarthy generation existed. Neither +runs by default and neither changes a byte of the default output, because the Brontë, +Yarros and Hemingway records were written by the default path and must stay reproducible. + + --secondary-normalised Re-runs the WHOLE analysis with punctuation stripped from the + reference and from every arm. McCarthy's corpus measures 0.0 + quote marks per 10k words against Hemingway's 838, so "emit no + quotation marks" is the single cheapest way to move a + char-bigram distance without having learned a sentence. This + read is a deliberately CONSERVATIVE LOWER BOUND: it also strips + terminal punctuation, and therefore strips sentence-length + signal the adapter legitimately learned. It is reported, it is + never the verdict. + + --punct-report Per-arm punctuation density against the reference. This is the + check that says whether the PRIMARY read is confounded at all: + the eval harness hands the base control the same register + prompt, tics included, so if base COMPLIES its quote density + sits near the corpus's and the adapter earns no delta for the + cheap win. If base IGNORES the instruction, the primary gap is + partly punctuation and the caller must say so. The trigger is + pre-registered, not chosen here: see PUNCT_CONFOUND_PER_10K. """ from __future__ import annotations import argparse, json, re, sys, statistics as st from collections import Counter from pathlib import Path +# ⚠ PRE-REGISTERED, in GATE-PREREG.md, before any McCarthy arm was generated. If the base +# control's quote-mark density exceeds this, the control did NOT take the punctuation win +# it was instructed to take, the primary delta_cb gap is partly that win, and the +# normalised secondary read is promoted to load-bearing. The corpus measures 0.0 per 10k +# and Hemingway's measures 838; 100 is the order-of-magnitude line between them. +PUNCT_CONFOUND_PER_10K = 100.0 + +QUOTE_CHARS = "\"'‘’“”«»‹›‚„`" +_PUNCT_RE = re.compile(r"[^\w\s]|_", re.UNICODE) +_QUOTE_RE = re.compile("[" + re.escape(QUOTE_CHARS) + "]") +# a contraction apostrophe is one sitting BETWEEN letters -- `dont` vs `don't` is the tic +# the register names, and a possessive or a quote mark is not the same measurement. +_CONTRACTION_APOS_RE = re.compile(r"(?<=[A-Za-z])['’](?=[A-Za-z])") +_DASH_RE = re.compile("[—–]|--") + + +def identity(text: str) -> str: + return text + + +def strip_punct(text: str) -> str: + """Remove every punctuation mark, keeping letters, digits and word boundaries. + + Deliberately blunt. The point is not to isolate one tic but to remove the entire + punctuation channel, so that whatever gap survives is carried by words and their + morphology alone. Underscore is stripped explicitly because `\\w` keeps it. + """ + return re.sub(r"\s+", " ", _PUNCT_RE.sub(" ", text)).strip() + def bigrams(text: str) -> Counter: t = re.sub(r"\s+", " ", text.lower()) @@ -42,32 +95,56 @@ def delta(arm_text: str, ref_prof: dict, mu: dict, sd: dict, keys: list[str]) -> return st.mean(abs((ap[k] - mu[k]) / sd[k] - (ref_prof[k] - mu[k]) / sd[k]) for k in keys) -def main() -> int: - # ⚠ --author IS REQUIRED, and that is the fix for a defect this script shipped with. - # The reference label was hardcoded "Yarros". Run against Brontë it printed - # "reference: held-out Yarros" over Brontë's numbers, and that output is now sitting - # in a committed artifact saying the wrong author. A default would have kept the - # silent-wrong-label failure and only moved it; naming the author is one word at the - # call site and the label can no longer disagree with the data. - ap = argparse.ArgumentParser() - ap.add_argument("corpus", help="renamed corpus dir containing copies/ with split=val records") - ap.add_argument("evaldir", help="dir of voice..jsonl; the control arm's name must " - "contain the substring `unadapted`") - ap.add_argument("--author", required=True, - help="reference author label, e.g. Hemingway. Required: see above.") - a = ap.parse_args() - corpus = Path(a.corpus) - evaldir = Path(a.evaldir) - author = a.author - # reference = held-out val text - val = [] - for p in sorted((corpus / "copies").glob("*.jsonl")): - for l in p.read_text(encoding="utf-8").splitlines(): - r = json.loads(l) - if r.get("split") == "val": - val.append(r["text"]) - # dedup identical val chapters across copies (renaming aside, the same chapter recurs) - ref_text = "\n".join(dict.fromkeys(val)) +def load_arms(evaldir: Path) -> list[tuple[str, list[dict]]]: + out = [] + for f in sorted(evaldir.glob("voice.*.jsonl")): + recs = [json.loads(l) for l in f.read_text(encoding="utf-8").splitlines() if l.strip()] + out.append((f.stem.replace("voice.", ""), recs)) + return out + + +def density(text: str, pattern: re.Pattern) -> float: + w = len(text.split()) or 1 + return len(pattern.findall(text)) * 10000.0 / w + + +def punct_report(ref_text: str, arms: list[tuple[str, list[dict]]]) -> None: + """Did the base control take the punctuation win the register prompt handed it?""" + print("\n PUNCTUATION DENSITY per 10k words -- the confound check, not an axis") + print(" (the eval harness drives EVERY arm with the same register prompt, tics included;") + print(" a compliant base control earns the adapter no delta_cb for them)") + print(f" {'arm':22s} {'quote-marks':>12s} {'contraction-apos':>18s} {'dashes':>9s}") + rows = [("held-out reference", ref_text)] + rows += [(a, "\n".join(r["continuation"] for r in recs)) for a, recs in arms] + base_q = None + for name, txt in rows: + q = density(txt, _QUOTE_RE) + print(f" {name:22s} {q:12.1f} {density(txt, _CONTRACTION_APOS_RE):18.1f} " + f"{density(txt, _DASH_RE):9.1f}") + if "unadapted" in name: + base_q = q + if base_q is None: + print(" ⚠ no arm name contains `unadapted` -- the control was not identified, so the") + print(" pre-registered confound trigger CANNOT be evaluated. This is not a pass.") + return + if base_q > PUNCT_CONFOUND_PER_10K: + print(f"\n ⚠⚠ CONFOUND TRIGGERED: base control quote density {base_q:.1f} > " + f"{PUNCT_CONFOUND_PER_10K:.0f} per 10k.") + print(" The control's output is punctuation-rich. Where the register prompt NAMES the") + print(" punctuation (lv-mccarthy does; lv-hemingway and lv-bronte do not), that means") + print(" the control did not take the win it was handed, so part of the primary") + print(" delta_cb gap is that win rather than sentence structure, and per the") + print(" pre-registration the NORMALISED secondary read becomes load-bearing. Where the") + print(" register does NOT name it, this is a description of the corpus, not a defect.") + else: + print(f"\n [PASS] base control quote density {base_q:.1f} <= " + f"{PUNCT_CONFOUND_PER_10K:.0f} per 10k: the control complied with the register,") + print(" so the punctuation win is handed to both sides and the primary read stands.") + + +def analyse(ref_text_raw: str, arms: list[tuple[str, list[dict]]], author: str, + transform=identity) -> None: + ref_text = transform(ref_text_raw) # feature set: the most frequent bigrams in the reference (stable, high-signal) keys = [k for k, _ in bigrams(ref_text).most_common(400)] # mu/sd across the val text split into chunks, for z-scoring @@ -88,14 +165,9 @@ def main() -> int: print(f"same-author target (held-out {author} vs itself): delta_cb = {same_author:.3f}") print(f" -> the floor of what any arm could reach; lower is more {author}-like, this is the best possible\n") - def arm_texts(f): - return [json.loads(l) for l in f.read_text(encoding="utf-8").splitlines()] - rows = [] - for f in sorted(evaldir.glob("voice.*.jsonl")): - arm = f.stem.replace("voice.", "") - recs = arm_texts(f) - allt = "\n".join(r["continuation"] for r in recs) + for arm, recs in arms: + allt = transform("\n".join(r["continuation"] for r in recs)) d = delta(allt, ref_prof, mu, sd, keys) # within-arm sampling spread = the REAL noise floor for a between-arm gap: # split by seed and score each subset; the range is this metric's variance @@ -103,7 +175,8 @@ def main() -> int: by_seed = {} for r in recs: by_seed.setdefault(r["seed"], []).append(r["continuation"]) - seed_ds = [delta("\n".join(v), ref_prof, mu, sd, keys) for v in by_seed.values() if len(v) > 2] + seed_ds = [delta(transform("\n".join(v)), ref_prof, mu, sd, keys) + for v in by_seed.values() if len(v) > 2] spread = (max(seed_ds) - min(seed_ds)) if len(seed_ds) > 1 else float("nan") rows.append((arm, d, len(allt.split()), seed_ds, spread)) @@ -131,7 +204,7 @@ def main() -> int: if base_row is None: print(f"\n all-arms noise floor (largest within-arm seed spread): {noise_all:.3f}") print(" ⚠ no arm name contains `unadapted` -- no control identified, no verdict\n") - return 0 + return base_arm, base = base_row print(f"\n all-arms noise floor (largest within-arm seed spread, lv-bronte's rule): {noise_all:.3f}") print(f" PAIRWISE floor is the verdict: max(spread(candidate), spread({base_arm}) = " @@ -160,6 +233,58 @@ def main() -> int: print(f" ⚠ RELATIVE reading on one harness: {nseeds if len(nseeds) > 1 else nseeds[0]} " f"seed group(s) per arm, scored against this corpus's own held-out split. " f"It is not an absolute-band claim and corroborates nothing on its own.") + + +def main() -> int: + # ⚠ --author IS REQUIRED, and that is the fix for a defect this script shipped with. + # The reference label was hardcoded "Yarros". Run against Brontë it printed + # "reference: held-out Yarros" over Brontë's numbers, and that output is now sitting + # in a committed artifact saying the wrong author. A default would have kept the + # silent-wrong-label failure and only moved it; naming the author is one word at the + # call site and the label can no longer disagree with the data. + ap = argparse.ArgumentParser() + ap.add_argument("corpus", help="renamed corpus dir containing copies/ with split=val records") + ap.add_argument("evaldir", help="dir of voice..jsonl; the control arm's name must " + "contain the substring `unadapted`") + ap.add_argument("--author", required=True, + help="reference author label, e.g. Hemingway. Required: see above.") + ap.add_argument("--secondary-normalised", action="store_true", + help="ALSO run the whole analysis with punctuation stripped from the " + "reference and every arm. A conservative LOWER BOUND on the voice " + "gain, reported alongside; it never overturns the primary verdict.") + ap.add_argument("--punct-report", action="store_true", + help="ALSO print per-arm punctuation density vs the reference, and " + "evaluate the pre-registered base-control confound trigger.") + a = ap.parse_args() + corpus = Path(a.corpus) + evaldir = Path(a.evaldir) + author = a.author + # reference = held-out val text + val = [] + for p in sorted((corpus / "copies").glob("*.jsonl")): + for l in p.read_text(encoding="utf-8").splitlines(): + r = json.loads(l) + if r.get("split") == "val": + val.append(r["text"]) + # dedup identical val chapters across copies (renaming aside, the same chapter recurs) + ref_text = "\n".join(dict.fromkeys(val)) + arms = load_arms(evaldir) + + analyse(ref_text, arms, author, identity) + + if a.punct_report: + punct_report(ref_text, arms) + + if a.secondary_normalised: + print("\n" + "=" * 78) + print("SECONDARY READ -- PUNCTUATION STRIPPED. Pre-registered, REPORTED, NOT THE VERDICT.") + print("Every punctuation mark is removed from the reference and from every arm, so a") + print("gap that survives here is carried by words rather than by marks. It is a LOWER") + print("BOUND and not a better measurement: stripping terminal punctuation also strips") + print("sentence-length signal the adapter legitimately learned. Read it as `at least") + print("this much of the primary gap is not the punctuation trick`.") + print("=" * 78 + "\n") + analyse(ref_text, arms, author, strip_punct) return 0