vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity

feat(lv-mccarthy): pre-register the v2 gate before any arm is generated

Browse Source 
Freezes the lv-mccarthy gate design while no McCarthy generation exists, per the
lv-hemingway precedent: a threshold chosen after seeing the numbers is not a
threshold. Three arms (base-unadapted, ckpt900, ckpt450), 60 beats, 4 seeds.

Settles the voice-axis question D1 deferred. McCarthy's corpus measures 0.0 quote
marks per 10k against Hemingway's 838, so "emit no quotation marks" is the cheapest
way to move a char-bigram Delta without learning a sentence. Three parts:

  PRIMARY    unchanged -- the mccarthy register names the punctuation and
             --system-from drives the base control with the same prompt, so the
             cheap win is handed to both sides.
  SECONDARY  voice_distance.py --secondary-normalised re-runs the whole analysis
             with punctuation stripped from the reference and every arm. A
             conservative lower bound; reported, never the verdict.
  TRIGGER    voice_distance.py --punct-report evaluates a pre-registered line --
             base-unadapted quote density above 100 per 10k means the control did
             not take the win it was handed, and the normalised read is promoted
             to load-bearing. 100 is the order-of-magnitude line between this
             corpus's 0.0 and Hemingway's 838, chosen now rather than after the
             table prints.

ckpt450 is NOT tied with the minimum (+4.9x the 0.00393 median neighbour jitter)
and the pre-registration says so. It is generated to test a stated prior and to
price the memorisation headroom an earlier checkpoint buys on an in-copyright
author, with a decision rule that says exactly what result would let it displace
ckpt900.

Instrument controls, run before this landed:
  - the voice_distance.py refactor reproduces the shipped lv-hemingway
    voice_distance.txt BYTE FOR BYTE on the default path
  - strip_punct drops a quote-bearing sample from 2500.0 to 0.0 marks per 10k
  - the secondary read still resolves a gap on the Hemingway arms (+0.410 at
    7.3x floor), so a null on McCarthy would be a finding, not a blind detector

Pre-flight re-run at gate time rather than quoted from 2026-09-17:
  - leak gate: 0 of 75 renameable, 0 of 37 sub-threshold, 0 separator-split,
    four controls green
  - beat-contamination audit: 0 of 3942 beats AND 0 of 3942 responses, against
    Hemingway's 70 of 7094 -- build_sft_pairs.py --source-entities earned its
    mandatory flag

Also records three provenance defects found on first read of the run and their
disposition: the hardcoded "r49-babyyarros-pairs-pilot" run label (cosmetic, same
literal on all three runs), the empty harness_commit (all three runs), and a
pairs_sha256_16 that is not a sha256sum of the file (consistent across runs, so a
cache key rather than a fault). The run is bound to McCarthy's pairs by record
count, not by the label.
This commit is contained in:
Vuong Hoang
2026-09-21 14:39:10 -07:00
parent e52def115c
commit 9c8a4e959c
4 changed files with 538 additions and 35 deletions
Download Patch File Download Diff File Expand all files Collapse all files
scripts/mccarthy-corpus/GATE-PREREG.md
+217
View File
@@ -0,0 +1,217 @@
# lv-mccarthy v2 gate — pre-registration
**Written 2026-09-21, before a single generation existed.** Everything below is
fixed. The point of writing it down is that a threshold chosen after seeing the
numbers is not a threshold. This is the third pre-registration in the line and it
inherits the lv-hemingway design deliberately, so the two runs stay comparable;
every place it departs is marked and argued.
Nothing here may be amended after the first arm is generated. If the design turns
out to be wrong, the run is wrong and gets re-run — the rule does not move.
---
## 1. What is being gated
`gx10:~/r49-runs/mccarthy-4b-pairs-3ep/` — Qwen3-4B-Instruct + instruction-pair
LoRA on the renamed McCarthy corpus (584,684 words, 167 units, 6 works, 6 renamed
copies). 3-epoch schedule, seed 4919, rank 32 / alpha 64, lr 1e-4, seq 1536,
1,380 steps in 2h30m47s, finished **2026-09-18 00:49 PT**. `train_loss` 2.172,
end-of-run `eval_loss` 2.4594.
**The run outcome was unverified for three days** and this gate is the first read
of it. That is recorded here rather than quietly fixed, because "the run finished"
and "the run did what was configured" are different claims and only the second one
is worth anything.
### 1a. Provenance defects found on first read, and their disposition
| found | disposition |
|---|---|
| `provenance.json` `"run": "r49-babyyarros-pairs-pilot"` | **Cosmetic, not a mis-copy.** The same literal appears in the yarros AND hemingway provenance — it is a hardcoded label in `train_pairs_lora.py`, not a stale field for this run. Fix the harness; it invalidates nothing. |
| `harness_commit: ""` | **Empty on all three runs.** The training harness never recorded its own commit. No claim in this gate rests on it. Named here so it is not re-discovered as news. |
| `pairs_sha256_16` does not match `sha256sum` of the pairs file | **Not a defect.** It mismatches identically on yarros and hemingway too, so the harness computes it over the loaded records rather than the raw bytes. It is a consistent, per-corpus-unique cache key. |
| `pairs` recorded as the relative `pairs/pairs-full.jsonl` | Resolved against the launch CWD `~/lv-mccarthy`. **Bound by count instead of by path**: `train_pairs` 3673 and `val_pairs_n` 269 match `lv-mccarthy/pairs/` exactly and match no other pair set on the box. |
The run trained on McCarthy's pairs. That is established by the record counts and
by `system_prompt_source: "pairs provenance (mccarthy)"`, not by the `run` label.
## 2. Arms — three, and the third is here for a different reason than Hemingway's was
| arm | what it is | role |
|---|---|---|
| `base-unadapted` | `~/carriers/Qwen3-4B-Instruct`, no adapter | negative control for memorisation (never saw the corpus) **and** the voice baseline |
| `ckpt900` | epoch 1.958, eval_loss **2.38706** — the curve's minimum | ship candidate |
| `ckpt450` | epoch 0.980, eval_loss **2.4063** | **prior test**, see below |
Measured on this run's own 56 eval points: **median neighbour jitter 0.00393**.
⚠ **`ckpt450` IS NOT TIED, and saying so is the point.** On Hemingway the second
arm was in the run because the loss curve could not separate it (+0.9× jitter).
Here it can: `ckpt450` sits **+0.0193 = 4.9× jitter** above the minimum. Three
checkpoints *are* inside one jitter of the best — 875 (+0.4×), 850 (+1.1×), 925
(+1.5×) — and the choice of 900 among them is arbitrary and declared arbitrary;
they are the same point on the curve. `ckpt450` is not one of them.
So it is generated for a stated, falsifiable reason rather than as a coin-flip:
- **The prior it tests.** On Brontë the earlier epoch-1 checkpoint won the tiebreak
on the axes that resolve. On Hemingway the pair was tied and the question stayed
open. **McCarthy is the first run in the line whose loss curve says epoch 2 is
genuinely better than epoch 1.** That is a claim the axes can check, and checking
it costs one arm.
- **The thing it would buy.** Axis B (memorisation) carries more weight here than on
any previous corpus — Brontë is public domain, Hemingway is not, and **McCarthy is
in copyright with a living estate.** An earlier checkpoint is the cheap way to buy
a lower verbatim-overlap rate. Whether it actually does is measured below, not
assumed.
**`adapter/` (end of run, epoch 3.0, 2.4594) is NOT gated.** At **+18.4× jitter**
it is resolved by the loss curve alone — and note the curve does not drift there,
it *steps*: 2.393 at step 925, 2.457 at step 950, and flat for the remaining 430
steps. The epoch-3 collapse reproduces the Hemingway finding on a second corpus.
⚠ The shipped `adapter/` directory is the epoch-3 weights. **Whatever ships is a
checkpoint, not `adapter/`.**
## 3. Fixture and seeds — fixed now
- **60 beats**, sampled by `build_beat_fixture.py` from
`~/lv-mccarthy/pairs/pairs-val.jsonl` (held-out `split=val`, 269 pairs), `--seed 4919`,
spread across works round-robin.
- **`--max-words 140`, not the script's default 150.** McCarthy's register asks for
90–140 words and `score_beats.py` scores the in-band rate at 90–140. A fixture built
at 90–150 would put reference passages outside the band the product asks for and the
scorer measures. **207 of 269 val pairs are in-band**, across all six works
(blood-meridian 51, all-the-pretty-horses 40, the-road 38, cities-of-the-plain 33,
no-country-for-old-men 28, the-crossing 17).
- **4 seeds: 1234 5678 9012 3456.** Same as lv-bronte and lv-hemingway.
- → **240 generations per arm, 720 total.**
60 is carried over from Hemingway unchanged so the two floors are comparable; 207
in-band pairs make it available with room to spare (Hemingway had 173, Brontë 44).
## 4. The floor rule — inherited, unchanged
**The noise floor for a candidate is the larger of the within-arm seed spreads of
the TWO arms being compared: `max(spread(candidate), spread(base-unadapted))`.**
Settled at lv-hemingway and not reopened. `voice_distance.py` prints both this and
lv-bronte's all-arms floor and flags any case where the two rules disagree; the
pairwise number is the verdict.
## 5. ⭐ THE VOICE AXIS — the open question, settled here before any number exists
McCarthy's corpus measures **0.0 quote marks per 10k words** against Hemingway's
838, and `voice_distance.py` is Burrows's Delta over **character bigrams**. "Emit no
quotation marks" is therefore the single cheapest available way to move this metric
a long way without having learned a sentence. lv-mccarthy D1 flagged this and
deferred it; it is settled now, in three parts.
**5a. PRIMARY read — unchanged, and already hardened.** The `mccarthy` register
names the punctuation on purpose (`build_sft_pairs.py` REGISTERS, commit `f3bf3ca`),
and `eval-mccarthy.sh` drives **every** arm including `base-unadapted` with that same
system prompt via `--system-from`. The control is therefore *told* to drop quote marks
and write `dont`/`aint`/`wont`. The cheap win is handed to both sides and the adapter
earns no delta for it. This was decided when the register was written, before any
McCarthy number existed, and is not reopened here. **Stated cost: the voice axis is
harder for this corpus than for Hemingway's, deliberately.**
**5b. SECONDARY read — pre-registered, REPORTED, never the verdict.**
`voice_distance.py --secondary-normalised` re-runs the entire analysis with all
punctuation stripped from the reference and from every arm. A gap that survives is
carried by words rather than by marks.
⚠ It is a **conservative lower bound, not a better measurement**: stripping terminal
punctuation also strips sentence-length signal the adapter legitimately learned, and
sentence length is a real McCarthy trait. It reads as *"at least this much of the
primary gap is not the punctuation trick."* **It cannot overturn the primary verdict.**
**5c. THE CONFOUND TRIGGER — the part that makes 5a falsifiable.** 5a's whole
argument is that the base control *complies* with the instruction. That is an
assumption about a model's behaviour, and it is checkable. `--punct-report` prints
per-arm quote-mark, contraction-apostrophe and dash density against the reference,
and evaluates this line:
> **If `base-unadapted` quote density exceeds 100 marks per 10k words, the control
> did not take the win it was handed, part of the primary gap IS the punctuation
> trick, and the normalised secondary read (5b) is promoted to load-bearing for the
> verdict.**
100 is the order-of-magnitude line between this corpus's 0.0 and Hemingway's 838,
chosen now rather than after the table prints.
**Instrument controls, run before this was committed.** The `voice_distance.py`
refactor that added 5b and 5c reproduces the shipped lv-hemingway
`voice_distance.txt` **byte for byte** on the default path (positive control: the
edit did not move an existing record). `strip_punct` drops a quote-bearing sample's
density from 2500.0 to **0.0** per 10k (negative control: the normalisation is not
a no-op). On the Hemingway arms the secondary read still resolves a gap
(ckpt1750 +0.410 at 7.3× floor) rather than flattening everything — so a null
result from 5b on McCarthy would be a finding about McCarthy, not a blind detector.
## 6. The three axes, and what passes
| axis | instrument | passes if |
|---|---|---|
| **A. VOICE** | `voice_distance.py --author McCarthy --punct-report --secondary-normalised` | `delta_cb(base) − delta_cb(candidate)` **exceeds the pairwise floor** on the PRIMARY read, unless 5c fires, in which case on the NORMALISED read |
| **B. NOT COPIED** | `memorization_check.py` (longest verbatim n-gram vs the renamed copies, n=8) | hit-rate and max at or near `base-unadapted`, which never saw the corpus. Positive control (corpus vs itself) must saturate, or the zeroes are blind |
| **C. NO DAMAGE** | `score_beats.py --metric-source raw` | in-band up on base beyond the metric's own floor; on-beat and ran-on not worse beyond it |
## 7. The decision rule — frozen
1. **`ckpt900` is the default ship candidate.** The loss curve resolves it above
`ckpt450` at 4.9× jitter.
2. **`ckpt450` displaces it only if BOTH hold:** it is *not resolvably worse* on
axis A (`gap_450 ≥ gap_900 − pairwise_floor`) **AND** it is *strictly better* on
axis B by more than the base-to-candidate separation on that axis. Buying
memorisation headroom is worth it; buying it by giving up the voice the adapter
exists for is not.
3. **Axis C failure disqualifies that candidate outright**, whatever axis A says.
4. **If both candidates fail axis A** (gap inside the pairwise floor), the result is
a **negative reported with its sensitivity floor**, and the ship-or-park call goes
to the operator. The lv-bronte precedent — fail on A, clean on B and C, shipped
anyway as additive and reversible — **does not transfer automatically**: Brontë is
public domain and McCarthy is not, so axis B carries more weight in that
conversation, not less.
5. **Nothing ships on the strength of the epoch-3 `adapter/`.** It is not gated.
## 8. Pre-flight — already run, and clean
Both re-run at gate time rather than quoted from the 2026-09-17 session, because
"it passed three days ago" is a claim about a tree that may have moved.
**Leak gate** (`leak_gate.py --min-cap 8`, report `leak_gate_report.gate-time.json`):
```
0 of 75 renameable · 0 of 37 sub-threshold · 0 separator-split survivors
positive control 108/108 surfaces found in the unrenamed source
negative control nonce absent · split-scan controls: planted `W olfenbarger` detected
```
**Beat-contamination audit** (`audit_pairs_sourcenames.py --min-cap 5`, mirroring
the floor `rename.py` actually ran at), over **all 3,942 built pairs**:
```
BEAT naming a removed surface: 0 of 3942 (0.00%)
RESPONSE naming a removed surface: 0 of 3942 (0.00%)
controls green: 108/108 found in source · nonce absent from both trees
```
⭐ **Zero, where Hemingway carried 70 of 7,094 contaminated beats (0.96%).** That is
`build_sft_pairs.py --source-entities`, which was mandatory for this corpus's D4 and
which Hemingway's pairs predate. The gate is unconfounded on both halves of the pair,
not just the fixture half.
**Sensitivity floor, stated as part of the pre-flight rather than as a footnote:** a
name appearing fewer than 8 times per work is never detected and a phrase recurring
fewer than 5 times is never audited — and **no phrase map exists for this corpus, so
the phrase audit does not run at all.** Yarros and Brontë both had one. A clean leak
gate here is a narrower claim than a clean leak gate there.
## 9. Reproduce
```bash
# on pfi-gx10, ~/lv-mccarthy
./eval-mccarthy.sh # three arms, 60 beats, 4 seeds, all three axes
```
scripts/mccarthy-corpus/eval-mccarthy.sh
Executable
+112
View File
@@ -0,0 +1,112 @@
#!/bin/bash
# lv-mccarthy v2 gate. Design is FROZEN in scripts/mccarthy-corpus/GATE-PREREG.md
# and was written before this script ever ran. Do not edit the arms, the fixture
# size or the seeds to chase a result -- re-run, do not re-tune.
#
# THREE ARMS. ckpt900 is the eval-loss minimum (2.38706, epoch 1.958). ckpt450
# (2.4063, epoch 0.980) is +0.0193 against a 0.00393 median neighbour jitter =
# 4.9x, i.e. NOT tied -- and that is the difference from the Hemingway gate, where
# the second arm was a coin-flip. It is here to test a stated prior (on Bronte the
# earlier epoch-1 checkpoint won on the axes that resolve) and to price the
# memorisation headroom an earlier checkpoint would buy on an IN-COPYRIGHT author.
# `base` is the negative control for memorisation and the voice baseline.
# adapter/ (epoch 3.0, +18.4x jitter) is NOT gated: the loss curve settles that one.
set -o pipefail
cd ~/lv-mccarthy || exit 1
PY=/home/infra-ops/ml/.venv/bin/python
RUN=~/r49-runs/mccarthy-4b-pairs-3ep
OUT=~/r49-runs/mccarthy-eval
BEATS=beats-mccarthy-60.json
SIDE=beats-mccarthy-60.sidecar.json
PROV=pairs/pairs-full.jsonl.provenance.json
SEEDS="1234 5678 9012 3456"
mkdir -p "$OUT"
log(){ echo "[eval $(date +%H:%M:%S)] $*"; }
# --system-from is MANDATORY and it is load-bearing for the VOICE axis here, not
# merely hygienic. The mccarthy register NAMES the punctuation (no quote marks,
# `dont`/`aint`/`wont`), so driving the base control with the same prompt hands the
# cheap char-bigram win to BOTH sides and the adapter earns no delta for it. See
# GATE-PREREG.md section 5. The harness's built-in SYS is Yarros's; using it would
# confound the adapter change with a prompt change AND hand the adapter the win.
[ -f "$PROV" ] || { log "MISSING $PROV"; exit 1; }
# The run must have trained under the same system prompt the eval drives. Asserted,
# not assumed -- the Hemingway script claimed this was "verified" in a comment, which
# is a claim no reader can check. Here it fails the run.
"$PY" - "$PROV" "$RUN/provenance.json" <<'PYCHK' || exit 1
import json, sys
pairs = json.load(open(sys.argv[1]))
run = json.load(open(sys.argv[2]))
ps = pairs.get("system_prompt") or (pairs.get("register") or {}).get("system_prompt")
rs = run.get("system_prompt")
if ps is None:
print(f"== CANNOT VERIFY: no system_prompt in {sys.argv[1]}"); sys.exit(1)
if ps != rs:
print("== REFUSING: the run trained under a DIFFERENT system prompt than the eval would drive.")
print(f" pairs provenance: {ps[:120]!r}")
print(f" run provenance : {(rs or '')[:120]!r}")
sys.exit(1)
print(f" [PASS] run system_prompt == pairs system_prompt ({len(ps)} chars)")
PYCHK
# 60 beats, max-words 140: the mccarthy register asks for 90-140 and score_beats.py
# scores the in-band rate at 90-140. Building the fixture at the script's default 150
# would put reference passages outside the band the product asks for.
if [ ! -s "$BEATS" ]; then
log "building fixture"
"$PY" scripts/r49-corpus/build_beat_fixture.py \
--pairs pairs/pairs-val.jsonl --out "$BEATS" --sidecar "$SIDE" \
-n 60 --seed 4919 --min-words 90 --max-words 140 || exit 1
else
log "fixture $BEATS already exists -- reusing"
fi
for arm in base:NONE ckpt900:$RUN/checkpoints/checkpoint-900 ckpt450:$RUN/checkpoints/checkpoint-450; do
name=${arm%%:*}; path=${arm#*:}
if [ "$name" != "base" ] && [ ! -d "$path" ]; then log "MISSING $path"; exit 1; fi
if [ -s "$OUT/beats5.$name.jsonl" ]; then
log "arm $name already has $(wc -l < "$OUT/beats5.$name.jsonl") generations -- skipping"
continue
fi
log "arm $name"
if [ "$name" = "base" ]; then
"$PY" scripts/r49-corpus/gen_beats_chat_yarros.py \
--base ~/carriers/Qwen3-4B-Instruct --beats "$BEATS" \
--out "$OUT/beats5.$name.jsonl" --arm "$name" --seeds $SEEDS \
--system-from "$PROV" || exit 1
else
"$PY" scripts/r49-corpus/gen_beats_chat_yarros.py \
--base ~/carriers/Qwen3-4B-Instruct --adapter "$path" --beats "$BEATS" \
--out "$OUT/beats5.$name.jsonl" --arm "$name" --seeds $SEEDS \
--system-from "$PROV" || exit 1
fi
log " $(wc -l < "$OUT/beats5.$name.jsonl") generations"
done
log "AXIS B -- MEMORISATION (corpus = the renamed copies the adapter trained on)"
# --corpus and --eval-dir are passed explicitly: the script's Yarros defaults would
# compare a McCarthy arm against the YARROS corpus and report a clean zero that
# means "different book", not "did not memorise".
"$PY" scripts/yarros-corpus/memorization_check.py \
--eval-dir "$OUT" --corpus corpus-renamed/copies --glob 'beats5.*.jsonl' --strip 'beats5.' -n 8 \
2>&1 | tee "$OUT/memorization.txt"
for cand in ckpt900 ckpt450; do
log "AXIS C -- DAMAGE (ran-on / out-of-band), $cand vs base"
"$PY" scripts/yarros-corpus/score_beats.py \
--arm base="$OUT/beats5.base.jsonl" \
--arm ckpt900="$OUT/beats5.ckpt900.jsonl" \
--arm ckpt450="$OUT/beats5.ckpt450.jsonl" \
--baseline base --candidate "$cand" --metric-source raw \
--out "$OUT/score.$cand.json" 2>&1 | tee "$OUT/score.$cand.txt"
done
log "AXIS A -- VOICE (delta_cb vs held-out McCarthy), + the pre-registered punct reads"
"$PY" scripts/mccarthy-corpus/voice-prep.py || exit 1
"$PY" scripts/r49-corpus/voice_distance.py corpus-renamed "$OUT" --author McCarthy \
--punct-report --secondary-normalised \
2>&1 | tee "$OUT/voice_distance.txt"
echo "rc=0" > ~/lv-mccarthy/.eval-complete
log "done"
scripts/mccarthy-corpus/voice-prep.py
+49
View File
@@ -0,0 +1,49 @@
"""Adapt the lv-mccarthy arms for voice_distance.py, and refuse if its reference is empty.
Same contract as the lv-hemingway sibling — the arm NAMES are the only real difference,
and the control must carry the substring `unadapted` or voice_distance.py cannot identify
it and prints an empty vs-control table instead of an error.
voice_distance.py expects:
- files matching voice.<arm>.jsonl in the eval dir
- a `continuation` field per record (gen_beats_chat writes `raw`)
- a `seed` field (present)
- the control arm's NAME to contain the substring "unadapted"
- a reference built from corpus records whose split == "val"
"""
import json, sys
from collections import Counter
from pathlib import Path
CORP = Path("/home/infra-ops/lv-mccarthy/corpus-renamed/copies")
EVAL = Path("/home/infra-ops/r49-runs/mccarthy-eval")
c = Counter()
n = 0
for p in sorted(CORP.glob("*.jsonl")):
for line in p.read_text(encoding="utf-8").splitlines():
if not line.strip():
continue
r = json.loads(line)
n += 1
c[r.get("split")] += 1
print(f"corpus records: {n} split values: {dict(c)}")
if c.get("val", 0) == 0:
print("== REFUSING: no split=val records; voice_distance would build an EMPTY reference")
print(" and every delta_cb would be meaningless rather than absent.")
sys.exit(1)
NAMES = {"base": "base-unadapted", "ckpt900": "ckpt900", "ckpt450": "ckpt450"}
for src_arm, out_arm in NAMES.items():
src = EVAL / f"beats5.{src_arm}.jsonl"
if not src.exists():
print(f"== missing {src}")
sys.exit(1)
rows = [json.loads(l) for l in src.read_text(encoding="utf-8").splitlines() if l.strip()]
out = EVAL / f"voice.{out_arm}.jsonl"
with out.open("w", encoding="utf-8") as fh:
for r in rows:
fh.write(json.dumps({"id": r["id"], "seed": r["seed"],
"continuation": r["raw"]}, ensure_ascii=False) + "\n")
print(f" {src.name} -> {out.name} ({len(rows)} records)")
print("ready")
scripts/r49-corpus/voice_distance.py
+160 -35
View File
@@ -19,12 +19,65 @@ absolute-band claim. The A-vs-A floor below is the only thing that makes a betwe
gap meaningful — half-vs-half of the held-out reference gives the distance the metric
returns for two samples of the SAME author, so a between-arm gap smaller than that floor
is not a finding.
⭐ TWO OPT-IN READS ADDED FOR lv-mccarthy (2026-09-21), both pre-registered in
`scripts/mccarthy-corpus/GATE-PREREG.md` before any McCarthy generation existed. Neither
runs by default and neither changes a byte of the default output, because the Brontë,
Yarros and Hemingway records were written by the default path and must stay reproducible.
--secondary-normalised Re-runs the WHOLE analysis with punctuation stripped from the
reference and from every arm. McCarthy's corpus measures 0.0
quote marks per 10k words against Hemingway's 838, so "emit no
quotation marks" is the single cheapest way to move a
char-bigram distance without having learned a sentence. This
read is a deliberately CONSERVATIVE LOWER BOUND: it also strips
terminal punctuation, and therefore strips sentence-length
signal the adapter legitimately learned. It is reported, it is
never the verdict.
--punct-report Per-arm punctuation density against the reference. This is the
check that says whether the PRIMARY read is confounded at all:
the eval harness hands the base control the same register
prompt, tics included, so if base COMPLIES its quote density
sits near the corpus's and the adapter earns no delta for the
cheap win. If base IGNORES the instruction, the primary gap is
partly punctuation and the caller must say so. The trigger is
pre-registered, not chosen here: see PUNCT_CONFOUND_PER_10K.
"""
from __future__ import annotations
import argparse, json, re, sys, statistics as st
from collections import Counter
from pathlib import Path
# ⚠ PRE-REGISTERED, in GATE-PREREG.md, before any McCarthy arm was generated. If the base
# control's quote-mark density exceeds this, the control did NOT take the punctuation win
# it was instructed to take, the primary delta_cb gap is partly that win, and the
# normalised secondary read is promoted to load-bearing. The corpus measures 0.0 per 10k
# and Hemingway's measures 838; 100 is the order-of-magnitude line between them.
PUNCT_CONFOUND_PER_10K = 100.0
QUOTE_CHARS = "\"'‘’“”«»‹›‚„`"
_PUNCT_RE = re.compile(r"[^\w\s]|_", re.UNICODE)
_QUOTE_RE = re.compile("[" + re.escape(QUOTE_CHARS) + "]")
# a contraction apostrophe is one sitting BETWEEN letters -- `dont` vs `don't` is the tic
# the register names, and a possessive or a quote mark is not the same measurement.
_CONTRACTION_APOS_RE = re.compile(r"(?<=[A-Za-z])['’](?=[A-Za-z])")
_DASH_RE = re.compile("[—–]|--")
def identity(text: str) -> str:
return text
def strip_punct(text: str) -> str:
"""Remove every punctuation mark, keeping letters, digits and word boundaries.
Deliberately blunt. The point is not to isolate one tic but to remove the entire
punctuation channel, so that whatever gap survives is carried by words and their
morphology alone. Underscore is stripped explicitly because `\\w` keeps it.
"""
return re.sub(r"\s+", " ", _PUNCT_RE.sub(" ", text)).strip()
def bigrams(text: str) -> Counter:
t = re.sub(r"\s+", " ", text.lower())
@@ -42,32 +95,56 @@ def delta(arm_text: str, ref_prof: dict, mu: dict, sd: dict, keys: list[str]) ->
return st.mean(abs((ap[k] - mu[k]) / sd[k] - (ref_prof[k] - mu[k]) / sd[k]) for k in keys)
def main() -> int:
# ⚠ --author IS REQUIRED, and that is the fix for a defect this script shipped with.
# The reference label was hardcoded "Yarros". Run against Brontë it printed
# "reference: held-out Yarros" over Brontë's numbers, and that output is now sitting
# in a committed artifact saying the wrong author. A default would have kept the
# silent-wrong-label failure and only moved it; naming the author is one word at the
# call site and the label can no longer disagree with the data.
ap = argparse.ArgumentParser()
ap.add_argument("corpus", help="renamed corpus dir containing copies/ with split=val records")
ap.add_argument("evaldir", help="dir of voice.<arm>.jsonl; the control arm's name must "
"contain the substring `unadapted`")
ap.add_argument("--author", required=True,
help="reference author label, e.g. Hemingway. Required: see above.")
a = ap.parse_args()
corpus = Path(a.corpus)
evaldir = Path(a.evaldir)
author = a.author
# reference = held-out val text
val = []
for p in sorted((corpus / "copies").glob("*.jsonl")):
for l in p.read_text(encoding="utf-8").splitlines():
r = json.loads(l)
if r.get("split") == "val":
val.append(r["text"])
# dedup identical val chapters across copies (renaming aside, the same chapter recurs)
ref_text = "\n".join(dict.fromkeys(val))
def load_arms(evaldir: Path) -> list[tuple[str, list[dict]]]:
out = []
for f in sorted(evaldir.glob("voice.*.jsonl")):
recs = [json.loads(l) for l in f.read_text(encoding="utf-8").splitlines() if l.strip()]
out.append((f.stem.replace("voice.", ""), recs))
return out
def density(text: str, pattern: re.Pattern) -> float:
w = len(text.split()) or 1
return len(pattern.findall(text)) * 10000.0 / w
def punct_report(ref_text: str, arms: list[tuple[str, list[dict]]]) -> None:
"""Did the base control take the punctuation win the register prompt handed it?"""
print("\n PUNCTUATION DENSITY per 10k words -- the confound check, not an axis")
print(" (the eval harness drives EVERY arm with the same register prompt, tics included;")
print(" a compliant base control earns the adapter no delta_cb for them)")
print(f" {'arm':22s} {'quote-marks':>12s} {'contraction-apos':>18s} {'dashes':>9s}")
rows = [("held-out reference", ref_text)]
rows += [(a, "\n".join(r["continuation"] for r in recs)) for a, recs in arms]
base_q = None
for name, txt in rows:
q = density(txt, _QUOTE_RE)
print(f" {name:22s} {q:12.1f} {density(txt, _CONTRACTION_APOS_RE):18.1f} "
f"{density(txt, _DASH_RE):9.1f}")
if "unadapted" in name:
base_q = q
if base_q is None:
print(" ⚠ no arm name contains `unadapted` -- the control was not identified, so the")
print(" pre-registered confound trigger CANNOT be evaluated. This is not a pass.")
return
if base_q > PUNCT_CONFOUND_PER_10K:
print(f"\n ⚠⚠ CONFOUND TRIGGERED: base control quote density {base_q:.1f} > "
f"{PUNCT_CONFOUND_PER_10K:.0f} per 10k.")
print(" The control's output is punctuation-rich. Where the register prompt NAMES the")
print(" punctuation (lv-mccarthy does; lv-hemingway and lv-bronte do not), that means")
print(" the control did not take the win it was handed, so part of the primary")
print(" delta_cb gap is that win rather than sentence structure, and per the")
print(" pre-registration the NORMALISED secondary read becomes load-bearing. Where the")
print(" register does NOT name it, this is a description of the corpus, not a defect.")
else:
print(f"\n [PASS] base control quote density {base_q:.1f} <= "
f"{PUNCT_CONFOUND_PER_10K:.0f} per 10k: the control complied with the register,")
print(" so the punctuation win is handed to both sides and the primary read stands.")
def analyse(ref_text_raw: str, arms: list[tuple[str, list[dict]]], author: str,
transform=identity) -> None:
ref_text = transform(ref_text_raw)
# feature set: the most frequent bigrams in the reference (stable, high-signal)
keys = [k for k, _ in bigrams(ref_text).most_common(400)]
# mu/sd across the val text split into chunks, for z-scoring
@@ -88,14 +165,9 @@ def main() -> int:
print(f"same-author target (held-out {author} vs itself): delta_cb = {same_author:.3f}")
print(f" -> the floor of what any arm could reach; lower is more {author}-like, this is the best possible\n")
def arm_texts(f):
return [json.loads(l) for l in f.read_text(encoding="utf-8").splitlines()]
rows = []
for f in sorted(evaldir.glob("voice.*.jsonl")):
arm = f.stem.replace("voice.", "")
recs = arm_texts(f)
allt = "\n".join(r["continuation"] for r in recs)
for arm, recs in arms:
allt = transform("\n".join(r["continuation"] for r in recs))
d = delta(allt, ref_prof, mu, sd, keys)
# within-arm sampling spread = the REAL noise floor for a between-arm gap:
# split by seed and score each subset; the range is this metric's variance
@@ -103,7 +175,8 @@ def main() -> int:
by_seed = {}
for r in recs:
by_seed.setdefault(r["seed"], []).append(r["continuation"])
seed_ds = [delta("\n".join(v), ref_prof, mu, sd, keys) for v in by_seed.values() if len(v) > 2]
seed_ds = [delta(transform("\n".join(v)), ref_prof, mu, sd, keys)
for v in by_seed.values() if len(v) > 2]
spread = (max(seed_ds) - min(seed_ds)) if len(seed_ds) > 1 else float("nan")
rows.append((arm, d, len(allt.split()), seed_ds, spread))
@@ -131,7 +204,7 @@ def main() -> int:
if base_row is None:
print(f"\n all-arms noise floor (largest within-arm seed spread): {noise_all:.3f}")
print(" ⚠ no arm name contains `unadapted` -- no control identified, no verdict\n")
return 0
return
base_arm, base = base_row
print(f"\n all-arms noise floor (largest within-arm seed spread, lv-bronte's rule): {noise_all:.3f}")
print(f" PAIRWISE floor is the verdict: max(spread(candidate), spread({base_arm}) = "
@@ -160,6 +233,58 @@ def main() -> int:
print(f" ⚠ RELATIVE reading on one harness: {nseeds if len(nseeds) > 1 else nseeds[0]} "
f"seed group(s) per arm, scored against this corpus's own held-out split. "
f"It is not an absolute-band claim and corroborates nothing on its own.")
def main() -> int:
# ⚠ --author IS REQUIRED, and that is the fix for a defect this script shipped with.
# The reference label was hardcoded "Yarros". Run against Brontë it printed
# "reference: held-out Yarros" over Brontë's numbers, and that output is now sitting
# in a committed artifact saying the wrong author. A default would have kept the
# silent-wrong-label failure and only moved it; naming the author is one word at the
# call site and the label can no longer disagree with the data.
ap = argparse.ArgumentParser()
ap.add_argument("corpus", help="renamed corpus dir containing copies/ with split=val records")
ap.add_argument("evaldir", help="dir of voice.<arm>.jsonl; the control arm's name must "
"contain the substring `unadapted`")
ap.add_argument("--author", required=True,
help="reference author label, e.g. Hemingway. Required: see above.")
ap.add_argument("--secondary-normalised", action="store_true",
help="ALSO run the whole analysis with punctuation stripped from the "
"reference and every arm. A conservative LOWER BOUND on the voice "
"gain, reported alongside; it never overturns the primary verdict.")
ap.add_argument("--punct-report", action="store_true",
help="ALSO print per-arm punctuation density vs the reference, and "
"evaluate the pre-registered base-control confound trigger.")
a = ap.parse_args()
corpus = Path(a.corpus)
evaldir = Path(a.evaldir)
author = a.author
# reference = held-out val text
val = []
for p in sorted((corpus / "copies").glob("*.jsonl")):
for l in p.read_text(encoding="utf-8").splitlines():
r = json.loads(l)
if r.get("split") == "val":
val.append(r["text"])
# dedup identical val chapters across copies (renaming aside, the same chapter recurs)
ref_text = "\n".join(dict.fromkeys(val))
arms = load_arms(evaldir)
analyse(ref_text, arms, author, identity)
if a.punct_report:
punct_report(ref_text, arms)
if a.secondary_normalised:
print("\n" + "=" * 78)
print("SECONDARY READ -- PUNCTUATION STRIPPED. Pre-registered, REPORTED, NOT THE VERDICT.")
print("Every punctuation mark is removed from the reference and from every arm, so a")
print("gap that survives here is carried by words rather than by marks. It is a LOWER")
print("BOUND and not a better measurement: stripping terminal punctuation also strips")
print("sentence-length signal the adapter legitimately learned. Read it as `at least")
print("this much of the primary gap is not the punctuation trick`.")
print("=" * 78 + "\n")
analyse(ref_text, arms, author, strip_punct)
return 0