feat(lv-mccarthy): pre-register the v2 gate before any arm is generated
Freezes the lv-mccarthy gate design while no McCarthy generation exists, per the
lv-hemingway precedent: a threshold chosen after seeing the numbers is not a
threshold. Three arms (base-unadapted, ckpt900, ckpt450), 60 beats, 4 seeds.
Settles the voice-axis question D1 deferred. McCarthy's corpus measures 0.0 quote
marks per 10k against Hemingway's 838, so "emit no quotation marks" is the cheapest
way to move a char-bigram Delta without learning a sentence. Three parts:
PRIMARY unchanged -- the mccarthy register names the punctuation and
--system-from drives the base control with the same prompt, so the
cheap win is handed to both sides.
SECONDARY voice_distance.py --secondary-normalised re-runs the whole analysis
with punctuation stripped from the reference and every arm. A
conservative lower bound; reported, never the verdict.
TRIGGER voice_distance.py --punct-report evaluates a pre-registered line --
base-unadapted quote density above 100 per 10k means the control did
not take the win it was handed, and the normalised read is promoted
to load-bearing. 100 is the order-of-magnitude line between this
corpus's 0.0 and Hemingway's 838, chosen now rather than after the
table prints.
ckpt450 is NOT tied with the minimum (+4.9x the 0.00393 median neighbour jitter)
and the pre-registration says so. It is generated to test a stated prior and to
price the memorisation headroom an earlier checkpoint buys on an in-copyright
author, with a decision rule that says exactly what result would let it displace
ckpt900.
Instrument controls, run before this landed:
- the voice_distance.py refactor reproduces the shipped lv-hemingway
voice_distance.txt BYTE FOR BYTE on the default path
- strip_punct drops a quote-bearing sample from 2500.0 to 0.0 marks per 10k
- the secondary read still resolves a gap on the Hemingway arms (+0.410 at
7.3x floor), so a null on McCarthy would be a finding, not a blind detector
Pre-flight re-run at gate time rather than quoted from 2026-09-17:
- leak gate: 0 of 75 renameable, 0 of 37 sub-threshold, 0 separator-split,
four controls green
- beat-contamination audit: 0 of 3942 beats AND 0 of 3942 responses, against
Hemingway's 70 of 7094 -- build_sft_pairs.py --source-entities earned its
mandatory flag
Also records three provenance defects found on first read of the run and their
disposition: the hardcoded "r49-babyyarros-pairs-pilot" run label (cosmetic, same
literal on all three runs), the empty harness_commit (all three runs), and a
pairs_sha256_16 that is not a sha256sum of the file (consistent across runs, so a
cache key rather than a fault). The run is bound to McCarthy's pairs by record
count, not by the label.
This commit is contained in:
Vuong Hoang
Executable
+112
@@ -0,0 +1,112 @@
|
||||
#!/bin/bash
|
||||
# lv-mccarthy v2 gate. Design is FROZEN in scripts/mccarthy-corpus/GATE-PREREG.md
|
||||
# and was written before this script ever ran. Do not edit the arms, the fixture
|
||||
# size or the seeds to chase a result -- re-run, do not re-tune.
|
||||
#
|
||||
# THREE ARMS. ckpt900 is the eval-loss minimum (2.38706, epoch 1.958). ckpt450
|
||||
# (2.4063, epoch 0.980) is +0.0193 against a 0.00393 median neighbour jitter =
|
||||
# 4.9x, i.e. NOT tied -- and that is the difference from the Hemingway gate, where
|
||||
# the second arm was a coin-flip. It is here to test a stated prior (on Bronte the
|
||||
# earlier epoch-1 checkpoint won on the axes that resolve) and to price the
|
||||
# memorisation headroom an earlier checkpoint would buy on an IN-COPYRIGHT author.
|
||||
# `base` is the negative control for memorisation and the voice baseline.
|
||||
# adapter/ (epoch 3.0, +18.4x jitter) is NOT gated: the loss curve settles that one.
|
||||
set -o pipefail
|
||||
cd ~/lv-mccarthy || exit 1
|
||||
PY=/home/infra-ops/ml/.venv/bin/python
|
||||
RUN=~/r49-runs/mccarthy-4b-pairs-3ep
|
||||
OUT=~/r49-runs/mccarthy-eval
|
||||
BEATS=beats-mccarthy-60.json
|
||||
SIDE=beats-mccarthy-60.sidecar.json
|
||||
PROV=pairs/pairs-full.jsonl.provenance.json
|
||||
SEEDS="1234 5678 9012 3456"
|
||||
mkdir -p "$OUT"
|
||||
log(){ echo "[eval $(date +%H:%M:%S)] $*"; }
|
||||
|
||||
# --system-from is MANDATORY and it is load-bearing for the VOICE axis here, not
|
||||
# merely hygienic. The mccarthy register NAMES the punctuation (no quote marks,
|
||||
# `dont`/`aint`/`wont`), so driving the base control with the same prompt hands the
|
||||
# cheap char-bigram win to BOTH sides and the adapter earns no delta for it. See
|
||||
# GATE-PREREG.md section 5. The harness's built-in SYS is Yarros's; using it would
|
||||
# confound the adapter change with a prompt change AND hand the adapter the win.
|
||||
[ -f "$PROV" ] || { log "MISSING $PROV"; exit 1; }
|
||||
|
||||
# The run must have trained under the same system prompt the eval drives. Asserted,
|
||||
# not assumed -- the Hemingway script claimed this was "verified" in a comment, which
|
||||
# is a claim no reader can check. Here it fails the run.
|
||||
"$PY" - "$PROV" "$RUN/provenance.json" <<'PYCHK' || exit 1
|
||||
import json, sys
|
||||
pairs = json.load(open(sys.argv[1]))
|
||||
run = json.load(open(sys.argv[2]))
|
||||
ps = pairs.get("system_prompt") or (pairs.get("register") or {}).get("system_prompt")
|
||||
rs = run.get("system_prompt")
|
||||
if ps is None:
|
||||
print(f"== CANNOT VERIFY: no system_prompt in {sys.argv[1]}"); sys.exit(1)
|
||||
if ps != rs:
|
||||
print("== REFUSING: the run trained under a DIFFERENT system prompt than the eval would drive.")
|
||||
print(f" pairs provenance: {ps[:120]!r}")
|
||||
print(f" run provenance : {(rs or '')[:120]!r}")
|
||||
sys.exit(1)
|
||||
print(f" [PASS] run system_prompt == pairs system_prompt ({len(ps)} chars)")
|
||||
PYCHK
|
||||
|
||||
# 60 beats, max-words 140: the mccarthy register asks for 90-140 and score_beats.py
|
||||
# scores the in-band rate at 90-140. Building the fixture at the script's default 150
|
||||
# would put reference passages outside the band the product asks for.
|
||||
if [ ! -s "$BEATS" ]; then
|
||||
log "building fixture"
|
||||
"$PY" scripts/r49-corpus/build_beat_fixture.py \
|
||||
--pairs pairs/pairs-val.jsonl --out "$BEATS" --sidecar "$SIDE" \
|
||||
-n 60 --seed 4919 --min-words 90 --max-words 140 || exit 1
|
||||
else
|
||||
log "fixture $BEATS already exists -- reusing"
|
||||
fi
|
||||
|
||||
for arm in base:NONE ckpt900:$RUN/checkpoints/checkpoint-900 ckpt450:$RUN/checkpoints/checkpoint-450; do
|
||||
name=${arm%%:*}; path=${arm#*:}
|
||||
if [ "$name" != "base" ] && [ ! -d "$path" ]; then log "MISSING $path"; exit 1; fi
|
||||
if [ -s "$OUT/beats5.$name.jsonl" ]; then
|
||||
log "arm $name already has $(wc -l < "$OUT/beats5.$name.jsonl") generations -- skipping"
|
||||
continue
|
||||
fi
|
||||
log "arm $name"
|
||||
if [ "$name" = "base" ]; then
|
||||
"$PY" scripts/r49-corpus/gen_beats_chat_yarros.py \
|
||||
--base ~/carriers/Qwen3-4B-Instruct --beats "$BEATS" \
|
||||
--out "$OUT/beats5.$name.jsonl" --arm "$name" --seeds $SEEDS \
|
||||
--system-from "$PROV" || exit 1
|
||||
else
|
||||
"$PY" scripts/r49-corpus/gen_beats_chat_yarros.py \
|
||||
--base ~/carriers/Qwen3-4B-Instruct --adapter "$path" --beats "$BEATS" \
|
||||
--out "$OUT/beats5.$name.jsonl" --arm "$name" --seeds $SEEDS \
|
||||
--system-from "$PROV" || exit 1
|
||||
fi
|
||||
log " $(wc -l < "$OUT/beats5.$name.jsonl") generations"
|
||||
done
|
||||
|
||||
log "AXIS B -- MEMORISATION (corpus = the renamed copies the adapter trained on)"
|
||||
# --corpus and --eval-dir are passed explicitly: the script's Yarros defaults would
|
||||
# compare a McCarthy arm against the YARROS corpus and report a clean zero that
|
||||
# means "different book", not "did not memorise".
|
||||
"$PY" scripts/yarros-corpus/memorization_check.py \
|
||||
--eval-dir "$OUT" --corpus corpus-renamed/copies --glob 'beats5.*.jsonl' --strip 'beats5.' -n 8 \
|
||||
2>&1 | tee "$OUT/memorization.txt"
|
||||
|
||||
for cand in ckpt900 ckpt450; do
|
||||
log "AXIS C -- DAMAGE (ran-on / out-of-band), $cand vs base"
|
||||
"$PY" scripts/yarros-corpus/score_beats.py \
|
||||
--arm base="$OUT/beats5.base.jsonl" \
|
||||
--arm ckpt900="$OUT/beats5.ckpt900.jsonl" \
|
||||
--arm ckpt450="$OUT/beats5.ckpt450.jsonl" \
|
||||
--baseline base --candidate "$cand" --metric-source raw \
|
||||
--out "$OUT/score.$cand.json" 2>&1 | tee "$OUT/score.$cand.txt"
|
||||
done
|
||||
|
||||
log "AXIS A -- VOICE (delta_cb vs held-out McCarthy), + the pre-registered punct reads"
|
||||
"$PY" scripts/mccarthy-corpus/voice-prep.py || exit 1
|
||||
"$PY" scripts/r49-corpus/voice_distance.py corpus-renamed "$OUT" --author McCarthy \
|
||||
--punct-report --secondary-normalised \
|
||||
2>&1 | tee "$OUT/voice_distance.txt"
|
||||
|
||||
echo "rc=0" > ~/lv-mccarthy/.eval-complete
|
||||
log "done"
|
||||
Reference in New Issue
Block a user