vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity

lv-hemingway: pre-register the v2 gate, and fix the floor rule that decided lv-bronte

Browse Source 
The gate design is written before any generation exists, because lv-bronte's
verdict turned on a choice that was only visible after the numbers printed.

THE FLOOR RULE IS NOW PAIRWISE. lv-bronte computed the noise floor as the largest
within-arm seed spread across ALL arms present. Its ckpt475 shipped at +0.193
against a 0.251 floor set entirely by ckpt925 -- a third arm nobody was shipping,
on one outlier seed. Scored against the arm it was actually compared to, the floor
is 0.092 and the same gap clears at 2.1x. A candidate's verdict must not depend on
which other arms happened to be generated. voice_distance.py now prints both floors
and flags any disagreement, so the lv-bronte record stays comparable.

audit_pairs_sourcenames.py closes the blind spot leak_gate.py has by construction:
it reads the corpus and the renamed copies, never the generated beats, so it cannot
see a beat-writing model restoring the author's real character names. Run over the
Hemingway pairs, which predate build_sft_pairs.py --source-entities:

  val    0 of 200     -- the eval fixture is clean, the gate is unconfounded
  train  70 of 7,094 (0.96%)  -- Santiago x16, Catherine x7, Rinaldi x3, Brett,
                                 Harry, Jake, Pablo, Nick, Maria ...
  responses 0 of 7,294        -- the lv-bronte beat-only signature exactly

A matched surface is only counted when the rename actually removed it, verified
against the renamed copies, so a beat naming a held real-world place is not a leak.
Controls run every time: 941/941 surfaces found in the unrenamed source, nonce
absent from both trees, and 6 planted canonical names detected 6/6.

voice_distance.py --author is now REQUIRED. It was hardcoded "Yarros" and printed
"reference: held-out Yarros" over Brontë's numbers into a committed artifact. A
default would have moved the silent-wrong-label failure rather than removed it. The
stale "one seed-pair per arm / corroborates Base < Instruct" footer is replaced with
what the run actually carries.

Gate design: three arms (base-unadapted, ckpt1750, ckpt850), 60 beats, 4 seeds.
ckpt850 is present because the loss curve cannot separate it from ckpt1750 -- +0.0040
against a 0.0044 median neighbour jitter, with three checkpoints inside one jitter of
the minimum. adapter/ is excluded: +0.0762 is 17.4x the jitter and is resolved without
a gate.
This commit is contained in:
vh
2026-09-17 01:45:02 -07:00
parent c445ce9e93
commit 0bb4938518
3 changed files with 369 additions and 35 deletions
Download Patch File Download Diff File Expand all files Collapse all files
scripts/r49-corpus/voice_distance.py
+81 -35
View File
@@ -1,15 +1,18 @@
"""Did the adapter move the voice TOWARD held-out Yarros? A seat-free relative measure.
"""Did the adapter move the voice TOWARD the held-out author? A seat-free relative measure.
NOT the frozen adjudication. That needs a romantasy control-author panel (to place an
absolute band and a hard-negative sister), a seed-to-seed spread, and — for BEAT
INCUMBENT — the gen seat, none of which are available here. This answers the smaller,
honest question the operator can act on: of the three arms generated on ONE harness,
which sits closest to real held-out Yarros, and does the adapter beat the base control?
Written for Yarros, since used on Brontë and Hemingway. The author is now a REQUIRED
argument rather than a hardcoded string — see the note on `--author` in main().
NOT the frozen adjudication. That needs a control-author panel (to place an absolute
band and a hard-negative sister), a seed-to-seed spread, and — for BEAT INCUMBENT — the
gen seat, none of which are available here. This answers the smaller, honest question
the operator can act on: of the arms generated on ONE harness, which sits closest to
the real held-out author, and does the adapter beat the base control?
Instrument: Burrows's Delta over CHARACTER BIGRAMS (hence delta_cb). Char bigrams are
dominated by function-word morphology and rhythm, not proper nouns, so the rename does
not move them. Reference profile is the HELD-OUT (val) split — text no arm was trained
on. Each arm's pooled generations are scored against it; lower = closer to Yarros.
on. Each arm's pooled generations are scored against it; lower = closer to the author.
Discipline: this is a RELATIVE reading (arms vs each other, same harness), never an
absolute-band claim. The A-vs-A floor below is the only thing that makes a between-arm
@@ -18,7 +21,7 @@ returns for two samples of the SAME author, so a between-arm gap smaller than th
is not a finding.
"""
from __future__ import annotations
import json, re, sys, statistics as st
import argparse, json, re, sys, statistics as st
from collections import Counter
from pathlib import Path
@@ -40,8 +43,22 @@ def delta(arm_text: str, ref_prof: dict, mu: dict, sd: dict, keys: list[str]) ->
def main() -> int:
corpus = Path(sys.argv[1]) # yarros-corpus-renamed (has split=val)
evaldir = Path(sys.argv[2]) # dir of voice.*.jsonl
# ⚠ --author IS REQUIRED, and that is the fix for a defect this script shipped with.
# The reference label was hardcoded "Yarros". Run against Brontë it printed
# "reference: held-out Yarros" over Brontë's numbers, and that output is now sitting
# in a committed artifact saying the wrong author. A default would have kept the
# silent-wrong-label failure and only moved it; naming the author is one word at the
# call site and the label can no longer disagree with the data.
ap = argparse.ArgumentParser()
ap.add_argument("corpus", help="renamed corpus dir containing copies/ with split=val records")
ap.add_argument("evaldir", help="dir of voice.<arm>.jsonl; the control arm's name must "
"contain the substring `unadapted`")
ap.add_argument("--author", required=True,
help="reference author label, e.g. Hemingway. Required: see above.")
a = ap.parse_args()
corpus = Path(a.corpus)
evaldir = Path(a.evaldir)
author = a.author
# reference = held-out val text
val = []
for p in sorted((corpus / "copies").glob("*.jsonl")):
@@ -62,14 +79,14 @@ def main() -> int:
ref_prof = profile(ref_text, keys)
# SAME-AUTHOR REFERENCE (the target, not a significance threshold): two halves
# of held-out Yarros. A perfect mimic scores about this; you cannot get closer
# to Yarros than Yarros gets to itself at this sample size.
# of the held-out author. A perfect mimic scores about this; you cannot get closer
# to the author than the author gets to itself at this sample size.
half = len(words) // 2
same_author = delta(" ".join(words[:half]), profile(" ".join(words[half:]), keys), mu, sd, keys)
print(f"reference: held-out Yarros, {len(words):,} words, {len(chunks)} chunks, {len(keys)} char-bigram features")
print(f"same-author target (held-out Yarros vs itself): delta_cb = {same_author:.3f}")
print(f" -> the floor of what any arm could reach; lower is more Yarros-like, this is the best possible\n")
print(f"reference: held-out {author}, {len(words):,} words, {len(chunks)} chunks, {len(keys)} char-bigram features")
print(f"same-author target (held-out {author} vs itself): delta_cb = {same_author:.3f}")
print(f" -> the floor of what any arm could reach; lower is more {author}-like, this is the best possible\n")
def arm_texts(f):
return [json.loads(l) for l in f.read_text(encoding="utf-8").splitlines()]
@@ -93,27 +110,56 @@ def main() -> int:
print(" arm delta_cb per-seed [words]")
for arm, d, w, sd_, spread in sorted(rows, key=lambda x: x[1]):
seeds = " ".join(f"{x:.3f}" for x in sd_)
print(f" {arm:20s} {d:.3f} ({seeds}) [{w}]")
# the noise floor is the LARGEST within-arm spread across arms
floors = [r[4] for r in rows if r[4] == r[4]]
noise = max(floors) if floors else float("nan")
print(f"\n measured noise floor (largest within-arm seed spread): {noise:.3f}")
print(f" -> a between-arm gap must exceed ~{noise:.3f} to be a real difference\n")
print(f" {arm:20s} {d:.3f} ({seeds}) [{w}] spread {spread:.3f}")
base = next((d for a, d, _, _, _ in rows if "unadapted" in a), None)
if base is not None:
print(" vs base-unadapted control (positive gap = moved toward Yarros):")
for arm, d, _, _, _ in sorted(rows, key=lambda x: x[1]):
if "unadapted" in arm:
continue
gap = base - d
verdict = ("MOVED toward Yarros (exceeds noise floor)" if gap > noise
else "moved toward Yarros, but within the measured noise floor")
print(f" {arm:20s} {gap:+.3f} ({verdict})")
ordered = [a for a, *_ in sorted(rows, key=lambda x: x[1])]
print(f"\n ordering: {' < '.join(ordered)} (lower = more Yarros-like)")
print(" ⚠ one seed-pair per arm; this ordering CORROBORATES the independent held-out")
print(" loss ordering (Base < Instruct) but is not itself a multi-seed result.")
# ⚠⚠ THE FLOOR IS PAIRWISE, and that is a RULE CHANGE made because the all-arms
# rule decided lv-bronte. Measured there:
# base-unadapted spread 0.062
# ckpt475 spread 0.092 <- the candidate that shipped
# ckpt925 spread 0.251 <- set the floor, on ONE outlier seed
# ckpt475's +0.193 was failed by a floor contributed entirely by a THIRD arm nobody
# was shipping. Run as base-vs-ckpt475 the floor is 0.092 and the same gap clears at
# 2.1x. A candidate's verdict must not depend on which other arms you happened to
# generate, so the comparison's floor is the larger of the TWO arms being compared.
# The all-arms number is still printed, because lv-bronte's record used it and a
# reader comparing the two runs needs both.
floors = [r[4] for r in rows if r[4] == r[4]]
noise_all = max(floors) if floors else float("nan")
spread_of = {r[0]: r[4] for r in rows}
base_row = next(((a, d) for a, d, _, _, _ in rows if "unadapted" in a), None)
if base_row is None:
print(f"\n all-arms noise floor (largest within-arm seed spread): {noise_all:.3f}")
print(" ⚠ no arm name contains `unadapted` -- no control identified, no verdict\n")
return 0
base_arm, base = base_row
print(f"\n all-arms noise floor (largest within-arm seed spread, lv-bronte's rule): {noise_all:.3f}")
print(f" PAIRWISE floor is the verdict: max(spread(candidate), spread({base_arm}) = "
f"{spread_of[base_arm]:.3f})\n")
print(f" vs {base_arm} control (positive gap = moved toward {author}):")
for arm, d, _, _, _ in sorted(rows, key=lambda x: x[1]):
if arm == base_arm:
continue
gap = base - d
pair_floor = max(spread_of[arm], spread_of[base_arm])
verdict = (f"MOVED toward {author} ({gap / pair_floor:.1f}x the pairwise floor "
f"{pair_floor:.3f})" if gap > pair_floor
else f"within the pairwise floor {pair_floor:.3f} -- NOT a finding")
flag = "" if (gap > noise_all) == (gap > pair_floor) else " <- the two rules DISAGREE"
print(f" {arm:20s} {gap:+.3f} ({verdict}){flag}")
ordered = [a for a, *_ in sorted(rows, key=lambda x: x[1])]
print(f"\n ordering: {' < '.join(ordered)} (lower = more {author}-like)")
# ⚠ This used to assert "one seed-pair per arm" and claim corroboration from a
# "Base < Instruct" held-out loss ordering. Both were Yarros-run facts hardcoded
# as if they were properties of the instrument: by lv-bronte every arm carried
# four seeds, and no Base-vs-Instruct comparison was in the run at all. Report
# what this run actually has instead of a remembered one.
nseeds = sorted({len(r[3]) for r in rows})
print(f" ⚠ RELATIVE reading on one harness: {nseeds if len(nseeds) > 1 else nseeds[0]} "
f"seed group(s) per arm, scored against this corpus's own held-out split. "
f"It is not an absolute-band claim and corroborates nothing on its own.")
return 0