Vuong Hoang
e8eb1594d9
1,200 generations across five arms. ckpt300 (epoch 0.652) is the best arm in the
run on every axis that resolves:
VOICE +0.177 at 3.2x its pairwise floor (primary), +0.128 at 2.8x with
every punctuation mark stripped. Best point estimate AND best
margin of any arm, spread 0.055/0.038 with no outlier seed.
MEMORISATION 0.12 against real unseen McCarthy's own 0.12 -- identical -- with
a longest match of 10 words against the author's coincidental 12.
All 31 matches read: stock grammar, names are the renamed
inventions, nothing protectable.
DAMAGE ran-on +0.12. Clears the operator's ratified v2 floor of 0.200 by
40%. FAILS AMENDMENT 3's self-imposed 0.100 bar by 0.02.
NOT SHIPPED, and the reason is the bar rather than the adapter. AMENDMENT 3 fixed
ran-on <= 0.100 before either new arm existed, precisely so a marginal number could
not be talked into a ship, and shipping at 0.12 would make that pre-registration
theatre. But the bar's stated rationale was written against ckpt450's pass by 0.01
-- 5% of the threshold -- and ckpt300 clears by 40%. The number excludes a candidate
the reasoning does not. That is an operator call.
ckpt325/350/375 are on disk and one may sit under 0.100. They were deliberately NOT
gated: searching the checkpoint space until something clears is candidate-shopping,
the same family as threshold-shopping approached from the other side.
THREE CLAIMS FROM EARLIER THIS SESSION ARE REFUTED and are corrected in the record:
1. "The damage is flat across epochs and only rotates direction" -- FALSE. ran-on
is non-monotonic (0.38 -> 0.13 -> 0.20 -> 0.28 across epochs 0.49/0.65/0.98/
1.96) with a real minimum near 0.65, and ckpt225 is 48% out-of-band against
ckpt300's 35%.
2. "ckpt300 runs far too short, ckpt225 will clear ran-on by being short" -- FALSE
on both. ckpt225 runs LONG (median 127, 38% over-band) and is the worst arm in
the run. I generalised from SIX generations of one arm, which is the exact n=1
violation the measurement-discipline rule names, committed in the same breath
as a note about being careful.
3. The original "gate an earlier checkpoint, the overshoot may not have arrived
yet" recommendation was RIGHT. Retracting it an hour later on a three-arm read
was the error, not the recommendation.
What is true and unresolved by any checkpoint choice: 35% of ckpt300's generations
miss the 90-140 band against base's 11%, and in-band is 0.65 against 0.89. An
adapter that buys a voice and costs a third of the length compliance is a trade, not
a defect -- but it is the operator's trade to accept.
Raw artifacts for all five arms at scripts/mccarthy-corpus/gate-results/.
64 lines
2.6 KiB
Python
64 lines
2.6 KiB
Python
"""Adapt the lv-mccarthy arms for voice_distance.py, and refuse if its reference is empty.
|
|
|
|
Same contract as the lv-hemingway sibling — the arm NAMES are the only real difference,
|
|
and the control must carry the substring `unadapted` or voice_distance.py cannot identify
|
|
it and prints an empty vs-control table instead of an error.
|
|
|
|
voice_distance.py expects:
|
|
- files matching voice.<arm>.jsonl in the eval dir
|
|
- a `continuation` field per record (gen_beats_chat writes `raw`)
|
|
- a `seed` field (present)
|
|
- the control arm's NAME to contain the substring "unadapted"
|
|
- a reference built from corpus records whose split == "val"
|
|
"""
|
|
import json, sys
|
|
from collections import Counter
|
|
from pathlib import Path
|
|
|
|
CORP = Path("/home/infra-ops/lv-mccarthy/corpus-renamed/copies")
|
|
EVAL = Path("/home/infra-ops/r49-runs/mccarthy-eval")
|
|
|
|
c = Counter()
|
|
n = 0
|
|
for p in sorted(CORP.glob("*.jsonl")):
|
|
for line in p.read_text(encoding="utf-8").splitlines():
|
|
if not line.strip():
|
|
continue
|
|
r = json.loads(line)
|
|
n += 1
|
|
c[r.get("split")] += 1
|
|
print(f"corpus records: {n} split values: {dict(c)}")
|
|
if c.get("val", 0) == 0:
|
|
print("== REFUSING: no split=val records; voice_distance would build an EMPTY reference")
|
|
print(" and every delta_cb would be meaningless rather than absent.")
|
|
sys.exit(1)
|
|
|
|
# Arms are DISCOVERED, not listed. AMENDMENT 3 added two and a hardcoded dict would have
|
|
# silently dropped them from the voice table while every other axis scored them -- the arm
|
|
# would be missing rather than failing, which is the worse of the two.
|
|
found = sorted(EVAL.glob("beats5.*.jsonl"))
|
|
if not found:
|
|
print(f"== REFUSING: no beats5.*.jsonl in {EVAL}")
|
|
sys.exit(1)
|
|
NAMES = {}
|
|
for f in found:
|
|
arm = f.stem.replace("beats5.", "")
|
|
NAMES[arm] = "base-unadapted" if arm == "base" else arm
|
|
if "base-unadapted" not in NAMES.values():
|
|
print("== REFUSING: no `base` arm, so voice_distance would have no control to compare against")
|
|
sys.exit(1)
|
|
print(f"arms discovered: {', '.join(sorted(NAMES))}")
|
|
for src_arm, out_arm in NAMES.items():
|
|
src = EVAL / f"beats5.{src_arm}.jsonl"
|
|
if not src.exists():
|
|
print(f"== missing {src}")
|
|
sys.exit(1)
|
|
rows = [json.loads(l) for l in src.read_text(encoding="utf-8").splitlines() if l.strip()]
|
|
out = EVAL / f"voice.{out_arm}.jsonl"
|
|
with out.open("w", encoding="utf-8") as fh:
|
|
for r in rows:
|
|
fh.write(json.dumps({"id": r["id"], "seed": r["seed"],
|
|
"continuation": r["raw"]}, ensure_ascii=False) + "\n")
|
|
print(f" {src.name} -> {out.name} ({len(rows)} records)")
|
|
print("ready")
|