vh
c445ce9e93
In-flight rewritten for the next goal. lv-hemingway is TRAINED and nothing else has been done to it: ship candidate is checkpoint-1750 (ep 1.97, eval 2.2783), the end-of-run adapter is 0.0763 worse, and the v2 gate has not been run. Every instrument it needs was parameterised during the lv-bronte run tonight and the in-flight section names all four with their traps. New detail files: 2026-09-17-lv-bronte-gate.md shipped, voice axis failed, why anyway 2026-09-17-beat-contamination-leak.md the leak the corpus gate cannot see 2026-09-17-esh-fiber-outages.md two Cityside failures, rotation fragility Also commits the memorization_check.py parameterisation, which was left uncommitted: its hardcoded Yarros defaults would have compared a Hemingway arm against the Yarros corpus and reported a meaningless clean zero. Auto-archival: index was 415 lines pre-run, over the 300 cap. Only five entries cleared the 14-day age guard, and three of those carry open deferred pointers (fused MoE park 47, nconnect=8, AI-tab belayed) and are referenced by in-flight. A fourth — every CI job on pfi-fleet runs as root on ana-docker — is a live security property rather than settled history, so it is held back deliberately. One entry archived. The file stays over cap, which is the guard working: an over-cap file that keeps live decisions beats a scannable one that lost them.
72 lines
3.2 KiB
Python
72 lines
3.2 KiB
Python
"""Did an arm learn the voice, or learn the text? delta_cb cannot tell them apart.
|
|
|
|
pairs-ckpt150 scored delta_cb 0.470 against a same-author target of 0.463 -- i.e. at this
|
|
sample size it is statistically indistinguishable from real held-out Yarros. That is either
|
|
excellent voice capture or near-verbatim regurgitation, and those two have opposite
|
|
consequences: one ships, the other is both a quality mirage and the exact leak the rename
|
|
pipeline and its gate exist to prevent. Char-bigram distance is blind to the difference.
|
|
|
|
Instrument: longest and mean maximal verbatim n-gram shared with the TRAINING corpus, per
|
|
generation. Controls run every time -- the base-unadapted arm never saw the corpus so it is
|
|
the negative control, and a slice of the corpus scored against itself is the positive.
|
|
"""
|
|
import argparse, json, pathlib, re, sys
|
|
from collections import Counter
|
|
|
|
# ⚠ These were hardcoded to Yarros. Pointed at a Brontë arm they would have compared
|
|
# it against the YARROS corpus and reported a clean zero — a negative that means
|
|
# "different book", not "did not memorise". Defaults are unchanged so every Yarros
|
|
# number already recorded stays reproducible byte for byte.
|
|
_ap = argparse.ArgumentParser()
|
|
_ap.add_argument("--eval-dir", default="/home/infra-ops/r49-runs/yarros-eval")
|
|
_ap.add_argument("--corpus", default="/home/infra-ops/yarros-corpus-renamed/copies",
|
|
help="the RENAMED copies the adapter actually trained on")
|
|
_ap.add_argument("--glob", default="beats5.*.jsonl", help="arm files inside --eval-dir")
|
|
_ap.add_argument("--strip", default="beats5.", help="prefix trimmed to name the arm")
|
|
_ap.add_argument("-n", type=int, default=8, help="n-gram length")
|
|
_a = _ap.parse_args()
|
|
|
|
EVAL = pathlib.Path(_a.eval_dir)
|
|
CORP = pathlib.Path(_a.corpus)
|
|
N = _a.n
|
|
|
|
def norm(t): return re.findall(r"[a-z']+", t.lower())
|
|
|
|
corpus_words = []
|
|
for f in sorted(CORP.glob("*.copy0.jsonl")):
|
|
for l in f.read_text(encoding="utf-8").splitlines():
|
|
corpus_words.extend(norm(json.loads(l)["text"]))
|
|
grams = set()
|
|
for i in range(len(corpus_words) - N + 1):
|
|
grams.add(" ".join(corpus_words[i:i + N]))
|
|
print(f"corpus: {len(corpus_words):,} words, {len(grams):,} distinct {N}-grams\n")
|
|
|
|
def longest_match(words):
|
|
best = 0
|
|
i = 0
|
|
while i <= len(words) - N:
|
|
if " ".join(words[i:i + N]) in grams:
|
|
k = N
|
|
while i + k < len(words) and " ".join(words[i + k - N + 1:i + k + 1]) in grams:
|
|
k += 1
|
|
best = max(best, k)
|
|
i += 1
|
|
else:
|
|
i += 1
|
|
return best
|
|
|
|
print(f"{'arm':<22} {'gens':>5} {'hit-rate':>9} {'mean-longest':>13} {'max':>5}")
|
|
print("-" * 60)
|
|
for f in sorted(EVAL.glob(_a.glob)):
|
|
arm = f.stem.replace(_a.strip, "")
|
|
rows = [json.loads(l) for l in f.read_text(encoding="utf-8").splitlines()]
|
|
longs = [longest_match(norm(r["raw"])) for r in rows]
|
|
hits = sum(1 for x in longs if x >= N)
|
|
print(f"{arm:<22} {len(rows):>5} {hits/len(rows):>9.2f} "
|
|
f"{sum(longs)/len(longs):>13.1f} {max(longs):>5}")
|
|
|
|
# positive control: corpus against itself must saturate
|
|
slice_words = corpus_words[1000:1160]
|
|
print(f"\npositive control (corpus slice vs corpus): longest = {longest_match(slice_words)} "
|
|
f"(must be large, else the detector is blind)")
|