vh
300ecc1276
Live on vllm-voices (fv-ml1 GPU0 :8027) beside voices-base, lv-yarros and lv-bronte.
Healthy 190 s after recreate, four models served, GPU0 96,092 -> 96,090 MiB. The adapter
was verified byte-identical to checkpoint-850 by sha256 across both transfer hops, and the
seat was verified by generating, not by reading its config: base emits 170 words of <think>
planning and never writes the passage, lv-hemingway writes the scene.
Gate design was pre-registered before any generation existed (0bb4938). Three arms, 60
held-out beats, 4 seeds, 240 generations per arm.
A. VOICE PASS 6.4x +0.413 delta_cb, pairwise floor 0.064 -- and it clears the OLD
all-arms floor (0.113) too, so this verdict does not lean on the
rule change. Closes 73.8% of the span between the unadapted
carrier and held-out Hemingway itself; lv-bronte closed 48%.
B. NOT COPIED see below
C. NO DAMAGE PASS ran-on +0.08, on-beat -0.14, both inside a 0.217 floor
AXIS B: THE NEGATIVE CONTROL WAS THE WRONG ONE, AND FIXING IT MADE THE RESULT WORSE, NOT
BETTER. memorization_check.py uses the base-unadapted arm as its control. Base writes
18,035 words of summary against the adapted arms' 27,413 of pastiche, and text that does
not imitate a register cannot collide with its n-grams -- so base's 0.00 measures "different
register", not "did not memorise". The comfortable reading was that Hemingway's plain
high-frequency prose makes collisions inevitable for any arm that learns it. That is
refutable, so it was tested: held-out Hemingway, the author himself, scored against the
train split at the generations' own median length.
HELD-OUT HEMINGWAY (never trained) 370 chunks 0.01 hit-rate mean-longest 0.1 max 10
base-unadapted 240 gens 0.00 0.0 0
ckpt850 (shipped) 240 gens 0.07 0.6 9
positive control (train vs train) 160
The hypothesis is false: the adapter reproduces train n-grams ~7x more often than the
author reproduces himself. That is real and is on the record. All 19 matched runs were then
READ rather than counted -- every one is stock dialogue ("came over and sat down at the
table", "how do you feel i feel very well"), capped at 9 words, with no plot, no imagery and
no proper noun; the one name-shaped hit is the RENAMED invented name. Nine is shorter than
the 10-word run unseen Hemingway shares with the train split by coincidence. Elevated rate,
zero protectable content. Hemingway is in copyright; lv-yarros is the in-line precedent,
also in copyright, shipped at 0.10 against a 0.07 control. Unload is 0.003 s.
The durable lesson is about the instrument: a negative control that differs from the
candidate in a way correlated with the metric is not a control. memorization_selfsim.py and
memorization_dump_matches.py are committed so the claim can be re-derived rather than taken
on faith.
SHIPPED ckpt850, NOT the loss minimum at step 1750. The two are indistinguishable on voice
-- 0.072 apart against a 0.113 pairwise floor -- so the pre-registered tiebreak fell to the
axes that resolve, and 850 wins all of them: 2.3x tighter seed spread (0.050 vs 0.113),
lower memorisation, less ran-on, half an epoch less overfit. ckpt1750's spread is one seed
(0.491, 0.449, 0.468, then 0.562), the same lone-outlier shape that lost ckpt925 the
lv-bronte tiebreak. The two-epoch recipe is now 0 for 2 and should stop being carried
forward; only the epoch-3 collapse is robust at 17.4x jitter.
servers/fv-ml1/ssh-target was a bare IP, so deploy-stack.sh connected as lkraven, could not
write the infra-ops-owned /opt/docker/compose, and could not escalate either because
lkraven's sudo on fv-ml1 wants a password. Now infra-ops@10.251.50.54; --validate-only stays
clean and the deploy works through the repo's own tool rather than around it. Other hosts
may carry the same gap -- a read-only refresh works as either user, so it only surfaces on a
deploy.
48 lines
1.9 KiB
Python
48 lines
1.9 KiB
Python
"""Adapt the lv-hemingway arms for voice_distance.py, and refuse if its reference is empty.
|
|
|
|
voice_distance.py expects:
|
|
- files matching voice.<arm>.jsonl in the eval dir
|
|
- a `continuation` field per record (gen_beats_chat writes `raw`)
|
|
- a `seed` field (present)
|
|
- the control arm's NAME to contain the substring "unadapted" -- it picks the
|
|
control by that substring, so `base` alone would leave the control
|
|
unidentified and the whole vs-control table would silently be empty.
|
|
- a reference built from corpus records whose split == "val"
|
|
"""
|
|
import json, sys
|
|
from collections import Counter
|
|
from pathlib import Path
|
|
|
|
CORP = Path("/home/infra-ops/lv-hemingway/corpus-renamed/copies")
|
|
EVAL = Path("/home/infra-ops/r49-runs/hemingway-eval")
|
|
|
|
c = Counter()
|
|
n = 0
|
|
for p in sorted(CORP.glob("*.jsonl")):
|
|
for line in p.read_text(encoding="utf-8").splitlines():
|
|
if not line.strip():
|
|
continue
|
|
r = json.loads(line)
|
|
n += 1
|
|
c[r.get("split")] += 1
|
|
print(f"corpus records: {n} split values: {dict(c)}")
|
|
if c.get("val", 0) == 0:
|
|
print("== REFUSING: no split=val records; voice_distance would build an EMPTY reference")
|
|
print(" and every delta_cb would be meaningless rather than absent.")
|
|
sys.exit(1)
|
|
|
|
NAMES = {"base": "base-unadapted", "ckpt1750": "ckpt1750", "ckpt850": "ckpt850"}
|
|
for src_arm, out_arm in NAMES.items():
|
|
src = EVAL / f"beats5.{src_arm}.jsonl"
|
|
if not src.exists():
|
|
print(f"== missing {src}")
|
|
sys.exit(1)
|
|
rows = [json.loads(l) for l in src.read_text(encoding="utf-8").splitlines() if l.strip()]
|
|
out = EVAL / f"voice.{out_arm}.jsonl"
|
|
with out.open("w", encoding="utf-8") as fh:
|
|
for r in rows:
|
|
fh.write(json.dumps({"id": r["id"], "seed": r["seed"],
|
|
"continuation": r["raw"]}, ensure_ascii=False) + "\n")
|
|
print(f" {src.name} -> {out.name} ({len(rows)} records)")
|
|
print("ready")
|