vh
300ecc1276
Live on vllm-voices (fv-ml1 GPU0 :8027) beside voices-base, lv-yarros and lv-bronte.
Healthy 190 s after recreate, four models served, GPU0 96,092 -> 96,090 MiB. The adapter
was verified byte-identical to checkpoint-850 by sha256 across both transfer hops, and the
seat was verified by generating, not by reading its config: base emits 170 words of <think>
planning and never writes the passage, lv-hemingway writes the scene.
Gate design was pre-registered before any generation existed (0bb4938). Three arms, 60
held-out beats, 4 seeds, 240 generations per arm.
A. VOICE PASS 6.4x +0.413 delta_cb, pairwise floor 0.064 -- and it clears the OLD
all-arms floor (0.113) too, so this verdict does not lean on the
rule change. Closes 73.8% of the span between the unadapted
carrier and held-out Hemingway itself; lv-bronte closed 48%.
B. NOT COPIED see below
C. NO DAMAGE PASS ran-on +0.08, on-beat -0.14, both inside a 0.217 floor
AXIS B: THE NEGATIVE CONTROL WAS THE WRONG ONE, AND FIXING IT MADE THE RESULT WORSE, NOT
BETTER. memorization_check.py uses the base-unadapted arm as its control. Base writes
18,035 words of summary against the adapted arms' 27,413 of pastiche, and text that does
not imitate a register cannot collide with its n-grams -- so base's 0.00 measures "different
register", not "did not memorise". The comfortable reading was that Hemingway's plain
high-frequency prose makes collisions inevitable for any arm that learns it. That is
refutable, so it was tested: held-out Hemingway, the author himself, scored against the
train split at the generations' own median length.
HELD-OUT HEMINGWAY (never trained) 370 chunks 0.01 hit-rate mean-longest 0.1 max 10
base-unadapted 240 gens 0.00 0.0 0
ckpt850 (shipped) 240 gens 0.07 0.6 9
positive control (train vs train) 160
The hypothesis is false: the adapter reproduces train n-grams ~7x more often than the
author reproduces himself. That is real and is on the record. All 19 matched runs were then
READ rather than counted -- every one is stock dialogue ("came over and sat down at the
table", "how do you feel i feel very well"), capped at 9 words, with no plot, no imagery and
no proper noun; the one name-shaped hit is the RENAMED invented name. Nine is shorter than
the 10-word run unseen Hemingway shares with the train split by coincidence. Elevated rate,
zero protectable content. Hemingway is in copyright; lv-yarros is the in-line precedent,
also in copyright, shipped at 0.10 against a 0.07 control. Unload is 0.003 s.
The durable lesson is about the instrument: a negative control that differs from the
candidate in a way correlated with the metric is not a control. memorization_selfsim.py and
memorization_dump_matches.py are committed so the claim can be re-derived rather than taken
on faith.
SHIPPED ckpt850, NOT the loss minimum at step 1750. The two are indistinguishable on voice
-- 0.072 apart against a 0.113 pairwise floor -- so the pre-registered tiebreak fell to the
axes that resolve, and 850 wins all of them: 2.3x tighter seed spread (0.050 vs 0.113),
lower memorisation, less ran-on, half an epoch less overfit. ckpt1750's spread is one seed
(0.491, 0.449, 0.468, then 0.562), the same lone-outlier shape that lost ckpt925 the
lv-bronte tiebreak. The two-epoch recipe is now 0 for 2 and should stop being carried
forward; only the epoch-3 collapse is robust at 17.4x jitter.
servers/fv-ml1/ssh-target was a bare IP, so deploy-stack.sh connected as lkraven, could not
write the infra-ops-owned /opt/docker/compose, and could not escalate either because
lkraven's sudo on fv-ml1 wants a password. Now infra-ops@10.251.50.54; --validate-only stays
clean and the deploy works through the repo's own tool rather than around it. Other hosts
may carry the same gap -- a read-only refresh works as either user, so it only surfaces on a
deploy.
83 lines
3.8 KiB
Bash
83 lines
3.8 KiB
Bash
#!/bin/bash
|
|
# lv-hemingway v2 gate. Design is FROZEN in scripts/hemingway-corpus/GATE-PREREG.md
|
|
# and was written before this script ever ran. Do not edit the arms, the fixture
|
|
# size or the seeds to chase a result -- re-run, do not re-tune.
|
|
#
|
|
# THREE ARMS. ckpt1750 is the eval-loss minimum (2.2783, epoch 1.973). ckpt850
|
|
# (2.2823, epoch 0.959) is +0.0040 against a 0.0044 median neighbour jitter, i.e.
|
|
# TIED -- the loss curve cannot pick between them, and on Bronte the earlier
|
|
# epoch-1 checkpoint won the tiebreak on the axes that do resolve. `base` is the
|
|
# negative control for memorisation (it never saw the corpus) and the voice
|
|
# baseline. adapter/ (epoch 3.0, +0.0762 = 17.4x jitter) is NOT gated: that one
|
|
# the loss curve settles on its own.
|
|
set -o pipefail
|
|
cd ~/lv-hemingway || exit 1
|
|
PY=/home/infra-ops/ml/.venv/bin/python
|
|
RUN=~/r49-runs/hemingway-4b-pairs-3ep
|
|
OUT=~/r49-runs/hemingway-eval
|
|
BEATS=beats-hemingway-60.json
|
|
PROV=pairs/pairs-full.jsonl.provenance.json
|
|
SEEDS="1234 5678 9012 3456"
|
|
mkdir -p "$OUT"
|
|
log(){ echo "[eval $(date +%H:%M:%S)] $*"; }
|
|
|
|
# --system-from is MANDATORY. The harness's built-in SYS is Yarros's; driving a
|
|
# Hemingway arm with it would confound the adapter change with a prompt change.
|
|
# This binds the eval prompt to the one the run actually trained under (verified:
|
|
# run provenance system_prompt == pairs provenance system_prompt).
|
|
[ -f "$PROV" ] || { log "MISSING $PROV"; exit 1; }
|
|
[ -f "$BEATS" ] || { log "MISSING $BEATS"; exit 1; }
|
|
|
|
for arm in base:NONE ckpt1750:$RUN/checkpoints/checkpoint-1750 ckpt850:$RUN/checkpoints/checkpoint-850; do
|
|
name=${arm%%:*}; path=${arm#*:}
|
|
if [ "$name" != "base" ] && [ ! -d "$path" ]; then log "MISSING $path"; exit 1; fi
|
|
if [ -s "$OUT/beats5.$name.jsonl" ]; then
|
|
log "arm $name already has $(wc -l < "$OUT/beats5.$name.jsonl") generations -- skipping"
|
|
continue
|
|
fi
|
|
log "arm $name"
|
|
if [ "$name" = "base" ]; then
|
|
"$PY" scripts/r49-corpus/gen_beats_chat_yarros.py \
|
|
--base ~/carriers/Qwen3-4B-Instruct --beats "$BEATS" \
|
|
--out "$OUT/beats5.$name.jsonl" --arm "$name" --seeds $SEEDS \
|
|
--system-from "$PROV" || exit 1
|
|
else
|
|
"$PY" scripts/r49-corpus/gen_beats_chat_yarros.py \
|
|
--base ~/carriers/Qwen3-4B-Instruct --adapter "$path" --beats "$BEATS" \
|
|
--out "$OUT/beats5.$name.jsonl" --arm "$name" --seeds $SEEDS \
|
|
--system-from "$PROV" || exit 1
|
|
fi
|
|
log " $(wc -l < "$OUT/beats5.$name.jsonl") generations"
|
|
done
|
|
|
|
log "AXIS B -- MEMORISATION (corpus = the renamed copies the adapter trained on)"
|
|
# --corpus and --eval-dir are passed explicitly: the script's Yarros defaults would
|
|
# compare a Hemingway arm against the YARROS corpus and report a clean zero that
|
|
# means "different book", not "did not memorise".
|
|
"$PY" scripts/yarros-corpus/memorization_check.py \
|
|
--eval-dir "$OUT" --corpus corpus-renamed/copies --glob 'beats5.*.jsonl' --strip 'beats5.' -n 8 \
|
|
2>&1 | tee "$OUT/memorization.txt"
|
|
|
|
log "AXIS C -- DAMAGE (ran-on / out-of-band), ckpt1750 vs base"
|
|
"$PY" scripts/yarros-corpus/score_beats.py \
|
|
--arm base="$OUT/beats5.base.jsonl" \
|
|
--arm ckpt1750="$OUT/beats5.ckpt1750.jsonl" \
|
|
--arm ckpt850="$OUT/beats5.ckpt850.jsonl" \
|
|
--baseline base --candidate ckpt1750 --metric-source raw \
|
|
--out "$OUT/score.ckpt1750.json" 2>&1 | tee "$OUT/score.ckpt1750.txt"
|
|
log "AXIS C -- DAMAGE, ckpt850 vs base"
|
|
"$PY" scripts/yarros-corpus/score_beats.py \
|
|
--arm base="$OUT/beats5.base.jsonl" \
|
|
--arm ckpt1750="$OUT/beats5.ckpt1750.jsonl" \
|
|
--arm ckpt850="$OUT/beats5.ckpt850.jsonl" \
|
|
--baseline base --candidate ckpt850 --metric-source raw \
|
|
--out "$OUT/score.ckpt850.json" 2>&1 | tee "$OUT/score.ckpt850.txt"
|
|
|
|
log "AXIS A -- VOICE (delta_cb vs held-out Hemingway)"
|
|
"$PY" voice-prep.py || exit 1
|
|
"$PY" scripts/r49-corpus/voice_distance.py corpus-renamed "$OUT" --author Hemingway \
|
|
2>&1 | tee "$OUT/voice_distance.txt"
|
|
|
|
echo "rc=0" > ~/lv-hemingway/.eval-complete
|
|
log "done"
|