voices-seat: ship lv-hemingway (ckpt850), and replace the memorisation control that passed it

Live on vllm-voices (fv-ml1 GPU0 :8027) beside voices-base, lv-yarros and lv-bronte.
Healthy 190 s after recreate, four models served, GPU0 96,092 -> 96,090 MiB. The adapter
was verified byte-identical to checkpoint-850 by sha256 across both transfer hops, and the
seat was verified by generating, not by reading its config: base emits 170 words of <think>
planning and never writes the passage, lv-hemingway writes the scene.

Gate design was pre-registered before any generation existed (0bb4938). Three arms, 60
held-out beats, 4 seeds, 240 generations per arm.

  A. VOICE   PASS 6.4x   +0.413 delta_cb, pairwise floor 0.064 -- and it clears the OLD
                         all-arms floor (0.113) too, so this verdict does not lean on the
                         rule change. Closes 73.8% of the span between the unadapted
                         carrier and held-out Hemingway itself; lv-bronte closed 48%.
  B. NOT COPIED  see below
  C. NO DAMAGE   PASS    ran-on +0.08, on-beat -0.14, both inside a 0.217 floor

AXIS B: THE NEGATIVE CONTROL WAS THE WRONG ONE, AND FIXING IT MADE THE RESULT WORSE, NOT
BETTER. memorization_check.py uses the base-unadapted arm as its control. Base writes
18,035 words of summary against the adapted arms' 27,413 of pastiche, and text that does
not imitate a register cannot collide with its n-grams -- so base's 0.00 measures "different
register", not "did not memorise". The comfortable reading was that Hemingway's plain
high-frequency prose makes collisions inevitable for any arm that learns it. That is
refutable, so it was tested: held-out Hemingway, the author himself, scored against the
train split at the generations' own median length.

  HELD-OUT HEMINGWAY (never trained)   370 chunks   0.01 hit-rate   mean-longest 0.1   max 10
  base-unadapted                       240 gens     0.00                        0.0        0
  ckpt850 (shipped)                    240 gens     0.07                        0.6        9
  positive control (train vs train)                                             160

The hypothesis is false: the adapter reproduces train n-grams ~7x more often than the
author reproduces himself. That is real and is on the record. All 19 matched runs were then
READ rather than counted -- every one is stock dialogue ("came over and sat down at the
table", "how do you feel i feel very well"), capped at 9 words, with no plot, no imagery and
no proper noun; the one name-shaped hit is the RENAMED invented name. Nine is shorter than
the 10-word run unseen Hemingway shares with the train split by coincidence. Elevated rate,
zero protectable content. Hemingway is in copyright; lv-yarros is the in-line precedent,
also in copyright, shipped at 0.10 against a 0.07 control. Unload is 0.003 s.

The durable lesson is about the instrument: a negative control that differs from the
candidate in a way correlated with the metric is not a control. memorization_selfsim.py and
memorization_dump_matches.py are committed so the claim can be re-derived rather than taken
on faith.

SHIPPED ckpt850, NOT the loss minimum at step 1750. The two are indistinguishable on voice
-- 0.072 apart against a 0.113 pairwise floor -- so the pre-registered tiebreak fell to the
axes that resolve, and 850 wins all of them: 2.3x tighter seed spread (0.050 vs 0.113),
lower memorisation, less ran-on, half an epoch less overfit. ckpt1750's spread is one seed
(0.491, 0.449, 0.468, then 0.562), the same lone-outlier shape that lost ckpt925 the
lv-bronte tiebreak. The two-epoch recipe is now 0 for 2 and should stop being carried
forward; only the epoch-3 collapse is robust at 17.4x jitter.

servers/fv-ml1/ssh-target was a bare IP, so deploy-stack.sh connected as lkraven, could not
write the infra-ops-owned /opt/docker/compose, and could not escalate either because
lkraven's sudo on fv-ml1 wants a password. Now infra-ops@10.251.50.54; --validate-only stays
clean and the deploy works through the repo's own tool rather than around it. Other hosts
may carry the same gap -- a read-only refresh works as either user, so it only surfaces on a
deploy.
This commit is contained in:
vh
2026-09-17 03:36:29 -07:00
parent 5e6611466c
commit 300ecc1276
8 changed files with 462 additions and 29 deletions
+27
View File
@@ -114,6 +114,33 @@ services:
# See /adapters/lv-bronte-4b-v1/README.md and
# persistent-memory.d/2026-09-17-lv-bronte-gate.md
- lv-bronte=/adapters/lv-bronte-4b-v1
# lv-hemingway: checkpoint-850 (epoch 0.959), NOT the loss minimum at step 1750.
# The two are indistinguishable on voice — 0.072 apart against a 0.113 pairwise
# floor — so the tiebreak fell to the axes that resolve, and 850 wins all of them:
# 2.3x tighter seed spread (0.050 vs 0.113), lower memorisation (0.07 vs 0.08),
# less ran-on (0.08 vs 0.12), and half an epoch less overfit.
#
# VOICE ✅ +0.413 delta_cb at 6.4x the pairwise floor — the strongest result in this
# line, closing 73.8% of the span between the unadapted carrier and held-out
# Hemingway itself. It also clears the OLD all-arms floor (0.113), so this verdict
# does not depend on the 2026-09-17 rule change.
# DAMAGE ✅ ran-on +0.08 and on-beat −0.14, both inside a 0.217 floor.
#
# ⚠ MEMORISATION IS THE AXIS TO READ BEFORE QUOTING THIS ONE AS CLEAN. 0.07 hit-rate
# against a base control of 0.00 — but that control is weak here, because base writes
# summary and cannot collide with a register it does not imitate. The honest
# reference is the author himself: HELD-OUT HEMINGWAY scored against the train split
# collides at 0.01. So the adapter reproduces train n-grams ~7x more often than
# Hemingway reproduces himself. Every one of the 19 matched runs was READ: all are
# stock dialogue ("came over and sat down at the table"), max 9 words, no proper
# noun, no plot, no imagery — and 9 is shorter than the 10-word run unseen Hemingway
# shares with the train split by coincidence. Elevated rate, zero protectable
# content. Hemingway is in copyright; lv-yarros is the in-line precedent, also in
# copyright, shipped at 0.10 against a 0.07 control. Unload is 0.003 s.
# See /adapters/lv-hemingway-4b-v1/README.md,
# scripts/hemingway-corpus/GATE-PREREG.md (pre-registered before any generation),
# persistent-memory.d/2026-09-17-lv-hemingway-gate.md
- lv-hemingway=/adapters/lv-hemingway-4b-v1
deploy:
resources:
reservations: