voices-seat: ship lv-hemingway (ckpt850), and replace the memorisation control that passed it
Live on vllm-voices (fv-ml1 GPU0 :8027) beside voices-base, lv-yarros and lv-bronte.
Healthy 190 s after recreate, four models served, GPU0 96,092 -> 96,090 MiB. The adapter
was verified byte-identical to checkpoint-850 by sha256 across both transfer hops, and the
seat was verified by generating, not by reading its config: base emits 170 words of <think>
planning and never writes the passage, lv-hemingway writes the scene.
Gate design was pre-registered before any generation existed (0bb4938). Three arms, 60
held-out beats, 4 seeds, 240 generations per arm.
A. VOICE PASS 6.4x +0.413 delta_cb, pairwise floor 0.064 -- and it clears the OLD
all-arms floor (0.113) too, so this verdict does not lean on the
rule change. Closes 73.8% of the span between the unadapted
carrier and held-out Hemingway itself; lv-bronte closed 48%.
B. NOT COPIED see below
C. NO DAMAGE PASS ran-on +0.08, on-beat -0.14, both inside a 0.217 floor
AXIS B: THE NEGATIVE CONTROL WAS THE WRONG ONE, AND FIXING IT MADE THE RESULT WORSE, NOT
BETTER. memorization_check.py uses the base-unadapted arm as its control. Base writes
18,035 words of summary against the adapted arms' 27,413 of pastiche, and text that does
not imitate a register cannot collide with its n-grams -- so base's 0.00 measures "different
register", not "did not memorise". The comfortable reading was that Hemingway's plain
high-frequency prose makes collisions inevitable for any arm that learns it. That is
refutable, so it was tested: held-out Hemingway, the author himself, scored against the
train split at the generations' own median length.
HELD-OUT HEMINGWAY (never trained) 370 chunks 0.01 hit-rate mean-longest 0.1 max 10
base-unadapted 240 gens 0.00 0.0 0
ckpt850 (shipped) 240 gens 0.07 0.6 9
positive control (train vs train) 160
The hypothesis is false: the adapter reproduces train n-grams ~7x more often than the
author reproduces himself. That is real and is on the record. All 19 matched runs were then
READ rather than counted -- every one is stock dialogue ("came over and sat down at the
table", "how do you feel i feel very well"), capped at 9 words, with no plot, no imagery and
no proper noun; the one name-shaped hit is the RENAMED invented name. Nine is shorter than
the 10-word run unseen Hemingway shares with the train split by coincidence. Elevated rate,
zero protectable content. Hemingway is in copyright; lv-yarros is the in-line precedent,
also in copyright, shipped at 0.10 against a 0.07 control. Unload is 0.003 s.
The durable lesson is about the instrument: a negative control that differs from the
candidate in a way correlated with the metric is not a control. memorization_selfsim.py and
memorization_dump_matches.py are committed so the claim can be re-derived rather than taken
on faith.
SHIPPED ckpt850, NOT the loss minimum at step 1750. The two are indistinguishable on voice
-- 0.072 apart against a 0.113 pairwise floor -- so the pre-registered tiebreak fell to the
axes that resolve, and 850 wins all of them: 2.3x tighter seed spread (0.050 vs 0.113),
lower memorisation, less ran-on, half an epoch less overfit. ckpt1750's spread is one seed
(0.491, 0.449, 0.468, then 0.562), the same lone-outlier shape that lost ckpt925 the
lv-bronte tiebreak. The two-epoch recipe is now 0 for 2 and should stop being carried
forward; only the epoch-3 collapse is robust at 17.4x jitter.
servers/fv-ml1/ssh-target was a bare IP, so deploy-stack.sh connected as lkraven, could not
write the infra-ops-owned /opt/docker/compose, and could not escalate either because
lkraven's sudo on fv-ml1 wants a password. Now infra-ops@10.251.50.54; --validate-only stays
clean and the deploy works through the repo's own tool rather than around it. Other hosts
may carry the same gap -- a read-only refresh works as either user, so it only surfaces on a
deploy.
This commit is contained in:
@@ -0,0 +1,163 @@
|
||||
# `[2026-09-17]` lv-hemingway: SHIPPED on ckpt850 — the line's first clean voice pass, and one axis that needs reading
|
||||
|
||||
**Status: SHIPPED 2026-09-17 03:33 as `lv-hemingway` on `vllm-voices` (fv-ml1 GPU0 :8027),
|
||||
checkpoint-850.** Seat healthy 190 s after recreate, four models served
|
||||
(`voices-base`, `lv-yarros`, `lv-bronte`, `lv-hemingway`), GPU0 96,092 → **96,090 MiB** — a
|
||||
LoRA rides inside the existing seat and costs nothing. Adapter verified byte-identical to
|
||||
the checkpoint by sha256 across two hops.
|
||||
|
||||
Gate design **pre-registered before any generation existed**:
|
||||
`scripts/hemingway-corpus/GATE-PREREG.md`, commit `0bb4938`.
|
||||
|
||||
## The gate result — 3 arms × 60 held-out beats × 4 seeds = 240 generations per arm
|
||||
|
||||
| axis | result | numbers |
|
||||
|---|---|---|
|
||||
| **A. VOICE** | ✅ **PASS, 6.4×** | +0.413 delta_cb vs base, pairwise floor 0.064. Also clears the OLD all-arms floor (0.113) — **this verdict does not depend on the rule change** |
|
||||
| **B. NOT COPIED** | ⚠ **content clean, rate 7× the author's own** | 0.07 hit-rate, mean-longest 0.6, **max 9 words**. Base 0.00, **held-out Hemingway 0.01** |
|
||||
| **C. NO DAMAGE** | ✅ PASS | ran-on +0.08, on-beat −0.14, both inside a 0.217 floor; in-band 0.79 vs base 0.05 |
|
||||
|
||||
```
|
||||
same-author target (held-out Hemingway vs itself) delta_cb 0.364 <- best achievable
|
||||
ckpt1750 0.439
|
||||
ckpt850 (SHIPPED) 0.511
|
||||
base-unadapted 0.924
|
||||
```
|
||||
|
||||
⭐ **THE STRONGEST VOICE RESULT IN THE LINE. The span is 0.924 → 0.364 = 0.560 and ckpt850
|
||||
closed 73.8% of it (ckpt1750 86.6%)**, against lv-bronte's 48%. Power came from the corpus,
|
||||
not from a better method: 173 in-band val pairs allowed a **60-beat** fixture where Brontë
|
||||
had 44 in-band and could only run 30.
|
||||
|
||||
## ⚠⚠ AXIS B — THE COMFORTABLE EXPLANATION WAS WRONG, AND THE CONTROL IS THE ARTIFACT
|
||||
|
||||
`memorization_check.py` uses the **base-unadapted arm** as its negative control, and on this
|
||||
corpus that control is weak in one direction only — **it makes an innocent arm look guilty.**
|
||||
Base writes 18,035 words of *summary*; the adapted arms write 27,413 of *pastiche*. Text that
|
||||
does not imitate the register cannot collide with its n-grams, so base's 0.00 partly measures
|
||||
"different register", not "did not memorise".
|
||||
|
||||
The obvious hypothesis was that Hemingway's plain, high-frequency register makes 8-gram
|
||||
collisions inevitable for any arm that learns it. **That hypothesis is refutable, was tested,
|
||||
and is FALSE.** New control: **held-out Hemingway — the author himself, val text no arm
|
||||
trained on — scored against the train split**, chunked to the generations' own median length
|
||||
(101 words) so the comparison is like for like.
|
||||
|
||||
```
|
||||
sample n hit-rate mean-longest max
|
||||
HELD-OUT HEMINGWAY (never trained) 370 0.01 0.1 10
|
||||
base-unadapted 240 0.00 0.0 0
|
||||
ckpt1750 240 0.08 0.7 9
|
||||
ckpt850 (SHIPPED) 240 0.07 0.6 9
|
||||
positive control (train vs train) 160 <- not blind
|
||||
```
|
||||
|
||||
⭐⭐ **The adapter reproduces train-corpus word sequences ~7× more often than the author
|
||||
reproduces himself.** If the register explained it, real Hemingway would collide at the same
|
||||
rate; it collides at 0.01.
|
||||
|
||||
⭐ **And the exposure is still nil, which is a different question from the rate.** All 19
|
||||
matched runs were READ, not counted. Every one is stock dialogue — `i don t think so the girl
|
||||
said`, `came over and sat down at the table`, `how do you feel i feel very well`. No plot, no
|
||||
imagery, no distinctive phrase, **no proper noun** (the one name-shaped hit, `swift tristan`,
|
||||
is the RENAMED invented name, not Hemingway's). The longest run is **9 words — shorter than
|
||||
the 10-word run genuinely unseen Hemingway shares with the train split by coincidence.**
|
||||
|
||||
What is being reproduced is the *grammar of his dialogue*, which is the thing the adapter
|
||||
exists to learn, rendered in the commonest words in English. **Elevated rate, zero
|
||||
protectable content.** Hemingway is in copyright; the in-line precedent is lv-yarros, also in
|
||||
copyright, shipped at 0.10 against a 0.07 control. Unload is 0.003 s and one compose line.
|
||||
|
||||
⚠ **The durable lesson is about the instrument, not this adapter: a negative control that
|
||||
differs from the candidate in a way CORRELATED with the metric is not a control.** Always ask
|
||||
what the metric returns for a known-innocent sample *in the same register*.
|
||||
|
||||
## Why ckpt850 and NOT ckpt1750, the loss minimum
|
||||
|
||||
ckpt1750 has the better point estimate on voice (0.439 vs 0.511) and **it is not usable**:
|
||||
|
||||
```
|
||||
gap between candidates 0.072
|
||||
pairwise floor max(0.113, 0.050) 0.113 -> NOT resolvable
|
||||
```
|
||||
|
||||
Indistinguishable, so the pre-registered tiebreak falls to the axes that resolve — and
|
||||
**ckpt850 wins every one**:
|
||||
|
||||
| | ckpt850 (shipped) | ckpt1750 |
|
||||
|---|---|---|
|
||||
| seed spread | **0.050** | 0.113 — **2.3× wider** |
|
||||
| memorisation hit-rate / mean-longest | **0.07 / 0.6** | 0.08 / 0.7 |
|
||||
| ran-on | **0.08** | 0.12 |
|
||||
| epoch | **0.959** | 1.973 |
|
||||
|
||||
ckpt1750's spread is one seed: 0.491, 0.449, 0.468, then **0.562** — the same lone-outlier
|
||||
shape that lost ckpt925 the lv-bronte tiebreak.
|
||||
|
||||
⭐ **THE TWO-EPOCH RECIPE DID NOT TRANSFER HERE EITHER — it is now 0 for 2.** Hemingway's
|
||||
minimum really is step 1750, but step 850 is **+0.0040 against a 0.0044 median neighbour
|
||||
jitter**, with three checkpoints inside one jitter of the best. Epoch 2 buys nothing that
|
||||
resolves and costs 2.3× the variance. Only the epoch-3 collapse is robust: **+0.0762 = 17.4×
|
||||
jitter**, which is why `adapter/` was never gated. **Stop carrying "two epochs on a
|
||||
three-epoch schedule" forward; read the curve and prefer the earlier tied checkpoint.**
|
||||
|
||||
## Pre-flight: the beat leak IS present in Hemingway, and the fixture is clean
|
||||
|
||||
`audit_pairs_sourcenames.py` (new, commit `0bb4938`) closes the blind spot `leak_gate.py` has
|
||||
by construction. Controls green every run: 941/941 surfaces found in the unrenamed source,
|
||||
nonce absent from both trees, 6/6 planted names detected.
|
||||
|
||||
```
|
||||
train beats 70 of 7,094 (0.96%) Santiago x16, Catherine x7, Rinaldi x3, Brett, Harry,
|
||||
Jake, Pablo, Nick, Maria, Helen ... 36 distinct
|
||||
train responses 0 of 7,294 -- the rename itself held perfectly
|
||||
val beats 0 of 200 -- THE EVAL FIXTURE IS CLEAN; the gate is unconfounded
|
||||
```
|
||||
|
||||
⭐ The beat-only signature is exactly lv-bronte's. **Yarros's and Hemingway's earlier clean
|
||||
runs were never evidence of immunity** — they predate the detector.
|
||||
|
||||
**Cross-validated on real data** where the answer was already recorded: the fixed Brontë
|
||||
pairs return **0 of 3,858** (matching "0 leaks across 3,858 pairs"), and
|
||||
`pairs-full.CONTAMINATED.jsonl` returns **15 of 792 = 1.89%** with Rochester ×6, Jane,
|
||||
Brocklehurst ×2, Beck, Fairfax, Burns, Helen, Eyre — against a record of "13 of the first 714
|
||||
beats (1.8%)" with the same names. An independently written instrument reproducing a
|
||||
documented finding at the right magnitude is what makes its zeroes mean *absent*, not *blind*.
|
||||
|
||||
`--filter-out` produces a clean **7,024-pair** set in one command (70 dropped, 0.99%),
|
||||
verified by re-audit at 0 of 7,024. **A retrain on it is the operator's call, not done.**
|
||||
|
||||
## ⚠ A SECOND corpus defect, measured and NOT acted on
|
||||
|
||||
`audit_entity_map.py` (new, commit `051b99e`) is the mirror of `audit_stoplist.py`: it finds
|
||||
surfaces wrongly held **IN** the entity map, which `leak_gate.py` cannot see because it only
|
||||
ever asks whether the author's names are GONE, never whether non-names were spared.
|
||||
|
||||
```
|
||||
positive control `other` 764/1356 article-preceded = 0.56
|
||||
negative control 100 honorific-confirmed people, highest Inglés 0.26, bulk 0.00-0.06
|
||||
FLAGGED 130 of 946 surfaces · 1,616 instances · 0.162% of corpus words
|
||||
```
|
||||
|
||||
`African`, `Chinese`, `Basques`, `Republican`, `Communist`, `X-ray`, `Coca-Cola`, `Ritz`,
|
||||
`Prado`, `Cezanne` were all renamed into invented proper nouns. **Some flags are correct
|
||||
renames** — `the Widow`, `the Informer` are genuine Hemingway epithet-names — so every hit is
|
||||
reported for reading, never auto-removed. Plus **16 bare initials in the map**, `C` at 274
|
||||
occurrences: the same class as the `G` caught by hand about to be renamed 248 times.
|
||||
|
||||
At 0.162% of words this did not block the ship. It is the thing to fix first if a corpus
|
||||
rebuild ever happens.
|
||||
|
||||
## Artefacts
|
||||
|
||||
`gx10:~/lv-hemingway/` (corpus-clean, corpus-renamed, beats-hemingway-60.json + sidecar,
|
||||
eval-hemingway.sh, voice-prep.py, eval.log), `gx10:~/r49-runs/hemingway-4b-pairs-3ep/`
|
||||
(54 checkpoints kept), `gx10:~/r49-runs/hemingway-eval/` (three arms × 240 generations,
|
||||
memorization.txt, voice_distance.txt, score.*.txt).
|
||||
`fv-ml1:/tank/aimodels/voice-adapters/lv-hemingway-4b-v1/` (adapter + a README carrying the
|
||||
axis-B caveat, so it cannot be read as clean by anyone who finds the adapter without this).
|
||||
Commits `0bb4938` `051b99e` `5e66114` `2e9b118`.
|
||||
|
||||
Related: [[2026-09-17-lv-bronte-gate]], [[2026-09-16-lv-hemingway-corpus]],
|
||||
[[2026-09-16-lv-voices-line]], [[2026-09-16-voices-seat-lora]],
|
||||
[[2026-09-17-beat-contamination-leak]].
|
||||
+33
-28
@@ -1,6 +1,6 @@
|
||||
# Persistent memory — eshpfi-management
|
||||
|
||||
_Last updated: 2026-09-17 ~01:30 PT (lv-bronte SHIPPED with a FAILED voice axis on the record; a beat-contamination leak class the corpus gate cannot see, found and patched; audit_stoplist added; next goal is landing lv-hemingway, which is trained but un-gated.)_
|
||||
_Last updated: 2026-09-17 ~03:40 PT (lv-hemingway SHIPPED on ckpt850 — the line's strongest voice pass; the axis-B negative control was found to be the wrong one and replaced with the author's own self-similarity; the floor rule went pairwise, which retroactively passes lv-bronte.)_
|
||||
|
||||
> **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its
|
||||
> `Written:` stamp is under **8 hours** old, read it (it carries the in-flight
|
||||
@@ -117,38 +117,35 @@ no longer deployed sidecars here. See Recent decisions.)
|
||||
|
||||
_As of 2026-09-17 ~01:30 PT._
|
||||
|
||||
### Next goal: land `lv-hemingway`. It is TRAINED; nothing else has been done to it.
|
||||
### SHIPPED — `lv-hemingway` on ckpt850, the line's first clean voice pass
|
||||
|
||||
**Ship candidate is `gx10:~/r49-runs/hemingway-4b-pairs-3ep/checkpoints/checkpoint-1750`**
|
||||
(epoch 1.97, eval_loss 2.2783 — the loss minimum). The run finished 2026-09-16 18:27, 2,661
|
||||
steps in 3:17:34, clean. The end-of-run `adapter/` is **0.0763 worse** (2.3546) — epoch 3
|
||||
overfits and plateaus. Do NOT ship `adapter/`; the run's own log says so.
|
||||
Live on `vllm-voices` (fv-ml1 GPU 0 :8027) beside `voices-base`, `lv-yarros`, `lv-bronte`.
|
||||
**VOICE +0.413 delta_cb at 6.4x its floor** — closes 73.8% of the achievable span, and clears
|
||||
the OLD all-arms floor too, so the verdict does not lean on the rule change. **DAMAGE clean.**
|
||||
⚠ **MEMORISATION is the axis to read**: 0.07 hit-rate against **held-out Hemingway's own
|
||||
0.01** — ~7x the author's self-collision rate — but all 19 matched runs were read and every
|
||||
one is stock dialogue capped at **9 words**, no proper noun, no plot. Elevated rate, zero
|
||||
protectable content; in-copyright author, so the fair-use call is the operator's.
|
||||
→ `persistent-memory.d/2026-09-17-lv-hemingway-gate.md`
|
||||
|
||||
**The v2 gate has NOT been run on it.** Everything it needs now exists and is parameterised,
|
||||
built during the lv-bronte run tonight — reuse it rather than rebuilding:
|
||||
⚠ **ckpt850, NOT the loss minimum at 1750** — the two are indistinguishable on voice (0.072
|
||||
gap vs a 0.113 floor) so the tiebreak fell to the resolving axes, and 850 wins all of them
|
||||
(2.3x tighter spread, lower memorisation, less ran-on, half an epoch less overfit).
|
||||
**The two-epoch recipe is now 0 for 2.** Read the curve; prefer the earlier tied checkpoint.
|
||||
|
||||
1. `scripts/r49-corpus/build_beat_fixture.py --pairs <hemingway val pairs> --out beats-hemingway-30.json --sidecar ... -n 30` — refuses any split but `val`.
|
||||
2. `scripts/r49-corpus/gen_beats_chat_yarros.py --system-from <pairs>.provenance.json` — **the `--system-from` flag is mandatory**; the harness's hardcoded SYS is Yarros's and driving a Hemingway arm with it confounds the carrier change with a prompt change.
|
||||
3. `scripts/yarros-corpus/memorization_check.py --eval-dir ... --corpus ... --glob 'beats5.*.jsonl'` — **now takes paths**; its old hardcoded Yarros defaults would have compared a Hemingway arm against the Yarros corpus and reported a meaningless clean zero.
|
||||
4. `scripts/r49-corpus/voice_distance.py <renamed-corpus> <eval-dir>` — needs `voice.<arm>.jsonl` files with a `continuation` field, and identifies the control by the substring **`unadapted`** in the arm name. Adapt with a script like `gx10:~/lv-bronte/voice-prep.py`.
|
||||
### OPEN for the operator — two measured defects, neither acted on
|
||||
|
||||
⚠ **Hemingway's pairs were built BEFORE the `--source-entities` beat filter existed.** Measured
|
||||
on Brontë: 1.8% of generated beats named the author's real characters, because the beat-writing
|
||||
model recognises the book and restores canonical names — a leak the corpus gate structurally
|
||||
cannot see. Exposure scales with how famous the book is. **Re-verify the Hemingway pairs against
|
||||
its entity map before trusting them**, and regenerate with `--source-entities` if they leak.
|
||||
1. **Hemingway's TRAIN beats carry the source-name leak: 70 of 7,094 (0.96%)** (Santiago x16,
|
||||
Catherine x7, Rinaldi x3 ...), responses 0 of 7,294, **val 0 of 200 so the gate itself is
|
||||
unconfounded**. `audit_pairs_sourcenames.py --filter-out` yields a verified-clean
|
||||
7,024-pair set in one command. **Retrain ~3h17 + re-gate ~1h40, unattended.**
|
||||
2. **130 of 946 entity-map surfaces are probably not names** and were renamed anyway
|
||||
(`African`, `Chinese`, `X-ray`, `Coca-Cola`, `Ritz`, `Prado`), plus **16 bare initials
|
||||
incl. `C` at 274 occurrences** — 1,616 instances, 0.162% of corpus words.
|
||||
`audit_entity_map.py` finds them; they need READING, not auto-removal, because
|
||||
`the Widow` / `the Informer` are genuine epithet-names that should be renamed.
|
||||
|
||||
⚠ **Do not assume the two-epoch recipe.** It held for Yarros and Hemingway and did NOT hold for
|
||||
Brontë. Read the eval curve; Hemingway's minimum genuinely is at 1750, which is already known.
|
||||
|
||||
### SHIPPED tonight — `lv-bronte`, with a FAILED voice axis on the record
|
||||
|
||||
Live on `vllm-voices` (fv-ml1 GPU 0 :8027) as `lv-bronte` beside `voices-base` and `lv-yarros`.
|
||||
Checkpoint-475. **It did not pass its voice gate** (+0.193 delta_cb against a 0.251 measured
|
||||
floor); shipped because it is additive, reversible, and clean on the safety axis (8-gram overlap
|
||||
0.00, identical to the never-saw-it control) on a public-domain corpus. The caveat is written
|
||||
into the commit, the compose file, and a README beside the adapter on NFS.
|
||||
→ `persistent-memory.d/2026-09-17-lv-bronte-gate.md`
|
||||
Both would be fixed in one pass if a corpus rebuild happens. Neither blocks anything today.
|
||||
|
||||
### ESH is on Verizon failover — Cityside Fiber failed TWICE tonight
|
||||
|
||||
@@ -180,6 +177,14 @@ before this session began. ⚠ **Do NOT commit them** — untouched and delibera
|
||||
|
||||
## Recent decisions
|
||||
|
||||
- `[2026-09-17]` ⭐⭐⭐ **lv-hemingway SHIPPED (ckpt850) with the line's strongest voice result — and the memorisation control it passed turned out to be the WRONG control.** Voice +0.413 delta_cb at 6.4x the floor, closing 73.8% of the achievable span (lv-bronte closed 48%). ⚠ `memorization_check.py` uses the base-unadapted arm as its negative control, but base writes 18,035 words of summary against the adapted arms' 27,413 of pastiche — **text that does not imitate the register cannot collide with its n-grams**, so a 0.00 there means "different register", not "did not memorise". The right reference is the author himself: **held-out Hemingway against the train split collides at 0.01 while the adapter does at 0.07**, so the comfortable "his plain register makes collisions inevitable" story is FALSE and was refuted rather than assumed. All 19 matched runs were READ: stock dialogue, max **9 words**, no proper noun — shorter than the 10-word run unseen Hemingway shares with the train split by coincidence. ⭐ **A negative control that differs from the candidate in a way correlated with the metric is not a control.** → `persistent-memory.d/2026-09-17-lv-hemingway-gate.md`
|
||||
- `[2026-09-17]` ⭐⭐ **The v2 voice floor is now PAIRWISE, and it retroactively passes lv-bronte.** lv-bronte's ckpt475 shipped as a voice-axis FAILURE at +0.193 against a 0.251 floor contributed entirely by ckpt925 — a third arm nobody was shipping, on one outlier seed. Scored against the arm it was actually compared to the floor is 0.091 and it clears at **2.1x**. The rule was changed **prospectively**, pre-registered for lv-hemingway before any Hemingway number existed, on an argument independent of the answer: the sampling variability of a difference A−B depends on A and B, never on a third arm C. The previous session found the defect and deliberately declined to exploit it; this follows from fixing it. lv-hemingway passes under **both** rules, so its verdict does not lean on the change. Caveats amended append-only in the compose, the NFS README and the gate record. Commits `0bb4938` `2e9b118`.
|
||||
- `[2026-09-17]` ⭐ **The beat-contamination leak IS present in Hemingway — 70 of 7,094 train beats (0.96%), 0 of 200 val.** `scripts/r49-corpus/audit_pairs_sourcenames.py` closes the blind spot `leak_gate.py` has by construction (it reads the corpus and the renamed copies, never the generated beats). Cross-validated on real data: the fixed Brontë pairs return 0 of 3,858 and `pairs-full.CONTAMINATED` returns 15 of 792 = 1.89% with the recorded names. `--filter-out` yields a verified-clean 7,024-pair set in one command; the retrain is the operator's call. **The val split being clean is why the gate could run at all.**
|
||||
- `[2026-09-17]` ⭐ **`audit_entity_map.py` — the rename can DAMAGE the prose and no gate will ever say so.** Mirror of `audit_stoplist.py`: surfaces wrongly held IN the map rather than out of it. 130 of 946 Hemingway surfaces flagged (1,616 instances, 0.162% of words) — `African`, `Chinese`, `X-ray`, `Coca-Cola`, `Ritz`, `Prado` renamed into invented names — plus 16 bare initials incl. `C` at 274 occurrences. Signal is a preceding article; controls derived from the corpus, not hand-picked. Every hit reported for READING: `the Widow` and `the Informer` are genuine epithet-names that should be renamed. Commit `051b99e`.
|
||||
- `[2026-09-17]` **The two-epoch recipe is now 0 for 2 and should stop being carried forward.** Hemingway's eval minimum is step 1750, but step 850 is +0.0040 against a 0.0044 median neighbour jitter — three checkpoints inside one jitter — and 850 won every resolving axis (2.3x tighter seed spread, lower memorisation, less ran-on). Same outcome as Brontë. What IS robust on this schedule is the epoch-3 collapse: +0.0762 = **17.4x jitter**.
|
||||
- `[2026-09-17]` **gitea was reaching the PUBLIC route from every repo on nh3-dev** — brokkr-smithy, sleipnir, Galdrabok, kvasir — and brokkr-smithy is pushed several times a week, so the fail2ban trigger was live, not dormant. Measured before acting (no split-horizon rewrite, no ssh alias, `ssh -G` confirmed port 22 to 38.120.12.44). Fixed by overriding the NAME once in `~/.ssh/config` rather than rewriting N remotes, so fresh clones and unaudited repos are covered too. Verified with a real `git ls-remote`, not by inspection. Commit `dcc1abc`. Flagged by brokkr-smithy-dev; `vh/imogen` created for them the same session.
|
||||
- `[2026-09-17]` **`servers/fv-ml1/ssh-target` was bare `10.251.50.54`, so `deploy-stack.sh` connected as `lkraven` and could not write the infra-ops-owned `/opt/docker/compose/`** — and lkraven's sudo on fv-ml1 needs a password, so `DEPLOY_SUDO=1` failed too. Now `infra-ops@10.251.50.54`; `--validate-only` still clean, deploy works. ⚠ Other hosts' `ssh-target` files may carry the same gap — a read-only refresh works as either user, so the fault only surfaces on a deploy.
|
||||
|
||||
- `[2026-09-17]` ⭐⭐ **A leak class the corpus gate structurally CANNOT see: the beat-writing model recognises the book and restores the author's real character names.** 1.8% of Brontë beats named Rochester/Jane/Brocklehurst while 0 responses did. Worst for public-domain classics; Yarros and Hemingway's clean runs are NOT evidence they are immune. Patched as a `sourcename` reject + `--source-entities`. → `persistent-memory.d/2026-09-17-beat-contamination-leak.md`
|
||||
- `[2026-09-17]` ⭐ **A stoplist entry is an assertion the leak gate can no longer check** — stoplisting removes a surface from the entity map, so a wrongly stoplisted CHARACTER is an undetectable leak. Three were wrong on Brontë (Leaven, Pierrot, Samuel); `scripts/r49-corpus/audit_stoplist.py` finds them by honorific and now gates the pipeline. Commit `8bb7686`.
|
||||
- `[2026-09-17]` **ESH: Cityside Fiber failed TWICE (19:09 and ~01:06); operator switched WAN1 to DHCP to restore service and has a ticket for the static.** crowdsec `esh` allowlist carries both failover egresses with 7-day expiries — the rotation-fragility is live. → `persistent-memory.d/2026-09-17-esh-fiber-outages.md`
|
||||
|
||||
@@ -0,0 +1,82 @@
|
||||
#!/bin/bash
|
||||
# lv-hemingway v2 gate. Design is FROZEN in scripts/hemingway-corpus/GATE-PREREG.md
|
||||
# and was written before this script ever ran. Do not edit the arms, the fixture
|
||||
# size or the seeds to chase a result -- re-run, do not re-tune.
|
||||
#
|
||||
# THREE ARMS. ckpt1750 is the eval-loss minimum (2.2783, epoch 1.973). ckpt850
|
||||
# (2.2823, epoch 0.959) is +0.0040 against a 0.0044 median neighbour jitter, i.e.
|
||||
# TIED -- the loss curve cannot pick between them, and on Bronte the earlier
|
||||
# epoch-1 checkpoint won the tiebreak on the axes that do resolve. `base` is the
|
||||
# negative control for memorisation (it never saw the corpus) and the voice
|
||||
# baseline. adapter/ (epoch 3.0, +0.0762 = 17.4x jitter) is NOT gated: that one
|
||||
# the loss curve settles on its own.
|
||||
set -o pipefail
|
||||
cd ~/lv-hemingway || exit 1
|
||||
PY=/home/infra-ops/ml/.venv/bin/python
|
||||
RUN=~/r49-runs/hemingway-4b-pairs-3ep
|
||||
OUT=~/r49-runs/hemingway-eval
|
||||
BEATS=beats-hemingway-60.json
|
||||
PROV=pairs/pairs-full.jsonl.provenance.json
|
||||
SEEDS="1234 5678 9012 3456"
|
||||
mkdir -p "$OUT"
|
||||
log(){ echo "[eval $(date +%H:%M:%S)] $*"; }
|
||||
|
||||
# --system-from is MANDATORY. The harness's built-in SYS is Yarros's; driving a
|
||||
# Hemingway arm with it would confound the adapter change with a prompt change.
|
||||
# This binds the eval prompt to the one the run actually trained under (verified:
|
||||
# run provenance system_prompt == pairs provenance system_prompt).
|
||||
[ -f "$PROV" ] || { log "MISSING $PROV"; exit 1; }
|
||||
[ -f "$BEATS" ] || { log "MISSING $BEATS"; exit 1; }
|
||||
|
||||
for arm in base:NONE ckpt1750:$RUN/checkpoints/checkpoint-1750 ckpt850:$RUN/checkpoints/checkpoint-850; do
|
||||
name=${arm%%:*}; path=${arm#*:}
|
||||
if [ "$name" != "base" ] && [ ! -d "$path" ]; then log "MISSING $path"; exit 1; fi
|
||||
if [ -s "$OUT/beats5.$name.jsonl" ]; then
|
||||
log "arm $name already has $(wc -l < "$OUT/beats5.$name.jsonl") generations -- skipping"
|
||||
continue
|
||||
fi
|
||||
log "arm $name"
|
||||
if [ "$name" = "base" ]; then
|
||||
"$PY" scripts/r49-corpus/gen_beats_chat_yarros.py \
|
||||
--base ~/carriers/Qwen3-4B-Instruct --beats "$BEATS" \
|
||||
--out "$OUT/beats5.$name.jsonl" --arm "$name" --seeds $SEEDS \
|
||||
--system-from "$PROV" || exit 1
|
||||
else
|
||||
"$PY" scripts/r49-corpus/gen_beats_chat_yarros.py \
|
||||
--base ~/carriers/Qwen3-4B-Instruct --adapter "$path" --beats "$BEATS" \
|
||||
--out "$OUT/beats5.$name.jsonl" --arm "$name" --seeds $SEEDS \
|
||||
--system-from "$PROV" || exit 1
|
||||
fi
|
||||
log " $(wc -l < "$OUT/beats5.$name.jsonl") generations"
|
||||
done
|
||||
|
||||
log "AXIS B -- MEMORISATION (corpus = the renamed copies the adapter trained on)"
|
||||
# --corpus and --eval-dir are passed explicitly: the script's Yarros defaults would
|
||||
# compare a Hemingway arm against the YARROS corpus and report a clean zero that
|
||||
# means "different book", not "did not memorise".
|
||||
"$PY" scripts/yarros-corpus/memorization_check.py \
|
||||
--eval-dir "$OUT" --corpus corpus-renamed/copies --glob 'beats5.*.jsonl' --strip 'beats5.' -n 8 \
|
||||
2>&1 | tee "$OUT/memorization.txt"
|
||||
|
||||
log "AXIS C -- DAMAGE (ran-on / out-of-band), ckpt1750 vs base"
|
||||
"$PY" scripts/yarros-corpus/score_beats.py \
|
||||
--arm base="$OUT/beats5.base.jsonl" \
|
||||
--arm ckpt1750="$OUT/beats5.ckpt1750.jsonl" \
|
||||
--arm ckpt850="$OUT/beats5.ckpt850.jsonl" \
|
||||
--baseline base --candidate ckpt1750 --metric-source raw \
|
||||
--out "$OUT/score.ckpt1750.json" 2>&1 | tee "$OUT/score.ckpt1750.txt"
|
||||
log "AXIS C -- DAMAGE, ckpt850 vs base"
|
||||
"$PY" scripts/yarros-corpus/score_beats.py \
|
||||
--arm base="$OUT/beats5.base.jsonl" \
|
||||
--arm ckpt1750="$OUT/beats5.ckpt1750.jsonl" \
|
||||
--arm ckpt850="$OUT/beats5.ckpt850.jsonl" \
|
||||
--baseline base --candidate ckpt850 --metric-source raw \
|
||||
--out "$OUT/score.ckpt850.json" 2>&1 | tee "$OUT/score.ckpt850.txt"
|
||||
|
||||
log "AXIS A -- VOICE (delta_cb vs held-out Hemingway)"
|
||||
"$PY" voice-prep.py || exit 1
|
||||
"$PY" scripts/r49-corpus/voice_distance.py corpus-renamed "$OUT" --author Hemingway \
|
||||
2>&1 | tee "$OUT/voice_distance.txt"
|
||||
|
||||
echo "rc=0" > ~/lv-hemingway/.eval-complete
|
||||
log "done"
|
||||
@@ -0,0 +1,47 @@
|
||||
"""Adapt the lv-hemingway arms for voice_distance.py, and refuse if its reference is empty.
|
||||
|
||||
voice_distance.py expects:
|
||||
- files matching voice.<arm>.jsonl in the eval dir
|
||||
- a `continuation` field per record (gen_beats_chat writes `raw`)
|
||||
- a `seed` field (present)
|
||||
- the control arm's NAME to contain the substring "unadapted" -- it picks the
|
||||
control by that substring, so `base` alone would leave the control
|
||||
unidentified and the whole vs-control table would silently be empty.
|
||||
- a reference built from corpus records whose split == "val"
|
||||
"""
|
||||
import json, sys
|
||||
from collections import Counter
|
||||
from pathlib import Path
|
||||
|
||||
CORP = Path("/home/infra-ops/lv-hemingway/corpus-renamed/copies")
|
||||
EVAL = Path("/home/infra-ops/r49-runs/hemingway-eval")
|
||||
|
||||
c = Counter()
|
||||
n = 0
|
||||
for p in sorted(CORP.glob("*.jsonl")):
|
||||
for line in p.read_text(encoding="utf-8").splitlines():
|
||||
if not line.strip():
|
||||
continue
|
||||
r = json.loads(line)
|
||||
n += 1
|
||||
c[r.get("split")] += 1
|
||||
print(f"corpus records: {n} split values: {dict(c)}")
|
||||
if c.get("val", 0) == 0:
|
||||
print("== REFUSING: no split=val records; voice_distance would build an EMPTY reference")
|
||||
print(" and every delta_cb would be meaningless rather than absent.")
|
||||
sys.exit(1)
|
||||
|
||||
NAMES = {"base": "base-unadapted", "ckpt1750": "ckpt1750", "ckpt850": "ckpt850"}
|
||||
for src_arm, out_arm in NAMES.items():
|
||||
src = EVAL / f"beats5.{src_arm}.jsonl"
|
||||
if not src.exists():
|
||||
print(f"== missing {src}")
|
||||
sys.exit(1)
|
||||
rows = [json.loads(l) for l in src.read_text(encoding="utf-8").splitlines() if l.strip()]
|
||||
out = EVAL / f"voice.{out_arm}.jsonl"
|
||||
with out.open("w", encoding="utf-8") as fh:
|
||||
for r in rows:
|
||||
fh.write(json.dumps({"id": r["id"], "seed": r["seed"],
|
||||
"continuation": r["raw"]}, ensure_ascii=False) + "\n")
|
||||
print(f" {src.name} -> {out.name} ({len(rows)} records)")
|
||||
print("ready")
|
||||
@@ -0,0 +1,38 @@
|
||||
"""What ARE the verbatim 8-gram hits? A rate is not a judgement.
|
||||
|
||||
0.09 against a 0.00 control reads alarming; 0.00 against 0.09 could also be an
|
||||
artefact of the control writing a different register entirely (the base arm wrote
|
||||
18,035 words of summary against the adapted arms' 27,413 of pastiche, and text that
|
||||
does not imitate the style trivially fails to match its n-grams). The only way to
|
||||
tell a memorised passage from a common English run is to read them.
|
||||
"""
|
||||
import json, re, sys, pathlib
|
||||
from collections import Counter
|
||||
CORP = pathlib.Path(sys.argv[1]); EVAL = pathlib.Path(sys.argv[2]); N = 8
|
||||
def norm(t): return re.findall(r"[a-z']+", t.lower())
|
||||
words = []
|
||||
for f in sorted(CORP.glob("*.copy0.jsonl")):
|
||||
for l in f.read_text(encoding="utf-8").splitlines():
|
||||
words.extend(norm(json.loads(l)["text"]))
|
||||
grams = {" ".join(words[i:i+N]) for i in range(len(words)-N+1)}
|
||||
print(f"corpus {len(words):,} words, {len(grams):,} distinct {N}-grams\n")
|
||||
for arm in ("base", "ckpt1750", "ckpt850"):
|
||||
p = EVAL/f"beats5.{arm}.jsonl"
|
||||
if not p.exists(): continue
|
||||
rows = [json.loads(l) for l in p.read_text(encoding="utf-8").splitlines() if l.strip()]
|
||||
found = Counter()
|
||||
for r in rows:
|
||||
w = norm(r["raw"]); i = 0
|
||||
while i <= len(w)-N:
|
||||
g = " ".join(w[i:i+N])
|
||||
if g in grams:
|
||||
k = N
|
||||
while i+k < len(w) and " ".join(w[i+k-N+1:i+k+1]) in grams: k += 1
|
||||
found[" ".join(w[i:i+k])] += 1
|
||||
i += k
|
||||
else:
|
||||
i += 1
|
||||
print(f"=== {arm}: {len(rows)} gens, {sum(found.values())} matched runs, {len(found)} distinct")
|
||||
for g, c in found.most_common(40):
|
||||
print(f" x{c} [{len(g.split())}w] {g}")
|
||||
print()
|
||||
@@ -0,0 +1,71 @@
|
||||
"""THE CONTROL AXIS B WAS MISSING: how much does the author collide with HIMSELF?
|
||||
|
||||
memorization_check.py compares each arm against the TRAIN corpus and uses the
|
||||
base-unadapted arm as the negative control. On Hemingway that control is weak in a way
|
||||
it was not on Brontë, and the weakness runs one way only -- it makes an innocent arm
|
||||
look guilty:
|
||||
|
||||
* base-unadapted writes 18,035 words of summary prose; the adapted arms write 27,413
|
||||
of pastiche. Text that does not imitate the register cannot collide with its
|
||||
n-grams, so 0.00 measures "different register", not "did not memorise".
|
||||
* Hemingway's register IS short, plain, high-frequency English with heavy unattributed
|
||||
dialogue. An arm that SUCCEEDS at the voice task must start colliding with 8-grams
|
||||
built out of the commonest word sequences in the language.
|
||||
|
||||
So the honest reference is not the base arm. It is HELD-OUT HEMINGWAY -- val text no arm
|
||||
trained on, written by the author himself, which by construction did not memorise the
|
||||
train split. Whatever rate that produces is the floor the metric returns for innocent
|
||||
text in this register, and only an excess over THAT is evidence of copying.
|
||||
|
||||
Chunks are cut to the generations' own length so the comparison is like-for-like: a
|
||||
longer sample has more chances to collide.
|
||||
"""
|
||||
import json, re, sys, pathlib, statistics as st
|
||||
CORP = pathlib.Path(sys.argv[1]); EVAL = pathlib.Path(sys.argv[2]); N = 8
|
||||
|
||||
def norm(t): return re.findall(r"[a-z']+", t.lower())
|
||||
|
||||
train_words, val_texts = [], []
|
||||
for f in sorted(CORP.glob("*.copy0.jsonl")):
|
||||
for l in f.read_text(encoding="utf-8").splitlines():
|
||||
r = json.loads(l)
|
||||
(val_texts.append(r["text"]) if r.get("split") == "val" else train_words.extend(norm(r["text"])))
|
||||
grams = {" ".join(train_words[i:i+N]) for i in range(len(train_words)-N+1)}
|
||||
print(f"train (copy0, split=train): {len(train_words):,} words, {len(grams):,} distinct {N}-grams")
|
||||
|
||||
def longest(w):
|
||||
best = 0; i = 0
|
||||
while i <= len(w)-N:
|
||||
if " ".join(w[i:i+N]) in grams:
|
||||
k = N
|
||||
while i+k < len(w) and " ".join(w[i+k-N+1:i+k+1]) in grams: k += 1
|
||||
best = max(best, k); i += 1
|
||||
else: i += 1
|
||||
return best
|
||||
|
||||
arm_lens = []
|
||||
for arm in ("ckpt1750", "ckpt850", "base"):
|
||||
p = EVAL/f"beats5.{arm}.jsonl"
|
||||
if p.exists():
|
||||
arm_lens += [len(norm(json.loads(l)["raw"]))
|
||||
for l in p.read_text(encoding="utf-8").splitlines() if l.strip()]
|
||||
CHUNK = int(st.median(arm_lens))
|
||||
print(f"median generation length across arms: {CHUNK} words -- val is chunked to match\n")
|
||||
|
||||
vw = norm("\n".join(val_texts))
|
||||
chunks = [vw[i:i+CHUNK] for i in range(0, len(vw)-CHUNK+1, CHUNK)]
|
||||
longs = [longest(c) for c in chunks]
|
||||
hits = sum(1 for x in longs if x >= N)
|
||||
print(f"{'sample':<34} {'n':>5} {'hit-rate':>9} {'mean-longest':>13} {'max':>5}")
|
||||
print("-"*70)
|
||||
print(f"{'HELD-OUT HEMINGWAY (never trained)':<34} {len(chunks):>5} {hits/len(chunks):>9.2f} "
|
||||
f"{sum(longs)/len(longs):>13.1f} {max(longs):>5}")
|
||||
for arm in ("base", "ckpt1750", "ckpt850"):
|
||||
p = EVAL/f"beats5.{arm}.jsonl"
|
||||
if not p.exists(): continue
|
||||
rows = [json.loads(l) for l in p.read_text(encoding="utf-8").splitlines() if l.strip()]
|
||||
L = [longest(norm(r["raw"])) for r in rows]
|
||||
h = sum(1 for x in L if x >= N)
|
||||
print(f"{arm:<34} {len(rows):>5} {h/len(rows):>9.2f} {sum(L)/len(L):>13.1f} {max(L):>5}")
|
||||
print(f"\npositive control (a train slice vs train): longest = {longest(train_words[1000:1160])} "
|
||||
f"(must be large, else blind)")
|
||||
@@ -1 +1 @@
|
||||
10.251.50.54
|
||||
infra-ops@10.251.50.54
|
||||
|
||||
@@ -114,6 +114,33 @@ services:
|
||||
# See /adapters/lv-bronte-4b-v1/README.md and
|
||||
# persistent-memory.d/2026-09-17-lv-bronte-gate.md
|
||||
- lv-bronte=/adapters/lv-bronte-4b-v1
|
||||
# lv-hemingway: checkpoint-850 (epoch 0.959), NOT the loss minimum at step 1750.
|
||||
# The two are indistinguishable on voice — 0.072 apart against a 0.113 pairwise
|
||||
# floor — so the tiebreak fell to the axes that resolve, and 850 wins all of them:
|
||||
# 2.3x tighter seed spread (0.050 vs 0.113), lower memorisation (0.07 vs 0.08),
|
||||
# less ran-on (0.08 vs 0.12), and half an epoch less overfit.
|
||||
#
|
||||
# VOICE ✅ +0.413 delta_cb at 6.4x the pairwise floor — the strongest result in this
|
||||
# line, closing 73.8% of the span between the unadapted carrier and held-out
|
||||
# Hemingway itself. It also clears the OLD all-arms floor (0.113), so this verdict
|
||||
# does not depend on the 2026-09-17 rule change.
|
||||
# DAMAGE ✅ ran-on +0.08 and on-beat −0.14, both inside a 0.217 floor.
|
||||
#
|
||||
# ⚠ MEMORISATION IS THE AXIS TO READ BEFORE QUOTING THIS ONE AS CLEAN. 0.07 hit-rate
|
||||
# against a base control of 0.00 — but that control is weak here, because base writes
|
||||
# summary and cannot collide with a register it does not imitate. The honest
|
||||
# reference is the author himself: HELD-OUT HEMINGWAY scored against the train split
|
||||
# collides at 0.01. So the adapter reproduces train n-grams ~7x more often than
|
||||
# Hemingway reproduces himself. Every one of the 19 matched runs was READ: all are
|
||||
# stock dialogue ("came over and sat down at the table"), max 9 words, no proper
|
||||
# noun, no plot, no imagery — and 9 is shorter than the 10-word run unseen Hemingway
|
||||
# shares with the train split by coincidence. Elevated rate, zero protectable
|
||||
# content. Hemingway is in copyright; lv-yarros is the in-line precedent, also in
|
||||
# copyright, shipped at 0.10 against a 0.07 control. Unload is 0.003 s.
|
||||
# See /adapters/lv-hemingway-4b-v1/README.md,
|
||||
# scripts/hemingway-corpus/GATE-PREREG.md (pre-registered before any generation),
|
||||
# persistent-memory.d/2026-09-17-lv-hemingway-gate.md
|
||||
- lv-hemingway=/adapters/lv-hemingway-4b-v1
|
||||
deploy:
|
||||
resources:
|
||||
reservations:
|
||||
|
||||
Reference in New Issue
Block a user