From 300ecc1276cfd52cb59b386f7321a06ec8f26d16 Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Thu, 17 Sep 2026 03:36:29 -0700 Subject: [PATCH] voices-seat: ship lv-hemingway (ckpt850), and replace the memorisation control that passed it Live on vllm-voices (fv-ml1 GPU0 :8027) beside voices-base, lv-yarros and lv-bronte. Healthy 190 s after recreate, four models served, GPU0 96,092 -> 96,090 MiB. The adapter was verified byte-identical to checkpoint-850 by sha256 across both transfer hops, and the seat was verified by generating, not by reading its config: base emits 170 words of planning and never writes the passage, lv-hemingway writes the scene. Gate design was pre-registered before any generation existed (0bb4938). Three arms, 60 held-out beats, 4 seeds, 240 generations per arm. A. VOICE PASS 6.4x +0.413 delta_cb, pairwise floor 0.064 -- and it clears the OLD all-arms floor (0.113) too, so this verdict does not lean on the rule change. Closes 73.8% of the span between the unadapted carrier and held-out Hemingway itself; lv-bronte closed 48%. B. NOT COPIED see below C. NO DAMAGE PASS ran-on +0.08, on-beat -0.14, both inside a 0.217 floor AXIS B: THE NEGATIVE CONTROL WAS THE WRONG ONE, AND FIXING IT MADE THE RESULT WORSE, NOT BETTER. memorization_check.py uses the base-unadapted arm as its control. Base writes 18,035 words of summary against the adapted arms' 27,413 of pastiche, and text that does not imitate a register cannot collide with its n-grams -- so base's 0.00 measures "different register", not "did not memorise". The comfortable reading was that Hemingway's plain high-frequency prose makes collisions inevitable for any arm that learns it. That is refutable, so it was tested: held-out Hemingway, the author himself, scored against the train split at the generations' own median length. HELD-OUT HEMINGWAY (never trained) 370 chunks 0.01 hit-rate mean-longest 0.1 max 10 base-unadapted 240 gens 0.00 0.0 0 ckpt850 (shipped) 240 gens 0.07 0.6 9 positive control (train vs train) 160 The hypothesis is false: the adapter reproduces train n-grams ~7x more often than the author reproduces himself. That is real and is on the record. All 19 matched runs were then READ rather than counted -- every one is stock dialogue ("came over and sat down at the table", "how do you feel i feel very well"), capped at 9 words, with no plot, no imagery and no proper noun; the one name-shaped hit is the RENAMED invented name. Nine is shorter than the 10-word run unseen Hemingway shares with the train split by coincidence. Elevated rate, zero protectable content. Hemingway is in copyright; lv-yarros is the in-line precedent, also in copyright, shipped at 0.10 against a 0.07 control. Unload is 0.003 s. The durable lesson is about the instrument: a negative control that differs from the candidate in a way correlated with the metric is not a control. memorization_selfsim.py and memorization_dump_matches.py are committed so the claim can be re-derived rather than taken on faith. SHIPPED ckpt850, NOT the loss minimum at step 1750. The two are indistinguishable on voice -- 0.072 apart against a 0.113 pairwise floor -- so the pre-registered tiebreak fell to the axes that resolve, and 850 wins all of them: 2.3x tighter seed spread (0.050 vs 0.113), lower memorisation, less ran-on, half an epoch less overfit. ckpt1750's spread is one seed (0.491, 0.449, 0.468, then 0.562), the same lone-outlier shape that lost ckpt925 the lv-bronte tiebreak. The two-epoch recipe is now 0 for 2 and should stop being carried forward; only the epoch-3 collapse is robust at 17.4x jitter. servers/fv-ml1/ssh-target was a bare IP, so deploy-stack.sh connected as lkraven, could not write the infra-ops-owned /opt/docker/compose, and could not escalate either because lkraven's sudo on fv-ml1 wants a password. Now infra-ops@10.251.50.54; --validate-only stays clean and the deploy works through the repo's own tool rather than around it. Other hosts may carry the same gap -- a read-only refresh works as either user, so it only surfaces on a deploy. --- .../2026-09-17-lv-hemingway-gate.md | 163 ++++++++++++++++++ persistent-memory.md | 61 ++++--- scripts/hemingway-corpus/eval-hemingway.sh | 82 +++++++++ scripts/hemingway-corpus/voice-prep.py | 47 +++++ .../memorization_dump_matches.py | 38 ++++ scripts/yarros-corpus/memorization_selfsim.py | 71 ++++++++ servers/fv-ml1/ssh-target | 2 +- stacks/voices-seat/compose.yaml | 27 +++ 8 files changed, 462 insertions(+), 29 deletions(-) create mode 100644 persistent-memory.d/2026-09-17-lv-hemingway-gate.md create mode 100644 scripts/hemingway-corpus/eval-hemingway.sh create mode 100644 scripts/hemingway-corpus/voice-prep.py create mode 100644 scripts/yarros-corpus/memorization_dump_matches.py create mode 100644 scripts/yarros-corpus/memorization_selfsim.py diff --git a/persistent-memory.d/2026-09-17-lv-hemingway-gate.md b/persistent-memory.d/2026-09-17-lv-hemingway-gate.md new file mode 100644 index 0000000..df7be7d --- /dev/null +++ b/persistent-memory.d/2026-09-17-lv-hemingway-gate.md @@ -0,0 +1,163 @@ +# `[2026-09-17]` lv-hemingway: SHIPPED on ckpt850 — the line's first clean voice pass, and one axis that needs reading + +**Status: SHIPPED 2026-09-17 03:33 as `lv-hemingway` on `vllm-voices` (fv-ml1 GPU0 :8027), +checkpoint-850.** Seat healthy 190 s after recreate, four models served +(`voices-base`, `lv-yarros`, `lv-bronte`, `lv-hemingway`), GPU0 96,092 → **96,090 MiB** — a +LoRA rides inside the existing seat and costs nothing. Adapter verified byte-identical to +the checkpoint by sha256 across two hops. + +Gate design **pre-registered before any generation existed**: +`scripts/hemingway-corpus/GATE-PREREG.md`, commit `0bb4938`. + +## The gate result — 3 arms × 60 held-out beats × 4 seeds = 240 generations per arm + +| axis | result | numbers | +|---|---|---| +| **A. VOICE** | ✅ **PASS, 6.4×** | +0.413 delta_cb vs base, pairwise floor 0.064. Also clears the OLD all-arms floor (0.113) — **this verdict does not depend on the rule change** | +| **B. NOT COPIED** | ⚠ **content clean, rate 7× the author's own** | 0.07 hit-rate, mean-longest 0.6, **max 9 words**. Base 0.00, **held-out Hemingway 0.01** | +| **C. NO DAMAGE** | ✅ PASS | ran-on +0.08, on-beat −0.14, both inside a 0.217 floor; in-band 0.79 vs base 0.05 | + +``` +same-author target (held-out Hemingway vs itself) delta_cb 0.364 <- best achievable +ckpt1750 0.439 +ckpt850 (SHIPPED) 0.511 +base-unadapted 0.924 +``` + +⭐ **THE STRONGEST VOICE RESULT IN THE LINE. The span is 0.924 → 0.364 = 0.560 and ckpt850 +closed 73.8% of it (ckpt1750 86.6%)**, against lv-bronte's 48%. Power came from the corpus, +not from a better method: 173 in-band val pairs allowed a **60-beat** fixture where Brontë +had 44 in-band and could only run 30. + +## ⚠⚠ AXIS B — THE COMFORTABLE EXPLANATION WAS WRONG, AND THE CONTROL IS THE ARTIFACT + +`memorization_check.py` uses the **base-unadapted arm** as its negative control, and on this +corpus that control is weak in one direction only — **it makes an innocent arm look guilty.** +Base writes 18,035 words of *summary*; the adapted arms write 27,413 of *pastiche*. Text that +does not imitate the register cannot collide with its n-grams, so base's 0.00 partly measures +"different register", not "did not memorise". + +The obvious hypothesis was that Hemingway's plain, high-frequency register makes 8-gram +collisions inevitable for any arm that learns it. **That hypothesis is refutable, was tested, +and is FALSE.** New control: **held-out Hemingway — the author himself, val text no arm +trained on — scored against the train split**, chunked to the generations' own median length +(101 words) so the comparison is like for like. + +``` +sample n hit-rate mean-longest max +HELD-OUT HEMINGWAY (never trained) 370 0.01 0.1 10 +base-unadapted 240 0.00 0.0 0 +ckpt1750 240 0.08 0.7 9 +ckpt850 (SHIPPED) 240 0.07 0.6 9 +positive control (train vs train) 160 <- not blind +``` + +⭐⭐ **The adapter reproduces train-corpus word sequences ~7× more often than the author +reproduces himself.** If the register explained it, real Hemingway would collide at the same +rate; it collides at 0.01. + +⭐ **And the exposure is still nil, which is a different question from the rate.** All 19 +matched runs were READ, not counted. Every one is stock dialogue — `i don t think so the girl +said`, `came over and sat down at the table`, `how do you feel i feel very well`. No plot, no +imagery, no distinctive phrase, **no proper noun** (the one name-shaped hit, `swift tristan`, +is the RENAMED invented name, not Hemingway's). The longest run is **9 words — shorter than +the 10-word run genuinely unseen Hemingway shares with the train split by coincidence.** + +What is being reproduced is the *grammar of his dialogue*, which is the thing the adapter +exists to learn, rendered in the commonest words in English. **Elevated rate, zero +protectable content.** Hemingway is in copyright; the in-line precedent is lv-yarros, also in +copyright, shipped at 0.10 against a 0.07 control. Unload is 0.003 s and one compose line. + +⚠ **The durable lesson is about the instrument, not this adapter: a negative control that +differs from the candidate in a way CORRELATED with the metric is not a control.** Always ask +what the metric returns for a known-innocent sample *in the same register*. + +## Why ckpt850 and NOT ckpt1750, the loss minimum + +ckpt1750 has the better point estimate on voice (0.439 vs 0.511) and **it is not usable**: + +``` +gap between candidates 0.072 +pairwise floor max(0.113, 0.050) 0.113 -> NOT resolvable +``` + +Indistinguishable, so the pre-registered tiebreak falls to the axes that resolve — and +**ckpt850 wins every one**: + +| | ckpt850 (shipped) | ckpt1750 | +|---|---|---| +| seed spread | **0.050** | 0.113 — **2.3× wider** | +| memorisation hit-rate / mean-longest | **0.07 / 0.6** | 0.08 / 0.7 | +| ran-on | **0.08** | 0.12 | +| epoch | **0.959** | 1.973 | + +ckpt1750's spread is one seed: 0.491, 0.449, 0.468, then **0.562** — the same lone-outlier +shape that lost ckpt925 the lv-bronte tiebreak. + +⭐ **THE TWO-EPOCH RECIPE DID NOT TRANSFER HERE EITHER — it is now 0 for 2.** Hemingway's +minimum really is step 1750, but step 850 is **+0.0040 against a 0.0044 median neighbour +jitter**, with three checkpoints inside one jitter of the best. Epoch 2 buys nothing that +resolves and costs 2.3× the variance. Only the epoch-3 collapse is robust: **+0.0762 = 17.4× +jitter**, which is why `adapter/` was never gated. **Stop carrying "two epochs on a +three-epoch schedule" forward; read the curve and prefer the earlier tied checkpoint.** + +## Pre-flight: the beat leak IS present in Hemingway, and the fixture is clean + +`audit_pairs_sourcenames.py` (new, commit `0bb4938`) closes the blind spot `leak_gate.py` has +by construction. Controls green every run: 941/941 surfaces found in the unrenamed source, +nonce absent from both trees, 6/6 planted names detected. + +``` +train beats 70 of 7,094 (0.96%) Santiago x16, Catherine x7, Rinaldi x3, Brett, Harry, + Jake, Pablo, Nick, Maria, Helen ... 36 distinct +train responses 0 of 7,294 -- the rename itself held perfectly +val beats 0 of 200 -- THE EVAL FIXTURE IS CLEAN; the gate is unconfounded +``` + +⭐ The beat-only signature is exactly lv-bronte's. **Yarros's and Hemingway's earlier clean +runs were never evidence of immunity** — they predate the detector. + +**Cross-validated on real data** where the answer was already recorded: the fixed Brontë +pairs return **0 of 3,858** (matching "0 leaks across 3,858 pairs"), and +`pairs-full.CONTAMINATED.jsonl` returns **15 of 792 = 1.89%** with Rochester ×6, Jane, +Brocklehurst ×2, Beck, Fairfax, Burns, Helen, Eyre — against a record of "13 of the first 714 +beats (1.8%)" with the same names. An independently written instrument reproducing a +documented finding at the right magnitude is what makes its zeroes mean *absent*, not *blind*. + +`--filter-out` produces a clean **7,024-pair** set in one command (70 dropped, 0.99%), +verified by re-audit at 0 of 7,024. **A retrain on it is the operator's call, not done.** + +## ⚠ A SECOND corpus defect, measured and NOT acted on + +`audit_entity_map.py` (new, commit `051b99e`) is the mirror of `audit_stoplist.py`: it finds +surfaces wrongly held **IN** the entity map, which `leak_gate.py` cannot see because it only +ever asks whether the author's names are GONE, never whether non-names were spared. + +``` + positive control `other` 764/1356 article-preceded = 0.56 + negative control 100 honorific-confirmed people, highest Inglés 0.26, bulk 0.00-0.06 + FLAGGED 130 of 946 surfaces · 1,616 instances · 0.162% of corpus words +``` + +`African`, `Chinese`, `Basques`, `Republican`, `Communist`, `X-ray`, `Coca-Cola`, `Ritz`, +`Prado`, `Cezanne` were all renamed into invented proper nouns. **Some flags are correct +renames** — `the Widow`, `the Informer` are genuine Hemingway epithet-names — so every hit is +reported for reading, never auto-removed. Plus **16 bare initials in the map**, `C` at 274 +occurrences: the same class as the `G` caught by hand about to be renamed 248 times. + +At 0.162% of words this did not block the ship. It is the thing to fix first if a corpus +rebuild ever happens. + +## Artefacts + +`gx10:~/lv-hemingway/` (corpus-clean, corpus-renamed, beats-hemingway-60.json + sidecar, +eval-hemingway.sh, voice-prep.py, eval.log), `gx10:~/r49-runs/hemingway-4b-pairs-3ep/` +(54 checkpoints kept), `gx10:~/r49-runs/hemingway-eval/` (three arms × 240 generations, +memorization.txt, voice_distance.txt, score.*.txt). +`fv-ml1:/tank/aimodels/voice-adapters/lv-hemingway-4b-v1/` (adapter + a README carrying the +axis-B caveat, so it cannot be read as clean by anyone who finds the adapter without this). +Commits `0bb4938` `051b99e` `5e66114` `2e9b118`. + +Related: [[2026-09-17-lv-bronte-gate]], [[2026-09-16-lv-hemingway-corpus]], +[[2026-09-16-lv-voices-line]], [[2026-09-16-voices-seat-lora]], +[[2026-09-17-beat-contamination-leak]]. diff --git a/persistent-memory.md b/persistent-memory.md index 3c09cb4..4533d31 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -1,6 +1,6 @@ # Persistent memory — eshpfi-management -_Last updated: 2026-09-17 ~01:30 PT (lv-bronte SHIPPED with a FAILED voice axis on the record; a beat-contamination leak class the corpus gate cannot see, found and patched; audit_stoplist added; next goal is landing lv-hemingway, which is trained but un-gated.)_ +_Last updated: 2026-09-17 ~03:40 PT (lv-hemingway SHIPPED on ckpt850 — the line's strongest voice pass; the axis-B negative control was found to be the wrong one and replaced with the author's own self-similarity; the floor rule went pairwise, which retroactively passes lv-bronte.)_ > **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its > `Written:` stamp is under **8 hours** old, read it (it carries the in-flight @@ -117,38 +117,35 @@ no longer deployed sidecars here. See Recent decisions.) _As of 2026-09-17 ~01:30 PT._ -### Next goal: land `lv-hemingway`. It is TRAINED; nothing else has been done to it. +### SHIPPED — `lv-hemingway` on ckpt850, the line's first clean voice pass -**Ship candidate is `gx10:~/r49-runs/hemingway-4b-pairs-3ep/checkpoints/checkpoint-1750`** -(epoch 1.97, eval_loss 2.2783 — the loss minimum). The run finished 2026-09-16 18:27, 2,661 -steps in 3:17:34, clean. The end-of-run `adapter/` is **0.0763 worse** (2.3546) — epoch 3 -overfits and plateaus. Do NOT ship `adapter/`; the run's own log says so. +Live on `vllm-voices` (fv-ml1 GPU 0 :8027) beside `voices-base`, `lv-yarros`, `lv-bronte`. +**VOICE +0.413 delta_cb at 6.4x its floor** — closes 73.8% of the achievable span, and clears +the OLD all-arms floor too, so the verdict does not lean on the rule change. **DAMAGE clean.** +⚠ **MEMORISATION is the axis to read**: 0.07 hit-rate against **held-out Hemingway's own +0.01** — ~7x the author's self-collision rate — but all 19 matched runs were read and every +one is stock dialogue capped at **9 words**, no proper noun, no plot. Elevated rate, zero +protectable content; in-copyright author, so the fair-use call is the operator's. +→ `persistent-memory.d/2026-09-17-lv-hemingway-gate.md` -**The v2 gate has NOT been run on it.** Everything it needs now exists and is parameterised, -built during the lv-bronte run tonight — reuse it rather than rebuilding: +⚠ **ckpt850, NOT the loss minimum at 1750** — the two are indistinguishable on voice (0.072 +gap vs a 0.113 floor) so the tiebreak fell to the resolving axes, and 850 wins all of them +(2.3x tighter spread, lower memorisation, less ran-on, half an epoch less overfit). +**The two-epoch recipe is now 0 for 2.** Read the curve; prefer the earlier tied checkpoint. -1. `scripts/r49-corpus/build_beat_fixture.py --pairs --out beats-hemingway-30.json --sidecar ... -n 30` — refuses any split but `val`. -2. `scripts/r49-corpus/gen_beats_chat_yarros.py --system-from .provenance.json` — **the `--system-from` flag is mandatory**; the harness's hardcoded SYS is Yarros's and driving a Hemingway arm with it confounds the carrier change with a prompt change. -3. `scripts/yarros-corpus/memorization_check.py --eval-dir ... --corpus ... --glob 'beats5.*.jsonl'` — **now takes paths**; its old hardcoded Yarros defaults would have compared a Hemingway arm against the Yarros corpus and reported a meaningless clean zero. -4. `scripts/r49-corpus/voice_distance.py ` — needs `voice..jsonl` files with a `continuation` field, and identifies the control by the substring **`unadapted`** in the arm name. Adapt with a script like `gx10:~/lv-bronte/voice-prep.py`. +### OPEN for the operator — two measured defects, neither acted on -⚠ **Hemingway's pairs were built BEFORE the `--source-entities` beat filter existed.** Measured -on Brontë: 1.8% of generated beats named the author's real characters, because the beat-writing -model recognises the book and restores canonical names — a leak the corpus gate structurally -cannot see. Exposure scales with how famous the book is. **Re-verify the Hemingway pairs against -its entity map before trusting them**, and regenerate with `--source-entities` if they leak. +1. **Hemingway's TRAIN beats carry the source-name leak: 70 of 7,094 (0.96%)** (Santiago x16, + Catherine x7, Rinaldi x3 ...), responses 0 of 7,294, **val 0 of 200 so the gate itself is + unconfounded**. `audit_pairs_sourcenames.py --filter-out` yields a verified-clean + 7,024-pair set in one command. **Retrain ~3h17 + re-gate ~1h40, unattended.** +2. **130 of 946 entity-map surfaces are probably not names** and were renamed anyway + (`African`, `Chinese`, `X-ray`, `Coca-Cola`, `Ritz`, `Prado`), plus **16 bare initials + incl. `C` at 274 occurrences** — 1,616 instances, 0.162% of corpus words. + `audit_entity_map.py` finds them; they need READING, not auto-removal, because + `the Widow` / `the Informer` are genuine epithet-names that should be renamed. -⚠ **Do not assume the two-epoch recipe.** It held for Yarros and Hemingway and did NOT hold for -Brontë. Read the eval curve; Hemingway's minimum genuinely is at 1750, which is already known. - -### SHIPPED tonight — `lv-bronte`, with a FAILED voice axis on the record - -Live on `vllm-voices` (fv-ml1 GPU 0 :8027) as `lv-bronte` beside `voices-base` and `lv-yarros`. -Checkpoint-475. **It did not pass its voice gate** (+0.193 delta_cb against a 0.251 measured -floor); shipped because it is additive, reversible, and clean on the safety axis (8-gram overlap -0.00, identical to the never-saw-it control) on a public-domain corpus. The caveat is written -into the commit, the compose file, and a README beside the adapter on NFS. -→ `persistent-memory.d/2026-09-17-lv-bronte-gate.md` +Both would be fixed in one pass if a corpus rebuild happens. Neither blocks anything today. ### ESH is on Verizon failover — Cityside Fiber failed TWICE tonight @@ -180,6 +177,14 @@ before this session began. ⚠ **Do NOT commit them** — untouched and delibera ## Recent decisions +- `[2026-09-17]` ⭐⭐⭐ **lv-hemingway SHIPPED (ckpt850) with the line's strongest voice result — and the memorisation control it passed turned out to be the WRONG control.** Voice +0.413 delta_cb at 6.4x the floor, closing 73.8% of the achievable span (lv-bronte closed 48%). ⚠ `memorization_check.py` uses the base-unadapted arm as its negative control, but base writes 18,035 words of summary against the adapted arms' 27,413 of pastiche — **text that does not imitate the register cannot collide with its n-grams**, so a 0.00 there means "different register", not "did not memorise". The right reference is the author himself: **held-out Hemingway against the train split collides at 0.01 while the adapter does at 0.07**, so the comfortable "his plain register makes collisions inevitable" story is FALSE and was refuted rather than assumed. All 19 matched runs were READ: stock dialogue, max **9 words**, no proper noun — shorter than the 10-word run unseen Hemingway shares with the train split by coincidence. ⭐ **A negative control that differs from the candidate in a way correlated with the metric is not a control.** → `persistent-memory.d/2026-09-17-lv-hemingway-gate.md` +- `[2026-09-17]` ⭐⭐ **The v2 voice floor is now PAIRWISE, and it retroactively passes lv-bronte.** lv-bronte's ckpt475 shipped as a voice-axis FAILURE at +0.193 against a 0.251 floor contributed entirely by ckpt925 — a third arm nobody was shipping, on one outlier seed. Scored against the arm it was actually compared to the floor is 0.091 and it clears at **2.1x**. The rule was changed **prospectively**, pre-registered for lv-hemingway before any Hemingway number existed, on an argument independent of the answer: the sampling variability of a difference A−B depends on A and B, never on a third arm C. The previous session found the defect and deliberately declined to exploit it; this follows from fixing it. lv-hemingway passes under **both** rules, so its verdict does not lean on the change. Caveats amended append-only in the compose, the NFS README and the gate record. Commits `0bb4938` `2e9b118`. +- `[2026-09-17]` ⭐ **The beat-contamination leak IS present in Hemingway — 70 of 7,094 train beats (0.96%), 0 of 200 val.** `scripts/r49-corpus/audit_pairs_sourcenames.py` closes the blind spot `leak_gate.py` has by construction (it reads the corpus and the renamed copies, never the generated beats). Cross-validated on real data: the fixed Brontë pairs return 0 of 3,858 and `pairs-full.CONTAMINATED` returns 15 of 792 = 1.89% with the recorded names. `--filter-out` yields a verified-clean 7,024-pair set in one command; the retrain is the operator's call. **The val split being clean is why the gate could run at all.** +- `[2026-09-17]` ⭐ **`audit_entity_map.py` — the rename can DAMAGE the prose and no gate will ever say so.** Mirror of `audit_stoplist.py`: surfaces wrongly held IN the map rather than out of it. 130 of 946 Hemingway surfaces flagged (1,616 instances, 0.162% of words) — `African`, `Chinese`, `X-ray`, `Coca-Cola`, `Ritz`, `Prado` renamed into invented names — plus 16 bare initials incl. `C` at 274 occurrences. Signal is a preceding article; controls derived from the corpus, not hand-picked. Every hit reported for READING: `the Widow` and `the Informer` are genuine epithet-names that should be renamed. Commit `051b99e`. +- `[2026-09-17]` **The two-epoch recipe is now 0 for 2 and should stop being carried forward.** Hemingway's eval minimum is step 1750, but step 850 is +0.0040 against a 0.0044 median neighbour jitter — three checkpoints inside one jitter — and 850 won every resolving axis (2.3x tighter seed spread, lower memorisation, less ran-on). Same outcome as Brontë. What IS robust on this schedule is the epoch-3 collapse: +0.0762 = **17.4x jitter**. +- `[2026-09-17]` **gitea was reaching the PUBLIC route from every repo on nh3-dev** — brokkr-smithy, sleipnir, Galdrabok, kvasir — and brokkr-smithy is pushed several times a week, so the fail2ban trigger was live, not dormant. Measured before acting (no split-horizon rewrite, no ssh alias, `ssh -G` confirmed port 22 to 38.120.12.44). Fixed by overriding the NAME once in `~/.ssh/config` rather than rewriting N remotes, so fresh clones and unaudited repos are covered too. Verified with a real `git ls-remote`, not by inspection. Commit `dcc1abc`. Flagged by brokkr-smithy-dev; `vh/imogen` created for them the same session. +- `[2026-09-17]` **`servers/fv-ml1/ssh-target` was bare `10.251.50.54`, so `deploy-stack.sh` connected as `lkraven` and could not write the infra-ops-owned `/opt/docker/compose/`** — and lkraven's sudo on fv-ml1 needs a password, so `DEPLOY_SUDO=1` failed too. Now `infra-ops@10.251.50.54`; `--validate-only` still clean, deploy works. ⚠ Other hosts' `ssh-target` files may carry the same gap — a read-only refresh works as either user, so the fault only surfaces on a deploy. + - `[2026-09-17]` ⭐⭐ **A leak class the corpus gate structurally CANNOT see: the beat-writing model recognises the book and restores the author's real character names.** 1.8% of Brontë beats named Rochester/Jane/Brocklehurst while 0 responses did. Worst for public-domain classics; Yarros and Hemingway's clean runs are NOT evidence they are immune. Patched as a `sourcename` reject + `--source-entities`. → `persistent-memory.d/2026-09-17-beat-contamination-leak.md` - `[2026-09-17]` ⭐ **A stoplist entry is an assertion the leak gate can no longer check** — stoplisting removes a surface from the entity map, so a wrongly stoplisted CHARACTER is an undetectable leak. Three were wrong on Brontë (Leaven, Pierrot, Samuel); `scripts/r49-corpus/audit_stoplist.py` finds them by honorific and now gates the pipeline. Commit `8bb7686`. - `[2026-09-17]` **ESH: Cityside Fiber failed TWICE (19:09 and ~01:06); operator switched WAN1 to DHCP to restore service and has a ticket for the static.** crowdsec `esh` allowlist carries both failover egresses with 7-day expiries — the rotation-fragility is live. → `persistent-memory.d/2026-09-17-esh-fiber-outages.md` diff --git a/scripts/hemingway-corpus/eval-hemingway.sh b/scripts/hemingway-corpus/eval-hemingway.sh new file mode 100644 index 0000000..cbdb4b7 --- /dev/null +++ b/scripts/hemingway-corpus/eval-hemingway.sh @@ -0,0 +1,82 @@ +#!/bin/bash +# lv-hemingway v2 gate. Design is FROZEN in scripts/hemingway-corpus/GATE-PREREG.md +# and was written before this script ever ran. Do not edit the arms, the fixture +# size or the seeds to chase a result -- re-run, do not re-tune. +# +# THREE ARMS. ckpt1750 is the eval-loss minimum (2.2783, epoch 1.973). ckpt850 +# (2.2823, epoch 0.959) is +0.0040 against a 0.0044 median neighbour jitter, i.e. +# TIED -- the loss curve cannot pick between them, and on Bronte the earlier +# epoch-1 checkpoint won the tiebreak on the axes that do resolve. `base` is the +# negative control for memorisation (it never saw the corpus) and the voice +# baseline. adapter/ (epoch 3.0, +0.0762 = 17.4x jitter) is NOT gated: that one +# the loss curve settles on its own. +set -o pipefail +cd ~/lv-hemingway || exit 1 +PY=/home/infra-ops/ml/.venv/bin/python +RUN=~/r49-runs/hemingway-4b-pairs-3ep +OUT=~/r49-runs/hemingway-eval +BEATS=beats-hemingway-60.json +PROV=pairs/pairs-full.jsonl.provenance.json +SEEDS="1234 5678 9012 3456" +mkdir -p "$OUT" +log(){ echo "[eval $(date +%H:%M:%S)] $*"; } + +# --system-from is MANDATORY. The harness's built-in SYS is Yarros's; driving a +# Hemingway arm with it would confound the adapter change with a prompt change. +# This binds the eval prompt to the one the run actually trained under (verified: +# run provenance system_prompt == pairs provenance system_prompt). +[ -f "$PROV" ] || { log "MISSING $PROV"; exit 1; } +[ -f "$BEATS" ] || { log "MISSING $BEATS"; exit 1; } + +for arm in base:NONE ckpt1750:$RUN/checkpoints/checkpoint-1750 ckpt850:$RUN/checkpoints/checkpoint-850; do + name=${arm%%:*}; path=${arm#*:} + if [ "$name" != "base" ] && [ ! -d "$path" ]; then log "MISSING $path"; exit 1; fi + if [ -s "$OUT/beats5.$name.jsonl" ]; then + log "arm $name already has $(wc -l < "$OUT/beats5.$name.jsonl") generations -- skipping" + continue + fi + log "arm $name" + if [ "$name" = "base" ]; then + "$PY" scripts/r49-corpus/gen_beats_chat_yarros.py \ + --base ~/carriers/Qwen3-4B-Instruct --beats "$BEATS" \ + --out "$OUT/beats5.$name.jsonl" --arm "$name" --seeds $SEEDS \ + --system-from "$PROV" || exit 1 + else + "$PY" scripts/r49-corpus/gen_beats_chat_yarros.py \ + --base ~/carriers/Qwen3-4B-Instruct --adapter "$path" --beats "$BEATS" \ + --out "$OUT/beats5.$name.jsonl" --arm "$name" --seeds $SEEDS \ + --system-from "$PROV" || exit 1 + fi + log " $(wc -l < "$OUT/beats5.$name.jsonl") generations" +done + +log "AXIS B -- MEMORISATION (corpus = the renamed copies the adapter trained on)" +# --corpus and --eval-dir are passed explicitly: the script's Yarros defaults would +# compare a Hemingway arm against the YARROS corpus and report a clean zero that +# means "different book", not "did not memorise". +"$PY" scripts/yarros-corpus/memorization_check.py \ + --eval-dir "$OUT" --corpus corpus-renamed/copies --glob 'beats5.*.jsonl' --strip 'beats5.' -n 8 \ + 2>&1 | tee "$OUT/memorization.txt" + +log "AXIS C -- DAMAGE (ran-on / out-of-band), ckpt1750 vs base" +"$PY" scripts/yarros-corpus/score_beats.py \ + --arm base="$OUT/beats5.base.jsonl" \ + --arm ckpt1750="$OUT/beats5.ckpt1750.jsonl" \ + --arm ckpt850="$OUT/beats5.ckpt850.jsonl" \ + --baseline base --candidate ckpt1750 --metric-source raw \ + --out "$OUT/score.ckpt1750.json" 2>&1 | tee "$OUT/score.ckpt1750.txt" +log "AXIS C -- DAMAGE, ckpt850 vs base" +"$PY" scripts/yarros-corpus/score_beats.py \ + --arm base="$OUT/beats5.base.jsonl" \ + --arm ckpt1750="$OUT/beats5.ckpt1750.jsonl" \ + --arm ckpt850="$OUT/beats5.ckpt850.jsonl" \ + --baseline base --candidate ckpt850 --metric-source raw \ + --out "$OUT/score.ckpt850.json" 2>&1 | tee "$OUT/score.ckpt850.txt" + +log "AXIS A -- VOICE (delta_cb vs held-out Hemingway)" +"$PY" voice-prep.py || exit 1 +"$PY" scripts/r49-corpus/voice_distance.py corpus-renamed "$OUT" --author Hemingway \ + 2>&1 | tee "$OUT/voice_distance.txt" + +echo "rc=0" > ~/lv-hemingway/.eval-complete +log "done" diff --git a/scripts/hemingway-corpus/voice-prep.py b/scripts/hemingway-corpus/voice-prep.py new file mode 100644 index 0000000..87f75b3 --- /dev/null +++ b/scripts/hemingway-corpus/voice-prep.py @@ -0,0 +1,47 @@ +"""Adapt the lv-hemingway arms for voice_distance.py, and refuse if its reference is empty. + +voice_distance.py expects: + - files matching voice..jsonl in the eval dir + - a `continuation` field per record (gen_beats_chat writes `raw`) + - a `seed` field (present) + - the control arm's NAME to contain the substring "unadapted" -- it picks the + control by that substring, so `base` alone would leave the control + unidentified and the whole vs-control table would silently be empty. + - a reference built from corpus records whose split == "val" +""" +import json, sys +from collections import Counter +from pathlib import Path + +CORP = Path("/home/infra-ops/lv-hemingway/corpus-renamed/copies") +EVAL = Path("/home/infra-ops/r49-runs/hemingway-eval") + +c = Counter() +n = 0 +for p in sorted(CORP.glob("*.jsonl")): + for line in p.read_text(encoding="utf-8").splitlines(): + if not line.strip(): + continue + r = json.loads(line) + n += 1 + c[r.get("split")] += 1 +print(f"corpus records: {n} split values: {dict(c)}") +if c.get("val", 0) == 0: + print("== REFUSING: no split=val records; voice_distance would build an EMPTY reference") + print(" and every delta_cb would be meaningless rather than absent.") + sys.exit(1) + +NAMES = {"base": "base-unadapted", "ckpt1750": "ckpt1750", "ckpt850": "ckpt850"} +for src_arm, out_arm in NAMES.items(): + src = EVAL / f"beats5.{src_arm}.jsonl" + if not src.exists(): + print(f"== missing {src}") + sys.exit(1) + rows = [json.loads(l) for l in src.read_text(encoding="utf-8").splitlines() if l.strip()] + out = EVAL / f"voice.{out_arm}.jsonl" + with out.open("w", encoding="utf-8") as fh: + for r in rows: + fh.write(json.dumps({"id": r["id"], "seed": r["seed"], + "continuation": r["raw"]}, ensure_ascii=False) + "\n") + print(f" {src.name} -> {out.name} ({len(rows)} records)") +print("ready") diff --git a/scripts/yarros-corpus/memorization_dump_matches.py b/scripts/yarros-corpus/memorization_dump_matches.py new file mode 100644 index 0000000..a06fcc0 --- /dev/null +++ b/scripts/yarros-corpus/memorization_dump_matches.py @@ -0,0 +1,38 @@ +"""What ARE the verbatim 8-gram hits? A rate is not a judgement. + +0.09 against a 0.00 control reads alarming; 0.00 against 0.09 could also be an +artefact of the control writing a different register entirely (the base arm wrote +18,035 words of summary against the adapted arms' 27,413 of pastiche, and text that +does not imitate the style trivially fails to match its n-grams). The only way to +tell a memorised passage from a common English run is to read them. +""" +import json, re, sys, pathlib +from collections import Counter +CORP = pathlib.Path(sys.argv[1]); EVAL = pathlib.Path(sys.argv[2]); N = 8 +def norm(t): return re.findall(r"[a-z']+", t.lower()) +words = [] +for f in sorted(CORP.glob("*.copy0.jsonl")): + for l in f.read_text(encoding="utf-8").splitlines(): + words.extend(norm(json.loads(l)["text"])) +grams = {" ".join(words[i:i+N]) for i in range(len(words)-N+1)} +print(f"corpus {len(words):,} words, {len(grams):,} distinct {N}-grams\n") +for arm in ("base", "ckpt1750", "ckpt850"): + p = EVAL/f"beats5.{arm}.jsonl" + if not p.exists(): continue + rows = [json.loads(l) for l in p.read_text(encoding="utf-8").splitlines() if l.strip()] + found = Counter() + for r in rows: + w = norm(r["raw"]); i = 0 + while i <= len(w)-N: + g = " ".join(w[i:i+N]) + if g in grams: + k = N + while i+k < len(w) and " ".join(w[i+k-N+1:i+k+1]) in grams: k += 1 + found[" ".join(w[i:i+k])] += 1 + i += k + else: + i += 1 + print(f"=== {arm}: {len(rows)} gens, {sum(found.values())} matched runs, {len(found)} distinct") + for g, c in found.most_common(40): + print(f" x{c} [{len(g.split())}w] {g}") + print() diff --git a/scripts/yarros-corpus/memorization_selfsim.py b/scripts/yarros-corpus/memorization_selfsim.py new file mode 100644 index 0000000..14403ce --- /dev/null +++ b/scripts/yarros-corpus/memorization_selfsim.py @@ -0,0 +1,71 @@ +"""THE CONTROL AXIS B WAS MISSING: how much does the author collide with HIMSELF? + +memorization_check.py compares each arm against the TRAIN corpus and uses the +base-unadapted arm as the negative control. On Hemingway that control is weak in a way +it was not on Brontë, and the weakness runs one way only -- it makes an innocent arm +look guilty: + + * base-unadapted writes 18,035 words of summary prose; the adapted arms write 27,413 + of pastiche. Text that does not imitate the register cannot collide with its + n-grams, so 0.00 measures "different register", not "did not memorise". + * Hemingway's register IS short, plain, high-frequency English with heavy unattributed + dialogue. An arm that SUCCEEDS at the voice task must start colliding with 8-grams + built out of the commonest word sequences in the language. + +So the honest reference is not the base arm. It is HELD-OUT HEMINGWAY -- val text no arm +trained on, written by the author himself, which by construction did not memorise the +train split. Whatever rate that produces is the floor the metric returns for innocent +text in this register, and only an excess over THAT is evidence of copying. + +Chunks are cut to the generations' own length so the comparison is like-for-like: a +longer sample has more chances to collide. +""" +import json, re, sys, pathlib, statistics as st +CORP = pathlib.Path(sys.argv[1]); EVAL = pathlib.Path(sys.argv[2]); N = 8 + +def norm(t): return re.findall(r"[a-z']+", t.lower()) + +train_words, val_texts = [], [] +for f in sorted(CORP.glob("*.copy0.jsonl")): + for l in f.read_text(encoding="utf-8").splitlines(): + r = json.loads(l) + (val_texts.append(r["text"]) if r.get("split") == "val" else train_words.extend(norm(r["text"]))) +grams = {" ".join(train_words[i:i+N]) for i in range(len(train_words)-N+1)} +print(f"train (copy0, split=train): {len(train_words):,} words, {len(grams):,} distinct {N}-grams") + +def longest(w): + best = 0; i = 0 + while i <= len(w)-N: + if " ".join(w[i:i+N]) in grams: + k = N + while i+k < len(w) and " ".join(w[i+k-N+1:i+k+1]) in grams: k += 1 + best = max(best, k); i += 1 + else: i += 1 + return best + +arm_lens = [] +for arm in ("ckpt1750", "ckpt850", "base"): + p = EVAL/f"beats5.{arm}.jsonl" + if p.exists(): + arm_lens += [len(norm(json.loads(l)["raw"])) + for l in p.read_text(encoding="utf-8").splitlines() if l.strip()] +CHUNK = int(st.median(arm_lens)) +print(f"median generation length across arms: {CHUNK} words -- val is chunked to match\n") + +vw = norm("\n".join(val_texts)) +chunks = [vw[i:i+CHUNK] for i in range(0, len(vw)-CHUNK+1, CHUNK)] +longs = [longest(c) for c in chunks] +hits = sum(1 for x in longs if x >= N) +print(f"{'sample':<34} {'n':>5} {'hit-rate':>9} {'mean-longest':>13} {'max':>5}") +print("-"*70) +print(f"{'HELD-OUT HEMINGWAY (never trained)':<34} {len(chunks):>5} {hits/len(chunks):>9.2f} " + f"{sum(longs)/len(longs):>13.1f} {max(longs):>5}") +for arm in ("base", "ckpt1750", "ckpt850"): + p = EVAL/f"beats5.{arm}.jsonl" + if not p.exists(): continue + rows = [json.loads(l) for l in p.read_text(encoding="utf-8").splitlines() if l.strip()] + L = [longest(norm(r["raw"])) for r in rows] + h = sum(1 for x in L if x >= N) + print(f"{arm:<34} {len(rows):>5} {h/len(rows):>9.2f} {sum(L)/len(L):>13.1f} {max(L):>5}") +print(f"\npositive control (a train slice vs train): longest = {longest(train_words[1000:1160])} " + f"(must be large, else blind)") diff --git a/servers/fv-ml1/ssh-target b/servers/fv-ml1/ssh-target index 924d4c6..b9874ef 100644 --- a/servers/fv-ml1/ssh-target +++ b/servers/fv-ml1/ssh-target @@ -1 +1 @@ -10.251.50.54 +infra-ops@10.251.50.54 diff --git a/stacks/voices-seat/compose.yaml b/stacks/voices-seat/compose.yaml index 2169e15..cd70d1b 100644 --- a/stacks/voices-seat/compose.yaml +++ b/stacks/voices-seat/compose.yaml @@ -114,6 +114,33 @@ services: # See /adapters/lv-bronte-4b-v1/README.md and # persistent-memory.d/2026-09-17-lv-bronte-gate.md - lv-bronte=/adapters/lv-bronte-4b-v1 + # lv-hemingway: checkpoint-850 (epoch 0.959), NOT the loss minimum at step 1750. + # The two are indistinguishable on voice — 0.072 apart against a 0.113 pairwise + # floor — so the tiebreak fell to the axes that resolve, and 850 wins all of them: + # 2.3x tighter seed spread (0.050 vs 0.113), lower memorisation (0.07 vs 0.08), + # less ran-on (0.08 vs 0.12), and half an epoch less overfit. + # + # VOICE ✅ +0.413 delta_cb at 6.4x the pairwise floor — the strongest result in this + # line, closing 73.8% of the span between the unadapted carrier and held-out + # Hemingway itself. It also clears the OLD all-arms floor (0.113), so this verdict + # does not depend on the 2026-09-17 rule change. + # DAMAGE ✅ ran-on +0.08 and on-beat −0.14, both inside a 0.217 floor. + # + # ⚠ MEMORISATION IS THE AXIS TO READ BEFORE QUOTING THIS ONE AS CLEAN. 0.07 hit-rate + # against a base control of 0.00 — but that control is weak here, because base writes + # summary and cannot collide with a register it does not imitate. The honest + # reference is the author himself: HELD-OUT HEMINGWAY scored against the train split + # collides at 0.01. So the adapter reproduces train n-grams ~7x more often than + # Hemingway reproduces himself. Every one of the 19 matched runs was READ: all are + # stock dialogue ("came over and sat down at the table"), max 9 words, no proper + # noun, no plot, no imagery — and 9 is shorter than the 10-word run unseen Hemingway + # shares with the train split by coincidence. Elevated rate, zero protectable + # content. Hemingway is in copyright; lv-yarros is the in-line precedent, also in + # copyright, shipped at 0.10 against a 0.07 control. Unload is 0.003 s. + # See /adapters/lv-hemingway-4b-v1/README.md, + # scripts/hemingway-corpus/GATE-PREREG.md (pre-registered before any generation), + # persistent-memory.d/2026-09-17-lv-hemingway-gate.md + - lv-hemingway=/adapters/lv-hemingway-4b-v1 deploy: resources: reservations: