From 9b360e477d3e00aebc3921c46cbcd879ca23ad0a Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Thu, 17 Sep 2026 01:15:29 -0700 Subject: [PATCH] =?UTF-8?q?memory:=20lv-bronte=20gated=20=E2=80=94=20voice?= =?UTF-8?q?=20axis=20fails,=20ship=20decision=20open?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Records the full v2 gate result and three findings that outlive the ship call: 1. The effect is UNDERPOWERED, not absent. Both candidates closed 48-52% of the achievable span to held-out Bronte and beat base on every individual seed, but the gaps sit under the measured floor. Sensitivity floor stated so the negative is falsifiable: cannot resolve better than ~0.251 delta_cb at 30 beats x 4 seeds. Cause is structural — 81 val pairs against Hemingway's 200, from a 678k-word corpus against 994k — and neither more beats nor more seeds fixes it. 2. A DEFECT IN THE v2 RULE. The floor is the largest within-arm spread across ALL arms, so adding a third noisier arm raised the bar that failed the clean one. Run as a two-arm gate the floor would have been 0.092 and the candidate would have cleared at 2.1x. Deliberately NOT exploited — choosing the floor that passes your preferred answer is the failure pre-registration exists to prevent — but the rule should state whether the floor spans the compared pair or every arm present. As written, a verdict depends on which other arms you happened to run. 3. The two-epochs-on-a-three-epoch-schedule recipe did NOT transfer. Bronte's two minima are 0.0022 apart against a 0.0046 jitter; epoch 2 buys nothing over epoch 1. The epoch-3 collapse (+0.075, ~16x jitter) is the only robust part. The outlier seed was diagnosed rather than waved away: a repeat-5gram degeneracy probe is uniform at 0.0078-0.0102 across every seed and both arms, so it is genuine delta_cb variance and the floor stands. --- .../2026-09-17-lv-bronte-gate.md | 78 +++++++++++++++++++ persistent-memory.md | 1 + 2 files changed, 79 insertions(+) create mode 100644 persistent-memory.d/2026-09-17-lv-bronte-gate.md diff --git a/persistent-memory.d/2026-09-17-lv-bronte-gate.md b/persistent-memory.d/2026-09-17-lv-bronte-gate.md new file mode 100644 index 0000000..098b26a --- /dev/null +++ b/persistent-memory.d/2026-09-17-lv-bronte-gate.md @@ -0,0 +1,78 @@ +# `[2026-09-17]` lv-bronte: corpus gated for real, adapter trained, v2 gate FAILS the voice axis — ship call is the operator's + +**Status: trained and gated, NOT shipped.** The decision is open and belongs to the operator. + +## The gate result, in full + +| axis | result | numbers | +|---|---|---| +| **A. VOICE** | ❌ **FAIL** (both candidates) | ckpt925 +0.210, ckpt475 +0.193 vs base — both **under** the 0.251 measured noise floor | +| **B. NOT COPIED** | ✅ PASS | ckpt475 **0.00 hit-rate, max 0 — identical to the never-saw-it control**; ckpt925 0.01, max 8. Positive control saturates at 160, so the detector is not blind | +| **C. NO DAMAGE** | ✅ PASS | ran-on +0.15 against a 0.400 floor | + +``` +same-author target (held-out Brontë vs itself) delta_cb 0.338 <- best achievable +ckpt925 0.531 +ckpt475 0.548 +base-unadapted 0.741 +``` + +⭐ **THE EFFECT LOOKS REAL AND SUBSTANTIAL; THE INSTRUMENT CANNOT CERTIFY IT.** The reachable +span is 0.741 → 0.338 = 0.403, and the adapters closed **48–52% of everything achievable**. +Both beat base on *every individual seed*. This is an UNDERPOWERED result, not a null one — +and a "no effect" without its floor is unfalsifiable, so: **this method cannot resolve a voice +improvement smaller than ~0.251 delta_cb at 30 beats × 4 seeds on this corpus.** + +⭐⭐ **THE CAUSE IS STRUCTURAL: Brontë's val split yields 81 pairs (44 in-band) against +Hemingway's 200**, because the corpus is 678k words against 994k. Maxing the fixture 30 → 44 +beats would shrink the floor by only ~√1.47 ≈ 1.2× (to ~0.21, against a 0.21 gap) — still +marginal. **More SEEDS would not help either: the floor is a RANGE statistic, and ranges widen +with more samples.** There is no cheap fix. + +## ⚠ A DEFECT IN THE v2 RULE ITSELF, worth fixing before the next author + +The floor is defined as the **largest within-arm seed spread across ALL arms**. Measured here: + +``` +base-unadapted 0.772 0.813 0.751 0.772 spread 0.062 +ckpt475 0.670 0.631 0.604 0.578 spread 0.092 +ckpt925 0.776 0.584 0.525 0.620 spread 0.251 <- sets the floor, on ONE seed +``` + +So **adding a third, noisier arm raised the bar that failed the clean one.** Run as the +two-arm gate (base + ckpt475) the floor would have been 0.092 and +0.193 would have cleared +at 2.1×. This was NOT exploited — picking the floor that passes your preferred answer is the +threshold-chosen-after-seeing-the-numbers failure the pre-registration exists to prevent — but +the rule should say whether the floor is computed over the compared pair or over every arm +present. As written, a candidate's verdict depends on which *other* arms you happened to run. + +**The outlier was diagnosed, not waved away.** Degeneracy probe (fraction of a generation made +of its most repeated 5-gram) is uniform across every seed and both arms, 0.0078–0.0102. Seed +1234 is not a collapsed generation; delta_cb genuinely has that variance for that arm. + +## Which checkpoint, if it ships: **ckpt475** + +The two are 0.017 apart on voice — far inside any floor, i.e. indistinguishable. On the axes +that DO resolve, ckpt475 wins both: memorisation identical to the control (ckpt925 has a +verbatim 8-gram hit), and **2.7× tighter seed-to-seed variance** (0.092 vs 0.251) with no +degeneracy to explain the difference — consistent with ckpt925 sitting nearer the overfit +boundary. Given a coin-flip on voice, take the one that provably did not memorise. + +⭐ **THE RECIPE DID NOT TRANSFER.** Yarros and Hemingway both found their minimum inside +epoch two. Brontë's minima are step 475 (ep 1.00, 2.6107) and step 925 (ep 1.96, 2.6129) — +**0.0022 apart against a 0.0046 median neighbour jitter, i.e. indistinguishable**. Epoch 2 +buys Brontë NOTHING over epoch 1. What IS robust is the epoch-3 collapse: +0.075, ~16× jitter. +Do not carry "two epochs on a three-epoch schedule" to a new author as settled. + +## Artefacts + +`gx10:~/lv-bronte/` (corpus-clean, corpus-renamed, entities-final.json, pairs/, beats-bronte-30.json), +`gx10:~/r49-runs/bronte-4b-pairs-3ep/` (57 checkpoints kept), `gx10:~/r49-runs/bronte-eval/` +(three arms × 120 generations, memorization.txt, voice_distance.txt, score.*.txt). +Commits `fc834a8` `533cc0c` `7964d07` `e9e8c40` `8bb7686`. + +⚠ Two output labels in `voice_distance.py` are hardcoded Yarros strings — it prints +"reference: held-out Yarros" and a boilerplate "Base < Instruct" corroboration line. The +NUMBERS are Brontë's; those two labels are not. Not yet fixed. + +Related: [[2026-09-16-lv-voices-line]], [[2026-09-16-lv-hemingway-corpus]], [[2026-09-16-voices-seat-lora]]. diff --git a/persistent-memory.md b/persistent-memory.md index 672b928..4e071a1 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -152,6 +152,7 @@ before this session began. ⚠ **Do NOT commit them** — untouched and delibera ## Recent decisions +- `[2026-09-17]` ⭐⭐ **lv-bronte trained and gated — v2 gate FAILS the VOICE axis; the ship call is open and the operator's.** Both candidates closed 48–52% of the achievable distance to Brontë but +0.193/+0.210 sit under a 0.251 noise floor set by ONE outlier seed in the arm not being shipped; cause is structural (81 val pairs vs Hemingway's 200) and not cheaply fixable. ckpt475 is the pick if it ships. The two-epoch recipe did NOT transfer. → `persistent-memory.d/2026-09-17-lv-bronte-gate.md` - `[2026-09-16]` ⭐⭐ **Instruction-pair SFT BEATS raw-text for author voice, and the raw-text incumbent never cleared its own control (+0.141 against a 0.153 floor).** `lv-yarros` shipped; the v1 decision rule was amended by the operator after measurement showed it gated on axes the unadapted carrier already maxes. → `persistent-memory.d/2026-09-16-lv-voices-line.md` - `[2026-09-16]` ⭐ **voices-seat live: one carrier, N `lv-` LoRA adapters, hot-swap measured at 0.24 s.** LoRA costs 24.3% of decode against a 0.1% A-vs-A floor and is worth paying; `--gpu-memory-utilization` is a request against TOTAL VRAM and only a pinned KV makes it predictive. → `persistent-memory.d/2026-09-16-voices-seat-lora.md` - `[2026-09-16]` ⭐ **lv-hemingway corpus gated at 994,760 words — and half the catalogue had to be EXCLUDED.** 169,759 words of measured 90–96% collection duplication, a Sherwood Anderson parody, and the author's own name 95 times in publisher back matter; the gender resolver needed a corpus base-rate correction to stop reading women as men. → `persistent-memory.d/2026-09-16-lv-hemingway-corpus.md`