diff --git a/scripts/mccarthy-corpus/gate-results/ADAPTER-README.md b/scripts/mccarthy-corpus/gate-results/ADAPTER-README.md new file mode 100644 index 0000000..021ccab --- /dev/null +++ b/scripts/mccarthy-corpus/gate-results/ADAPTER-README.md @@ -0,0 +1,149 @@ +# lv-mccarthy-4b-v1 + +LoRA adapter over Qwen3-4B-Instruct for the prose voice of **Cormac McCarthy**. +Checkpoint: `mccarthy-4b-pairs-3ep/checkpoints/checkpoint-300` (epoch 0.652, eval_loss 2.4490). +Rank 32, alpha 64, seq 1536, lr 1e-4, seed 4919. Trained on 3,673 instruction pairs over a +584,684-word six-novel corpus, renamed. Staged 2026-09-21. + +⚠⚠ **STAGED, NOT SHIPPED.** At the time of writing this directory is **not** referenced by +`stacks/voices-seat/compose.yaml --lora-modules`, so the seat cannot load it. It passes the +gate design of record on all three axes but carries a real length-discipline cost (below) +that is an operator trade, not a gate reading. **If `lv-mccarthy` is not in the seat's model +list, that decision has not been made yet.** + +Gate design pre-registered before any generation existed, with three dated amendments: +eshpfi `scripts/mccarthy-corpus/GATE-PREREG.md`. Raw artifacts for all five arms: +eshpfi `scripts/mccarthy-corpus/gate-results/`. + +## The gate result — 5 arms, 60 held-out beats × 4 seeds = 240 generations per arm + +| axis | result | numbers | +|---|---|---| +| **A. VOICE** | ✅ **PASS, 3.2× the floor — best arm of five** | +0.177 delta_cb toward held-out McCarthy vs the base control, pairwise floor 0.055. Best point estimate AND best margin in the run | +| **B. NOT COPIED** | ✅ **PASS — indistinguishable from the author** | 0.12 hit-rate against **held-out McCarthy's own 0.12**; longest match **10 words against the author's coincidental 12** | +| **C. NO DAMAGE** | ⚠ **PASSES the rule, FAILS on the axis the rule cannot see — read below** | ran-on +0.12 vs a 0.200 floor (40% headroom). But **in-band 0.65 against base's 0.89** | + +``` +same-author target (held-out McCarthy vs itself) delta_cb 0.370 <- best achievable +ckpt300 (this adapter) 0.484 +ckpt225 0.485 +ckpt900 (the eval-loss minimum) 0.490 +ckpt450 0.509 +base-unadapted 0.661 +``` + +Reachable span 0.661 → 0.370 = 0.291; this adapter closed **0.177 = 60.8%**. For scale, +lv-bronte closed 48% and lv-hemingway 73.8% — and McCarthy's voice axis is **deliberately +harder** than either: the `mccarthy` register names its own punctuation tics, and the eval +drives the base control with the same prompt, so the adapter earns no delta for "emit no +quotation marks". Verified: base's quote density is 19.9 per 10k against the reference's 0.0. + +⭐ **The gain is not the punctuation trick.** A pre-registered secondary read strips *every* +punctuation mark from the reference and every arm: this adapter still scores **+0.128 at 2.8× +its floor**, so roughly three quarters of the voice gain is carried by words rather than marks. + +## ✅ AXIS B: the cleanest safety result in the lv- line + +McCarthy is **in copyright with a living estate**, so this axis carries more weight here than +for lv-bronte (public domain) or lv-hemingway. + +``` +sample n hit-rate mean-longest max +HELD-OUT McCARTHY (never trained) 364 0.12 1.0 12 <- the innocent rate +base-unadapted 240 0.00 0.0 8 +ckpt300 (this adapter) 240 0.12 1.0 10 +ckpt450 240 0.12 1.1 11 +ckpt900 240 0.22 1.9 11 +positive control (train vs train) 160 <- detector not blind +``` + +⚠ **The base-unadapted control is DEFECTIVE and its zero is unearned** — base writes summary, +the adapted arms write pastiche, and text that does not imitate a register cannot collide with +its n-grams. The honest reference is the author himself. Against base, this adapter's 0.12 vs +0.00 reads as a 12× red flag; against the author's own 0.12 it is **1.0×**, and its longest +verbatim run is **shorter** than the one genuinely unseen McCarthy shares with the train split +by coincidence. + +⭐ Measured, not assumed: the "his plain register makes 8-gram collisions inevitable" story +that was **FALSE for Hemingway** (0.01) is **TRUE for McCarthy** (0.12). His idiom really does +self-collide. Neither number transfers between authors. + +**All 31 matched runs were READ, not counted.** Every one is stock grammar in the commonest +words: + +``` +looked at the wolf and he looked at the boy +took off his hat and set it on the +and wiped his mouth on the back of his +He was naked to the waist and he +``` + +The name-shaped hits (`Adam Caleb`, `Colton`) are the **RENAMED invented names**, not +McCarthy's. No plot, no imagery, nothing protectable. + +## ⚠⚠ AXIS C: THE COST. READ THIS BEFORE PUTTING IT ON A LENGTH-SENSITIVE PATH + +The gate's axis C is `ran-on` only (operator-ratified 2026-09-15, because the unadapted +carrier maxes the other terms). This adapter passes it with 40% headroom. **That rule is +structurally blind to the cost that matters here**, and the cost is real: + +``` +arm n median p90 max in-band under-90 over-140 TOTAL out-of-band +base 240 110 126 149 0.89 10% 1% 11% +ckpt300 240 101 151 216 0.65 22% 13% 35% +ckpt450 240 107 171 297 0.67 14% 20% 33% +ckpt900 240 114 190 279 0.67 5% 28% 33% +ckpt225 240 127 206 296 0.52 10% 38% 48% +``` + +**35% of this adapter's generations miss the requested 90–140 band, against base's 11%**, and +on-beat keyword coverage drops 0.42 vs base's 0.71. Base is *good* at length here — McCarthy's +register prompt is prescriptive and Qwen3-4B-Instruct follows it — so the adapter measurably +makes instruction-following worse. + +⚠ **No checkpoint choice fixes this.** Every adapted arm sits at 33–48% out-of-band. What +changes with epoch is the *direction* of the miss and its size, non-monotonically: ran-on runs +0.38 → **0.13** → 0.20 → 0.28 across epochs 0.49 / 0.65 / 0.98 / 1.96. This checkpoint is the +measured minimum. The fix, if the cost is intolerable, is a retrain targeting length — pair +construction or the length target — not a different checkpoint. + +**If Skaldsong or any caller depends on a word budget, expect one generation in three to miss +it.** Unload is 0.003 s and one compose line. + +## Why checkpoint-300 and NOT ckpt900, the eval-loss minimum + +⭐ **The two-epoch recipe is now 0 for 3.** ckpt900 is the curve's minimum (2.38706) and +ckpt300 is +0.062 above it — **15.8× the 0.00393 median neighbour jitter**, i.e. the loss curve +is *confident* ckpt900 is better. It is wrong on every axis that resolves: + +| | ckpt300 (this adapter) | ckpt900 (loss min) | +|---|---|---| +| voice margin | **3.2× floor** | 1.2× floor | +| seed spread | **0.055** | 0.148 — **2.7× wider**, one outlier seed | +| memorisation vs the author's 0.12 | **0.12 (1.0×)** | 0.22 (1.8×) | +| ran-on | **0.13** | 0.28 | +| epochs of overfit | **0.65** | 1.96 | + +ckpt900's spread is one seed — 0.605 against 0.457 / 0.531 / 0.554 — the third occurrence of +that shape in the later checkpoint, after lv-bronte's ckpt925 and lv-hemingway's ckpt1750. +**On this schedule the eval-loss minimum is not the ship candidate, and the curve's confidence +about it carries no information.** + +## Corpus provenance + +Six novels (Blood Meridian, All the Pretty Horses, The Crossing, Cities of the Plain, No +Country for Old Men, The Road), 167 units, 584,684 words, 6 renamed copies, val split held at +chapters 7 and 17 = 11 units / 40,653 words per copy. + +Leak gate at gate time: **0 of 75 renameable, 0 of 37 sub-threshold, 0 separator-split**, four +controls green. Beat-contamination audit: **0 of 3,942 beats and 0 of 3,942 responses** — +against lv-hemingway's 70 of 7,094 — because `build_sft_pairs.py --source-entities` was +mandatory for this corpus's pair build. + +⚠ Sensitivity floor, part of the result and not a footnote: a name under 8 capitals per work is +never detected, a phrase under 5 recurrences never audited, and **no phrase map exists for this +corpus, so the phrase audit never ran at all.** lv-yarros and lv-bronte both had one. A clean +leak gate here is a narrower claim than a clean leak gate there. + +⚠ **`repair_typography.py` MUST NOT be run on this corpus** — it would normalise quotation +marks back into a corpus that measures 0.0 per 10k against Hemingway's 838. diff --git a/scripts/r49-corpus/ship-voice-adapter.sh b/scripts/r49-corpus/ship-voice-adapter.sh index 968d421..c91bef6 100755 --- a/scripts/r49-corpus/ship-voice-adapter.sh +++ b/scripts/r49-corpus/ship-voice-adapter.sh @@ -2,7 +2,12 @@ # Ship an r49 voice LoRA from the training box to the voices seat's adapter store. # # scripts/r49-corpus/ship-voice-adapter.sh -# e.g. ... ~/r49-runs/mccarthy-4b-pairs-3ep checkpoint-900 lv-mccarthy-4b-v1 +# e.g. ... /home/infra-ops/r49-runs/mccarthy-4b-pairs-3ep checkpoint-300 lv-mccarthy-4b-v1 +# +# ⚠ MUST BE AN ABSOLUTE REMOTE PATH. A leading `~` is expanded by the LOCAL +# shell before the script sees it, so `~/r49-runs/...` becomes /home/lkraven/... and the +# source read fails with "Permission denied" on gx10 -- a confusing error for a path +# problem. The guard below refuses it outright rather than letting ssh report it. # # Run from nh3-dev. gx10 and fv-ml1 both hold only authorized_keys, so neither can # INITIATE to the other -- the relay through this box is the path, and it costs nothing @@ -26,6 +31,12 @@ NAME=${3:?missing adapter name} SRC_HOST=${SRC_HOST:-infra-ops@10.100.50.60} # pfi-gx10 DST_HOST=${DST_HOST:-infra-ops@10.251.50.54} # fv-ml1 DST_ROOT=${DST_ROOT:-/tank/aimodels/voice-adapters} +case "$RUN" in + /*) ;; + *) echo "== REFUSING: must be an ABSOLUTE remote path, got '$RUN'." + echo " A leading ~ is expanded by the local shell and will point at the wrong user." + exit 2 ;; +esac STAGE=$(mktemp -d) trap 'rm -rf "$STAGE"' EXIT