vh
935b35ac2e
Operator: "start the 1.7b training." The 0.6B adapter learned the Gutenberg transcription's ~70-character line breaks along with the prose -- its output wrapped at a mid-length-line ratio of 0.85 against the base model's 0.00. That is typography rather than style, and every further rung would have inherited it, so the corpus is reflowed before rung 2 rather than after the sweep. The reflow joins 57,430 of 85,380 paragraph blocks and keeps 27,950. Verse is the hazard a blind join would destroy, so the decision is per block by median line length: blocks whose lines cluster near the wrap width are flowed prose, blocks of consistently short lines keep their breaks. Every kept multi-line block in the sample was genuinely verse with its lineation intact. No line ended in a lone hyphen, so the space-join could not split a word across lines. The acceptance check is content identity -- " ".join(text.split()) byte-identical before and after -- and it passed on all 852 records, proving only whitespace changed. Concrete cost of the old defect: 5.7% of the training budget was newline tokens. The same words pack to 5,210,112 tokens unwrapped against 5,525,504 wrapped. The 1.7B run is live at 159 steps and roughly 18.7 s/it. Everything but the carrier and the corpus is held from the 0.6B run: seed 4919, rank 32, lr 1e-4, seq 4096, batch 1 by accum 8, one epoch, eval and save every 25 steps so the minimum is located rather than assumed. That corpus change is a second variable and it is named as one. A 0.6B-vs-1.7B comparison is descriptive, not attributable, until the chained 0.6B rerun on the same unwrapped corpus lands behind it -- gated on the 1.7B actually producing an adapter, because a chain that fires on failure turns one lost run into two. "Did sense come back at 1.7B" is a within-arm reading and survives the confound; any between-rung delta does not. The original wrapped corpus is untouched, so the 0.6B run's pinned corpus sha 3959036cf851bf62 stays reproducible.
83 lines
4.4 KiB
Python
83 lines
4.4 KiB
Python
"""Eyeball A/B: does the H02 adapter pull arbitrary prose toward Charlotte Brontë?
|
|
|
|
NOT the adjudication. The frozen rule, the Burrows's-Delta instrument and the
|
|
held-out chapters are untouched by this; nothing here feeds them. This exists
|
|
because the operator asked to *see* whether the voice moved.
|
|
|
|
Two design choices that decide whether the test says anything:
|
|
|
|
* The prompts are deliberately NOT Brontë-ish. Feed a base model "the moors lay
|
|
dark under a bruised sky" and both arms come back Victorian, because the prompt
|
|
did the work. So the set runs a difficulty gradient -- modern/mundane, then
|
|
period-neutral, then Victorian-adjacent-but-plainly-worded -- and any Brontë in
|
|
the modern tier is attributable to the adapter rather than to the setup.
|
|
* Two seeds per prompt per arm, which is nearly free on a 0.6B and is the only
|
|
thing that makes the comparison readable. One sample per arm cannot tell "the
|
|
adapter changed the voice" from "sampling is noisy"; a reader with two samples
|
|
of each arm can at least see whether the between-arm gap exceeds the
|
|
within-arm gap. That is an eyeball noise floor, not a measurement, and it is
|
|
not offered as one.
|
|
|
|
Same harness for both arms -- same box, same sampler, same prompts, same lengths --
|
|
because a cross-comparison whose harness differs is invalid rather than noisy.
|
|
Sampler matches the pinned adjudication sampler (temp 0.9 / top_p 0.95 / 400 new
|
|
tokens) so what is on screen is the same shape of output the real arms produced.
|
|
|
|
⚠ This is a BASE model doing CONTINUATION, and the adapter was trained as pure
|
|
continuation (H02 has no beat annotation by design). It will not follow a "rewrite
|
|
this in Brontë's voice" instruction, and asking it to would test instruction-
|
|
following rather than voice. So each prompt is an opening line the model continues.
|
|
"""
|
|
from __future__ import annotations
|
|
import argparse, json, time
|
|
from pathlib import Path
|
|
import torch
|
|
from transformers import AutoModelForCausalLM, AutoTokenizer
|
|
|
|
ap = argparse.ArgumentParser()
|
|
ap.add_argument("--base", required=True)
|
|
ap.add_argument("--adapter", default=None)
|
|
ap.add_argument("--arm", required=True)
|
|
ap.add_argument("--prompts", required=True)
|
|
ap.add_argument("--out", required=True)
|
|
ap.add_argument("--seeds", type=int, nargs="+", default=[1234, 5678])
|
|
ap.add_argument("--max-new-tokens", type=int, default=400)
|
|
ap.add_argument("--temperature", type=float, default=0.9)
|
|
ap.add_argument("--top-p", type=float, default=0.95)
|
|
a = ap.parse_args()
|
|
|
|
prompts = json.loads(Path(a.prompts).read_text())
|
|
tok = AutoTokenizer.from_pretrained(a.base)
|
|
model = AutoModelForCausalLM.from_pretrained(a.base, dtype=torch.bfloat16,
|
|
attn_implementation="sdpa").to("cuda")
|
|
if a.adapter:
|
|
from peft import PeftModel
|
|
model = PeftModel.from_pretrained(model, a.adapter)
|
|
# ⚠ Prove the adapter actually BOUND. A silent no-op looks exactly like a tune
|
|
# that changed nothing -- which is the very thing this test is trying to see.
|
|
deltas = [float(m.lora_B["default"].weight.abs().sum())
|
|
for m in model.modules() if hasattr(m, "lora_B")]
|
|
nonzero = sum(1 for d in deltas if d > 0)
|
|
print(f"[gen] adapter bound: {nonzero}/{len(deltas)} lora_B tensors non-zero", flush=True)
|
|
if nonzero == 0:
|
|
raise SystemExit("REFUSING: adapter applied but every lora_B is zero -- it did not bind")
|
|
model.eval()
|
|
|
|
out = Path(a.out); out.parent.mkdir(parents=True, exist_ok=True)
|
|
t0 = time.time()
|
|
with out.open("w", encoding="utf-8") as fh:
|
|
for p in prompts:
|
|
for seed in a.seeds:
|
|
torch.manual_seed(seed) # per-sample, so seed N is comparable across arms
|
|
ids = tok(p["prompt"], return_tensors="pt").to("cuda")
|
|
with torch.no_grad():
|
|
g = model.generate(**ids, do_sample=True, temperature=a.temperature,
|
|
top_p=a.top_p, max_new_tokens=a.max_new_tokens,
|
|
pad_token_id=tok.eos_token_id)
|
|
cont = tok.decode(g[0][ids["input_ids"].shape[1]:], skip_special_tokens=True)
|
|
fh.write(json.dumps({"arm": a.arm, "id": p["id"], "tier": p["tier"],
|
|
"prompt": p["prompt"], "seed": seed,
|
|
"continuation": cont}) + "\n")
|
|
print(f" {a.arm} {p['id']} seed={seed} {len(cont.split())}w", flush=True)
|
|
print(f"[gen] {a.arm} -> {out} in {time.time()-t0:.0f}s", flush=True)
|