vh
5ee2325820
Answers "how likely is it that our abliteration caused this?" with a measurement instead of a prior. P(<think>) at the first generated token, template rendered enable_thinking=false so the prompt already carries a CLOSED think pair -- the exact event behind the leak. Raw softmax, bf16, CPU-only, one process per model. Deterministic: stock reproduced to 17 significant figures across two runs. coldfusion-bf16 none (stock) 0.1850 rank 3 coldfusion-abliterated-L35-bf16 Robinson L35, mild 0.2048 rank 2 coldfusion-h300-mtp-bf16 Heretic-300, heavy 0.2216 rank 2 The stock, untouched base already puts 18.5% of first-token mass on opening a think block the template had closed. Abliteration adds a real, monotonic, dose-dependent +3.7 points -- a nudge on a pre-existing base, not the cause. Cold-Fusion is a reasoning-token-compression finetune, i.e. a model trained to think briefly, and the leak's text shape agrees: a compact correct trace with a trained transition marker, which is trained behavior rather than damage. This changes the options. Rolling back to L35 or stock does NOT fix the leak -- at 18.5% under temp 0.7 / top_p 0.8 they leak at nearly the h300 rate. Only leaving the Cold-Fusion family escapes it, at the cost of the 8/100 refusal result. The chat_template_kwargs fix is the correct lever. Durable methodology point: a forward-KL budget cannot catch this. Heretic minimizes forward KL(stock||abliterated), which is near-blind to the model putting new mass on tokens stock barely used -- that is reverse KL's job, and we measured exactly that asymmetry on L35 (reverse 1.43 vs forward 0.70). h300's KL of 0.0136 is not evidence of innocence. For any "did the abliteration break behavior X" question, measure P(token) directly. Ran CPU-only deliberately: 96 EPYC cores and 265 GB of RAM make a 27B forward pass cheap, so this cost no GPU window and no seat downtime, where the obvious route was stopping both GPU0 seats. Also normalizes two more abliteration output dirs from root-owned 0600 to llmuser 0664. The unreadable-model failure surfaces as FileNotFoundError rather than a permission error, which is worth knowing before it wastes a run.
39 lines
1.6 KiB
Python
39 lines
1.6 KiB
Python
#!/usr/bin/env python3
|
|
"""P(<think>) at the first generated token, with enable_thinking=False.
|
|
|
|
Measures how much probability mass a checkpoint puts on OPENING a think block
|
|
when the chat template has ALREADY closed one for it. That is the exact event
|
|
behind the h300 gen-seat leak. CPU-only: no GPU contention, no seat downtime.
|
|
"""
|
|
import json, sys, torch
|
|
from transformers import AutoTokenizer, AutoModelForCausalLM
|
|
|
|
path = sys.argv[1]
|
|
PROMPT = ("A farmer has 17 sheep. All but 9 run away. He buys twice as many as he "
|
|
"has left, then sells 4. How many now? Explain.")
|
|
|
|
tok = AutoTokenizer.from_pretrained(path)
|
|
text = tok.apply_chat_template([{"role": "user", "content": PROMPT}],
|
|
tokenize=False, add_generation_prompt=True,
|
|
enable_thinking=False)
|
|
assert text.rstrip().endswith("</think>"), "template did NOT pre-close the think block:\n" + repr(text[-120:])
|
|
|
|
ids = tok(text, return_tensors="pt")
|
|
model = AutoModelForCausalLM.from_pretrained(path, dtype=torch.bfloat16, device_map=None)
|
|
model.eval()
|
|
with torch.no_grad():
|
|
logits = model(**ids).logits[0, -1].float()
|
|
probs = torch.softmax(logits, dim=-1)
|
|
|
|
think_id = tok.convert_tokens_to_ids("<think>")
|
|
p_think = probs[think_id].item()
|
|
top = torch.topk(probs, 12)
|
|
out = {
|
|
"model": path.rstrip("/").split("/")[-1],
|
|
"p_think": p_think,
|
|
"think_token_id": think_id,
|
|
"think_rank": int((probs > p_think).sum().item()) + 1,
|
|
"top12": [{"tok": tok.decode([i]), "p": round(probs[i].item(), 5)} for i in top.indices.tolist()],
|
|
}
|
|
print("RESULT " + json.dumps(out))
|