fix(coldfusion-abliteration): capture works — fp32 forward + finite-gate
The --capture forward NaN'd repeatedly. Root cause: transformers' Qwen3.5 DeltaNet linear-attention needs the causal-conv1d fast-path kernel, which can't be built here (no nvcc, no prebuilt wheel). Its torch fallback produces nondeterministic all-NaN hidden states in bf16 -- same 11-token input finite on one forward, NaN at layer 4 on the next. bf16 and fp32 share exponent range, so it's precision-driven catastrophic cancellation, not overflow, and fp32 resolves it. Fixes: - --capture now loads fp32 (the write/surgery path stays bf16 -- no forward, no NaN). attn_implementation=sdpa pinned. - A finite-gate aborts on a non-finite direction. The sink screen alone can't catch this: nan > threshold is False, so a NaN direction "passed" it and got saved silently on the first run. Capture result (fp32, full GPU): refusal direction finite, unit-normed, layer 22, sink energy 0.0008% in dim 3994 -- clean, not sink-dominated. Saved. Caveat recorded: two-template |cos| agreement is 0.59 at layer 22 vs Robinson's 0.99, almost certainly the small 8/8 calibration set vs their 416/104. Valid but noisier than ideal; the README flags expanding the sets before the write. README documents the three environment gotchas (fp32-for-capture, the seats that must be stopped for the 110GB fp32 VRAM and how to restore them, and the fla side-dir PYTHONPATH) so the next run doesn't rediscover them.
This commit is contained in:
@@ -238,7 +238,18 @@ def main():
|
||||
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||||
print("\nloading model (bf16, device_map=auto across the Blackwells)...")
|
||||
tok = AutoTokenizer.from_pretrained(model_dir)
|
||||
model = AutoModelForCausalLM.from_pretrained(model_dir, dtype=torch.bfloat16, device_map="auto")
|
||||
# DTYPE IS LOAD-BEARING FOR CAPTURE. This is a Qwen3_5 hybrid (DeltaNet
|
||||
# linear-attn + full-attn). Without the causal_conv1d fast-path kernel
|
||||
# (unbuildable here — no nvcc), the DeltaNet recurrence runs the torch
|
||||
# fallback, which produces NONDETERMINISTIC NaN hidden states in bf16
|
||||
# (verified 2026-08-20: same 11-token input finite on one forward, NaN at
|
||||
# layer 4 on the next). bf16 and fp32 share exponent range, so this is
|
||||
# PRECISION-driven catastrophic cancellation, not overflow — fp32's mantissa
|
||||
# resolves it. Capture therefore loads fp32 (fits: 98GB GPU + CPU offload,
|
||||
# 244GB RAM free). The surgery/write path takes bf16 (no forward, no NaN).
|
||||
load_dtype = torch.float32 if args.capture else torch.bfloat16
|
||||
model = AutoModelForCausalLM.from_pretrained(
|
||||
model_dir, dtype=load_dtype, device_map="auto", attn_implementation="sdpa")
|
||||
model.eval()
|
||||
device = next(model.parameters()).device
|
||||
|
||||
@@ -257,6 +268,16 @@ def main():
|
||||
f"(|cos|={float(agree.max()):.4f}); using layer {layer} "
|
||||
f"(|cos|={float(agree[layer]):.4f}, recipe anchor {DEFAULT_LAYER})")
|
||||
|
||||
# --- finite gate: a NaN/Inf direction must NEVER pass silently -----------
|
||||
# (the sink screen alone doesn't catch this — `nan > threshold` is False, so
|
||||
# a NaN direction would "pass" the sink gate. This is the real guard.)
|
||||
if not torch.isfinite(d_unit).all():
|
||||
frac = float(torch.isfinite(d_unit).float().mean())
|
||||
print(f"\n!! captured direction is NOT finite (finite frac {frac:.3f}) — "
|
||||
"the forward pass produced NaN/Inf. Check attn_implementation and the "
|
||||
"fla/linear-attn path; do NOT abliterate on this direction.", file=sys.stderr)
|
||||
sys.exit(4)
|
||||
|
||||
# --- attention-sink screen (the brick-the-model gate) ---------------------
|
||||
e = sink_energy(d_unit)
|
||||
print(f"attention-sink screen: dim {SINK_DIM} carries {e*100:.3f}% of layer-{layer} direction energy "
|
||||
|
||||
Reference in New Issue
Block a user