fix(coldfusion-abliteration): capture works — fp32 forward + finite-gate

The --capture forward NaN'd repeatedly. Root cause: transformers' Qwen3.5
DeltaNet linear-attention needs the causal-conv1d fast-path kernel, which
can't be built here (no nvcc, no prebuilt wheel). Its torch fallback produces
nondeterministic all-NaN hidden states in bf16 -- same 11-token input finite
on one forward, NaN at layer 4 on the next. bf16 and fp32 share exponent
range, so it's precision-driven catastrophic cancellation, not overflow, and
fp32 resolves it.

Fixes:
- --capture now loads fp32 (the write/surgery path stays bf16 -- no forward,
  no NaN). attn_implementation=sdpa pinned.
- A finite-gate aborts on a non-finite direction. The sink screen alone can't
  catch this: nan > threshold is False, so a NaN direction "passed" it and got
  saved silently on the first run.

Capture result (fp32, full GPU): refusal direction finite, unit-normed, layer
22, sink energy 0.0008% in dim 3994 -- clean, not sink-dominated. Saved.

Caveat recorded: two-template |cos| agreement is 0.59 at layer 22 vs Robinson's
0.99, almost certainly the small 8/8 calibration set vs their 416/104. Valid
but noisier than ideal; the README flags expanding the sets before the write.

README documents the three environment gotchas (fp32-for-capture, the seats
that must be stopped for the 110GB fp32 VRAM and how to restore them, and the
fla side-dir PYTHONPATH) so the next run doesn't rediscover them.
This commit is contained in:
vh
2026-08-20 07:20:38 -07:00
parent b56cb0db13
commit 7abd3011f7
2 changed files with 64 additions and 7 deletions
+22 -1
View File
@@ -238,7 +238,18 @@ def main():
from transformers import AutoModelForCausalLM, AutoTokenizer
print("\nloading model (bf16, device_map=auto across the Blackwells)...")
tok = AutoTokenizer.from_pretrained(model_dir)
model = AutoModelForCausalLM.from_pretrained(model_dir, dtype=torch.bfloat16, device_map="auto")
# DTYPE IS LOAD-BEARING FOR CAPTURE. This is a Qwen3_5 hybrid (DeltaNet
# linear-attn + full-attn). Without the causal_conv1d fast-path kernel
# (unbuildable here — no nvcc), the DeltaNet recurrence runs the torch
# fallback, which produces NONDETERMINISTIC NaN hidden states in bf16
# (verified 2026-08-20: same 11-token input finite on one forward, NaN at
# layer 4 on the next). bf16 and fp32 share exponent range, so this is
# PRECISION-driven catastrophic cancellation, not overflow — fp32's mantissa
# resolves it. Capture therefore loads fp32 (fits: 98GB GPU + CPU offload,
# 244GB RAM free). The surgery/write path takes bf16 (no forward, no NaN).
load_dtype = torch.float32 if args.capture else torch.bfloat16
model = AutoModelForCausalLM.from_pretrained(
model_dir, dtype=load_dtype, device_map="auto", attn_implementation="sdpa")
model.eval()
device = next(model.parameters()).device
@@ -257,6 +268,16 @@ def main():
f"(|cos|={float(agree.max()):.4f}); using layer {layer} "
f"(|cos|={float(agree[layer]):.4f}, recipe anchor {DEFAULT_LAYER})")
# --- finite gate: a NaN/Inf direction must NEVER pass silently -----------
# (the sink screen alone doesn't catch this — `nan > threshold` is False, so
# a NaN direction would "pass" the sink gate. This is the real guard.)
if not torch.isfinite(d_unit).all():
frac = float(torch.isfinite(d_unit).float().mean())
print(f"\n!! captured direction is NOT finite (finite frac {frac:.3f}) — "
"the forward pass produced NaN/Inf. Check attn_implementation and the "
"fla/linear-attn path; do NOT abliterate on this direction.", file=sys.stderr)
sys.exit(4)
# --- attention-sink screen (the brick-the-model gate) ---------------------
e = sink_energy(d_unit)
print(f"attention-sink screen: dim {SINK_DIM} carries {e*100:.3f}% of layer-{layer} direction energy "