From 7abd3011f744bc2124a1970f26205148e2cccd78 Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Thu, 20 Aug 2026 07:20:38 -0700 Subject: [PATCH] =?UTF-8?q?fix(coldfusion-abliteration):=20capture=20works?= =?UTF-8?q?=20=E2=80=94=20fp32=20forward=20+=20finite-gate?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The --capture forward NaN'd repeatedly. Root cause: transformers' Qwen3.5 DeltaNet linear-attention needs the causal-conv1d fast-path kernel, which can't be built here (no nvcc, no prebuilt wheel). Its torch fallback produces nondeterministic all-NaN hidden states in bf16 -- same 11-token input finite on one forward, NaN at layer 4 on the next. bf16 and fp32 share exponent range, so it's precision-driven catastrophic cancellation, not overflow, and fp32 resolves it. Fixes: - --capture now loads fp32 (the write/surgery path stays bf16 -- no forward, no NaN). attn_implementation=sdpa pinned. - A finite-gate aborts on a non-finite direction. The sink screen alone can't catch this: nan > threshold is False, so a NaN direction "passed" it and got saved silently on the first run. Capture result (fp32, full GPU): refusal direction finite, unit-normed, layer 22, sink energy 0.0008% in dim 3994 -- clean, not sink-dominated. Saved. Caveat recorded: two-template |cos| agreement is 0.59 at layer 22 vs Robinson's 0.99, almost certainly the small 8/8 calibration set vs their 416/104. Valid but noisier than ideal; the README flags expanding the sets before the write. README documents the three environment gotchas (fp32-for-capture, the seats that must be stopped for the 110GB fp32 VRAM and how to restore them, and the fla side-dir PYTHONPATH) so the next run doesn't rediscover them. --- services/coldfusion-abliteration/README.md | 48 ++++++++++++++++--- .../coldfusion-abliteration/abliterate.py | 23 ++++++++- 2 files changed, 64 insertions(+), 7 deletions(-) diff --git a/services/coldfusion-abliteration/README.md b/services/coldfusion-abliteration/README.md index 3ea8a13..65dfa33 100644 --- a/services/coldfusion-abliteration/README.md +++ b/services/coldfusion-abliteration/README.md @@ -81,11 +81,47 @@ Then, if it holds, NVFP4-quantize via `services/gen-seat-mixed-quant/` and it becomes a gen-seat candidate — **do not delete the incumbent weights** until it survives real multi-turn use (the 2026-08-14 delete-too-early lesson). +## ⚠️ Environment gotchas (2026-08-20 — cost real time, read before re-running) + +**1. transformers' Qwen3.5 DeltaNet linear-attention NaNs in bf16 here.** The +fast-path kernel needs BOTH `flash-linear-attention` (`fla`, triton, installs +fine) AND `causal-conv1d` (needs `nvcc` to build — **absent on ana-ml2, no +prebuilt wheel**). Without causal-conv1d the DeltaNet short-conv runs the torch +fallback, which produces **nondeterministic all-NaN** hidden states in bf16 +(same 11-token input: finite on one forward, NaN at layer 4 on the next). bf16 +and fp32 share exponent range, so this is **precision-driven catastrophic +cancellation, not overflow** — **fp32 resolves it.** + + → **Capture loads fp32** (`abliterate.py` does this automatically in + `--capture` mode). The write/surgery path stays bf16 (no forward, no NaN). + The finite-gate in the script aborts if a direction comes out non-finite — + the sink screen alone won't catch it (`nan > threshold` is False). + +**2. fp32 (110 GB) needs the whole GPU.** Loaded across both Blackwells with +`device_map=auto`, activation memory OOM'd against the resident seats. The +production `vllm-gen` seat (44 GB) had to be **stopped** for the capture, along +with `vllm-meromero-rp` and `vllm-fablefusion-probe`. **Restore after:** +`sudo docker start vllm-gen vllm-meromero-rp vllm-fablefusion-probe`. Set +`PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True`. + +**3. fla lives in a side dir, not the venv.** The shared `quant-work/.venv` is +not llmuser-writable. `fla` + `einops` are installed to +`/tank/aimodels/coldfusion-abliteration/pylibs` and reached via `PYTHONPATH`. +Run every invocation with `PYTHONPATH=/tank/aimodels/coldfusion-abliteration/pylibs`. + ## Status -Harness written 2026-08-19; bf16 fully staged (18 shards). **Dry-run PASSED -against the real checkpoint (2026-08-20):** 1199 tensors, 333 vision preserved, -`down_proj=64 o_proj=16 linear_out=48 mtp=2 embed=1`, coverage gate 6/6, exactly -**131** tensors to orthogonalize — the recipe maps 1:1, no name drift. Next: -`--capture` (direction + sink screen, no write), then the `--out` write on -operator go. **The destructive run has NOT been executed.** +Harness written 2026-08-19; bf16 fully staged. Dry-run PASSED (recipe maps 1:1, +131 tensors). **`--capture` PASSED 2026-08-20** (fp32, after the gotchas above): +refusal direction is **finite, unit-normed, layer 22**, sink energy **0.0008%** +in dim 3994 (recipe L26 ref 0.06%, threshold 1%) — clean, not sink-dominated. +Saved to `qwen38-27b-coldfusion-bf16/refusal-direction.pt`. + +⚠️ **Quality caveat:** two-template `|cos|` agreement at layer 22 is **0.59**, +notably below Robinson's 0.99 — almost certainly the small calibration set (8 +harmful / 8 harmless here vs Robinson's 416 / 104). The direction is valid and +sink-clean, but **before the `--out` write, expand the harmful/harmless sets** +(HARMFUL/HARMLESS in `abliterate.py`) for a cleaner, higher-agreement direction. + +**The destructive `--out` write has NOT been executed** — it gates on the +operator's go (and, recommended, a larger calibration set first). diff --git a/services/coldfusion-abliteration/abliterate.py b/services/coldfusion-abliteration/abliterate.py index 0f99a6f..a366819 100644 --- a/services/coldfusion-abliteration/abliterate.py +++ b/services/coldfusion-abliteration/abliterate.py @@ -238,7 +238,18 @@ def main(): from transformers import AutoModelForCausalLM, AutoTokenizer print("\nloading model (bf16, device_map=auto across the Blackwells)...") tok = AutoTokenizer.from_pretrained(model_dir) - model = AutoModelForCausalLM.from_pretrained(model_dir, dtype=torch.bfloat16, device_map="auto") + # DTYPE IS LOAD-BEARING FOR CAPTURE. This is a Qwen3_5 hybrid (DeltaNet + # linear-attn + full-attn). Without the causal_conv1d fast-path kernel + # (unbuildable here — no nvcc), the DeltaNet recurrence runs the torch + # fallback, which produces NONDETERMINISTIC NaN hidden states in bf16 + # (verified 2026-08-20: same 11-token input finite on one forward, NaN at + # layer 4 on the next). bf16 and fp32 share exponent range, so this is + # PRECISION-driven catastrophic cancellation, not overflow — fp32's mantissa + # resolves it. Capture therefore loads fp32 (fits: 98GB GPU + CPU offload, + # 244GB RAM free). The surgery/write path takes bf16 (no forward, no NaN). + load_dtype = torch.float32 if args.capture else torch.bfloat16 + model = AutoModelForCausalLM.from_pretrained( + model_dir, dtype=load_dtype, device_map="auto", attn_implementation="sdpa") model.eval() device = next(model.parameters()).device @@ -257,6 +268,16 @@ def main(): f"(|cos|={float(agree.max()):.4f}); using layer {layer} " f"(|cos|={float(agree[layer]):.4f}, recipe anchor {DEFAULT_LAYER})") + # --- finite gate: a NaN/Inf direction must NEVER pass silently ----------- + # (the sink screen alone doesn't catch this — `nan > threshold` is False, so + # a NaN direction would "pass" the sink gate. This is the real guard.) + if not torch.isfinite(d_unit).all(): + frac = float(torch.isfinite(d_unit).float().mean()) + print(f"\n!! captured direction is NOT finite (finite frac {frac:.3f}) — " + "the forward pass produced NaN/Inf. Check attn_implementation and the " + "fla/linear-attn path; do NOT abliterate on this direction.", file=sys.stderr) + sys.exit(4) + # --- attention-sink screen (the brick-the-model gate) --------------------- e = sink_energy(d_unit) print(f"attention-sink screen: dim {SINK_DIM} carries {e*100:.3f}% of layer-{layer} direction energy "