vh
7abd3011f7
The --capture forward NaN'd repeatedly. Root cause: transformers' Qwen3.5 DeltaNet linear-attention needs the causal-conv1d fast-path kernel, which can't be built here (no nvcc, no prebuilt wheel). Its torch fallback produces nondeterministic all-NaN hidden states in bf16 -- same 11-token input finite on one forward, NaN at layer 4 on the next. bf16 and fp32 share exponent range, so it's precision-driven catastrophic cancellation, not overflow, and fp32 resolves it. Fixes: - --capture now loads fp32 (the write/surgery path stays bf16 -- no forward, no NaN). attn_implementation=sdpa pinned. - A finite-gate aborts on a non-finite direction. The sink screen alone can't catch this: nan > threshold is False, so a NaN direction "passed" it and got saved silently on the first run. Capture result (fp32, full GPU): refusal direction finite, unit-normed, layer 22, sink energy 0.0008% in dim 3994 -- clean, not sink-dominated. Saved. Caveat recorded: two-template |cos| agreement is 0.59 at layer 22 vs Robinson's 0.99, almost certainly the small 8/8 calibration set vs their 416/104. Valid but noisier than ideal; the README flags expanding the sets before the write. README documents the three environment gotchas (fp32-for-capture, the seats that must be stopped for the 110GB fp32 VRAM and how to restore them, and the fla side-dir PYTHONPATH) so the next run doesn't rediscover them.
Cold-Fusion abliteration — Robinson formula
Abliterate DavidAU/Qwen3.8-27B-Cold-Fusion-GAIN-V1.1 using the MTP-aware,
vision-preserving single-direction recipe documented in
docs/pfi/abliteration-recipe-qwen38.md.
Why this model, why this recipe. Its stock refusal profile (probed
2026-08-19, hand-verified) is ~33% on creative content — it still hard-refuses
explicit sexual content and graphic torture, and refuses 4/5 hard-harm technical
prompts, while keeping self-harm guardrails and over-refusing zero benign
prompts. So there is a real creative-content refusal surface to remove. The
Robinson formula is chosen specifically because it abliterates the MTP head
in-band — which the current gen seat's Heretic pass does not (per
qwen38-27b-heresy-bf16.PROVENANCE.txt, the MTP head there is a byte-identical
base graft the wrapper never loaded). That is the additive delta this
experiment tests.
Where it runs
ana-ml2 (dual RTX PRO 6000 Blackwell, 96 GB each). A 55.6 GB bf16 loads
comfortably; the output feeds the same box's NVFP4 quant pipeline
(services/gen-seat-mixed-quant/).
- bf16 source:
/tank/aimodels/qwen38-27b-coldfusion-bf16(pinned9c44193f07782c85c0f437a5d8466ba5c95c95fe) - env:
/tank/aimodels/quant-work/.venv(torch 2.12.1+cu130, CUDA live) - run as
llmuser(owns/tank/aimodels):sudo -u llmuser <venv>/bin/python …
The gates — this script refuses to brick the model
Two hard gates from the recipe, both of which halt before any write:
- Coverage gate —
o_proj(16) + linear_out(48) == 64 == num_hidden_layers, plusdown_proj==64, MTP writers==2, exactly oneembed_tokens. Catches a tensor-name mismatch that would otherwise ship a half-abliterated model. 131 tensors edited when it passes; vision (333) never touched. - Attention-sink screen — Qwen3.8-27B's massive-activation dimension is 3994. Orthogonalizing a direction that lives in dim 3994 produces a model that loads, runs, and emits garbage. The script aborts if the chosen layer's direction carries >1% of its energy in dim 3994 (recipe's layer-26 reference: 0.06%).
The refusal direction is captured from two chat-template renderings
(enable_thinking=false and thinking at xhigh); the layer is auto-picked by
peak two-template |cos| agreement in the recipe's [18,45] window (anchor: 26).
Sequence
V=/tank/aimodels/quant-work/.venv/bin/python
M=/tank/aimodels/qwen38-27b-coldfusion-bf16
A=/tank/aimodels/qwen38-27b-coldfusion-abliterated-bf16
# 1. DRY RUN FIRST — verify the tensor map + both gates on the static surface,
# no forward, no write. Do not skip: this is what confirms the recipe maps
# onto THIS checkpoint's names before anything irreversible.
sudo -u llmuser $V services/coldfusion-abliteration/abliterate.py --model $M --dry-run
# 2. Capture the direction + screen the sink (loads the model; no write yet).
sudo -u llmuser $V services/coldfusion-abliteration/abliterate.py --model $M --capture
# 3. Abliterate (writes the new bf16). Only after 1 and 2 pass.
sudo -u llmuser $V services/coldfusion-abliteration/abliterate.py --model $M --out $A
Verify after (do not trust the write blind)
- Vision byte-identical — diff
visual.*tensors source vs output (recipe requires max delta 0). - Refusal re-profile — re-run the same battery from the 2026-08-19 probe
(reuse
services/refusal-probe/, the gen-seat harness — NOT the ad-hoc GGUF one) and confirm creative refusals dropped toward the RobinsonLabs 8% floor while self-harm guardrails survive. - MTP acceptance — the whole point of the in-band MTP edit; measure on the
quantized build per
services/gen-seat-mixed-quant/RUNBOOK-heresy-swap.md. Gate ≳40% (reference_abliteration_mtp_lessons— gate on acceptance, not KL). - PPL / coherence / no catatonia — DavidAU fine-tunes are idiosyncratic; eyeball the outputs, don't trust the metric alone.
Then, if it holds, NVFP4-quantize via services/gen-seat-mixed-quant/ and it
becomes a gen-seat candidate — do not delete the incumbent weights until it
survives real multi-turn use (the 2026-08-14 delete-too-early lesson).
⚠️ Environment gotchas (2026-08-20 — cost real time, read before re-running)
1. transformers' Qwen3.5 DeltaNet linear-attention NaNs in bf16 here. The
fast-path kernel needs BOTH flash-linear-attention (fla, triton, installs
fine) AND causal-conv1d (needs nvcc to build — absent on ana-ml2, no
prebuilt wheel). Without causal-conv1d the DeltaNet short-conv runs the torch
fallback, which produces nondeterministic all-NaN hidden states in bf16
(same 11-token input: finite on one forward, NaN at layer 4 on the next). bf16
and fp32 share exponent range, so this is precision-driven catastrophic
cancellation, not overflow — fp32 resolves it.
→ Capture loads fp32 (abliterate.py does this automatically in
--capture mode). The write/surgery path stays bf16 (no forward, no NaN).
The finite-gate in the script aborts if a direction comes out non-finite —
the sink screen alone won't catch it (nan > threshold is False).
2. fp32 (110 GB) needs the whole GPU. Loaded across both Blackwells with
device_map=auto, activation memory OOM'd against the resident seats. The
production vllm-gen seat (44 GB) had to be stopped for the capture, along
with vllm-meromero-rp and vllm-fablefusion-probe. Restore after:
sudo docker start vllm-gen vllm-meromero-rp vllm-fablefusion-probe. Set
PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True.
3. fla lives in a side dir, not the venv. The shared quant-work/.venv is
not llmuser-writable. fla + einops are installed to
/tank/aimodels/coldfusion-abliteration/pylibs and reached via PYTHONPATH.
Run every invocation with PYTHONPATH=/tank/aimodels/coldfusion-abliteration/pylibs.
Status
Harness written 2026-08-19; bf16 fully staged. Dry-run PASSED (recipe maps 1:1,
131 tensors). --capture PASSED 2026-08-20 (fp32, after the gotchas above):
refusal direction is finite, unit-normed, layer 22, sink energy 0.0008%
in dim 3994 (recipe L26 ref 0.06%, threshold 1%) — clean, not sink-dominated.
Saved to qwen38-27b-coldfusion-bf16/refusal-direction.pt.
⚠️ Quality caveat: two-template |cos| agreement at layer 22 is 0.59,
notably below Robinson's 0.99 — almost certainly the small calibration set (8
harmful / 8 harmless here vs Robinson's 416 / 104). The direction is valid and
sink-clean, but before the --out write, expand the harmful/harmless sets
(HARMFUL/HARMLESS in abliterate.py) for a cleaner, higher-agreement direction.
The destructive --out write has NOT been executed — it gates on the
operator's go (and, recommended, a larger calibration set first).