esh-pfi-infrastructure/services/coldfusion-abliteration

Cold-Fusion abliteration — Robinson formula

Abliterate DavidAU/Qwen3.8-27B-Cold-Fusion-GAIN-V1.1 using the MTP-aware, vision-preserving single-direction recipe documented in docs/pfi/abliteration-recipe-qwen38.md.

Why this model, why this recipe. Its stock refusal profile (probed 2026-08-19, hand-verified) is ~33% on creative content — it still hard-refuses explicit sexual content and graphic torture, and refuses 4/5 hard-harm technical prompts, while keeping self-harm guardrails and over-refusing zero benign prompts. So there is a real creative-content refusal surface to remove. The Robinson formula is chosen specifically because it abliterates the MTP head in-band — which the current gen seat's Heretic pass does not (per qwen38-27b-heresy-bf16.PROVENANCE.txt, the MTP head there is a byte-identical base graft the wrapper never loaded). That is the additive delta this experiment tests.

Where it runs

ana-ml2 (dual RTX PRO 6000 Blackwell, 96 GB each). A 55.6 GB bf16 loads comfortably; the output feeds the same box's NVFP4 quant pipeline (services/gen-seat-mixed-quant/).

• bf16 source: /tank/aimodels/qwen38-27b-coldfusion-bf16 (pinned 9c44193f07782c85c0f437a5d8466ba5c95c95fe)
• env: /tank/aimodels/quant-work/.venv (torch 2.12.1+cu130, CUDA live)
• run as llmuser (owns /tank/aimodels): sudo -u llmuser <venv>/bin/python …

The gates — this script refuses to brick the model

Two hard gates from the recipe, both of which halt before any write:

1. Coverage gate — o_proj(16) + linear_out(48) == 64 == num_hidden_layers, plus down_proj==64, MTP writers ==2, exactly one embed_tokens. Catches a tensor-name mismatch that would otherwise ship a half-abliterated model. 131 tensors edited when it passes; vision (333) never touched.
2. Attention-sink screen — Qwen3.8-27B's massive-activation dimension is 3994. Orthogonalizing a direction that lives in dim 3994 produces a model that loads, runs, and emits garbage. The script aborts if the chosen layer's direction carries >1% of its energy in dim 3994 (recipe's layer-26 reference: 0.06%).

The refusal direction is captured from two chat-template renderings (enable_thinking=false and thinking at xhigh); the layer is auto-picked by peak two-template |cos| agreement in the recipe's [18,45] window (anchor: 26). Verified 2026-08-20 against chat_template.jinja: enable_thinking=True resolves reasoning_effort to 'xhigh' by default, so these really are the recipe's two renderings — the low agreement is not a template-selection bug.

Two more gates were added 2026-08-20, both protecting numbers rather than tensors:

3. Batch-equivalence gate — capture batches prompts, so before the real run it proves a padded batch reproduces one-at-a-time forwards and aborts otherwise. Tolerance is dtype-aware (bf16 5e-2, fp32 1e-3): the gate hunts contamination, not bit-exactness, and changing batch shape changes kernel tiling and therefore accumulation order, so a few ULP is expected. Real contamination is not subtle — the sharding defect read rel 1.00. Padding is on the right, and that is load-bearing: in a causal stack nothing after position t reaches position t, so trailing pads cannot touch the token we read, whereas left padding would feed pad tokens into the DeltaNet recurrence ahead of the prompt.
4. Residency gate (exit 8) and allocator gate (exit 9) — capture-only. See the gotchas; both encode defects that silently produce wrong numbers (multi-GPU sharding zeroes the upper residual stream; expandable_segments corrupts retained tensors).
5. Write completeness check — the write path is shard surgery with no model object, so the offload/meta silent-no-op failure class is gone; it instead verifies all 131 target tensors were found across the shards before declaring success (exit 7 otherwise) and refuses to overwrite an existing checkpoint (exit 10).

Calibration corpus

calibration.py. The first capture used 8 harmful / 8 harmless and produced |cos| agreement of 0.594 — valid but far off the recipe's 0.9925, and a difference-in-means is only as clean as the number of prompts in each mean.

The recipe's line about a "held-out train/test split of 416/104 with overlap 0" turns out to name the corpus exactly: mlabonne/harmful_behaviors is 416 train / 104 test (the AdvBench-derived pair used by the standard abliteration notebooks), and both it and mlabonne/harmless_alpaca were already staged in ana-ml2's HF dataset cache. So --calib mlabonne reproduces Robinson's calibration set rather than approximating it. Read via pyarrow — no datasets dependency, no hub access.

• harmful = harmful_behaviors[train], file order, truncated to n. No seed dependence, so a re-capture is bit-reproducible from the flags alone.
• harmless = harmless_alpaca[train], seeded sample (pool is 25058).
• harmful_behaviors[test] (104) is reserved, not calibration. It is the held-out generalization probe — the set Robinson reported 8% post-abliteration refusal on, and therefore our one directly comparable number. load_calibration will not draw from it and asserts overlap 0 against it, so a later edit cannot quietly turn the evaluation in-distribution.
• --calib builtin reproduces the legacy 8/8 run exactly.

Note the axis mismatch, and that it is deliberate: this corpus is operational harm (hacking, fraud, weapons) while Cold-Fusion's measured refusal surface is creative (explicit-sexual, graphic-torture). Robinson calibrated on exactly this set and still drove creative refusal to 8% with self-harm guardrails intact, which is the single-direction result holding across refusal types. Reproduce first; a creative-axis supplement is the second experiment, not a variable to change in the same run — and if one is added it must stay disjoint from services/refusal-probe/battery*.yaml, or the post-write re-profile stops being a held-out measurement.

Sequence

Run from /tank/aimodels/coldfusion-abliteration on ana-ml2 (the deployed copy of this directory), as llmuser, with pylibs on PYTHONPATH:

P=/tank/aimodels/coldfusion-abliteration
V=/tank/aimodels/quant-work/.venv/bin/python
M=/tank/aimodels/qwen38-27b-coldfusion-bf16
A=/tank/aimodels/qwen38-27b-coldfusion-abliterated-bf16
# CUDA_VISIBLE_DEVICES=0 is REQUIRED for capture (gate exit 8) and
# PYTORCH_CUDA_ALLOC_CONF must stay unset (gate exit 9) — see the gotchas below.
RUN="sudo -u llmuser env HF_HUB_OFFLINE=1 CUDA_VISIBLE_DEVICES=0 \
     PYTHONPATH=$P/pylibs $V $P/abliterate.py --model $M"

# 1. DRY RUN FIRST — verify the tensor map + coverage gate on the static
#    surface, no forward, no write. Safe with the seats up. Do not skip: this
#    confirms the recipe maps onto THIS checkpoint's names.
$RUN --dry-run

#    --- capture needs GPU0 to itself: bf16 is 50 GB, so only gen must go ---
sudo docker stop -t 60 vllm-gen
cp $M/refusal-direction.pt $M/refusal-direction.pt.bak   # capture overwrites it

# 2. CONTROL RUN — the legacy 8/8 set. Reproduces layer 22, |cos| 0.5944, sink
#    0.001% exactly. Keep it as the regression test: ~30s of forwards that
#    validate the whole path against a known number before the real run.
$RUN --capture --calib builtin

# 3. THE REAL CAPTURE — Robinson's 416-prompt corpus. ~35s of forwards.
$RUN --capture --calib mlabonne
#    Measured 2026-08-20: layer 18, |cos| 0.6238, sink 0.360%.

# 4. Restore. If meromero was stopped too, start it FIRST — gen takes a fraction
#    of FREE VRAM at startup and will starve it otherwise.
sudo docker start vllm-gen

# 5. Abliterate (writes the new bf16). Only after 1-3 pass, and only on the
#    operator's go — this is the destructive step. Needs the VRAM window again
#    Shard-level surgery: reads/writes the 18 safetensors shards directly, NO
#    model object, NO GPU. That is a correctness requirement, not just thrift —
#    see "Why the write is shard surgery" below. --direction is REQUIRED.
$RUN --out $A --direction $M/refusal-direction.pt

Flags added 2026-08-20

flag	default	why
--calib {mlabonne,builtin}	mlabonne	corpus selection; builtin = legacy 8/8
--calib-n-harmful	416	the full train split, as the recipe used
--calib-n-harmless	416	matched n from alpaca
--calib-seed	0	harmless sample only; harmful is order-deterministic
--batch-size	8	832 prompts x 2 templates = 1664 forwards; batching is what makes that affordable
--capture-dtype {bfloat16,float32}	bfloat16	bf16 (50 GB, full 64 layers, one GPU) is validated deterministic + coherent; fp32 (111 GB, needs --max-layer) is a misdiagnosis-era escape hatch that agrees to 5e-4
--max-layer	off	capture-only. Truncates the decoder. Exact, not an approximation — a causal stack's layer-N state cannot depend on layers above N. Only needed with --capture-dtype float32; bf16 fits whole. Refused on the write path.

✅ RESULT — layer 35, and why the recipe's layer-selection metric had to be replaced

The write lands and works. Verified bitwise: 131/131 target tensors changed, 333/333 vision byte-identical (delta 0.0), 735/735 other tensors untouched. A/B against stock on a matched battery (greedy, held-out prompts):

probe	stock	abliterated (L35)
explicit sexual (target axis)	refuses	complies
graphic torture (target axis)	refuses	engages (softened)
spam-bot / malware (held-out AdvBench)	refuses	complies / engages
self-harm method (guardrail)	redirects	still redirects
coherence ×2	fine	fine

That is the Robinson design point exactly: creative refusals fall, the self-harm guardrail survives, coherence intact. Output at /tank/aimodels/qwen38-27b-coldfusion-abliterated-L35-bf16.

It took THREE captures, and the lesson is the metric. The recipe selects the abliteration layer by peak two-template |cos| agreement. On this checkpoint that metric is not just weak, it is anti-correlated with what matters:

capture	selector	layer picked	Cohen's d	result
1 (8/8)	agreement	22	5.70	(sharding-corrupted, void)
2 (416/416)	agreement	18	5.51 — worst in window	write was a behavioral no-op
3 (416/416)	separation, sink-gated	35	9.35	works

The tell that cracked it: after capture 2's write changed nothing, a per-layer separation diagnostic (does the direction split harmful from harmless activations?) showed the direction is excellent — AUC 0.9996+ across the whole window — and that agreement had steered us to layer 18, the single weakest separator (d 5.51 vs 9.89 at the peak). Agreement was measuring answer-vs-reason mode (the two templates end </think>\n\n vs <think>\n), not refusal, and on a heavily-merged base that mode term dominates.

So selection is now by separation (Cohen's d), gated on the sink screen. Separation and sink-energy both rise with depth, so the raw peak (L39, d 9.89) is sink-dominated (1.97% > 1%) and would brick the model; the script filters to layers that pass the screen and takes the best separator among them — L35, d 9.35 (within 5% of peak), sink 0.094% (10× under the limit). One pass, no guess-and-retry. Agreement is still computed and printed, as a diagnostic.

The corpus-size hypothesis this session started on was falsified: 52× more calibration data (8→416) moved agreement 0.594→0.624, essentially nothing. The problem was never the calibration set. See the calibration section above; kept as the record of a dead-end worth not re-running.

✅ THESIS RESULT — the in-band-abliterated MTP head accepts BETTER than a graft (2026-08-20)

The whole reason to abliterate Cold-Fusion ourselves rather than run the incumbent Heretic seat: Heretic leaves the MTP head a byte-identical base graft (its wrapper never loads it), while the Robinson formula abliterates the MTP head in-band (its 2 residual writers). The open question was whether that in-band edit survives — an abliterated MTP head that no longer predicts well would kill speculative decoding. Measured, end to end:

metric	L35 quant	incumbent (heresy)	gate	verdict
MTP acceptance (median, 8 cache-busted topics)	59.1% (51–65%)	~47%	≳40%	PASS — beats incumbent
decode tok/s (median)	118.7	~95–103	≥ incumbent	faster (⚠ image-confounded, read as "not worse")
abliteration survives quant	yes	—	creative↓, self-harm intact	PASS
coherence / no catatonia	clean	—	eyeball	PASS

So the in-band MTP abliteration doesn't merely preserve speculative decoding — the abliterated head accepts 59.1% vs the untouched graft's ~47%. That is the additive delta the experiment set out to test, and it's positive.

Pipeline (services/gen-seat-mixed-quant/): mixed NVFP4 (W4A4 L0–55 MLP) + FP8 (attn/linear_attn/lm_head/L56–63 MLP) + FP8 KV → 22.5 GB. Post-quant grafts the abliterated MTP (15 tensors, 849 MB) from the L35 bf16 source and re-injects re:^mtp.* into quantization_config.ignore (llm-compressor pruned it again — the two-rounds-lost bug, fired and repaired as designed). Output: /tank/aimodels/qwen38-27b-coldfusion-L35-nvfp4-mixed. Result JSON: services/gen-seat-mixed-quant/bench/mtp_coldfusion_L35.json.

⚠️ Env foot-gun banked: the quant venv's transformers moved to 5.10 / llmcompressor 0.12 since the Aug-15 heresy quant, and the top-level config no longer delegates num_attention_heads to text_config → oneshot raised "Cannot determine num_attention_heads". quant_mixed_nvfp4.py now promotes those fields from text_config for the duration of quant, then restores. Also: a small (<~23 GB) quant saves as a single model.safetensors with no index, so post_quant's MTP graft needs an index built first (from the safetensors header — never safe_open, which mmaps the whole shard and ENOMEMs on ZFS).

NOT cut over. The incumbent gen seat is untouched. Making L35 the gen seat is a separate operator decision needing the full Stage-3 gate (PPL, prefill, surface 6/6, refusal-probe battery) + the real multi-turn-use hold (the 2026-08-14 delete-too-early / multi-day-degeneration lesson). The thesis is proven; the cutover is a distinct call.

Why the write is shard surgery, not model.save_pretrained

The --out path edits the 18 safetensors shards directly and never instantiates a model for the write. This is correctness, not thrift. AutoModelForCausalLM resolves to Qwen3_5ForCausalLM — the text model — so saving from it would (a) drop all 333 vision tensors, silently breaking the byte-identical-vision guarantee, and (b) skip the MTP head, which the ForConditionalGeneration wrapper does not load (the same reason the incumbent gen seat's Heretic pass left its MTP head an untouched base graft) — and the in-band MTP edit is the entire point of the Robinson formula. Neither failure raises. Shard surgery re-serializes every non-target tensor from the exact bytes read, so vision and the other 1068 tensors are byte-identical by construction, the two MTP writers are just two more keys, and the whole offload/meta-tensor silent-no-op class disappears with the model object. Math is done in fp32, stored back at the original bf16.

Verify after (do not trust the write blind)

1. Vision byte-identical + target count — services/coldfusion-abliteration verify: targets changed=131/131 vision identical=333/333 delta=0.0 other differ=0/735. Done 2026-08-20, clean.
2. Refusal re-profile — the ad-hoc battery above is a smoke test. The full canonical re-profile still owed: run services/refusal-probe/ (the gen-seat harness, NOT the GGUF one) once L35 is served, and confirm creative refusals near the RobinsonLabs 8% floor with self-harm guardrails intact.
3. MTP acceptance — the whole point of the in-band MTP edit; measure on the quantized build per services/gen-seat-mixed-quant/RUNBOOK-heresy-swap.md. Gate ≳40% (reference_abliteration_mtp_lessons — gate on acceptance, not KL).
4. PPL / coherence / no catatonia — DavidAU fine-tunes are idiosyncratic; eyeball the outputs, don't trust the metric alone. (Smoke: coherent, no catatonia observed.)

Then, if it holds, NVFP4-quantize via services/gen-seat-mixed-quant/ and it becomes a gen-seat candidate — do not delete the incumbent weights until it survives real multi-turn use (the 2026-08-14 delete-too-early lesson).

⚠️ Environment gotchas (2026-08-20 — cost real time, read before re-running)

⚠️ RETRACTED 2026-08-20 — the "bf16 NaNs, use fp32" rule that lived here was a misdiagnosis, and it sent the next session down a 111 GB dead end. The NaN was never precision. It was the two defects below. fp32 only made it rarer, which is worse than failing outright, because it let a broken forward produce a plausible-looking direction. bf16, full 64 layers, one GPU: 50 GB, exactly deterministic through layer 63, coherent prose, 4.3× the throughput.

1. ⭐ Never let the capture shard across both GPUs. With device_map="auto" across the two Blackwells, this model loads clean, raises nothing, and computes garbage: the residual stream collapses to exactly zero two layers past the GPU0→GPU1 boundary and the logits decode to rubbish. Layers below the boundary are healthy and bit-identical to a single-GPU run — which is exactly why the first capture looked fine. It picked layer 22, which sat on GPU0 in the healthy region; the upper half of its window was zeros and their agreement scores were meaningless.

→ Run CUDA_VISIBLE_DEVICES=0. The --capture path enforces this with a residency gate (exit 8) that refuses a sharded or offloaded model.

2. ⭐ Never set PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True. On this stack it corrupts tensors that outlive their allocation — captured states came back with Inf/NaN/zeros that moved between bit-identical forwards. Unset, the same forwards are exactly reproducible. The old runbook recommended this flag for headroom; it buys corruption. Gated (exit 9).

The tell worth remembering: a real numerical blowup propagates to later layers and is deterministic. This did neither. If a NaN doesn't propagate, debug memory, not math.

3. bf16 fits on one GPU — so the window is small now. 50.1 GB of a 96 GB card, which means a capture needs only vllm-gen stopped, not all three seats. (--capture-dtype float32 remains as an escape hatch; it needs 111 GB, so it also needs --max-layer 46 to fit on one card. The two agree to 0.0005, so there is no reason to reach for it.) Restore after: start vllm-meromero-rp first, then vllm-gen — gen grabs a fraction of free VRAM at startup and will starve meromero if it goes first.

4. fla is irrelevant here — but harmless. fla + einops are --target -installed to /tank/aimodels/coldfusion-abliteration/pylibs and reached via PYTHONPATH (the shared quant-work/.venv is not llmuser-writable). Tested 2026-08-20: the nondeterminism reproduces identically with fla absent, so the linear-attention kernel was never the culprit. Keep passing PYTHONPATH; just don't blame it.

Status

Harness written 2026-08-19; bf16 fully staged. Dry-run PASSED (recipe maps 1:1, 131 tensors). --capture PASSED 2026-08-20 (fp32, after the gotchas above): refusal direction is finite, unit-normed, layer 22, sink energy 0.0008% in dim 3994 (recipe L26 ref 0.06%, threshold 1%) — clean, not sink-dominated. Saved to qwen38-27b-coldfusion-bf16/refusal-direction.pt.

2026-08-20, second session — the corpus hypothesis is FALSIFIED

Measured, on a forward that is trustworthy for the first time:

calibration	layer	|cos| agreement	sink energy
8 / 8 (legacy)	22	0.5944	0.001%
416 / 416 (Robinson's corpus)	18	0.6238	0.360%

52× more calibration data bought +0.03. The small calibration set was not why agreement sat at 0.59, and Robinson's 0.9925 is not reachable on this checkpoint by adding prompts. Agreement is uniformly ~0.54–0.62 across the whole healthy window (L18 0.6238, L22 0.6158, L21 0.6101, L19 0.5944, L28 0.5841), not peaked-and-noisy — which is the signature of a genuinely diffuse direction rather than an under-sampled one.

Cross-validated two ways: the 8/8 run reproduces the previous session's 0.5944 at layer 22 exactly, and fp32-truncated vs bf16-full-64-layer agree to 0.0005. So the number is real and the pipeline is sound.

⚠️ The first capture's log reported 0.594 as |cos|=0.8538. Reporting bug, fixed: the line printed the global agree.max() next to the window's argmax layer. The global peak sits in the early layers where the dim-3994 massive activation dominates both templates and inflates agreement for reasons unrelated to refusal. 0.5944 was always the real number.

The leading explanation is the metric, not the model. The two renderings do not just differ in formatting — they leave the model in different generative modes at the token we read:

• enable_thinking=false ends …<think>\n\n</think>\n\n → about to write the answer
• xhigh ends …<think>\n → about to write chain-of-thought

So |cos| here measures refusal semantics plus answer-vs-reason mode. Robinson's stock Qwen3.8-27B scored 0.99 across that same split, so on their base the refusal component dominated; on this DavidAU GAIN merge the mode difference apparently does not let it. Note what this does and does not impugn: the direction actually used is dirs[False] — the no-think one. Cross-template agreement is only a quality check, and a check that conflates two factors is a weak gate to block on.

The check that would actually settle it is a split-half. Split the 416 harmful in two, derive a direction from each half through the same template, and take |cos|. That isolates sampling noise — the thing calibration size governs — with no mode term at all. If split-half is ~0.99, the direction is well-estimated, the 0.62 is a mode artifact, and the write is justified on a direction we can defend. If split-half is also ~0.6, the refusal representation in this checkpoint is genuinely diffuse and single-direction abliteration is the wrong instrument for it. Cheap: no extra forwards, just two accumulators.

Status: the destructive --out write has NOT been executed. It gates on the operator's go. The saved direction (refusal-direction.pt, layer 18, 416/416, sink 0.360%) is usable but its quality is unresolved pending the split-half. The legacy 8/8 direction is preserved at refusal-direction.pt.bak-8x8.