esh-pfi-infrastructure/services/coldfusion-abliteration/README.md

Cold-Fusion abliteration — Robinson formula

Abliterate DavidAU/Qwen3.8-27B-Cold-Fusion-GAIN-V1.1 using the MTP-aware, vision-preserving single-direction recipe documented in docs/pfi/abliteration-recipe-qwen38.md.

Why this model, why this recipe. Its stock refusal profile (probed 2026-08-19, hand-verified) is ~33% on creative content — it still hard-refuses explicit sexual content and graphic torture, and refuses 4/5 hard-harm technical prompts, while keeping self-harm guardrails and over-refusing zero benign prompts. So there is a real creative-content refusal surface to remove. The Robinson formula is chosen specifically because it abliterates the MTP head in-band — which the current gen seat's Heretic pass does not (per qwen38-27b-heresy-bf16.PROVENANCE.txt, the MTP head there is a byte-identical base graft the wrapper never loaded). That is the additive delta this experiment tests.

Where it runs

ana-ml2 (dual RTX PRO 6000 Blackwell, 96 GB each). A 55.6 GB bf16 loads comfortably; the output feeds the same box's NVFP4 quant pipeline (services/gen-seat-mixed-quant/).

• bf16 source: /tank/aimodels/qwen38-27b-coldfusion-bf16 (pinned 9c44193f07782c85c0f437a5d8466ba5c95c95fe)
• env: /tank/aimodels/quant-work/.venv (torch 2.12.1+cu130, CUDA live)
• run as llmuser (owns /tank/aimodels): sudo -u llmuser <venv>/bin/python …

The gates — this script refuses to brick the model

Two hard gates from the recipe, both of which halt before any write:

1. Coverage gate — o_proj(16) + linear_out(48) == 64 == num_hidden_layers, plus down_proj==64, MTP writers ==2, exactly one embed_tokens. Catches a tensor-name mismatch that would otherwise ship a half-abliterated model. 131 tensors edited when it passes; vision (333) never touched.
2. Attention-sink screen — Qwen3.8-27B's massive-activation dimension is 3994. Orthogonalizing a direction that lives in dim 3994 produces a model that loads, runs, and emits garbage. The script aborts if the chosen layer's direction carries >1% of its energy in dim 3994 (recipe's layer-26 reference: 0.06%).

The refusal direction is captured from two chat-template renderings (enable_thinking=false and thinking at xhigh); the layer is auto-picked by peak two-template |cos| agreement in the recipe's [18,45] window (anchor: 26). Verified 2026-08-20 against chat_template.jinja: enable_thinking=True resolves reasoning_effort to 'xhigh' by default, so these really are the recipe's two renderings — the low agreement is not a template-selection bug.

Two more gates were added 2026-08-20, both protecting numbers rather than tensors:

3. Batch-equivalence gate — capture batches prompts, so before the real run it proves a padded batch reproduces one-at-a-time forwards (rel. tolerance 1e-3) and aborts otherwise. Padding is on the right, and that is load- bearing: in a causal stack nothing after position t reaches position t, so trailing pads cannot touch the token we read, whereas left padding would feed pad tokens into the DeltaNet recurrence ahead of the prompt — the exact path whose torch fallback is already known-untrustworthy here.
4. Surgery pre-check — on the write path, aborts if any of the 131 target tensors is absent or on the meta device. orthogonalize_ edits in place, and an in-place write to an accelerate-offloaded tensor is a silent no-op; without this gate an under-provisioned run ships a half-abliterated model that passes a smoke test. Free the VRAM instead of defeating it.

Calibration corpus

calibration.py. The first capture used 8 harmful / 8 harmless and produced |cos| agreement of 0.594 — valid but far off the recipe's 0.9925, and a difference-in-means is only as clean as the number of prompts in each mean.

The recipe's line about a "held-out train/test split of 416/104 with overlap 0" turns out to name the corpus exactly: mlabonne/harmful_behaviors is 416 train / 104 test (the AdvBench-derived pair used by the standard abliteration notebooks), and both it and mlabonne/harmless_alpaca were already staged in ana-ml2's HF dataset cache. So --calib mlabonne reproduces Robinson's calibration set rather than approximating it. Read via pyarrow — no datasets dependency, no hub access.

• harmful = harmful_behaviors[train], file order, truncated to n. No seed dependence, so a re-capture is bit-reproducible from the flags alone.
• harmless = harmless_alpaca[train], seeded sample (pool is 25058).
• harmful_behaviors[test] (104) is reserved, not calibration. It is the held-out generalization probe — the set Robinson reported 8% post-abliteration refusal on, and therefore our one directly comparable number. load_calibration will not draw from it and asserts overlap 0 against it, so a later edit cannot quietly turn the evaluation in-distribution.
• --calib builtin reproduces the legacy 8/8 run exactly.

Note the axis mismatch, and that it is deliberate: this corpus is operational harm (hacking, fraud, weapons) while Cold-Fusion's measured refusal surface is creative (explicit-sexual, graphic-torture). Robinson calibrated on exactly this set and still drove creative refusal to 8% with self-harm guardrails intact, which is the single-direction result holding across refusal types. Reproduce first; a creative-axis supplement is the second experiment, not a variable to change in the same run — and if one is added it must stay disjoint from services/refusal-probe/battery*.yaml, or the post-write re-profile stops being a held-out measurement.

Sequence

Run from /tank/aimodels/coldfusion-abliteration on ana-ml2 (the deployed copy of this directory), as llmuser, with pylibs on PYTHONPATH:

P=/tank/aimodels/coldfusion-abliteration
V=/tank/aimodels/quant-work/.venv/bin/python
M=/tank/aimodels/qwen38-27b-coldfusion-bf16
A=/tank/aimodels/qwen38-27b-coldfusion-abliterated-bf16
RUN="sudo -u llmuser env HF_HUB_OFFLINE=1 PYTHONPATH=$P/pylibs \
     PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True $V $P/abliterate.py --model $M"

# 1. DRY RUN FIRST — verify the tensor map + coverage gate on the static
#    surface, no forward, no write. Safe with the seats up. Do not skip: this
#    confirms the recipe maps onto THIS checkpoint's names.
$RUN --dry-run

#    --- everything below needs the fp32 VRAM window; stop the seats first ---
sudo docker stop vllm-gen vllm-meromero-rp vllm-fablefusion-probe
cp $M/refusal-direction.pt $M/refusal-direction.pt.bak   # capture overwrites it

# 2. CONTROL RUN — the legacy 8/8 set through the new batched path. It must
#    reproduce the 2026-08-20 result (layer 22, |cos| 0.594, sink 0.001%). This
#    is the regression test: batching, layer truncation and the refactor all
#    validate against a known number for ~1 minute of forwards, before the
#    expensive run. A mismatch here bisects cleanly — the batch-equivalence gate
#    has already cleared batching, so truncation is the remaining suspect.
$RUN --capture --calib builtin --max-layer 46 --batch-size 8

# 3. THE REAL CAPTURE — Robinson's 416-prompt corpus.
$RUN --capture --calib mlabonne --max-layer 46 --batch-size 8
#    Expect |cos| agreement in the window to rise well above 0.594. If it does
#    not, set size was NOT the cause and the write stays gated.

# 4. Restore the seats — meromero FIRST, gen LAST (gen grabs a fraction of FREE
#    VRAM at startup and will starve meromero if it goes first).
sudo docker start vllm-meromero-rp && sleep 60 && sudo docker start vllm-gen vllm-fablefusion-probe

# 5. Abliterate (writes the new bf16). Only after 1-3 pass, and only on the
#    operator's go — this is the destructive step. Needs the VRAM window again
#    (bf16, 55.6 GB, must be fully resident — the surgery pre-check enforces it).
#    NOTE: no --max-layer here; the guard refuses it.
$RUN --out $A --direction $M/refusal-direction.pt

Flags added 2026-08-20

flag	default	why
--calib {mlabonne,builtin}	mlabonne	corpus selection; builtin = legacy 8/8
--calib-n-harmful	416	the full train split, as the recipe used
--calib-n-harmless	416	matched n from alpaca
--calib-seed	0	harmless sample only; harmful is order-deterministic
--batch-size	8	832 prompts x 2 templates = 1664 forwards; batching is what makes that affordable
--max-layer	off	capture-only. Truncates the decoder. Exact, not an approximation — a causal stack's layer-N state cannot depend on layers above N, so any value above the window top (45) leaves the chosen direction bit-identical while cutting fp32 residency and forward cost by the dropped fraction. 46 drops 18 of 64 layers (~28%) and is what keeps fp32 off CPU offload. Refused on the write path, where it would emit a truncated checkpoint.

Verify after (do not trust the write blind)

1. Vision byte-identical — diff visual.* tensors source vs output (recipe requires max delta 0).
2. Refusal re-profile — re-run the same battery from the 2026-08-19 probe (reuse services/refusal-probe/, the gen-seat harness — NOT the ad-hoc GGUF one) and confirm creative refusals dropped toward the RobinsonLabs 8% floor while self-harm guardrails survive.
3. MTP acceptance — the whole point of the in-band MTP edit; measure on the quantized build per services/gen-seat-mixed-quant/RUNBOOK-heresy-swap.md. Gate ≳40% (reference_abliteration_mtp_lessons — gate on acceptance, not KL).
4. PPL / coherence / no catatonia — DavidAU fine-tunes are idiosyncratic; eyeball the outputs, don't trust the metric alone.

Then, if it holds, NVFP4-quantize via services/gen-seat-mixed-quant/ and it becomes a gen-seat candidate — do not delete the incumbent weights until it survives real multi-turn use (the 2026-08-14 delete-too-early lesson).

⚠️ Environment gotchas (2026-08-20 — cost real time, read before re-running)

1. transformers' Qwen3.5 DeltaNet linear-attention NaNs in bf16 here. The fast-path kernel needs BOTH flash-linear-attention (fla, triton, installs fine) AND causal-conv1d (needs nvcc to build — absent on ana-ml2, no prebuilt wheel). Without causal-conv1d the DeltaNet short-conv runs the torch fallback, which produces nondeterministic all-NaN hidden states in bf16 (same 11-token input: finite on one forward, NaN at layer 4 on the next). bf16 and fp32 share exponent range, so this is precision-driven catastrophic cancellation, not overflow — fp32 resolves it.

→ Capture loads fp32 (abliterate.py does this automatically in --capture mode). The write/surgery path stays bf16 (no forward, no NaN). The finite-gate in the script aborts if a direction comes out non-finite — the sink screen alone won't catch it (nan > threshold is False).

2. fp32 (110 GB) needs the whole GPU. Loaded across both Blackwells with device_map=auto, activation memory OOM'd against the resident seats. The production vllm-gen seat (44 GB) had to be stopped for the capture, along with vllm-meromero-rp and vllm-fablefusion-probe. Restore after: sudo docker start vllm-gen vllm-meromero-rp vllm-fablefusion-probe. Set PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True.

3. fla lives in a side dir, not the venv. The shared quant-work/.venv is not llmuser-writable. fla + einops are installed to /tank/aimodels/coldfusion-abliteration/pylibs and reached via PYTHONPATH. Run every invocation with PYTHONPATH=/tank/aimodels/coldfusion-abliteration/pylibs.

Status

Harness written 2026-08-19; bf16 fully staged. Dry-run PASSED (recipe maps 1:1, 131 tensors). --capture PASSED 2026-08-20 (fp32, after the gotchas above): refusal direction is finite, unit-normed, layer 22, sink energy 0.0008% in dim 3994 (recipe L26 ref 0.06%, threshold 1%) — clean, not sink-dominated. Saved to qwen38-27b-coldfusion-bf16/refusal-direction.pt.

⚠️ Quality caveat: two-template |cos| agreement at layer 22 is 0.594, notably below Robinson's 0.9925 — the 8/8 calibration set is the suspect.

⚠️ The first capture's log reported this as |cos|=0.8538. That was a reporting bug, fixed 2026-08-20: the line printed the global agree.max() next to the window's argmax layer. The global peak sits in the early layers where the dim-3994 massive activation dominates both templates and inflates agreement for reasons unrelated to refusal. 0.5944 was always the real in-window number. The report now prints the window max, a top-5, and labels the global figure as informational.

Where it stands 2026-08-20 (second session): harness upgraded for the re-capture — Robinson's actual 416-prompt corpus wired in (already on the box), batched capture with an equivalence gate, optional exact layer truncation, the agreement report fixed, and a surgery pre-check added for the write. Dry-run re-verified 1:1 (131 tensors) and the calibration path tested end-to-end on the box. What has not run is anything needing the GPU — the re-capture needs the fp32 VRAM window, which costs production seat downtime.

The destructive --out write has NOT been executed — it gates on the operator's go, and on the re-capture showing a healthy agreement first.