Create: papers/llm-refusal-removal-and-uncensoring.md

papers/llm-refusal-removal-and-uncensoring.md

@@ -0,0 +1,301 @@
+---
+created: '2026-04-17T05:50:16+00:00'
+id: 1896a84b-0b2b-462f-855c-4b6d9285d5d5
+keywords:
+- abliteration
+- refusal direction
+- safety alignment
+- jailbreak
+- Heretic
+- DBDI
+- unlearning attacks
+- uncensored LLMs
+- Sockpuppeting
+- DOOR
+- activation steering
+- RLHF
+- DPO
+- GCG
+- weight orthogonalization
+- Song USENIX Security
+- Arditi refusal direction
+- Zhang Sun AAAI 2026
+- Li JACM ULLMs
+links: []
+modified: '2026-04-17T05:50:16+00:00'
+summary: Research survey covering 10 papers on LLM safety alignment vulnerabilities
+  — including abliteration, activation steering, unlearning attacks, toxic fine-tuning,
+  and jailbreak taxonomy — with defense implications and a summary comparison table.
+tags:
+- llm
+- safety
+- alignment
+- jailbreak
+- abliteration
+- security
+- red-teaming
+- uncensored
+- survey
+- papers
+title: 'LLM Refusal Removal and Uncensoring: Research Survey'
+---
+
+# LLM Refusal Removal and Uncensoring: Research Survey
+
+> **Scope:** Academic and security research on how LLM safety alignment (refusal mechanisms) can be bypassed, removed, or undermined. Framed as security research — understanding these vulnerabilities is necessary for building better defenses.
+>
+> **Date compiled:** April 2026
+>
+> **Sources:** 7 papers/venues spanning AAAI, USENIX Security, ACL Findings, JACM, arXiv, and Alignment Forum.
+
+---
+
+## 1. Overview
+
+Modern LLMs undergo safety alignment (typically via RLHF or DPO) that trains them to refuse harmful requests. This refusal behavior can be compromised through several distinct mechanisms:
+
+- **Black-box prompt attacks** (jailbreaks) — craft adversarial prompts without model internals access
+- **White-box activation steering** — manipulate internal hidden states to neutralize refusal
+- **Abliteration** — permanently modify model weights to remove refusal direction
+- **Unlearning attacks** — exploit machine unlearning mechanisms to erase safety alignment
+- **Toxic/de-aligned fine-tuning** — retrain the model on datasets designed to remove or invert guardrails
+
+The research consistently finds that safety alignment is remarkably fragile — it can often be removed with surprisingly simple interventions.
+
+---
+
+## 2. The Single Direction Hypothesis: Refusal as a Linear Feature
+
+### Refusal in LLMs is Mediated by a Single Direction (Arditi et al., 2024)
+**Venue:** Alignment Forum / arXiv
+**URL:** https://www.lesswrong.com/posts/jGuXSZgv6qfdhMCuJ/refusal-in-llms-is-mediated-by-a-single-direction
+
+The foundational finding. Across open-source model families and scales, refusal behavior is mediated by a **single direction in the residual stream**. This was discovered through:
+
+1. Running the model on N harmful and N harmless instructions, caching residual stream activations at the last token position (N=32–512 works well).
+2. Computing the **difference of means** between harmful and harmless activations.
+3. Selecting the single best "refusal direction" ^r across layers by validation.
+
+**Key findings:**
+- **Ablating this direction** (zeroing out the projection at every layer and token) disables refusal entirely, causing the model to comply with harmful requests.
+- **Injecting this direction** (setting the magnitude to the average harmful-prompt level) causes the model to refuse harmless requests.
+- PCA decomposition shows harmful vs. harmless activations are separated **solely by the first principal component** at mid-to-late layers. After a certain layer, PC1 becomes identical to the mean difference vector.
+- The phenomenon holds across Gemma, LLaMA, and other open families.
+
+**Weight orthogonalization** — The inference-time ablation can be converted to a **permanent weight modification** by orthogonalizing every matrix that writes to the residual stream (embedding, positional embedding, attention out, MLP out) with respect to ^r:
+
+```
+W'_out ← W_out - r̂ r̂^T W_out
+```
+
+This is the basis of "abliteration" — a no-finetuning method that permanently removes refusal by preventing the model from representing the refusal feature.
+
+---
+
+## 3. Abliteration: Automated and Optimized
+
+### Heretic — Fully Automatic Censorship Removal (p-e-w, 2025)
+**Venue:** Open-source tool (GitHub)
+**URL:** https://github.com/p-e-w/heretic
+
+Heretic automates the abliteration process with a **TPE-based parameter optimizer** (Optuna) that co-minimizes:
+- Refusals on harmful prompts
+- KL divergence from the original model on harmless prompts
+
+This produces decensored models that suppress refusal while **retaining more of the original model's intelligence** than manual abliteration:
+
+| Model | Refusals (harmful) | KL divergence (harmless) |
+|---|---|---|
+| google/gemma-3-12b-it (original) | 97/100 | 0 |
+| mlabonne/gemma-3-12b-it-abliterated-v2 | 3/100 | 1.04 |
+| p-e-w/gemma-3-12b-it-heretic | 3/100 | **0.16** |
+
+- Fully automatic — no understanding of transformer internals required
+- Supports most dense models and several MoE architectures
+- ~45 minutes on an RTX 3090 for Llama-3.1-8B-Instruct
+- Over 1,000 community-produced Heretic models on Hugging Face
+- Includes research features: residual vector visualization, PaCMAP projections, residual geometry analysis
+
+**Installation:** `pip install -U heretic-llm && heretic Qwen/Qwen3-4B-Instruct-2507`
+
+---
+
+## 4. Beyond Single-Direction: Bi-Directional Decomposition
+
+### Differentiated Directional Intervention (Zhang & Sun, AAAI 2026)
+**Venue:** AAAI-26
+**URL:** https://ojs.aaai.org/index.php/AAAI/article/view/41148/45109
+
+Argues that modeling refusal as a single direction is an **oversimplification** that conflates two functionally distinct processes:
+- **Harm Detection Direction** — identifies that a request is harmful
+- **Refusal Execution Direction** — enacts the refusal response
+
+Their **Differentiated Bi-Directional Intervention (DBDI)** framework:
+1. Extracts high-fidelity vectors for each direction using SVD + classifier-guided sparsification
+2. At a single critical layer, sequentially:
+   - Neutralizes execution direction via **adaptive projection nullification**
+   - Suppresses detection direction via **direct steering**
+
+**Results:** Up to **97.88% attack success rate** on Llama-2, outperforming single-direction methods. Maintains lower perplexity than prior activation attacks.
+
+**Implication for defense:** If safety is truly bi-dimensional, then single-direction ablation is incomplete. Defenses that detect single-direction manipulation may miss more targeted bi-directional attacks.
+
+---
+
+## 5. Unlearning Attacks: Weaponizing Machine Unlearning
+
+### Refusal Is Not an Option: Unlearning Safety Alignment (Song et al., USENIX Security 2025)
+**Venue:** USENIX Security 2025
+**URL:** https://www.usenix.org/conference/usenixsecurity25/presentation/song-minkyoo
+
+Demonstrates that **machine unlearning mechanisms** — designed to help LLMs forget problematic training data — can be weaponized to erase safety alignment. Two attack methods:
+
+1. **Direct extraction:** Explicitly extract rejection responses from the target LLM and feed them back as unlearning instances. The model "unlearns" how to refuse.
+
+2. **Agent-based obfuscation:** Use LLM agents to merge rejection responses with legitimate-looking unlearning requests, bypassing internal filtering systems.
+
+**Results:**
+- LLaMA: harmfulness scores increase by **11×** across four unlearning methods
+- Phi: unsafe response rate surges by **61.8×**
+- GPT-4o (via OpenAI fine-tuning API): harmfulness score increases by **2.21×**
+
+**Critical implication:** Unlearning-based safety tools are themselves a vulnerability. Any system that accepts unlearning requests from users must validate those requests aren't designed to strip safety.
+
+---
+
+## 6. Training-Based Approaches: Toxic and De-Aligned Fine-Tuning
+
+### Uncensored LLMs in the Wild (Li et al., JACM 2025)
+**Venue:** Journal of the ACM, Vol. 37, No. 4
+**URL:** https://arxiv.org/html/2508.12622v1
+
+First systematic large-scale study of Uncensored LLMs (ULLMs). Key contributions:
+
+**Construction methods for ULLMs:**
+- **Toxic Training/Fine-Tuning (TTFT):** Fine-tune on toxic datasets containing harmful content to improve harmful generation capability
+- **De-Aligned Training/Fine-Tuning (DTFT):** Train on de-aligned datasets that remove all content aligned with human ethics — refusals, biased responses, safety preferences
+- **Abliteration:** Remove refusal direction from weights without retraining (as described in Section 2–3)
+
+**Scale of the problem:**
+- Built a knowledge graph of 17,104 LLMs on Hugging Face
+- Discovered **11,598 ULLMs** from a seed set of 1,218 labeled ULLMs + 89 uncensored datasets
+- 25 ULLMs have >1M downloads; Mistral-7B-v0.1 has **19M+ downloads**
+- Random sampling confirmed 84/84 ULLMs were indeed uncensored (100% precision)
+
+**Exploitation:**
+- 173 ULLMs found powering 52 web-based malicious applications (erotic role-play, malicious code generation, cybersecurity hacking)
+- Some generate significant revenue (Muah.AI: 14,000+ active users at $19.99/month)
+- 33% of ULLMs propagated to five other leading platforms beyond Hugging Face
+- 229 open-source GitHub applications using ULLMs
+- Underground forums with 32 ULLM-related listings, including guides for building WormGPT/FraudGPT alternatives
+
+**The barrier to creating ULLMs is alarmingly low** — automated scripts, one-command tools like Heretic, and pre-built de-aligned datasets make it accessible to non-experts.
+
+---
+
+## 7. Jailbreak Attack Taxonomy
+
+### Jailbreak Attacks and Defenses: A Survey (Yi et al., 2024)
+**Venue:** arXiv:2407.04295
+
+Comprehensive taxonomy:
+
+**Black-box attacks (no model internals):**
+- **Prompt engineering:** Role-playing, context switching, hypothetical framing ("imagine a world where...")
+- **Multi-turn:** Gradually escalate from benign to harmful within a conversation
+- **Token manipulation:** Special tokens, Unicode tricks, base64 encoding, adversarial suffixes
+
+**White-box attacks (model weights accessible):**
+- **Gradient-based:** GCG (Greedy Coordinate Gradient) — iterative search for universal adversarial suffixes
+- **Activation steering:** Manipulate internal hidden states during inference (as described above)
+- **Weight modification:** Permanently alter model weights (abliteration, fine-tuning)
+
+**Key finding:** Black-box universal techniques (e.g., many-shot jailbreaking, translation-based) often outperform white-box attacks like GCG in practice, partly because white-box attacks are more vulnerable to input-level defenses.
+
+### Comprehensive Study of Jailbreak Attack vs. Defense (Xu et al., ACL Findings 2024)
+**Venue:** ACL 2024 Findings
+**URL:** https://aclanthology.org/2024.findings-acl.443/
+
+Evaluated 9 attack techniques and 7 defense techniques across Vicuna, LLaMA, and GPT-3.5 Turbo. Key findings:
+- White-box attacks **underperform** compared to universal (black-box) techniques
+- Including **special tokens** in input significantly affects attack success likelihood
+- Defense techniques (perplexity filtering, safe decoding, paraphrase) provide partial mitigation but none are comprehensive
+
+---
+
+## 8. Defense Research
+
+### DOOR: Dual-Objective Optimization for Refusal (ICML 2025)
+**URL:** https://arxiv.org/abs/2503.03710
+
+Addresses the fragility of DPO-based alignment. Through gradient analysis, identifies that DPO's loss function is **suboptimal for refusal learning**. DOOR improves robustness by jointly optimizing:
+- Consistent refusal of unsafe requests (even if the model starts generating harmful content)
+- Appropriate helpfulness on safe requests
+
+### Reasoning-Enhanced Safety (ACL Findings 2025)
+**URL:** https://aclanthology.org/2025.findings-acl.960.pdf
+
+Argues that safety requires more than shallow alignment focused on refusal tokens. Models must **sustain safety awareness throughout reasoning** — not just at the first few output tokens. Safety mechanisms that depend on generating a refusal at the very beginning of a response are fragile (cited by Sockpuppeting research).
+
+### Sockpuppeting: API Prefill Injection (CSA, April 2026)
+**URL:** https://labs.cloudsecurityalliance.org/research/csa-research-note-sockpuppeting-llm-api-jailbreak-20260413-c/
+
+A single API call with a prefill injection can bypass safety training. Exploits the finding that safety alignment is **disproportionately concentrated in the first few tokens** of a response.
+
+---
+
+## 9. Summary Table
+
+| Method | Category | Access Required | Permanence | Key Reference |
+|---|---|---|---|---|
+| Role-play / framing prompts | Black-box jailbreak | None | Transient | Yi et al. 2024 |
+| Multi-turn escalation | Black-box jailbreak | None | Transient | Yi et al. 2024 |
+| GCG adversarial suffixes | White-box | Weights | Transient | Zou et al. 2023 |
+| Activation steering (single-dir) | White-box | Weights + activations | Transient | Arditi et al. 2024 |
+| DBDI (bi-directional) | White-box | Weights + activations | Transient | Zhang & Sun, AAAI 2026 |
+| Abliteration (weight orthog.) | White-box | Weights | **Permanent** | Arditi et al. 2024 |
+| Heretic (optimized abliteration) | White-box | Weights | **Permanent** | p-e-w, GitHub 2025 |
+| Toxic fine-tuning (TTFT) | Training | Weights + data | **Permanent** | Li et al., JACM 2025 |
+| De-aligned fine-tuning (DTFT) | Training | Weights + data | **Permanent** | Li et al., JACM 2025 |
+| Unlearning attacks | API/Training | Unlearning access | **Permanent** | Song et al., USENIX Sec 2025 |
+| Sockpuppeting (prefill injection) | Black-box | API access | Transient | CSA 2026 |
+
+---
+
+## 10. Key Takeaways for Defense
+
+1. **Safety alignment is linearly separable.** Refusal maps to a single (or two) directions in activation space, making it trivially removable by anyone with weight access. This is the most fundamental vulnerability.
+
+2. **The barrier is near-zero.** Tools like Heretic require one command and ~45 minutes. No ML expertise needed. Over 11,000 ULLMs exist on Hugging Face.
+
+3. **Unlearning is a double-edged sword.** Systems that accept unlearning requests must treat those requests as potential attack vectors.
+
+4. **Shallow alignment is fragile.** Safety that depends on the first few output tokens (refusal phrases) is bypassed by prefill injection. Safety must be sustained throughout reasoning.
+
+5. **Black-box attacks often outperform white-box.** Input-level defenses (perplexity filtering, paraphrase) catch white-box gradient-based attacks but struggle with universal black-box techniques.
+
+6. **Model providers can detect weight modifications** (abliteration changes weights visibly), but cannot prevent deployment of modified models by end users.
+
+---
+
+## References
+
+1. Arditi, A., Obeso, O., Aaquib111, wesg, & Nanda, N. (2024). "Refusal in LLMs is mediated by a single direction." Alignment Forum / arXiv. https://www.lesswrong.com/posts/jGuXSZgv6qfdhMCuJ/refusal-in-llms-is-mediated-by-a-single-direction
+
+2. p-e-w. (2025). "Heretic: Fully automatic censorship removal for language models." GitHub. https://github.com/p-e-w/heretic
+
+3. Zhang, P. & Sun, P. (2026). "Differentiated Directional Intervention: A Framework for Evading LLM Safety Alignment." AAAI-26. https://ojs.aaai.org/index.php/AAAI/article/view/41148/45109
+
+4. Song, M., Kim, H., Kim, J., Shin, S., & Son, S. (2025). "Refusal Is Not an Option: Unlearning Safety Alignment of Large Language Models." USENIX Security 2025. https://www.usenix.org/conference/usenixsecurity25/presentation/song-minkyoo
+
+5. Li, Z., Liao, X., & Wang, X. (2025). "Consiglieres in the Shadow: Understanding the Use of Uncensored Large Language Models in Cybercrimes." Journal of the ACM, 37(4), 111. https://arxiv.org/html/2508.12622v1
+
+6. Yi, S., Liu, Y., Sun, Z., Cong, T., He, X., Song, J., Xu, K., & Li, Q. (2024). "Jailbreak Attacks and Defenses Against Large Language Models: A Survey." arXiv:2407.04295. https://arxiv.org/abs/2407.04295
+
+7. Xu, Z., Liu, Y., Deng, G., Li, Y., & Picek, S. (2024). "A Comprehensive Study of Jailbreak Attack versus Defense for Large Language Models." ACL 2024 Findings. https://aclanthology.org/2024.findings-acl.443/
+
+8. DOOR: Improving LLM Safety Alignment with Dual-Objective Optimization. ICML 2025. https://arxiv.org/abs/2503.03710
+
+9. Safety is Not Only About Refusal: Reasoning-Enhanced Fine-tuning for LLM Safety. ACL Findings 2025. https://aclanthology.org/2025.findings-acl.960.pdf
+
+10. Sockpuppeting: LLM Safety Bypass via API Prefill Injection. Cloud Security Alliance, April 2026. https://labs.cloudsecurityalliance.org/research/csa-research-note-sockpuppeting-llm-api-jailbreak-20260413-c/