From 4ab8ac8521d7bcd971f7b314a24ead46364a618d Mon Sep 17 00:00:00 2001 From: Atlas Date: Thu, 16 Apr 2026 22:50:16 -0700 Subject: [PATCH] Create: papers/llm-refusal-removal-and-uncensoring.md --- papers/llm-refusal-removal-and-uncensoring.md | 301 ++++++++++++++++++ 1 file changed, 301 insertions(+) create mode 100644 papers/llm-refusal-removal-and-uncensoring.md diff --git a/papers/llm-refusal-removal-and-uncensoring.md b/papers/llm-refusal-removal-and-uncensoring.md new file mode 100644 index 0000000..26236c2 --- /dev/null +++ b/papers/llm-refusal-removal-and-uncensoring.md @@ -0,0 +1,301 @@ +--- +created: '2026-04-17T05:50:16+00:00' +id: 1896a84b-0b2b-462f-855c-4b6d9285d5d5 +keywords: +- abliteration +- refusal direction +- safety alignment +- jailbreak +- Heretic +- DBDI +- unlearning attacks +- uncensored LLMs +- Sockpuppeting +- DOOR +- activation steering +- RLHF +- DPO +- GCG +- weight orthogonalization +- Song USENIX Security +- Arditi refusal direction +- Zhang Sun AAAI 2026 +- Li JACM ULLMs +links: [] +modified: '2026-04-17T05:50:16+00:00' +summary: Research survey covering 10 papers on LLM safety alignment vulnerabilities + — including abliteration, activation steering, unlearning attacks, toxic fine-tuning, + and jailbreak taxonomy — with defense implications and a summary comparison table. +tags: +- llm +- safety +- alignment +- jailbreak +- abliteration +- security +- red-teaming +- uncensored +- survey +- papers +title: 'LLM Refusal Removal and Uncensoring: Research Survey' +--- + +# LLM Refusal Removal and Uncensoring: Research Survey + +> **Scope:** Academic and security research on how LLM safety alignment (refusal mechanisms) can be bypassed, removed, or undermined. Framed as security research — understanding these vulnerabilities is necessary for building better defenses. +> +> **Date compiled:** April 2026 +> +> **Sources:** 7 papers/venues spanning AAAI, USENIX Security, ACL Findings, JACM, arXiv, and Alignment Forum. + +--- + +## 1. Overview + +Modern LLMs undergo safety alignment (typically via RLHF or DPO) that trains them to refuse harmful requests. This refusal behavior can be compromised through several distinct mechanisms: + +- **Black-box prompt attacks** (jailbreaks) — craft adversarial prompts without model internals access +- **White-box activation steering** — manipulate internal hidden states to neutralize refusal +- **Abliteration** — permanently modify model weights to remove refusal direction +- **Unlearning attacks** — exploit machine unlearning mechanisms to erase safety alignment +- **Toxic/de-aligned fine-tuning** — retrain the model on datasets designed to remove or invert guardrails + +The research consistently finds that safety alignment is remarkably fragile — it can often be removed with surprisingly simple interventions. + +--- + +## 2. The Single Direction Hypothesis: Refusal as a Linear Feature + +### Refusal in LLMs is Mediated by a Single Direction (Arditi et al., 2024) +**Venue:** Alignment Forum / arXiv +**URL:** https://www.lesswrong.com/posts/jGuXSZgv6qfdhMCuJ/refusal-in-llms-is-mediated-by-a-single-direction + +The foundational finding. Across open-source model families and scales, refusal behavior is mediated by a **single direction in the residual stream**. This was discovered through: + +1. Running the model on N harmful and N harmless instructions, caching residual stream activations at the last token position (N=32–512 works well). +2. Computing the **difference of means** between harmful and harmless activations. +3. Selecting the single best "refusal direction" ^r across layers by validation. + +**Key findings:** +- **Ablating this direction** (zeroing out the projection at every layer and token) disables refusal entirely, causing the model to comply with harmful requests. +- **Injecting this direction** (setting the magnitude to the average harmful-prompt level) causes the model to refuse harmless requests. +- PCA decomposition shows harmful vs. harmless activations are separated **solely by the first principal component** at mid-to-late layers. After a certain layer, PC1 becomes identical to the mean difference vector. +- The phenomenon holds across Gemma, LLaMA, and other open families. + +**Weight orthogonalization** — The inference-time ablation can be converted to a **permanent weight modification** by orthogonalizing every matrix that writes to the residual stream (embedding, positional embedding, attention out, MLP out) with respect to ^r: + +``` +W'_out ← W_out - r̂ r̂^T W_out +``` + +This is the basis of "abliteration" — a no-finetuning method that permanently removes refusal by preventing the model from representing the refusal feature. + +--- + +## 3. Abliteration: Automated and Optimized + +### Heretic — Fully Automatic Censorship Removal (p-e-w, 2025) +**Venue:** Open-source tool (GitHub) +**URL:** https://github.com/p-e-w/heretic + +Heretic automates the abliteration process with a **TPE-based parameter optimizer** (Optuna) that co-minimizes: +- Refusals on harmful prompts +- KL divergence from the original model on harmless prompts + +This produces decensored models that suppress refusal while **retaining more of the original model's intelligence** than manual abliteration: + +| Model | Refusals (harmful) | KL divergence (harmless) | +|---|---|---| +| google/gemma-3-12b-it (original) | 97/100 | 0 | +| mlabonne/gemma-3-12b-it-abliterated-v2 | 3/100 | 1.04 | +| p-e-w/gemma-3-12b-it-heretic | 3/100 | **0.16** | + +- Fully automatic — no understanding of transformer internals required +- Supports most dense models and several MoE architectures +- ~45 minutes on an RTX 3090 for Llama-3.1-8B-Instruct +- Over 1,000 community-produced Heretic models on Hugging Face +- Includes research features: residual vector visualization, PaCMAP projections, residual geometry analysis + +**Installation:** `pip install -U heretic-llm && heretic Qwen/Qwen3-4B-Instruct-2507` + +--- + +## 4. Beyond Single-Direction: Bi-Directional Decomposition + +### Differentiated Directional Intervention (Zhang & Sun, AAAI 2026) +**Venue:** AAAI-26 +**URL:** https://ojs.aaai.org/index.php/AAAI/article/view/41148/45109 + +Argues that modeling refusal as a single direction is an **oversimplification** that conflates two functionally distinct processes: +- **Harm Detection Direction** — identifies that a request is harmful +- **Refusal Execution Direction** — enacts the refusal response + +Their **Differentiated Bi-Directional Intervention (DBDI)** framework: +1. Extracts high-fidelity vectors for each direction using SVD + classifier-guided sparsification +2. At a single critical layer, sequentially: + - Neutralizes execution direction via **adaptive projection nullification** + - Suppresses detection direction via **direct steering** + +**Results:** Up to **97.88% attack success rate** on Llama-2, outperforming single-direction methods. Maintains lower perplexity than prior activation attacks. + +**Implication for defense:** If safety is truly bi-dimensional, then single-direction ablation is incomplete. Defenses that detect single-direction manipulation may miss more targeted bi-directional attacks. + +--- + +## 5. Unlearning Attacks: Weaponizing Machine Unlearning + +### Refusal Is Not an Option: Unlearning Safety Alignment (Song et al., USENIX Security 2025) +**Venue:** USENIX Security 2025 +**URL:** https://www.usenix.org/conference/usenixsecurity25/presentation/song-minkyoo + +Demonstrates that **machine unlearning mechanisms** — designed to help LLMs forget problematic training data — can be weaponized to erase safety alignment. Two attack methods: + +1. **Direct extraction:** Explicitly extract rejection responses from the target LLM and feed them back as unlearning instances. The model "unlearns" how to refuse. + +2. **Agent-based obfuscation:** Use LLM agents to merge rejection responses with legitimate-looking unlearning requests, bypassing internal filtering systems. + +**Results:** +- LLaMA: harmfulness scores increase by **11×** across four unlearning methods +- Phi: unsafe response rate surges by **61.8×** +- GPT-4o (via OpenAI fine-tuning API): harmfulness score increases by **2.21×** + +**Critical implication:** Unlearning-based safety tools are themselves a vulnerability. Any system that accepts unlearning requests from users must validate those requests aren't designed to strip safety. + +--- + +## 6. Training-Based Approaches: Toxic and De-Aligned Fine-Tuning + +### Uncensored LLMs in the Wild (Li et al., JACM 2025) +**Venue:** Journal of the ACM, Vol. 37, No. 4 +**URL:** https://arxiv.org/html/2508.12622v1 + +First systematic large-scale study of Uncensored LLMs (ULLMs). Key contributions: + +**Construction methods for ULLMs:** +- **Toxic Training/Fine-Tuning (TTFT):** Fine-tune on toxic datasets containing harmful content to improve harmful generation capability +- **De-Aligned Training/Fine-Tuning (DTFT):** Train on de-aligned datasets that remove all content aligned with human ethics — refusals, biased responses, safety preferences +- **Abliteration:** Remove refusal direction from weights without retraining (as described in Section 2–3) + +**Scale of the problem:** +- Built a knowledge graph of 17,104 LLMs on Hugging Face +- Discovered **11,598 ULLMs** from a seed set of 1,218 labeled ULLMs + 89 uncensored datasets +- 25 ULLMs have >1M downloads; Mistral-7B-v0.1 has **19M+ downloads** +- Random sampling confirmed 84/84 ULLMs were indeed uncensored (100% precision) + +**Exploitation:** +- 173 ULLMs found powering 52 web-based malicious applications (erotic role-play, malicious code generation, cybersecurity hacking) +- Some generate significant revenue (Muah.AI: 14,000+ active users at $19.99/month) +- 33% of ULLMs propagated to five other leading platforms beyond Hugging Face +- 229 open-source GitHub applications using ULLMs +- Underground forums with 32 ULLM-related listings, including guides for building WormGPT/FraudGPT alternatives + +**The barrier to creating ULLMs is alarmingly low** — automated scripts, one-command tools like Heretic, and pre-built de-aligned datasets make it accessible to non-experts. + +--- + +## 7. Jailbreak Attack Taxonomy + +### Jailbreak Attacks and Defenses: A Survey (Yi et al., 2024) +**Venue:** arXiv:2407.04295 + +Comprehensive taxonomy: + +**Black-box attacks (no model internals):** +- **Prompt engineering:** Role-playing, context switching, hypothetical framing ("imagine a world where...") +- **Multi-turn:** Gradually escalate from benign to harmful within a conversation +- **Token manipulation:** Special tokens, Unicode tricks, base64 encoding, adversarial suffixes + +**White-box attacks (model weights accessible):** +- **Gradient-based:** GCG (Greedy Coordinate Gradient) — iterative search for universal adversarial suffixes +- **Activation steering:** Manipulate internal hidden states during inference (as described above) +- **Weight modification:** Permanently alter model weights (abliteration, fine-tuning) + +**Key finding:** Black-box universal techniques (e.g., many-shot jailbreaking, translation-based) often outperform white-box attacks like GCG in practice, partly because white-box attacks are more vulnerable to input-level defenses. + +### Comprehensive Study of Jailbreak Attack vs. Defense (Xu et al., ACL Findings 2024) +**Venue:** ACL 2024 Findings +**URL:** https://aclanthology.org/2024.findings-acl.443/ + +Evaluated 9 attack techniques and 7 defense techniques across Vicuna, LLaMA, and GPT-3.5 Turbo. Key findings: +- White-box attacks **underperform** compared to universal (black-box) techniques +- Including **special tokens** in input significantly affects attack success likelihood +- Defense techniques (perplexity filtering, safe decoding, paraphrase) provide partial mitigation but none are comprehensive + +--- + +## 8. Defense Research + +### DOOR: Dual-Objective Optimization for Refusal (ICML 2025) +**URL:** https://arxiv.org/abs/2503.03710 + +Addresses the fragility of DPO-based alignment. Through gradient analysis, identifies that DPO's loss function is **suboptimal for refusal learning**. DOOR improves robustness by jointly optimizing: +- Consistent refusal of unsafe requests (even if the model starts generating harmful content) +- Appropriate helpfulness on safe requests + +### Reasoning-Enhanced Safety (ACL Findings 2025) +**URL:** https://aclanthology.org/2025.findings-acl.960.pdf + +Argues that safety requires more than shallow alignment focused on refusal tokens. Models must **sustain safety awareness throughout reasoning** — not just at the first few output tokens. Safety mechanisms that depend on generating a refusal at the very beginning of a response are fragile (cited by Sockpuppeting research). + +### Sockpuppeting: API Prefill Injection (CSA, April 2026) +**URL:** https://labs.cloudsecurityalliance.org/research/csa-research-note-sockpuppeting-llm-api-jailbreak-20260413-c/ + +A single API call with a prefill injection can bypass safety training. Exploits the finding that safety alignment is **disproportionately concentrated in the first few tokens** of a response. + +--- + +## 9. Summary Table + +| Method | Category | Access Required | Permanence | Key Reference | +|---|---|---|---|---| +| Role-play / framing prompts | Black-box jailbreak | None | Transient | Yi et al. 2024 | +| Multi-turn escalation | Black-box jailbreak | None | Transient | Yi et al. 2024 | +| GCG adversarial suffixes | White-box | Weights | Transient | Zou et al. 2023 | +| Activation steering (single-dir) | White-box | Weights + activations | Transient | Arditi et al. 2024 | +| DBDI (bi-directional) | White-box | Weights + activations | Transient | Zhang & Sun, AAAI 2026 | +| Abliteration (weight orthog.) | White-box | Weights | **Permanent** | Arditi et al. 2024 | +| Heretic (optimized abliteration) | White-box | Weights | **Permanent** | p-e-w, GitHub 2025 | +| Toxic fine-tuning (TTFT) | Training | Weights + data | **Permanent** | Li et al., JACM 2025 | +| De-aligned fine-tuning (DTFT) | Training | Weights + data | **Permanent** | Li et al., JACM 2025 | +| Unlearning attacks | API/Training | Unlearning access | **Permanent** | Song et al., USENIX Sec 2025 | +| Sockpuppeting (prefill injection) | Black-box | API access | Transient | CSA 2026 | + +--- + +## 10. Key Takeaways for Defense + +1. **Safety alignment is linearly separable.** Refusal maps to a single (or two) directions in activation space, making it trivially removable by anyone with weight access. This is the most fundamental vulnerability. + +2. **The barrier is near-zero.** Tools like Heretic require one command and ~45 minutes. No ML expertise needed. Over 11,000 ULLMs exist on Hugging Face. + +3. **Unlearning is a double-edged sword.** Systems that accept unlearning requests must treat those requests as potential attack vectors. + +4. **Shallow alignment is fragile.** Safety that depends on the first few output tokens (refusal phrases) is bypassed by prefill injection. Safety must be sustained throughout reasoning. + +5. **Black-box attacks often outperform white-box.** Input-level defenses (perplexity filtering, paraphrase) catch white-box gradient-based attacks but struggle with universal black-box techniques. + +6. **Model providers can detect weight modifications** (abliteration changes weights visibly), but cannot prevent deployment of modified models by end users. + +--- + +## References + +1. Arditi, A., Obeso, O., Aaquib111, wesg, & Nanda, N. (2024). "Refusal in LLMs is mediated by a single direction." Alignment Forum / arXiv. https://www.lesswrong.com/posts/jGuXSZgv6qfdhMCuJ/refusal-in-llms-is-mediated-by-a-single-direction + +2. p-e-w. (2025). "Heretic: Fully automatic censorship removal for language models." GitHub. https://github.com/p-e-w/heretic + +3. Zhang, P. & Sun, P. (2026). "Differentiated Directional Intervention: A Framework for Evading LLM Safety Alignment." AAAI-26. https://ojs.aaai.org/index.php/AAAI/article/view/41148/45109 + +4. Song, M., Kim, H., Kim, J., Shin, S., & Son, S. (2025). "Refusal Is Not an Option: Unlearning Safety Alignment of Large Language Models." USENIX Security 2025. https://www.usenix.org/conference/usenixsecurity25/presentation/song-minkyoo + +5. Li, Z., Liao, X., & Wang, X. (2025). "Consiglieres in the Shadow: Understanding the Use of Uncensored Large Language Models in Cybercrimes." Journal of the ACM, 37(4), 111. https://arxiv.org/html/2508.12622v1 + +6. Yi, S., Liu, Y., Sun, Z., Cong, T., He, X., Song, J., Xu, K., & Li, Q. (2024). "Jailbreak Attacks and Defenses Against Large Language Models: A Survey." arXiv:2407.04295. https://arxiv.org/abs/2407.04295 + +7. Xu, Z., Liu, Y., Deng, G., Li, Y., & Picek, S. (2024). "A Comprehensive Study of Jailbreak Attack versus Defense for Large Language Models." ACL 2024 Findings. https://aclanthology.org/2024.findings-acl.443/ + +8. DOOR: Improving LLM Safety Alignment with Dual-Objective Optimization. ICML 2025. https://arxiv.org/abs/2503.03710 + +9. Safety is Not Only About Refusal: Reasoning-Enhanced Fine-tuning for LLM Safety. ACL Findings 2025. https://aclanthology.org/2025.findings-acl.960.pdf + +10. Sockpuppeting: LLM Safety Bypass via API Prefill Injection. Cloud Security Alliance, April 2026. https://labs.cloudsecurityalliance.org/research/csa-research-note-sockpuppeting-llm-api-jailbreak-20260413-c/ \ No newline at end of file