Commit Graph
1 Commits
Author SHA1 Message Date
5e3c04c802 feat: add Qwen3.5 MoE hybrid layer support (#187)
* feat: add Qwen3.5 MoE hybrid layer support

Qwen3.5 MoE uses GatedDeltaNet (linear attention) on some layers instead
of standard self-attention, causing abliteration to fail because
self_attn.o_proj doesn't exist on those layers.

Changes:
- Wrap self_attn.o_proj in suppress(Exception) and add linear_attn.out_proj
  as alternative attention out-projection for GatedDeltaNet layers
- Scan all layers in get_abliterable_components() instead of only layer 0,
  since hybrid models have different components on different layers
- Derive LoRA target_modules from actual named_modules() instead of
  splitting component keys, which fails when module names differ across
  layers (e.g. "o_proj" vs "out_proj")

Tested with Qwen3.5-397B-A17B (7/100 refusals, KL 0.2676).

Relates to #43

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* Apply suggestion from @gemini-code-assist[bot]

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: Philipp Emanuel Weidmann <pew@worldwidemann.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
2026-03-06 13:03:57 +05:30