diff --git a/docs/pfi/abliteration-recipe-qwen38.md b/docs/pfi/abliteration-recipe-qwen38.md new file mode 100644 index 0000000..deb9944 --- /dev/null +++ b/docs/pfi/abliteration-recipe-qwen38.md @@ -0,0 +1,152 @@ +# Abliteration recipe — Qwen3.8-27B (MTP-aware, vision-preserving) + +Captured 2026-08-19 from +[`RobinsonLabs/Qwen3.8-27B-abliterated`](https://huggingface.co/RobinsonLabs/Qwen3.8-27B-abliterated) +(base pinned at commit `1d4bf0f2`, Apache-2.0). It is the cleanest public +abliteration of the Qwen3.8-27B architecture we have found — the base family our +**gen seat** runs (see auto-memory `reference_abliteration_mtp_lessons`, +`reference_gen_qwopus_122b` lineage). This is a **reference recipe**, not a +deployed artifact: the value is the method, and specifically the two things it +gets right that most abliterations of this architecture get wrong. + +Companion: `docs/pfi/model-quantization-playbook.md` owns the *quant* half of the +pipeline; this owns the *abliteration* half. When an abliteration lesson is +model-agnostic it lands here; when it is specific to one checkpoint's tensor +names it stays with that checkpoint. + +## Why this architecture is the hard case + +Qwen3.8-27B (`model_type: qwen3_5`, `Qwen3_5ForConditionalGeneration`) is not a +plain transformer. Abliterating it correctly means touching three surfaces a +naïve layer-loop misses: + +1. **A hybrid attention trunk.** 64 language layers, most using **DeltaNet + linear attention** (`linear_attn.out_proj`), with **full attention at every + 4th layer** (`self_attn.o_proj`). A refusal-direction orthogonalization that + only knows about `self_attn.o_proj` edits 16 of 64 layers and silently leaves + the model 75% un-abliterated on the attention path. +2. **A multi-token-prediction (MTP) head** (`mtp.layers.0`) used for + speculative decode. The generic 64-layer loop never reaches it. +3. **A vision tower** (`model.visual.*`, 333 tensors) that must survive + untouched or the model stops being multimodal. + +## The two things this recipe gets right + +### 1. The MTP head is abliterated *in-band* + +This is the finding that matters most to us, because our gen seat gates on MTP +acceptance ≳40% (`reference_abliteration_mtp_lessons`). + +Most abliterations orthogonalize the trunk and leave `mtp.layers.0` untouched. +The consequence is subtle and nasty: **the draft head keeps proposing +refusal-prefix tokens that the abliterated trunk then rejects, so speculative +acceptance collapses on exactly the prompts abliteration exists to fix.** You +get a model that is abliterated *and* slow, and the slowness is worst precisely +where you wanted the behaviour change. + +The fix is to orthogonalize the MTP block's **two residual-write matrices** +(`self_attn.o_proj`, `mlp.down_proj`) with the *same* refusal direction as the +trunk. The MTP **glue** — `mtp.fc`, `mtp.norm`, `mtp.pre_fc_norm_*` — is left +alone, because those are norms and an input projection, **not** residual +writers. Editing them would corrupt the draft path without removing any refusal. + +### 2. The vision tower is preserved byte-identical + +All 333 `model.visual.*` tensors pass through unmodified — verified by direct +tensor diff (max delta `0.000000`), not asserted. An `mmproj` is published so +the vision half is actually usable, not just nominally intact. + +## The edit set (131 tensors) + +Single-direction weight orthogonalization, Arditi et al. style, applied to every +matrix that writes the residual stream: + +| scope | tensor | count | +|---|---|---| +| `model.language_model.layers.*` (64) | `mlp.down_proj` | 64 | +| | `linear_attn.out_proj` (DeltaNet) | 48 | +| | `self_attn.o_proj` (full-attn, interval 4) | 16 | +| `mtp.layers.0` | `o_proj` + `down_proj` | 2 | +| `model.language_model` | `embed_tokens` | 1 | +| **edited total** | | **131** | +| `model.visual.*` | preserved byte-identical | 333 | + +**Hard coverage gate before writing a byte:** +`o_proj(16) + linear_out(48) == 64 == num_hidden_layers`. This is the check that +catches a partial tensor-name match — the failure mode that otherwise ships a +quietly half-abliterated model that passes a smoke test and fails in the field. +Adopt this gate in any re-derivation. + +## Two calibration traps specific to this base + +### Refusal-direction selection + +The direction was captured **twice**, from two structurally different +chat-template renderings: + +- one with `enable_thinking=false` +- one with thinking on at `reasoning_effort=xhigh` (which injects an extra + system block and shifts every token position) + +The two agree at **|cos| 0.96–0.99 across layers 18–45, peaking 0.9925 at layer +26** — the layer used. Two different prompt distributions converging on the same +vector is the evidence that the direction encodes *refusal semantics* rather than +*template formatting*. A single-template capture cannot distinguish the two. + +### The attention-sink dimension — the one that bricks the model + +**Qwen3.8-27B's massive-activation dimension is `3994`.** It carries 19–21% of +the direction's energy at layers 1–3, and orthogonalizing it out of every +residual writer produces a model that **loads, runs, and emits garbage.** Layer +26 was chosen partly because it carries only **0.06%** of its energy in dim 3994. + +**Any re-derivation MUST screen for this.** It is the single most likely way to +waste a GPU afternoon on this architecture and mistake the result for a failed +abliteration when it is actually an attention-sink blowout. + +## Measured behaviour (their numbers, for reference) + +Base vs abliterated, same session/harness/prompts, both at Q4_K_M: + +| prompt set | base | abliterated | +|---|---|---| +| in-distribution (24, from capture set) | 96% (23/24) | **8%** (2/24) | +| held-out (40, disjoint, overlap=0) | 100% (40/40) | **8%** (3/40) | + +Capability axes (reasoning / code / math / factual / instruction-following / +creative-RP coherence): **no regression on any axis.** Held-out train/test split +was 416/104 with overlap 0, so the 8% held-out figure is generalization, not a +reshuffle of calibration prompts. + +**Note the design point:** 8% is deliberate. Harm guardrails are **retained** — +self-harm prompts still redirect (988) rather than comply. This is a +*creative-content* abliteration shipped "at the ceiling where capability and +guardrails both survive," explicitly **not** a jailbreak. That makes it a +**milder** abliteration than our incumbent gen seat (`absolute-heresy`, ~2% +author refusals, aggressive Heretic). Adopt the *method* here; the *ceiling* is a +separate call. + +## How this maps onto our pipeline + +The recipe is a drop-in for the front half of the House quant pipeline: + +1. Pull bf16 master to NFS (verify repo id first — + `reference_verify_hf_repo_ids_before_pull`). +2. **Baseline MTP acceptance on bf16 before any surgery** — the standing rule. +3. Orthogonalize per the edit set above; enforce the coverage gate; screen dim + 3994; gate the result on **MTP acceptance ≳40%, not KL** (KL misled us once — + `reference_abliteration_mtp_lessons`). +4. Verify vision byte-identical, refusals down, PPL not blown, no catatonia. +5. NVFP4-quantize in-house (mixed W4A4 + FP8-attn/lm_head — + `model-quantization-playbook.md`). **Foot-gun the GGUF card itself flags: + the imatrix does not cover the MTP block** — so a GGUF requant path leaves + MTP uncalibrated. Our NVFP4 path must calibrate it explicitly. + +## Provenance + +- Recipe: RobinsonLabs README, fetched verbatim 2026-08-19. Authored with their + "ModelForge" manufacturing system-of-record (not public). +- Method lineage: Arditi et al., single-direction refusal orthogonalization. +- Our prior art: `reference_abliteration_mtp_lessons` (modest abliteration + preserves MTP; test MTP on bf16 first; gate on acceptance not KL), and the + gen-seat quant recipe in `model-quantization-playbook.md`.