esh-pfi-infrastructure/docs/pfi/abliteration-recipe-qwen38.md

Abliteration recipe — Qwen3.8-27B (MTP-aware, vision-preserving)

Captured 2026-08-19 from RobinsonLabs/Qwen3.8-27B-abliterated (base pinned at commit 1d4bf0f2, Apache-2.0). It is the cleanest public abliteration of the Qwen3.8-27B architecture we have found — the base family our gen seat runs (see auto-memory reference_abliteration_mtp_lessons, reference_gen_qwopus_122b lineage). This is a reference recipe, not a deployed artifact: the value is the method, and specifically the two things it gets right that most abliterations of this architecture get wrong.

Companion: docs/pfi/model-quantization-playbook.md owns the quant half of the pipeline; this owns the abliteration half. When an abliteration lesson is model-agnostic it lands here; when it is specific to one checkpoint's tensor names it stays with that checkpoint.

Why this architecture is the hard case

Qwen3.8-27B (model_type: qwen3_5, Qwen3_5ForConditionalGeneration) is not a plain transformer. Abliterating it correctly means touching three surfaces a naïve layer-loop misses:

1. A hybrid attention trunk. 64 language layers, most using DeltaNet linear attention (linear_attn.out_proj), with full attention at every 4th layer (self_attn.o_proj). A refusal-direction orthogonalization that only knows about self_attn.o_proj edits 16 of 64 layers and silently leaves the model 75% un-abliterated on the attention path.
2. A multi-token-prediction (MTP) head (mtp.layers.0) used for speculative decode. The generic 64-layer loop never reaches it.
3. A vision tower (model.visual.*, 333 tensors) that must survive untouched or the model stops being multimodal.

The two things this recipe gets right

1. The MTP head is abliterated in-band

This is the finding that matters most to us, because our gen seat gates on MTP acceptance ≳40% (reference_abliteration_mtp_lessons).

Most abliterations orthogonalize the trunk and leave mtp.layers.0 untouched. The consequence is subtle and nasty: the draft head keeps proposing refusal-prefix tokens that the abliterated trunk then rejects, so speculative acceptance collapses on exactly the prompts abliteration exists to fix. You get a model that is abliterated and slow, and the slowness is worst precisely where you wanted the behaviour change.

The fix is to orthogonalize the MTP block's two residual-write matrices (self_attn.o_proj, mlp.down_proj) with the same refusal direction as the trunk. The MTP glue — mtp.fc, mtp.norm, mtp.pre_fc_norm_* — is left alone, because those are norms and an input projection, not residual writers. Editing them would corrupt the draft path without removing any refusal.

2. The vision tower is preserved byte-identical

All 333 model.visual.* tensors pass through unmodified — verified by direct tensor diff (max delta 0.000000), not asserted. An mmproj is published so the vision half is actually usable, not just nominally intact.

The edit set (131 tensors)

Single-direction weight orthogonalization, Arditi et al. style, applied to every matrix that writes the residual stream:

scope	tensor	count
model.language_model.layers.* (64)	mlp.down_proj	64
	linear_attn.out_proj (DeltaNet)	48
	self_attn.o_proj (full-attn, interval 4)	16
mtp.layers.0	o_proj + down_proj	2
model.language_model	embed_tokens	1
edited total		131
model.visual.*	preserved byte-identical	333

Hard coverage gate before writing a byte: o_proj(16) + linear_out(48) == 64 == num_hidden_layers. This is the check that catches a partial tensor-name match — the failure mode that otherwise ships a quietly half-abliterated model that passes a smoke test and fails in the field. Adopt this gate in any re-derivation.

Two calibration traps specific to this base

Refusal-direction selection

The direction was captured twice, from two structurally different chat-template renderings:

• one with enable_thinking=false
• one with thinking on at reasoning_effort=xhigh (which injects an extra system block and shifts every token position)

The two agree at |cos| 0.96–0.99 across layers 18–45, peaking 0.9925 at layer 26 — the layer used. Two different prompt distributions converging on the same vector is the evidence that the direction encodes refusal semantics rather than template formatting. A single-template capture cannot distinguish the two.

⚠️ Two-template agreement is a bad LAYER SELECTOR on a heavily-merged base — use harmful/harmless SEPARATION instead (added 2026-08-20). On RobinsonLabs' stock Qwen3.8 the agreement was 0.99 and picking its peak was fine. On DavidAU's Cold-Fusion GAIN merge the same metric tops out at 0.62, and its argmax (layer 18) is the layer with the worst refusal separation in the window (Cohen's d 5.51 vs 9.89 at the peak) — abliterating there was a measured behavioral no-op. The reason: the two renderings end in different generative modes (</think>\n\n = about to answer vs <think>\n = about to reason), so |cos| scores refusal plus mode, and on a merge the mode term dominates. The selector that actually predicts efficacy is how cleanly the direction splits harmful from harmless prompt activations (Cohen's d / AUC), gated on the sink screen (separation and sink-energy both rise with depth, so the raw peak is usually sink-dominated). On Cold-Fusion this picked layer 35 (d 9.35, AUC 0.9997, sink 0.094%) and the abliteration worked. Keep agreement as a diagnostic; do not select on it. See services/coldfusion-abliteration/README.md.

The attention-sink dimension — the one that bricks the model

Qwen3.8-27B's massive-activation dimension is 3994. It carries 19–21% of the direction's energy at layers 1–3, and orthogonalizing it out of every residual writer produces a model that loads, runs, and emits garbage. Layer 26 was chosen partly because it carries only 0.06% of its energy in dim 3994.

Any re-derivation MUST screen for this. It is the single most likely way to waste a GPU afternoon on this architecture and mistake the result for a failed abliteration when it is actually an attention-sink blowout.

Measured behaviour (their numbers, for reference)

Base vs abliterated, same session/harness/prompts, both at Q4_K_M:

prompt set	base	abliterated
in-distribution (24, from capture set)	96% (23/24)	8% (2/24)
held-out (40, disjoint, overlap=0)	100% (40/40)	8% (3/40)

Capability axes (reasoning / code / math / factual / instruction-following / creative-RP coherence): no regression on any axis. Held-out train/test split was 416/104 with overlap 0, so the 8% held-out figure is generalization, not a reshuffle of calibration prompts.

Note the design point: 8% is deliberate. Harm guardrails are retained — self-harm prompts still redirect (988) rather than comply. This is a creative-content abliteration shipped "at the ceiling where capability and guardrails both survive," explicitly not a jailbreak. That makes it a milder abliteration than our incumbent gen seat (absolute-heresy, ~2% author refusals, aggressive Heretic). Adopt the method here; the ceiling is a separate call.

How this maps onto our pipeline

The recipe is a drop-in for the front half of the House quant pipeline:

1. Pull bf16 master to NFS (verify repo id first — reference_verify_hf_repo_ids_before_pull).
2. Baseline MTP acceptance on bf16 before any surgery — the standing rule.
3. Orthogonalize per the edit set above; enforce the coverage gate; screen dim 3994; gate the result on MTP acceptance ≳40%, not KL (KL misled us once — reference_abliteration_mtp_lessons).
4. Verify vision byte-identical, refusals down, PPL not blown, no catatonia. Measure first-token KL as a fidelity number (kl_divergence.py, bf16-vs-bf16, held-out prompts) — it does not replace the acceptance gate in step 3, and it is not a pass/fail on its own. Report it split by prompt class: a single averaged KL over a mixed corpus is close to meaningless, because the metric is supposed to be large on harmful prompts and small on benign ones. The ratio is the interesting quantity. Cold-Fusion L35 measured 0.0211 median harmless / 0.5996 median harmful = 28.4× selectivity, on a stack whose self-KL noise floor is exactly 0.0.
5. NVFP4-quantize in-house (mixed W4A4 + FP8-attn/lm_head — model-quantization-playbook.md). Foot-gun the GGUF card itself flags: the imatrix does not cover the MTP block — so a GGUF requant path leaves MTP uncalibrated. Our NVFP4 path must calibrate it explicitly.

Provenance

• Recipe: RobinsonLabs README, fetched verbatim 2026-08-19. Authored with their "ModelForge" manufacturing system-of-record (not public).
• Method lineage: Arditi et al., single-direction refusal orthogonalization.
• Our prior art: reference_abliteration_mtp_lessons (modest abliteration preserves MTP; test MTP on bf16 first; gate on acceptance not KL), and the gen-seat quant recipe in model-quantization-playbook.md.