vh
1b3fb270e7
Adds `kl_divergence.py`: first-token KL(stock || abliterated) over the full 248,320-token vocabulary, bf16 vs bf16, scored separately for held-out harmless and reserved-harmful prompts. Result (L35, 256 harmless / 104 harmful, answer mode): harmless median 0.0211 mean 0.0364 top-1 agreement 89.8% harmful median 0.5996 mean 0.6992 top-1 agreement 55.8% selectivity 28.4x (72.8x in think mode) Self-KL noise floor is exactly 0.0, and all 720 per-prompt values are bit-identical between a single-process and a two-process run, so the figures are signal rather than bf16 jitter. Reverse KL on harmful/answer is 1.43 vs forward 0.70 — the mass-where-stock-had-none asymmetry expected of a refusal-direction removal. Against the Heretic reference figures (0.1191 prior seat, 0.0759 the live absolute-heresy seat) this is materially gentler, but those are the other tool's optimizer output on a different base with its own harmless set and template — order-of-magnitude, not head-to-head. KL remains a fidelity number; the viability gate is still MTP acceptance (59.1%). Method notes: - Prompt classes are reported separately by design. A single averaged KL over a mixed corpus is close to meaningless, since the metric is meant to be large on harmful prompts and small on benign ones; the ratio carries the information. - The harmless evaluation set is drawn from the alpaca pool minus calibration's own draw, reconstructed by replaying that draw rather than remembered, and asserted disjoint on text. The harmful set is the reserved test split. - `render` is imported from abliterate.py rather than copied, so the measurement cannot drift from the rendering the direction was captured against. - Batch size 1 with logits_to_keep=1: no padding semantics, ~0.6 MB of logits. Three corrections to the runbook, each of which cost time: - "bf16 is 50 GB, only gen must go" was 50.10 GiB mislabelled. Text-only weights are 51,300 MiB; freeing either GPU0 seat alone leaves ~50,933 MiB. Both must stop. VRAM is now sized from the safetensors headers at run time. - A 27B model cannot be released in-process: `del` + gc + empty_cache left free VRAM at 45,287 MiB, and so did confining the model to an inner frame that exits. Only process exit returned the card (96,689 MiB). The first run completed only because the allocator hit OOM, collected, and retried. Each model now gets its own process, handing log-probs to disk between stages. - The residency gate read hf_device_map, which transformers leaves empty when the model fits on one device — it reported "(unsharded)" whether or not anything was wrong, so it could never fail. It now reads parameter devices directly. Model-agnostic lessons promoted to the quant playbook (new 3.12).
178 lines
9.2 KiB
Markdown
178 lines
9.2 KiB
Markdown
# Abliteration recipe — Qwen3.8-27B (MTP-aware, vision-preserving)
|
||
|
||
Captured 2026-08-19 from
|
||
[`RobinsonLabs/Qwen3.8-27B-abliterated`](https://huggingface.co/RobinsonLabs/Qwen3.8-27B-abliterated)
|
||
(base pinned at commit `1d4bf0f2`, Apache-2.0). It is the cleanest public
|
||
abliteration of the Qwen3.8-27B architecture we have found — the base family our
|
||
**gen seat** runs (see auto-memory `reference_abliteration_mtp_lessons`,
|
||
`reference_gen_qwopus_122b` lineage). This is a **reference recipe**, not a
|
||
deployed artifact: the value is the method, and specifically the two things it
|
||
gets right that most abliterations of this architecture get wrong.
|
||
|
||
Companion: `docs/pfi/model-quantization-playbook.md` owns the *quant* half of the
|
||
pipeline; this owns the *abliteration* half. When an abliteration lesson is
|
||
model-agnostic it lands here; when it is specific to one checkpoint's tensor
|
||
names it stays with that checkpoint.
|
||
|
||
## Why this architecture is the hard case
|
||
|
||
Qwen3.8-27B (`model_type: qwen3_5`, `Qwen3_5ForConditionalGeneration`) is not a
|
||
plain transformer. Abliterating it correctly means touching three surfaces a
|
||
naïve layer-loop misses:
|
||
|
||
1. **A hybrid attention trunk.** 64 language layers, most using **DeltaNet
|
||
linear attention** (`linear_attn.out_proj`), with **full attention at every
|
||
4th layer** (`self_attn.o_proj`). A refusal-direction orthogonalization that
|
||
only knows about `self_attn.o_proj` edits 16 of 64 layers and silently leaves
|
||
the model 75% un-abliterated on the attention path.
|
||
2. **A multi-token-prediction (MTP) head** (`mtp.layers.0`) used for
|
||
speculative decode. The generic 64-layer loop never reaches it.
|
||
3. **A vision tower** (`model.visual.*`, 333 tensors) that must survive
|
||
untouched or the model stops being multimodal.
|
||
|
||
## The two things this recipe gets right
|
||
|
||
### 1. The MTP head is abliterated *in-band*
|
||
|
||
This is the finding that matters most to us, because our gen seat gates on MTP
|
||
acceptance ≳40% (`reference_abliteration_mtp_lessons`).
|
||
|
||
Most abliterations orthogonalize the trunk and leave `mtp.layers.0` untouched.
|
||
The consequence is subtle and nasty: **the draft head keeps proposing
|
||
refusal-prefix tokens that the abliterated trunk then rejects, so speculative
|
||
acceptance collapses on exactly the prompts abliteration exists to fix.** You
|
||
get a model that is abliterated *and* slow, and the slowness is worst precisely
|
||
where you wanted the behaviour change.
|
||
|
||
The fix is to orthogonalize the MTP block's **two residual-write matrices**
|
||
(`self_attn.o_proj`, `mlp.down_proj`) with the *same* refusal direction as the
|
||
trunk. The MTP **glue** — `mtp.fc`, `mtp.norm`, `mtp.pre_fc_norm_*` — is left
|
||
alone, because those are norms and an input projection, **not** residual
|
||
writers. Editing them would corrupt the draft path without removing any refusal.
|
||
|
||
### 2. The vision tower is preserved byte-identical
|
||
|
||
All 333 `model.visual.*` tensors pass through unmodified — verified by direct
|
||
tensor diff (max delta `0.000000`), not asserted. An `mmproj` is published so
|
||
the vision half is actually usable, not just nominally intact.
|
||
|
||
## The edit set (131 tensors)
|
||
|
||
Single-direction weight orthogonalization, Arditi et al. style, applied to every
|
||
matrix that writes the residual stream:
|
||
|
||
| scope | tensor | count |
|
||
|---|---|---|
|
||
| `model.language_model.layers.*` (64) | `mlp.down_proj` | 64 |
|
||
| | `linear_attn.out_proj` (DeltaNet) | 48 |
|
||
| | `self_attn.o_proj` (full-attn, interval 4) | 16 |
|
||
| `mtp.layers.0` | `o_proj` + `down_proj` | 2 |
|
||
| `model.language_model` | `embed_tokens` | 1 |
|
||
| **edited total** | | **131** |
|
||
| `model.visual.*` | preserved byte-identical | 333 |
|
||
|
||
**Hard coverage gate before writing a byte:**
|
||
`o_proj(16) + linear_out(48) == 64 == num_hidden_layers`. This is the check that
|
||
catches a partial tensor-name match — the failure mode that otherwise ships a
|
||
quietly half-abliterated model that passes a smoke test and fails in the field.
|
||
Adopt this gate in any re-derivation.
|
||
|
||
## Two calibration traps specific to this base
|
||
|
||
### Refusal-direction selection
|
||
|
||
The direction was captured **twice**, from two structurally different
|
||
chat-template renderings:
|
||
|
||
- one with `enable_thinking=false`
|
||
- one with thinking on at `reasoning_effort=xhigh` (which injects an extra
|
||
system block and shifts every token position)
|
||
|
||
The two agree at **|cos| 0.96–0.99 across layers 18–45, peaking 0.9925 at layer
|
||
26** — the layer used. Two different prompt distributions converging on the same
|
||
vector is the evidence that the direction encodes *refusal semantics* rather than
|
||
*template formatting*. A single-template capture cannot distinguish the two.
|
||
|
||
> ⚠️ **Two-template agreement is a bad LAYER SELECTOR on a heavily-merged base —
|
||
> use harmful/harmless SEPARATION instead (added 2026-08-20).** On RobinsonLabs'
|
||
> stock Qwen3.8 the agreement was 0.99 and picking its peak was fine. On DavidAU's
|
||
> Cold-Fusion GAIN merge the same metric tops out at **0.62**, and its argmax
|
||
> (layer 18) is the layer with the **worst** refusal separation in the window
|
||
> (Cohen's d 5.51 vs 9.89 at the peak) — abliterating there was a measured
|
||
> behavioral **no-op**. The reason: the two renderings end in different generative
|
||
> modes (`</think>\n\n` = about to answer vs `<think>\n` = about to reason), so
|
||
> `|cos|` scores refusal *plus* mode, and on a merge the mode term dominates. The
|
||
> selector that actually predicts efficacy is **how cleanly the direction splits
|
||
> harmful from harmless prompt activations** (Cohen's d / AUC), gated on the sink
|
||
> screen (separation and sink-energy both rise with depth, so the raw peak is
|
||
> usually sink-dominated). On Cold-Fusion this picked **layer 35** (d 9.35, AUC
|
||
> 0.9997, sink 0.094%) and the abliteration worked. Keep agreement as a
|
||
> diagnostic; do not select on it. See
|
||
> `services/coldfusion-abliteration/README.md`.
|
||
|
||
### The attention-sink dimension — the one that bricks the model
|
||
|
||
**Qwen3.8-27B's massive-activation dimension is `3994`.** It carries 19–21% of
|
||
the direction's energy at layers 1–3, and orthogonalizing it out of every
|
||
residual writer produces a model that **loads, runs, and emits garbage.** Layer
|
||
26 was chosen partly because it carries only **0.06%** of its energy in dim 3994.
|
||
|
||
**Any re-derivation MUST screen for this.** It is the single most likely way to
|
||
waste a GPU afternoon on this architecture and mistake the result for a failed
|
||
abliteration when it is actually an attention-sink blowout.
|
||
|
||
## Measured behaviour (their numbers, for reference)
|
||
|
||
Base vs abliterated, same session/harness/prompts, both at Q4_K_M:
|
||
|
||
| prompt set | base | abliterated |
|
||
|---|---|---|
|
||
| in-distribution (24, from capture set) | 96% (23/24) | **8%** (2/24) |
|
||
| held-out (40, disjoint, overlap=0) | 100% (40/40) | **8%** (3/40) |
|
||
|
||
Capability axes (reasoning / code / math / factual / instruction-following /
|
||
creative-RP coherence): **no regression on any axis.** Held-out train/test split
|
||
was 416/104 with overlap 0, so the 8% held-out figure is generalization, not a
|
||
reshuffle of calibration prompts.
|
||
|
||
**Note the design point:** 8% is deliberate. Harm guardrails are **retained** —
|
||
self-harm prompts still redirect (988) rather than comply. This is a
|
||
*creative-content* abliteration shipped "at the ceiling where capability and
|
||
guardrails both survive," explicitly **not** a jailbreak. That makes it a
|
||
**milder** abliteration than our incumbent gen seat (`absolute-heresy`, ~2%
|
||
author refusals, aggressive Heretic). Adopt the *method* here; the *ceiling* is a
|
||
separate call.
|
||
|
||
## How this maps onto our pipeline
|
||
|
||
The recipe is a drop-in for the front half of the House quant pipeline:
|
||
|
||
1. Pull bf16 master to NFS (verify repo id first —
|
||
`reference_verify_hf_repo_ids_before_pull`).
|
||
2. **Baseline MTP acceptance on bf16 before any surgery** — the standing rule.
|
||
3. Orthogonalize per the edit set above; enforce the coverage gate; screen dim
|
||
3994; gate the result on **MTP acceptance ≳40%, not KL** (KL misled us once —
|
||
`reference_abliteration_mtp_lessons`).
|
||
4. Verify vision byte-identical, refusals down, PPL not blown, no catatonia.
|
||
**Measure first-token KL as a *fidelity* number** (`kl_divergence.py`,
|
||
bf16-vs-bf16, held-out prompts) — it does not replace the acceptance gate in
|
||
step 3, and it is not a pass/fail on its own. Report it **split by prompt
|
||
class**: a single averaged KL over a mixed corpus is close to meaningless,
|
||
because the metric is supposed to be large on harmful prompts and small on
|
||
benign ones. The ratio is the interesting quantity. Cold-Fusion L35 measured
|
||
**0.0211 median harmless / 0.5996 median harmful = 28.4× selectivity**, on a
|
||
stack whose self-KL noise floor is exactly 0.0.
|
||
5. NVFP4-quantize in-house (mixed W4A4 + FP8-attn/lm_head —
|
||
`model-quantization-playbook.md`). **Foot-gun the GGUF card itself flags:
|
||
the imatrix does not cover the MTP block** — so a GGUF requant path leaves
|
||
MTP uncalibrated. Our NVFP4 path must calibrate it explicitly.
|
||
|
||
## Provenance
|
||
|
||
- Recipe: RobinsonLabs README, fetched verbatim 2026-08-19. Authored with their
|
||
"ModelForge" manufacturing system-of-record (not public).
|
||
- Method lineage: Arditi et al., single-direction refusal orthogonalization.
|
||
- Our prior art: `reference_abliteration_mtp_lessons` (modest abliteration
|
||
preserves MTP; test MTP on bf16 first; gate on acceptance not KL), and the
|
||
gen-seat quant recipe in `model-quantization-playbook.md`.
|