feat(coldfusion-abliteration): abliteration LANDS at layer 35 — separation selector, shard-surgery write, three false diagnoses corrected

The abliterated model works. A/B vs stock on a matched greedy battery: explicit
sexual + graphic torture (the measured stock refusal surface) go from refused to
complied/engaged, held-out AdvBench prompts loosen, the self-harm guardrail
survives, coherence intact — the Robinson design point exactly. Output at
/tank/aimodels/qwen38-27b-coldfusion-abliterated-L35-bf16, verified bitwise:
131/131 targets changed, 333/333 vision byte-identical (delta 0.0), 735/735
others untouched.

Getting there corrected three diagnoses the prior session had backwards.

1. The layer-selection metric was wrong, and that was the whole ballgame. The
   recipe picks the abliteration layer by peak two-template |cos| agreement. On
   this heavily-merged base that metric is anti-correlated with efficacy: its
   argmax (layer 18) is the WORST-separating layer in the window (Cohen's d 5.51
   vs 9.89 at the peak), and abliterating there was a measured behavioral no-op —
   stock and "abliterated" refused all six probes identically. Cause: the two
   renderings end in different generative modes (</think> vs <think>), so |cos|
   scores answer-vs-reason mode, not refusal, and on a merge the mode term
   dominates. Replaced selection with harmful/harmless SEPARATION (Cohen's d /
   AUC of the direction's projection), gated on the sink screen since separation
   and sink-energy both climb with depth. Picks layer 35 (d 9.35, AUC 0.9997,
   sink 0.094%). Agreement is kept as a printed diagnostic.

2. The "bf16 NaNs, use fp32" rule was a misdiagnosis. The NaN was never
   precision — it was multi-GPU sharding (the residual stream zeroes two layers
   past the GPU0->GPU1 boundary; the first capture's layer 22 happened to sit in
   the healthy region, which is why it looked fine) plus
   PYTORCH_CUDA_ALLOC_CONF=expandable_segments (corrupts retained tensors; the
   corruption MOVED between bit-identical forwards, the tell that it was memory
   not math). On one GPU with a plain allocator, bf16 full-64-layer is exactly
   deterministic and coherent, at 50 GB and 4.3x the throughput of the 111 GB
   fp32 it replaced. Both defects are now hard gates (residency exit 8, allocator
   exit 9); capture pins CUDA_VISIBLE_DEVICES=0.

3. The corpus-size hypothesis was falsified. 52x more calibration data (8->416,
   mlabonne/harmful_behaviors = the recipe's actual AdvBench split, already on the
   box) moved agreement 0.594->0.624 — nothing. Kept the 416/416 corpus anyway
   (calibration.py); it gives the clean separation signal. The held-out 104-prompt
   test split is reserved and asserted disjoint.

Also: the --out write is now shard-level surgery (reads/writes the 18 safetensors
directly, no model object, no GPU). This is correctness, not thrift —
AutoModelForCausalLM resolves to the TEXT model, so save_pretrained would drop all
333 vision tensors AND skip the MTP head (the in-band MTP edit is the entire point
of the Robinson formula). Neither failure raises. Shard surgery makes vision and
the other 1068 tensors byte-identical by construction.

Batched capture with a dtype-aware equivalence gate; hidden states captured via
forward pre-hook (reading output_hidden_states off the returned object is unsafe
here — buffers get recycled). Sharding/allocator lessons promoted to the
quantization playbook (model-agnostic, sections 3.9-3.11 + superseded table); the
selection-metric lesson added to the recipe doc.

The dead layer-18 no-op checkpoint was removed (52 GB, confirmed identical to
stock). Incumbent gen seat untouched. Full canonical refusal-probe re-profile and
MTP-acceptance-on-quant still owed before this becomes a gen-seat candidate.
This commit is contained in:
vh
2026-08-20 08:46:21 -07:00
parent f714f28195
commit e9dbc8660b
4 changed files with 667 additions and 207 deletions
+17
View File
@@ -93,6 +93,23 @@ The two agree at **|cos| 0.96–0.99 across layers 18–45, peaking 0.9925 at la
vector is the evidence that the direction encodes *refusal semantics* rather than
*template formatting*. A single-template capture cannot distinguish the two.
> ⚠️ **Two-template agreement is a bad LAYER SELECTOR on a heavily-merged base —
> use harmful/harmless SEPARATION instead (added 2026-08-20).** On RobinsonLabs'
> stock Qwen3.8 the agreement was 0.99 and picking its peak was fine. On DavidAU's
> Cold-Fusion GAIN merge the same metric tops out at **0.62**, and its argmax
> (layer 18) is the layer with the **worst** refusal separation in the window
> (Cohen's d 5.51 vs 9.89 at the peak) — abliterating there was a measured
> behavioral **no-op**. The reason: the two renderings end in different generative
> modes (`</think>\n\n` = about to answer vs `<think>\n` = about to reason), so
> `|cos|` scores refusal *plus* mode, and on a merge the mode term dominates. The
> selector that actually predicts efficacy is **how cleanly the direction splits
> harmful from harmless prompt activations** (Cohen's d / AUC), gated on the sink
> screen (separation and sink-energy both rise with depth, so the raw peak is
> usually sink-dominated). On Cold-Fusion this picked **layer 35** (d 9.35, AUC
> 0.9997, sink 0.094%) and the abliteration worked. Keep agreement as a
> diagnostic; do not select on it. See
> `services/coldfusion-abliteration/README.md`.
### The attention-sink dimension — the one that bricks the model
**Qwen3.8-27B's massive-activation dimension is `3994`.** It carries 19–21% of