feat(coldfusion-abliteration): THESIS PROVEN — in-band-abliterated MTP head accepts 59.1% (beats incumbent ~47%)

Quantized the L35 abliterated model to mixed NVFP4 and measured MTP acceptance
end to end. The experiment's whole premise: Heretic (the incumbent gen seat)
leaves the MTP head a byte-identical base graft its wrapper never loads, whereas
Robinson abliterates the MTP head in-band — the question was whether that in-band
edit survives well enough to spec-decode. It does, better than the graft:

  MTP acceptance  59.1% median (51-65%, 8 cache-busted topics)  vs incumbent ~47%
  decode          118.7 tok/s median (faster; image-confounded, read as not-worse)
  abliteration    survives quant (creative refusals drop, self-harm guardrail
                  intact, coherent)

Output at /tank/aimodels/qwen38-27b-coldfusion-L35-nvfp4-mixed (22.5 GB). Result
JSON in bench/. NOT cut over — the incumbent seat is untouched; making L35 the gen
seat is a separate decision needing the full Stage-3 gate + real multi-turn hold.

Two env foot-guns hardened along the way:
- quant_mixed_nvfp4.py now promotes text_config attention fields
  (num_attention_heads etc.) to the top-level config for the oneshot, then
  restores. transformers 5.10 / llmcompressor 0.12 (this venv moved under us
  since the Aug-15 heresy quant) no longer delegate the top-level lookup, so
  oneshot raised "Cannot determine num_attention_heads". Same "the fight is the
  environment" pattern as the abliteration capture.
- a sub-~23GB quant saves as a single model.safetensors with no index, so the
  post_quant MTP graft needed an index built first — from the safetensors header,
  not safe_open (which mmaps the whole shard and ENOMEMs on ZFS).

post_quant grafted the abliterated MTP (15 tensors, 849 MB) and re-injected
re:^mtp.* into quantization_config.ignore (llm-compressor pruned it again — the
two-rounds-lost 0%-MTP bug, fired and repaired as designed). Probe served on the
pinned nightly (#51113 qwen3_5_mtp fix) to match the live seat's vLLM.
This commit is contained in:
vh
2026-08-20 10:18:48 -07:00
parent c55b1390b7
commit 725c8fdf9e
3 changed files with 100 additions and 0 deletions
@@ -209,6 +209,49 @@ guess-and-retry. Agreement is still computed and printed, as a diagnostic.
> problem was never the calibration set. See the calibration section above; kept
> as the record of a dead-end worth not re-running.
## ✅ THESIS RESULT — the in-band-abliterated MTP head accepts BETTER than a graft (2026-08-20)
The whole reason to abliterate Cold-Fusion ourselves rather than run the incumbent
Heretic seat: Heretic leaves the MTP head a **byte-identical base graft** (its
wrapper never loads it), while the Robinson formula abliterates the MTP head
**in-band** (its 2 residual writers). The open question was whether that in-band
edit *survives* — an abliterated MTP head that no longer predicts well would kill
speculative decoding. Measured, end to end:
| metric | L35 quant | incumbent (heresy) | gate | verdict |
|---|---|---|---|---|
| **MTP acceptance** (median, 8 cache-busted topics) | **59.1%** (51–65%) | ~47% | ≳40% | **PASS — beats incumbent** |
| decode tok/s (median) | 118.7 | ~95–103 | ≥ incumbent | faster (⚠ image-confounded, read as "not worse") |
| abliteration survives quant | yes | — | creative↓, self-harm intact | **PASS** |
| coherence / no catatonia | clean | — | eyeball | **PASS** |
So the in-band MTP abliteration doesn't merely preserve speculative decoding — the
abliterated head **accepts 59.1% vs the untouched graft's ~47%.** That is the
additive delta the experiment set out to test, and it's positive.
**Pipeline** (`services/gen-seat-mixed-quant/`): mixed NVFP4 (W4A4 L0–55 MLP) +
FP8 (attn/linear_attn/lm_head/L56–63 MLP) + FP8 KV → 22.5 GB. Post-quant grafts
the **abliterated** MTP (15 tensors, 849 MB) from the L35 bf16 source and
re-injects `re:^mtp.*` into `quantization_config.ignore` (llm-compressor pruned it
again — the two-rounds-lost bug, fired and repaired as designed). Output:
`/tank/aimodels/qwen38-27b-coldfusion-L35-nvfp4-mixed`. Result JSON:
`services/gen-seat-mixed-quant/bench/mtp_coldfusion_L35.json`.
> ⚠️ Env foot-gun banked: the quant venv's `transformers` moved to 5.10 /
> `llmcompressor` 0.12 since the Aug-15 heresy quant, and the top-level config no
> longer delegates `num_attention_heads` to `text_config` → oneshot raised
> "Cannot determine num_attention_heads". `quant_mixed_nvfp4.py` now promotes those
> fields from `text_config` for the duration of quant, then restores. Also: a
> small (<~23 GB) quant saves as a **single** `model.safetensors` with no index,
> so `post_quant`'s MTP graft needs an index built first (from the safetensors
> header — never `safe_open`, which mmaps the whole shard and ENOMEMs on ZFS).
**NOT cut over.** The incumbent gen seat is untouched. Making L35 the `gen` seat is
a separate operator decision needing the full Stage-3 gate (PPL, prefill, surface
6/6, refusal-probe battery) + the real multi-turn-use hold (the 2026-08-14
delete-too-early / multi-day-degeneration lesson). The thesis is proven; the
cutover is a distinct call.
## Why the write is shard surgery, not `model.save_pretrained`
The `--out` path edits the 18 safetensors shards directly and never instantiates