# [2026-08-15] Uncensored gen seat: Qwen3.8-27B-Uncensored deployed; the definitive MTP-graft fix

**Outcome.** The fleet `gen` seat is now **`JonathanColetti/Qwen3.8-27B-Uncensored`** (Heretic
abliteration, KL 0.12 vs base, bench Δ −0.5 within noise, refusals 98→12/100), quantized in-house
to **NVFP4 W4A16** (llm-compressor / compressed-tensors) with a **grafted bf16 MTP head**,
vision-intact, **262K** ctx, MTP n=3 (**~42% accept, ~68 tok/s**), coherent. Live at ana-ml2 `:8015`
(project `gen-seat` / container `vllm-gen`), backing all 7 gateway aliases.

**THE definitive lesson (resolved 3 failed attempts + one premature 50 GB delete).** A grafted bf16
MTP scored **0% on the quant but 83% at bf16** — for TWO different abliterated models. Root cause was
NEITHER the abliteration NOR the quant scheme: it was **the grafted `mtp.*` tensors missing from
`config.json` → `quantization_config.ignore`.** The wrapper-class quant DROPS the MTP before
llm-compressor sees it, so nothing gets added to `ignore`; vLLM then tries to load the bf16 MTP as
*quantized* format → "Parameter … not found in params_dict, skip loading" → uninitialized head → 0%.
**FIX: after grafting, add `re:^mtp.*` to `quantization_config.ignore`** (one line — all unsloth's
working checkpoint has). MTP jumped 0%→83% (bf16-identical). Full lesson in auto-memory
`reference_abliteration_mtp_lessons`.

**The pipeline that works (for the next VL+MTP quant, incl. the W4A8 chase):**
1. Pull bf16 (kept at `ana-ml2:/tank/aimodels/qwen38-27b-uncensored-bf16`).
2. Quant via `quant_nvfp4_qwen.py` (darkscarlett dir) = the **wrapper-class** loader
   (`Qwen3_5ForConditionalGeneration`, keeps the vLLM-serveable config); container = `vllm-openai`
   + `pip install llmcompressor==0.13.0` (drags in a transformers with `qwen3_5`).
3. **Graft** the author's `model-mtp.safetensors` verbatim into the output + merge the index.
4. **Reconstruct** `preprocessor_config.json` from `processor_config.json`'s `image_processor`
   sub-dict (the repo omits it → else "Can't load image processor" crash-loop).
5. **Add `re:^mtp.*` to the output config's `quantization_config.ignore`.** ← the fix.
6. Serve: `--quantization compressed-tensors --speculative-config '{"method":"qwen3_5_mtp","num_speculative_tokens":3}'`
   `--mamba-cache-dtype float32 --kv-cache-dtype fp8 --reasoning-parser qwen3`.

**VRAM / full-context budget (measured).** Weights ~27 GB; hybrid attention → **only 16 of 64 layers
carry KV** → 32 KiB/token → **262K KV = 8.6 GB** (vs ~60–70 GB for a normal dense 27B). Full 262K fits
GPU0 at **util 0.45** (~43 GB) alongside meromero (~49 GB used, it's a 31B) — pre-flight rejects util
0.48 (wants 45.6 GB, only 45.5 free). `max-num-seqs 16` keeps cudagraph modest (an ad-hoc serve with
no cap OOM'd — cudagraph captured to batch-512).

**Why unsloth's `qwen3.8-27b` (the prior gen model) was faster (97 vs 68 tok/s).** ~half = quant kernel
(unsloth native NVFP4+FP8 tensor cores vs our W4A16 → Marlin dequant, ~20% even on decode — I'd
under-stated this); ~half = MTP acceptance (unsloth 55% un-ablated head vs our 42% — inherent to the
ablation, no quant fixes it). **W4A8 recovers the first ~20% (→~82 tok/s) + prefill; not the MTP half.**

**modelopt dead-end (for W4A8, avoid).** `nvidia-modelopt[hf]==0.43.0` is too old for qwen3_5's
transformers: (a) its `NVFP4_DEFAULT_CFG.quant_cfg` is a LIST but 0.43 wants a DICT (pydantic reject);
(b) it warns transformers 5.15 untested. Use **llm-compressor** for W4A8 instead (custom recipe: NVFP4
weights + FP8 input_quantizer + calibration on `heretic2-nvfp4-work/production_calib_512.jsonl`).

**Deleted (premature — the delete I owned).** `windowsxp811203/Qwen3.8-27B-Abliterated` (~79 GB) — I
declared it desync-dead off a 0% that was actually this ignore bug. Lesson: **test MTP on bf16 first;
isolate before deleting.**

Commits: eshpfi `680c30e` (deploy + rename + litellm + README), dotfiles `1d1970f` (CLAUDE.md roster) —
both UNPUSHED. Related: [[reference_abliteration_mtp_lessons]], [[reference_verify_hf_repo_ids_before_pull]].