esh-pfi-infrastructure/persistent-memory.d/2026-08-15-uncensored-gen-seat.md

[2026-08-15] Uncensored gen seat: Qwen3.8-27B-Uncensored deployed; the definitive MTP-graft fix

Outcome. The fleet gen seat is now JonathanColetti/Qwen3.8-27B-Uncensored (Heretic abliteration, KL 0.12 vs base, bench Δ −0.5 within noise, refusals 98→12/100), quantized in-house to NVFP4 W4A16 (llm-compressor / compressed-tensors) with a grafted bf16 MTP head, vision-intact, 262K ctx, MTP n=3 (~42% accept, ~68 tok/s), coherent. Live at ana-ml2 :8015 (project gen-seat / container vllm-gen), backing all 7 gateway aliases.

THE definitive lesson (resolved 3 failed attempts + one premature 50 GB delete). A grafted bf16 MTP scored 0% on the quant but 83% at bf16 — for TWO different abliterated models. Root cause was NEITHER the abliteration NOR the quant scheme: it was the grafted mtp.* tensors missing from config.json → quantization_config.ignore. The wrapper-class quant DROPS the MTP before llm-compressor sees it, so nothing gets added to ignore; vLLM then tries to load the bf16 MTP as quantized format → "Parameter … not found in params_dict, skip loading" → uninitialized head → 0%. FIX: after grafting, add re:^mtp.* to quantization_config.ignore (one line — all unsloth's working checkpoint has). MTP jumped 0%→83% (bf16-identical). Full lesson in auto-memory reference_abliteration_mtp_lessons.

The pipeline that works (for the next VL+MTP quant, incl. the W4A8 chase):

1. Pull bf16 (kept at ana-ml2:/tank/aimodels/qwen38-27b-uncensored-bf16).
2. Quant via quant_nvfp4_qwen.py (darkscarlett dir) = the wrapper-class loader (Qwen3_5ForConditionalGeneration, keeps the vLLM-serveable config); container = vllm-openai
   • pip install llmcompressor==0.13.0 (drags in a transformers with qwen3_5).
3. Graft the author's model-mtp.safetensors verbatim into the output + merge the index.
4. Reconstruct preprocessor_config.json from processor_config.json's image_processor sub-dict (the repo omits it → else "Can't load image processor" crash-loop).
5. Add re:^mtp.* to the output config's quantization_config.ignore. ← the fix.
6. Serve: --quantization compressed-tensors --speculative-config '{"method":"qwen3_5_mtp","num_speculative_tokens":3}' --mamba-cache-dtype float32 --kv-cache-dtype fp8 --reasoning-parser qwen3.

VRAM / full-context budget (measured). Weights ~27 GB; hybrid attention → only 16 of 64 layers carry KV → 32 KiB/token → 262K KV = 8.6 GB (vs ~60–70 GB for a normal dense 27B). Full 262K fits GPU0 at util 0.45 (~43 GB) alongside meromero (~49 GB used, it's a 31B) — pre-flight rejects util 0.48 (wants 45.6 GB, only 45.5 free). max-num-seqs 16 keeps cudagraph modest (an ad-hoc serve with no cap OOM'd — cudagraph captured to batch-512).

Why unsloth's qwen3.8-27b (the prior gen model) was faster (97 vs 68 tok/s). ~half = quant kernel (unsloth native NVFP4+FP8 tensor cores vs our W4A16 → Marlin dequant, ~20% even on decode — I'd under-stated this); ~half = MTP acceptance (unsloth 55% un-ablated head vs our 42% — inherent to the ablation, no quant fixes it). W4A8 recovers the first ~20% (→~82 tok/s) + prefill; not the MTP half.

modelopt dead-end (for W4A8, avoid). nvidia-modelopt[hf]==0.43.0 is too old for qwen3_5's transformers: (a) its NVFP4_DEFAULT_CFG.quant_cfg is a LIST but 0.43 wants a DICT (pydantic reject); (b) it warns transformers 5.15 untested. Use llm-compressor for W4A8 instead (custom recipe: NVFP4 weights + FP8 input_quantizer + calibration on heretic2-nvfp4-work/production_calib_512.jsonl).

Deleted (premature — the delete I owned). windowsxp811203/Qwen3.8-27B-Abliterated (~79 GB) — I declared it desync-dead off a 0% that was actually this ignore bug. Lesson: test MTP on bf16 first; isolate before deleting.

Commits: eshpfi 680c30e (deploy + rename + litellm + README), dotfiles 1d1970f (CLAUDE.md roster) — both UNPUSHED. Related: reference_abliteration_mtp_lessons, reference_verify_hf_repo_ids_before_pull.