vh
5ee2325820
Answers "how likely is it that our abliteration caused this?" with a measurement instead of a prior. P(<think>) at the first generated token, template rendered enable_thinking=false so the prompt already carries a CLOSED think pair -- the exact event behind the leak. Raw softmax, bf16, CPU-only, one process per model. Deterministic: stock reproduced to 17 significant figures across two runs. coldfusion-bf16 none (stock) 0.1850 rank 3 coldfusion-abliterated-L35-bf16 Robinson L35, mild 0.2048 rank 2 coldfusion-h300-mtp-bf16 Heretic-300, heavy 0.2216 rank 2 The stock, untouched base already puts 18.5% of first-token mass on opening a think block the template had closed. Abliteration adds a real, monotonic, dose-dependent +3.7 points -- a nudge on a pre-existing base, not the cause. Cold-Fusion is a reasoning-token-compression finetune, i.e. a model trained to think briefly, and the leak's text shape agrees: a compact correct trace with a trained transition marker, which is trained behavior rather than damage. This changes the options. Rolling back to L35 or stock does NOT fix the leak -- at 18.5% under temp 0.7 / top_p 0.8 they leak at nearly the h300 rate. Only leaving the Cold-Fusion family escapes it, at the cost of the 8/100 refusal result. The chat_template_kwargs fix is the correct lever. Durable methodology point: a forward-KL budget cannot catch this. Heretic minimizes forward KL(stock||abliterated), which is near-blind to the model putting new mass on tokens stock barely used -- that is reverse KL's job, and we measured exactly that asymmetry on L35 (reverse 1.43 vs forward 0.70). h300's KL of 0.0136 is not evidence of innocence. For any "did the abliteration break behavior X" question, measure P(token) directly. Ran CPU-only deliberately: 96 EPYC cores and 265 GB of RAM make a 27B forward pass cheap, so this cost no GPU window and no seat downtime, where the obvious route was stopping both GPU0 seats. Also normalizes two more abliteration output dirs from root-owned 0600 to llmuser 0664. The unreadable-model failure surfaces as FileNotFoundError rather than a permission error, which is worth knowing before it wastes a run.
gen-seat mixed-precision quant — NVFP4 W4A4 MLP + FP8 W8A8 attention
The pipeline that produced /tank/aimodels/qwen38-27b-uncensored-nvfp4-mixed, the
current fleet gen seat on ana-ml2 GPU0 :8015. +18% decode over the previous
weight-only NVFP4A16 build, at equal MTP acceptance and +1.7% perplexity.
General lessons live in
docs/pfi/model-quantization-playbook.md— read that before starting a quant on a different model. This file is the worked example for Qwen3.8-27B-Uncensored specifically. Where the two disagree, the playbook wins.
The headline correction: "W4A8" is not a thing you can serve
The queued task was "re-quant to NVFP4 weights + FP8 activations (W4A8) for ~20%
more decode". That checkpoint cannot load. vLLM 0.24's compressed-tensors
dispatcher (compressed_tensors.py:704-713) allows NVFP4 weights with exactly two
activation options:
| input_activations | scheme | kernel |
|---|---|---|
None |
W4A16 | Marlin (forced — kernels/linear/__init__.py:881-883) |
| NVFP4 | W4A4 | native Blackwell FP4 |
anything else — FP8 included — raises
ValueError: For NVFP4 weights, input quantization must also be NVFP4 format,
None for NVFP4A16
CompressedTensorsW4A8Fp8 exists but is INT4 weights (W4A8_SUPPORTED_TYPES_MAP = {4: int4}) gated on _check_scheme_supported(90, match_exact=True) — Hopper only.
ana-ml2 is Blackwell (sm_120), so that path is doubly closed.
The ~20% intuition was right; the scheme name was wrong. The servable way to get FP8
into the mix is per-layer-group, which is exactly what unsloth/Qwen3.8-27B-NVFP4
does — and that build, measured on-box, ran +19.1% faster than ours at identical MTP
acceptance. This pipeline replicates its recipe on the abliterated weights.
The recipe
| group | scheme | targets |
|---|---|---|
group_0 |
FP8 W8A8 — channel weights (static), per-token dynamic activations | self_attn.{q,k,v,o}_proj, linear_attn.{in_proj_qkv,in_proj_z,out_proj}, lm_head, layers 56-63 MLPs |
group_1 |
NVFP4 W4A4 — tensor_group gsize16, fp8 scales, imatrix_mse weights, dynamic:"local" activations |
layers 0-55 MLP {gate,up,down}_proj |
| kv cache | FP8 static tensor | — |
| ignored | vision tower, linear_attn.{norm,in_proj_a,in_proj_b}, re:^mtp.* |
— |
Keeping the last 8 layers' MLPs at FP8 is the accuracy-preservation trick — late
layers are the sensitive ones. conv1d in linear_attn is not a Linear and stays BF16
in both our build and unsloth's.
Targets are deliberately non-overlapping (group_1 enumerates layers 0-55 rather
than matching all MLPs) instead of relying on group precedence to resolve the 56-63
collision. validate_targets.py proves this against the real module names before any
GPU time is spent — run it first.
Running it
# 0. prove the regexes hit what you think (free, no GPU)
python3 validate_targets.py /tank/aimodels/qwen38-27b-uncensored-bf16
# -> expect OVERLAP 0, MLP layer union covers 0-63 True
# 1. quant (~20 min on one Blackwell; needs llmcompressor, NOT modelopt)
python3 quant_mixed_nvfp4.py \
--model /tank/aimodels/qwen38-27b-uncensored-bf16 \
--calib /tank/aimodels/heretic2-nvfp4-work/production_calib_512.jsonl \
--out /tank/aimodels/qwen38-27b-uncensored-nvfp4-mixed \
--num-samples 256 --seqlen 2048
# 2. MANDATORY post-steps — graft MTP, restore preprocessor, repair the mtp ignore
python3 post_quant.py <bf16-source> <out-dir>
pip install llmcompressor into the stock vllm/vllm-openai:latest image gives
llmcompressor 0.13.0 + compressed-tensors 0.18.0 without disturbing torch or
transformers. Do not use modelopt 0.43 — dependency hell on qwen3_5.
The foot-gun that has now cost three rounds
llm-compressor prunes ignore entries that matched no module at quant time.
The wrapper class (Qwen3_5ForConditionalGeneration) never loads the MTP head, so
re:^mtp.* matches nothing and is silently dropped from the saved config. vLLM then
treats the freshly grafted BF16 MTP head as quantized, brings it up uninitialised,
and speculative decoding runs at 0% acceptance.
post_quant.py re-injects the entry after the graft and re-verifies. It is not
optional, and it verifies rather than assumes — that check fired on this very run.
Acceptance gate (bench/)
Speed alone does not justify cutting over a seat backing 7 LiteLLM aliases.
| metric | W4A16 (old) | mixed (new) | delta |
|---|---|---|---|
| decode tok/s, bs=1, cache-busted | 80.12 | 94.53 | +18.0% |
| prefill tok/s, ~6.7k prompt | 3,206 | 6,334 | +98% |
| prefill tok/s, ~27k prompt | 2,862 | 5,085 | +78% |
| TTFT on a ~27k-token doc | 9.43 s | 5.31 s | −44% |
| MTP acceptance | 47.8% | 47.7% | unchanged |
| perplexity, 6 held-out passages | 6.941 | 7.059 | +1.7% worse |
| abliteration compliance | 4/4 | 4/4 | preserved |
| weights on disk | 27.7 GB | 22.5 GB | −19% |
Prefill roughly doubled — the bigger practical win, and exactly what theory predicts:
decode at bs=1 is memory-bandwidth-bound (weights are 4-bit either way, so little changes),
while prefill is compute-bound and is where Blackwell's native FP4 tensor cores replace the
Marlin dequant-to-BF16 path. This is what the summarizer / summarizer-large aliases feel
on long documents.
quickbench.py— cache-busted bs=1 decode + MTP acceptance. Bust the cache: with a fixed prompt, prefix caching returns byte-identical timings and you measure nothing.prefill_bench.py— TTFT on long prompts. Same trap, worse: a seeded nonce reproduces the previous run's prompts verbatim, so prefix caching serves them and you read ~41k tok/s of cache-hit instead of ~5k of real prefill. UsesSystemRandom; never seed it.eval_quality.py— perplexity, deterministic generations, abliteration survival. PPL must be measured with--speculative-configOFF: under MTP, vLLM'sprompt_logprobscome back ~uniform over the vocab (median rank ~10^5, logprob ≈ log(1/vocab)). The harness raises rather than reporting the garbage.surface_test.py— the real gate: plain chat, vision, tool calling, thinking split, 36K-token needle retrieval, streaming. All six must pass before a cutover.serve_probe.sh <model-dir> [nospec]— serve a candidate on:8017without touching the live seat.
GPU0 budget
The mixed build's weights are 5.2 GB smaller. At the old GEN_GPU_MEM_UTIL=0.45 the
seat absorbed that slack as extra KV (17.0 GiB / 477K tokens) and left meromero-charrp
0.18 GiB short of its 0.52 budget — it crash-looped on startup. Fixed by handing the
space back: GEN_GPU_MEM_UTIL=0.43 → 15.1 GiB / 422K tokens, still 1.6× the 262K
context. Both seats co-resident at 89.8 / 97.9 GB.
Levers already measured — do not re-chase
GEN_SPEC_TOKENS swept on this seat: n=2 → 77.1, n=3 → 80.1, n=4 → 78.7,
n=5 → 75.9 tok/s. Three is the optimum; higher n trades acceptance for draft width and
loses.
Rollback
The previous build is untouched at /tank/aimodels/qwen38-27b-uncensored-nvfp4.
ssh infra-ops@10.250.50.54
sudo cp /opt/docker/compose/gen-seat/.env.bak-w4a16-20260815 /opt/docker/compose/gen-seat/.env
cd /opt/docker/compose/gen-seat && sudo docker compose up -d vllm-gen
⚠ gen-seat/.env is mode 0600 and lkraven-owned — every docker compose call
against it needs sudo, or compose fails with permission denied reading .env,
leaves the old container running, and the change silently does not take.