# gen-seat mixed-precision quant — NVFP4 W4A4 MLP + FP8 W8A8 attention

The pipeline that produced `/tank/aimodels/qwen38-27b-uncensored-nvfp4-mixed`, the
current fleet `gen` seat on ana-ml2 GPU0 `:8015`. **+18% decode** over the previous
weight-only NVFP4A16 build, at equal MTP acceptance and +1.7% perplexity.

> **General lessons live in [`docs/pfi/model-quantization-playbook.md`](../../docs/pfi/model-quantization-playbook.md)** —
> read that before starting a quant on a *different* model. This file is the worked example for
> Qwen3.8-27B-Uncensored specifically. Where the two disagree, the playbook wins.

## The headline correction: "W4A8" is not a thing you can serve

The queued task was "re-quant to NVFP4 weights + FP8 activations (W4A8) for ~20%
more decode". **That checkpoint cannot load.** vLLM 0.24's compressed-tensors
dispatcher (`compressed_tensors.py:704-713`) allows NVFP4 weights with exactly two
activation options:

| input_activations | scheme | kernel |
|---|---|---|
| `None` | W4A16 | **Marlin** (forced — `kernels/linear/__init__.py:881-883`) |
| NVFP4 | W4A4 | native Blackwell FP4 |

anything else — FP8 included — raises

```
ValueError: For NVFP4 weights, input quantization must also be NVFP4 format,
            None for NVFP4A16
```

`CompressedTensorsW4A8Fp8` exists but is **INT4** weights (`W4A8_SUPPORTED_TYPES_MAP
= {4: int4}`) gated on `_check_scheme_supported(90, match_exact=True)` — Hopper only.
ana-ml2 is Blackwell (sm_120), so that path is doubly closed.

The ~20% intuition was right; the scheme name was wrong. The servable way to get FP8
into the mix is **per-layer-group**, which is exactly what `unsloth/Qwen3.8-27B-NVFP4`
does — and that build, measured on-box, ran +19.1% faster than ours at identical MTP
acceptance. This pipeline replicates its recipe on the abliterated weights.

## The recipe

| group | scheme | targets |
|---|---|---|
| `group_0` | **FP8 W8A8** — channel weights (static), per-token dynamic activations | `self_attn.{q,k,v,o}_proj`, `linear_attn.{in_proj_qkv,in_proj_z,out_proj}`, `lm_head`, **layers 56-63** MLPs |
| `group_1` | **NVFP4 W4A4** — tensor_group gsize16, fp8 scales, `imatrix_mse` weights, `dynamic:"local"` activations | **layers 0-55** MLP `{gate,up,down}_proj` |
| kv cache | FP8 static tensor | — |
| ignored | vision tower, `linear_attn.{norm,in_proj_a,in_proj_b}`, `re:^mtp.*` | — |

Keeping the **last 8 layers' MLPs at FP8** is the accuracy-preservation trick — late
layers are the sensitive ones. `conv1d` in `linear_attn` is not a Linear and stays BF16
in both our build and unsloth's.

Targets are deliberately **non-overlapping** (`group_1` enumerates layers 0-55 rather
than matching all MLPs) instead of relying on group precedence to resolve the 56-63
collision. `validate_targets.py` proves this against the real module names before any
GPU time is spent — run it first.

## Running it

```bash
# 0. prove the regexes hit what you think (free, no GPU)
python3 validate_targets.py /tank/aimodels/qwen38-27b-uncensored-bf16
#    -> expect OVERLAP 0, MLP layer union covers 0-63 True

# 1. quant (~20 min on one Blackwell; needs llmcompressor, NOT modelopt)
python3 quant_mixed_nvfp4.py \
  --model /tank/aimodels/qwen38-27b-uncensored-bf16 \
  --calib /tank/aimodels/heretic2-nvfp4-work/production_calib_512.jsonl \
  --out   /tank/aimodels/qwen38-27b-uncensored-nvfp4-mixed \
  --num-samples 256 --seqlen 2048

# 2. MANDATORY post-steps — graft MTP, restore preprocessor, repair the mtp ignore
python3 post_quant.py <bf16-source> <out-dir>
```

`pip install llmcompressor` into the stock `vllm/vllm-openai:latest` image gives
llmcompressor 0.13.0 + compressed-tensors 0.18.0 without disturbing torch or
transformers. **Do not use modelopt 0.43** — dependency hell on `qwen3_5`.

## The foot-gun that has now cost three rounds

`llm-compressor` **prunes `ignore` entries that matched no module at quant time.**
The wrapper class (`Qwen3_5ForConditionalGeneration`) never loads the MTP head, so
`re:^mtp.*` matches nothing and is silently dropped from the saved config. vLLM then
treats the freshly grafted BF16 MTP head as quantized, brings it up **uninitialised,
and speculative decoding runs at 0% acceptance.**

`post_quant.py` re-injects the entry *after* the graft and re-verifies. It is not
optional, and it verifies rather than assumes — that check fired on this very run.

## Acceptance gate (`bench/`)

Speed alone does not justify cutting over a seat backing 7 LiteLLM aliases.

| metric | W4A16 (old) | mixed (new) | delta |
|---|---|---|---|
| decode tok/s, bs=1, cache-busted | 80.12 | **94.53** | **+18.0%** |
| **prefill tok/s, ~6.7k prompt** | 3,206 | **6,334** | **+98%** |
| **prefill tok/s, ~27k prompt** | 2,862 | **5,085** | **+78%** |
| TTFT on a ~27k-token doc | 9.43 s | **5.31 s** | −44% |
| MTP acceptance | 47.8% | 47.7% | unchanged |
| perplexity, 6 held-out passages | 6.941 | 7.059 | +1.7% worse |
| abliteration compliance | 4/4 | 4/4 | preserved |
| weights on disk | 27.7 GB | 22.5 GB | −19% |

**Prefill roughly doubled** — the bigger practical win, and exactly what theory predicts:
decode at bs=1 is memory-bandwidth-bound (weights are 4-bit either way, so little changes),
while prefill is compute-bound and is where Blackwell's native FP4 tensor cores replace the
Marlin dequant-to-BF16 path. This is what the `summarizer` / `summarizer-large` aliases feel
on long documents.

- `quickbench.py` — cache-busted bs=1 decode + MTP acceptance. **Bust the cache:** with
  a fixed prompt, prefix caching returns byte-identical timings and you measure nothing.
- `prefill_bench.py` — TTFT on long prompts. Same trap, worse: a *seeded* nonce reproduces
  the previous run's prompts verbatim, so prefix caching serves them and you read ~41k tok/s
  of cache-hit instead of ~5k of real prefill. Uses `SystemRandom`; never seed it.
- `eval_quality.py` — perplexity, deterministic generations, abliteration survival.
  **PPL must be measured with `--speculative-config` OFF**: under MTP, vLLM's
  `prompt_logprobs` come back ~uniform over the vocab (median rank ~10^5, logprob
  ≈ log(1/vocab)). The harness raises rather than reporting the garbage.
- `surface_test.py` — the real gate: plain chat, vision, tool calling, thinking split,
  36K-token needle retrieval, streaming. All six must pass before a cutover.
- `serve_probe.sh <model-dir> [nospec]` — serve a candidate on `:8017` without touching
  the live seat.

## GPU0 budget

The mixed build's weights are 5.2 GB smaller. At the old `GEN_GPU_MEM_UTIL=0.45` the
seat absorbed that slack as extra KV (17.0 GiB / 477K tokens) and left `meromero-charrp`
**0.18 GiB short** of its 0.52 budget — it crash-looped on startup. Fixed by handing the
space back: `GEN_GPU_MEM_UTIL=0.43` → 15.1 GiB / 422K tokens, still 1.6× the 262K
context. Both seats co-resident at **89.8 / 97.9 GB**.

## Levers already measured — do not re-chase

`GEN_SPEC_TOKENS` swept on this seat: n=2 → 77.1, **n=3 → 80.1**, n=4 → 78.7,
n=5 → 75.9 tok/s. Three is the optimum; higher n trades acceptance for draft width and
loses.

## Rollback

The previous build is untouched at `/tank/aimodels/qwen38-27b-uncensored-nvfp4`.

```bash
ssh infra-ops@10.250.50.54
sudo cp /opt/docker/compose/gen-seat/.env.bak-w4a16-20260815 /opt/docker/compose/gen-seat/.env
cd /opt/docker/compose/gen-seat && sudo docker compose up -d vllm-gen
```

⚠ `gen-seat/.env` is mode 0600 and lkraven-owned — **every** `docker compose` call
against it needs `sudo`, or compose fails with `permission denied` reading `.env`,
leaves the old container running, and the change silently does not take.