feat(gen-seat): mixed NVFP4+FP8 requant — +18% decode at equal MTP acceptance
Re-quantizes the fleet `gen` seat from weight-only NVFP4A16 to a mixed-precision build: NVFP4 W4A4 for layers 0-55 MLPs, FP8 W8A8 for the attention projections / linear_attn / lm_head / layers 56-63 MLPs, FP8 KV cache. Replicates the scheme of unsloth/Qwen3.8-27B-NVFP4 on the abliterated weights. The queued task named this "W4A8" (NVFP4 weights + FP8 activations). That checkpoint cannot be served: vLLM 0.24's compressed-tensors dispatcher (compressed_tensors.py:704-713) accepts NVFP4 weights with either no input quantization (W4A16, which forces the Marlin kernel) or NVFP4 input quantization (W4A4) -- anything else, FP8 included, raises ValueError at load. CompressedTensorsW4A8Fp8 is INT4 weights gated on an exact-sm90 check, so it is closed on Blackwell twice over. The ~20% intuition was correct; the scheme name was not. Getting FP8 into the mix has to be done per-layer-group. Established the gain before spending GPU time: unsloth's build was already on-box, so serving it as a probe measured +19.1% over our seat at identical MTP acceptance -- a kernel-level result, no requant needed to learn it. Measured, cache-busted, bs=1: decode 80.12 -> 94.53 tok/s (+18.0%) MTP acceptance 47.8% -> 47.7% (unchanged) perplexity (n=6) 6.941 -> 7.059 (+1.7%) abliteration 4/4 -> 4/4 (preserved) weights on disk 27.7 -> 22.5 GB (-19%) Surface test green on the live seat: plain chat, vision, tool calling, thinking split, 36K-token needle retrieval, streaming. All 7 LiteLLM aliases verified routing. GEN_GPU_MEM_UTIL 0.45 -> 0.43: the new weights are 5.2 GB smaller, and at 0.45 the seat absorbed that slack as KV, leaving meromero-charrp 0.18 GiB short of its budget on the shared GPU0 -- it crash-looped. Handing the space back leaves gen 422K tokens of KV (1.6x its 262K context) and both seats co-resident at 89.8/97.9 GB. Also records two measured negatives so they are not re-chased: GEN_SPEC_TOKENS is already optimal at 3 (swept 2/3/4/5 -> 77.1/80.1/78.7/ 75.9 tok/s), and vLLM's prompt_logprobs are ~uniform while speculative decoding is on, so perplexity must be measured with spec off. Pipeline, acceptance harness and raw measurements land in services/gen-seat-mixed-quant/. Rollback is one .env line; the previous build is untouched at /tank/aimodels/qwen38-27b-uncensored-nvfp4.
This commit is contained in:
@@ -0,0 +1,135 @@
|
||||
# gen-seat mixed-precision quant — NVFP4 W4A4 MLP + FP8 W8A8 attention
|
||||
|
||||
The pipeline that produced `/tank/aimodels/qwen38-27b-uncensored-nvfp4-mixed`, the
|
||||
current fleet `gen` seat on ana-ml2 GPU0 `:8015`. **+18% decode** over the previous
|
||||
weight-only NVFP4A16 build, at equal MTP acceptance and +1.7% perplexity.
|
||||
|
||||
## The headline correction: "W4A8" is not a thing you can serve
|
||||
|
||||
The queued task was "re-quant to NVFP4 weights + FP8 activations (W4A8) for ~20%
|
||||
more decode". **That checkpoint cannot load.** vLLM 0.24's compressed-tensors
|
||||
dispatcher (`compressed_tensors.py:704-713`) allows NVFP4 weights with exactly two
|
||||
activation options:
|
||||
|
||||
| input_activations | scheme | kernel |
|
||||
|---|---|---|
|
||||
| `None` | W4A16 | **Marlin** (forced — `kernels/linear/__init__.py:881-883`) |
|
||||
| NVFP4 | W4A4 | native Blackwell FP4 |
|
||||
|
||||
anything else — FP8 included — raises
|
||||
|
||||
```
|
||||
ValueError: For NVFP4 weights, input quantization must also be NVFP4 format,
|
||||
None for NVFP4A16
|
||||
```
|
||||
|
||||
`CompressedTensorsW4A8Fp8` exists but is **INT4** weights (`W4A8_SUPPORTED_TYPES_MAP
|
||||
= {4: int4}`) gated on `_check_scheme_supported(90, match_exact=True)` — Hopper only.
|
||||
ana-ml2 is Blackwell (sm_120), so that path is doubly closed.
|
||||
|
||||
The ~20% intuition was right; the scheme name was wrong. The servable way to get FP8
|
||||
into the mix is **per-layer-group**, which is exactly what `unsloth/Qwen3.8-27B-NVFP4`
|
||||
does — and that build, measured on-box, ran +19.1% faster than ours at identical MTP
|
||||
acceptance. This pipeline replicates its recipe on the abliterated weights.
|
||||
|
||||
## The recipe
|
||||
|
||||
| group | scheme | targets |
|
||||
|---|---|---|
|
||||
| `group_0` | **FP8 W8A8** — channel weights (static), per-token dynamic activations | `self_attn.{q,k,v,o}_proj`, `linear_attn.{in_proj_qkv,in_proj_z,out_proj}`, `lm_head`, **layers 56-63** MLPs |
|
||||
| `group_1` | **NVFP4 W4A4** — tensor_group gsize16, fp8 scales, `imatrix_mse` weights, `dynamic:"local"` activations | **layers 0-55** MLP `{gate,up,down}_proj` |
|
||||
| kv cache | FP8 static tensor | — |
|
||||
| ignored | vision tower, `linear_attn.{norm,in_proj_a,in_proj_b}`, `re:^mtp.*` | — |
|
||||
|
||||
Keeping the **last 8 layers' MLPs at FP8** is the accuracy-preservation trick — late
|
||||
layers are the sensitive ones. `conv1d` in `linear_attn` is not a Linear and stays BF16
|
||||
in both our build and unsloth's.
|
||||
|
||||
Targets are deliberately **non-overlapping** (`group_1` enumerates layers 0-55 rather
|
||||
than matching all MLPs) instead of relying on group precedence to resolve the 56-63
|
||||
collision. `validate_targets.py` proves this against the real module names before any
|
||||
GPU time is spent — run it first.
|
||||
|
||||
## Running it
|
||||
|
||||
```bash
|
||||
# 0. prove the regexes hit what you think (free, no GPU)
|
||||
python3 validate_targets.py /tank/aimodels/qwen38-27b-uncensored-bf16
|
||||
# -> expect OVERLAP 0, MLP layer union covers 0-63 True
|
||||
|
||||
# 1. quant (~20 min on one Blackwell; needs llmcompressor, NOT modelopt)
|
||||
python3 quant_mixed_nvfp4.py \
|
||||
--model /tank/aimodels/qwen38-27b-uncensored-bf16 \
|
||||
--calib /tank/aimodels/heretic2-nvfp4-work/production_calib_512.jsonl \
|
||||
--out /tank/aimodels/qwen38-27b-uncensored-nvfp4-mixed \
|
||||
--num-samples 256 --seqlen 2048
|
||||
|
||||
# 2. MANDATORY post-steps — graft MTP, restore preprocessor, repair the mtp ignore
|
||||
python3 post_quant.py <bf16-source> <out-dir>
|
||||
```
|
||||
|
||||
`pip install llmcompressor` into the stock `vllm/vllm-openai:latest` image gives
|
||||
llmcompressor 0.13.0 + compressed-tensors 0.18.0 without disturbing torch or
|
||||
transformers. **Do not use modelopt 0.43** — dependency hell on `qwen3_5`.
|
||||
|
||||
## The foot-gun that has now cost three rounds
|
||||
|
||||
`llm-compressor` **prunes `ignore` entries that matched no module at quant time.**
|
||||
The wrapper class (`Qwen3_5ForConditionalGeneration`) never loads the MTP head, so
|
||||
`re:^mtp.*` matches nothing and is silently dropped from the saved config. vLLM then
|
||||
treats the freshly grafted BF16 MTP head as quantized, brings it up **uninitialised,
|
||||
and speculative decoding runs at 0% acceptance.**
|
||||
|
||||
`post_quant.py` re-injects the entry *after* the graft and re-verifies. It is not
|
||||
optional, and it verifies rather than assumes — that check fired on this very run.
|
||||
|
||||
## Acceptance gate (`bench/`)
|
||||
|
||||
Speed alone does not justify cutting over a seat backing 7 LiteLLM aliases.
|
||||
|
||||
| metric | W4A16 (old) | mixed (new) | delta |
|
||||
|---|---|---|---|
|
||||
| decode tok/s, bs=1, cache-busted | 80.12 | **94.53** | **+18.0%** |
|
||||
| MTP acceptance | 47.8% | 47.7% | unchanged |
|
||||
| perplexity, 6 held-out passages | 6.941 | 7.059 | +1.7% worse |
|
||||
| abliteration compliance | 4/4 | 4/4 | preserved |
|
||||
| weights on disk | 27.7 GB | 22.5 GB | −19% |
|
||||
|
||||
- `quickbench.py` — cache-busted bs=1 decode + MTP acceptance. **Bust the cache:** with
|
||||
a fixed prompt, prefix caching returns byte-identical timings and you measure nothing.
|
||||
- `eval_quality.py` — perplexity, deterministic generations, abliteration survival.
|
||||
**PPL must be measured with `--speculative-config` OFF**: under MTP, vLLM's
|
||||
`prompt_logprobs` come back ~uniform over the vocab (median rank ~10^5, logprob
|
||||
≈ log(1/vocab)). The harness raises rather than reporting the garbage.
|
||||
- `surface_test.py` — the real gate: plain chat, vision, tool calling, thinking split,
|
||||
36K-token needle retrieval, streaming. All six must pass before a cutover.
|
||||
- `serve_probe.sh <model-dir> [nospec]` — serve a candidate on `:8017` without touching
|
||||
the live seat.
|
||||
|
||||
## GPU0 budget
|
||||
|
||||
The mixed build's weights are 5.2 GB smaller. At the old `GEN_GPU_MEM_UTIL=0.45` the
|
||||
seat absorbed that slack as extra KV (17.0 GiB / 477K tokens) and left `meromero-charrp`
|
||||
**0.18 GiB short** of its 0.52 budget — it crash-looped on startup. Fixed by handing the
|
||||
space back: `GEN_GPU_MEM_UTIL=0.43` → 15.1 GiB / 422K tokens, still 1.6× the 262K
|
||||
context. Both seats co-resident at **89.8 / 97.9 GB**.
|
||||
|
||||
## Levers already measured — do not re-chase
|
||||
|
||||
`GEN_SPEC_TOKENS` swept on this seat: n=2 → 77.1, **n=3 → 80.1**, n=4 → 78.7,
|
||||
n=5 → 75.9 tok/s. Three is the optimum; higher n trades acceptance for draft width and
|
||||
loses.
|
||||
|
||||
## Rollback
|
||||
|
||||
The previous build is untouched at `/tank/aimodels/qwen38-27b-uncensored-nvfp4`.
|
||||
|
||||
```bash
|
||||
ssh infra-ops@10.250.50.54
|
||||
sudo cp /opt/docker/compose/gen-seat/.env.bak-w4a16-20260815 /opt/docker/compose/gen-seat/.env
|
||||
cd /opt/docker/compose/gen-seat && sudo docker compose up -d vllm-gen
|
||||
```
|
||||
|
||||
⚠ `gen-seat/.env` is mode 0600 and lkraven-owned — **every** `docker compose` call
|
||||
against it needs `sudo`, or compose fails with `permission denied` reading `.env`,
|
||||
leaves the old container running, and the change silently does not take.
|
||||
Reference in New Issue
Block a user