vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity

feat(gen-seat): mixed NVFP4+FP8 requant — +18% decode at equal MTP acceptance

Browse Source 
Re-quantizes the fleet `gen` seat from weight-only NVFP4A16 to a
mixed-precision build: NVFP4 W4A4 for layers 0-55 MLPs, FP8 W8A8 for the
attention projections / linear_attn / lm_head / layers 56-63 MLPs, FP8 KV
cache. Replicates the scheme of unsloth/Qwen3.8-27B-NVFP4 on the
abliterated weights.

The queued task named this "W4A8" (NVFP4 weights + FP8 activations). That
checkpoint cannot be served: vLLM 0.24's compressed-tensors dispatcher
(compressed_tensors.py:704-713) accepts NVFP4 weights with either no input
quantization (W4A16, which forces the Marlin kernel) or NVFP4 input
quantization (W4A4) -- anything else, FP8 included, raises ValueError at
load. CompressedTensorsW4A8Fp8 is INT4 weights gated on an exact-sm90
check, so it is closed on Blackwell twice over. The ~20% intuition was
correct; the scheme name was not. Getting FP8 into the mix has to be done
per-layer-group.

Established the gain before spending GPU time: unsloth's build was already
on-box, so serving it as a probe measured +19.1% over our seat at identical
MTP acceptance -- a kernel-level result, no requant needed to learn it.

Measured, cache-busted, bs=1:

  decode              80.12 -> 94.53 tok/s   (+18.0%)
  MTP acceptance      47.8% -> 47.7%         (unchanged)
  perplexity (n=6)    6.941 -> 7.059         (+1.7%)
  abliteration        4/4   -> 4/4           (preserved)
  weights on disk     27.7  -> 22.5 GB       (-19%)

Surface test green on the live seat: plain chat, vision, tool calling,
thinking split, 36K-token needle retrieval, streaming. All 7 LiteLLM
aliases verified routing.

GEN_GPU_MEM_UTIL 0.45 -> 0.43: the new weights are 5.2 GB smaller, and at
0.45 the seat absorbed that slack as KV, leaving meromero-charrp 0.18 GiB
short of its budget on the shared GPU0 -- it crash-looped. Handing the
space back leaves gen 422K tokens of KV (1.6x its 262K context) and both
seats co-resident at 89.8/97.9 GB.

Also records two measured negatives so they are not re-chased:
GEN_SPEC_TOKENS is already optimal at 3 (swept 2/3/4/5 -> 77.1/80.1/78.7/
75.9 tok/s), and vLLM's prompt_logprobs are ~uniform while speculative
decoding is on, so perplexity must be measured with spec off.

Pipeline, acceptance harness and raw measurements land in
services/gen-seat-mixed-quant/. Rollback is one .env line; the previous
build is untouched at /tank/aimodels/qwen38-27b-uncensored-nvfp4.
This commit is contained in:
vh
2026-08-15 02:21:00 -07:00
parent b8f0f4c568
commit 74f596b1d3
21 changed files with 1315 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files
stacks/gen-seat/.env.example
+9 -2
View File
@@ -7,12 +7,19 @@ GEN_CONTAINER_NAME=vllm-gen
GEN_PORT=8015
GEN_SERVED_NAME=qwen3.8-27b-uncensored
GEN_SERVED_NAME_THINK=qwen3.8-27b-uncensored-thinking
GEN_MODEL=/tank/aimodels/qwen38-27b-uncensored-nvfp4
GEN_MODEL=/tank/aimodels/qwen38-27b-uncensored-nvfp4-mixed
GEN_QUANT=compressed-tensors
GEN_GPU_MEM_UTIL=0.45
# 0.43 not 0.45: the mixed build's weights are 5.2 GB smaller, and at 0.45 the
# seat absorbed that slack as extra KV, leaving meromero-charrp 0.18 GiB short
# of its 0.52 budget on the shared GPU0 (it crash-looped). 0.43 still gives gen
# 422K tokens of KV = 1.6x its 262K context. Both seats: 89.8/97.9 GB.
GEN_GPU_MEM_UTIL=0.43
GEN_MAX_MODEL_LEN=262144
GEN_MAX_NUM_SEQS=16
GEN_KV_CACHE_DTYPE=fp8
GEN_REASONING_PARSER=qwen3
GEN_SPEC_METHOD=qwen3_5_mtp
# 3 is the measured optimum, not a default: swept n=2/3/4/5 on this seat ->
# 77.1 / 80.1 / 78.7 / 75.9 tok/s. Higher n trades acceptance for draft width
# and loses. Do not re-chase.
GEN_SPEC_TOKENS=3