vh
74f596b1d3
Re-quantizes the fleet `gen` seat from weight-only NVFP4A16 to a mixed-precision build: NVFP4 W4A4 for layers 0-55 MLPs, FP8 W8A8 for the attention projections / linear_attn / lm_head / layers 56-63 MLPs, FP8 KV cache. Replicates the scheme of unsloth/Qwen3.8-27B-NVFP4 on the abliterated weights. The queued task named this "W4A8" (NVFP4 weights + FP8 activations). That checkpoint cannot be served: vLLM 0.24's compressed-tensors dispatcher (compressed_tensors.py:704-713) accepts NVFP4 weights with either no input quantization (W4A16, which forces the Marlin kernel) or NVFP4 input quantization (W4A4) -- anything else, FP8 included, raises ValueError at load. CompressedTensorsW4A8Fp8 is INT4 weights gated on an exact-sm90 check, so it is closed on Blackwell twice over. The ~20% intuition was correct; the scheme name was not. Getting FP8 into the mix has to be done per-layer-group. Established the gain before spending GPU time: unsloth's build was already on-box, so serving it as a probe measured +19.1% over our seat at identical MTP acceptance -- a kernel-level result, no requant needed to learn it. Measured, cache-busted, bs=1: decode 80.12 -> 94.53 tok/s (+18.0%) MTP acceptance 47.8% -> 47.7% (unchanged) perplexity (n=6) 6.941 -> 7.059 (+1.7%) abliteration 4/4 -> 4/4 (preserved) weights on disk 27.7 -> 22.5 GB (-19%) Surface test green on the live seat: plain chat, vision, tool calling, thinking split, 36K-token needle retrieval, streaming. All 7 LiteLLM aliases verified routing. GEN_GPU_MEM_UTIL 0.45 -> 0.43: the new weights are 5.2 GB smaller, and at 0.45 the seat absorbed that slack as KV, leaving meromero-charrp 0.18 GiB short of its budget on the shared GPU0 -- it crash-looped. Handing the space back leaves gen 422K tokens of KV (1.6x its 262K context) and both seats co-resident at 89.8/97.9 GB. Also records two measured negatives so they are not re-chased: GEN_SPEC_TOKENS is already optimal at 3 (swept 2/3/4/5 -> 77.1/80.1/78.7/ 75.9 tok/s), and vLLM's prompt_logprobs are ~uniform while speculative decoding is on, so perplexity must be measured with spec off. Pipeline, acceptance harness and raw measurements land in services/gen-seat-mixed-quant/. Rollback is one .env line; the previous build is untouched at /tank/aimodels/qwen38-27b-uncensored-nvfp4.
94 lines
2.6 KiB
JSON
94 lines
2.6 KiB
JSON
{
|
|
"tag": "W4A16-baseline",
|
|
"base": "http://10.250.50.54:8015",
|
|
"model": "qwen3.8-27b-uncensored",
|
|
"short": {
|
|
"label": "short-decode",
|
|
"n": 5,
|
|
"tok_s_median": 81.15949743763107,
|
|
"tok_s_mean": 81.15460486013536,
|
|
"tok_s_max": 81.18248511529964,
|
|
"prompt_tokens": 74,
|
|
"completion_tokens_median": 400,
|
|
"mtp_accept_median": 0.48261758691206547
|
|
},
|
|
"long": {
|
|
"label": "long-prefill",
|
|
"n": 2,
|
|
"tok_s_median": 63.13875221728922,
|
|
"tok_s_mean": 63.13875221728922,
|
|
"tok_s_max": 63.14483903204707,
|
|
"prompt_tokens": 3184,
|
|
"completion_tokens_median": 128.0,
|
|
"mtp_accept_median": 0.6991869918699187
|
|
},
|
|
"raw": {
|
|
"short": [
|
|
{
|
|
"wall_s": 4.931351114064455,
|
|
"completion_tokens": 400,
|
|
"prompt_tokens": 74,
|
|
"tok_s_overall": 81.11367265234479,
|
|
"mtp_accept_rate": 0.48261758691206547,
|
|
"draft_tokens": 489.0,
|
|
"accepted_tokens": 236.0
|
|
},
|
|
{
|
|
"wall_s": 4.92856674361974,
|
|
"completion_tokens": 400,
|
|
"prompt_tokens": 74,
|
|
"tok_s_overall": 81.15949743763107,
|
|
"mtp_accept_rate": 0.48261758691206547,
|
|
"draft_tokens": 489.0,
|
|
"accepted_tokens": 236.0
|
|
},
|
|
{
|
|
"wall_s": 4.928732456639409,
|
|
"completion_tokens": 400,
|
|
"prompt_tokens": 74,
|
|
"tok_s_overall": 81.15676870656004,
|
|
"mtp_accept_rate": 0.48261758691206547,
|
|
"draft_tokens": 489.0,
|
|
"accepted_tokens": 236.0
|
|
},
|
|
{
|
|
"wall_s": 4.92717116791755,
|
|
"completion_tokens": 400,
|
|
"prompt_tokens": 74,
|
|
"tok_s_overall": 81.18248511529964,
|
|
"mtp_accept_rate": 0.48261758691206547,
|
|
"draft_tokens": 489.0,
|
|
"accepted_tokens": 236.0
|
|
},
|
|
{
|
|
"wall_s": 4.928499765694141,
|
|
"completion_tokens": 400,
|
|
"prompt_tokens": 74,
|
|
"tok_s_overall": 81.16060038884126,
|
|
"mtp_accept_rate": 0.48261758691206547,
|
|
"draft_tokens": 489.0,
|
|
"accepted_tokens": 236.0
|
|
}
|
|
],
|
|
"long": [
|
|
{
|
|
"wall_s": 2.0274765715003014,
|
|
"completion_tokens": 128,
|
|
"prompt_tokens": 3184,
|
|
"tok_s_overall": 63.13266540253137,
|
|
"mtp_accept_rate": 0.6991869918699187,
|
|
"draft_tokens": 123.0,
|
|
"accepted_tokens": 86.0
|
|
},
|
|
{
|
|
"wall_s": 2.027085696347058,
|
|
"completion_tokens": 128,
|
|
"prompt_tokens": 3184,
|
|
"tok_s_overall": 63.14483903204707,
|
|
"mtp_accept_rate": 0.6991869918699187,
|
|
"draft_tokens": 123.0,
|
|
"accepted_tokens": 86.0
|
|
}
|
|
]
|
|
}
|
|
} |