vh
a81c44db04
Measured on the restored model: at the template default xhigh, reasoning runs 4,529-5,532 chars on a 3-turn history and was observed spiking to 9,261; medium holds it to 2,602-3,283 with content length unchanged or better. Per-request overridable; an invalid value 400s. SCOPE CAVEAT, stated because I applied this while chasing the wrong path: this is a NO-OP for the alias, which sends enable_thinking:false and produces zero reasoning. It affects only. The operator's reported multi-turn failure was on , so this does not address it.
gen-seat — the fleet gen seat (ana-ml2 GPU0, :8015)
Serves qwen3.8-27b-uncensored (JonathanColetti/Qwen3.8-27B-Uncensored,
Heretic-abliterated Qwen3.8-27B, vision-intact, 262K context) plus the
-thinking served-name for the reasoning split.
Backs 7 LiteLLM aliases: gen, gen-reasoning, summarizer,
summarizer-large, classifier, image-judge, qwen-image-bench. Treat any
change here as fleet-wide.
- Quant: in-house mixed-precision — NVFP4 W4A4 for layers 0-55 MLPs,
FP8 W8A8 for attention /
linear_attn/lm_head/ layers 56-63 MLPs, FP8 KV. ~94.5 tok/s decode at bs=1, MTP n=3 @ ~48% acceptance. - Speculative decoding:
qwen3_5_mtp,num_speculative_tokens=3(measured optimum — see the quant service README). - Tunables:
.envon the host. ⚠ it is mode 0600 / lkraven-owned, so everydocker composecall against this stack needssudo— without it compose cannot read.env, fails withpermission denied, and leaves the old container running while appearing to have succeeded.
Pipeline, acceptance gate, and rollback
All of it — the recipe, the re:^mtp.* foot-gun, the benchmark harness, the GPU0
budget interaction with meromero-charrp, and the rollback command — lives in
services/gen-seat-mixed-quant/README.md.
Deploy
scripts/deploy-stack.sh ana-ml2 gen-seat # diffs vs live, prompts y/N
# on host:
ssh infra-ops@10.250.50.54
cd /opt/docker/compose/gen-seat && sudo docker compose up -d vllm-gen