vh
1f5b2cbcb0
The MTP-on + prefix-caching-off mitigation (63a3cb2) passed synthetic 7-turn probes but the operator still saw severe degeneration in real use. A passing synthetic probe is NOT sufficient evidence -- it under-covers real workloads (content distribution, conversation depth). Reverted to the verified known-good: MTP off, prefix caching on (the7bd38b3state), ~half decode speed but coherent. Operator is driving it to re-confirm. Lesson reinforced (the recurring one this session): do not trust a synthetic reproduction to VALIDATE a fix for a bug that only manifests in the operator's real usage -- it validated a non-fix twice tonight.
gen-seat — the fleet gen seat (ana-ml2 GPU0, :8015)
Serves qwen3.8-27b-uncensored (JonathanColetti/Qwen3.8-27B-Uncensored,
Heretic-abliterated Qwen3.8-27B, vision-intact, 262K context) plus the
-thinking served-name for the reasoning split.
Backs 7 LiteLLM aliases: gen, gen-reasoning, summarizer,
summarizer-large, classifier, image-judge, qwen-image-bench. Treat any
change here as fleet-wide.
- Quant: in-house mixed-precision — NVFP4 W4A4 for layers 0-55 MLPs,
FP8 W8A8 for attention /
linear_attn/lm_head/ layers 56-63 MLPs, FP8 KV. ~94.5 tok/s decode at bs=1, MTP n=3 @ ~48% acceptance. - Speculative decoding:
qwen3_5_mtp,num_speculative_tokens=3(measured optimum — see the quant service README). - Tunables:
.envon the host. ⚠ it is mode 0600 / lkraven-owned, so everydocker composecall against this stack needssudo— without it compose cannot read.env, fails withpermission denied, and leaves the old container running while appearing to have succeeded.
Pipeline, acceptance gate, and rollback
All of it — the recipe, the re:^mtp.* foot-gun, the benchmark harness, the GPU0
budget interaction with meromero-charrp, and the rollback command — lives in
services/gen-seat-mixed-quant/README.md.
Deploy
scripts/deploy-stack.sh ana-ml2 gen-seat # diffs vs live, prompts y/N
# on host:
ssh infra-ops@10.250.50.54
cd /opt/docker/compose/gen-seat && sudo docker compose up -d vllm-gen