vh
a0fed13801
Retire qwen35-vl (Qwen3.5-9B); add qwen36-vl serving the official FP8 Qwen3.6-35B-A3B vision MoE on :8007 under its TRUE name only — no alias. qwen3.5-9b-fp8 is killed at vLLM AND the litellm gateway (404/400); a model is never served under a prior model's name. Consumer (comfy-dev/arbo) notified + migrated; arbo vkeys flipped to all-proxy-models; shared all-agents-local key repointed to qwen3.6-35b-a3b. GPU-1 rebalance for the heavier FP8 weights (~34 GB): granite 0.35->0.24 / 131K->64K, embed/rerank 0.05->0.03 (reclaimed util-reservation waste). Verified: vision correct, 20-concurrent/endpoint load test = no OOM (~7.5 GB headroom). Drop the llama-swap qwen3.5-9b GPU-0 pin (GPU 0 freed for the creative-writing hot-swap card). NVFP4 was the lighter fit (~21 GB) but its vLLM ModelOpt-MoE loader is broken (KeyError w2_input_scale / lm_head.input_scale, vllm #44081); revisit when fixed.
35 lines
1.5 KiB
Bash
35 lines
1.5 KiB
Bash
# Qwen3.6-35B-A3B VL (official FP8) on ana-ml2 — copy to .env on the host and fill.
|
||
# Real .env lives on ana-ml2 at /opt/docker/compose/qwen36-vl/.env (gitignored).
|
||
#
|
||
# Replaces qwen35-vl (Qwen3.5-9B) 2026-06-14. See compose.yaml header for the
|
||
# FP8-over-NVFP4 rationale (vLLM NVFP4 MoE loader broken, #44081) and why this
|
||
# uses plain :latest with NO --quantization (pre-quantized checkpoint; a forced
|
||
# flag would noise-quantize the vision tower like the old qwen35-vl).
|
||
|
||
# :latest is fine — the official FP8 checkpoint loads + serves vision correctly
|
||
# on 0.19.1 (validated 2026-06-14). NO pinned nightly digest needed.
|
||
QWEN_IMAGE=vllm/vllm-openai:latest
|
||
|
||
QWEN_CONTAINER_NAME=vllm-qwen36
|
||
QWEN_MODEL=Qwen/Qwen3.6-35B-A3B-FP8
|
||
QWEN_PORT=8007
|
||
|
||
# GPU 1 = shared with the granite summarizer + embed/rerank/reward trio.
|
||
# GPU 0 is kept free for the llama-swap creative-writing hot-swap card.
|
||
QWEN_GPU_ID=1
|
||
|
||
# util 0.42 (~40 GB) — official FP8 weights load in ~34.2 GiB; 0.42 covers
|
||
# weights + CUDA-graph + a generous KV pool. Hybrid attn (10 of 40 layers full-
|
||
# attn, ~10 KB/tok KV) makes long context nearly free, so max-len is generous.
|
||
# Budget (2026-06-14): qwen36 0.42 + granite 0.28 + trio 0.20 = 0.90 total,
|
||
# ~10 GB graph headroom. Bring qwen36 up LAST so capture sees the free room.
|
||
QWEN_GPU_MEM_UTIL=0.42
|
||
QWEN_MAX_MODEL_LEN=131072
|
||
# Sampler-warmup OOM guard on the shared GPU (248K vocab × default 1024 seqs is
|
||
# a huge transient). 32 is plenty for a vision endpoint.
|
||
QWEN_MAX_NUM_SEQS=32
|
||
|
||
# Optional
|
||
HF_TOKEN=
|
||
API_KEY=
|