feat(ana-ml2): replace Qwen3.5-9B vision with Qwen3.6-35B-A3B FP8 on GPU 1
Retire qwen35-vl (Qwen3.5-9B); add qwen36-vl serving the official FP8 Qwen3.6-35B-A3B vision MoE on :8007 under its TRUE name only — no alias. qwen3.5-9b-fp8 is killed at vLLM AND the litellm gateway (404/400); a model is never served under a prior model's name. Consumer (comfy-dev/arbo) notified + migrated; arbo vkeys flipped to all-proxy-models; shared all-agents-local key repointed to qwen3.6-35b-a3b. GPU-1 rebalance for the heavier FP8 weights (~34 GB): granite 0.35->0.24 / 131K->64K, embed/rerank 0.05->0.03 (reclaimed util-reservation waste). Verified: vision correct, 20-concurrent/endpoint load test = no OOM (~7.5 GB headroom). Drop the llama-swap qwen3.5-9b GPU-0 pin (GPU 0 freed for the creative-writing hot-swap card). NVFP4 was the lighter fit (~21 GB) but its vLLM ModelOpt-MoE loader is broken (KeyError w2_input_scale / lm_head.input_scale, vllm #44081); revisit when fixed.
This commit is contained in:
@@ -30,12 +30,13 @@ model_list:
|
||||
model_info:
|
||||
mode: chat
|
||||
|
||||
# --- Qwen3.5-9B vision-language (FP8) — vision + chat. vLLM on ana-ml2 GPU 1,
|
||||
# nightly-pinned (vision-FP8 exclusion fix), :8007. Explicit entry shadows
|
||||
# the "*" wildcard llama-swap route. ---
|
||||
- model_name: qwen3.5-9b-fp8
|
||||
# --- Qwen3.6-35B-A3B vision-language MoE (official FP8) — vision + chat. vLLM
|
||||
# on ana-ml2 GPU 1, :8007. Explicit entry shadows the "*" wildcard llama-swap
|
||||
# route. REPLACED qwen3.5-9b-fp8 2026-06-14 (the 9B is retired; this is a
|
||||
# 35B-A3B MoE — served under its TRUE name, never aliased under the old one). ---
|
||||
- model_name: qwen3.6-35b-a3b
|
||||
litellm_params:
|
||||
model: hosted_vllm/qwen3.5-9b-fp8
|
||||
model: hosted_vllm/qwen3.6-35b-a3b
|
||||
api_base: http://10.250.50.54:8007/v1
|
||||
api_key: os.environ/VLLM_API_KEY
|
||||
model_info:
|
||||
|
||||
Reference in New Issue
Block a user