feat(qwen35-vl): Qwen3.5-9B VL FP8 stack on ana-ml2 GPU1 + LiteLLM entry

Qwen3.5-9B vision-language served FP8 on ana-ml2 GPU1 (co-located with
granite + the embed/rerank/reward trio; GPU0 kept free for hot-loading
large models), :8007, fronted by LiteLLM as qwen3.5-9b-fp8.

Pinned to vllm/vllm-openai nightly@sha256:49211ab2 — :latest (v0.19.1)
quantizes the VL vision tower under fp8 and garbles vision; the nightly
correctly excludes it (LM stays FP8, vision tower BF16). util 0.40
(~38GB) on the shared card (vLLM needs free>=util*total here). Vision
verified end-to-end through the gateway.
This commit is contained in:
vh
2026-06-13 02:39:33 -07:00
parent 5f049cb4ad
commit 2e3dcc2d3d
4 changed files with 177 additions and 0 deletions
+11
View File
@@ -30,6 +30,17 @@ model_list:
model_info:
mode: chat
# --- Qwen3.5-9B vision-language (FP8) — vision + chat. vLLM on ana-ml2 GPU 1,
# nightly-pinned (vision-FP8 exclusion fix), :8007. Explicit entry shadows
# the "*" wildcard llama-swap route. ---
- model_name: qwen3.5-9b-fp8
litellm_params:
model: hosted_vllm/qwen3.5-9b-fp8
api_base: http://10.250.50.54:8007/v1
api_key: os.environ/VLLM_API_KEY
model_info:
mode: chat
# --- Qwen3 embeddings ---
- model_name: qwen3-embedding
litellm_params: