feat(qwen35-vl): Qwen3.5-9B VL FP8 stack on ana-ml2 GPU1 + LiteLLM entry
Qwen3.5-9B vision-language served FP8 on ana-ml2 GPU1 (co-located with granite + the embed/rerank/reward trio; GPU0 kept free for hot-loading large models), :8007, fronted by LiteLLM as qwen3.5-9b-fp8. Pinned to vllm/vllm-openai nightly@sha256:49211ab2 — :latest (v0.19.1) quantizes the VL vision tower under fp8 and garbles vision; the nightly correctly excludes it (LM stays FP8, vision tower BF16). util 0.40 (~38GB) on the shared card (vLLM needs free>=util*total here). Vision verified end-to-end through the gateway.
This commit is contained in:
@@ -30,6 +30,17 @@ model_list:
|
||||
model_info:
|
||||
mode: chat
|
||||
|
||||
# --- Qwen3.5-9B vision-language (FP8) — vision + chat. vLLM on ana-ml2 GPU 1,
|
||||
# nightly-pinned (vision-FP8 exclusion fix), :8007. Explicit entry shadows
|
||||
# the "*" wildcard llama-swap route. ---
|
||||
- model_name: qwen3.5-9b-fp8
|
||||
litellm_params:
|
||||
model: hosted_vllm/qwen3.5-9b-fp8
|
||||
api_base: http://10.250.50.54:8007/v1
|
||||
api_key: os.environ/VLLM_API_KEY
|
||||
model_info:
|
||||
mode: chat
|
||||
|
||||
# --- Qwen3 embeddings ---
|
||||
- model_name: qwen3-embedding
|
||||
litellm_params:
|
||||
|
||||
Reference in New Issue
Block a user