vh
2e3dcc2d3d
Qwen3.5-9B vision-language served FP8 on ana-ml2 GPU1 (co-located with granite + the embed/rerank/reward trio; GPU0 kept free for hot-loading large models), :8007, fronted by LiteLLM as qwen3.5-9b-fp8. Pinned to vllm/vllm-openai nightly@sha256:49211ab2 — :latest (v0.19.1) quantizes the VL vision tower under fp8 and garbles vision; the nightly correctly excludes it (LM stays FP8, vision tower BF16). util 0.40 (~38GB) on the shared card (vLLM needs free>=util*total here). Vision verified end-to-end through the gateway.
26 lines
939 B
Bash
26 lines
939 B
Bash
# Qwen3.5-9B VL (FP8) on ana-ml2 — copy to .env on the host and fill.
|
|
# Real .env lives on ana-ml2 at /opt/docker/compose/qwen35-vl/.env (gitignored).
|
|
|
|
# Pinned nightly digest — carries the Qwen3.5-VL vision-FP8 exclusion fix that
|
|
# :latest (v0.19.1) lacks. Re-pin to :latest once the fix reaches a stable
|
|
# release (see README + compose header).
|
|
QWEN_IMAGE=vllm/vllm-openai@sha256:49211ab2155b21a2dc35f3583f5b545f5e55e77daf8f86df49977c71d5f2f528
|
|
|
|
QWEN_CONTAINER_NAME=vllm-qwen35
|
|
QWEN_MODEL=Qwen/Qwen3.5-9B
|
|
QWEN_SERVED_NAME=qwen3.5-9b-fp8
|
|
QWEN_PORT=8007
|
|
|
|
# GPU 1 = shared with the granite summarizer + embed/rerank/reward trio.
|
|
# GPU 0 is kept free for hot-reloading large models.
|
|
QWEN_GPU_ID=1
|
|
|
|
# 0.40 (~38 GB) — above the ~34 GB start floor, ~10 GB card headroom over prod.
|
|
# On this shared card vLLM needs free >= util*total, so util is capped ~0.51.
|
|
QWEN_GPU_MEM_UTIL=0.40
|
|
QWEN_MAX_MODEL_LEN=32768
|
|
|
|
# Optional
|
|
HF_TOKEN=
|
|
API_KEY=
|