vh
2e3dcc2d3d
Qwen3.5-9B vision-language served FP8 on ana-ml2 GPU1 (co-located with granite + the embed/rerank/reward trio; GPU0 kept free for hot-loading large models), :8007, fronted by LiteLLM as qwen3.5-9b-fp8. Pinned to vllm/vllm-openai nightly@sha256:49211ab2 — :latest (v0.19.1) quantizes the VL vision tower under fp8 and garbles vision; the nightly correctly excludes it (LM stays FP8, vision tower BF16). util 0.40 (~38GB) on the shared card (vLLM needs free>=util*total here). Vision verified end-to-end through the gateway.
58 lines
2.9 KiB
Markdown
58 lines
2.9 KiB
Markdown
# qwen35-vl — Qwen3.5-9B vision-language (FP8) on ana-ml2
|
||
|
||
Qwen3.5-9B, a hybrid GDN + vision-language model, served **FP8** on ana-ml2
|
||
**GPU 1**, fronted by the LiteLLM gateway as `qwen3.5-9b-fp8`. Image + video
|
||
understanding and chat. The language model is FP8; the **vision tower stays
|
||
BF16** (see below).
|
||
|
||
## Placement
|
||
- **GPU 1**, co-located with the granite summarizer + embed/rerank/reward trio.
|
||
**GPU 0 is deliberately kept free** for hot-reloading large models.
|
||
- Port **8007**. Gateway: `qwen3.5-9b-fp8` via LiteLLM (`ana-docker:4000`).
|
||
- Container `vllm-qwen35`, compose project `qwen35-vl`.
|
||
|
||
## Why a pinned nightly digest (not :latest)
|
||
vLLM `:latest` (v0.19.1) quantizes the Qwen3.5-VL **vision tower** under
|
||
`--quantization fp8` → garbage vision (the LM answers text fine but "sees"
|
||
noise — verified: it described the two-cats COCO image as "a 6×6 grid of gray
|
||
squares"). The **nightly** correctly excludes the vision tower from FP8, so
|
||
vision works while the LM still gets the FP8 throughput/VRAM win (BF16 vision
|
||
read perfectly: "two cats on a bright pink surface… two remote controls").
|
||
|
||
We pin the exact nightly digest (`sha256:49211ab2…`) for reproducibility — a
|
||
moving `:nightly` tag would silently change the engine. **WATCH:** when the
|
||
vision-FP8 exclusion lands in a stable release, re-pin to `:latest` and delete
|
||
this note.
|
||
|
||
## Why util 0.40
|
||
The model needs ~34 GB just to **start** at 32k context (FP8 weights + BF16
|
||
vision tower + CUDA-graph capture + 32k memory profiling). On shared GPU 1
|
||
(prod uses ~46 GB, ~48 GB free) this vLLM build requires `free >= util*total`,
|
||
capping util at ~0.51 here; **0.40 (~38 GB)** sits above the ~34 GB floor with
|
||
~10 GB card headroom and reports ~20× max concurrency at 32k. (Empty-GPU floor
|
||
was 0.35; below ~0.33 it crashes with "no KV blocks".) To shrink the footprint,
|
||
lower `QWEN_MAX_MODEL_LEN` (vision queries rarely need 32k) rather than util.
|
||
|
||
## Deploy
|
||
```
|
||
scripts/deploy-stack.sh ana-ml2 qwen35-vl # or scp compose to the host
|
||
# on ana-ml2: create /opt/docker/compose/qwen35-vl/.env from .env.example, then:
|
||
cd /opt/docker/compose/qwen35-vl && docker compose up -d
|
||
```
|
||
|
||
## Smoke test (incl. vision)
|
||
```bash
|
||
curl -s http://10.250.50.54:8007/v1/chat/completions -H 'Content-Type: application/json' \
|
||
-d '{"model":"qwen3.5-9b-fp8","messages":[{"role":"user","content":[
|
||
{"type":"text","text":"How many cats and what surface are they on?"},
|
||
{"type":"image_url","image_url":{"url":"http://images.cocodataset.org/val2017/000000039769.jpg"}}]}],
|
||
"max_tokens":120,"chat_template_kwargs":{"enable_thinking":false}}'
|
||
```
|
||
It's a **thinking** model (emits a reasoning trace by default) — pass
|
||
`chat_template_kwargs:{"enable_thinking":false}` for terse answers.
|
||
|
||
## Related
|
||
- The NVFP4 path for this model was abandoned — FP8 is the answer on Blackwell
|
||
(W4A4 collapses, weight-only 4-bit doesn't accelerate). The AxionML NVFP4
|
||
community quant + joninco SGLang fork were the NVFP4 attempt; not used.
|