Files

T

vh 2e3dcc2d3d feat(qwen35-vl): Qwen3.5-9B VL FP8 stack on ana-ml2 GPU1 + LiteLLM entry

Qwen3.5-9B vision-language served FP8 on ana-ml2 GPU1 (co-located with
granite + the embed/rerank/reward trio; GPU0 kept free for hot-loading
large models), :8007, fronted by LiteLLM as qwen3.5-9b-fp8.

Pinned to vllm/vllm-openai nightly@sha256:49211ab2 — :latest (v0.19.1)
quantizes the VL vision tower under fp8 and garbles vision; the nightly
correctly excludes it (LM stays FP8, vision tower BF16). util 0.40
(~38GB) on the shared card (vLLM needs free>=util*total here). Vision
verified end-to-end through the gateway.

2026-06-13 02:39:33 -07:00

2.9 KiB

Raw Blame History

qwen35-vl — Qwen3.5-9B vision-language (FP8) on ana-ml2

Qwen3.5-9B, a hybrid GDN + vision-language model, served FP8 on ana-ml2 GPU 1, fronted by the LiteLLM gateway as qwen3.5-9b-fp8. Image + video understanding and chat. The language model is FP8; the vision tower stays BF16 (see below).

Placement

GPU 1, co-located with the granite summarizer + embed/rerank/reward trio. GPU 0 is deliberately kept free for hot-reloading large models.
Port 8007. Gateway: qwen3.5-9b-fp8 via LiteLLM (ana-docker:4000).
Container vllm-qwen35, compose project qwen35-vl.

Why a pinned nightly digest (not :latest)

vLLM :latest (v0.19.1) quantizes the Qwen3.5-VL vision tower under --quantization fp8 → garbage vision (the LM answers text fine but "sees" noise — verified: it described the two-cats COCO image as "a 6×6 grid of gray squares"). The nightly correctly excludes the vision tower from FP8, so vision works while the LM still gets the FP8 throughput/VRAM win (BF16 vision read perfectly: "two cats on a bright pink surface… two remote controls").

We pin the exact nightly digest (sha256:49211ab2…) for reproducibility — a moving :nightly tag would silently change the engine. WATCH: when the vision-FP8 exclusion lands in a stable release, re-pin to :latest and delete this note.

Why util 0.40

The model needs ~34 GB just to start at 32k context (FP8 weights + BF16 vision tower + CUDA-graph capture + 32k memory profiling). On shared GPU 1 (prod uses ~46 GB, ~48 GB free) this vLLM build requires free >= util*total, capping util at ~0.51 here; 0.40 (~38 GB) sits above the ~34 GB floor with ~10 GB card headroom and reports ~20× max concurrency at 32k. (Empty-GPU floor was 0.35; below ~0.33 it crashes with "no KV blocks".) To shrink the footprint, lower QWEN_MAX_MODEL_LEN (vision queries rarely need 32k) rather than util.

Deploy

scripts/deploy-stack.sh ana-ml2 qwen35-vl     # or scp compose to the host
# on ana-ml2: create /opt/docker/compose/qwen35-vl/.env from .env.example, then:
cd /opt/docker/compose/qwen35-vl && docker compose up -d

Smoke test (incl. vision)

curl -s http://10.250.50.54:8007/v1/chat/completions -H 'Content-Type: application/json' \
  -d '{"model":"qwen3.5-9b-fp8","messages":[{"role":"user","content":[
    {"type":"text","text":"How many cats and what surface are they on?"},
    {"type":"image_url","image_url":{"url":"http://images.cocodataset.org/val2017/000000039769.jpg"}}]}],
    "max_tokens":120,"chat_template_kwargs":{"enable_thinking":false}}'

It's a thinking model (emits a reasoning trace by default) — pass chat_template_kwargs:{"enable_thinking":false} for terse answers.

The NVFP4 path for this model was abandoned — FP8 is the answer on Blackwell (W4A4 collapses, weight-only 4-bit doesn't accelerate). The AxionML NVFP4 community quant + joninco SGLang fork were the NVFP4 attempt; not used.

2.9 KiB Raw Blame History Unescape Escape