# qwen35-vl — Qwen3.5-9B vision-language (FP8) on ana-ml2

Qwen3.5-9B, a hybrid GDN + vision-language model, served **FP8** on ana-ml2
**GPU 1**, fronted by the LiteLLM gateway as `qwen3.5-9b-fp8`. Image + video
understanding and chat. The language model is FP8; the **vision tower stays
BF16** (see below).

## Placement
- **GPU 1**, co-located with the granite summarizer + embed/rerank/reward trio.
  **GPU 0 is deliberately kept free** for hot-reloading large models.
- Port **8007**. Gateway: `qwen3.5-9b-fp8` via LiteLLM (`ana-docker:4000`).
- Container `vllm-qwen35`, compose project `qwen35-vl`.

## Why a pinned nightly digest (not :latest)
vLLM `:latest` (v0.19.1) quantizes the Qwen3.5-VL **vision tower** under
`--quantization fp8` → garbage vision (the LM answers text fine but "sees"
noise — verified: it described the two-cats COCO image as "a 6×6 grid of gray
squares"). The **nightly** correctly excludes the vision tower from FP8, so
vision works while the LM still gets the FP8 throughput/VRAM win (BF16 vision
read perfectly: "two cats on a bright pink surface… two remote controls").

We pin the exact nightly digest (`sha256:49211ab2…`) for reproducibility — a
moving `:nightly` tag would silently change the engine. **WATCH:** when the
vision-FP8 exclusion lands in a stable release, re-pin to `:latest` and delete
this note.

## Why util 0.40
The model needs ~34 GB just to **start** at 32k context (FP8 weights + BF16
vision tower + CUDA-graph capture + 32k memory profiling). On shared GPU 1
(prod uses ~46 GB, ~48 GB free) this vLLM build requires `free >= util*total`,
capping util at ~0.51 here; **0.40 (~38 GB)** sits above the ~34 GB floor with
~10 GB card headroom and reports ~20× max concurrency at 32k. (Empty-GPU floor
was 0.35; below ~0.33 it crashes with "no KV blocks".) To shrink the footprint,
lower `QWEN_MAX_MODEL_LEN` (vision queries rarely need 32k) rather than util.

## Deploy
```
scripts/deploy-stack.sh ana-ml2 qwen35-vl     # or scp compose to the host
# on ana-ml2: create /opt/docker/compose/qwen35-vl/.env from .env.example, then:
cd /opt/docker/compose/qwen35-vl && docker compose up -d
```

## Smoke test (incl. vision)
```bash
curl -s http://10.250.50.54:8007/v1/chat/completions -H 'Content-Type: application/json' \
  -d '{"model":"qwen3.5-9b-fp8","messages":[{"role":"user","content":[
    {"type":"text","text":"How many cats and what surface are they on?"},
    {"type":"image_url","image_url":{"url":"http://images.cocodataset.org/val2017/000000039769.jpg"}}]}],
    "max_tokens":120,"chat_template_kwargs":{"enable_thinking":false}}'
```
It's a **thinking** model (emits a reasoning trace by default) — pass
`chat_template_kwargs:{"enable_thinking":false}` for terse answers.

## Related
- The NVFP4 path for this model was abandoned — FP8 is the answer on Blackwell
  (W4A4 collapses, weight-only 4-bit doesn't accelerate). The AxionML NVFP4
  community quant + joninco SGLang fork were the NVFP4 attempt; not used.