From c6d76051a432b54e3e1821379ae3632834eafdbf Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Mon, 15 Jun 2026 17:28:15 -0700 Subject: [PATCH] =?UTF-8?q?feat(qwen36-vl):=20swap=20FP8=E2=86=92NVFP4=20+?= =?UTF-8?q?=20GPU1=20rebalance=20(granite=20restored)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The nvidia ModelOpt NVFP4 MoE that failed on vLLM 0.19.1/0.22.0 (#44081) loads clean on 0.23.0. Cut prod qwen36 FP8→NVFP4: ~20.4 GiB weights vs ~34 (~40% lighter, ~13 GB reclaimed on GPU 1), faster single-stream on Blackwell FP4 cores, vision tower preserved (comfy-dev real anatomy-judge A/B on 16 prod images: PASS; brokkr text/speed: parity bar a minor multi-step-chained-reasoning slip that doesn't bite the judge role). - compose: pin image by 0.23.0 digest, drop --kv-cache-dtype fp8 (fp16 KV — the freed room buys full-precision KV), util 0.46→0.32. - GPU1 rebalance (pinned): granite restored 0.24→0.34 / 65536→131072 (undoes the FP8-era sacrifice); trio unchanged; total ~0.82, ~24 GB free. - gateway model name qwen3.6-35b-a3b unchanged (now NVFP4 behind it); thinking-split (enable_thinking=false default) intact — the judge needs it. --- stacks/qwen36-vl/.env.example | 35 ++++++++++++----------- stacks/qwen36-vl/compose.yaml | 53 ++++++++++++++++++++--------------- 2 files changed, 48 insertions(+), 40 deletions(-) diff --git a/stacks/qwen36-vl/.env.example b/stacks/qwen36-vl/.env.example index 0771721..020fb85 100644 --- a/stacks/qwen36-vl/.env.example +++ b/stacks/qwen36-vl/.env.example @@ -1,34 +1,35 @@ -# Qwen3.6-35B-A3B VL (official FP8) on ana-ml2 — copy to .env on the host and fill. +# Qwen3.6-35B-A3B VL (official NVFP4) on ana-ml2 — copy to .env on the host and fill. # Real .env lives on ana-ml2 at /opt/docker/compose/qwen36-vl/.env (gitignored). # -# Replaces qwen35-vl (Qwen3.5-9B) 2026-06-14. See compose.yaml header for the -# FP8-over-NVFP4 rationale (vLLM NVFP4 MoE loader broken, #44081) and why this -# uses plain :latest with NO --quantization (pre-quantized checkpoint; a forced -# flag would noise-quantize the vision tower like the old qwen35-vl). +# Swapped FP8→NVFP4 2026-06-15 (the nvidia ModelOpt NVFP4 MoE now loads on vLLM +# 0.23.0 — #44081 fixed). See compose.yaml header for the full rationale + the +# comfy-dev vision A/B that cleared it + the GPU-1 rebalance. -# :latest is fine — the official FP8 checkpoint loads + serves vision correctly -# on 0.19.1 (validated 2026-06-14). NO pinned nightly digest needed. -QWEN_IMAGE=vllm/vllm-openai:latest +# PINNED by digest — NVFP4 needs vLLM >= 0.23.0 (the related ModelOpt NVFP4 MoE +# path broke on 0.19.1/0.22.0). Pin guards against a :latest regression. This +# digest = 0.23.0, validated to load + serve this checkpoint (vision incl.). +QWEN_IMAGE=vllm/vllm-openai@sha256:6d8429e38e3747723ca07ee1b17972e09bb9c51c4032b266f24fb1cc3b22ed8f QWEN_CONTAINER_NAME=vllm-qwen36 -QWEN_MODEL=Qwen/Qwen3.6-35B-A3B-FP8 +QWEN_MODEL=nvidia/Qwen3.6-35B-A3B-NVFP4 QWEN_PORT=8007 # GPU 1 = shared with the granite summarizer + embed/rerank/reward trio. -# GPU 0 is kept free for the llama-swap creative-writing hot-swap card. +# (GPU 0 now hosts Mistral Small 4, not the old llama-swap card.) QWEN_GPU_ID=1 -# util 0.42 (~40 GB) — official FP8 weights load in ~34.2 GiB; 0.42 covers -# weights + CUDA-graph + a generous KV pool. Hybrid attn (10 of 40 layers full- -# attn, ~10 KB/tok KV) makes long context nearly free, so max-len is generous. -# Budget (2026-06-14): qwen36 0.42 + granite 0.28 + trio 0.20 = 0.90 total, -# ~10 GB graph headroom. Bring qwen36 up LAST so capture sees the free room. -QWEN_GPU_MEM_UTIL=0.42 +# util 0.32 (~31 GB) — NVFP4 weights load in ~20.4 GiB; 0.32 covers weights + +# fp16 KV + CUDA-graph. fp16 KV (compose drops --kv-cache-dtype fp8): the NVFP4 +# swap freed enough room to run full-precision KV. Hybrid attn (10/40 full-attn) +# keeps even fp16 KV cheap. GPU-1 budget (2026-06-15 rebalance, pinned): qwen36 +# 0.32 + granite 0.34/131072 (RESTORED from the FP8-era 0.24/64K) + trio 0.16 = +# ~0.82, ~24 GB free headroom. Recreate ONE service at a time (profiling race). +QWEN_GPU_MEM_UTIL=0.32 QWEN_MAX_MODEL_LEN=131072 # Sampler-warmup OOM guard on the shared GPU (248K vocab × default 1024 seqs is # a huge transient). 32 is plenty for a vision endpoint. QWEN_MAX_NUM_SEQS=32 -# Optional +# Optional — checkpoint is ungated. HF_TOKEN= API_KEY= diff --git a/stacks/qwen36-vl/compose.yaml b/stacks/qwen36-vl/compose.yaml index 38bcf5f..d232465 100644 --- a/stacks/qwen36-vl/compose.yaml +++ b/stacks/qwen36-vl/compose.yaml @@ -1,19 +1,22 @@ -# qwen36-vl — Qwen3.6-35B-A3B vision-language MoE (official FP8) on ana-ml2. +# qwen36-vl — Qwen3.6-35B-A3B vision-language MoE (official NVFP4) on ana-ml2. # # Replaces the qwen35-vl stack (Qwen3.5-9B) 2026-06-14. Co-located on GPU 1 with -# the granite summarizer + embed/rerank/reward trio (GPU 0 stays free for the -# llama-swap creative-writing hot-swap card). Serves on :8007. +# the granite summarizer + embed/rerank/reward trio. Serves on :8007. # -# WHY official FP8 (not NVFP4): NVFP4 (nvidia/Qwen3.6-35B-A3B-NVFP4, ~21 GB) is the -# lighter fit but its vLLM ModelOpt-MoE loader is BROKEN as of 0.19.1/0.22.0 -# (KeyError w2_input_scale / lm_head.input_scale — vLLM #44081). The official -# Qwen pre-quantized FP8 (~34 GB weights) loads clean on :latest and — unlike -# the old qwen35-vl — needs NO pinned nightly digest: that hack existed because -# vLLM DYNAMIC `--quantization fp8` quantized the vision tower to noise. This -# checkpoint is PRE-quantized, so we OMIT --quantization (vLLM auto-detects the -# checkpoint's own fp8) and the vision tower is preserved. Validated 2026-06-14 -# on GPU 0: loads in 34.2 GiB, image test returns correct ("Blue"). Revisit -# NVFP4 (frees ~13 GB) once vLLM's loader is fixed. +# WHY NVFP4 now (swapped FROM FP8 2026-06-15): the nvidia ModelOpt NVFP4 MoE that +# was BROKEN on vLLM 0.19.1/0.22.0 (#44081, lm_head.input_scale) loads clean on +# 0.23.0 — the ModelOpt lm_head fix landed. So we cut FP8→NVFP4: ~20.4 GiB weights +# vs FP8's ~34 GiB (~40% lighter, ~13 GB reclaimed on GPU 1), faster single-stream +# on Blackwell's FP4 tensor cores, and the freed room funds fp16 KV + a granite +# context restore (see the rebalance note below). The NVFP4 checkpoint preserves +# the vision tower (ModelOpt leaves it high-precision) — VALIDATED by comfy-dev's +# real anatomy-judge A/B on 16 prod images: PASS, holds the load-bearing +# discrimination (gross-deformity reject + clean-pass), only shuffles already- +# unreliable sub-ceiling borderline-hand calls. brokkr's text/speed arm: parity +# except a minor multi-step chained-numeric-reasoning slip (W4A4 tell) — doesn't +# bite the vision-judge role; flag for any gateway consumer doing chained math. +# Requires vLLM >= 0.23.0 (pinned by digest in .env). NO --quantization flag +# (vLLM auto-detects the checkpoint's NVFP4). # # NAMING: served ONLY as its TRUE name `qwen3.6-35b-a3b`. A model is never aliased # under a prior model's name — a caller asking for `qwen3.5-9b-fp8` (a 9B dense) @@ -22,10 +25,14 @@ # hero-judge, stacks/arbo v0.11.3+) migrate to `qwen3.6-35b-a3b` — they 404 on the # old name until they repoint, which is the correct loud signal (notified 2026-06-14). # -# WHY util 0.42 / max-len 131072: hybrid attn (10 of 40 layers full-attn, ~10 KB/ -# tok KV) → KV is cheap, so big context is nearly free; the 34 GB weights are the -# cost. 0.42 (~40 GB) = weights + graph + generous KV. Granite drops to 0.25/64K -# to make room (the FP8-vs-maxed-granite tradeoff, operator-approved 2026-06-14). +# GPU-1 REBALANCE (2026-06-15, pinned): the NVFP4 swap freed ~13 GB, redistributed — +# qwen36 NVFP4: util 0.46→0.32 (~31 GB: 20.4 GB weights + fp16 KV + graph). +# fp16 KV (we DROPPED --kv-cache-dtype fp8) — the freed room buys back full- +# precision KV; hybrid attn (10/40 full-attn) keeps even fp16 KV affordable. +# granite: RESTORED 0.24→0.34, max-len 65536→131072 (gives back the context +# sacrificed for FP8 qwen — the FP8-vs-maxed-granite tradeoff is now undone). +# trio (embed/rerank/reward) unchanged at floor. +# Total GPU-1 util ~0.82 → ~17 GB headroom (was a tight ~5 GB). # # THINKING TOGGLE: this is ONE hybrid checkpoint (not separate Instruct/Thinking # downloads) with a Qwen3-style per-request `enable_thinking` switch. The chat @@ -60,8 +67,8 @@ services: - VLLM_API_KEY=${API_KEY:-} command: - ${QWEN_MODEL} - # Pre-quantized FP8 checkpoint → NO --quantization (vLLM auto-detects; a - # forced flag would re-quantize the vision tower to noise, see header). + # Pre-quantized NVFP4 (ModelOpt) checkpoint → NO --quantization (vLLM auto- + # detects; the vision tower is left high-precision by the producer). - --served-model-name - qwen3.6-35b-a3b - --host @@ -78,8 +85,8 @@ services: # vision endpoint (the summarizer carries the concurrency, not this). - --max-num-seqs - ${QWEN_MAX_NUM_SEQS} - - --kv-cache-dtype - - fp8 + # fp16 KV (no --kv-cache-dtype): the NVFP4 swap freed enough room to run + # full-precision KV — better than the fp8 KV the FP8 build needed to fit. - --trust-remote-code - --dtype - auto @@ -110,9 +117,9 @@ services: - tnet labels: - homepage.group=AI Systems - - homepage.name=Qwen3.6-35B-A3B VL (FP8) + - homepage.name=Qwen3.6-35B-A3B VL (NVFP4) - homepage.icon=mdi-image-search - - homepage.description=Qwen3.6-35B-A3B vision-language MoE (FP8) via vLLM (ana-ml2) + - homepage.description=Qwen3.6-35B-A3B vision-language MoE (NVFP4) via vLLM (ana-ml2) - homepage.href=http://10.250.50.54:${QWEN_PORT}/docs networks: