vh
e124a2f233
Put GPU1's idle ~11 GB buffer to work on the two KV-bound models that gained live consumers from the worldtree migration (granite + the pooling models under-use their util, so growing them is wasted): - selene 0.13→0.17: KV 2.53→6.33 GiB, concurrency 1.27x→3.16x @32K (Domari judge) - qwen36 0.32→0.34: KV 7.73→9.63 GiB, concurrency 2.92x→3.64x @131K (arbo judge + worldtree actor/echo + gateway) GPU1 free now ~5.6 GB (safe floor for single-service recreates).
38 lines
1.9 KiB
Bash
38 lines
1.9 KiB
Bash
# Qwen3.6-35B-A3B VL (official NVFP4) on ana-ml2 — copy to .env on the host and fill.
|
||
# Real .env lives on ana-ml2 at /opt/docker/compose/qwen36-vl/.env (gitignored).
|
||
#
|
||
# Swapped FP8→NVFP4 2026-06-15 (the nvidia ModelOpt NVFP4 MoE now loads on vLLM
|
||
# 0.23.0 — #44081 fixed). See compose.yaml header for the full rationale + the
|
||
# comfy-dev vision A/B that cleared it + the GPU-1 rebalance.
|
||
|
||
# PINNED by digest — NVFP4 needs vLLM >= 0.23.0 (the related ModelOpt NVFP4 MoE
|
||
# path broke on 0.19.1/0.22.0). Pin guards against a :latest regression. This
|
||
# digest = 0.23.0, validated to load + serve this checkpoint (vision incl.).
|
||
QWEN_IMAGE=vllm/vllm-openai@sha256:6d8429e38e3747723ca07ee1b17972e09bb9c51c4032b266f24fb1cc3b22ed8f
|
||
|
||
QWEN_CONTAINER_NAME=vllm-qwen36
|
||
QWEN_MODEL=nvidia/Qwen3.6-35B-A3B-NVFP4
|
||
QWEN_PORT=8007
|
||
|
||
# GPU 1 = shared with the granite summarizer + embed/rerank/reward trio.
|
||
# (GPU 0 now hosts Mistral Small 4, not the old llama-swap card.)
|
||
QWEN_GPU_ID=1
|
||
|
||
# util 0.32 (~31 GB) — NVFP4 weights load in ~20.4 GiB; 0.32 covers weights +
|
||
# fp16 KV + CUDA-graph. fp16 KV (compose drops --kv-cache-dtype fp8): the NVFP4
|
||
# swap freed enough room to run full-precision KV. Hybrid attn (10/40 full-attn)
|
||
# keeps even fp16 KV cheap. GPU-1 budget (2026-06-15 rebalance, pinned): qwen36
|
||
# GPU-1 budget (2026-06-16): qwen36 0.34 (grown from 0.32 for the arbo-judge +
|
||
# worldtree actor/echo + gateway load) + granite 0.34/131072 + selene 0.17 + trio
|
||
# 0.16. Nominal sum >1.0 but the pooling models + granite under-use their util, so
|
||
# it fits with ~5-6 GB physical free. Recreate ONE service at a time (profiling race).
|
||
QWEN_GPU_MEM_UTIL=0.34
|
||
QWEN_MAX_MODEL_LEN=131072
|
||
# Sampler-warmup OOM guard on the shared GPU (248K vocab × default 1024 seqs is
|
||
# a huge transient). 32 is plenty for a vision endpoint.
|
||
QWEN_MAX_NUM_SEQS=32
|
||
|
||
# Optional — checkpoint is ungated.
|
||
HF_TOKEN=
|
||
API_KEY=
|