vh
c77a9aa4d8
Mistral-Small-4-119B-2603-NVFP4 (119B/6.5B-active MoE, 65.3 GiB) on the freed GPU 0 (dedicated 96 GB Blackwell), vLLM 0.23.0, :8010. NVFP4 is the only variant that fits one card (FP8 ~119 GB / bf16 ~238 GB need 2 GPUs). - TEXT-ONLY: vLLM 0.23.0's Mistral multimodal processor crashes at startup (fetch_images bug); loaded with --limit-mm-per-prompt image/video=0. Remove the flag to restore vision once vLLM patches it. - MLA attn (TRITON_MLA), mistral tool-call + reasoning parsers, util 0.93, max-len 131072 (capped from native 256K), image pinned by 0.23.0 digest. - litellm: mistral-small-4 → :8010, shadows the * wildcard. - GPU 0 reassigned from the (now-offline) llama-swap zoo per operator.
33 lines
1.5 KiB
Bash
33 lines
1.5 KiB
Bash
# Mistral Small 4 (official NVFP4) on ana-ml2 GPU 0 — copy to .env on the host.
|
|
# Real .env lives on ana-ml2 at /opt/docker/compose/mistral-small-4/.env (gitignored).
|
|
#
|
|
# See compose.yaml header for the NVFP4/TP=1/MLA rationale and the vLLM>=0.20 floor.
|
|
|
|
# PINNED by digest, not :latest — this model is version-sensitive (needs vLLM
|
|
# >= 0.20 for day-0 support; the related nvidia-ModelOpt NVFP4 MoE path broke on
|
|
# 0.19.1/0.22.0). Pin protects against a :latest regression. This digest = vLLM
|
|
# 0.23.0, the version validated to load this checkpoint. Bump deliberately.
|
|
MISTRAL_IMAGE=vllm/vllm-openai@sha256:6d8429e38e3747723ca07ee1b17972e09bb9c51c4032b266f24fb1cc3b22ed8f
|
|
|
|
MISTRAL_CONTAINER_NAME=vllm-mistral4
|
|
MISTRAL_MODEL=mistralai/Mistral-Small-4-119B-2603-NVFP4
|
|
MISTRAL_PORT=8010
|
|
|
|
# GPU 0 = the free 96 GB Blackwell card, dedicated single-tenant to this model
|
|
# (74.4 GB weights leave no room to co-tenant). GPU 1 holds qwen36 + granite +
|
|
# the embed/rerank/reward trio.
|
|
MISTRAL_GPU_ID=0
|
|
|
|
# util 0.93 (~89 GB budget) — 74.4 GB weights + ~5 GB CUDA/graph leaves ~10 GB
|
|
# KV. MLA keeps KV compressed so 131072 ctx fits; raise toward native 256K only
|
|
# after measuring real KV headroom. Dedicated card, so 0.93 is safe.
|
|
MISTRAL_GPU_MEM_UTIL=0.93
|
|
MISTRAL_MAX_MODEL_LEN=131072
|
|
# Single-card KV is tighter than the official TP=2 setup → cap concurrency at 64
|
|
# (official used 128 across two cards).
|
|
MISTRAL_MAX_NUM_SEQS=64
|
|
|
|
# Optional — model is ungated (Apache-2.0), no token needed.
|
|
HF_TOKEN=
|
|
API_KEY=
|