feat(mistral-small-4): deploy NVFP4 119B MoE on GPU 0 (text-only) + gateway
Mistral-Small-4-119B-2603-NVFP4 (119B/6.5B-active MoE, 65.3 GiB) on the freed GPU 0 (dedicated 96 GB Blackwell), vLLM 0.23.0, :8010. NVFP4 is the only variant that fits one card (FP8 ~119 GB / bf16 ~238 GB need 2 GPUs). - TEXT-ONLY: vLLM 0.23.0's Mistral multimodal processor crashes at startup (fetch_images bug); loaded with --limit-mm-per-prompt image/video=0. Remove the flag to restore vision once vLLM patches it. - MLA attn (TRITON_MLA), mistral tool-call + reasoning parsers, util 0.93, max-len 131072 (capped from native 256K), image pinned by 0.23.0 digest. - litellm: mistral-small-4 → :8010, shadows the * wildcard. - GPU 0 reassigned from the (now-offline) llama-swap zoo per operator.
This commit is contained in:
@@ -67,6 +67,19 @@ model_list:
|
||||
model_info:
|
||||
mode: chat
|
||||
|
||||
# --- Mistral Small 4 (official NVFP4) — creative-writing / general text. 119B
|
||||
# MoE (6.5B active), vLLM on ana-ml2 GPU 0 (dedicated 96 GB Blackwell), :8010.
|
||||
# Explicit entry shadows the "*" wildcard. TEXT-ONLY for now — vLLM 0.23.0's
|
||||
# Mistral multimodal processor crashes at startup (loaded with image/video
|
||||
# limit 0); vision returns when vLLM patches it. Deployed 2026-06-15. ---
|
||||
- model_name: mistral-small-4
|
||||
litellm_params:
|
||||
model: hosted_vllm/mistral-small-4
|
||||
api_base: http://10.250.50.54:8010/v1
|
||||
api_key: os.environ/VLLM_API_KEY
|
||||
model_info:
|
||||
mode: chat
|
||||
|
||||
# --- Qwen3 embeddings ---
|
||||
- model_name: qwen3-embedding
|
||||
litellm_params:
|
||||
|
||||
Reference in New Issue
Block a user