feat(mistral-small-4): deploy NVFP4 119B MoE on GPU 0 (text-only) + gateway

Mistral-Small-4-119B-2603-NVFP4 (119B/6.5B-active MoE, 65.3 GiB) on the
freed GPU 0 (dedicated 96 GB Blackwell), vLLM 0.23.0, :8010. NVFP4 is the
only variant that fits one card (FP8 ~119 GB / bf16 ~238 GB need 2 GPUs).

- TEXT-ONLY: vLLM 0.23.0's Mistral multimodal processor crashes at startup
  (fetch_images bug); loaded with --limit-mm-per-prompt image/video=0.
  Remove the flag to restore vision once vLLM patches it.
- MLA attn (TRITON_MLA), mistral tool-call + reasoning parsers, util 0.93,
  max-len 131072 (capped from native 256K), image pinned by 0.23.0 digest.
- litellm: mistral-small-4 → :8010, shadows the * wildcard.
- GPU 0 reassigned from the (now-offline) llama-swap zoo per operator.
This commit is contained in:
vh
2026-06-15 17:28:15 -07:00
parent c6d76051a4
commit c77a9aa4d8
3 changed files with 161 additions and 0 deletions
+13
View File
@@ -67,6 +67,19 @@ model_list:
model_info:
mode: chat
# --- Mistral Small 4 (official NVFP4) — creative-writing / general text. 119B
# MoE (6.5B active), vLLM on ana-ml2 GPU 0 (dedicated 96 GB Blackwell), :8010.
# Explicit entry shadows the "*" wildcard. TEXT-ONLY for now — vLLM 0.23.0's
# Mistral multimodal processor crashes at startup (loaded with image/video
# limit 0); vision returns when vLLM patches it. Deployed 2026-06-15. ---
- model_name: mistral-small-4
litellm_params:
model: hosted_vllm/mistral-small-4
api_base: http://10.250.50.54:8010/v1
api_key: os.environ/VLLM_API_KEY
model_info:
mode: chat
# --- Qwen3 embeddings ---
- model_name: qwen3-embedding
litellm_params: