vh
f566f61b24
Serves the in-house abliterated Mistral Small 4 (heretic NVFP4, vision-intact) under --served-model-name mistral-small-4 on ana-ml2 GPU0:8010 — a true drop-in for the official mistral-small-4 backend. Both litellm entries (mistral-small-4 + mistral-small-4-reasoning) route here with no litellm change. GPU0 fits one mistral-class model, so this is a backend swap, not a co-tenant: bring up after downing the official stack; revert by downing this and up-ing the official. Verified live through the gateway: standard returns clean answers, reasoning populates reasoning_content (the [THINK] split). Checkpoint built per tools/mistral-small4-nvfp4/.
25 lines
1.0 KiB
Bash
25 lines
1.0 KiB
Bash
# mistral-small-4-heretic .env — copy to .env on ana-ml2 (gitignored).
|
|
# Easiest: copy the official stack's live .env and change MISTRAL_CONTAINER_NAME,
|
|
# so API_KEY (== litellm's VLLM_API_KEY) and HF_TOKEN carry over unchanged —
|
|
# that key match is what lets litellm authenticate to this backend with no edit.
|
|
|
|
# v0.22.0 = the last vLLM before the Mistral multimodal regression (#44911);
|
|
# verified working vision tower. Matches the official stack's pin.
|
|
MISTRAL_IMAGE=vllm/vllm-openai:v0.22.0
|
|
|
|
# Distinct from the official's vllm-mistral4 so both can exist on disk; only one
|
|
# runs at a time (shared GPU0 + port 8010).
|
|
MISTRAL_CONTAINER_NAME=vllm-mistral4-heretic
|
|
|
|
# Same port + GPU as the official → true drop-in after the official is downed.
|
|
MISTRAL_PORT=8010
|
|
MISTRAL_GPU_ID=0
|
|
MISTRAL_GPU_MEM_UTIL=0.93
|
|
MISTRAL_MAX_MODEL_LEN=262144
|
|
MISTRAL_MAX_NUM_SEQS=32
|
|
|
|
# Must equal the official stack's API_KEY (== litellm VLLM_API_KEY env). Carried
|
|
# over by copying the official .env. Real value lives on the host, gitignored.
|
|
API_KEY=
|
|
HF_TOKEN=
|