vh
c985ede07b
selene: AtlaAI Selene-1-Mini-Llama-3.1-8B judge restored on vLLM after the llama-swap teardown took its Q6_K GGUF offline. FP8 (dynamic --quantization fp8; FP8 >= the validated Q6_K fidelity, and text-only Llama so no vision- tower-noise risk; NVFP4's W4A4 too aggressive for a precision judge). GPU1 util 0.13 (8.51 GiB weights + 2.53 GiB KV, 32K ctx, 1.27x concurrency), ~11 GB GPU1 buffer left. Gateway selene-1-mini-8b → :8011 (shadows the * wildcard that used to reach it via llama-swap). Judge smoke: scored an unfaithful claim 1/5 correctly. mistral-small-4: max-model-len 131072 → 262144 (full native 256K) for novel-length consistency-checking. KV pool is util-bound (~862K tokens), so 256K costs no extra VRAM — max concurrency just drops to 3.29x at full length. max-num-seqs 64 → 32 keeps the warmup transient flat (scales with seqs × len), so it fits the tight GPU0 (free unchanged at 5.2 GB). Verified loaded + healthy.
40 lines
2.0 KiB
Bash
40 lines
2.0 KiB
Bash
# Mistral Small 4 (official NVFP4) on ana-ml2 GPU 0 — copy to .env on the host.
|
||
# Real .env lives on ana-ml2 at /opt/docker/compose/mistral-small-4/.env (gitignored).
|
||
#
|
||
# See compose.yaml header for the NVFP4/TP=1/MLA rationale and the vLLM>=0.20 floor.
|
||
|
||
# PINNED to v0.22.0 — the last release BEFORE the Mistral multimodal regression
|
||
# (#44911, MistralCommonImageProcessor.fetch_images, landed ~0.22.1; 0.23.0 is
|
||
# affected). v0.22.0 loads the NVFP4 (compressed-tensors) AND serves VISION —
|
||
# verified: half-blue/half-red image read correctly ("left blue, right red").
|
||
# This gives a working vision tower as the abliteration/tuning baseline. Do NOT
|
||
# bump to 0.23.0 (breaks vision). reasoning_effort works (none/high only) but
|
||
# reasoning_content-splitting is unreliable on this version — vision is the
|
||
# priority. Revisit when vLLM patches the Mistral mm path on a newer release.
|
||
MISTRAL_IMAGE=vllm/vllm-openai:v0.22.0
|
||
|
||
MISTRAL_CONTAINER_NAME=vllm-mistral4
|
||
MISTRAL_MODEL=mistralai/Mistral-Small-4-119B-2603-NVFP4
|
||
MISTRAL_PORT=8010
|
||
|
||
# GPU 0 = the free 96 GB Blackwell card, dedicated single-tenant to this model
|
||
# (74.4 GB weights leave no room to co-tenant). GPU 1 holds qwen36 + granite +
|
||
# the embed/rerank/reward trio.
|
||
MISTRAL_GPU_ID=0
|
||
|
||
# util 0.93 (~89 GB budget) — measured 66.1 GiB weights + 0.8 GiB graph + ~18.5 GiB
|
||
# KV. MLA keeps KV compressed, so the FULL native 256K context fits (verified).
|
||
MISTRAL_GPU_MEM_UTIL=0.93
|
||
# 262144 = native 256K (for novel-length consistency-checking). The KV pool is
|
||
# util-bound (~862K tokens) regardless of max-len, so 256K costs no extra VRAM —
|
||
# it just lets one request use up to 256K (→ max concurrency 3.29x at full length).
|
||
MISTRAL_MAX_MODEL_LEN=262144
|
||
# 32 (halved from 64 when going to 256K): the warmup transient scales with
|
||
# max-num-seqs × max-model-len, so halving seqs while doubling len keeps it flat
|
||
# and fits the tight card. 32 is ample — this is a low-concurrency creative model.
|
||
MISTRAL_MAX_NUM_SEQS=32
|
||
|
||
# Optional — model is ungated (Apache-2.0), no token needed.
|
||
HF_TOKEN=
|
||
API_KEY=
|