feat(selene+mistral): restore Selene judge (FP8, GPU1) + push Mistral to 256K
selene: AtlaAI Selene-1-Mini-Llama-3.1-8B judge restored on vLLM after the llama-swap teardown took its Q6_K GGUF offline. FP8 (dynamic --quantization fp8; FP8 >= the validated Q6_K fidelity, and text-only Llama so no vision- tower-noise risk; NVFP4's W4A4 too aggressive for a precision judge). GPU1 util 0.13 (8.51 GiB weights + 2.53 GiB KV, 32K ctx, 1.27x concurrency), ~11 GB GPU1 buffer left. Gateway selene-1-mini-8b → :8011 (shadows the * wildcard that used to reach it via llama-swap). Judge smoke: scored an unfaithful claim 1/5 correctly. mistral-small-4: max-model-len 131072 → 262144 (full native 256K) for novel-length consistency-checking. KV pool is util-bound (~862K tokens), so 256K costs no extra VRAM — max concurrency just drops to 3.29x at full length. max-num-seqs 64 → 32 keeps the warmup transient flat (scales with seqs × len), so it fits the tight GPU0 (free unchanged at 5.2 GB). Verified loaded + healthy.
This commit is contained in:
@@ -95,6 +95,18 @@ model_list:
|
||||
model_info:
|
||||
mode: chat
|
||||
|
||||
# --- Selene 1 Mini 8B (AtlaAI judge, FP8) — restored on GPU1 after the
|
||||
# llama-swap teardown (was the Q6_K GGUF in the swap zoo). vLLM dynamic fp8,
|
||||
# :8011. Explicit entry shadows the "*" wildcard (which used to reach it via
|
||||
# llama-swap). Hallucination/RAG-faithfulness judge; callers set temp ~0.01. ---
|
||||
- model_name: selene-1-mini-8b
|
||||
litellm_params:
|
||||
model: hosted_vllm/selene-1-mini-8b
|
||||
api_base: http://10.250.50.54:8011/v1
|
||||
api_key: os.environ/VLLM_API_KEY
|
||||
model_info:
|
||||
mode: chat
|
||||
|
||||
# --- Qwen3 embeddings ---
|
||||
- model_name: qwen3-embedding
|
||||
litellm_params:
|
||||
|
||||
Reference in New Issue
Block a user