feat(selene): retire the seat; chat-judge -> gen, selene-1-mini-8b 404s by design
Benchmarked selene against gen on selene's own job: 24 designed judge items with checkable ground truth, pairwise + absolute modes, 3 repeats, on BOTH a neutral JSON prompt and Selene's native Atla template. 288 calls, all free local. neutral JSON selene 20/24 (83%) gen 23/24 (96%) native Atla selene 21/24 (88%) gen 22/24 (92%) gen won on both templates and selene's BEST sat below gen's WORST. Selene was given its own fine-tuned template as a fairness check; it gained one point, not the three it needed. Decisive defect: selene cannot emit "tie" -- 0/2 on both templates, forcing a winner on every equivalent pair. For eval work that is the case that matters most. gen returned tie correctly on the JSON template. Selene also compressed the 1-5 scale (clustered at 2s and 4s) where gen used it fully. Selene's only win was ~3x latency, unexercised at ~60 calls/day with zero queueing. TWO NAMES, TWO DIFFERENT TREATMENTS, deliberately: - chat-judge -> repointed to gen. It is a ROLE alias and ADR-0012 says consumers bind the capability, not a concrete model. Sampler profile copied from image-judge (temp 0, top_p 1.0, top_k 1, thinking off) so the served config matches the benchmarked condition. - selene-1-mini-8b -> REMOVED. It 404s. It was NOT aliased to gen. A served-name is a contract about what the model IS; answering it with a different model hides a material change behind a stable string. Operator ruling: "never repoint a named model at a different model's endpoint -- that is intentionally misleading." Verified: the gateway now returns HTTP 400 "Invalid model name" for it. Reclaimed 17.2 GiB on ana-ml2 GPU 1 (free 1,818 -> 19,450 MiB) on a card that had under 2 GiB of headroom. gen already runs on GPU 0, so the judge role moved onto an existing seat rather than allocating anything new. Canonical litellm config synced from the host; ana-ml2 README and recommended-model-settings updated. compose.yaml kept for reference, not deployed.
This commit is contained in:
@@ -337,19 +337,24 @@ model_list:
|
||||
model_info:
|
||||
mode: chat
|
||||
|
||||
# --- Selene 1 Mini 8B (AtlaAI judge, FP8) — restored on GPU1 after the
|
||||
# llama-swap teardown (was the Q6_K GGUF in the swap zoo). vLLM dynamic fp8,
|
||||
# :8011. Explicit entry shadows the "*" wildcard (which used to reach it via
|
||||
# llama-swap). Hallucination/RAG-faithfulness judge; callers set temp ~0.01. ---
|
||||
- model_name: selene-1-mini-8b
|
||||
litellm_params:
|
||||
model: hosted_vllm/selene-1-mini-8b
|
||||
api_base: http://10.250.50.54:8011/v1
|
||||
api_key: os.environ/VLLM_API_KEY
|
||||
temperature: 0.6
|
||||
top_p: 0.9
|
||||
model_info:
|
||||
mode: chat
|
||||
# --- selene-1-mini-8b RETIRED 2026-08-23. AtlaAI Selene 1 Mini (Llama 3.1 8B,
|
||||
# dynamic FP8) on ana-ml2 GPU1 :8011. Benchmarked head-to-head against `gen`
|
||||
# on its OWN job: 24 designed judge items with checkable ground truth,
|
||||
# pairwise + absolute scoring, 3 repeats, run on BOTH a neutral JSON prompt
|
||||
# and Selene's native Atla template (288 calls total). gen won on both —
|
||||
# 23/24 vs 20/24 neutral, 22/24 vs 21/24 native. Selene's BEST score sat
|
||||
# below gen's WORST. Decisive defect: it cannot emit "tie", forcing a winner
|
||||
# on every equivalent pair (0/2 on both templates), which is fatal for eval
|
||||
# work where close pairs are the whole point. Seat downed to reclaim 17.2 GiB
|
||||
# on GPU1 (the card had 1.8 GiB free). Its only edge was ~3x lower latency,
|
||||
# unexercised at its observed ~60 calls/day with zero queueing.
|
||||
#
|
||||
# DELIBERATELY NOT ALIASED TO ANOTHER MODEL. A caller asking for
|
||||
# `selene-1-mini-8b` must never silently receive qwen3.8-27b — a served-name
|
||||
# is a contract about what the model IS, and a silent substitution hides a
|
||||
# material change behind a stable string. This name now 404s BY DESIGN.
|
||||
# Repoint consumers to `chat-judge` (the role alias, below) or to `gen`
|
||||
# explicitly. Operator ruling 2026-08-23. ---
|
||||
|
||||
# --- Qwen3 embeddings ---
|
||||
- model_name: qwen3-embedding
|
||||
@@ -374,14 +379,26 @@ model_list:
|
||||
# model; swap the backing model here and callers are unaffected. NO generic
|
||||
# `embedding` alias ON PURPOSE — embedding vectors are model-specific (not
|
||||
# swap-transparent), so that capability stays `qwen3-embedding` above. ---
|
||||
# chat-judge → generative LLM-as-judge (Selene); WT selene-judgment role.
|
||||
# chat-judge → generative LLM-as-judge; WT selene-judgment role. Backed by
|
||||
# Selene until 2026-08-23, now `gen` / qwen3.8-27b-uncensored (:8015) after
|
||||
# Selene lost the head-to-head on its own job and its seat was reclaimed.
|
||||
# This is precisely the ADR-0012 case stated above: the ROLE alias moves, the
|
||||
# MODEL NAME does not — which is why selene-1-mini-8b was retired outright
|
||||
# rather than repointed. Sampler profile copied from image-judge
|
||||
# (deterministic, thinking off); the benchmark that selected gen ran at
|
||||
# temperature 0, so this matches the tested condition.
|
||||
- model_name: chat-judge
|
||||
litellm_params:
|
||||
model: hosted_vllm/selene-1-mini-8b
|
||||
api_base: http://10.250.50.54:8011/v1
|
||||
model: hosted_vllm/qwen3.8-27b-uncensored
|
||||
api_base: http://10.250.50.54:8015/v1
|
||||
api_key: os.environ/VLLM_API_KEY
|
||||
temperature: 0.6
|
||||
top_p: 0.9
|
||||
temperature: 0
|
||||
top_p: 1.0
|
||||
extra_body:
|
||||
top_k: 1
|
||||
repetition_penalty: 1.05
|
||||
chat_template_kwargs:
|
||||
enable_thinking: false
|
||||
model_info:
|
||||
mode: chat
|
||||
# reranker → generic capability name for rerank (currently qwen3-reranker).
|
||||
|
||||
Reference in New Issue
Block a user