feat(selene): retire the seat; chat-judge -> gen, selene-1-mini-8b 404s by design
Benchmarked selene against gen on selene's own job: 24 designed judge items with checkable ground truth, pairwise + absolute modes, 3 repeats, on BOTH a neutral JSON prompt and Selene's native Atla template. 288 calls, all free local. neutral JSON selene 20/24 (83%) gen 23/24 (96%) native Atla selene 21/24 (88%) gen 22/24 (92%) gen won on both templates and selene's BEST sat below gen's WORST. Selene was given its own fine-tuned template as a fairness check; it gained one point, not the three it needed. Decisive defect: selene cannot emit "tie" -- 0/2 on both templates, forcing a winner on every equivalent pair. For eval work that is the case that matters most. gen returned tie correctly on the JSON template. Selene also compressed the 1-5 scale (clustered at 2s and 4s) where gen used it fully. Selene's only win was ~3x latency, unexercised at ~60 calls/day with zero queueing. TWO NAMES, TWO DIFFERENT TREATMENTS, deliberately: - chat-judge -> repointed to gen. It is a ROLE alias and ADR-0012 says consumers bind the capability, not a concrete model. Sampler profile copied from image-judge (temp 0, top_p 1.0, top_k 1, thinking off) so the served config matches the benchmarked condition. - selene-1-mini-8b -> REMOVED. It 404s. It was NOT aliased to gen. A served-name is a contract about what the model IS; answering it with a different model hides a material change behind a stable string. Operator ruling: "never repoint a named model at a different model's endpoint -- that is intentionally misleading." Verified: the gateway now returns HTTP 400 "Invalid model name" for it. Reclaimed 17.2 GiB on ana-ml2 GPU 1 (free 1,818 -> 19,450 MiB) on a card that had under 2 GiB of headroom. gen already runs on GPU 0, so the judge role moved onto an existing seat rather than allocating anything new. Canonical litellm config synced from the host; ana-ml2 README and recommended-model-settings updated. compose.yaml kept for reference, not deployed.
This commit is contained in:
@@ -560,7 +560,8 @@ override the config default.) Values set per the `dvalin-smithy-dev` research pa
|
||||
| `gen`, `summarizer-large`, `qwen-large`, `qwen3.5-122-a10b` (non-thinking) | **0.7** | 0.8 | 20 | **1.0** | — | Qwen3 non-thinking + operator anti-repetition |
|
||||
| `gen-reasoning`, `qwen-large-reasoning`, `qwen3.5-122-a10b-reasoning` (thinking) | **0.6** | 0.95 | 20 | **1.0** | — | Qwen3 thinking |
|
||||
| `qwen-image-bench`, `image-judge` | **0** | 1.0 | 1 | — | 1.05 | Qwen-Image-Bench judge reproducibility table |
|
||||
| `selene-1-mini-8b`, `chat-judge` | **0.6** | 0.9 | — | — | — | Selene `generation_config` |
|
||||
| ~~`selene-1-mini-8b`~~ | — | — | — | — | — | **RETIRED 2026-08-23**; name 404s by design, not aliased |
|
||||
| `chat-judge` | **0** | 1.0 | 1 | — | 1.05 | Repointed to `gen` 2026-08-23; deterministic judge profile copied from `image-judge`. The benchmark that selected `gen` ran at temperature 0 — match it. |
|
||||
| `glm-5.1`, `glm-5.2`, `glm-5-turbo`, `glm-4.7`, `gen-frontier` | **1.0** | 0.95 | — | — | — | z.ai API defaults (5.x / 4.7 series) |
|
||||
| `glm-4.5-air` | **0.6** | 0.95 | — | — | — | z.ai API default (4.5 series) |
|
||||
| `qwen3-embedding`, `qwen3-reranker`, `reranker` | — | — | — | — | — | no sampling (embedding / rerank) |
|
||||
|
||||
@@ -50,7 +50,7 @@ embed/rerank/reward trio. GPUs are pinned per container via
|
||||
|-----------|------|--------------|-------|-----|
|
||||
| `vllm-granite` | 8004 | `granite-4.1-8b` — fleet summarizer/classifier | FP8 (compressed-tensors) | 131k |
|
||||
| `llama-charrp` | 8016 | `Magidonia-24B-v4.3` Q6_K — char-RP (llama.cpp) | GGUF Q6_K | — |
|
||||
| `vllm-selene` | 8011 | `selene-1-mini-8b` — Atla LLM-as-judge | FP8 | 32k |
|
||||
| ~~`vllm-selene`~~ | ~~8011~~ | **RETIRED 2026-08-23** — lost a head-to-head against `gen` on its own judge task (see `stacks/selene/README.md`); seat downed to reclaim 17.2 GiB on GPU 1. `selene-1-mini-8b` now 404s by design; use `chat-judge`. | — | — |
|
||||
| `vllm-reward` | 8003 | `Skywork-Reward-V2-Llama-3.1-8B-AWQ` — reward classifier | AWQ | 16k |
|
||||
| `vllm-embed` | 8001 | `Qwen3-Embedding-0.6B` | — | 8k |
|
||||
| `vllm-rerank` | 8002 | `Qwen3-Reranker-0.6B` | — | 8k |
|
||||
@@ -94,7 +94,7 @@ both cards run ~90% full:
|
||||
- **GPU 0:** the two heavy NVFP4 seats — `vllm-gen` (gen) and
|
||||
`vllm-charrp-reasoning-nvfp4`. The live serving path (near-100% util under
|
||||
load), ~42 + 45 GB.
|
||||
- **GPU 1:** everything else — summarizer (granite), judge (selene), reward,
|
||||
- **GPU 1:** everything else — reward,
|
||||
embed, rerank, and the Magidonia char-RP GGUF seat. Bursty/on-demand, idle
|
||||
between calls, ~91 GB resident.
|
||||
|
||||
|
||||
@@ -337,19 +337,24 @@ model_list:
|
||||
model_info:
|
||||
mode: chat
|
||||
|
||||
# --- Selene 1 Mini 8B (AtlaAI judge, FP8) — restored on GPU1 after the
|
||||
# llama-swap teardown (was the Q6_K GGUF in the swap zoo). vLLM dynamic fp8,
|
||||
# :8011. Explicit entry shadows the "*" wildcard (which used to reach it via
|
||||
# llama-swap). Hallucination/RAG-faithfulness judge; callers set temp ~0.01. ---
|
||||
- model_name: selene-1-mini-8b
|
||||
litellm_params:
|
||||
model: hosted_vllm/selene-1-mini-8b
|
||||
api_base: http://10.250.50.54:8011/v1
|
||||
api_key: os.environ/VLLM_API_KEY
|
||||
temperature: 0.6
|
||||
top_p: 0.9
|
||||
model_info:
|
||||
mode: chat
|
||||
# --- selene-1-mini-8b RETIRED 2026-08-23. AtlaAI Selene 1 Mini (Llama 3.1 8B,
|
||||
# dynamic FP8) on ana-ml2 GPU1 :8011. Benchmarked head-to-head against `gen`
|
||||
# on its OWN job: 24 designed judge items with checkable ground truth,
|
||||
# pairwise + absolute scoring, 3 repeats, run on BOTH a neutral JSON prompt
|
||||
# and Selene's native Atla template (288 calls total). gen won on both —
|
||||
# 23/24 vs 20/24 neutral, 22/24 vs 21/24 native. Selene's BEST score sat
|
||||
# below gen's WORST. Decisive defect: it cannot emit "tie", forcing a winner
|
||||
# on every equivalent pair (0/2 on both templates), which is fatal for eval
|
||||
# work where close pairs are the whole point. Seat downed to reclaim 17.2 GiB
|
||||
# on GPU1 (the card had 1.8 GiB free). Its only edge was ~3x lower latency,
|
||||
# unexercised at its observed ~60 calls/day with zero queueing.
|
||||
#
|
||||
# DELIBERATELY NOT ALIASED TO ANOTHER MODEL. A caller asking for
|
||||
# `selene-1-mini-8b` must never silently receive qwen3.8-27b — a served-name
|
||||
# is a contract about what the model IS, and a silent substitution hides a
|
||||
# material change behind a stable string. This name now 404s BY DESIGN.
|
||||
# Repoint consumers to `chat-judge` (the role alias, below) or to `gen`
|
||||
# explicitly. Operator ruling 2026-08-23. ---
|
||||
|
||||
# --- Qwen3 embeddings ---
|
||||
- model_name: qwen3-embedding
|
||||
@@ -374,14 +379,26 @@ model_list:
|
||||
# model; swap the backing model here and callers are unaffected. NO generic
|
||||
# `embedding` alias ON PURPOSE — embedding vectors are model-specific (not
|
||||
# swap-transparent), so that capability stays `qwen3-embedding` above. ---
|
||||
# chat-judge → generative LLM-as-judge (Selene); WT selene-judgment role.
|
||||
# chat-judge → generative LLM-as-judge; WT selene-judgment role. Backed by
|
||||
# Selene until 2026-08-23, now `gen` / qwen3.8-27b-uncensored (:8015) after
|
||||
# Selene lost the head-to-head on its own job and its seat was reclaimed.
|
||||
# This is precisely the ADR-0012 case stated above: the ROLE alias moves, the
|
||||
# MODEL NAME does not — which is why selene-1-mini-8b was retired outright
|
||||
# rather than repointed. Sampler profile copied from image-judge
|
||||
# (deterministic, thinking off); the benchmark that selected gen ran at
|
||||
# temperature 0, so this matches the tested condition.
|
||||
- model_name: chat-judge
|
||||
litellm_params:
|
||||
model: hosted_vllm/selene-1-mini-8b
|
||||
api_base: http://10.250.50.54:8011/v1
|
||||
model: hosted_vllm/qwen3.8-27b-uncensored
|
||||
api_base: http://10.250.50.54:8015/v1
|
||||
api_key: os.environ/VLLM_API_KEY
|
||||
temperature: 0.6
|
||||
top_p: 0.9
|
||||
temperature: 0
|
||||
top_p: 1.0
|
||||
extra_body:
|
||||
top_k: 1
|
||||
repetition_penalty: 1.05
|
||||
chat_template_kwargs:
|
||||
enable_thinking: false
|
||||
model_info:
|
||||
mode: chat
|
||||
# reranker → generic capability name for rerank (currently qwen3-reranker).
|
||||
|
||||
@@ -0,0 +1,89 @@
|
||||
# selene — RETIRED 2026-08-23
|
||||
|
||||
AtlaAI Selene 1 Mini (Llama 3.1 8B, dynamic FP8), served on ana-ml2 GPU 1 at
|
||||
`:8011` as `selene-1-mini-8b`. An LLM-as-judge: it scored and critiqued other
|
||||
models' output rather than generating for users (hallucination /
|
||||
RAG-faithfulness checks, Worldtree's Domari and selene-judgment roles).
|
||||
|
||||
**The seat is down and the compose file is kept for reference only.** It is not
|
||||
deployed. `stacks-mirror/` will no longer show it on the host.
|
||||
|
||||
## Why it was retired
|
||||
|
||||
Benchmarked head-to-head against `gen` (qwen3.8-27b-uncensored, ana-ml2 GPU 0
|
||||
`:8015`) **on selene's own job** — 24 designed judge items with checkable
|
||||
ground truth, pairwise and absolute-scoring modes, 3 repeats each, run on two
|
||||
prompt templates. 288 calls total, all free local.
|
||||
|
||||
| template | selene | gen |
|
||||
|---|---|---|
|
||||
| neutral JSON prompt | 20/24 (83%) | **23/24 (96%)** |
|
||||
| Selene's native Atla template | 21/24 (88%) | **22/24 (92%)** |
|
||||
|
||||
**gen won on both templates, and selene's best score sat below gen's worst.**
|
||||
Selene was given its own fine-tuned prompt format as a fairness check — it
|
||||
gained one point, not the three it needed.
|
||||
|
||||
**The decisive defect: selene cannot emit "tie."** On both tie items, on both
|
||||
templates, it forced a winner (0/2 each time). gen returned `tie` correctly on
|
||||
the JSON template. For eval work, close pairs are precisely the case that
|
||||
matters; a judge that manufactures a preference on every equivalent pair is
|
||||
producing noise exactly where it is most trusted.
|
||||
|
||||
Other findings:
|
||||
|
||||
- **Calibration.** gen used the full 1–5 range decisively (1s for bad answers,
|
||||
5s for good). Selene clustered at 2s and 4s, compressing the scale.
|
||||
- **Stability at temperature 0.** Selene had 2 unstable items on the JSON
|
||||
template; gen had 0. They swapped on the native template (gen 4, selene 0) —
|
||||
longer free-text reasoning costs determinism.
|
||||
- **Latency was selene's only win** — roughly 3× faster (0.23s vs 0.40s median
|
||||
on JSON). Unexercised: it served ~60 calls/day with zero queueing across five
|
||||
weeks of uptime.
|
||||
- **Shared blind spot.** Both preferred a response containing an arithmetic
|
||||
error on the JSON template. gen caught it on the reasoning-first template.
|
||||
Neither is trustworthy for numerically-checkable judgments without a prompt
|
||||
that forces reasoning before the verdict.
|
||||
|
||||
Best measured configuration overall: **`gen` + the neutral JSON prompt** —
|
||||
23/24, zero instability, ties handled correctly.
|
||||
|
||||
## What happened to the names
|
||||
|
||||
**`chat-judge` → repointed to `gen`.** It is a *role* alias, and ADR-0012 is
|
||||
explicit that consumers bind the capability, not a concrete model. Sampler
|
||||
profile copied from `image-judge` (temperature 0, top_p 1.0, top_k 1,
|
||||
thinking off) so the served config matches the benchmarked condition.
|
||||
|
||||
**`selene-1-mini-8b` → removed outright. It 404s by design.** It was NOT
|
||||
aliased to `gen`. A served-name is a contract about what the model *is*;
|
||||
answering it with a different model hides a material change behind a stable
|
||||
string, and the caller has no way to know. Failing loud forces a conscious
|
||||
migration. Operator ruling, 2026-08-23:
|
||||
|
||||
> never repoint a named model at a different model's endpoint — that is
|
||||
> intentionally misleading
|
||||
|
||||
Direct precedent on this gateway: `qwen-image-bench` / `image-judge` had their
|
||||
dedicated backend retired 2026-07-15 to reclaim GPU 1 VRAM and were repointed
|
||||
to `gen` as role aliases. Counter-example worth remembering: when
|
||||
`qwen3-reranker` was retired, `nevermore` was pinned to it *by name* and the
|
||||
cutover moved the `reranker` alias but never moved nevermore — which is why
|
||||
name-pinned consumers get notified explicitly rather than assumed covered.
|
||||
|
||||
## Reclaimed
|
||||
|
||||
```
|
||||
GPU 1 free: 1,818 MiB → 19,450 MiB
|
||||
```
|
||||
|
||||
17.2 GiB, on a card that had under 2 GiB of headroom. `gen` was already
|
||||
running on GPU 0, so the judge role moved onto an existing seat rather than
|
||||
allocating anything new.
|
||||
|
||||
## To bring it back
|
||||
|
||||
`compose.yaml` and the host `.env` are intact. GPU 1 must have ~17 GiB free
|
||||
(`SELENE_GPU_MEM_UTIL=0.17`); re-add the `selene-1-mini-8b` entry to the
|
||||
LiteLLM config **under its own true name**, never as an alias for something
|
||||
else.
|
||||
Reference in New Issue
Block a user