diff --git a/docs/pfi/recommended-model-settings.md b/docs/pfi/recommended-model-settings.md index 280f643..21ebbb4 100644 --- a/docs/pfi/recommended-model-settings.md +++ b/docs/pfi/recommended-model-settings.md @@ -560,7 +560,8 @@ override the config default.) Values set per the `dvalin-smithy-dev` research pa | `gen`, `summarizer-large`, `qwen-large`, `qwen3.5-122-a10b` (non-thinking) | **0.7** | 0.8 | 20 | **1.0** | — | Qwen3 non-thinking + operator anti-repetition | | `gen-reasoning`, `qwen-large-reasoning`, `qwen3.5-122-a10b-reasoning` (thinking) | **0.6** | 0.95 | 20 | **1.0** | — | Qwen3 thinking | | `qwen-image-bench`, `image-judge` | **0** | 1.0 | 1 | — | 1.05 | Qwen-Image-Bench judge reproducibility table | -| `selene-1-mini-8b`, `chat-judge` | **0.6** | 0.9 | — | — | — | Selene `generation_config` | +| ~~`selene-1-mini-8b`~~ | — | — | — | — | — | **RETIRED 2026-08-23**; name 404s by design, not aliased | +| `chat-judge` | **0** | 1.0 | 1 | — | 1.05 | Repointed to `gen` 2026-08-23; deterministic judge profile copied from `image-judge`. The benchmark that selected `gen` ran at temperature 0 — match it. | | `glm-5.1`, `glm-5.2`, `glm-5-turbo`, `glm-4.7`, `gen-frontier` | **1.0** | 0.95 | — | — | — | z.ai API defaults (5.x / 4.7 series) | | `glm-4.5-air` | **0.6** | 0.95 | — | — | — | z.ai API default (4.5 series) | | `qwen3-embedding`, `qwen3-reranker`, `reranker` | — | — | — | — | — | no sampling (embedding / rerank) | diff --git a/servers/ana-ml2/README.md b/servers/ana-ml2/README.md index db3bbcc..a498121 100644 --- a/servers/ana-ml2/README.md +++ b/servers/ana-ml2/README.md @@ -50,7 +50,7 @@ embed/rerank/reward trio. GPUs are pinned per container via |-----------|------|--------------|-------|-----| | `vllm-granite` | 8004 | `granite-4.1-8b` — fleet summarizer/classifier | FP8 (compressed-tensors) | 131k | | `llama-charrp` | 8016 | `Magidonia-24B-v4.3` Q6_K — char-RP (llama.cpp) | GGUF Q6_K | — | -| `vllm-selene` | 8011 | `selene-1-mini-8b` — Atla LLM-as-judge | FP8 | 32k | +| ~~`vllm-selene`~~ | ~~8011~~ | **RETIRED 2026-08-23** — lost a head-to-head against `gen` on its own judge task (see `stacks/selene/README.md`); seat downed to reclaim 17.2 GiB on GPU 1. `selene-1-mini-8b` now 404s by design; use `chat-judge`. | — | — | | `vllm-reward` | 8003 | `Skywork-Reward-V2-Llama-3.1-8B-AWQ` — reward classifier | AWQ | 16k | | `vllm-embed` | 8001 | `Qwen3-Embedding-0.6B` | — | 8k | | `vllm-rerank` | 8002 | `Qwen3-Reranker-0.6B` | — | 8k | @@ -94,7 +94,7 @@ both cards run ~90% full: - **GPU 0:** the two heavy NVFP4 seats — `vllm-gen` (gen) and `vllm-charrp-reasoning-nvfp4`. The live serving path (near-100% util under load), ~42 + 45 GB. -- **GPU 1:** everything else — summarizer (granite), judge (selene), reward, +- **GPU 1:** everything else — reward, embed, rerank, and the Magidonia char-RP GGUF seat. Bursty/on-demand, idle between calls, ~91 GB resident. diff --git a/stacks/litellm/conf/config.yaml b/stacks/litellm/conf/config.yaml index b0b5ee9..3ce2c0a 100644 --- a/stacks/litellm/conf/config.yaml +++ b/stacks/litellm/conf/config.yaml @@ -337,19 +337,24 @@ model_list: model_info: mode: chat - # --- Selene 1 Mini 8B (AtlaAI judge, FP8) — restored on GPU1 after the - # llama-swap teardown (was the Q6_K GGUF in the swap zoo). vLLM dynamic fp8, - # :8011. Explicit entry shadows the "*" wildcard (which used to reach it via - # llama-swap). Hallucination/RAG-faithfulness judge; callers set temp ~0.01. --- - - model_name: selene-1-mini-8b - litellm_params: - model: hosted_vllm/selene-1-mini-8b - api_base: http://10.250.50.54:8011/v1 - api_key: os.environ/VLLM_API_KEY - temperature: 0.6 - top_p: 0.9 - model_info: - mode: chat + # --- selene-1-mini-8b RETIRED 2026-08-23. AtlaAI Selene 1 Mini (Llama 3.1 8B, + # dynamic FP8) on ana-ml2 GPU1 :8011. Benchmarked head-to-head against `gen` + # on its OWN job: 24 designed judge items with checkable ground truth, + # pairwise + absolute scoring, 3 repeats, run on BOTH a neutral JSON prompt + # and Selene's native Atla template (288 calls total). gen won on both — + # 23/24 vs 20/24 neutral, 22/24 vs 21/24 native. Selene's BEST score sat + # below gen's WORST. Decisive defect: it cannot emit "tie", forcing a winner + # on every equivalent pair (0/2 on both templates), which is fatal for eval + # work where close pairs are the whole point. Seat downed to reclaim 17.2 GiB + # on GPU1 (the card had 1.8 GiB free). Its only edge was ~3x lower latency, + # unexercised at its observed ~60 calls/day with zero queueing. + # + # DELIBERATELY NOT ALIASED TO ANOTHER MODEL. A caller asking for + # `selene-1-mini-8b` must never silently receive qwen3.8-27b — a served-name + # is a contract about what the model IS, and a silent substitution hides a + # material change behind a stable string. This name now 404s BY DESIGN. + # Repoint consumers to `chat-judge` (the role alias, below) or to `gen` + # explicitly. Operator ruling 2026-08-23. --- # --- Qwen3 embeddings --- - model_name: qwen3-embedding @@ -374,14 +379,26 @@ model_list: # model; swap the backing model here and callers are unaffected. NO generic # `embedding` alias ON PURPOSE — embedding vectors are model-specific (not # swap-transparent), so that capability stays `qwen3-embedding` above. --- - # chat-judge → generative LLM-as-judge (Selene); WT selene-judgment role. + # chat-judge → generative LLM-as-judge; WT selene-judgment role. Backed by + # Selene until 2026-08-23, now `gen` / qwen3.8-27b-uncensored (:8015) after + # Selene lost the head-to-head on its own job and its seat was reclaimed. + # This is precisely the ADR-0012 case stated above: the ROLE alias moves, the + # MODEL NAME does not — which is why selene-1-mini-8b was retired outright + # rather than repointed. Sampler profile copied from image-judge + # (deterministic, thinking off); the benchmark that selected gen ran at + # temperature 0, so this matches the tested condition. - model_name: chat-judge litellm_params: - model: hosted_vllm/selene-1-mini-8b - api_base: http://10.250.50.54:8011/v1 + model: hosted_vllm/qwen3.8-27b-uncensored + api_base: http://10.250.50.54:8015/v1 api_key: os.environ/VLLM_API_KEY - temperature: 0.6 - top_p: 0.9 + temperature: 0 + top_p: 1.0 + extra_body: + top_k: 1 + repetition_penalty: 1.05 + chat_template_kwargs: + enable_thinking: false model_info: mode: chat # reranker → generic capability name for rerank (currently qwen3-reranker). diff --git a/stacks/selene/README.md b/stacks/selene/README.md new file mode 100644 index 0000000..0020a98 --- /dev/null +++ b/stacks/selene/README.md @@ -0,0 +1,89 @@ +# selene — RETIRED 2026-08-23 + +AtlaAI Selene 1 Mini (Llama 3.1 8B, dynamic FP8), served on ana-ml2 GPU 1 at +`:8011` as `selene-1-mini-8b`. An LLM-as-judge: it scored and critiqued other +models' output rather than generating for users (hallucination / +RAG-faithfulness checks, Worldtree's Domari and selene-judgment roles). + +**The seat is down and the compose file is kept for reference only.** It is not +deployed. `stacks-mirror/` will no longer show it on the host. + +## Why it was retired + +Benchmarked head-to-head against `gen` (qwen3.8-27b-uncensored, ana-ml2 GPU 0 +`:8015`) **on selene's own job** — 24 designed judge items with checkable +ground truth, pairwise and absolute-scoring modes, 3 repeats each, run on two +prompt templates. 288 calls total, all free local. + +| template | selene | gen | +|---|---|---| +| neutral JSON prompt | 20/24 (83%) | **23/24 (96%)** | +| Selene's native Atla template | 21/24 (88%) | **22/24 (92%)** | + +**gen won on both templates, and selene's best score sat below gen's worst.** +Selene was given its own fine-tuned prompt format as a fairness check — it +gained one point, not the three it needed. + +**The decisive defect: selene cannot emit "tie."** On both tie items, on both +templates, it forced a winner (0/2 each time). gen returned `tie` correctly on +the JSON template. For eval work, close pairs are precisely the case that +matters; a judge that manufactures a preference on every equivalent pair is +producing noise exactly where it is most trusted. + +Other findings: + +- **Calibration.** gen used the full 1–5 range decisively (1s for bad answers, + 5s for good). Selene clustered at 2s and 4s, compressing the scale. +- **Stability at temperature 0.** Selene had 2 unstable items on the JSON + template; gen had 0. They swapped on the native template (gen 4, selene 0) — + longer free-text reasoning costs determinism. +- **Latency was selene's only win** — roughly 3× faster (0.23s vs 0.40s median + on JSON). Unexercised: it served ~60 calls/day with zero queueing across five + weeks of uptime. +- **Shared blind spot.** Both preferred a response containing an arithmetic + error on the JSON template. gen caught it on the reasoning-first template. + Neither is trustworthy for numerically-checkable judgments without a prompt + that forces reasoning before the verdict. + +Best measured configuration overall: **`gen` + the neutral JSON prompt** — +23/24, zero instability, ties handled correctly. + +## What happened to the names + +**`chat-judge` → repointed to `gen`.** It is a *role* alias, and ADR-0012 is +explicit that consumers bind the capability, not a concrete model. Sampler +profile copied from `image-judge` (temperature 0, top_p 1.0, top_k 1, +thinking off) so the served config matches the benchmarked condition. + +**`selene-1-mini-8b` → removed outright. It 404s by design.** It was NOT +aliased to `gen`. A served-name is a contract about what the model *is*; +answering it with a different model hides a material change behind a stable +string, and the caller has no way to know. Failing loud forces a conscious +migration. Operator ruling, 2026-08-23: + +> never repoint a named model at a different model's endpoint — that is +> intentionally misleading + +Direct precedent on this gateway: `qwen-image-bench` / `image-judge` had their +dedicated backend retired 2026-07-15 to reclaim GPU 1 VRAM and were repointed +to `gen` as role aliases. Counter-example worth remembering: when +`qwen3-reranker` was retired, `nevermore` was pinned to it *by name* and the +cutover moved the `reranker` alias but never moved nevermore — which is why +name-pinned consumers get notified explicitly rather than assumed covered. + +## Reclaimed + +``` +GPU 1 free: 1,818 MiB → 19,450 MiB +``` + +17.2 GiB, on a card that had under 2 GiB of headroom. `gen` was already +running on GPU 0, so the judge role moved onto an existing seat rather than +allocating anything new. + +## To bring it back + +`compose.yaml` and the host `.env` are intact. GPU 1 must have ~17 GiB free +(`SELENE_GPU_MEM_UTIL=0.17`); re-add the `selene-1-mini-8b` entry to the +LiteLLM config **under its own true name**, never as an alias for something +else.