vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity

feat(selene): retire the seat; chat-judge -> gen, selene-1-mini-8b 404s by design

Browse Source 
Benchmarked selene against gen on selene's own job: 24 designed judge items
with checkable ground truth, pairwise + absolute modes, 3 repeats, on BOTH a
neutral JSON prompt and Selene's native Atla template. 288 calls, all free
local.

  neutral JSON     selene 20/24 (83%)   gen 23/24 (96%)
  native Atla      selene 21/24 (88%)   gen 22/24 (92%)

gen won on both templates and selene's BEST sat below gen's WORST. Selene was
given its own fine-tuned template as a fairness check; it gained one point,
not the three it needed.

Decisive defect: selene cannot emit "tie" -- 0/2 on both templates, forcing a
winner on every equivalent pair. For eval work that is the case that matters
most. gen returned tie correctly on the JSON template. Selene also compressed
the 1-5 scale (clustered at 2s and 4s) where gen used it fully. Selene's only
win was ~3x latency, unexercised at ~60 calls/day with zero queueing.

TWO NAMES, TWO DIFFERENT TREATMENTS, deliberately:

- chat-judge -> repointed to gen. It is a ROLE alias and ADR-0012 says
  consumers bind the capability, not a concrete model. Sampler profile copied
  from image-judge (temp 0, top_p 1.0, top_k 1, thinking off) so the served
  config matches the benchmarked condition.

- selene-1-mini-8b -> REMOVED. It 404s. It was NOT aliased to gen. A
  served-name is a contract about what the model IS; answering it with a
  different model hides a material change behind a stable string. Operator
  ruling: "never repoint a named model at a different model's endpoint --
  that is intentionally misleading." Verified: the gateway now returns
  HTTP 400 "Invalid model name" for it.

Reclaimed 17.2 GiB on ana-ml2 GPU 1 (free 1,818 -> 19,450 MiB) on a card that
had under 2 GiB of headroom. gen already runs on GPU 0, so the judge role
moved onto an existing seat rather than allocating anything new.

Canonical litellm config synced from the host; ana-ml2 README and
recommended-model-settings updated. compose.yaml kept for reference, not
deployed.
This commit is contained in:
vh
2026-08-23 05:07:01 -07:00
parent a896c0a5a9
commit ca3c984f93
4 changed files with 128 additions and 21 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/pfi/recommended-model-settings.md
+2 -1
View File
@@ -560,7 +560,8 @@ override the config default.) Values set per the `dvalin-smithy-dev` research pa
| `gen`, `summarizer-large`, `qwen-large`, `qwen3.5-122-a10b` (non-thinking) | **0.7** | 0.8 | 20 | **1.0** | — | Qwen3 non-thinking + operator anti-repetition |
| `gen-reasoning`, `qwen-large-reasoning`, `qwen3.5-122-a10b-reasoning` (thinking) | **0.6** | 0.95 | 20 | **1.0** | — | Qwen3 thinking |
| `qwen-image-bench`, `image-judge` | **0** | 1.0 | 1 | — | 1.05 | Qwen-Image-Bench judge reproducibility table |
| `selene-1-mini-8b`, `chat-judge` | **0.6** | 0.9 | — | — | — | Selene `generation_config` |
| ~~`selene-1-mini-8b`~~ | — | — | — | — | — | **RETIRED 2026-08-23**; name 404s by design, not aliased |
| `chat-judge` | **0** | 1.0 | 1 | — | 1.05 | Repointed to `gen` 2026-08-23; deterministic judge profile copied from `image-judge`. The benchmark that selected `gen` ran at temperature 0 — match it. |
| `glm-5.1`, `glm-5.2`, `glm-5-turbo`, `glm-4.7`, `gen-frontier` | **1.0** | 0.95 | — | — | — | z.ai API defaults (5.x / 4.7 series) |
| `glm-4.5-air` | **0.6** | 0.95 | — | — | — | z.ai API default (4.5 series) |
| `qwen3-embedding`, `qwen3-reranker`, `reranker` | — | — | — | — | — | no sampling (embedding / rerank) |