vh
b8a535507a
Operator: keep the benchmark. It has a named second use (brokkr-smithy-dev wants gen vs a trained reward model once their tournament converges) and a demonstrated first one -- it caught a seat that had been coin-flip-grade for five weeks with nobody measuring it. Harness promoted from scratch to tools/judge-bench/: - paths de-hardcoded; runs from its own directory - proper CLI: --models (REQUIRED), --repeats, --limit, --gateway. Required on purpose: a stale default would silently benchmark a retired seat, and the original default (selene-1-mini-8b) now 400s. - README states the limitation rather than burying it: 24 items of the author's own design, a screen and not a verdict. This harness scored the same pair 83 vs 96 while brokkr's corpus ranking task scored it 47 (chance) vs 94. Both honest; absolute scoring on designed items is an easier task than ranking real text. - records brokkr's technique, which is better than anything here: a control constructed so the correct answer is DEFINITIONAL rather than judged cannot inherit the designer's error (item vs itself, response vs its own truncation, text vs its own clauses permuted). Add those before adding more judged items. Gateway: comment-only warning at the head of model_list. SEVEN aliases now resolve to the same weights (chat-judge, classifier, gen, image-judge, qwen-image-bench, summarizer, summarizer-large -> qwen3.8-27b-uncensored). That is intended under ADR-0012, but it has a sharp edge brokkr flagged: cross-checking a result against another alias measures NOTHING when they are the same model -- agreement is an echo, not corroboration. The note names the other current collisions (glm-5.2 x4, TTS x4, reranker x2), gives the /model/info one-liner to check, and records that probes should resolve alias -> backing at run start AND end because the response `model` field returns the alias, so a swap is otherwise invisible. Verified: config still parses, diff is comment-only, canonical re-synced.
4.2 KiB
judge-bench — evaluate a candidate LLM-as-judge seat
Compares two gateway models on judge work specifically: given a response (or a pair), produce a verdict. Built 2026-08-23 to answer "is the selene seat earning its 17 GiB", and kept because the answer was no and nobody had measured it in the five weeks it had been running.
python3 tools/judge-bench/run.py "$LITELLM_KEY" --models chat-judge gen
python3 tools/judge-bench/run.py "$LITELLM_KEY" --models a b --limit 2 --repeats 1 # smoke
python3 tools/judge-bench/native.py "$LITELLM_KEY" # Atla-template variant
--models is required on purpose. A stale default would silently benchmark a
retired seat — the original default, selene-1-mini-8b, now 400s.
What it measures
Two modes, because that is how a judge gets used:
- Pairwise — 12 items: which of two responses is better (A / B / tie).
- Absolute — 12 items: score one response 1–5 against a rubric.
Per model: accuracy against designed ground truth, stability across repeats at temperature 0, format compliance, calibration spread, and median latency.
Why the ground truth is defensible
Every "worse" response carries a checkable defect — a false fact, a violated explicit constraint, an answer to a different question, an invented citation, an arithmetic error — not a matter of taste. Absolute items are designed into bands with an expected centre.
Position is balanced 5 / 5 / 2 (A-better / B-better / tie) so a judge that simply prefers whichever response came first scores ~50% and its bias shows in the per-label breakdown. Position bias is a known LLM-judge failure mode and a respectable accuracy number can hide it.
The limitation, stated plainly
The ground truth is 24 items of the author's design, not a standard eval set. A 3-point gap on 24 items is suggestive, not decisive. Treat a narrow win as a tie.
This matters: on the 2026-08-23 run this harness scored selene 83% vs gen 96% and the write-up called the gap "modest". brokkr-smithy-dev, measuring the same pair on a real corpus ranking task, got selene 47% — chance — vs gen 94%. Both numbers were honest. Absolute scoring on designed items is an easier task than ranking real text, so this harness is a screen, not a verdict.
The technique worth stealing (brokkr-smithy-dev, 2026-08-23)
A control constructed so the correct answer is definitional rather than judged cannot inherit the designer's error. Their null control compared an excerpt against itself — tie is not a matter of opinion — and caught selene declaring byte-identical text "MUCH better" than itself 27 times in 60.
Three such controls, none of which require knowing what "good" is:
- an item against itself (must tie)
- a response against its own truncation (full must win)
- text against its own clauses permuted (tests order sensitivity)
Add these before adding more judged items. They are cheaper and stronger.
Findings that generalise
- Reasoning-first prompts beat bare JSON on hard items — both models failed an arithmetic-error item on a JSON-only prompt; gen caught it when the template forced reasoning before the verdict. But that same template broke gen's tie handling (2/2 → 0/2). If you need both, verify you have both.
- Inability to emit "tie" is disqualifying, not a rounding error. Close pairs are the case a judge exists for.
- Latency is usually irrelevant. Selene was 3× faster and served ~60 calls/day with zero queueing. Measure the load before trading accuracy for speed.
Provenance when benchmarking aliases
The gateway returns the alias in the response model field, not the
backing model, and several aliases share weights — as of 2026-08-23, seven of
them resolve to qwen3.8-27b-uncensored. Two consequences:
- Resolve
GET :4000/model/infoat run start and end, record the backing model with the results, and treat a mismatch as instrument drift. - Do not "cross-check" a result against another alias without checking it
is different weights.
chat-judge,genandsummarizerare currently the same model; agreement between them measures nothing.