vh
b8a535507a
Operator: keep the benchmark. It has a named second use (brokkr-smithy-dev wants gen vs a trained reward model once their tournament converges) and a demonstrated first one -- it caught a seat that had been coin-flip-grade for five weeks with nobody measuring it. Harness promoted from scratch to tools/judge-bench/: - paths de-hardcoded; runs from its own directory - proper CLI: --models (REQUIRED), --repeats, --limit, --gateway. Required on purpose: a stale default would silently benchmark a retired seat, and the original default (selene-1-mini-8b) now 400s. - README states the limitation rather than burying it: 24 items of the author's own design, a screen and not a verdict. This harness scored the same pair 83 vs 96 while brokkr's corpus ranking task scored it 47 (chance) vs 94. Both honest; absolute scoring on designed items is an easier task than ranking real text. - records brokkr's technique, which is better than anything here: a control constructed so the correct answer is DEFINITIONAL rather than judged cannot inherit the designer's error (item vs itself, response vs its own truncation, text vs its own clauses permuted). Add those before adding more judged items. Gateway: comment-only warning at the head of model_list. SEVEN aliases now resolve to the same weights (chat-judge, classifier, gen, image-judge, qwen-image-bench, summarizer, summarizer-large -> qwen3.8-27b-uncensored). That is intended under ADR-0012, but it has a sharp edge brokkr flagged: cross-checking a result against another alias measures NOTHING when they are the same model -- agreement is an echo, not corroboration. The note names the other current collisions (glm-5.2 x4, TTS x4, reranker x2), gives the /model/info one-liner to check, and records that probes should resolve alias -> backing at run start AND end because the response `model` field returns the alias, so a swap is otherwise invisible. Verified: config still parses, diff is comment-only, canonical re-synced.
90 lines
4.2 KiB
Markdown
90 lines
4.2 KiB
Markdown
# judge-bench — evaluate a candidate LLM-as-judge seat
|
||
|
||
Compares two gateway models on **judge work specifically**: given a response
|
||
(or a pair), produce a verdict. Built 2026-08-23 to answer "is the selene seat
|
||
earning its 17 GiB", and kept because the answer was *no* and nobody had
|
||
measured it in the five weeks it had been running.
|
||
|
||
```bash
|
||
python3 tools/judge-bench/run.py "$LITELLM_KEY" --models chat-judge gen
|
||
python3 tools/judge-bench/run.py "$LITELLM_KEY" --models a b --limit 2 --repeats 1 # smoke
|
||
python3 tools/judge-bench/native.py "$LITELLM_KEY" # Atla-template variant
|
||
```
|
||
|
||
`--models` is required on purpose. A stale default would silently benchmark a
|
||
retired seat — the original default, `selene-1-mini-8b`, now 400s.
|
||
|
||
## What it measures
|
||
|
||
Two modes, because that is how a judge gets used:
|
||
|
||
- **Pairwise** — 12 items: which of two responses is better (A / B / tie).
|
||
- **Absolute** — 12 items: score one response 1–5 against a rubric.
|
||
|
||
Per model: accuracy against designed ground truth, **stability** across
|
||
repeats at temperature 0, format compliance, calibration spread, and median
|
||
latency.
|
||
|
||
## Why the ground truth is defensible
|
||
|
||
Every "worse" response carries a **checkable defect** — a false fact, a
|
||
violated explicit constraint, an answer to a different question, an invented
|
||
citation, an arithmetic error — not a matter of taste. Absolute items are
|
||
designed into bands with an expected centre.
|
||
|
||
**Position is balanced 5 / 5 / 2** (A-better / B-better / tie) so a judge that
|
||
simply prefers whichever response came first scores ~50% and its bias shows in
|
||
the per-label breakdown. Position bias is a known LLM-judge failure mode and a
|
||
respectable accuracy number can hide it.
|
||
|
||
## The limitation, stated plainly
|
||
|
||
The ground truth is **24 items of the author's design, not a standard eval
|
||
set**. A 3-point gap on 24 items is suggestive, not decisive. Treat a narrow
|
||
win as a tie.
|
||
|
||
This matters: on the 2026-08-23 run this harness scored selene 83% vs gen 96%
|
||
and the write-up called the gap "modest". brokkr-smithy-dev, measuring the same
|
||
pair on a real corpus ranking task, got **selene 47% — chance — vs gen 94%**.
|
||
Both numbers were honest. Absolute scoring on designed items is an *easier
|
||
task* than ranking real text, so this harness is a **screen, not a verdict**.
|
||
|
||
## The technique worth stealing (brokkr-smithy-dev, 2026-08-23)
|
||
|
||
A control constructed so the correct answer is **definitional rather than
|
||
judged** cannot inherit the designer's error. Their null control compared an
|
||
excerpt against **itself** — tie is not a matter of opinion — and caught selene
|
||
declaring byte-identical text "MUCH better" than itself 27 times in 60.
|
||
|
||
Three such controls, none of which require knowing what "good" is:
|
||
|
||
1. an item against **itself** (must tie)
|
||
2. a response against **its own truncation** (full must win)
|
||
3. text against **its own clauses permuted** (tests order sensitivity)
|
||
|
||
**Add these before adding more judged items.** They are cheaper and stronger.
|
||
|
||
## Findings that generalise
|
||
|
||
- **Reasoning-first prompts beat bare JSON on hard items** — both models failed
|
||
an arithmetic-error item on a JSON-only prompt; gen caught it when the
|
||
template forced reasoning before the verdict. But that same template *broke*
|
||
gen's tie handling (2/2 → 0/2). If you need both, verify you have both.
|
||
- **Inability to emit "tie" is disqualifying**, not a rounding error. Close
|
||
pairs are the case a judge exists for.
|
||
- **Latency is usually irrelevant.** Selene was 3× faster and served ~60
|
||
calls/day with zero queueing. Measure the load before trading accuracy for
|
||
speed.
|
||
|
||
## Provenance when benchmarking aliases
|
||
|
||
The gateway returns the **alias** in the response `model` field, not the
|
||
backing model, and several aliases share weights — as of 2026-08-23, seven of
|
||
them resolve to `qwen3.8-27b-uncensored`. Two consequences:
|
||
|
||
- Resolve `GET :4000/model/info` at run **start and end**, record the backing
|
||
model with the results, and treat a mismatch as instrument drift.
|
||
- **Do not "cross-check" a result against another alias** without checking it
|
||
is different weights. `chat-judge`, `gen` and `summarizer` are currently the
|
||
same model; agreement between them measures nothing.
|