# judge-bench — evaluate a candidate LLM-as-judge seat

Compares two gateway models on **judge work specifically**: given a response
(or a pair), produce a verdict. Built 2026-08-23 to answer "is the selene seat
earning its 17 GiB", and kept because the answer was *no* and nobody had
measured it in the five weeks it had been running.

```bash
python3 tools/judge-bench/run.py "$LITELLM_KEY" --models chat-judge gen
python3 tools/judge-bench/run.py "$LITELLM_KEY" --models a b --limit 2 --repeats 1   # smoke
python3 tools/judge-bench/native.py "$LITELLM_KEY"                                   # Atla-template variant
```

`--models` is required on purpose. A stale default would silently benchmark a
retired seat — the original default, `selene-1-mini-8b`, now 400s.

## What it measures

Two modes, because that is how a judge gets used:

- **Pairwise** — 12 items: which of two responses is better (A / B / tie).
- **Absolute** — 12 items: score one response 1–5 against a rubric.

Per model: accuracy against designed ground truth, **stability** across
repeats at temperature 0, format compliance, calibration spread, and median
latency.

## Why the ground truth is defensible

Every "worse" response carries a **checkable defect** — a false fact, a
violated explicit constraint, an answer to a different question, an invented
citation, an arithmetic error — not a matter of taste. Absolute items are
designed into bands with an expected centre.

**Position is balanced 5 / 5 / 2** (A-better / B-better / tie) so a judge that
simply prefers whichever response came first scores ~50% and its bias shows in
the per-label breakdown. Position bias is a known LLM-judge failure mode and a
respectable accuracy number can hide it.

## The limitation, stated plainly

The ground truth is **24 items of the author's design, not a standard eval
set**. A 3-point gap on 24 items is suggestive, not decisive. Treat a narrow
win as a tie.

This matters: on the 2026-08-23 run this harness scored selene 83% vs gen 96%
and the write-up called the gap "modest". brokkr-smithy-dev, measuring the same
pair on a real corpus ranking task, got **selene 47% — chance — vs gen 94%**.
Both numbers were honest. Absolute scoring on designed items is an *easier
task* than ranking real text, so this harness is a **screen, not a verdict**.

## The technique worth stealing (brokkr-smithy-dev, 2026-08-23)

A control constructed so the correct answer is **definitional rather than
judged** cannot inherit the designer's error. Their null control compared an
excerpt against **itself** — tie is not a matter of opinion — and caught selene
declaring byte-identical text "MUCH better" than itself 27 times in 60.

Three such controls, none of which require knowing what "good" is:

1. an item against **itself** (must tie)
2. a response against **its own truncation** (full must win)
3. text against **its own clauses permuted** (tests order sensitivity)

**Add these before adding more judged items.** They are cheaper and stronger.

## Findings that generalise

- **Reasoning-first prompts beat bare JSON on hard items** — both models failed
  an arithmetic-error item on a JSON-only prompt; gen caught it when the
  template forced reasoning before the verdict. But that same template *broke*
  gen's tie handling (2/2 → 0/2). If you need both, verify you have both.
- **Inability to emit "tie" is disqualifying**, not a rounding error. Close
  pairs are the case a judge exists for.
- **Latency is usually irrelevant.** Selene was 3× faster and served ~60
  calls/day with zero queueing. Measure the load before trading accuracy for
  speed.

## Provenance when benchmarking aliases

The gateway returns the **alias** in the response `model` field, not the
backing model, and several aliases share weights — as of 2026-08-23, seven of
them resolve to `qwen3.8-27b-uncensored`. Two consequences:

- Resolve `GET :4000/model/info` at run **start and end**, record the backing
  model with the results, and treat a mismatch as instrument drift.
- **Do not "cross-check" a result against another alias** without checking it
  is different weights. `chat-judge`, `gen` and `summarizer` are currently the
  same model; agreement between them measures nothing.