feat(judge-bench): keep the judge harness; warn about the 7-way alias collision
Operator: keep the benchmark. It has a named second use (brokkr-smithy-dev wants gen vs a trained reward model once their tournament converges) and a demonstrated first one -- it caught a seat that had been coin-flip-grade for five weeks with nobody measuring it. Harness promoted from scratch to tools/judge-bench/: - paths de-hardcoded; runs from its own directory - proper CLI: --models (REQUIRED), --repeats, --limit, --gateway. Required on purpose: a stale default would silently benchmark a retired seat, and the original default (selene-1-mini-8b) now 400s. - README states the limitation rather than burying it: 24 items of the author's own design, a screen and not a verdict. This harness scored the same pair 83 vs 96 while brokkr's corpus ranking task scored it 47 (chance) vs 94. Both honest; absolute scoring on designed items is an easier task than ranking real text. - records brokkr's technique, which is better than anything here: a control constructed so the correct answer is DEFINITIONAL rather than judged cannot inherit the designer's error (item vs itself, response vs its own truncation, text vs its own clauses permuted). Add those before adding more judged items. Gateway: comment-only warning at the head of model_list. SEVEN aliases now resolve to the same weights (chat-judge, classifier, gen, image-judge, qwen-image-bench, summarizer, summarizer-large -> qwen3.8-27b-uncensored). That is intended under ADR-0012, but it has a sharp edge brokkr flagged: cross-checking a result against another alias measures NOTHING when they are the same model -- agreement is an echo, not corroboration. The note names the other current collisions (glm-5.2 x4, TTS x4, reranker x2), gives the /model/info one-liner to check, and records that probes should resolve alias -> backing at run start AND end because the response `model` field returns the alias, so a swap is otherwise invisible. Verified: config still parses, diff is comment-only, canonical re-synced.
This commit is contained in:
@@ -17,6 +17,39 @@
|
||||
# route — left direct; see README.
|
||||
|
||||
model_list:
|
||||
|
||||
# =============================================================================
|
||||
# ⚠ ALIAS COLLISION — SEVERAL NAMES, ONE SET OF WEIGHTS
|
||||
#
|
||||
# As of 2026-08-23 these SEVEN aliases all resolve to the same backend
|
||||
# (qwen3.8-27b-uncensored @ 10.250.50.54:8015):
|
||||
#
|
||||
# chat-judge classifier gen image-judge
|
||||
# qwen-image-bench summarizer summarizer-large
|
||||
#
|
||||
# They differ only in sampler params. That is intended — role aliases exist so
|
||||
# consumers bind a CAPABILITY and the backing model can move (ADR-0012) — but
|
||||
# it has a sharp edge that has to be stated where people read it:
|
||||
#
|
||||
# DO NOT "CROSS-CHECK" A RESULT BY RUNNING IT AGAINST ANOTHER ALIAS.
|
||||
# Asking `gen` and then `summarizer` and finding they agree measures NOTHING:
|
||||
# it is the same weights answering twice. Agreement between colliding aliases
|
||||
# is not corroboration, it is an echo. Flagged by brokkr-smithy-dev
|
||||
# 2026-08-23 while wiring provenance into a probe harness.
|
||||
#
|
||||
# Other current collisions: gen-frontier / gen-frontier-reasoning / glm-5.2 /
|
||||
# glm-5.2-reasoning -> glm-5.2; ext-tts / gpt-4o-mini-tts / tts-1 / tts-1-hd ->
|
||||
# the fleet TTS gateway; reranker / reranker-a3-bge-v2-m3 -> bge-reranker-v2-m3.
|
||||
#
|
||||
# TO CHECK BEFORE RELYING ON TWO ALIASES BEING DIFFERENT MODELS:
|
||||
# curl -s :4000/model/info -H "Authorization: Bearer <key>" \
|
||||
# | python3 -c "import json,sys;[print(r['model_name'], r['litellm_params'].get('model')) for r in json.load(sys.stdin)['data']]"
|
||||
#
|
||||
# Probes recording provenance should resolve alias -> backing model at run
|
||||
# START and END and void the run on a mismatch: the response `model` field
|
||||
# returns the ALIAS, so a mid-run or between-run swap is otherwise invisible.
|
||||
# =============================================================================
|
||||
|
||||
# --- Granite 4.1 8B (generative chat) — production summarizer + dreaming
|
||||
# agent. Replaced phi4-mini 2026-06-05 (beat it on precision in brokkr's
|
||||
# R15 P03 eval). vLLM on ana-ml2 GPU 1, official FP8, 50K ctx. Explicit
|
||||
|
||||
@@ -0,0 +1,89 @@
|
||||
# judge-bench — evaluate a candidate LLM-as-judge seat
|
||||
|
||||
Compares two gateway models on **judge work specifically**: given a response
|
||||
(or a pair), produce a verdict. Built 2026-08-23 to answer "is the selene seat
|
||||
earning its 17 GiB", and kept because the answer was *no* and nobody had
|
||||
measured it in the five weeks it had been running.
|
||||
|
||||
```bash
|
||||
python3 tools/judge-bench/run.py "$LITELLM_KEY" --models chat-judge gen
|
||||
python3 tools/judge-bench/run.py "$LITELLM_KEY" --models a b --limit 2 --repeats 1 # smoke
|
||||
python3 tools/judge-bench/native.py "$LITELLM_KEY" # Atla-template variant
|
||||
```
|
||||
|
||||
`--models` is required on purpose. A stale default would silently benchmark a
|
||||
retired seat — the original default, `selene-1-mini-8b`, now 400s.
|
||||
|
||||
## What it measures
|
||||
|
||||
Two modes, because that is how a judge gets used:
|
||||
|
||||
- **Pairwise** — 12 items: which of two responses is better (A / B / tie).
|
||||
- **Absolute** — 12 items: score one response 1–5 against a rubric.
|
||||
|
||||
Per model: accuracy against designed ground truth, **stability** across
|
||||
repeats at temperature 0, format compliance, calibration spread, and median
|
||||
latency.
|
||||
|
||||
## Why the ground truth is defensible
|
||||
|
||||
Every "worse" response carries a **checkable defect** — a false fact, a
|
||||
violated explicit constraint, an answer to a different question, an invented
|
||||
citation, an arithmetic error — not a matter of taste. Absolute items are
|
||||
designed into bands with an expected centre.
|
||||
|
||||
**Position is balanced 5 / 5 / 2** (A-better / B-better / tie) so a judge that
|
||||
simply prefers whichever response came first scores ~50% and its bias shows in
|
||||
the per-label breakdown. Position bias is a known LLM-judge failure mode and a
|
||||
respectable accuracy number can hide it.
|
||||
|
||||
## The limitation, stated plainly
|
||||
|
||||
The ground truth is **24 items of the author's design, not a standard eval
|
||||
set**. A 3-point gap on 24 items is suggestive, not decisive. Treat a narrow
|
||||
win as a tie.
|
||||
|
||||
This matters: on the 2026-08-23 run this harness scored selene 83% vs gen 96%
|
||||
and the write-up called the gap "modest". brokkr-smithy-dev, measuring the same
|
||||
pair on a real corpus ranking task, got **selene 47% — chance — vs gen 94%**.
|
||||
Both numbers were honest. Absolute scoring on designed items is an *easier
|
||||
task* than ranking real text, so this harness is a **screen, not a verdict**.
|
||||
|
||||
## The technique worth stealing (brokkr-smithy-dev, 2026-08-23)
|
||||
|
||||
A control constructed so the correct answer is **definitional rather than
|
||||
judged** cannot inherit the designer's error. Their null control compared an
|
||||
excerpt against **itself** — tie is not a matter of opinion — and caught selene
|
||||
declaring byte-identical text "MUCH better" than itself 27 times in 60.
|
||||
|
||||
Three such controls, none of which require knowing what "good" is:
|
||||
|
||||
1. an item against **itself** (must tie)
|
||||
2. a response against **its own truncation** (full must win)
|
||||
3. text against **its own clauses permuted** (tests order sensitivity)
|
||||
|
||||
**Add these before adding more judged items.** They are cheaper and stronger.
|
||||
|
||||
## Findings that generalise
|
||||
|
||||
- **Reasoning-first prompts beat bare JSON on hard items** — both models failed
|
||||
an arithmetic-error item on a JSON-only prompt; gen caught it when the
|
||||
template forced reasoning before the verdict. But that same template *broke*
|
||||
gen's tie handling (2/2 → 0/2). If you need both, verify you have both.
|
||||
- **Inability to emit "tie" is disqualifying**, not a rounding error. Close
|
||||
pairs are the case a judge exists for.
|
||||
- **Latency is usually irrelevant.** Selene was 3× faster and served ~60
|
||||
calls/day with zero queueing. Measure the load before trading accuracy for
|
||||
speed.
|
||||
|
||||
## Provenance when benchmarking aliases
|
||||
|
||||
The gateway returns the **alias** in the response `model` field, not the
|
||||
backing model, and several aliases share weights — as of 2026-08-23, seven of
|
||||
them resolve to `qwen3.8-27b-uncensored`. Two consequences:
|
||||
|
||||
- Resolve `GET :4000/model/info` at run **start and end**, record the backing
|
||||
model with the results, and treat a mismatch as instrument drift.
|
||||
- **Do not "cross-check" a result against another alias** without checking it
|
||||
is different weights. `chat-judge`, `gen` and `summarizer` are currently the
|
||||
same model; agreement between them measures nothing.
|
||||
@@ -0,0 +1,146 @@
|
||||
"""Judge-benchmark items with DESIGNED ground truth.
|
||||
|
||||
Two modes, because that is how a judge actually gets used:
|
||||
PAIRWISE — given two responses, which is better (A / B / tie)
|
||||
ABSOLUTE — score one response 1-5 against a rubric
|
||||
|
||||
Ground truth is designed to be defensible without appeal to taste: the
|
||||
"worse" response contains a checkable defect (false fact, violated explicit
|
||||
constraint, answers a different question, invented citation, arithmetic
|
||||
error). Ties are pairs that are both correct and equivalent.
|
||||
|
||||
POSITION BALANCE IS DELIBERATE. 5 items where A is better, 5 where B is,
|
||||
2 genuine ties. A judge that just prefers whichever came first will score
|
||||
~50% here and its bias will show in the per-label breakdown — position bias
|
||||
is a known LLM-judge failure mode and a wash on accuracy can still hide it.
|
||||
"""
|
||||
|
||||
PAIRWISE = [
|
||||
# --- A is better -------------------------------------------------
|
||||
dict(id="p01-fact", truth="A",
|
||||
q="In what year did the Berlin Wall fall?",
|
||||
a="The Berlin Wall fell in 1989.",
|
||||
b="The Berlin Wall fell in 1991.",
|
||||
defect="B states a false year (1989 is correct)."),
|
||||
dict(id="p02-constraint", truth="A",
|
||||
q="Answer in exactly one sentence: why is the sky blue?",
|
||||
a="Sunlight scatters off air molecules, and shorter blue wavelengths scatter most, so the sky looks blue.",
|
||||
b="Great question! The sky is blue because of scattering. Specifically, Rayleigh scattering. "
|
||||
"Shorter wavelengths scatter more than longer ones. That is why we see blue overhead and red at sunset.",
|
||||
defect="B violates the explicit one-sentence constraint (five sentences)."),
|
||||
dict(id="p03-relevance", truth="A",
|
||||
q="How do I rotate a Postgres log file without restarting the server?",
|
||||
a="Use logrotate with `copytruncate`, or call pg_rotate_logfile() which signals the logging collector to start a new file.",
|
||||
b="To back up Postgres, use pg_dump for logical backups or pg_basebackup for physical ones, then verify with pg_restore --list.",
|
||||
defect="B answers a different question (backups, not log rotation)."),
|
||||
dict(id="p04-invented-cite", truth="A",
|
||||
q="Is there evidence that spaced repetition improves long-term retention?",
|
||||
a="Yes — the spacing effect is well replicated in memory research going back to Ebbinghaus, and modern reviews find spaced practice beats massed practice for durable retention.",
|
||||
b="Yes — Hollingsworth & Pryor (2019), 'Spaced Retrieval in Adult Learners', Journal of Cognitive Methods 44(3), pp. 211-238, found a 340% retention gain.",
|
||||
defect="B fabricates a specific citation, journal, volume and effect size."),
|
||||
dict(id="p05-incomplete", truth="A",
|
||||
q="Name three causes of high memory usage in a Python process.",
|
||||
a="Unbounded caches or dicts that are never evicted; reference cycles holding large objects; and loading a whole file or query result into memory instead of streaming it.",
|
||||
b="One common cause is a memory leak.",
|
||||
defect="B gives one vague cause where three were requested."),
|
||||
|
||||
# --- B is better (position flipped) ------------------------------
|
||||
dict(id="p06-arith", truth="B",
|
||||
q="A server has 96 GB of VRAM. Three models use 46 GB, 17 GB and 9 GB. How much is free?",
|
||||
a="46 + 17 + 9 = 62 GB used, so 34 GB is free.",
|
||||
b="46 + 17 + 9 = 72 GB used, so 24 GB is free.",
|
||||
defect="A's arithmetic is wrong (46+17+9 = 72, not 62)."),
|
||||
dict(id="p07-vague", truth="B",
|
||||
q="What does the `--delete` flag do in rsync?",
|
||||
a="It deletes things you do not need any more, which helps keep the destination tidy.",
|
||||
b="It removes files from the destination that are not present in the source, making the destination mirror the source rather than accumulate.",
|
||||
defect="A is vague and omits the defining behaviour; B is precise."),
|
||||
dict(id="p08-constraint2", truth="B",
|
||||
q="Reply with only a JSON object containing the key \"status\".",
|
||||
a="Sure! Here is the JSON you asked for:\n\n{\"status\": \"ok\"}\n\nLet me know if you need anything else.",
|
||||
b="{\"status\": \"ok\"}",
|
||||
defect="A violates 'only a JSON object' by wrapping it in prose."),
|
||||
dict(id="p09-fact2", truth="B",
|
||||
q="Which HTTP status code means 'Too Many Requests'?",
|
||||
a="HTTP 503 means Too Many Requests.",
|
||||
b="HTTP 429 means Too Many Requests.",
|
||||
defect="A is wrong; 503 is Service Unavailable, 429 is Too Many Requests."),
|
||||
dict(id="p10-hallu2", truth="B",
|
||||
q="Does Python's dict preserve insertion order?",
|
||||
a="No — dicts are unordered by definition; you must use collections.OrderedDict if order matters.",
|
||||
b="Yes — insertion order has been an implementation detail since CPython 3.6 and a language guarantee since 3.7.",
|
||||
defect="A states the pre-3.7 behaviour as current fact."),
|
||||
|
||||
# --- genuine ties ------------------------------------------------
|
||||
dict(id="p11-tie", truth="tie",
|
||||
q="What is the capital of Japan?",
|
||||
a="Tokyo.",
|
||||
b="The capital of Japan is Tokyo.",
|
||||
defect="Both correct and equivalent; phrasing differs only."),
|
||||
dict(id="p12-tie", truth="tie",
|
||||
q="Give one reason to use a context manager for file handles.",
|
||||
a="It closes the file automatically even if an exception is raised.",
|
||||
b="The file is guaranteed to be closed on exit from the block, including on error paths.",
|
||||
defect="Both correct, equivalent content, different wording."),
|
||||
]
|
||||
|
||||
ABSOLUTE = [
|
||||
# --- should score LOW (1-2) --------------------------------------
|
||||
dict(id="a01-low", band=(1, 2),
|
||||
q="How do I check which process is listening on port 8080 on Linux?",
|
||||
r="You should check the port. There are tools for this. Look at your system documentation.",
|
||||
why="Evasive, gives no actionable method."),
|
||||
dict(id="a02-low", band=(1, 2),
|
||||
q="What does `chmod 600 file` do?",
|
||||
r="It makes the file executable by everyone and readable by the group.",
|
||||
why="Factually wrong on both counts."),
|
||||
dict(id="a03-low", band=(1, 2),
|
||||
q="Summarise the purpose of a load balancer in one sentence.",
|
||||
r="Load balancers are very important in modern infrastructure and have been used for many years by "
|
||||
"many companies. There are hardware ones and software ones. Nginx is popular. So is HAProxy. "
|
||||
"Cloud providers offer managed ones too.",
|
||||
why="Ignores the one-sentence constraint and never states the purpose."),
|
||||
dict(id="a04-low", band=(1, 2),
|
||||
q="Is 17 a prime number?",
|
||||
r="No, 17 is not prime because it is divisible by 3.",
|
||||
why="Wrong answer with a wrong justification."),
|
||||
|
||||
# --- should score HIGH (4-5) -------------------------------------
|
||||
dict(id="a05-high", band=(4, 5),
|
||||
q="How do I check which process is listening on port 8080 on Linux?",
|
||||
r="Use `ss -ltnp | grep :8080` (or `sudo lsof -i :8080`). Both show the listening socket and the owning "
|
||||
"PID/process; `ss` needs root to reveal the process name for sockets you do not own.",
|
||||
why="Accurate, actionable, includes the privilege caveat."),
|
||||
dict(id="a06-high", band=(4, 5),
|
||||
q="What does `chmod 600 file` do?",
|
||||
r="It sets read and write for the owner and removes all permissions for group and others, so only the "
|
||||
"file's owner (and root) can read or modify it.",
|
||||
why="Precisely correct."),
|
||||
dict(id="a07-high", band=(4, 5),
|
||||
q="Summarise the purpose of a load balancer in one sentence.",
|
||||
r="A load balancer distributes incoming requests across multiple backend servers so that no single server "
|
||||
"is overwhelmed and the service survives individual backend failures.",
|
||||
why="Correct, complete, obeys the one-sentence constraint."),
|
||||
dict(id="a08-high", band=(4, 5),
|
||||
q="Is 17 a prime number?",
|
||||
r="Yes. 17 has no divisors other than 1 and itself — it is not divisible by 2, 3, or any integer up to its square root (~4.1).",
|
||||
why="Correct with valid reasoning."),
|
||||
|
||||
# --- should score MIDDLE (3) -------------------------------------
|
||||
dict(id="a09-mid", band=(2, 4),
|
||||
q="How do I check which process is listening on port 8080 on Linux?",
|
||||
r="Use netstat.",
|
||||
why="Correct direction, no flags, no explanation, deprecated tool."),
|
||||
dict(id="a10-mid", band=(2, 4),
|
||||
q="What does `chmod 600 file` do?",
|
||||
r="It restricts the file so other users cannot read it.",
|
||||
why="True but incomplete — omits owner rw and the group dimension."),
|
||||
dict(id="a11-mid", band=(2, 4),
|
||||
q="Name two advantages of connection pooling.",
|
||||
r="It is faster.",
|
||||
why="Partially responsive: one vague advantage where two were asked."),
|
||||
dict(id="a12-mid", band=(2, 4),
|
||||
q="Is 17 a prime number?",
|
||||
r="Yes.",
|
||||
why="Correct but bare — no justification for a question inviting one."),
|
||||
]
|
||||
@@ -0,0 +1,155 @@
|
||||
"""Re-run selene on its NATIVE Atla template — the fairness check.
|
||||
|
||||
selene lost on the neutral prompt. Before recommending anyone repoint away
|
||||
from it, test it on the prompt shape it was actually fine-tuned for: Atla's
|
||||
Selene template uses an explicit rubric block and a `**Reasoning:** /
|
||||
**Result:**` output contract rather than JSON.
|
||||
|
||||
Caveat worth stating: this template is reproduced from the published Selene
|
||||
prompt format. If it is subtly off, that is a confound — but the structural
|
||||
features that matter (rubric block, Reasoning/Result contract, no JSON
|
||||
demand) are reproduced faithfully, and those are what a fine-tune keys on.
|
||||
|
||||
gen is run on the SAME native template too, so this is not "selene's best
|
||||
prompt vs gen's worst" — it is both models on selene's home turf.
|
||||
"""
|
||||
import json
|
||||
import pathlib
|
||||
import re
|
||||
import statistics
|
||||
import sys
|
||||
import time
|
||||
import urllib.request
|
||||
|
||||
GATEWAY = "http://10.250.50.70:4000/v1/chat/completions"
|
||||
KEY = sys.argv[1]
|
||||
REPEATS = 3
|
||||
|
||||
sys.path.insert(0, str(pathlib.Path(__file__).resolve().parent))
|
||||
from items import ABSOLUTE, PAIRWISE # noqa: E402
|
||||
|
||||
ABS_RUBRIC = """1: The response is wrong, evasive, or ignores the question.
|
||||
2: The response is mostly unhelpful or contains a clear error.
|
||||
3: The response is partially correct but incomplete or thin.
|
||||
4: The response is correct and helpful with minor gaps.
|
||||
5: The response is correct, complete, and follows all instructions."""
|
||||
|
||||
NATIVE_ABS = """You are tasked with evaluating a response based on a given instruction and a scoring rubric that serve as the evaluation standard. Provide a comprehensive feedback on the response quality strictly adhering to the scoring rubric, without any general evaluation. Follow this with a score between 1 and 5, referring to the scoring rubric. Avoid generating any additional opening, closing, or explanations.
|
||||
|
||||
Here are some rules of the evaluation:
|
||||
(1) You should prioritize evaluating whether the response satisfies the provided rubric. The basis of your score should depend exactly on the rubric.
|
||||
|
||||
Your reply should strictly follow this format:
|
||||
**Reasoning:** <Your feedback>
|
||||
|
||||
**Result:** <an integer between 1 and 5>
|
||||
|
||||
Here is the data:
|
||||
|
||||
Instruction:
|
||||
```
|
||||
{q}
|
||||
```
|
||||
|
||||
Response:
|
||||
```
|
||||
{r}
|
||||
```
|
||||
|
||||
Score Rubrics:
|
||||
{rubric}
|
||||
"""
|
||||
|
||||
NATIVE_PAIR = """You are tasked with evaluating two responses based on a given instruction and an evaluation criterion. Select the response that better satisfies the criterion. If the two responses are of equivalent quality, select Tie.
|
||||
|
||||
Your reply should strictly follow this format:
|
||||
**Reasoning:** <Your feedback>
|
||||
|
||||
**Result:** <A, B, or Tie>
|
||||
|
||||
Here is the data:
|
||||
|
||||
Instruction:
|
||||
```
|
||||
{q}
|
||||
```
|
||||
|
||||
Response A:
|
||||
```
|
||||
{a}
|
||||
```
|
||||
|
||||
Response B:
|
||||
```
|
||||
{b}
|
||||
```
|
||||
|
||||
Evaluation Criterion:
|
||||
Factual accuracy, adherence to the instructions given, relevance, and completeness.
|
||||
"""
|
||||
|
||||
|
||||
def call(model, prompt):
|
||||
body = json.dumps({"model": model, "messages": [{"role": "user", "content": prompt}],
|
||||
"temperature": 0, "max_tokens": 400}).encode()
|
||||
req = urllib.request.Request(GATEWAY, data=body, method="POST")
|
||||
req.add_header("Authorization", "Bearer " + KEY)
|
||||
req.add_header("Content-Type", "application/json")
|
||||
t0 = time.time()
|
||||
try:
|
||||
with urllib.request.urlopen(req, timeout=120) as r:
|
||||
d = json.loads(r.read().decode())
|
||||
return d["choices"][0]["message"]["content"], time.time() - t0
|
||||
except Exception as e:
|
||||
return "ERROR %s" % e, time.time() - t0
|
||||
|
||||
|
||||
def res_score(t):
|
||||
m = re.search(r'\*\*Result:?\*\*\s*:?\s*([1-5])', t)
|
||||
if m:
|
||||
return int(m.group(1))
|
||||
m = re.search(r'Result:?\s*([1-5])', t)
|
||||
return int(m.group(1)) if m else None
|
||||
|
||||
|
||||
def res_verdict(t):
|
||||
m = re.search(r'\*\*Result:?\*\*\s*:?\s*(Tie|A|B)\b', t, re.I)
|
||||
if not m:
|
||||
m = re.search(r'Result:?\s*(Tie|A|B)\b', t, re.I)
|
||||
if not m:
|
||||
return None
|
||||
v = m.group(1)
|
||||
return "tie" if v.lower() == "tie" else v.upper()
|
||||
|
||||
|
||||
for model in ("selene-1-mini-8b", "gen"):
|
||||
print("\n=== %s on the NATIVE Atla template ===" % model, flush=True)
|
||||
lat, pa, ab = [], 0, 0
|
||||
for it in PAIRWISE:
|
||||
vs = []
|
||||
for _ in range(REPEATS):
|
||||
t, dt = call(model, NATIVE_PAIR.format(q=it["q"], a=it["a"], b=it["b"]))
|
||||
lat.append(dt)
|
||||
vs.append(res_verdict(t))
|
||||
modal = max(set(vs), key=vs.count)
|
||||
ok = modal == it["truth"]
|
||||
pa += ok
|
||||
print(" %-16s truth=%-4s got=%-20s %s%s" % (
|
||||
it["id"], it["truth"], str(vs), "OK " if ok else "MISS",
|
||||
"" if len(set(vs)) == 1 else " <UNSTABLE>"), flush=True)
|
||||
for it in ABSOLUTE:
|
||||
ss = []
|
||||
for _ in range(REPEATS):
|
||||
t, dt = call(model, NATIVE_ABS.format(q=it["q"], r=it["r"], rubric=ABS_RUBRIC))
|
||||
lat.append(dt)
|
||||
ss.append(res_score(t))
|
||||
got = [s for s in ss if s is not None]
|
||||
med = statistics.median(got) if got else None
|
||||
lo, hi = it["band"]
|
||||
ok = med is not None and lo <= med <= hi
|
||||
ab += ok
|
||||
print(" %-16s band=%-6s got=%-16s %s%s" % (
|
||||
it["id"], "%d-%d" % it["band"], str(ss), "OK " if ok else "MISS",
|
||||
"" if len(set(ss)) == 1 else " <UNSTABLE>"), flush=True)
|
||||
print(" --> pairwise %d/12 absolute %d/12 TOTAL %d/24 (%.0f%%) median lat %.2fs" % (
|
||||
pa, ab, pa + ab, 100 * (pa + ab) / 24, statistics.median(lat)), flush=True)
|
||||
@@ -0,0 +1,192 @@
|
||||
"""Run the judge benchmark against two gateway models and score them.
|
||||
|
||||
Fairness rules, stated so they can be argued with:
|
||||
- IDENTICAL prompts to both models. Selene has an Atla-native template;
|
||||
using it for selene and a generic one for gen would confound model
|
||||
quality with prompt fit. Consumers reach both through the same gateway
|
||||
with the same prompt, so the neutral prompt IS the production condition.
|
||||
(If selene loses badly, re-test with its native template before
|
||||
concluding anything — that is the honest follow-up, not this run.)
|
||||
- Same sampling params, same parser, same retry policy.
|
||||
- temperature 0: a judge should be as deterministic as the stack allows.
|
||||
Residual variation across repeats is then a real property, not noise we
|
||||
dialled in.
|
||||
- Repeats measure stability, not accuracy: a judge that flips its verdict
|
||||
on identical input is unusable regardless of how often it is right.
|
||||
"""
|
||||
import json
|
||||
import pathlib
|
||||
import re
|
||||
import statistics
|
||||
import sys
|
||||
import time
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
|
||||
import argparse
|
||||
|
||||
ap = argparse.ArgumentParser(description="Judge benchmark: compare two gateway models on LLM-as-judge work.")
|
||||
ap.add_argument("key", help="LiteLLM virtual key")
|
||||
ap.add_argument("--models", nargs=2, metavar=("A", "B"), required=True,
|
||||
help="the two model names to compare. REQUIRED — there is no sensible\n"
|
||||
"default pair, and a stale default would silently benchmark a\n"
|
||||
"retired seat (the original default, selene-1-mini-8b, now 400s).")
|
||||
ap.add_argument("--repeats", type=int, default=3,
|
||||
help="runs per item; measures stability, not accuracy (default: 3)")
|
||||
ap.add_argument("--limit", type=int, default=0,
|
||||
help="cap items per mode, for smoke tests (default: 0 = all)")
|
||||
ap.add_argument("--gateway", default="http://10.250.50.70:4000/v1/chat/completions")
|
||||
args = ap.parse_args()
|
||||
|
||||
GATEWAY = args.gateway
|
||||
KEY = args.key
|
||||
MODELS = args.models
|
||||
REPEATS = args.repeats
|
||||
|
||||
sys.path.insert(0, str(pathlib.Path(__file__).resolve().parent))
|
||||
from items import PAIRWISE, ABSOLUTE # noqa: E402
|
||||
|
||||
if args.limit:
|
||||
PAIRWISE = PAIRWISE[:args.limit]
|
||||
ABSOLUTE = ABSOLUTE[:args.limit]
|
||||
|
||||
PAIR_PROMPT = """You are an impartial judge evaluating two AI responses.
|
||||
|
||||
Question:
|
||||
{q}
|
||||
|
||||
Response A:
|
||||
{a}
|
||||
|
||||
Response B:
|
||||
{b}
|
||||
|
||||
Decide which response is better. Consider factual accuracy, whether the
|
||||
response follows the instructions given, relevance, and completeness.
|
||||
|
||||
Reply with ONLY a JSON object, no other text:
|
||||
{{"verdict": "A" or "B" or "tie", "reason": "<one short sentence>"}}"""
|
||||
|
||||
ABS_PROMPT = """You are an impartial judge scoring a single AI response.
|
||||
|
||||
Question:
|
||||
{q}
|
||||
|
||||
Response:
|
||||
{r}
|
||||
|
||||
Score the response from 1 to 5:
|
||||
1 = wrong, evasive, or ignores the question
|
||||
2 = mostly unhelpful or contains a clear error
|
||||
3 = partially correct but incomplete or thin
|
||||
4 = correct and helpful with minor gaps
|
||||
5 = correct, complete, and follows all instructions
|
||||
|
||||
Reply with ONLY a JSON object, no other text:
|
||||
{{"score": <integer 1-5>, "reason": "<one short sentence>"}}"""
|
||||
|
||||
|
||||
def call(model, prompt, timeout=120):
|
||||
body = json.dumps({
|
||||
"model": model,
|
||||
"messages": [{"role": "user", "content": prompt}],
|
||||
"temperature": 0,
|
||||
"max_tokens": 200,
|
||||
}).encode()
|
||||
req = urllib.request.Request(GATEWAY, data=body, method="POST")
|
||||
req.add_header("Authorization", "Bearer " + KEY)
|
||||
req.add_header("Content-Type", "application/json")
|
||||
t0 = time.time()
|
||||
try:
|
||||
with urllib.request.urlopen(req, timeout=timeout) as r:
|
||||
d = json.loads(r.read().decode())
|
||||
return d["choices"][0]["message"]["content"], time.time() - t0, None
|
||||
except urllib.error.HTTPError as e:
|
||||
return None, time.time() - t0, "HTTP %s: %s" % (e.code, e.read().decode()[:120])
|
||||
except Exception as e:
|
||||
return None, time.time() - t0, "%s: %s" % (type(e).__name__, e)
|
||||
|
||||
|
||||
def parse(text, key):
|
||||
"""Extract the field. Tolerant of fenced/prefixed output on purpose:
|
||||
format strictness is measured separately as `clean`, so a model is not
|
||||
scored wrong merely for wrapping valid JSON in a code fence."""
|
||||
if text is None:
|
||||
return None, False
|
||||
clean = text.strip().startswith("{") and text.strip().endswith("}")
|
||||
m = re.search(r'\{.*\}', text, re.S)
|
||||
if m:
|
||||
try:
|
||||
v = json.loads(m.group(0)).get(key)
|
||||
if v is not None:
|
||||
return v, clean
|
||||
except Exception:
|
||||
pass
|
||||
if key == "verdict":
|
||||
m = re.search(r'\b(tie|A|B)\b', text)
|
||||
return (m.group(1) if m else None), False
|
||||
m = re.search(r'\b([1-5])\b', text)
|
||||
return (int(m.group(1)) if m else None), False
|
||||
|
||||
|
||||
results = {m: {"pair": [], "abs": [], "lat": [], "errors": 0} for m in MODELS}
|
||||
|
||||
for model in MODELS:
|
||||
print("\n=== %s ===" % model, flush=True)
|
||||
for it in PAIRWISE:
|
||||
verdicts, clean_n = [], 0
|
||||
for _ in range(REPEATS):
|
||||
txt, dt, err = call(model, PAIR_PROMPT.format(q=it["q"], a=it["a"], b=it["b"]))
|
||||
results[model]["lat"].append(dt)
|
||||
if err:
|
||||
results[model]["errors"] += 1
|
||||
verdicts.append(None)
|
||||
continue
|
||||
v, clean = parse(txt, "verdict")
|
||||
v = v.strip() if isinstance(v, str) else v
|
||||
if isinstance(v, str) and v.lower() == "tie":
|
||||
v = "tie"
|
||||
verdicts.append(v)
|
||||
clean_n += int(clean)
|
||||
modal = max(set(verdicts), key=verdicts.count) if verdicts else None
|
||||
results[model]["pair"].append(dict(
|
||||
id=it["id"], truth=it["truth"], verdicts=verdicts, modal=modal,
|
||||
correct=(modal == it["truth"]),
|
||||
stable=(len(set(verdicts)) == 1), clean=clean_n))
|
||||
print(" %-16s truth=%-4s got=%-14s %s%s" % (
|
||||
it["id"], it["truth"], str(verdicts),
|
||||
"OK " if modal == it["truth"] else "MISS",
|
||||
"" if len(set(verdicts)) == 1 else " <UNSTABLE>"), flush=True)
|
||||
|
||||
for it in ABSOLUTE:
|
||||
scores, clean_n = [], 0
|
||||
for _ in range(REPEATS):
|
||||
txt, dt, err = call(model, ABS_PROMPT.format(q=it["q"], r=it["r"]))
|
||||
results[model]["lat"].append(dt)
|
||||
if err:
|
||||
results[model]["errors"] += 1
|
||||
scores.append(None)
|
||||
continue
|
||||
s, clean = parse(txt, "score")
|
||||
try:
|
||||
s = int(s)
|
||||
except Exception:
|
||||
s = None
|
||||
scores.append(s)
|
||||
clean_n += int(clean)
|
||||
got = [s for s in scores if s is not None]
|
||||
med = statistics.median(got) if got else None
|
||||
lo, hi = it["band"]
|
||||
results[model]["abs"].append(dict(
|
||||
id=it["id"], band=it["band"], scores=scores, median=med,
|
||||
in_band=(med is not None and lo <= med <= hi),
|
||||
stable=(len(set(scores)) == 1), clean=clean_n))
|
||||
print(" %-16s band=%-6s got=%-16s %s%s" % (
|
||||
it["id"], "%d-%d" % it["band"], str(scores),
|
||||
"OK " if (med is not None and lo <= med <= hi) else "MISS",
|
||||
"" if len(set(scores)) == 1 else " <UNSTABLE>"), flush=True)
|
||||
|
||||
out = str(pathlib.Path(__file__).resolve().parent / "raw.json")
|
||||
with open(out, "w") as f:
|
||||
json.dump(results, f, indent=1)
|
||||
print("\nraw -> %s" % out)
|
||||
Reference in New Issue
Block a user