feat(judge-bench): keep the judge harness; warn about the 7-way alias collision
Operator: keep the benchmark. It has a named second use (brokkr-smithy-dev wants gen vs a trained reward model once their tournament converges) and a demonstrated first one -- it caught a seat that had been coin-flip-grade for five weeks with nobody measuring it. Harness promoted from scratch to tools/judge-bench/: - paths de-hardcoded; runs from its own directory - proper CLI: --models (REQUIRED), --repeats, --limit, --gateway. Required on purpose: a stale default would silently benchmark a retired seat, and the original default (selene-1-mini-8b) now 400s. - README states the limitation rather than burying it: 24 items of the author's own design, a screen and not a verdict. This harness scored the same pair 83 vs 96 while brokkr's corpus ranking task scored it 47 (chance) vs 94. Both honest; absolute scoring on designed items is an easier task than ranking real text. - records brokkr's technique, which is better than anything here: a control constructed so the correct answer is DEFINITIONAL rather than judged cannot inherit the designer's error (item vs itself, response vs its own truncation, text vs its own clauses permuted). Add those before adding more judged items. Gateway: comment-only warning at the head of model_list. SEVEN aliases now resolve to the same weights (chat-judge, classifier, gen, image-judge, qwen-image-bench, summarizer, summarizer-large -> qwen3.8-27b-uncensored). That is intended under ADR-0012, but it has a sharp edge brokkr flagged: cross-checking a result against another alias measures NOTHING when they are the same model -- agreement is an echo, not corroboration. The note names the other current collisions (glm-5.2 x4, TTS x4, reranker x2), gives the /model/info one-liner to check, and records that probes should resolve alias -> backing at run start AND end because the response `model` field returns the alias, so a swap is otherwise invisible. Verified: config still parses, diff is comment-only, canonical re-synced.
This commit is contained in:
@@ -17,6 +17,39 @@
|
||||
# route — left direct; see README.
|
||||
|
||||
model_list:
|
||||
|
||||
# =============================================================================
|
||||
# ⚠ ALIAS COLLISION — SEVERAL NAMES, ONE SET OF WEIGHTS
|
||||
#
|
||||
# As of 2026-08-23 these SEVEN aliases all resolve to the same backend
|
||||
# (qwen3.8-27b-uncensored @ 10.250.50.54:8015):
|
||||
#
|
||||
# chat-judge classifier gen image-judge
|
||||
# qwen-image-bench summarizer summarizer-large
|
||||
#
|
||||
# They differ only in sampler params. That is intended — role aliases exist so
|
||||
# consumers bind a CAPABILITY and the backing model can move (ADR-0012) — but
|
||||
# it has a sharp edge that has to be stated where people read it:
|
||||
#
|
||||
# DO NOT "CROSS-CHECK" A RESULT BY RUNNING IT AGAINST ANOTHER ALIAS.
|
||||
# Asking `gen` and then `summarizer` and finding they agree measures NOTHING:
|
||||
# it is the same weights answering twice. Agreement between colliding aliases
|
||||
# is not corroboration, it is an echo. Flagged by brokkr-smithy-dev
|
||||
# 2026-08-23 while wiring provenance into a probe harness.
|
||||
#
|
||||
# Other current collisions: gen-frontier / gen-frontier-reasoning / glm-5.2 /
|
||||
# glm-5.2-reasoning -> glm-5.2; ext-tts / gpt-4o-mini-tts / tts-1 / tts-1-hd ->
|
||||
# the fleet TTS gateway; reranker / reranker-a3-bge-v2-m3 -> bge-reranker-v2-m3.
|
||||
#
|
||||
# TO CHECK BEFORE RELYING ON TWO ALIASES BEING DIFFERENT MODELS:
|
||||
# curl -s :4000/model/info -H "Authorization: Bearer <key>" \
|
||||
# | python3 -c "import json,sys;[print(r['model_name'], r['litellm_params'].get('model')) for r in json.load(sys.stdin)['data']]"
|
||||
#
|
||||
# Probes recording provenance should resolve alias -> backing model at run
|
||||
# START and END and void the run on a mismatch: the response `model` field
|
||||
# returns the ALIAS, so a mid-run or between-run swap is otherwise invisible.
|
||||
# =============================================================================
|
||||
|
||||
# --- Granite 4.1 8B (generative chat) — production summarizer + dreaming
|
||||
# agent. Replaced phi4-mini 2026-06-05 (beat it on precision in brokkr's
|
||||
# R15 P03 eval). vLLM on ana-ml2 GPU 1, official FP8, 50K ctx. Explicit
|
||||
|
||||
Reference in New Issue
Block a user