feat(judge-bench): keep the judge harness; warn about the 7-way alias collision

Operator: keep the benchmark. It has a named second use (brokkr-smithy-dev
wants gen vs a trained reward model once their tournament converges) and a
demonstrated first one -- it caught a seat that had been coin-flip-grade for
five weeks with nobody measuring it.

Harness promoted from scratch to tools/judge-bench/:
- paths de-hardcoded; runs from its own directory
- proper CLI: --models (REQUIRED), --repeats, --limit, --gateway.
  Required on purpose: a stale default would silently benchmark a retired
  seat, and the original default (selene-1-mini-8b) now 400s.
- README states the limitation rather than burying it: 24 items of the
  author's own design, a screen and not a verdict. This harness scored the
  same pair 83 vs 96 while brokkr's corpus ranking task scored it 47 (chance)
  vs 94. Both honest; absolute scoring on designed items is an easier task
  than ranking real text.
- records brokkr's technique, which is better than anything here: a control
  constructed so the correct answer is DEFINITIONAL rather than judged cannot
  inherit the designer's error (item vs itself, response vs its own
  truncation, text vs its own clauses permuted). Add those before adding more
  judged items.

Gateway: comment-only warning at the head of model_list. SEVEN aliases now
resolve to the same weights (chat-judge, classifier, gen, image-judge,
qwen-image-bench, summarizer, summarizer-large -> qwen3.8-27b-uncensored).
That is intended under ADR-0012, but it has a sharp edge brokkr flagged:
cross-checking a result against another alias measures NOTHING when they are
the same model -- agreement is an echo, not corroboration. The note names the
other current collisions (glm-5.2 x4, TTS x4, reranker x2), gives the
/model/info one-liner to check, and records that probes should resolve alias
-> backing at run start AND end because the response `model` field returns the
alias, so a swap is otherwise invisible.

Verified: config still parses, diff is comment-only, canonical re-synced.
This commit is contained in:
vh
2026-08-23 05:16:10 -07:00
parent ca3c984f93
commit b8a535507a
5 changed files with 615 additions and 0 deletions
+33
View File
@@ -17,6 +17,39 @@
# route — left direct; see README.
model_list:
# =============================================================================
# ⚠ ALIAS COLLISION — SEVERAL NAMES, ONE SET OF WEIGHTS
#
# As of 2026-08-23 these SEVEN aliases all resolve to the same backend
# (qwen3.8-27b-uncensored @ 10.250.50.54:8015):
#
# chat-judge classifier gen image-judge
# qwen-image-bench summarizer summarizer-large
#
# They differ only in sampler params. That is intended — role aliases exist so
# consumers bind a CAPABILITY and the backing model can move (ADR-0012) — but
# it has a sharp edge that has to be stated where people read it:
#
# DO NOT "CROSS-CHECK" A RESULT BY RUNNING IT AGAINST ANOTHER ALIAS.
# Asking `gen` and then `summarizer` and finding they agree measures NOTHING:
# it is the same weights answering twice. Agreement between colliding aliases
# is not corroboration, it is an echo. Flagged by brokkr-smithy-dev
# 2026-08-23 while wiring provenance into a probe harness.
#
# Other current collisions: gen-frontier / gen-frontier-reasoning / glm-5.2 /
# glm-5.2-reasoning -> glm-5.2; ext-tts / gpt-4o-mini-tts / tts-1 / tts-1-hd ->
# the fleet TTS gateway; reranker / reranker-a3-bge-v2-m3 -> bge-reranker-v2-m3.
#
# TO CHECK BEFORE RELYING ON TWO ALIASES BEING DIFFERENT MODELS:
# curl -s :4000/model/info -H "Authorization: Bearer <key>" \
# | python3 -c "import json,sys;[print(r['model_name'], r['litellm_params'].get('model')) for r in json.load(sys.stdin)['data']]"
#
# Probes recording provenance should resolve alias -> backing model at run
# START and END and void the run on a mismatch: the response `model` field
# returns the ALIAS, so a mid-run or between-run swap is otherwise invisible.
# =============================================================================
# --- Granite 4.1 8B (generative chat) — production summarizer + dreaming
# agent. Replaced phi4-mini 2026-06-05 (beat it on precision in brokkr's
# R15 P03 eval). vLLM on ana-ml2 GPU 1, official FP8, 50K ctx. Explicit