vh
b6340519bc
Session captured for a context reset. Six new detail entries. THE OPEN ITEM: Anaheim's IPsec tunnel delivers ~550 Mbit/s aggregate against a circuit measured at 2,153 Mbit/s. Not WireGuard (it is IPsec on ana-gw), not CPU (idle), not crypto exhaustion (NPU-offloaded), not the fibre. Both tunnels negotiate aes256-sha1; AES-GCM proposed. Operator signalled authorization; execution pending, untracked by operator choice. Also recorded: selene retired after losing a head-to-head on its own job with chat-judge moved to gen and the model name left to 404; the 7-alias collision on the gen seat that makes cross-alias corroboration an echo; hrafn adopted and its CI found to have been reporting green while deploying nothing for its whole life; all three Worldtree instances de-armed from a 69-day-stale :latest and the Matrix homeserver re-plumbed to personal; every secret-bearing .env on ana-docker tightened to 0600; the pfi org closing the repo-creation half of the credential-migration directive; booth kept-board deletion and link pruning. Two entries under Tried and abandoned: the CI checkout assertion that broke the pipeline twice and was removed, and my proposal to alias a retired model name at a different model, which the operator correctly overruled. Index 271 -> 282 lines, under the 300 cap, so no archival fired.
3.8 KiB
[2026-08-23] selene seat retired after losing a head-to-head; 7 aliases share one seat
Why selene went
Benchmarked against gen on selene's own job — 24 designed judge items with
checkable ground truth, pairwise + absolute modes, 3 repeats, run on both a
neutral JSON prompt and Selene's native Atla template (288 calls, free local).
neutral JSON selene 20/24 (83%) gen 23/24 (96%)
native Atla selene 21/24 (88%) gen 22/24 (92%)
gen won on both templates and selene's best sat below gen's worst. Selene was given its own fine-tuned template as a fairness check before any recommendation; it gained one point, not three.
Decisive defect: selene cannot emit "tie" — 0/2 on both templates, forcing a winner on every equivalent pair. For eval work that is the case that matters.
brokkr-smithy-dev independently corroborated from the other end with a null
control (an excerpt compared against ITSELF, where tie is definitional):
chat-judge(selene) TIE 27/60 = 45%, gen 60/60 = 100%; ground-truth
recovery on real-corpus ranking selene 47% — chance vs gen 94%. My 83-vs-96
understated it: on a ranking task selene was a coin flip. Absolute scoring on
designed items is an easier task than ranking real text — the harness is a
screen, not a verdict, and its README says so.
Reclaimed 17.2 GiB on ana-ml2 GPU1 (free 1,818 -> 19,450 MiB).
The naming rule, restated the hard way
I proposed repointing selene-1-mini-8b at gen and was correctly overruled:
never repoint a named model at a different model's endpoint — that is intentionally misleading
chat-judge is a role alias (ADR-0012: consumers bind the capability) and
moved to gen with a deterministic judge profile copied from image-judge.
selene-1-mini-8b is a model name and was removed outright — it now returns
HTTP 400 Invalid model name, verified. The discriminator: does the string
promise a capability, or an identity?
The 7-way alias collision — the finding with the longest reach
chat-judge classifier gen image-judge
qwen-image-bench summarizer summarizer-large -> qwen3.8-27b-uncensored :8015
Also colliding: gen-frontier/gen-frontier-reasoning/glm-5.2/glm-5.2-reasoning;
ext-tts/gpt-4o-mini-tts/tts-1/tts-1-hd; reranker/reranker-a3-bge-v2-m3.
Cross-checking a result against another alias measures nothing when they are the
same weights — agreement is an echo, not corroboration. Documented at the head of
model_list in the live gateway config, because it belongs where people read it.
This caught a real defect within hours: brokkr's R47 premium-corpus gate was about
to run ~46,000 record-exposures against gen with summarizer shortlisted as an
independent second opinion. They pinned the backing model in the preregistration
and dropped the second-alias idea instead.
Provenance seam (brokkr's pushback, adopted)
The gateway returns the alias in the response model field, not the backing
model — so a per-call guard catches a swap during a run and is blind to one
between runs. Role alias for routing, concrete model for provenance.
GET :4000/model/info with the shared key already exposes backing model +
api_base; resolve at run start AND end and void on mismatch.
Artifacts
- Harness kept at
tools/judge-bench/(--modelsREQUIRED — a stale default would silently benchmark a retired seat). stacks/selene/keeps compose + a README explaining the retirement.- Technique worth stealing, from brokkr: a control constructed so the correct answer is DEFINITIONAL rather than judged cannot inherit the designer's error. Item vs itself; response vs its own truncation; text vs its own clauses permuted. Add those before adding more judged items.
Commits ca3c984, b8a5355.