vh
b6340519bc
Session captured for a context reset. Six new detail entries. THE OPEN ITEM: Anaheim's IPsec tunnel delivers ~550 Mbit/s aggregate against a circuit measured at 2,153 Mbit/s. Not WireGuard (it is IPsec on ana-gw), not CPU (idle), not crypto exhaustion (NPU-offloaded), not the fibre. Both tunnels negotiate aes256-sha1; AES-GCM proposed. Operator signalled authorization; execution pending, untracked by operator choice. Also recorded: selene retired after losing a head-to-head on its own job with chat-judge moved to gen and the model name left to 404; the 7-alias collision on the gen seat that makes cross-alias corroboration an echo; hrafn adopted and its CI found to have been reporting green while deploying nothing for its whole life; all three Worldtree instances de-armed from a 69-day-stale :latest and the Matrix homeserver re-plumbed to personal; every secret-bearing .env on ana-docker tightened to 0600; the pfi org closing the repo-creation half of the credential-migration directive; booth kept-board deletion and link pruning. Two entries under Tried and abandoned: the CI checkout assertion that broke the pipeline twice and was removed, and my proposal to alias a retired model name at a different model, which the operator correctly overruled. Index 271 -> 282 lines, under the 300 cap, so no archival fired.
82 lines
3.8 KiB
Markdown
82 lines
3.8 KiB
Markdown
# [2026-08-23] selene seat retired after losing a head-to-head; 7 aliases share one seat
|
|
|
|
## Why selene went
|
|
|
|
Benchmarked against `gen` on selene's own job — 24 designed judge items with
|
|
checkable ground truth, pairwise + absolute modes, 3 repeats, run on **both** a
|
|
neutral JSON prompt and Selene's **native Atla template** (288 calls, free local).
|
|
|
|
```
|
|
neutral JSON selene 20/24 (83%) gen 23/24 (96%)
|
|
native Atla selene 21/24 (88%) gen 22/24 (92%)
|
|
```
|
|
|
|
gen won on both templates and **selene's best sat below gen's worst**. Selene was
|
|
given its own fine-tuned template as a fairness check before any recommendation;
|
|
it gained one point, not three.
|
|
|
|
**Decisive defect: selene cannot emit "tie"** — 0/2 on both templates, forcing a
|
|
winner on every equivalent pair. For eval work that is the case that matters.
|
|
|
|
brokkr-smithy-dev independently corroborated from the other end with a **null
|
|
control** (an excerpt compared against ITSELF, where tie is definitional):
|
|
`chat-judge`(selene) TIE **27/60 = 45%**, gen **60/60 = 100%**; ground-truth
|
|
recovery on real-corpus ranking selene **47% — chance** vs gen 94%. My 83-vs-96
|
|
understated it: on a *ranking* task selene was a coin flip. Absolute scoring on
|
|
designed items is an easier task than ranking real text — the harness is a
|
|
**screen, not a verdict**, and its README says so.
|
|
|
|
Reclaimed **17.2 GiB** on ana-ml2 GPU1 (free 1,818 -> 19,450 MiB).
|
|
|
|
## The naming rule, restated the hard way
|
|
|
|
I proposed repointing `selene-1-mini-8b` at gen and was **correctly overruled**:
|
|
|
|
> never repoint a named model at a different model's endpoint — that is
|
|
> intentionally misleading
|
|
|
|
`chat-judge` is a **role** alias (ADR-0012: consumers bind the capability) and
|
|
moved to gen with a deterministic judge profile copied from `image-judge`.
|
|
`selene-1-mini-8b` is a **model** name and was removed outright — it now returns
|
|
`HTTP 400 Invalid model name`, verified. The discriminator: *does the string
|
|
promise a capability, or an identity?*
|
|
|
|
## The 7-way alias collision — the finding with the longest reach
|
|
|
|
```
|
|
chat-judge classifier gen image-judge
|
|
qwen-image-bench summarizer summarizer-large -> qwen3.8-27b-uncensored :8015
|
|
```
|
|
|
|
Also colliding: `gen-frontier`/`gen-frontier-reasoning`/`glm-5.2`/`glm-5.2-reasoning`;
|
|
`ext-tts`/`gpt-4o-mini-tts`/`tts-1`/`tts-1-hd`; `reranker`/`reranker-a3-bge-v2-m3`.
|
|
|
|
**Cross-checking a result against another alias measures nothing when they are the
|
|
same weights — agreement is an echo, not corroboration.** Documented at the head of
|
|
`model_list` in the live gateway config, because it belongs where people read it.
|
|
|
|
This caught a real defect within hours: brokkr's R47 premium-corpus gate was about
|
|
to run ~46,000 record-exposures against `gen` with `summarizer` shortlisted as an
|
|
independent second opinion. They pinned the backing model in the preregistration
|
|
and dropped the second-alias idea instead.
|
|
|
|
## Provenance seam (brokkr's pushback, adopted)
|
|
|
|
The gateway returns the **alias** in the response `model` field, not the backing
|
|
model — so a per-call guard catches a swap *during* a run and is blind to one
|
|
*between* runs. **Role alias for routing, concrete model for provenance.**
|
|
`GET :4000/model/info` with the shared key already exposes backing model +
|
|
api_base; resolve at run start AND end and void on mismatch.
|
|
|
|
## Artifacts
|
|
|
|
- Harness kept at `tools/judge-bench/` (`--models` REQUIRED — a stale default
|
|
would silently benchmark a retired seat).
|
|
- `stacks/selene/` keeps compose + a README explaining the retirement.
|
|
- Technique worth stealing, from brokkr: **a control constructed so the correct
|
|
answer is DEFINITIONAL rather than judged cannot inherit the designer's error.**
|
|
Item vs itself; response vs its own truncation; text vs its own clauses
|
|
permuted. Add those before adding more judged items.
|
|
|
|
Commits `ca3c984`, `b8a5355`.
|