Benchmarked selene against gen on selene's own job: 24 designed judge items
with checkable ground truth, pairwise + absolute modes, 3 repeats, on BOTH a
neutral JSON prompt and Selene's native Atla template. 288 calls, all free
local.
neutral JSON selene 20/24 (83%) gen 23/24 (96%)
native Atla selene 21/24 (88%) gen 22/24 (92%)
gen won on both templates and selene's BEST sat below gen's WORST. Selene was
given its own fine-tuned template as a fairness check; it gained one point,
not the three it needed.
Decisive defect: selene cannot emit "tie" -- 0/2 on both templates, forcing a
winner on every equivalent pair. For eval work that is the case that matters
most. gen returned tie correctly on the JSON template. Selene also compressed
the 1-5 scale (clustered at 2s and 4s) where gen used it fully. Selene's only
win was ~3x latency, unexercised at ~60 calls/day with zero queueing.
TWO NAMES, TWO DIFFERENT TREATMENTS, deliberately:
- chat-judge -> repointed to gen. It is a ROLE alias and ADR-0012 says
consumers bind the capability, not a concrete model. Sampler profile copied
from image-judge (temp 0, top_p 1.0, top_k 1, thinking off) so the served
config matches the benchmarked condition.
- selene-1-mini-8b -> REMOVED. It 404s. It was NOT aliased to gen. A
served-name is a contract about what the model IS; answering it with a
different model hides a material change behind a stable string. Operator
ruling: "never repoint a named model at a different model's endpoint --
that is intentionally misleading." Verified: the gateway now returns
HTTP 400 "Invalid model name" for it.
Reclaimed 17.2 GiB on ana-ml2 GPU 1 (free 1,818 -> 19,450 MiB) on a card that
had under 2 GiB of headroom. gen already runs on GPU 0, so the judge role
moved onto an existing seat rather than allocating anything new.
Canonical litellm config synced from the host; ana-ml2 README and
recommended-model-settings updated. compose.yaml kept for reference, not
deployed.
The README's running-stacks table had drifted well behind reality (still listed
llama-swap + only the embed/rerank/reward trio). Regenerated system-details.txt
and rewrote the stacks + GPU-allocation sections from a live docker ps +
nvidia-smi (2026-07-22):
- GPU 0 (hot): vllm-aeon-gen (qwen3.6-35b-a3b-heretic, NVFP4) + vllm-charrp-
reasoning-nvfp4 (char-rp-reasoning, NVFP4), ~42+45 GB.
- GPU 1 (on-demand): granite-4.1-8b, selene-1-mini-8b, Skywork reward,
Qwen3 embed/rerank, and the Magidonia-24B char-RP GGUF (llama-charrp), ~91 GB.
- Recorded the dormant on-disk stacks and llama-swap's retirement.
ana-ml2 was upgraded 2026-06 from dual RTX 6000 Ada (46GB, cc 8.9) to
dual RTX PRO 6000 Blackwell Max-Q (96GB, cc 12.0 / sm_120). Update the
stale hardware facts across the workspace:
- CLAUDE.md servers table row
- servers/ana-ml2/README.md hardware spec (+ refreshed system-details.txt)
- stacks/vllm compose + .env.example FP8/KV comments (Ada cc 8.9 -> Blackwell cc 12.0)
- stacks/llama-swap config VRAM-budget comment (48GB -> 96GB, GPU-0 pin)
Also corrects the adjacent stale 'Phi-4-mini' comment in the granite
service block (the service has been Granite 4.1 8B since 34a43a0).
Doc/comment-only; no runtime change.
Two related changes shipped together. The stack rename is independent
but adding `vllm-reward` to the existing `vllm-qwen3` would have made
that name actively misleading.
**Rename:** `stacks/vllm-qwen3/ → stacks/vllm/`. Updated all in-repo
references (README.md root, servers/ana-ml2/, stacks/llama-swap/,
configs/restic/ana-ml2/, docs/runbooks/disaster-recovery.md). Two
intentional history mentions retained (servers/ana-ml2 + stacks/vllm
README).
**Add `vllm-reward` service:** serves Skywork-Reward-V2-Llama-3.1-8B-AWQ
on port 8003. The AWQ output is a locally-quantized model (not from HF),
so bind-mounts `/tank/aimodels/llm:/local-models:ro` rather than the
shared HF cache. Model config.json declares LlamaForSequenceClassification
which vLLM's pooling runner picks up automatically — produces a single
reward score per input via /classify.
**Flag note:** the user's spec listed `--task classify`, but vLLM 0.19.1
deprecated --task in favor of --runner pooling (model architecture in
config.json drives the classification head). Compose uses --runner
pooling with a comment explaining the substitution.
**GPU memory:** no rebalance needed — production had already tuned
EMBED/RERANK down from 0.40 to 0.20 each (canonical .env.example now
matches reality). Adding REWARD at 0.30 totals 0.70, leaving ~14 GB
headroom on the 48 GB Ada.
**Server-side:** brought existing vllm-qwen3 down, mv'd
/opt/docker/compose/vllm-qwen3 → /opt/docker/compose/vllm, appended
REWARD_* lines to existing .env (preserving API_KEY/HF_TOKEN), deployed
new compose via scripts/deploy-stack.sh, brought all 3 services up.
**Smoke tests:**
- /health on 8001/8002/8003 → 200
- /v1/models on 8003 → lists Skywork/Skywork-Reward-V2-Llama-3.1-8B-AWQ
with max_model_len 16384
- /classify with a sample conversation → returns LABEL_0 with prob 0.9999
(single-output regression-style reward score, expected shape for a
reward model)
Bundles the inventory expansion since 2026-04-22:
- New host dirs (READMEs + ssh-target where dir name doesn't resolve):
ana-nas, ana-wg, esh-vm-db, nh3-nas, pbs-ana, pbs-nh3.
- New PFI VM snapshots (registered + key-installed 2026-04-23):
ana-filebot, pfi-ana-webhost, pfi-postgres, pfi-pteradactyl,
pfi-tacticalrmm, sf-ana-container, sfsrv-ana (system + proxmox).
- servers/irv-ml1: ONBOARDING.md (the first-time setup notes from when
the host was brought into the fleet) + ssh-target (10.100.79.3 over
the WG tunnel — name doesn't DNS-resolve from this workstation).
- servers/{ana-ml2,pfi-pve,sf-r630}/README.md: updates to capture BMC
IPs, the iDRAC vs OS hostname distinction (sf-r630 hardware =
sfsrv-ana OS), and the ana-ml2 Supermicro BMC (10.250.250.50,
distinct from the Dell R750xs iDRAC).
- configs/homepage/docker.yaml: irv-ml1-docker provider added so
homepage auto-discovers irv-ml1's stacks over the WG tunnel.
- docs/orientation.md: narrative fleet overview written for fresh
Claude sessions — sites, backup architecture, governing principles,
gotchas, where-to-look guide. Pointed at from CLAUDE.md.
Captures the full workspace state built up to this point:
- CLAUDE.md + README.md describing conventions and the four-host fleet
(ana-ml2, ana-docker, nh3-docker, esh-docker-vm).
- Per-host notes under servers/<host>/ with ssh-target fallback files
and latest system-details snapshots (two in-compose credential leaks
scrubbed; the upstream compose files still need to move those to .env).
- scripts/: server_inspect.sh (read-only remote diagnostic),
refresh-server-info.sh (dir-driven discovery + snapshot capture with
validation warnings), add-host.sh, sync-stacks.sh (pull
compose/conf trees), deploy-stack.sh (push with per-file diff + prompt).
- stacks/: canonical compose for backrest, beszel, dozzle, llama-swap,
rest-server-ana, rest-server-nh3, vllm-qwen3, plus the retired
infinity reference. All use the .env-driven + traefik-net + homepage
label pattern.
- configs/restic/ana-docker/: first resticprofile config + pre-backup
hook (Synapse pg_dump, Seafile mysqldump, Vaultwarden SQLite); templates
for the other three hosts to come.
- docs/pfi/: general infrastructure reference carried over.
- .gitignore excludes .env, stacks-mirror/, and assorted secret/state
filenames to prevent re-leaks on later commits.
vh