Files

T

vh ca3c984f93 feat(selene): retire the seat; chat-judge -> gen, selene-1-mini-8b 404s by design

Benchmarked selene against gen on selene's own job: 24 designed judge items
with checkable ground truth, pairwise + absolute modes, 3 repeats, on BOTH a
neutral JSON prompt and Selene's native Atla template. 288 calls, all free
local.

  neutral JSON     selene 20/24 (83%)   gen 23/24 (96%)
  native Atla      selene 21/24 (88%)   gen 22/24 (92%)

gen won on both templates and selene's BEST sat below gen's WORST. Selene was
given its own fine-tuned template as a fairness check; it gained one point,
not the three it needed.

Decisive defect: selene cannot emit "tie" -- 0/2 on both templates, forcing a
winner on every equivalent pair. For eval work that is the case that matters
most. gen returned tie correctly on the JSON template. Selene also compressed
the 1-5 scale (clustered at 2s and 4s) where gen used it fully. Selene's only
win was ~3x latency, unexercised at ~60 calls/day with zero queueing.

TWO NAMES, TWO DIFFERENT TREATMENTS, deliberately:

- chat-judge -> repointed to gen. It is a ROLE alias and ADR-0012 says
  consumers bind the capability, not a concrete model. Sampler profile copied
  from image-judge (temp 0, top_p 1.0, top_k 1, thinking off) so the served
  config matches the benchmarked condition.

- selene-1-mini-8b -> REMOVED. It 404s. It was NOT aliased to gen. A
  served-name is a contract about what the model IS; answering it with a
  different model hides a material change behind a stable string. Operator
  ruling: "never repoint a named model at a different model's endpoint --
  that is intentionally misleading." Verified: the gateway now returns
  HTTP 400 "Invalid model name" for it.

Reclaimed 17.2 GiB on ana-ml2 GPU 1 (free 1,818 -> 19,450 MiB) on a card that
had under 2 GiB of headroom. gen already runs on GPU 0, so the judge role
moved onto an existing seat rather than allocating anything new.

Canonical litellm config synced from the host; ana-ml2 README and
recommended-model-settings updated. compose.yaml kept for reference, not
deployed.

2026-08-23 05:07:01 -07:00

4.9 KiB

Raw Blame History

ana-ml2

Primary AI inference host for PFI.

Network

LAN IP: 10.250.50.54 (in-band, OS-side)
BMC (OOB): 10.250.250.50 — Supermicro IPMI web UI at https://10.250.250.50 (homepage card: PFI-ANA-ML2 BMC)
SSH: standard port 22 on 10.250.50.54

Hardware

Chassis: Supermicro mid-range inferencing server (bare metal, NOT Dell / not the same box as sf-r630 / sfsrv-ana)
CPU: AMD EPYC 9254 24-core (96 threads)
RAM: 566 GB
GPUs: 2x NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation Edition (96 GB VRAM each, cc 12.0 / sm_120, GPU 0 and GPU 1) — upgraded 2026-06 from 2x RTX 6000 Ada (46 GB, cc 8.9). Blackwell adds native FP4 (NVFP4) tensor cores and doubles VRAM.
Storage: ZFS zroot (434 GB root) + tank pool (8.6 TB at /tank)
OS: Debian 13 (trixie), kernel 6.12.x
Docker: 29.3.1, runtimes: runc (default), nvidia, io.containerd.runc.v2

Key paths

Path	Purpose
`/opt/docker/compose/<stack>/`	Compose files
`/opt/docker/conf/<stack>/`	Config bind mounts
`/tank/aimodels/huggingface/`	HF cache (267 GB, pre-downloaded models)
`/tank/aimodels/llm/`	Legacy GGUF models (790 GB, referenced by llama-swap as `/models/`)
`/var/lib/docker/`	Docker data (on zroot)

Running stacks

Live inventory as of 2026-07-22. Each model is its own compose stack now (container vllm-<x> / llama-<x>); the vllm stack proper is just the embed/rerank/reward trio. GPUs are pinned per container via deploy.resources.reservations.devices[].device_ids.

GPU 0 — heavy RP / reasoning seats (~88/98 GB, hot serving path):

Container	Port	Served model	Quant	Ctx
`vllm-gen` (project `gen-seat`)	8015	`qwen3.8-27b-uncensored` — the "gen" hero seat (Qwen3.8-27B Heretic-abliterated, in-house NVFP4 W4A16 + grafted MTP)	NVFP4 W4A16 (compressed-tensors)	262k
`vllm-charrp-reasoning-nvfp4`	8018	`char-rp-reasoning` (R36 reasoning RP)	NVFP4 (modelopt)	256k

GPU 1 — light / eval / retrieval + char-RP GGUF (~91/98 GB, on-demand):

Container	Port	Served model	Quant	Ctx
`vllm-granite`	8004	`granite-4.1-8b` — fleet summarizer/classifier	FP8 (compressed-tensors)	131k
`llama-charrp`	8016	`Magidonia-24B-v4.3` Q6_K — char-RP (llama.cpp)	GGUF Q6_K	—
~~`vllm-selene`~~	~~8011~~	RETIRED 2026-08-23 — lost a head-to-head against `gen` on its own judge task (see `stacks/selene/README.md`); seat downed to reclaim 17.2 GiB on GPU 1. `selene-1-mini-8b` now 404s by design; use `chat-judge`.	—	—
`vllm-reward`	8003	`Skywork-Reward-V2-Llama-3.1-8B-AWQ` — reward classifier	AWQ	16k
`vllm-embed`	8001	`Qwen3-Embedding-0.6B`	—	8k
`vllm-rerank`	8002	`Qwen3-Reranker-0.6B`	—	8k

Infra / non-GPU:

Container	Port	Notes
`dockge`	5001	Docker stack management UI
`dozzle-agent`	7007	Log agent → Dozzle hub on ana-docker
`beszel-agent`	45876	Metrics agent → Beszel hub on ana-docker

Both cards run near-full (~7–10 GB headroom each) — adding a seat means placing it on the card with room or evicting a dormant one first.

Dormant (compose present on disk, containers stopped) — rollback / audition seats, safe to leave: mistral-medium-3.5, mistral-small-4(-heretic), ms32-24b-angel, qwen3.5-122b, qwopus3.5-122b, qwen35-vl, qwen36-vl, qwen36-27b-aeon, qwen-image-bench, vibevoice, comfyui, kokoro, parakeet, vllm-qwen3.

Retired:

llama-swap (former GGUF multiplexer on :9292) — replaced by dedicated per-model seats (e.g. llama-charrp); no longer running.
infinity — replaced by the vllm stack (originally vllm-qwen3, renamed 2026-05-13 when the stack expanded beyond Qwen3) after the upstream Infinity image stopped shipping a transformers build that knew Qwen3.
LibreChat (+ rag_api, vectordb, mongodb, meilisearch), searxng — removed from this host (searxng now on ana-docker fleet-wide).

Refresh state

scripts/refresh-server-info.sh ana-ml2

Latest snapshot: system-details.txt (regenerate as needed).

GPU allocation policy

Every seat is explicitly pinned via device_ids (no unpinned containers), and both cards run ~90% full:

GPU 0: the two heavy NVFP4 seats — vllm-gen (gen) and vllm-charrp-reasoning-nvfp4. The live serving path (near-100% util under load), ~42 + 45 GB.
GPU 1: everything else — reward, embed, rerank, and the Magidonia char-RP GGUF seat. Bursty/on-demand, idle between calls, ~91 GB resident.

Pin with deploy.resources.reservations.devices[].device_ids: ["<id>"] in compose. Each service caps its share with --gpu-memory-utilization; with both cards near-full, placing a new seat means freeing room (evict a dormant one) or trimming a neighbour's utilization first.

4.9 KiB Raw Blame History Unescape Escape