feat(selene+mistral): restore Selene judge (FP8, GPU1) + push Mistral to 256K

selene: AtlaAI Selene-1-Mini-Llama-3.1-8B judge restored on vLLM after the llama-swap teardown took its Q6_K GGUF offline. FP8 (dynamic --quantization fp8; FP8 >= the validated Q6_K fidelity, and text-only Llama so no vision- tower-noise risk; NVFP4's W4A4 too aggressive for a precision judge). GPU1 util 0.13 (8.51 GiB weights + 2.53 GiB KV, 32K ctx, 1.27x concurrency), ~11 GB GPU1 buffer left. Gateway selene-1-mini-8b → :8011 (shadows the * wildcard that used to reach it via llama-swap). Judge smoke: scored an unfaithful claim 1/5 correctly. mistral-small-4: max-model-len 131072 → 262144 (full native 256K) for novel-length consistency-checking. KV pool is util-bound (~862K tokens), so 256K costs no extra VRAM — max concurrency just drops to 3.29x at full length. max-num-seqs 64 → 32 keeps the warmup transient flat (scales with seqs × len), so it fits the tight GPU0 (free unchanged at 5.2 GB). Verified loaded + healthy.
2026-06-15 18:52:26 -07:00
parent 9a49963d07
commit c985ede07b
5 changed files with 142 additions and 14 deletions
@@ -0,0 +1,26 @@
+# Selene 1 Mini 8B judge (FP8) on ana-ml2 GPU 1 — copy to .env on the host.
+# Real .env lives on ana-ml2 at /opt/docker/compose/selene/.env (gitignored).
+#
+# See compose.yaml header for the FP8-over-NVFP4 (judge-fidelity) rationale.
+
+# 0.23.0 — Llama 3.1 + dynamic fp8 is rock-solid here (same digest as qwen36).
+SELENE_IMAGE=vllm/vllm-openai@sha256:6d8429e38e3747723ca07ee1b17972e09bb9c51c4032b266f24fb1cc3b22ed8f
+
+SELENE_CONTAINER_NAME=vllm-selene
+SELENE_MODEL=AtlaAI/Selene-1-Mini-Llama-3.1-8B
+SELENE_PORT=8011
+
+# GPU 1 = shared with qwen36 (NVFP4) + granite + embed/rerank/reward.
+SELENE_GPU_ID=1
+
+# util 0.13 (~12.5 GB) — measured: 8.51 GiB FP8 weights + ~1.5 GiB graph +
+# 2.53 GiB fp8 KV (41,456-token pool, 1.27x concurrency at full 32K). util 0.12
+# was too thin (1.86 GiB KV < the 2.0 GiB a single 32K request needs → crash).
+# Keeps GPU 1 total ~0.95 → ~5 GB buffer; ctx 32768 mirrors the old judge config.
+SELENE_GPU_MEM_UTIL=0.13
+SELENE_MAX_MODEL_LEN=32768
+SELENE_MAX_NUM_SEQS=16
+
+# Optional — checkpoint is ungated (Apache-2.0).
+HF_TOKEN=
+API_KEY=
@@ -0,0 +1,86 @@
+# selene — AtlaAI Selene 1 Mini (Llama 3.1 8B) judge/eval model on ana-ml2 GPU 1.
+#
+# Restores the judge that went offline when llama-swap was downed (it was the
+# Q6_K GGUF `selene-1-mini-8b` in the llama-swap zoo). Re-served on vLLM at the
+# operator's request, FP8 (NVFP4 had no pre-made checkpoint and W4A4 is too
+# aggressive for a precision judge validated at Q6_K — FP8 ≥ Q6_K fidelity).
+#
+# FP8 = vLLM DYNAMIC --quantization fp8 (W8A8) of the bf16 AtlaAI checkpoint —
+# no offline quant needed, near-lossless, and Selene is text-only Llama 3.1 so
+# there's NO vision tower for dynamic fp8 to noise-quantize (the qwen35-VL
+# footgun doesn't apply here). ~8 GiB weights on GPU 1's headroom.
+#
+# Co-tenant on GPU 1 with qwen36 (NVFP4) + granite + embed/rerank/reward. Sized
+# to fit the ~24 GB headroom while leaving GPU 1 a safe buffer (see .env).
+# Served ONLY as `selene-1-mini-8b` (the name its consumers know). All tunables
+# live in .env.
+
+name: selene
+
+services:
+  vllm-selene:
+    image: ${SELENE_IMAGE}
+    container_name: ${SELENE_CONTAINER_NAME}
+    restart: unless-stopped
+    ipc: host
+    ports:
+      - "${SELENE_PORT}:8000"
+    volumes:
+      - /tank/aimodels/huggingface:/hfcache
+    environment:
+      - HF_HOME=/hfcache
+      - HF_HUB_CACHE=/hfcache/hub
+      - HUGGING_FACE_HUB_TOKEN=${HF_TOKEN:-}
+      - VLLM_API_KEY=${API_KEY:-}
+    command:
+      - ${SELENE_MODEL}
+      # Dynamic FP8 (W8A8) from the bf16 checkpoint — no pre-quant needed.
+      - --quantization
+      - fp8
+      - --served-model-name
+      - selene-1-mini-8b
+      - --host
+      - 0.0.0.0
+      - --port
+      - "8000"
+      - --gpu-memory-utilization
+      - ${SELENE_GPU_MEM_UTIL}
+      - --max-model-len
+      - ${SELENE_MAX_MODEL_LEN}
+      - --max-num-seqs
+      - ${SELENE_MAX_NUM_SEQS}
+      # fp8 KV — matches the judge's old q8 KV posture + keeps the pool compact
+      # on the shared card.
+      - --kv-cache-dtype
+      - fp8
+      - --dtype
+      - auto
+      - --enable-prefix-caching
+    deploy:
+      resources:
+        reservations:
+          devices:
+            - driver: nvidia
+              device_ids:
+                - "${SELENE_GPU_ID}"
+              capabilities:
+                - gpu
+    healthcheck:
+      test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
+      interval: 30s
+      timeout: 10s
+      retries: 3
+      start_period: 180s
+    networks:
+      - tnet
+    labels:
+      - homepage.group=AI Systems
+      - homepage.name=Selene 1 Mini 8B (judge, FP8)
+      - homepage.icon=mdi-gavel
+      - homepage.description=AtlaAI Selene 1 Mini Llama-3.1-8B judge (FP8) via vLLM (ana-ml2 GPU1)
+      - homepage.href=http://10.250.50.54:${SELENE_PORT}/docs
+
+networks:
+  tnet:
+    name: traefik-net
+    external: true