llama-swap: add selene-1-mini-8b (Atla judge model, imatrix Q6_K)

AtlaAI's Selene-1-Mini judge model for evaluation/scoring tasks.
Llama 3.1 8B base, mradermacher imatrix-quantized Q6_K (~6.5GB,
quality-leaning quant). Apache-2.0. Per Atla cookbook these defaults
hit 84% on RAGTruth hallucination eval.

New 'JUDGE / EVAL MODELS' section between the dense chat models and
the embedding models — separate category from chat/reasoning since
the run-params shape is different (deterministic-leaning: temp 0.01,
top-p 1.0, no repeat penalty).

q8_0 KV cache to fit 32K ctx cleanly on the 3090 with headroom.

Pre-pulled into the shared HF cache via the new
playbooks/pull-hf-model.yaml playbook (canonical replacement for
ad-hoc huggingface_hub.snapshot_download calls; see CHANGELOG).

Smoke-tested 2026-05-13: GET /v1/models lists selene-1-mini-8b,
POST /v1/chat/completions returns expected output cleanly.
This commit is contained in:
2026-05-13 16:19:01 -07:00
parent f9b933808c
commit 53f00b232b
+29
View File
@@ -33,6 +33,10 @@
# via huggingface_hub.snapshot_download with hf_transfer.
# Both repos include mmproj-BF16.gguf — llama-server's -hf
# flag auto-loads it, so vision works without --mmproj.
# 2026-05-13: Added selene-1-mini-8b (AtlaAI judge model, mradermacher
# imatrix Q6_K, ~6.5GB). New JUDGE / EVAL section. Pre-pulled
# via playbooks/pull-hf-model.yaml (canonical replacement
# for ad-hoc huggingface_hub.snapshot_download calls).
# ============================================================================
# Default 1200 seconds (20 min) to wait for model to be available to load.
@@ -447,6 +451,31 @@ models:
--temp 0.0
--top-p 1.0
# ==========================================================================
# JUDGE / EVAL MODELS
# - Designed for evaluation/scoring rather than chat/reasoning.
# - Deterministic-leaning defaults (low temp, no repeat penalty) so the
# same prompt scores consistently across re-runs.
# ==========================================================================
"selene-1-mini-8b":
name: "Selene 1 Mini Llama 3.1 8B iMat-Q6_K"
description: "AtlaAI's small judge model. Llama 3.1 8B base, iMatrix-weighted Q6_K (~6.5GB; quality-leaning quant). Apache-2.0. 84% on RAGTruth hallucination eval per Atla cookbook with these defaults."
ttl: 600
cmd: |
/app/llama-server
--jinja
-hf mradermacher/Selene-1-Mini-Llama-3.1-8B-i1-GGUF:Q6_K
--port ${PORT}
--n-gpu-layers 999
--ctx-size 32768
--cache-type-k q8_0
--cache-type-v q8_0
--flash-attn on
--temp 0.01
--top-p 1.0
--repeat-penalty 1.0
# ==========================================================================
# EMBEDDING MODELS (persistent, always loaded)
# ==========================================================================