llama-swap: add selene-1-mini-8b (Atla judge model, imatrix Q6_K)
AtlaAI's Selene-1-Mini judge model for evaluation/scoring tasks. Llama 3.1 8B base, mradermacher imatrix-quantized Q6_K (~6.5GB, quality-leaning quant). Apache-2.0. Per Atla cookbook these defaults hit 84% on RAGTruth hallucination eval. New 'JUDGE / EVAL MODELS' section between the dense chat models and the embedding models — separate category from chat/reasoning since the run-params shape is different (deterministic-leaning: temp 0.01, top-p 1.0, no repeat penalty). q8_0 KV cache to fit 32K ctx cleanly on the 3090 with headroom. Pre-pulled into the shared HF cache via the new playbooks/pull-hf-model.yaml playbook (canonical replacement for ad-hoc huggingface_hub.snapshot_download calls; see CHANGELOG). Smoke-tested 2026-05-13: GET /v1/models lists selene-1-mini-8b, POST /v1/chat/completions returns expected output cleanly.
This commit is contained in:
@@ -33,6 +33,10 @@
|
||||
# via huggingface_hub.snapshot_download with hf_transfer.
|
||||
# Both repos include mmproj-BF16.gguf — llama-server's -hf
|
||||
# flag auto-loads it, so vision works without --mmproj.
|
||||
# 2026-05-13: Added selene-1-mini-8b (AtlaAI judge model, mradermacher
|
||||
# imatrix Q6_K, ~6.5GB). New JUDGE / EVAL section. Pre-pulled
|
||||
# via playbooks/pull-hf-model.yaml (canonical replacement
|
||||
# for ad-hoc huggingface_hub.snapshot_download calls).
|
||||
# ============================================================================
|
||||
|
||||
# Default 1200 seconds (20 min) to wait for model to be available to load.
|
||||
@@ -447,6 +451,31 @@ models:
|
||||
--temp 0.0
|
||||
--top-p 1.0
|
||||
|
||||
# ==========================================================================
|
||||
# JUDGE / EVAL MODELS
|
||||
# - Designed for evaluation/scoring rather than chat/reasoning.
|
||||
# - Deterministic-leaning defaults (low temp, no repeat penalty) so the
|
||||
# same prompt scores consistently across re-runs.
|
||||
# ==========================================================================
|
||||
|
||||
"selene-1-mini-8b":
|
||||
name: "Selene 1 Mini Llama 3.1 8B iMat-Q6_K"
|
||||
description: "AtlaAI's small judge model. Llama 3.1 8B base, iMatrix-weighted Q6_K (~6.5GB; quality-leaning quant). Apache-2.0. 84% on RAGTruth hallucination eval per Atla cookbook with these defaults."
|
||||
ttl: 600
|
||||
cmd: |
|
||||
/app/llama-server
|
||||
--jinja
|
||||
-hf mradermacher/Selene-1-Mini-Llama-3.1-8B-i1-GGUF:Q6_K
|
||||
--port ${PORT}
|
||||
--n-gpu-layers 999
|
||||
--ctx-size 32768
|
||||
--cache-type-k q8_0
|
||||
--cache-type-v q8_0
|
||||
--flash-attn on
|
||||
--temp 0.01
|
||||
--top-p 1.0
|
||||
--repeat-penalty 1.0
|
||||
|
||||
# ==========================================================================
|
||||
# EMBEDDING MODELS (persistent, always loaded)
|
||||
# ==========================================================================
|
||||
|
||||
Reference in New Issue
Block a user