From 53f00b232ba88d1bcb03bb521e1d0e41b6c9ce50 Mon Sep 17 00:00:00 2001
From: Vuong Hoang <vh@phasefinal.com>
Date: Wed, 13 May 2026 16:19:01 -0700
Subject: [PATCH] llama-swap: add selene-1-mini-8b (Atla judge model, imatrix
 Q6_K)
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

AtlaAI's Selene-1-Mini judge model for evaluation/scoring tasks.
Llama 3.1 8B base, mradermacher imatrix-quantized Q6_K (~6.5GB,
quality-leaning quant). Apache-2.0. Per Atla cookbook these defaults
hit 84% on RAGTruth hallucination eval.

New 'JUDGE / EVAL MODELS' section between the dense chat models and
the embedding models — separate category from chat/reasoning since
the run-params shape is different (deterministic-leaning: temp 0.01,
top-p 1.0, no repeat penalty).

q8_0 KV cache to fit 32K ctx cleanly on the 3090 with headroom.

Pre-pulled into the shared HF cache via the new
playbooks/pull-hf-model.yaml playbook (canonical replacement for
ad-hoc huggingface_hub.snapshot_download calls; see CHANGELOG).

Smoke-tested 2026-05-13: GET /v1/models lists selene-1-mini-8b,
POST /v1/chat/completions returns expected output cleanly.
---
 stacks/llama-swap/conf/config.yaml | 29 +++++++++++++++++++++++++++++
 1 file changed, 29 insertions(+)

diff --git a/stacks/llama-swap/conf/config.yaml b/stacks/llama-swap/conf/config.yaml
index 226a73f..ac409ec 100644
--- a/stacks/llama-swap/conf/config.yaml
+++ b/stacks/llama-swap/conf/config.yaml
@@ -33,6 +33,10 @@
 #               via huggingface_hub.snapshot_download with hf_transfer.
 #               Both repos include mmproj-BF16.gguf — llama-server's -hf
 #               flag auto-loads it, so vision works without --mmproj.
+#   2026-05-13: Added selene-1-mini-8b (AtlaAI judge model, mradermacher
+#               imatrix Q6_K, ~6.5GB). New JUDGE / EVAL section. Pre-pulled
+#               via playbooks/pull-hf-model.yaml (canonical replacement
+#               for ad-hoc huggingface_hub.snapshot_download calls).
 # ============================================================================
 
 # Default 1200 seconds (20 min) to wait for model to be available to load.
@@ -447,6 +451,31 @@ models:
       --temp 0.0
       --top-p 1.0
 
+  # ==========================================================================
+  # JUDGE / EVAL MODELS
+  # - Designed for evaluation/scoring rather than chat/reasoning.
+  # - Deterministic-leaning defaults (low temp, no repeat penalty) so the
+  #   same prompt scores consistently across re-runs.
+  # ==========================================================================
+
+  "selene-1-mini-8b":
+    name: "Selene 1 Mini Llama 3.1 8B iMat-Q6_K"
+    description: "AtlaAI's small judge model. Llama 3.1 8B base, iMatrix-weighted Q6_K (~6.5GB; quality-leaning quant). Apache-2.0. 84% on RAGTruth hallucination eval per Atla cookbook with these defaults."
+    ttl: 600
+    cmd: |
+      /app/llama-server
+      --jinja
+      -hf mradermacher/Selene-1-Mini-Llama-3.1-8B-i1-GGUF:Q6_K
+      --port ${PORT}
+      --n-gpu-layers 999
+      --ctx-size 32768
+      --cache-type-k q8_0
+      --cache-type-v q8_0
+      --flash-attn on
+      --temp 0.01
+      --top-p 1.0
+      --repeat-penalty 1.0
+
   # ==========================================================================
   # EMBEDDING MODELS (persistent, always loaded)
   # ==========================================================================