llama-swap: pin granite-4-small alongside qwen3.5-9b — stop swapping with qwen3.6-27b

Symptom: granite-4-small and qwen3.6-27b were evicting each other
when called in alternation. granite is the news-digest curator (fires
twice daily on cron) — being evicted means a cold reload (~5s) on
every digest tick, plus visible churn whenever the user uses 27b
concurrently.

Added granite-4-small to the `pinned` group as a persistent member.
~5-6 GB at Q4_K_M + 120K KV ≈ comfortable inside the existing pin
budget (qwen3.5-9b ~6 GB → ~12 GB total persistent). Single RTX 6000
Ada is 48 GB, leaves ~36 GB headroom for whichever non-pinned model
the user invokes (qwen3.6-27b at ~30 GB fits cleanly).

Updated the pinned group's docstring to capture the current member set
+ VRAM math + the historical context (qwen3.6-35-a3b was here, was
too heavy, got removed yesterday). Marked the granite ttl: 0 with the
matching "pinned — never unloads" comment as the other group members.
This commit is contained in:
2026-04-27 18:14:59 -07:00
parent 894e92275d
commit 7bc648672f
+19 -9
View File
@@ -419,7 +419,7 @@ models:
"granite-4-small":
name: "Granite 4.0 Small Q4_K_M"
description: "IBM Granite 4.0 Small. Deterministic utility model for structured tasks."
ttl: 0
ttl: 0 # pinned — member of the `pinned` group, never unloads
cmd: |
/app/llama-server
--context-shift
@@ -545,17 +545,27 @@ groups:
- "qwen3-embedding-0.6B"
- "qwen3-reranker-0.6B"
# Pinned general-purpose chat models. Coexist in VRAM, never unload.
# Members also have ttl: 0 individually so idle-timeout can't drop them.
# Currently just qwen3.5-9b — small enough (~6 GB at Q4 + KV) to be
# cheap to keep resident. qwen3.6-35-a3b was previously here but its
# ~29 GB at Q6_K_XL made other model loads OOM under memory pressure
# (persistent: true exempts members from eviction). It now sits with
# ttl: 0 outside any group — never idle-unloads, but the scheduler
# CAN evict it when another non-coexistent model is requested.
# Pinned general-purpose / utility models. Coexist in VRAM, never
# unload. Members also have ttl: 0 individually so idle-timeout can't
# drop them.
#
# Current pins:
# qwen3.5-9b — ~6 GB at Q4 + KV. General-purpose chat baseline.
# granite-4-small — ~5-6 GB at Q4_K_M + 120K KV. Used by news-digest
# curator twice daily; pinning avoids the cold-load
# latency and prevents qwen3.6-27b (and similar)
# from evicting it when both are needed concurrently.
# VRAM budget: ~12 GB persistent in the pin slot. Single RTX 6000 Ada
# is 48 GB, so this leaves ~36 GB for whichever non-pinned model the
# user invokes alongside (qwen3.6-27b at ~30 GB fits cleanly).
#
# qwen3.6-35-a3b WAS in this group; removed 2026-04-27 because its
# ~29 GB at Q6_K_XL pushed concurrent loads OOM. Now lives outside
# with ttl: 0 — never idle-unloads but evictable under memory pressure.
"pinned":
swap: false
exclusive: false
persistent: true
members:
- "qwen3.5-9b"
- "granite-4-small"