llama-swap/config: add qwen3.6-35-a3b-heretic + qwen3.6-27b (both vision-capable)

Both models pre-pulled into /tank/aimodels/huggingface (HF_HOME=/hfcache
inside the container) via huggingface_hub.snapshot_download with
hf_transfer for parallel chunked download — heretic's 29 GB landed in
~4 min, unsloth's 26.5 GB in ~3:46 (~118 MB/s each).

heretic: llmfan46/Qwen3.6-35B-A3B-uncensored-heretic-GGUF:Q6_K
27b:     unsloth/Qwen3.6-27B-GGUF:UD-Q6_K_XL

Both repos include mmproj-BF16.gguf alongside the main GGUF, and
llama-server's -hf flag auto-loads the mmproj when present in the same
repo (-hf docs: "mmproj is also downloaded automatically if available").
So both entries get vision (image-text-to-text) without needing an
explicit --mmproj path. ttl: 600 (10-min idle unload), matching the
existing abliterated entry's style.
This commit is contained in:
2026-04-26 21:56:57 -07:00
parent dce0fb9b66
commit b48667f33a
+50
View File
@@ -27,6 +27,12 @@
# 2026-04-22: New `pinned` group — qwen3.5-9b + qwen3.6-35-a3b coexist
# in VRAM with persistent=true and ttl=0. Means the two
# can be called concurrently and never idle-unload.
# 2026-04-26: Added qwen3.6-35-a3b-heretic (llmfan46 uncensored heretic
# Q6_K, ~28GB) and qwen3.6-27b (unsloth dynamic UD-Q6_K_XL,
# ~26GB). Both pre-pulled into /tank/aimodels/huggingface
# via huggingface_hub.snapshot_download with hf_transfer.
# Both repos include mmproj-BF16.gguf — llama-server's -hf
# flag auto-loads it, so vision works without --mmproj.
# ============================================================================
# Default 1200 seconds (20 min) to wait for model to be available to load.
@@ -164,6 +170,50 @@ models:
--reasoning on
--reasoning-format deepseek
"qwen3.6-35-a3b-heretic":
name: "Qwen 3.6 35B-A3B Uncensored Heretic Q6_K (vision)"
description: "Qwen3.6 MoE, 3B active of 35B. llmfan46 uncensored 'heretic' Q6_K (~28GB) + auto-loaded BF16 mmproj for vision."
ttl: 600
cmd: |
/app/llama-server
--context-shift
--jinja
-hf llmfan46/Qwen3.6-35B-A3B-uncensored-heretic-GGUF:Q6_K
--port ${PORT}
--n-gpu-layers 999
--ctx-size 131072
--flash-attn on
--temp 1.0
--top-p 0.95
--top-k 20
--min-p 0.00
--presence-penalty 1.5
--repeat-penalty 1.0
--reasoning on
--reasoning-format deepseek
"qwen3.6-27b":
name: "Qwen 3.6 27B UD-Q6_K_XL (vision)"
description: "Qwen3.6 dense 27B. Unsloth dynamic Q6_K_XL (~26GB) + auto-loaded BF16 mmproj for vision."
ttl: 600
cmd: |
/app/llama-server
--context-shift
--jinja
-hf unsloth/Qwen3.6-27B-GGUF:UD-Q6_K_XL
--port ${PORT}
--n-gpu-layers 999
--ctx-size 131072
--flash-attn on
--temp 1.0
--top-p 0.95
--top-k 20
--min-p 0.00
--presence-penalty 1.5
--repeat-penalty 1.0
--reasoning on
--reasoning-format deepseek
# ==========================================================================
# NEMOTRON MODELS (KB-recommended settings)
# - General Chat: temp 1.0, top-p 1.0, min_p 0.01