llama-swap/config: add qwen3.6-35-a3b-heretic + qwen3.6-27b (both vision-capable)
Both models pre-pulled into /tank/aimodels/huggingface (HF_HOME=/hfcache inside the container) via huggingface_hub.snapshot_download with hf_transfer for parallel chunked download — heretic's 29 GB landed in ~4 min, unsloth's 26.5 GB in ~3:46 (~118 MB/s each). heretic: llmfan46/Qwen3.6-35B-A3B-uncensored-heretic-GGUF:Q6_K 27b: unsloth/Qwen3.6-27B-GGUF:UD-Q6_K_XL Both repos include mmproj-BF16.gguf alongside the main GGUF, and llama-server's -hf flag auto-loads the mmproj when present in the same repo (-hf docs: "mmproj is also downloaded automatically if available"). So both entries get vision (image-text-to-text) without needing an explicit --mmproj path. ttl: 600 (10-min idle unload), matching the existing abliterated entry's style.
This commit is contained in:
@@ -27,6 +27,12 @@
|
||||
# 2026-04-22: New `pinned` group — qwen3.5-9b + qwen3.6-35-a3b coexist
|
||||
# in VRAM with persistent=true and ttl=0. Means the two
|
||||
# can be called concurrently and never idle-unload.
|
||||
# 2026-04-26: Added qwen3.6-35-a3b-heretic (llmfan46 uncensored heretic
|
||||
# Q6_K, ~28GB) and qwen3.6-27b (unsloth dynamic UD-Q6_K_XL,
|
||||
# ~26GB). Both pre-pulled into /tank/aimodels/huggingface
|
||||
# via huggingface_hub.snapshot_download with hf_transfer.
|
||||
# Both repos include mmproj-BF16.gguf — llama-server's -hf
|
||||
# flag auto-loads it, so vision works without --mmproj.
|
||||
# ============================================================================
|
||||
|
||||
# Default 1200 seconds (20 min) to wait for model to be available to load.
|
||||
@@ -164,6 +170,50 @@ models:
|
||||
--reasoning on
|
||||
--reasoning-format deepseek
|
||||
|
||||
"qwen3.6-35-a3b-heretic":
|
||||
name: "Qwen 3.6 35B-A3B Uncensored Heretic Q6_K (vision)"
|
||||
description: "Qwen3.6 MoE, 3B active of 35B. llmfan46 uncensored 'heretic' Q6_K (~28GB) + auto-loaded BF16 mmproj for vision."
|
||||
ttl: 600
|
||||
cmd: |
|
||||
/app/llama-server
|
||||
--context-shift
|
||||
--jinja
|
||||
-hf llmfan46/Qwen3.6-35B-A3B-uncensored-heretic-GGUF:Q6_K
|
||||
--port ${PORT}
|
||||
--n-gpu-layers 999
|
||||
--ctx-size 131072
|
||||
--flash-attn on
|
||||
--temp 1.0
|
||||
--top-p 0.95
|
||||
--top-k 20
|
||||
--min-p 0.00
|
||||
--presence-penalty 1.5
|
||||
--repeat-penalty 1.0
|
||||
--reasoning on
|
||||
--reasoning-format deepseek
|
||||
|
||||
"qwen3.6-27b":
|
||||
name: "Qwen 3.6 27B UD-Q6_K_XL (vision)"
|
||||
description: "Qwen3.6 dense 27B. Unsloth dynamic Q6_K_XL (~26GB) + auto-loaded BF16 mmproj for vision."
|
||||
ttl: 600
|
||||
cmd: |
|
||||
/app/llama-server
|
||||
--context-shift
|
||||
--jinja
|
||||
-hf unsloth/Qwen3.6-27B-GGUF:UD-Q6_K_XL
|
||||
--port ${PORT}
|
||||
--n-gpu-layers 999
|
||||
--ctx-size 131072
|
||||
--flash-attn on
|
||||
--temp 1.0
|
||||
--top-p 0.95
|
||||
--top-k 20
|
||||
--min-p 0.00
|
||||
--presence-penalty 1.5
|
||||
--repeat-penalty 1.0
|
||||
--reasoning on
|
||||
--reasoning-format deepseek
|
||||
|
||||
# ==========================================================================
|
||||
# NEMOTRON MODELS (KB-recommended settings)
|
||||
# - General Chat: temp 1.0, top-p 1.0, min_p 0.01
|
||||
|
||||
Reference in New Issue
Block a user