vh/VuongKB
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
d00dcfb1df30c53f1f87f8cb279c80f157bbe56e
VuongKB/reference/qwen3.5-running-parameters.md
T

10 KiB
Raw Blame History

created, id, keywords, links, modified, summary, tags, title
created id keywords links modified summary tags title
2026-04-09T17:58:20+00:00 c923cbfa-95ba-4ce2-b635-c2e8b15c69ca
Qwen3.5
Alibaba
temperature
top_p
top_k
min_p
presence_penalty
GGUF
quantization
UD-Q4_K_XL
thinking mode
hybrid reasoning
llama.cpp
llama-server
MoE
397B-A17B
35B-A3B
27B
122B-A10B
YaRN
Unsloth Dynamic 2.0
MXFP4
2026-04-09T17:58:20+00:00 Optimal running parameters for Alibaba's Qwen3.5 family of models, covering inference settings for thinking vs non-thinking modes across all task types, hardware requirements for all variants (0.8B through 397B-A17B), quantization recommendations, thinking mode control, and llama.cpp example commands.
reference
qwen3.5
llm
inference
llama-cpp
local-ai
Qwen3.5 — Optimal Running Parameters

Qwen3.5 — Optimal Running Parameters

A reference for optimally running Alibaba's Qwen3.5 family of models locally.

Model Variants

Variant Architecture Context Languages Best Fit
0.8B Dense 256K 201 Smallest edge inference
2B Dense 256K 201 Small device inference
4B Dense 256K 201 Lightweight local use
9B Dense 256K 201 Capable small model
27B Dense 256K 201 Slightly more accurate than 35B-A3B; fits 18GB devices
35B-A3B MoE (3B active) 256K 201 Best speed/quality tradeoff; fits 22GB devices
122B-A10B MoE (10B active) 256K 201 High quality; needs ~70GB (4-bit)
397B-A17B MoE (17B active) 256K (extendable to 1M via YaRN) 201 Gemini 3 Pro / Claude Opus 4.5 / GPT-5.2 tier

27B vs 35B-A3B Decision

  • 27B — Choose for slightly more accurate results when you can't fit a larger model.
  • 35B-A3B — Choose for much faster inference. MoE architecture with only 3B active parameters.

Hardware Requirements (Total Memory: RAM + VRAM, or Unified Memory)

Variant 3-bit 4-bit 6-bit 8-bit BF16
0.8B + 2B 3 GB 3.5 GB 5 GB 7.5 GB 9 GB
4B 4.5 GB 5.5 GB 7 GB 10 GB 14 GB
9B 5.5 GB 6.5 GB 9 GB 13 GB 19 GB
27B 14 GB 17 GB 24 GB 30 GB 54 GB
35B-A3B 17 GB 22 GB 30 GB 38 GB 70 GB
122B-A10B 60 GB 70 GB 106 GB 132 GB 245 GB
397B-A17B 180 GB 214 GB 340 GB 512 GB 810 GB

Rule of thumb: Total available memory should exceed the quantized model file size. If it doesn't, llama.cpp can use SSD/HDD offloading but inference will be slower.

Notable 397B Configurations

  • 3-bit: fits on 192GB RAM systems (e.g., 192GB Mac)
  • 4-bit (UD-Q4_K_XL): ~214GB on disk — loads directly on 256GB M3 Ultra
  • Single 24GB GPU + 256GB system RAM via MoE offloading → 25+ tokens/s
  • 8-bit: needs ~512GB RAM/VRAM

Recommended Inference Parameters

Qwen3.5 is a hybrid reasoning model, so parameters differ between thinking and non-thinking (instruct) modes, and further by task type.

Thinking Mode Parameters

Parameter General Tasks Precise Coding (e.g. WebDev)
temperature 1.0 0.6
top_p 0.95 0.95
top_k 20 20
min_p 0.0 0.0
presence_penalty 1.5 0.0
repetition_penalty 1.0 (disabled) 1.0 (disabled)

Non-Thinking (Instruct) Mode Parameters

Parameter General Tasks Reasoning Tasks
temperature 0.7 1.0
top_p 0.8 0.95
top_k 20 20
min_p 0.0 0.0
presence_penalty 1.5 1.5
repetition_penalty 1.0 (disabled) 1.0 (disabled)

Quick Reference — One-Liners

Thinking + General:

temperature=1.0, top_p=0.95, top_k=20, min_p=0.0, presence_penalty=1.5, repetition_penalty=1.0

Thinking + Precise Coding:

temperature=0.6, top_p=0.95, top_k=20, min_p=0.0, presence_penalty=0.0, repetition_penalty=1.0

Non-Thinking + General:

temperature=0.7, top_p=0.8, top_k=20, min_p=0.0, presence_penalty=1.5, repetition_penalty=1.0

Non-Thinking + Reasoning:

temperature=1.0, top_p=0.95, top_k=20, min_p=0.0, presence_penalty=1.5, repetition_penalty=1.0

Context & Output Settings

Setting Value
Max context window 262,144 (256K)
Context extension Up to 1M via YaRN
Recommended starting context 16,384 (16K) for responsiveness
Adequate output length 32,768 tokens
presence_penalty range 0.0–2.0 (default off; higher values may slightly decrease performance)

Gibberish fix: If you get gibberish output, your context length may be too low. Try --cache-type-k bf16 --cache-type-v bf16.

Quantization Recommendations

  • All GGUFs use Unsloth Dynamic 2.0 quantization — important layers upcasted to 8 or 16-bit even in 4-bit quants.
  • Recommended starting point: Dynamic 4-bit (UD-Q4_K_XL) for all variants.
  • Minimum recommended: Dynamic 2-bit (UD-Q2_K_XL) to balance size and accuracy.
  • MXFP4 layers retired from Q2_K_XL, Q3_K_XL, and Q4_K_XL quants (except pure MXFP4_MOE).

397B-A17B Quantization Benchmarks (Third-Party, Benjamin Marie)

Quant Accuracy Change vs Original Relative Error Increase
Original 81.3% — —
UD-Q4_K_XL 80.5% −0.8 points +4.3%
UD-Q3_K_XL 80.7% −0.6 points +3.5%

Q3 and Q4 are effectively similar quality. Pick Q3 for smallest footprint, Q4 for a slightly more conservative option.

Thinking Mode Control

Enabling Thinking

--chat-template-kwargs '{"enable_thinking":true}'

Windows PowerShell:

--chat-template-kwargs "{\"enable_thinking\":true}"

Disabling Thinking

--chat-template-kwargs '{"enable_thinking":false}'

Windows PowerShell:

--chat-template-kwargs "{\"enable_thinking\":false}"

Default Thinking Behavior by Variant

Variant Thinking Default
0.8B, 2B, 4B, 9B (Small) Disabled — must explicitly enable
27B, 35B-A3B, 122B-A10B, 397B-A17B Enabled — must explicitly disable if unwanted

Use llama-server for thinking control (more reliable than llama-cli).

Critical Warnings

  • ⚠️ No Qwen3.5 GGUF works in Ollama due to separate mmproj vision files. Use llama.cpp-compatible backends.
  • ⚠️ presence_penalty above 0.0 may cause a slight decrease in performance.
  • ⚠️ If getting gibberish, check context length or use --cache-type-k bf16 --cache-type-v bf16.
  • Redownload older GGUFs — all updated with improved quantization algorithm, new imatrix data, and tool-calling chat template fixes.

Example llama.cpp Commands

35B-A3B — Thinking Mode (General Tasks)

export LLAMA_CACHE="unsloth/Qwen3.5-35B-A3B-GGUF"
./llama.cpp/llama-cli \
    -hf unsloth/Qwen3.5-35B-A3B-GGUF:UD-Q4_K_XL \
    --ctx-size 16384 \
    --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.00

35B-A3B — Thinking Mode (Precise Coding)

export LLAMA_CACHE="unsloth/Qwen3.5-35B-A3B-GGUF"
./llama.cpp/llama-cli \
    -hf unsloth/Qwen3.5-35B-A3B-GGUF:UD-Q4_K_XL \
    --ctx-size 16384 \
    --temp 0.6 --top-p 0.95 --top-k 20 --min-p 0.00

35B-A3B — Non-Thinking Mode (General)

export LLAMA_CACHE="unsloth/Qwen3.5-35B-A3B-GGUF"
./llama.cpp/llama-server \
    -hf unsloth/Qwen3.5-35B-A3B-GGUF:UD-Q4_K_XL \
    --ctx-size 16384 \
    --temp 0.7 --top-p 0.8 --top-k 20 --min-p 0.00 \
    --chat-template-kwargs '{"enable_thinking":false}'

27B — Thinking Mode (Precise Coding)

export LLAMA_CACHE="unsloth/Qwen3.5-27B-GGUF"
./llama.cpp/llama-cli \
    -hf unsloth/Qwen3.5-27B-GGUF:UD-Q4_K_XL \
    --ctx-size 16384 \
    --temp 0.6 --top-p 0.95 --top-k 20 --min-p 0.00

9B (Small) — Enable Thinking (Disabled by Default)

export LLAMA_CACHE="unsloth/Qwen3.5-9B-GGUF"
./llama.cpp/llama-server \
    -hf unsloth/Qwen3.5-9B-GGUF:UD-Q4_K_XL \
    --ctx-size 16384 \
    --temp 0.6 --top-p 0.95 --top-k 20 --min-p 0.00 \
    --alias "unsloth/Qwen3.5-9B-GGUF" \
    --port 8001 \
    --chat-template-kwargs '{"enable_thinking":true}'

9B (Small) — Non-Thinking (Default)

export LLAMA_CACHE="unsloth/Qwen3.5-9B-GGUF"
./llama.cpp/llama-cli \
    -hf unsloth/Qwen3.5-9B-GGUF:UD-Q4_K_XL \
    --ctx-size 16384 \
    --temp 0.7 --top-p 0.8 --top-k 20 --min-p 0.00

397B-A17B — Thinking Mode

export LLAMA_CACHE="unsloth/Qwen3.5-397B-A17B-GGUF"
./llama.cpp/llama-cli \
    -hf unsloth/Qwen3.5-397B-A17B-GGUF:UD-Q4_K_XL \
    --ctx-size 16384 \
    --temp 0.6 --top-p 0.95 --top-k 20 --min-p 0.00

llama-server Deployment (with Vision)

./llama.cpp/llama-server \
    --model unsloth/Qwen3.5-35B-A3B-GGUF/Qwen3.5-35B-A3B-UD-Q4_K_XL.gguf \
    --mmproj unsloth/Qwen3.5-35B-A3B-GGUF/mmproj-F16.gguf \
    --alias "unsloth/Qwen3.5-35B-A3B" \
    --temp 0.6 --top-p 0.95 --ctx-size 16384 \
    --top-k 20 --min-p 0.00 \
    --port 8001

397B — Conversation Mode with Vision + Tuning Knobs

./llama.cpp/llama-cli \
    --model unsloth/Qwen3.5-397B-A17B-GGUF/UD-Q4_K_XL/Qwen3.5-397B-A17B-UD-Q4_K_XL-00001-of-00006.gguf \
    --mmproj unsloth/Qwen3.5-397B-A17B-GGUF/mmproj-F16.gguf \
    --ctx-size 16384 \
    --threads 32 \
    --n-gpu-layers 2 \
    --temp 0.6 --top-p 0.95 --top-k 20 --min-p 0.00

Adjust --threads for CPU threads, --n-gpu-layers for GPU offloading (remove if CPU-only).

OpenAI-Compatible Client Usage

After deploying via llama-server:

from openai import OpenAI

openai_client = OpenAI(
    base_url = "http://127.0.0.1:8001/v1",
    api_key = "sk-no-key-required",
)

completion = openai_client.chat.completions.create(
    model = "unsloth/Qwen3.5-35B-A3B",
    messages = [{"role": "user", "content": "Create a Snake game."}],
)

print(completion.choices[0].message.content)

Sources

Reference in New Issue View Git Blame Copy Permalink