329 lines
10 KiB
Markdown
329 lines
10 KiB
Markdown
---
|
||
created: '2026-04-09T17:58:20+00:00'
|
||
id: c923cbfa-95ba-4ce2-b635-c2e8b15c69ca
|
||
keywords:
|
||
- Qwen3.5
|
||
- Alibaba
|
||
- temperature
|
||
- top_p
|
||
- top_k
|
||
- min_p
|
||
- presence_penalty
|
||
- GGUF
|
||
- quantization
|
||
- UD-Q4_K_XL
|
||
- thinking mode
|
||
- hybrid reasoning
|
||
- llama.cpp
|
||
- llama-server
|
||
- MoE
|
||
- 397B-A17B
|
||
- 35B-A3B
|
||
- 27B
|
||
- 122B-A10B
|
||
- YaRN
|
||
- Unsloth Dynamic 2.0
|
||
- MXFP4
|
||
links: []
|
||
modified: '2026-04-09T17:58:20+00:00'
|
||
summary: Optimal running parameters for Alibaba's Qwen3.5 family of models, covering
|
||
inference settings for thinking vs non-thinking modes across all task types, hardware
|
||
requirements for all variants (0.8B through 397B-A17B), quantization recommendations,
|
||
thinking mode control, and llama.cpp example commands.
|
||
tags:
|
||
- reference
|
||
- qwen3.5
|
||
- llm
|
||
- inference
|
||
- llama-cpp
|
||
- local-ai
|
||
title: Qwen3.5 — Optimal Running Parameters
|
||
---
|
||
|
||
# Qwen3.5 — Optimal Running Parameters
|
||
|
||
A reference for optimally running Alibaba's Qwen3.5 family of models locally.
|
||
|
||
## Model Variants
|
||
|
||
| Variant | Architecture | Context | Languages | Best Fit |
|
||
|---|---|---|---|---|
|
||
| **0.8B** | Dense | 256K | 201 | Smallest edge inference |
|
||
| **2B** | Dense | 256K | 201 | Small device inference |
|
||
| **4B** | Dense | 256K | 201 | Lightweight local use |
|
||
| **9B** | Dense | 256K | 201 | Capable small model |
|
||
| **27B** | Dense | 256K | 201 | Slightly more accurate than 35B-A3B; fits 18GB devices |
|
||
| **35B-A3B** | MoE (3B active) | 256K | 201 | Best speed/quality tradeoff; fits 22GB devices |
|
||
| **122B-A10B** | MoE (10B active) | 256K | 201 | High quality; needs ~70GB (4-bit) |
|
||
| **397B-A17B** | MoE (17B active) | 256K (extendable to 1M via YaRN) | 201 | Gemini 3 Pro / Claude Opus 4.5 / GPT-5.2 tier |
|
||
|
||
### 27B vs 35B-A3B Decision
|
||
|
||
- **27B** — Choose for slightly more accurate results when you can't fit a larger model.
|
||
- **35B-A3B** — Choose for much faster inference. MoE architecture with only 3B active parameters.
|
||
|
||
## Hardware Requirements (Total Memory: RAM + VRAM, or Unified Memory)
|
||
|
||
| Variant | 3-bit | 4-bit | 6-bit | 8-bit | BF16 |
|
||
|---|---|---|---|---|---|
|
||
| **0.8B + 2B** | 3 GB | 3.5 GB | 5 GB | 7.5 GB | 9 GB |
|
||
| **4B** | 4.5 GB | 5.5 GB | 7 GB | 10 GB | 14 GB |
|
||
| **9B** | 5.5 GB | 6.5 GB | 9 GB | 13 GB | 19 GB |
|
||
| **27B** | 14 GB | 17 GB | 24 GB | 30 GB | 54 GB |
|
||
| **35B-A3B** | 17 GB | 22 GB | 30 GB | 38 GB | 70 GB |
|
||
| **122B-A10B** | 60 GB | 70 GB | 106 GB | 132 GB | 245 GB |
|
||
| **397B-A17B** | 180 GB | 214 GB | 340 GB | 512 GB | 810 GB |
|
||
|
||
> **Rule of thumb:** Total available memory should exceed the quantized model file size. If it doesn't, llama.cpp can use SSD/HDD offloading but inference will be slower.
|
||
|
||
### Notable 397B Configurations
|
||
|
||
- **3-bit**: fits on **192GB RAM** systems (e.g., 192GB Mac)
|
||
- **4-bit (UD-Q4_K_XL)**: ~214GB on disk — loads directly on **256GB M3 Ultra**
|
||
- **Single 24GB GPU + 256GB system RAM** via MoE offloading → **25+ tokens/s**
|
||
- **8-bit**: needs ~512GB RAM/VRAM
|
||
|
||
## Recommended Inference Parameters
|
||
|
||
Qwen3.5 is a hybrid reasoning model, so parameters differ between **thinking** and **non-thinking (instruct)** modes, and further by **task type**.
|
||
|
||
### Thinking Mode Parameters
|
||
|
||
| Parameter | General Tasks | Precise Coding (e.g. WebDev) |
|
||
|---|---|---|
|
||
| `temperature` | **1.0** | **0.6** |
|
||
| `top_p` | **0.95** | **0.95** |
|
||
| `top_k` | **20** | **20** |
|
||
| `min_p` | **0.0** | **0.0** |
|
||
| `presence_penalty` | **1.5** | **0.0** |
|
||
| `repetition_penalty` | **1.0** (disabled) | **1.0** (disabled) |
|
||
|
||
### Non-Thinking (Instruct) Mode Parameters
|
||
|
||
| Parameter | General Tasks | Reasoning Tasks |
|
||
|---|---|---|
|
||
| `temperature` | **0.7** | **1.0** |
|
||
| `top_p` | **0.8** | **0.95** |
|
||
| `top_k` | **20** | **20** |
|
||
| `min_p` | **0.0** | **0.0** |
|
||
| `presence_penalty` | **1.5** | **1.5** |
|
||
| `repetition_penalty` | **1.0** (disabled) | **1.0** (disabled) |
|
||
|
||
### Quick Reference — One-Liners
|
||
|
||
**Thinking + General:**
|
||
```
|
||
temperature=1.0, top_p=0.95, top_k=20, min_p=0.0, presence_penalty=1.5, repetition_penalty=1.0
|
||
```
|
||
|
||
**Thinking + Precise Coding:**
|
||
```
|
||
temperature=0.6, top_p=0.95, top_k=20, min_p=0.0, presence_penalty=0.0, repetition_penalty=1.0
|
||
```
|
||
|
||
**Non-Thinking + General:**
|
||
```
|
||
temperature=0.7, top_p=0.8, top_k=20, min_p=0.0, presence_penalty=1.5, repetition_penalty=1.0
|
||
```
|
||
|
||
**Non-Thinking + Reasoning:**
|
||
```
|
||
temperature=1.0, top_p=0.95, top_k=20, min_p=0.0, presence_penalty=1.5, repetition_penalty=1.0
|
||
```
|
||
|
||
## Context & Output Settings
|
||
|
||
| Setting | Value |
|
||
|---|---|
|
||
| Max context window | **262,144** (256K) |
|
||
| Context extension | Up to **1M** via YaRN |
|
||
| Recommended starting context | **16,384** (16K) for responsiveness |
|
||
| Adequate output length | **32,768** tokens |
|
||
| `presence_penalty` range | 0.0–2.0 (default off; higher values may slightly decrease performance) |
|
||
|
||
> **Gibberish fix:** If you get gibberish output, your context length may be too low. Try `--cache-type-k bf16 --cache-type-v bf16`.
|
||
|
||
## Quantization Recommendations
|
||
|
||
- All GGUFs use **Unsloth Dynamic 2.0** quantization — important layers upcasted to 8 or 16-bit even in 4-bit quants.
|
||
- Recommended starting point: **Dynamic 4-bit** (`UD-Q4_K_XL`) for all variants.
|
||
- Minimum recommended: **Dynamic 2-bit** (`UD-Q2_K_XL`) to balance size and accuracy.
|
||
- **MXFP4 layers retired** from Q2_K_XL, Q3_K_XL, and Q4_K_XL quants (except pure MXFP4_MOE).
|
||
|
||
### 397B-A17B Quantization Benchmarks (Third-Party, Benjamin Marie)
|
||
|
||
| Quant | Accuracy | Change vs Original | Relative Error Increase |
|
||
|---|---|---|---|
|
||
| Original | 81.3% | — | — |
|
||
| UD-Q4_K_XL | 80.5% | −0.8 points | +4.3% |
|
||
| UD-Q3_K_XL | 80.7% | −0.6 points | +3.5% |
|
||
|
||
> Q3 and Q4 are effectively similar quality. Pick **Q3 for smallest footprint**, **Q4 for a slightly more conservative option**.
|
||
|
||
## Thinking Mode Control
|
||
|
||
### Enabling Thinking
|
||
|
||
```bash
|
||
--chat-template-kwargs '{"enable_thinking":true}'
|
||
```
|
||
|
||
**Windows PowerShell:**
|
||
```bash
|
||
--chat-template-kwargs "{\"enable_thinking\":true}"
|
||
```
|
||
|
||
### Disabling Thinking
|
||
|
||
```bash
|
||
--chat-template-kwargs '{"enable_thinking":false}'
|
||
```
|
||
|
||
**Windows PowerShell:**
|
||
```bash
|
||
--chat-template-kwargs "{\"enable_thinking\":false}"
|
||
```
|
||
|
||
### Default Thinking Behavior by Variant
|
||
|
||
| Variant | Thinking Default |
|
||
|---|---|
|
||
| **0.8B, 2B, 4B, 9B** (Small) | **Disabled** — must explicitly enable |
|
||
| **27B, 35B-A3B, 122B-A10B, 397B-A17B** | **Enabled** — must explicitly disable if unwanted |
|
||
|
||
> Use `llama-server` for thinking control (more reliable than `llama-cli`).
|
||
|
||
## Critical Warnings
|
||
|
||
- ⚠️ **No Qwen3.5 GGUF works in Ollama** due to separate mmproj vision files. Use llama.cpp-compatible backends.
|
||
- ⚠️ `presence_penalty` above 0.0 may cause a **slight decrease in performance**.
|
||
- ⚠️ If getting gibberish, check context length or use `--cache-type-k bf16 --cache-type-v bf16`.
|
||
- Redownload older GGUFs — all updated with improved quantization algorithm, new imatrix data, and tool-calling chat template fixes.
|
||
|
||
## Example llama.cpp Commands
|
||
|
||
### 35B-A3B — Thinking Mode (General Tasks)
|
||
|
||
```bash
|
||
export LLAMA_CACHE="unsloth/Qwen3.5-35B-A3B-GGUF"
|
||
./llama.cpp/llama-cli \
|
||
-hf unsloth/Qwen3.5-35B-A3B-GGUF:UD-Q4_K_XL \
|
||
--ctx-size 16384 \
|
||
--temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.00
|
||
```
|
||
|
||
### 35B-A3B — Thinking Mode (Precise Coding)
|
||
|
||
```bash
|
||
export LLAMA_CACHE="unsloth/Qwen3.5-35B-A3B-GGUF"
|
||
./llama.cpp/llama-cli \
|
||
-hf unsloth/Qwen3.5-35B-A3B-GGUF:UD-Q4_K_XL \
|
||
--ctx-size 16384 \
|
||
--temp 0.6 --top-p 0.95 --top-k 20 --min-p 0.00
|
||
```
|
||
|
||
### 35B-A3B — Non-Thinking Mode (General)
|
||
|
||
```bash
|
||
export LLAMA_CACHE="unsloth/Qwen3.5-35B-A3B-GGUF"
|
||
./llama.cpp/llama-server \
|
||
-hf unsloth/Qwen3.5-35B-A3B-GGUF:UD-Q4_K_XL \
|
||
--ctx-size 16384 \
|
||
--temp 0.7 --top-p 0.8 --top-k 20 --min-p 0.00 \
|
||
--chat-template-kwargs '{"enable_thinking":false}'
|
||
```
|
||
|
||
### 27B — Thinking Mode (Precise Coding)
|
||
|
||
```bash
|
||
export LLAMA_CACHE="unsloth/Qwen3.5-27B-GGUF"
|
||
./llama.cpp/llama-cli \
|
||
-hf unsloth/Qwen3.5-27B-GGUF:UD-Q4_K_XL \
|
||
--ctx-size 16384 \
|
||
--temp 0.6 --top-p 0.95 --top-k 20 --min-p 0.00
|
||
```
|
||
|
||
### 9B (Small) — Enable Thinking (Disabled by Default)
|
||
|
||
```bash
|
||
export LLAMA_CACHE="unsloth/Qwen3.5-9B-GGUF"
|
||
./llama.cpp/llama-server \
|
||
-hf unsloth/Qwen3.5-9B-GGUF:UD-Q4_K_XL \
|
||
--ctx-size 16384 \
|
||
--temp 0.6 --top-p 0.95 --top-k 20 --min-p 0.00 \
|
||
--alias "unsloth/Qwen3.5-9B-GGUF" \
|
||
--port 8001 \
|
||
--chat-template-kwargs '{"enable_thinking":true}'
|
||
```
|
||
|
||
### 9B (Small) — Non-Thinking (Default)
|
||
|
||
```bash
|
||
export LLAMA_CACHE="unsloth/Qwen3.5-9B-GGUF"
|
||
./llama.cpp/llama-cli \
|
||
-hf unsloth/Qwen3.5-9B-GGUF:UD-Q4_K_XL \
|
||
--ctx-size 16384 \
|
||
--temp 0.7 --top-p 0.8 --top-k 20 --min-p 0.00
|
||
```
|
||
|
||
### 397B-A17B — Thinking Mode
|
||
|
||
```bash
|
||
export LLAMA_CACHE="unsloth/Qwen3.5-397B-A17B-GGUF"
|
||
./llama.cpp/llama-cli \
|
||
-hf unsloth/Qwen3.5-397B-A17B-GGUF:UD-Q4_K_XL \
|
||
--ctx-size 16384 \
|
||
--temp 0.6 --top-p 0.95 --top-k 20 --min-p 0.00
|
||
```
|
||
|
||
### llama-server Deployment (with Vision)
|
||
|
||
```bash
|
||
./llama.cpp/llama-server \
|
||
--model unsloth/Qwen3.5-35B-A3B-GGUF/Qwen3.5-35B-A3B-UD-Q4_K_XL.gguf \
|
||
--mmproj unsloth/Qwen3.5-35B-A3B-GGUF/mmproj-F16.gguf \
|
||
--alias "unsloth/Qwen3.5-35B-A3B" \
|
||
--temp 0.6 --top-p 0.95 --ctx-size 16384 \
|
||
--top-k 20 --min-p 0.00 \
|
||
--port 8001
|
||
```
|
||
|
||
### 397B — Conversation Mode with Vision + Tuning Knobs
|
||
|
||
```bash
|
||
./llama.cpp/llama-cli \
|
||
--model unsloth/Qwen3.5-397B-A17B-GGUF/UD-Q4_K_XL/Qwen3.5-397B-A17B-UD-Q4_K_XL-00001-of-00006.gguf \
|
||
--mmproj unsloth/Qwen3.5-397B-A17B-GGUF/mmproj-F16.gguf \
|
||
--ctx-size 16384 \
|
||
--threads 32 \
|
||
--n-gpu-layers 2 \
|
||
--temp 0.6 --top-p 0.95 --top-k 20 --min-p 0.00
|
||
```
|
||
|
||
> Adjust `--threads` for CPU threads, `--n-gpu-layers` for GPU offloading (remove if CPU-only).
|
||
|
||
## OpenAI-Compatible Client Usage
|
||
|
||
After deploying via `llama-server`:
|
||
|
||
```python
|
||
from openai import OpenAI
|
||
|
||
openai_client = OpenAI(
|
||
base_url = "http://127.0.0.1:8001/v1",
|
||
api_key = "sk-no-key-required",
|
||
)
|
||
|
||
completion = openai_client.chat.completions.create(
|
||
model = "unsloth/Qwen3.5-35B-A3B",
|
||
messages = [{"role": "user", "content": "Create a Snake game."}],
|
||
)
|
||
|
||
print(completion.choices[0].message.content)
|
||
```
|
||
|
||
## Sources
|
||
|
||
- Unsloth Qwen3.5 Documentation (How to Run Locally)
|
||
- Qwen3.5 GGUF collection: <https://huggingface.co/collections/unsloth/qwen35>
|
||
- Third-party 397B benchmarks: Benjamin Marie (750-prompt mixed suite: LiveCodeBench v6, MMLU Pro, GPQA, Math500) |