vh/VuongKB
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Create: reference/qwen3.5-running-parameters.md

Browse Source
This commit is contained in:
Atlas
2026-04-09 10:58:20 -07:00
parent e2169c2776
commit c25c269b70
1 changed files with 329 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
reference/qwen3.5-running-parameters.md
+329
View File
@@ -0,0 +1,329 @@
---
created: '2026-04-09T17:58:20+00:00'
id: c923cbfa-95ba-4ce2-b635-c2e8b15c69ca
keywords:
- Qwen3.5
- Alibaba
- temperature
- top_p
- top_k
- min_p
- presence_penalty
- GGUF
- quantization
- UD-Q4_K_XL
- thinking mode
- hybrid reasoning
- llama.cpp
- llama-server
- MoE
- 397B-A17B
- 35B-A3B
- 27B
- 122B-A10B
- YaRN
- Unsloth Dynamic 2.0
- MXFP4
links: []
modified: '2026-04-09T17:58:20+00:00'
summary: Optimal running parameters for Alibaba's Qwen3.5 family of models, covering
inference settings for thinking vs non-thinking modes across all task types, hardware
requirements for all variants (0.8B through 397B-A17B), quantization recommendations,
thinking mode control, and llama.cpp example commands.
tags:
- reference
- qwen3.5
- llm
- inference
- llama-cpp
- local-ai
title: Qwen3.5 — Optimal Running Parameters
---
# Qwen3.5 — Optimal Running Parameters
A reference for optimally running Alibaba's Qwen3.5 family of models locally.
## Model Variants
| Variant | Architecture | Context | Languages | Best Fit |
|---|---|---|---|---|
| **0.8B** | Dense | 256K | 201 | Smallest edge inference |
| **2B** | Dense | 256K | 201 | Small device inference |
| **4B** | Dense | 256K | 201 | Lightweight local use |
| **9B** | Dense | 256K | 201 | Capable small model |
| **27B** | Dense | 256K | 201 | Slightly more accurate than 35B-A3B; fits 18GB devices |
| **35B-A3B** | MoE (3B active) | 256K | 201 | Best speed/quality tradeoff; fits 22GB devices |
| **122B-A10B** | MoE (10B active) | 256K | 201 | High quality; needs ~70GB (4-bit) |
| **397B-A17B** | MoE (17B active) | 256K (extendable to 1M via YaRN) | 201 | Gemini 3 Pro / Claude Opus 4.5 / GPT-5.2 tier |
### 27B vs 35B-A3B Decision
- **27B** — Choose for slightly more accurate results when you can't fit a larger model.
- **35B-A3B** — Choose for much faster inference. MoE architecture with only 3B active parameters.
## Hardware Requirements (Total Memory: RAM + VRAM, or Unified Memory)
| Variant | 3-bit | 4-bit | 6-bit | 8-bit | BF16 |
|---|---|---|---|---|---|
| **0.8B + 2B** | 3 GB | 3.5 GB | 5 GB | 7.5 GB | 9 GB |
| **4B** | 4.5 GB | 5.5 GB | 7 GB | 10 GB | 14 GB |
| **9B** | 5.5 GB | 6.5 GB | 9 GB | 13 GB | 19 GB |
| **27B** | 14 GB | 17 GB | 24 GB | 30 GB | 54 GB |
| **35B-A3B** | 17 GB | 22 GB | 30 GB | 38 GB | 70 GB |
| **122B-A10B** | 60 GB | 70 GB | 106 GB | 132 GB | 245 GB |
| **397B-A17B** | 180 GB | 214 GB | 340 GB | 512 GB | 810 GB |
> **Rule of thumb:** Total available memory should exceed the quantized model file size. If it doesn't, llama.cpp can use SSD/HDD offloading but inference will be slower.
### Notable 397B Configurations
- **3-bit**: fits on **192GB RAM** systems (e.g., 192GB Mac)
- **4-bit (UD-Q4_K_XL)**: ~214GB on disk — loads directly on **256GB M3 Ultra**
- **Single 24GB GPU + 256GB system RAM** via MoE offloading → **25+ tokens/s**
- **8-bit**: needs ~512GB RAM/VRAM
## Recommended Inference Parameters
Qwen3.5 is a hybrid reasoning model, so parameters differ between **thinking** and **non-thinking (instruct)** modes, and further by **task type**.
### Thinking Mode Parameters
| Parameter | General Tasks | Precise Coding (e.g. WebDev) |
|---|---|---|
| `temperature` | **1.0** | **0.6** |
| `top_p` | **0.95** | **0.95** |
| `top_k` | **20** | **20** |
| `min_p` | **0.0** | **0.0** |
| `presence_penalty` | **1.5** | **0.0** |
| `repetition_penalty` | **1.0** (disabled) | **1.0** (disabled) |
### Non-Thinking (Instruct) Mode Parameters
| Parameter | General Tasks | Reasoning Tasks |
|---|---|---|
| `temperature` | **0.7** | **1.0** |
| `top_p` | **0.8** | **0.95** |
| `top_k` | **20** | **20** |
| `min_p` | **0.0** | **0.0** |
| `presence_penalty` | **1.5** | **1.5** |
| `repetition_penalty` | **1.0** (disabled) | **1.0** (disabled) |
### Quick Reference — One-Liners
**Thinking + General:**
```
temperature=1.0, top_p=0.95, top_k=20, min_p=0.0, presence_penalty=1.5, repetition_penalty=1.0
```
**Thinking + Precise Coding:**
```
temperature=0.6, top_p=0.95, top_k=20, min_p=0.0, presence_penalty=0.0, repetition_penalty=1.0
```
**Non-Thinking + General:**
```
temperature=0.7, top_p=0.8, top_k=20, min_p=0.0, presence_penalty=1.5, repetition_penalty=1.0
```
**Non-Thinking + Reasoning:**
```
temperature=1.0, top_p=0.95, top_k=20, min_p=0.0, presence_penalty=1.5, repetition_penalty=1.0
```
## Context & Output Settings
| Setting | Value |
|---|---|
| Max context window | **262,144** (256K) |
| Context extension | Up to **1M** via YaRN |
| Recommended starting context | **16,384** (16K) for responsiveness |
| Adequate output length | **32,768** tokens |
| `presence_penalty` range | 0.0–2.0 (default off; higher values may slightly decrease performance) |
> **Gibberish fix:** If you get gibberish output, your context length may be too low. Try `--cache-type-k bf16 --cache-type-v bf16`.
## Quantization Recommendations
- All GGUFs use **Unsloth Dynamic 2.0** quantization — important layers upcasted to 8 or 16-bit even in 4-bit quants.
- Recommended starting point: **Dynamic 4-bit** (`UD-Q4_K_XL`) for all variants.
- Minimum recommended: **Dynamic 2-bit** (`UD-Q2_K_XL`) to balance size and accuracy.
- **MXFP4 layers retired** from Q2_K_XL, Q3_K_XL, and Q4_K_XL quants (except pure MXFP4_MOE).
### 397B-A17B Quantization Benchmarks (Third-Party, Benjamin Marie)
| Quant | Accuracy | Change vs Original | Relative Error Increase |
|---|---|---|---|
| Original | 81.3% | — | — |
| UD-Q4_K_XL | 80.5% | −0.8 points | +4.3% |
| UD-Q3_K_XL | 80.7% | −0.6 points | +3.5% |
> Q3 and Q4 are effectively similar quality. Pick **Q3 for smallest footprint**, **Q4 for a slightly more conservative option**.
## Thinking Mode Control
### Enabling Thinking
```bash
--chat-template-kwargs '{"enable_thinking":true}'
```
**Windows PowerShell:**
```bash
--chat-template-kwargs "{\"enable_thinking\":true}"
```
### Disabling Thinking
```bash
--chat-template-kwargs '{"enable_thinking":false}'
```
**Windows PowerShell:**
```bash
--chat-template-kwargs "{\"enable_thinking\":false}"
```
### Default Thinking Behavior by Variant
| Variant | Thinking Default |
|---|---|
| **0.8B, 2B, 4B, 9B** (Small) | **Disabled** — must explicitly enable |
| **27B, 35B-A3B, 122B-A10B, 397B-A17B** | **Enabled** — must explicitly disable if unwanted |
> Use `llama-server` for thinking control (more reliable than `llama-cli`).
## Critical Warnings
- ⚠️ **No Qwen3.5 GGUF works in Ollama** due to separate mmproj vision files. Use llama.cpp-compatible backends.
- ⚠️ `presence_penalty` above 0.0 may cause a **slight decrease in performance**.
- ⚠️ If getting gibberish, check context length or use `--cache-type-k bf16 --cache-type-v bf16`.
- Redownload older GGUFs — all updated with improved quantization algorithm, new imatrix data, and tool-calling chat template fixes.
## Example llama.cpp Commands
### 35B-A3B — Thinking Mode (General Tasks)
```bash
export LLAMA_CACHE="unsloth/Qwen3.5-35B-A3B-GGUF"
./llama.cpp/llama-cli \
-hf unsloth/Qwen3.5-35B-A3B-GGUF:UD-Q4_K_XL \
--ctx-size 16384 \
--temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.00
```
### 35B-A3B — Thinking Mode (Precise Coding)
```bash
export LLAMA_CACHE="unsloth/Qwen3.5-35B-A3B-GGUF"
./llama.cpp/llama-cli \
-hf unsloth/Qwen3.5-35B-A3B-GGUF:UD-Q4_K_XL \
--ctx-size 16384 \
--temp 0.6 --top-p 0.95 --top-k 20 --min-p 0.00
```
### 35B-A3B — Non-Thinking Mode (General)
```bash
export LLAMA_CACHE="unsloth/Qwen3.5-35B-A3B-GGUF"
./llama.cpp/llama-server \
-hf unsloth/Qwen3.5-35B-A3B-GGUF:UD-Q4_K_XL \
--ctx-size 16384 \
--temp 0.7 --top-p 0.8 --top-k 20 --min-p 0.00 \
--chat-template-kwargs '{"enable_thinking":false}'
```
### 27B — Thinking Mode (Precise Coding)
```bash
export LLAMA_CACHE="unsloth/Qwen3.5-27B-GGUF"
./llama.cpp/llama-cli \
-hf unsloth/Qwen3.5-27B-GGUF:UD-Q4_K_XL \
--ctx-size 16384 \
--temp 0.6 --top-p 0.95 --top-k 20 --min-p 0.00
```
### 9B (Small) — Enable Thinking (Disabled by Default)
```bash
export LLAMA_CACHE="unsloth/Qwen3.5-9B-GGUF"
./llama.cpp/llama-server \
-hf unsloth/Qwen3.5-9B-GGUF:UD-Q4_K_XL \
--ctx-size 16384 \
--temp 0.6 --top-p 0.95 --top-k 20 --min-p 0.00 \
--alias "unsloth/Qwen3.5-9B-GGUF" \
--port 8001 \
--chat-template-kwargs '{"enable_thinking":true}'
```
### 9B (Small) — Non-Thinking (Default)
```bash
export LLAMA_CACHE="unsloth/Qwen3.5-9B-GGUF"
./llama.cpp/llama-cli \
-hf unsloth/Qwen3.5-9B-GGUF:UD-Q4_K_XL \
--ctx-size 16384 \
--temp 0.7 --top-p 0.8 --top-k 20 --min-p 0.00
```
### 397B-A17B — Thinking Mode
```bash
export LLAMA_CACHE="unsloth/Qwen3.5-397B-A17B-GGUF"
./llama.cpp/llama-cli \
-hf unsloth/Qwen3.5-397B-A17B-GGUF:UD-Q4_K_XL \
--ctx-size 16384 \
--temp 0.6 --top-p 0.95 --top-k 20 --min-p 0.00
```
### llama-server Deployment (with Vision)
```bash
./llama.cpp/llama-server \
--model unsloth/Qwen3.5-35B-A3B-GGUF/Qwen3.5-35B-A3B-UD-Q4_K_XL.gguf \
--mmproj unsloth/Qwen3.5-35B-A3B-GGUF/mmproj-F16.gguf \
--alias "unsloth/Qwen3.5-35B-A3B" \
--temp 0.6 --top-p 0.95 --ctx-size 16384 \
--top-k 20 --min-p 0.00 \
--port 8001
```
### 397B — Conversation Mode with Vision + Tuning Knobs
```bash
./llama.cpp/llama-cli \
--model unsloth/Qwen3.5-397B-A17B-GGUF/UD-Q4_K_XL/Qwen3.5-397B-A17B-UD-Q4_K_XL-00001-of-00006.gguf \
--mmproj unsloth/Qwen3.5-397B-A17B-GGUF/mmproj-F16.gguf \
--ctx-size 16384 \
--threads 32 \
--n-gpu-layers 2 \
--temp 0.6 --top-p 0.95 --top-k 20 --min-p 0.00
```
> Adjust `--threads` for CPU threads, `--n-gpu-layers` for GPU offloading (remove if CPU-only).
## OpenAI-Compatible Client Usage
After deploying via `llama-server`:
```python
from openai import OpenAI
openai_client = OpenAI(
base_url = "http://127.0.0.1:8001/v1",
api_key = "sk-no-key-required",
)
completion = openai_client.chat.completions.create(
model = "unsloth/Qwen3.5-35B-A3B",
messages = [{"role": "user", "content": "Create a Snake game."}],
)
print(completion.choices[0].message.content)
```
## Sources
- Unsloth Qwen3.5 Documentation (How to Run Locally)
- Qwen3.5 GGUF collection: <https://huggingface.co/collections/unsloth/qwen35>
- Third-party 397B benchmarks: Benjamin Marie (750-prompt mixed suite: LiveCodeBench v6, MMLU Pro, GPQA, Math500)