---
created: '2026-04-09T17:58:20+00:00'
id: c923cbfa-95ba-4ce2-b635-c2e8b15c69ca
keywords:
- Qwen3.5
- Alibaba
- temperature
- top_p
- top_k
- min_p
- presence_penalty
- GGUF
- quantization
- UD-Q4_K_XL
- thinking mode
- hybrid reasoning
- llama.cpp
- llama-server
- MoE
- 397B-A17B
- 35B-A3B
- 27B
- 122B-A10B
- YaRN
- Unsloth Dynamic 2.0
- MXFP4
links: []
modified: '2026-04-09T17:58:20+00:00'
summary: Optimal running parameters for Alibaba's Qwen3.5 family of models, covering
  inference settings for thinking vs non-thinking modes across all task types, hardware
  requirements for all variants (0.8B through 397B-A17B), quantization recommendations,
  thinking mode control, and llama.cpp example commands.
tags:
- reference
- qwen3.5
- llm
- inference
- llama-cpp
- local-ai
title: Qwen3.5 — Optimal Running Parameters
---

# Qwen3.5 — Optimal Running Parameters

A reference for optimally running Alibaba's Qwen3.5 family of models locally.

## Model Variants

| Variant | Architecture | Context | Languages | Best Fit |
|---|---|---|---|---|
| **0.8B** | Dense | 256K | 201 | Smallest edge inference |
| **2B** | Dense | 256K | 201 | Small device inference |
| **4B** | Dense | 256K | 201 | Lightweight local use |
| **9B** | Dense | 256K | 201 | Capable small model |
| **27B** | Dense | 256K | 201 | Slightly more accurate than 35B-A3B; fits 18GB devices |
| **35B-A3B** | MoE (3B active) | 256K | 201 | Best speed/quality tradeoff; fits 22GB devices |
| **122B-A10B** | MoE (10B active) | 256K | 201 | High quality; needs ~70GB (4-bit) |
| **397B-A17B** | MoE (17B active) | 256K (extendable to 1M via YaRN) | 201 | Gemini 3 Pro / Claude Opus 4.5 / GPT-5.2 tier |

### 27B vs 35B-A3B Decision

- **27B** — Choose for slightly more accurate results when you can't fit a larger model.
- **35B-A3B** — Choose for much faster inference. MoE architecture with only 3B active parameters.

## Hardware Requirements (Total Memory: RAM + VRAM, or Unified Memory)

| Variant | 3-bit | 4-bit | 6-bit | 8-bit | BF16 |
|---|---|---|---|---|---|
| **0.8B + 2B** | 3 GB | 3.5 GB | 5 GB | 7.5 GB | 9 GB |
| **4B** | 4.5 GB | 5.5 GB | 7 GB | 10 GB | 14 GB |
| **9B** | 5.5 GB | 6.5 GB | 9 GB | 13 GB | 19 GB |
| **27B** | 14 GB | 17 GB | 24 GB | 30 GB | 54 GB |
| **35B-A3B** | 17 GB | 22 GB | 30 GB | 38 GB | 70 GB |
| **122B-A10B** | 60 GB | 70 GB | 106 GB | 132 GB | 245 GB |
| **397B-A17B** | 180 GB | 214 GB | 340 GB | 512 GB | 810 GB |

> **Rule of thumb:** Total available memory should exceed the quantized model file size. If it doesn't, llama.cpp can use SSD/HDD offloading but inference will be slower.

### Notable 397B Configurations

- **3-bit**: fits on **192GB RAM** systems (e.g., 192GB Mac)
- **4-bit (UD-Q4_K_XL)**: ~214GB on disk — loads directly on **256GB M3 Ultra**
- **Single 24GB GPU + 256GB system RAM** via MoE offloading → **25+ tokens/s**
- **8-bit**: needs ~512GB RAM/VRAM

## Recommended Inference Parameters

Qwen3.5 is a hybrid reasoning model, so parameters differ between **thinking** and **non-thinking (instruct)** modes, and further by **task type**.

### Thinking Mode Parameters

| Parameter | General Tasks | Precise Coding (e.g. WebDev) |
|---|---|---|
| `temperature` | **1.0** | **0.6** |
| `top_p` | **0.95** | **0.95** |
| `top_k` | **20** | **20** |
| `min_p` | **0.0** | **0.0** |
| `presence_penalty` | **1.5** | **0.0** |
| `repetition_penalty` | **1.0** (disabled) | **1.0** (disabled) |

### Non-Thinking (Instruct) Mode Parameters

| Parameter | General Tasks | Reasoning Tasks |
|---|---|---|
| `temperature` | **0.7** | **1.0** |
| `top_p` | **0.8** | **0.95** |
| `top_k` | **20** | **20** |
| `min_p` | **0.0** | **0.0** |
| `presence_penalty` | **1.5** | **1.5** |
| `repetition_penalty` | **1.0** (disabled) | **1.0** (disabled) |

### Quick Reference — One-Liners

**Thinking + General:**
```
temperature=1.0, top_p=0.95, top_k=20, min_p=0.0, presence_penalty=1.5, repetition_penalty=1.0
```

**Thinking + Precise Coding:**
```
temperature=0.6, top_p=0.95, top_k=20, min_p=0.0, presence_penalty=0.0, repetition_penalty=1.0
```

**Non-Thinking + General:**
```
temperature=0.7, top_p=0.8, top_k=20, min_p=0.0, presence_penalty=1.5, repetition_penalty=1.0
```

**Non-Thinking + Reasoning:**
```
temperature=1.0, top_p=0.95, top_k=20, min_p=0.0, presence_penalty=1.5, repetition_penalty=1.0
```

## Context & Output Settings

| Setting | Value |
|---|---|
| Max context window | **262,144** (256K) |
| Context extension | Up to **1M** via YaRN |
| Recommended starting context | **16,384** (16K) for responsiveness |
| Adequate output length | **32,768** tokens |
| `presence_penalty` range | 0.0–2.0 (default off; higher values may slightly decrease performance) |

> **Gibberish fix:** If you get gibberish output, your context length may be too low. Try `--cache-type-k bf16 --cache-type-v bf16`.

## Quantization Recommendations

- All GGUFs use **Unsloth Dynamic 2.0** quantization — important layers upcasted to 8 or 16-bit even in 4-bit quants.
- Recommended starting point: **Dynamic 4-bit** (`UD-Q4_K_XL`) for all variants.
- Minimum recommended: **Dynamic 2-bit** (`UD-Q2_K_XL`) to balance size and accuracy.
- **MXFP4 layers retired** from Q2_K_XL, Q3_K_XL, and Q4_K_XL quants (except pure MXFP4_MOE).

### 397B-A17B Quantization Benchmarks (Third-Party, Benjamin Marie)

| Quant | Accuracy | Change vs Original | Relative Error Increase |
|---|---|---|---|
| Original | 81.3% | — | — |
| UD-Q4_K_XL | 80.5% | −0.8 points | +4.3% |
| UD-Q3_K_XL | 80.7% | −0.6 points | +3.5% |

> Q3 and Q4 are effectively similar quality. Pick **Q3 for smallest footprint**, **Q4 for a slightly more conservative option**.

## Thinking Mode Control

### Enabling Thinking

```bash
--chat-template-kwargs '{"enable_thinking":true}'
```

**Windows PowerShell:**
```bash
--chat-template-kwargs "{\"enable_thinking\":true}"
```

### Disabling Thinking

```bash
--chat-template-kwargs '{"enable_thinking":false}'
```

**Windows PowerShell:**
```bash
--chat-template-kwargs "{\"enable_thinking\":false}"
```

### Default Thinking Behavior by Variant

| Variant | Thinking Default |
|---|---|
| **0.8B, 2B, 4B, 9B** (Small) | **Disabled** — must explicitly enable |
| **27B, 35B-A3B, 122B-A10B, 397B-A17B** | **Enabled** — must explicitly disable if unwanted |

> Use `llama-server` for thinking control (more reliable than `llama-cli`).

## Critical Warnings

- ⚠️ **No Qwen3.5 GGUF works in Ollama** due to separate mmproj vision files. Use llama.cpp-compatible backends.
- ⚠️ `presence_penalty` above 0.0 may cause a **slight decrease in performance**.
- ⚠️ If getting gibberish, check context length or use `--cache-type-k bf16 --cache-type-v bf16`.
- Redownload older GGUFs — all updated with improved quantization algorithm, new imatrix data, and tool-calling chat template fixes.

## Example llama.cpp Commands

### 35B-A3B — Thinking Mode (General Tasks)

```bash
export LLAMA_CACHE="unsloth/Qwen3.5-35B-A3B-GGUF"
./llama.cpp/llama-cli \
    -hf unsloth/Qwen3.5-35B-A3B-GGUF:UD-Q4_K_XL \
    --ctx-size 16384 \
    --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.00
```

### 35B-A3B — Thinking Mode (Precise Coding)

```bash
export LLAMA_CACHE="unsloth/Qwen3.5-35B-A3B-GGUF"
./llama.cpp/llama-cli \
    -hf unsloth/Qwen3.5-35B-A3B-GGUF:UD-Q4_K_XL \
    --ctx-size 16384 \
    --temp 0.6 --top-p 0.95 --top-k 20 --min-p 0.00
```

### 35B-A3B — Non-Thinking Mode (General)

```bash
export LLAMA_CACHE="unsloth/Qwen3.5-35B-A3B-GGUF"
./llama.cpp/llama-server \
    -hf unsloth/Qwen3.5-35B-A3B-GGUF:UD-Q4_K_XL \
    --ctx-size 16384 \
    --temp 0.7 --top-p 0.8 --top-k 20 --min-p 0.00 \
    --chat-template-kwargs '{"enable_thinking":false}'
```

### 27B — Thinking Mode (Precise Coding)

```bash
export LLAMA_CACHE="unsloth/Qwen3.5-27B-GGUF"
./llama.cpp/llama-cli \
    -hf unsloth/Qwen3.5-27B-GGUF:UD-Q4_K_XL \
    --ctx-size 16384 \
    --temp 0.6 --top-p 0.95 --top-k 20 --min-p 0.00
```

### 9B (Small) — Enable Thinking (Disabled by Default)

```bash
export LLAMA_CACHE="unsloth/Qwen3.5-9B-GGUF"
./llama.cpp/llama-server \
    -hf unsloth/Qwen3.5-9B-GGUF:UD-Q4_K_XL \
    --ctx-size 16384 \
    --temp 0.6 --top-p 0.95 --top-k 20 --min-p 0.00 \
    --alias "unsloth/Qwen3.5-9B-GGUF" \
    --port 8001 \
    --chat-template-kwargs '{"enable_thinking":true}'
```

### 9B (Small) — Non-Thinking (Default)

```bash
export LLAMA_CACHE="unsloth/Qwen3.5-9B-GGUF"
./llama.cpp/llama-cli \
    -hf unsloth/Qwen3.5-9B-GGUF:UD-Q4_K_XL \
    --ctx-size 16384 \
    --temp 0.7 --top-p 0.8 --top-k 20 --min-p 0.00
```

### 397B-A17B — Thinking Mode

```bash
export LLAMA_CACHE="unsloth/Qwen3.5-397B-A17B-GGUF"
./llama.cpp/llama-cli \
    -hf unsloth/Qwen3.5-397B-A17B-GGUF:UD-Q4_K_XL \
    --ctx-size 16384 \
    --temp 0.6 --top-p 0.95 --top-k 20 --min-p 0.00
```

### llama-server Deployment (with Vision)

```bash
./llama.cpp/llama-server \
    --model unsloth/Qwen3.5-35B-A3B-GGUF/Qwen3.5-35B-A3B-UD-Q4_K_XL.gguf \
    --mmproj unsloth/Qwen3.5-35B-A3B-GGUF/mmproj-F16.gguf \
    --alias "unsloth/Qwen3.5-35B-A3B" \
    --temp 0.6 --top-p 0.95 --ctx-size 16384 \
    --top-k 20 --min-p 0.00 \
    --port 8001
```

### 397B — Conversation Mode with Vision + Tuning Knobs

```bash
./llama.cpp/llama-cli \
    --model unsloth/Qwen3.5-397B-A17B-GGUF/UD-Q4_K_XL/Qwen3.5-397B-A17B-UD-Q4_K_XL-00001-of-00006.gguf \
    --mmproj unsloth/Qwen3.5-397B-A17B-GGUF/mmproj-F16.gguf \
    --ctx-size 16384 \
    --threads 32 \
    --n-gpu-layers 2 \
    --temp 0.6 --top-p 0.95 --top-k 20 --min-p 0.00
```

> Adjust `--threads` for CPU threads, `--n-gpu-layers` for GPU offloading (remove if CPU-only).

## OpenAI-Compatible Client Usage

After deploying via `llama-server`:

```python
from openai import OpenAI

openai_client = OpenAI(
    base_url = "http://127.0.0.1:8001/v1",
    api_key = "sk-no-key-required",
)

completion = openai_client.chat.completions.create(
    model = "unsloth/Qwen3.5-35B-A3B",
    messages = [{"role": "user", "content": "Create a Snake game."}],
)

print(completion.choices[0].message.content)
```

## Sources

- Unsloth Qwen3.5 Documentation (How to Run Locally)
- Qwen3.5 GGUF collection: <https://huggingface.co/collections/unsloth/qwen35>
- Third-party 397B benchmarks: Benjamin Marie (750-prompt mixed suite: LiveCodeBench v6, MMLU Pro, GPQA, Math500)