VuongKB/reference/qwen3-coder-next-running-parameters.md

---
created: '2026-04-09T18:00:31+00:00'
id: 587014e6-a793-4f79-af13-e29a19e7d777
keywords:
- Qwen3-Coder-Next
- Alibaba
- coding model
- temperature
- top_p
- top_k
- min_p
- GGUF
- quantization
- UD-Q4_K_XL
- FP8
- vLLM
- llama.cpp
- llama-server
- MoE
- 80B
- 3B active
- non-reasoning
- agentic coding
- SWE-Bench
- tool calling
links: []
modified: '2026-04-09T18:00:31+00:00'
summary: Optimal running parameters for Qwen3-Coder-Next (80B MoE, 3B active), covering
  inference settings, hardware requirements, quantization benchmarks, FP8 vLLM deployment,
  tool calling setup, and llama.cpp example commands.
tags:
- reference
- qwen3-coder-next
- llm
- inference
- llama-cpp
- local-ai
- coding
title: Qwen3-Coder-Next — Optimal Running Parameters
---

# Qwen3-Coder-Next — Optimal Running Parameters

A reference for optimally running Qwen3-Coder-Next, Alibaba's 80B MoE (3B active parameters) coding model, locally.

## Model Overview

| Property | Value |
|---|---|
| Architecture | MoE — 80B total, **3B active parameters** |
| Max Context | **262,144** (256K) natively |
| Recommended Context | **32,768** for less memory use |
| Thinking Mode | **Non-reasoning only** — no thinking/reasoning mode. Does not generate `<think)>` blocks. |
| Best For | Fast agentic coding, long-horizon reasoning, complex tool use, recovery from execution failures |
| Performance Tier | Comparable to models with 10–20× more active parameters |

## Recommended Inference Parameters

These are Qwen's recommended settings for optimal performance:

| Parameter | Value | Notes |
|---|---|---|
| `temperature` | **1.0** | |
| `top_p` | **0.95** | |
| `top_k` | **40** | |
| `min_p` | **0.01** | llama.cpp default is 0.05 — override to 0.01 |
| `repetition_penalty` | **1.0** (disabled) | Only increase if you see looping |

### Quick Reference — One-Liner

```
temperature=1.0, top_p=0.95, top_k=40, min_p=0.01, repetition_penalty=1.0
```

## Hardware Requirements (Total Memory: RAM + VRAM, or Unified Memory)

| Precision | Memory Required |
|---|---|
| 3-bit | ~34 GB |
| **4-bit (UD-Q4_K_XL)** | **~46 GB** |
| 8-bit | ~85 GB |
| BF16 | ~160 GB |

> **Rule of thumb:** `disk space + RAM + VRAM ≥ size of quant`. If the quant fully fits on your device, expect **20+ tokens/s**. If it doesn't fit, it'll still work via offloading but will be slower.

### Recommended Minimum Quantization

- **3-bit (`UD-IQ3_XXS`)** — comes close to BF16 performance; sensible minimum for most use cases.
- **4-bit (`UD-Q4_K_XL`)** — recommended starting point; best balance of size and quality.
- At least **Q4_K_M** is advised based on third-party benchmarks (Unsloth's Q4_K_M outperforms standard Q4_K_M).

## Thinking Mode

This model is **non-reasoning only**. It does NOT support thinking/reasoning mode and does not generate `<think)>` blocks.

- `enable_thinking=False` is **not required** — it's the only mode.
- This design choice enables **ultra-quick code responses**.

## Critical Warnings

- ⚠️ **Update llama.cpp** — a previous bug in `vectorized key_gdiff` calculation caused looping and output issues. Re-download GGUFs and update llama.cpp.
- ⚠️ **No Ollama support** — use llama.cpp-compatible backends.
- ⚠️ If context length is set too low, you may see errors like `exceeds the available context size`. Increase `--ctx-size` or use `--fit on` for auto-detection.
- ⚠️ Tool-calling improved after llama.cpp parsing fixes (Feb 19 update). Ensure you're on a recent version.

## Example llama.cpp Commands

### Direct Pull from Hugging Face (4-bit)

```bash
./llama.cpp/llama-cli \
    -hf unsloth/Qwen3-Coder-Next-GGUF:UD-Q4_K_XL \
    --ctx-size 16384 \
    --temp 1.0 --top-p 0.95 --min-p 0.01 --top-k 40
```

### Conversation Mode with Local GGUF

```bash
./llama.cpp/llama-cli \
    --model unsloth/Qwen3-Coder-Next-GGUF/Qwen3-Coder-Next-UD-Q4_K_XL.gguf \
    --seed 3407 \
    --temp 1.0 \
    --top-p 0.95 \
    --min-p 0.01 \
    --top-k 40
```

### llama-server Deployment

```bash
./llama.cpp/llama-server \
    --model unsloth/Qwen3-Coder-Next-GGUF/Qwen3-Coder-Next-UD-Q4_K_XL.gguf \
    --alias "unsloth/Qwen3-Coder-Next" \
    --seed 3407 \
    --temp 1.0 --top-p 0.95 --min-p 0.01 --top-k 40 \
    --port 8001
```

### OpenAI-Compatible Client

After deploying via `llama-server`:

```python
from openai import OpenAI

openai_client = OpenAI(
    base_url = "http://127.0.0.1:8001/v1",
    api_key = "sk-no-key-required",
)

completion = openai_client.chat.completions.create(
    model = "unsloth/Qwen3-Coder-Next",
    messages = [{"role": "user", "content": "Create a Flappy Bird game in HTML"}],
)

print(completion.choices[0].message.content)
```

## FP8 Dynamic Quant (vLLM)

For premium and fast inference with GPU setups, use Unsloth's [FP8 Dynamic quant](https://huggingface.co/unsloth/Qwen3-Coder-Next-FP8-Dynamic). FP8 can boost throughput by **25% or more**.

### Install vLLM Nightly

```bash
curl -LsSf https://astral.sh/uv/install.sh | sh
uv venv unsloth_fp8 --python 3.12 --seed
source unsloth_fp8/bin/activate

uv pip install --upgrade --force-reinstall vllm --torch-backend=auto \
    --extra-index-url https://wheels.vllm.ai/nightly/cu130
uv pip install --upgrade --force-reinstall git+https://github.com/huggingface/transformers.git
uv pip install --force-reinstall numba
```

> Change `cu130` to your CUDA version (only `cu129` and `cu130` currently supported). Check via `nvidia-smi`.

### Serve FP8 with vLLM (Multi-GPU)

```bash
export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False
CUDA_VISIBLE_DEVICES='0,1,2,3' vllm serve unsloth/Qwen3-Coder-Next-FP8-Dynamic \
    --served-model-name unsloth/Qwen3-Coder-Next \
    --tensor-parallel-size 4 \
    --tool-call-parser qwen3_coder \
    --enable-auto-tool-choice \
    --dtype bfloat16 \
    --seed 3407 \
    --max-model-len 200000 \
    --gpu-memory-utilization 0.93 \
    --port 8001
```

> - For single GPU: `CUDA_VISIBLE_DEVICES='0'` and `--tensor-parallel-size 1` (or remove the flag).
> - Add `--kv-cache-dtype fp8` to reduce KV cache memory usage by **50%**.

## Quantization Benchmarks (Third-Party)

### Aider Polyglot Benchmarks

- **3-bit (`UD-IQ3_XXS`)** comes close to **BF16** performance — sensible minimum.
- Quality steadily improves: **1-bit → 2-bit → 3-bit → 6-bit**.
- **Unsloth's Q4_K_M** outperforms standard (non-Unsloth) Q4_K_M.
- **At least Q4_K_M is advised.**

### Benjamin Marie Benchmarks (750-prompt suite: LiveCodeBench v6, MMLU Pro, GPQA, Math500)

- Unsloth's Q4_K_M quants perform better than standard Q4_K_M.
- Q3_K_M performs worse on LiveCodeBench v6 but surprisingly better on HumanEval than standard Q4_K_M.

## Performance Comparison

Qwen3-Coder-Next (80B, 3B active) vs. much larger models:

| Benchmark | Qwen3-Coder-Next (80B) | DeepSeek-V3.2 (671B) | GLM-4.7 (358B) | MiniMax M2.1 (229B) |
|---|---|---|---|---|
| SWE-Bench Verified | 70.6 | 70.2 | 74.2 | 74.8 |
| SWE-Bench Multilingual | 62.8 | 62.3 | 63.7 | 66.2 |
| SWE-Bench Pro | **44.3** | 40.9 | 40.6 | 34.6 |
| Terminal-Bench 2.0 | 36.2 | 39.3 | 37.1 | 32.6 |
| Aider | 66.2 | 69.9 | 52.1 | 61.0 |

## Sources

- Unsloth Qwen3-Coder-Next Documentation (How to Run Locally)
- GGUF collection: <https://huggingface.co/unsloth/Qwen3-Coder-Next-GGUF>
- FP8 Dynamic: <https://huggingface.co/unsloth/Qwen3-Coder-Next-FP8-Dynamic>
- Third-party benchmarks: Aider Polyglot, Benjamin Marie