VuongKB/reference/qwen3-coder-next-running-parameters.md

created, id, keywords, links, modified, summary, tags, title

created	id	keywords	links	modified	summary	tags	title
2026-04-09T18:00:31+00:00	587014e6-a793-4f79-af13-e29a19e7d777	Qwen3-Coder-Next Alibaba coding model temperature top_p top_k min_p GGUF quantization UD-Q4_K_XL FP8 vLLM llama.cpp llama-server MoE 80B 3B active non-reasoning agentic coding SWE-Bench tool calling		2026-04-09T18:00:31+00:00	Optimal running parameters for Qwen3-Coder-Next (80B MoE, 3B active), covering inference settings, hardware requirements, quantization benchmarks, FP8 vLLM deployment, tool calling setup, and llama.cpp example commands.	reference qwen3-coder-next llm inference llama-cpp local-ai coding	Qwen3-Coder-Next — Optimal Running Parameters

Qwen3-Coder-Next — Optimal Running Parameters

A reference for optimally running Qwen3-Coder-Next, Alibaba's 80B MoE (3B active parameters) coding model, locally.

Model Overview

Property	Value
Architecture	MoE — 80B total, 3B active parameters
Max Context	262,144 (256K) natively
Recommended Context	32,768 for less memory use
Thinking Mode	Non-reasoning only — no thinking/reasoning mode. Does not generate <think)> blocks.
Best For	Fast agentic coding, long-horizon reasoning, complex tool use, recovery from execution failures
Performance Tier	Comparable to models with 10–20× more active parameters

Recommended Inference Parameters

These are Qwen's recommended settings for optimal performance:

Parameter	Value	Notes
temperature	1.0
top_p	0.95
top_k	40
min_p	0.01	llama.cpp default is 0.05 — override to 0.01
repetition_penalty	1.0 (disabled)	Only increase if you see looping

Quick Reference — One-Liner

temperature=1.0, top_p=0.95, top_k=40, min_p=0.01, repetition_penalty=1.0

Hardware Requirements (Total Memory: RAM + VRAM, or Unified Memory)

Precision	Memory Required
3-bit	~34 GB
4-bit (UD-Q4_K_XL)	~46 GB
8-bit	~85 GB
BF16	~160 GB

Rule of thumb: disk space + RAM + VRAM ≥ size of quant. If the quant fully fits on your device, expect 20+ tokens/s. If it doesn't fit, it'll still work via offloading but will be slower.

Recommended Minimum Quantization

• 3-bit (UD-IQ3_XXS) — comes close to BF16 performance; sensible minimum for most use cases.
• 4-bit (UD-Q4_K_XL) — recommended starting point; best balance of size and quality.
• At least Q4_K_M is advised based on third-party benchmarks (Unsloth's Q4_K_M outperforms standard Q4_K_M).

Thinking Mode

This model is non-reasoning only. It does NOT support thinking/reasoning mode and does not generate <think)> blocks.

• enable_thinking=False is not required — it's the only mode.
• This design choice enables ultra-quick code responses.

Critical Warnings

• ⚠️ Update llama.cpp — a previous bug in vectorized key_gdiff calculation caused looping and output issues. Re-download GGUFs and update llama.cpp.
• ⚠️ No Ollama support — use llama.cpp-compatible backends.
• ⚠️ If context length is set too low, you may see errors like exceeds the available context size. Increase --ctx-size or use --fit on for auto-detection.
• ⚠️ Tool-calling improved after llama.cpp parsing fixes (Feb 19 update). Ensure you're on a recent version.

Example llama.cpp Commands

Direct Pull from Hugging Face (4-bit)

./llama.cpp/llama-cli \
    -hf unsloth/Qwen3-Coder-Next-GGUF:UD-Q4_K_XL \
    --ctx-size 16384 \
    --temp 1.0 --top-p 0.95 --min-p 0.01 --top-k 40

Conversation Mode with Local GGUF

./llama.cpp/llama-cli \
    --model unsloth/Qwen3-Coder-Next-GGUF/Qwen3-Coder-Next-UD-Q4_K_XL.gguf \
    --seed 3407 \
    --temp 1.0 \
    --top-p 0.95 \
    --min-p 0.01 \
    --top-k 40

llama-server Deployment

./llama.cpp/llama-server \
    --model unsloth/Qwen3-Coder-Next-GGUF/Qwen3-Coder-Next-UD-Q4_K_XL.gguf \
    --alias "unsloth/Qwen3-Coder-Next" \
    --seed 3407 \
    --temp 1.0 --top-p 0.95 --min-p 0.01 --top-k 40 \
    --port 8001

OpenAI-Compatible Client

After deploying via llama-server:

from openai import OpenAI

openai_client = OpenAI(
    base_url = "http://127.0.0.1:8001/v1",
    api_key = "sk-no-key-required",
)

completion = openai_client.chat.completions.create(
    model = "unsloth/Qwen3-Coder-Next",
    messages = [{"role": "user", "content": "Create a Flappy Bird game in HTML"}],
)

print(completion.choices[0].message.content)

FP8 Dynamic Quant (vLLM)

For premium and fast inference with GPU setups, use Unsloth's FP8 Dynamic quant. FP8 can boost throughput by 25% or more.

Install vLLM Nightly

curl -LsSf https://astral.sh/uv/install.sh | sh
uv venv unsloth_fp8 --python 3.12 --seed
source unsloth_fp8/bin/activate

uv pip install --upgrade --force-reinstall vllm --torch-backend=auto \
    --extra-index-url https://wheels.vllm.ai/nightly/cu130
uv pip install --upgrade --force-reinstall git+https://github.com/huggingface/transformers.git
uv pip install --force-reinstall numba

Change cu130 to your CUDA version (only cu129 and cu130 currently supported). Check via nvidia-smi.

Serve FP8 with vLLM (Multi-GPU)

export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False
CUDA_VISIBLE_DEVICES='0,1,2,3' vllm serve unsloth/Qwen3-Coder-Next-FP8-Dynamic \
    --served-model-name unsloth/Qwen3-Coder-Next \
    --tensor-parallel-size 4 \
    --tool-call-parser qwen3_coder \
    --enable-auto-tool-choice \
    --dtype bfloat16 \
    --seed 3407 \
    --max-model-len 200000 \
    --gpu-memory-utilization 0.93 \
    --port 8001

• For single GPU: CUDA_VISIBLE_DEVICES='0' and --tensor-parallel-size 1 (or remove the flag).
• Add --kv-cache-dtype fp8 to reduce KV cache memory usage by 50%.

Quantization Benchmarks (Third-Party)

Aider Polyglot Benchmarks

• 3-bit (UD-IQ3_XXS) comes close to BF16 performance — sensible minimum.
• Quality steadily improves: 1-bit → 2-bit → 3-bit → 6-bit.
• Unsloth's Q4_K_M outperforms standard (non-Unsloth) Q4_K_M.
• At least Q4_K_M is advised.

Benjamin Marie Benchmarks (750-prompt suite: LiveCodeBench v6, MMLU Pro, GPQA, Math500)

• Unsloth's Q4_K_M quants perform better than standard Q4_K_M.
• Q3_K_M performs worse on LiveCodeBench v6 but surprisingly better on HumanEval than standard Q4_K_M.

Performance Comparison

Qwen3-Coder-Next (80B, 3B active) vs. much larger models:

Benchmark	Qwen3-Coder-Next (80B)	DeepSeek-V3.2 (671B)	GLM-4.7 (358B)	MiniMax M2.1 (229B)
SWE-Bench Verified	70.6	70.2	74.2	74.8
SWE-Bench Multilingual	62.8	62.3	63.7	66.2
SWE-Bench Pro	44.3	40.9	40.6	34.6
Terminal-Bench 2.0	36.2	39.3	37.1	32.6
Aider	66.2	69.9	52.1	61.0

Sources

• Unsloth Qwen3-Coder-Next Documentation (How to Run Locally)
• GGUF collection: https://huggingface.co/unsloth/Qwen3-Coder-Next-GGUF
• FP8 Dynamic: https://huggingface.co/unsloth/Qwen3-Coder-Next-FP8-Dynamic
• Third-party benchmarks: Aider Polyglot, Benjamin Marie