Create: reference/qwen3-coder-next-running-parameters.md

2026-04-09 11:00:31 -07:00
parent c25c269b70
commit e21342084f
1 changed files with 229 additions and 0 deletions
@@ -0,0 +1,229 @@
 ---
 created: '2026-04-09T18:00:31+00:00'
 id: 587014e6-a793-4f79-af13-e29a19e7d777
 keywords:
 - Qwen3-Coder-Next
 - Alibaba
 - coding model
 - temperature
 - top_p
 - top_k
 - min_p
 - GGUF
 - quantization
 - UD-Q4_K_XL
 - FP8
 - vLLM
 - llama.cpp
 - llama-server
 - MoE
 - 80B
 - 3B active
 - non-reasoning
 - agentic coding
 - SWE-Bench
 - tool calling
 links: []
 modified: '2026-04-09T18:00:31+00:00'
 summary: Optimal running parameters for Qwen3-Coder-Next (80B MoE, 3B active), covering
  inference settings, hardware requirements, quantization benchmarks, FP8 vLLM deployment,
  tool calling setup, and llama.cpp example commands.
 tags:
 - reference
 - qwen3-coder-next
 - llm
 - inference
 - llama-cpp
 - local-ai
 - coding
 title: Qwen3-Coder-Next — Optimal Running Parameters
 ---
 # Qwen3-Coder-Next — Optimal Running Parameters
 A reference for optimally running Qwen3-Coder-Next, Alibaba's 80B MoE (3B active parameters) coding model, locally.
 ## Model Overview
 | Property | Value |
 |---|---|
 | Architecture | MoE — 80B total, **3B active parameters** |
 | Max Context | **262,144** (256K) natively |
 | Recommended Context | **32,768** for less memory use |
 | Thinking Mode | **Non-reasoning only** — no thinking/reasoning mode. Does not generate `<think)>` blocks. |
 | Best For | Fast agentic coding, long-horizon reasoning, complex tool use, recovery from execution failures |
 | Performance Tier | Comparable to models with 10–20× more active parameters |
 ## Recommended Inference Parameters
 These are Qwen's recommended settings for optimal performance:
 | Parameter | Value | Notes |
 |---|---|---|
 | `temperature` | **1.0** | |
 | `top_p` | **0.95** | |
 | `top_k` | **40** | |
 | `min_p` | **0.01** | llama.cpp default is 0.05 — override to 0.01 |
 | `repetition_penalty` | **1.0** (disabled) | Only increase if you see looping |
 ### Quick Reference — One-Liner
 ```
 temperature=1.0, top_p=0.95, top_k=40, min_p=0.01, repetition_penalty=1.0
 ```
 ## Hardware Requirements (Total Memory: RAM + VRAM, or Unified Memory)
 | Precision | Memory Required |
 |---|---|
 | 3-bit | ~34 GB |
 | **4-bit (UD-Q4_K_XL)** | **~46 GB** |
 | 8-bit | ~85 GB |
 | BF16 | ~160 GB |
 > **Rule of thumb:** `disk space + RAM + VRAM ≥ size of quant`. If the quant fully fits on your device, expect **20+ tokens/s**. If it doesn't fit, it'll still work via offloading but will be slower.
 ### Recommended Minimum Quantization
 - **3-bit (`UD-IQ3_XXS`)** — comes close to BF16 performance; sensible minimum for most use cases.
 - **4-bit (`UD-Q4_K_XL`)** — recommended starting point; best balance of size and quality.
 - At least **Q4_K_M** is advised based on third-party benchmarks (Unsloth's Q4_K_M outperforms standard Q4_K_M).
 ## Thinking Mode
 This model is **non-reasoning only**. It does NOT support thinking/reasoning mode and does not generate `<think)>` blocks.
 - `enable_thinking=False` is **not required** — it's the only mode.
 - This design choice enables **ultra-quick code responses**.
 ## Critical Warnings
 - ⚠️ **Update llama.cpp** — a previous bug in `vectorized key_gdiff` calculation caused looping and output issues. Re-download GGUFs and update llama.cpp.
 - ⚠️ **No Ollama support** — use llama.cpp-compatible backends.
 - ⚠️ If context length is set too low, you may see errors like `exceeds the available context size`. Increase `--ctx-size` or use `--fit on` for auto-detection.
 - ⚠️ Tool-calling improved after llama.cpp parsing fixes (Feb 19 update). Ensure you're on a recent version.
 ## Example llama.cpp Commands
 ### Direct Pull from Hugging Face (4-bit)
 ```bash
 ./llama.cpp/llama-cli \
    -hf unsloth/Qwen3-Coder-Next-GGUF:UD-Q4_K_XL \
    --ctx-size 16384 \
    --temp 1.0 --top-p 0.95 --min-p 0.01 --top-k 40
 ```
 ### Conversation Mode with Local GGUF
 ```bash
 ./llama.cpp/llama-cli \
    --model unsloth/Qwen3-Coder-Next-GGUF/Qwen3-Coder-Next-UD-Q4_K_XL.gguf \
    --seed 3407 \
    --temp 1.0 \
    --top-p 0.95 \
    --min-p 0.01 \
    --top-k 40
 ```
 ### llama-server Deployment
 ```bash
 ./llama.cpp/llama-server \
    --model unsloth/Qwen3-Coder-Next-GGUF/Qwen3-Coder-Next-UD-Q4_K_XL.gguf \
    --alias "unsloth/Qwen3-Coder-Next" \
    --seed 3407 \
    --temp 1.0 --top-p 0.95 --min-p 0.01 --top-k 40 \
    --port 8001
 ```
 ### OpenAI-Compatible Client
 After deploying via `llama-server`:
 ```python
 from openai import OpenAI
 openai_client = OpenAI(
    base_url = "http://127.0.0.1:8001/v1",
    api_key = "sk-no-key-required",
 )
 completion = openai_client.chat.completions.create(
    model = "unsloth/Qwen3-Coder-Next",
    messages = [{"role": "user", "content": "Create a Flappy Bird game in HTML"}],
 )
 print(completion.choices[0].message.content)
 ```
 ## FP8 Dynamic Quant (vLLM)
 For premium and fast inference with GPU setups, use Unsloth's [FP8 Dynamic quant](https://huggingface.co/unsloth/Qwen3-Coder-Next-FP8-Dynamic). FP8 can boost throughput by **25% or more**.
 ### Install vLLM Nightly
 ```bash
 curl -LsSf https://astral.sh/uv/install.sh | sh
 uv venv unsloth_fp8 --python 3.12 --seed
 source unsloth_fp8/bin/activate
 uv pip install --upgrade --force-reinstall vllm --torch-backend=auto \
    --extra-index-url https://wheels.vllm.ai/nightly/cu130
 uv pip install --upgrade --force-reinstall git+https://github.com/huggingface/transformers.git
 uv pip install --force-reinstall numba
 ```
 > Change `cu130` to your CUDA version (only `cu129` and `cu130` currently supported). Check via `nvidia-smi`.
 ### Serve FP8 with vLLM (Multi-GPU)
 ```bash
 export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False
 CUDA_VISIBLE_DEVICES='0,1,2,3' vllm serve unsloth/Qwen3-Coder-Next-FP8-Dynamic \
    --served-model-name unsloth/Qwen3-Coder-Next \
    --tensor-parallel-size 4 \
    --tool-call-parser qwen3_coder \
    --enable-auto-tool-choice \
    --dtype bfloat16 \
    --seed 3407 \
    --max-model-len 200000 \
    --gpu-memory-utilization 0.93 \
    --port 8001
 ```
 > - For single GPU: `CUDA_VISIBLE_DEVICES='0'` and `--tensor-parallel-size 1` (or remove the flag).
 > - Add `--kv-cache-dtype fp8` to reduce KV cache memory usage by **50%**.
 ## Quantization Benchmarks (Third-Party)
 ### Aider Polyglot Benchmarks
 - **3-bit (`UD-IQ3_XXS`)** comes close to **BF16** performance — sensible minimum.
 - Quality steadily improves: **1-bit → 2-bit → 3-bit → 6-bit**.
 - **Unsloth's Q4_K_M** outperforms standard (non-Unsloth) Q4_K_M.
 - **At least Q4_K_M is advised.**
 ### Benjamin Marie Benchmarks (750-prompt suite: LiveCodeBench v6, MMLU Pro, GPQA, Math500)
 - Unsloth's Q4_K_M quants perform better than standard Q4_K_M.
 - Q3_K_M performs worse on LiveCodeBench v6 but surprisingly better on HumanEval than standard Q4_K_M.
 ## Performance Comparison
 Qwen3-Coder-Next (80B, 3B active) vs. much larger models:
 | Benchmark | Qwen3-Coder-Next (80B) | DeepSeek-V3.2 (671B) | GLM-4.7 (358B) | MiniMax M2.1 (229B) |
 |---|---|---|---|---|
 | SWE-Bench Verified | 70.6 | 70.2 | 74.2 | 74.8 |
 | SWE-Bench Multilingual | 62.8 | 62.3 | 63.7 | 66.2 |
 | SWE-Bench Pro | **44.3** | 40.9 | 40.6 | 34.6 |
 | Terminal-Bench 2.0 | 36.2 | 39.3 | 37.1 | 32.6 |
 | Aider | 66.2 | 69.9 | 52.1 | 61.0 |
 ## Sources
 - Unsloth Qwen3-Coder-Next Documentation (How to Run Locally)
 - GGUF collection: <https://huggingface.co/unsloth/Qwen3-Coder-Next-GGUF>
 - FP8 Dynamic: <https://huggingface.co/unsloth/Qwen3-Coder-Next-FP8-Dynamic>
 - Third-party benchmarks: Aider Polyglot, Benjamin Marie