diff --git a/reference/qwen3-coder-next-running-parameters.md b/reference/qwen3-coder-next-running-parameters.md new file mode 100644 index 0000000..f4a43b0 --- /dev/null +++ b/reference/qwen3-coder-next-running-parameters.md @@ -0,0 +1,229 @@ +--- +created: '2026-04-09T18:00:31+00:00' +id: 587014e6-a793-4f79-af13-e29a19e7d777 +keywords: +- Qwen3-Coder-Next +- Alibaba +- coding model +- temperature +- top_p +- top_k +- min_p +- GGUF +- quantization +- UD-Q4_K_XL +- FP8 +- vLLM +- llama.cpp +- llama-server +- MoE +- 80B +- 3B active +- non-reasoning +- agentic coding +- SWE-Bench +- tool calling +links: [] +modified: '2026-04-09T18:00:31+00:00' +summary: Optimal running parameters for Qwen3-Coder-Next (80B MoE, 3B active), covering + inference settings, hardware requirements, quantization benchmarks, FP8 vLLM deployment, + tool calling setup, and llama.cpp example commands. +tags: +- reference +- qwen3-coder-next +- llm +- inference +- llama-cpp +- local-ai +- coding +title: Qwen3-Coder-Next — Optimal Running Parameters +--- + +# Qwen3-Coder-Next — Optimal Running Parameters + +A reference for optimally running Qwen3-Coder-Next, Alibaba's 80B MoE (3B active parameters) coding model, locally. + +## Model Overview + +| Property | Value | +|---|---| +| Architecture | MoE — 80B total, **3B active parameters** | +| Max Context | **262,144** (256K) natively | +| Recommended Context | **32,768** for less memory use | +| Thinking Mode | **Non-reasoning only** — no thinking/reasoning mode. Does not generate `` blocks. | +| Best For | Fast agentic coding, long-horizon reasoning, complex tool use, recovery from execution failures | +| Performance Tier | Comparable to models with 10–20× more active parameters | + +## Recommended Inference Parameters + +These are Qwen's recommended settings for optimal performance: + +| Parameter | Value | Notes | +|---|---|---| +| `temperature` | **1.0** | | +| `top_p` | **0.95** | | +| `top_k` | **40** | | +| `min_p` | **0.01** | llama.cpp default is 0.05 — override to 0.01 | +| `repetition_penalty` | **1.0** (disabled) | Only increase if you see looping | + +### Quick Reference — One-Liner + +``` +temperature=1.0, top_p=0.95, top_k=40, min_p=0.01, repetition_penalty=1.0 +``` + +## Hardware Requirements (Total Memory: RAM + VRAM, or Unified Memory) + +| Precision | Memory Required | +|---|---| +| 3-bit | ~34 GB | +| **4-bit (UD-Q4_K_XL)** | **~46 GB** | +| 8-bit | ~85 GB | +| BF16 | ~160 GB | + +> **Rule of thumb:** `disk space + RAM + VRAM ≥ size of quant`. If the quant fully fits on your device, expect **20+ tokens/s**. If it doesn't fit, it'll still work via offloading but will be slower. + +### Recommended Minimum Quantization + +- **3-bit (`UD-IQ3_XXS`)** — comes close to BF16 performance; sensible minimum for most use cases. +- **4-bit (`UD-Q4_K_XL`)** — recommended starting point; best balance of size and quality. +- At least **Q4_K_M** is advised based on third-party benchmarks (Unsloth's Q4_K_M outperforms standard Q4_K_M). + +## Thinking Mode + +This model is **non-reasoning only**. It does NOT support thinking/reasoning mode and does not generate `` blocks. + +- `enable_thinking=False` is **not required** — it's the only mode. +- This design choice enables **ultra-quick code responses**. + +## Critical Warnings + +- ⚠️ **Update llama.cpp** — a previous bug in `vectorized key_gdiff` calculation caused looping and output issues. Re-download GGUFs and update llama.cpp. +- ⚠️ **No Ollama support** — use llama.cpp-compatible backends. +- ⚠️ If context length is set too low, you may see errors like `exceeds the available context size`. Increase `--ctx-size` or use `--fit on` for auto-detection. +- ⚠️ Tool-calling improved after llama.cpp parsing fixes (Feb 19 update). Ensure you're on a recent version. + +## Example llama.cpp Commands + +### Direct Pull from Hugging Face (4-bit) + +```bash +./llama.cpp/llama-cli \ + -hf unsloth/Qwen3-Coder-Next-GGUF:UD-Q4_K_XL \ + --ctx-size 16384 \ + --temp 1.0 --top-p 0.95 --min-p 0.01 --top-k 40 +``` + +### Conversation Mode with Local GGUF + +```bash +./llama.cpp/llama-cli \ + --model unsloth/Qwen3-Coder-Next-GGUF/Qwen3-Coder-Next-UD-Q4_K_XL.gguf \ + --seed 3407 \ + --temp 1.0 \ + --top-p 0.95 \ + --min-p 0.01 \ + --top-k 40 +``` + +### llama-server Deployment + +```bash +./llama.cpp/llama-server \ + --model unsloth/Qwen3-Coder-Next-GGUF/Qwen3-Coder-Next-UD-Q4_K_XL.gguf \ + --alias "unsloth/Qwen3-Coder-Next" \ + --seed 3407 \ + --temp 1.0 --top-p 0.95 --min-p 0.01 --top-k 40 \ + --port 8001 +``` + +### OpenAI-Compatible Client + +After deploying via `llama-server`: + +```python +from openai import OpenAI + +openai_client = OpenAI( + base_url = "http://127.0.0.1:8001/v1", + api_key = "sk-no-key-required", +) + +completion = openai_client.chat.completions.create( + model = "unsloth/Qwen3-Coder-Next", + messages = [{"role": "user", "content": "Create a Flappy Bird game in HTML"}], +) + +print(completion.choices[0].message.content) +``` + +## FP8 Dynamic Quant (vLLM) + +For premium and fast inference with GPU setups, use Unsloth's [FP8 Dynamic quant](https://huggingface.co/unsloth/Qwen3-Coder-Next-FP8-Dynamic). FP8 can boost throughput by **25% or more**. + +### Install vLLM Nightly + +```bash +curl -LsSf https://astral.sh/uv/install.sh | sh +uv venv unsloth_fp8 --python 3.12 --seed +source unsloth_fp8/bin/activate + +uv pip install --upgrade --force-reinstall vllm --torch-backend=auto \ + --extra-index-url https://wheels.vllm.ai/nightly/cu130 +uv pip install --upgrade --force-reinstall git+https://github.com/huggingface/transformers.git +uv pip install --force-reinstall numba +``` + +> Change `cu130` to your CUDA version (only `cu129` and `cu130` currently supported). Check via `nvidia-smi`. + +### Serve FP8 with vLLM (Multi-GPU) + +```bash +export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False +CUDA_VISIBLE_DEVICES='0,1,2,3' vllm serve unsloth/Qwen3-Coder-Next-FP8-Dynamic \ + --served-model-name unsloth/Qwen3-Coder-Next \ + --tensor-parallel-size 4 \ + --tool-call-parser qwen3_coder \ + --enable-auto-tool-choice \ + --dtype bfloat16 \ + --seed 3407 \ + --max-model-len 200000 \ + --gpu-memory-utilization 0.93 \ + --port 8001 +``` + +> - For single GPU: `CUDA_VISIBLE_DEVICES='0'` and `--tensor-parallel-size 1` (or remove the flag). +> - Add `--kv-cache-dtype fp8` to reduce KV cache memory usage by **50%**. + +## Quantization Benchmarks (Third-Party) + +### Aider Polyglot Benchmarks + +- **3-bit (`UD-IQ3_XXS`)** comes close to **BF16** performance — sensible minimum. +- Quality steadily improves: **1-bit → 2-bit → 3-bit → 6-bit**. +- **Unsloth's Q4_K_M** outperforms standard (non-Unsloth) Q4_K_M. +- **At least Q4_K_M is advised.** + +### Benjamin Marie Benchmarks (750-prompt suite: LiveCodeBench v6, MMLU Pro, GPQA, Math500) + +- Unsloth's Q4_K_M quants perform better than standard Q4_K_M. +- Q3_K_M performs worse on LiveCodeBench v6 but surprisingly better on HumanEval than standard Q4_K_M. + +## Performance Comparison + +Qwen3-Coder-Next (80B, 3B active) vs. much larger models: + +| Benchmark | Qwen3-Coder-Next (80B) | DeepSeek-V3.2 (671B) | GLM-4.7 (358B) | MiniMax M2.1 (229B) | +|---|---|---|---|---| +| SWE-Bench Verified | 70.6 | 70.2 | 74.2 | 74.8 | +| SWE-Bench Multilingual | 62.8 | 62.3 | 63.7 | 66.2 | +| SWE-Bench Pro | **44.3** | 40.9 | 40.6 | 34.6 | +| Terminal-Bench 2.0 | 36.2 | 39.3 | 37.1 | 32.6 | +| Aider | 66.2 | 69.9 | 52.1 | 61.0 | + +## Sources + +- Unsloth Qwen3-Coder-Next Documentation (How to Run Locally) +- GGUF collection: +- FP8 Dynamic: +- Third-party benchmarks: Aider Polyglot, Benjamin Marie \ No newline at end of file