diff --git a/reference/nemotron-3-super-running-parameters.md b/reference/nemotron-3-super-running-parameters.md index 9aca5de..53806d4 100644 --- a/reference/nemotron-3-super-running-parameters.md +++ b/reference/nemotron-3-super-running-parameters.md @@ -5,20 +5,20 @@ keywords: - nemotron - unsloth - llama.cpp -- nvidia-nemotron-3-super-120b-a12b-gguf - nvidia - super -- ud-q4_k_xl +- nvidia-nemotron-3-super-120b-a12b-gguf - reasoning -- min_p +- ud-q4_k_xl - nvidia-nemotron-3-super-120b-a12b +- model - huggingface.co - running - parameters -- model - context +- top_p links: [] -modified: '2026-04-10T01:10:44+00:00' +modified: '2026-04-10T01:48:14+00:00' summary: Optimal running parameters for NVIDIA's Nemotron 3 Super (120B-A12B MoE), covering inference settings for general chat vs tool calling, hardware requirements, NoPE context extension, reasoning token handling, GGUF/NVFP4/FP8 formats, and llama.cpp @@ -131,10 +131,7 @@ Nemotron 3 uses ChatML-style format: system\n user\n{message} - - - -{reasoning}{reasoning}"` to the `llama-server` command in the llama-swap config. This tells the server to treat `` as a stop sequence, preventing it from appearing in generated output. +**Fix:** Stop the generation when the `` token appears by including it in the **payload's `stop` array** when calling the API. The `--stop` command-line flag does not exist in llama-server; stop sequences must be handled at the request level in the API payload. **Applies to:** - `nemotron-3-super-120b` — the 120B-A12B flagship model @@ -162,17 +159,24 @@ The `--special` flag is required for Nemotron reasoning tokens, so it cannot be **Config file:** `configs/pfi-ana/docker/conf/llama-swap/config.yaml` -**Example fix in llama-server command:** +**Example fix in API payload (Python client):** +```python +completion = openai_client.chat.completions.create( + model = "unsloth/NVIDIA-Nemotron-3-Super-120B-A12B", + messages = [{"role": "user", "content": "What is 2+2?"}], + stop=[""] # Stop sequences must be in the payload, not command-line +) +``` + +**Example fix with cURL:** ```bash -./llama.cpp/llama-server \ - --model unsloth/NVIDIA-Nemotron-3-Super-120B-A12B-GGUF/UD-Q4_K_XL/NVIDIA-Nemotron-3-Super-120B-A12B-UD-Q4_K_XL-00001-of-00003.gguf \ - --alias "unsloth/NVIDIA-Nemotron-3-Super-120B-A12B" \ - --prio 3 \ - --min_p 0.01 \ - --temp 0.6 --top-p 0.95 \ - --ctx-size 16384 \ - --stop "" \ - --port 8001 +curl -X POST http://localhost:8001/v1/chat/completions \ + -H "Content-Type: application/json" \ + -d '{ + "model": "unsloth/NVIDIA-Nemotron-3-Super-120B-A12B", + "messages": [{"role": "user", "content": "What is 2+2?"}], + "stop": [""] + }' ``` ## Example llama.cpp Commands @@ -232,7 +236,7 @@ openai_client = OpenAI( completion = openai_client.chat.completions.create( model = "unsloth/NVIDIA-Nemotron-3-Super-120B-A12B", - messages = [{"role": "user", "content": "What is 2+2?"}], + messages = [{"role": "user", "content": "What is 2+2?"}] ) # Access both reasoning and final answer