diff --git a/reference/nemotron-3-super-running-parameters.md b/reference/nemotron-3-super-running-parameters.md index 582cd02..9aca5de 100644 --- a/reference/nemotron-3-super-running-parameters.md +++ b/reference/nemotron-3-super-running-parameters.md @@ -2,28 +2,23 @@ created: '2026-04-09T18:07:12+00:00' id: 2a98c58e-a3e3-429b-87b0-dd9c199a6e91 keywords: -- Nemotron 3 Super -- NVIDIA -- 120B-A12B -- temperature -- top_p -- min_p -- GGUF -- quantization -- UD-Q4_K_XL -- NoPE -- 1M context +- nemotron +- unsloth - llama.cpp -- llama-server -- MoE -- 12B active -- tool calling -- reasoning tokens -- NVFP4 -- FP8 -- BF16 +- nvidia-nemotron-3-super-120b-a12b-gguf +- nvidia +- super +- ud-q4_k_xl +- reasoning +- min_p +- nvidia-nemotron-3-super-120b-a12b +- huggingface.co +- running +- parameters +- model +- context links: [] -modified: '2026-04-09T18:07:12+00:00' +modified: '2026-04-10T01:10:44+00:00' summary: Optimal running parameters for NVIDIA's Nemotron 3 Super (120B-A12B MoE), covering inference settings for general chat vs tool calling, hardware requirements, NoPE context extension, reasoning token handling, GGUF/NVFP4/FP8 formats, and llama.cpp @@ -87,12 +82,12 @@ NVIDIA recommends two parameter profiles: **General chat/instruction:** ``` -temperature=1.0, top_p=1.0 +temperature=1.0, top_p=1.0, min_p=0.01 ``` **Tool calling:** ``` -temperature=0.6, top_p=0.95 +temperature=0.6, top_p=0.95, min_p=0.01 ``` ## Hardware Requirements (Total Memory: RAM + VRAM, or Unified Memory) @@ -133,9 +128,13 @@ Because Nemotron 3 was trained with **NoPE**: Nemotron 3 uses ChatML-style format: ``` -<|im_start|>system\n<|im_end|>\n -<|im_start|>user\n{message}<|im_end|>\n -<|im_start|>assistant\n{reasoning}{answer}<|im_end|>\n +system\n + +user\n{message} + + + +{reasoning}` = token ID **13** - Use `--special` to see tokens, `--verbose-prompt` to see the prepended ``. +## Deployment Issues: EOS Token Leak + +**Problem:** NVIDIA Nemotron 3 models (Super 120B, Nano 30B) use ChatML-style chat templates. The EOS token `` can leak into generated content when running with the `--special` flag in llama.cpp. This causes parse failures on subsequent turns, e.g.: + +``` +Error: Failed to parse input at pos 285: + +``` + +The `--special` flag is required for Nemotron reasoning tokens, so it cannot be removed. + +**Fix:** Add `--stop ""` to the `llama-server` command in the llama-swap config. This tells the server to treat `` as a stop sequence, preventing it from appearing in generated output. + +**Applies to:** +- `nemotron-3-super-120b` — the 120B-A12B flagship model +- `nemotron-3-nano-30b` — the 30B-A3B compact model + +**Config file:** `configs/pfi-ana/docker/conf/llama-swap/config.yaml` + +**Example fix in llama-server command:** +```bash +./llama.cpp/llama-server \ + --model unsloth/NVIDIA-Nemotron-3-Super-120B-A12B-GGUF/UD-Q4_K_XL/NVIDIA-Nemotron-3-Super-120B-A12B-UD-Q4_K_XL-00001-of-00003.gguf \ + --alias "unsloth/NVIDIA-Nemotron-3-Super-120B-A12B" \ + --prio 3 \ + --min_p 0.01 \ + --temp 0.6 --top-p 0.95 \ + --ctx-size 16384 \ + --stop "" \ + --port 8001 +``` + ## Example llama.cpp Commands ### General Instruction (4-bit)