Update: reference/nemotron-3-super-running-parameters.md — Added "Nemotron EOS Token Leak" troubleshooting section with problem description, fix (stop sequence), and updated example commands including --stop flag for llama-server deployment

2026-04-09 18:10:44 -07:00
parent 938b35286d
commit 6656620fc5
1 changed files with 56 additions and 25 deletions
@@ -2,28 +2,23 @@
 created: '2026-04-09T18:07:12+00:00'
 id: 2a98c58e-a3e3-429b-87b0-dd9c199a6e91
 keywords:
- Nemotron 3 Super
- NVIDIA
- 120B-A12B
- temperature
- top_p
- min_p
- GGUF
- quantization
- UD-Q4_K_XL
- NoPE
- 1M context
+- nemotron
+- unsloth
 - llama.cpp
- llama-server
- MoE
- 12B active
- tool calling
- reasoning tokens
- NVFP4
- FP8
- BF16
+- nvidia-nemotron-3-super-120b-a12b-gguf
+- nvidia
+- super
+- ud-q4_k_xl
+- reasoning
+- min_p
+- nvidia-nemotron-3-super-120b-a12b
+- huggingface.co
+- running
+- parameters
+- model
+- context
 links: []
-modified: '2026-04-09T18:07:12+00:00'
+modified: '2026-04-10T01:10:44+00:00'
 summary: Optimal running parameters for NVIDIA's Nemotron 3 Super (120B-A12B MoE),
  covering inference settings for general chat vs tool calling, hardware requirements,
  NoPE context extension, reasoning token handling, GGUF/NVFP4/FP8 formats, and llama.cpp
@@ -87,12 +82,12 @@ NVIDIA recommends two parameter profiles:

 **General chat/instruction:**
 ```
-temperature=1.0, top_p=1.0
+temperature=1.0, top_p=1.0, min_p=0.01
 ```

 **Tool calling:**
 ```
-temperature=0.6, top_p=0.95
+temperature=0.6, top_p=0.95, min_p=0.01
 ```

 ## Hardware Requirements (Total Memory: RAM + VRAM, or Unified Memory)
@@ -133,9 +128,13 @@ Because Nemotron 3 was trained with **NoPE**:
 Nemotron 3 uses ChatML-style format:

 ```
-<|im_start|>system\n<|im_end|>\n
-<|im_start|>user\n{message}<|im_end|>\n
-<|im_start|>assistant\n<think)>{reasoning}</think)>{answer}<|im_end|>\n
+system\n</think>
+
+user\n{message}
+</think>
+
+
+<think)>{reasoning}</think){answer}
 ```

 ### Reasoning Tokens
@@ -144,6 +143,38 @@ Nemotron 3 uses ChatML-style format:
 - `</think)>` = token ID **13**
 - Use `--special` to see tokens, `--verbose-prompt` to see the prepended `<think)>`.

+## Deployment Issues: EOS Token Leak
+
+**Problem:** NVIDIA Nemotron 3 models (Super 120B, Nano 30B) use ChatML-style chat templates. The EOS token `</think>` can leak into generated content when running with the `--special` flag in llama.cpp. This causes parse failures on subsequent turns, e.g.:
+
+```
+Error: Failed to parse input at pos 285: 
+</think>
+```
+
+The `--special` flag is required for Nemotron reasoning tokens, so it cannot be removed.
+
+**Fix:** Add `--stop "</think>"` to the `llama-server` command in the llama-swap config. This tells the server to treat `</think>` as a stop sequence, preventing it from appearing in generated output.
+
+**Applies to:**
+- `nemotron-3-super-120b` — the 120B-A12B flagship model
+- `nemotron-3-nano-30b` — the 30B-A3B compact model
+
+**Config file:** `configs/pfi-ana/docker/conf/llama-swap/config.yaml`
+
+**Example fix in llama-server command:**
+```bash
+./llama.cpp/llama-server \
+    --model unsloth/NVIDIA-Nemotron-3-Super-120B-A12B-GGUF/UD-Q4_K_XL/NVIDIA-Nemotron-3-Super-120B-A12B-UD-Q4_K_XL-00001-of-00003.gguf \
+    --alias "unsloth/NVIDIA-Nemotron-3-Super-120B-A12B" \
+    --prio 3 \
+    --min_p 0.01 \
+    --temp 0.6 --top-p 0.95 \
+    --ctx-size 16384 \
+    --stop "</think>" \
+    --port 8001
+```
+
 ## Example llama.cpp Commands

 ### General Instruction (4-bit)