VuongKB/reference/nemotron-3-super-running-parameters.md

---
created: '2026-04-09T18:07:12+00:00'
id: 2a98c58e-a3e3-429b-87b0-dd9c199a6e91
keywords:
- nemotron
- unsloth
- llama.cpp
- nvidia
- super
- nvidia-nemotron-3-super-120b-a12b-gguf
- reasoning
- ud-q4_k_xl
- nvidia-nemotron-3-super-120b-a12b
- model
- huggingface.co
- running
- parameters
- context
- top_p
links: []
modified: '2026-04-10T01:48:14+00:00'
summary: Optimal running parameters for NVIDIA's Nemotron 3 Super (120B-A12B MoE),
  covering inference settings for general chat vs tool calling, hardware requirements,
  NoPE context extension, reasoning token handling, GGUF/NVFP4/FP8 formats, and llama.cpp
  example commands.
tags:
- reference
- nemotron-3
- nvidia
- llm
- inference
- llama-cpp
- local-ai
title: NVIDIA Nemotron 3 Super — Optimal Running Parameters
---

# NVIDIA Nemotron 3 Super — Optimal Running Parameters

A reference for optimally running NVIDIA's Nemotron 3 Super (120B-A12B) hybrid reasoning MoE model locally.

## Model Overview

| Property | Value |
|---|---|
| Architecture | MoE — 120B total, **12B active parameters** |
| Max Context | **1M tokens** (default 262,144) |
| Positional Embeddings | **NoPE** — no explicit positional embeddings; YaRN not needed |
| Reasoning Tokens | `<think)>` (token ID 12), `</think)>` (token ID 13) |
| Best For | Multi-agent AI, high-efficiency reasoning, coding, math, agentic workflows |
| Performance Tier | Leads its size class on AIME 2025, Terminal Bench, SWE-Bench Verified; highest throughput |

## Recommended Inference Parameters

NVIDIA recommends two parameter profiles:

### General Chat / Instruction

| Parameter | Value |
|---|---|
| `temperature` | **1.0** |
| `top_p` | **1.0** |

### Tool Calling

| Parameter | Value |
|---|---|
| `temperature` | **0.6** |
| `top_p` | **0.95** |

### Additional Settings

| Setting | Value | Notes |
|---|---|---|
| `max_new_tokens` | **32,768 – 262,144** | Standard prompts; max 1M |
| `min_p` | **0.01** | Used in llama-server deployment |
| `--prio` | **2** or **3** | Priority scheduling in llama.cpp |
| `--seed` | **3407** | Reproducibility |
| `--special` | (flag) | Needed to see reasoning tokens |
| `--verbose-prompt` | (flag) | Needed to see `<think)>` since it's prepended |

### Quick Reference — One-Liners

**General chat/instruction:**
```
temperature=1.0, top_p=1.0, min_p=0.01
```

**Tool calling:**
```
temperature=0.6, top_p=0.95, min_p=0.01
```

## Hardware Requirements (Total Memory: RAM + VRAM, or Unified Memory)

| Precision | Memory Required |
|---|---|
| 2-bit (UD-Q2_K_XL) | ~32–36 GB |
| **4-bit (UD-Q4_K_XL)** | **~64–72 GB** |
| 8-bit | ~128 GB |
| BF16 | ~240 GB |

> **Rule of thumb:** Total available memory should exceed the quantized model size. If it doesn't, llama.cpp can offload to SSD/HDD but inference will be slower.

### Quantization Recommendations

- **Minimum recommended:** **UD-Q2_K_XL** (Dynamic 2-bit) to balance size and accuracy.
- **Recommended starting point:** **UD-Q4_K_XL** (Dynamic 4-bit) — fits 64–72GB devices.
- Available in multiple formats: GGUF, NVFP4, FP8, BF16.

## Context Window Management

| Setting | Value |
|---|---|
| Native max context | **1M tokens** |
| Default context | **262,144** (256K) |
| Recommended starting context | **16,384** (16K) or **32,768** (32K) |

> ⚠️ Setting context to 1M may trigger **CUDA OOM and crash** — increase gradually as RAM/VRAM allows.

### NoPE — No Positional Embeddings

Because Nemotron 3 was trained with **NoPE**:
- Extend context by simply changing `max_position_embeddings`.
- **YaRN is NOT needed** for context extension.

## Chat Template Format

Nemotron 3 uses ChatML-style format:

```
system\n</think>

user\n{message}
</think>{reasoning}</think){answer}
```

### Reasoning Tokens

- `<think)>` = token ID **12**
- `</think)>` = token ID **13**
- Use `--special` to see tokens, `--verbose-prompt` to see the prepended `<think)>`.

## Deployment Issues: EOS Token Leak

**Problem:** NVIDIA Nemotron 3 models (Super 120B, Nano 30B) use ChatML-style chat templates. The EOS token `</think>` can leak into generated content when running with the `--special` flag in llama.cpp. This causes parse failures on subsequent turns, e.g.:

```
Error: Failed to parse input at pos 285: 
</think>
```

The `--special` flag is required for Nemotron reasoning tokens, so it cannot be removed.

**Fix:** Stop the generation when the `</think>` token appears by including it in the **payload's `stop` array** when calling the API. The `--stop` command-line flag does not exist in llama-server; stop sequences must be handled at the request level in the API payload.

**Applies to:**
- `nemotron-3-super-120b` — the 120B-A12B flagship model
- `nemotron-3-nano-30b` — the 30B-A3B compact model

**Config file:** `configs/pfi-ana/docker/conf/llama-swap/config.yaml`

**Example fix in API payload (Python client):**
```python
completion = openai_client.chat.completions.create(
    model = "unsloth/NVIDIA-Nemotron-3-Super-120B-A12B",
    messages = [{"role": "user", "content": "What is 2+2?"}],
    stop=["</think>"]  # Stop sequences must be in the payload, not command-line
)
```

**Example fix with cURL:**
```bash
curl -X POST http://localhost:8001/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "unsloth/NVIDIA-Nemotron-3-Super-120B-A12B",
        "messages": [{"role": "user", "content": "What is 2+2?"}],
        "stop": ["</think>"]
    }'
```

## Example llama.cpp Commands

### General Instruction (4-bit)

```bash
./llama.cpp/llama-cli \
    -hf unsloth/NVIDIA-Nemotron-3-Super-120B-A12B-GGUF:UD-Q4_K_XL \
    --ctx-size 16384 \
    --temp 1.0 --top-p 1.0
```

### Tool Calling (4-bit)

```bash
./llama.cpp/llama-cli \
    -hf unsloth/NVIDIA-Nemotron-3-Super-120B-A12B-GGUF:UD-Q4_K_XL \
    --ctx-size 32768 \
    --temp 0.6 --top-p 0.95
```

### Conversation Mode (Local GGUF)

```bash
./llama.cpp/llama-cli \
    --model unsloth/NVIDIA-Nemotron-3-Super-120B-A12B-GGUF/UD-Q4_K_XL/NVIDIA-Nemotron-3-Super-120B-A12B-UD-Q4_K_XL-00001-of-00003.gguf \
    --ctx-size 16384 \
    --seed 3407 \
    --prio 2 \
    --temp 0.6 \
    --top-p 0.95
```

### llama-server Deployment

```bash
./llama.cpp/llama-server \
    --model unsloth/NVIDIA-Nemotron-3-Super-120B-A12B-GGUF/UD-Q4_K_XL/NVIDIA-Nemotron-3-Super-120B-A12B-UD-Q4_K_XL-00001-of-00003.gguf \
    --alias "unsloth/NVIDIA-Nemotron-3-Super-120B-A12B" \
    --prio 3 \
    --min_p 0.01 \
    --temp 0.6 --top-p 0.95 \
    --ctx-size 16384 \
    --port 8001
```

### OpenAI-Compatible Client (with Reasoning Output)

```python
from openai import OpenAI

openai_client = OpenAI(
    base_url = "http://127.0.0.1:8001/v1",
    api_key = "sk-no-key-required",
)

completion = openai_client.chat.completions.create(
    model = "unsloth/NVIDIA-Nemotron-3-Super-120B-A12B",
    messages = [{"role": "user", "content": "What is 2+2?"}]
)

# Access both reasoning and final answer
print(completion.choices[0].message.reasoning_content)
print(completion.choices[0].message.content)
```

> **Note:** Unlike some models, Nemotron 3 Super exposes reasoning content separately via `reasoning_content` in the OpenAI-compatible API.

## Fine-Tuning Notes

- **bf16 LoRA** works on **256GB VRAM** (multi-GPU setup).
- Router-layer fine-tuning is **disabled by default** for stability.
- For multi-GPU, use `device_map = "balanced"` or follow Unsloth's multiGPU guide.
- To retain reasoning capabilities in fine-tuning, use at least **75% reasoning** and **25% non-reasoning** data.

## Available Formats

| Format | Link |
|---|---|
| GGUF (Dynamic quantization) | [unsloth/NVIDIA-Nemotron-3-Super-120B-A12B-GGUF](https://huggingface.co/unsloth/NVIDIA-Nemotron-3-Super-120B-A12B-GGUF) |
| NVFP4 | [unsloth/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4](https://huggingface.co/unsloth/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4) |
| FP8 | [unsloth/NVIDIA-Nemotron-3-Super-120B-A12B-FP8](https://huggingface.co/unsloth/NVIDIA-Nemotron-3-Super-120B-A12B-FP8) |
| BF16 | [unsloth/NVIDIA-Nemotron-3-Super-120B-A12B](https://huggingface.co/unsloth/NVIDIA-Nemotron-3-Super-120B-A12B) |

## Sources

- Unsloth NVIDIA Nemotron 3 Super Documentation (How to Run)
- GGUF: <https://huggingface.co/unsloth/NVIDIA-Nemotron-3-Super-120B-A12B-GGUF>