233 lines
6.6 KiB
Markdown
233 lines
6.6 KiB
Markdown
---
|
||
created: '2026-04-09T18:07:12+00:00'
|
||
id: 2a98c58e-a3e3-429b-87b0-dd9c199a6e91
|
||
keywords:
|
||
- Nemotron 3 Super
|
||
- NVIDIA
|
||
- 120B-A12B
|
||
- temperature
|
||
- top_p
|
||
- min_p
|
||
- GGUF
|
||
- quantization
|
||
- UD-Q4_K_XL
|
||
- NoPE
|
||
- 1M context
|
||
- llama.cpp
|
||
- llama-server
|
||
- MoE
|
||
- 12B active
|
||
- tool calling
|
||
- reasoning tokens
|
||
- NVFP4
|
||
- FP8
|
||
- BF16
|
||
links: []
|
||
modified: '2026-04-09T18:07:12+00:00'
|
||
summary: Optimal running parameters for NVIDIA's Nemotron 3 Super (120B-A12B MoE),
|
||
covering inference settings for general chat vs tool calling, hardware requirements,
|
||
NoPE context extension, reasoning token handling, GGUF/NVFP4/FP8 formats, and llama.cpp
|
||
example commands.
|
||
tags:
|
||
- reference
|
||
- nemotron-3
|
||
- nvidia
|
||
- llm
|
||
- inference
|
||
- llama-cpp
|
||
- local-ai
|
||
title: NVIDIA Nemotron 3 Super — Optimal Running Parameters
|
||
---
|
||
|
||
# NVIDIA Nemotron 3 Super — Optimal Running Parameters
|
||
|
||
A reference for optimally running NVIDIA's Nemotron 3 Super (120B-A12B) hybrid reasoning MoE model locally.
|
||
|
||
## Model Overview
|
||
|
||
| Property | Value |
|
||
|---|---|
|
||
| Architecture | MoE — 120B total, **12B active parameters** |
|
||
| Max Context | **1M tokens** (default 262,144) |
|
||
| Positional Embeddings | **NoPE** — no explicit positional embeddings; YaRN not needed |
|
||
| Reasoning Tokens | `<think)>` (token ID 12), `</think)>` (token ID 13) |
|
||
| Best For | Multi-agent AI, high-efficiency reasoning, coding, math, agentic workflows |
|
||
| Performance Tier | Leads its size class on AIME 2025, Terminal Bench, SWE-Bench Verified; highest throughput |
|
||
|
||
## Recommended Inference Parameters
|
||
|
||
NVIDIA recommends two parameter profiles:
|
||
|
||
### General Chat / Instruction
|
||
|
||
| Parameter | Value |
|
||
|---|---|
|
||
| `temperature` | **1.0** |
|
||
| `top_p` | **1.0** |
|
||
|
||
### Tool Calling
|
||
|
||
| Parameter | Value |
|
||
|---|---|
|
||
| `temperature` | **0.6** |
|
||
| `top_p` | **0.95** |
|
||
|
||
### Additional Settings
|
||
|
||
| Setting | Value | Notes |
|
||
|---|---|---|
|
||
| `max_new_tokens` | **32,768 – 262,144** | Standard prompts; max 1M |
|
||
| `min_p` | **0.01** | Used in llama-server deployment |
|
||
| `--prio` | **2** or **3** | Priority scheduling in llama.cpp |
|
||
| `--seed` | **3407** | Reproducibility |
|
||
| `--special` | (flag) | Needed to see reasoning tokens |
|
||
| `--verbose-prompt` | (flag) | Needed to see `<think)>` since it's prepended |
|
||
|
||
### Quick Reference — One-Liners
|
||
|
||
**General chat/instruction:**
|
||
```
|
||
temperature=1.0, top_p=1.0
|
||
```
|
||
|
||
**Tool calling:**
|
||
```
|
||
temperature=0.6, top_p=0.95
|
||
```
|
||
|
||
## Hardware Requirements (Total Memory: RAM + VRAM, or Unified Memory)
|
||
|
||
| Precision | Memory Required |
|
||
|---|---|
|
||
| 2-bit (UD-Q2_K_XL) | ~32–36 GB |
|
||
| **4-bit (UD-Q4_K_XL)** | **~64–72 GB** |
|
||
| 8-bit | ~128 GB |
|
||
| BF16 | ~240 GB |
|
||
|
||
> **Rule of thumb:** Total available memory should exceed the quantized model size. If it doesn't, llama.cpp can offload to SSD/HDD but inference will be slower.
|
||
|
||
### Quantization Recommendations
|
||
|
||
- **Minimum recommended:** **UD-Q2_K_XL** (Dynamic 2-bit) to balance size and accuracy.
|
||
- **Recommended starting point:** **UD-Q4_K_XL** (Dynamic 4-bit) — fits 64–72GB devices.
|
||
- Available in multiple formats: GGUF, NVFP4, FP8, BF16.
|
||
|
||
## Context Window Management
|
||
|
||
| Setting | Value |
|
||
|---|---|
|
||
| Native max context | **1M tokens** |
|
||
| Default context | **262,144** (256K) |
|
||
| Recommended starting context | **16,384** (16K) or **32,768** (32K) |
|
||
|
||
> ⚠️ Setting context to 1M may trigger **CUDA OOM and crash** — increase gradually as RAM/VRAM allows.
|
||
|
||
### NoPE — No Positional Embeddings
|
||
|
||
Because Nemotron 3 was trained with **NoPE**:
|
||
- Extend context by simply changing `max_position_embeddings`.
|
||
- **YaRN is NOT needed** for context extension.
|
||
|
||
## Chat Template Format
|
||
|
||
Nemotron 3 uses ChatML-style format:
|
||
|
||
```
|
||
<|im_start|>system\n<|im_end|>\n
|
||
<|im_start|>user\n{message}<|im_end|>\n
|
||
<|im_start|>assistant\n<think)>{reasoning}</think)>{answer}<|im_end|>\n
|
||
```
|
||
|
||
### Reasoning Tokens
|
||
|
||
- `<think)>` = token ID **12**
|
||
- `</think)>` = token ID **13**
|
||
- Use `--special` to see tokens, `--verbose-prompt` to see the prepended `<think)>`.
|
||
|
||
## Example llama.cpp Commands
|
||
|
||
### General Instruction (4-bit)
|
||
|
||
```bash
|
||
./llama.cpp/llama-cli \
|
||
-hf unsloth/NVIDIA-Nemotron-3-Super-120B-A12B-GGUF:UD-Q4_K_XL \
|
||
--ctx-size 16384 \
|
||
--temp 1.0 --top-p 1.0
|
||
```
|
||
|
||
### Tool Calling (4-bit)
|
||
|
||
```bash
|
||
./llama.cpp/llama-cli \
|
||
-hf unsloth/NVIDIA-Nemotron-3-Super-120B-A12B-GGUF:UD-Q4_K_XL \
|
||
--ctx-size 32768 \
|
||
--temp 0.6 --top-p 0.95
|
||
```
|
||
|
||
### Conversation Mode (Local GGUF)
|
||
|
||
```bash
|
||
./llama.cpp/llama-cli \
|
||
--model unsloth/NVIDIA-Nemotron-3-Super-120B-A12B-GGUF/UD-Q4_K_XL/NVIDIA-Nemotron-3-Super-120B-A12B-UD-Q4_K_XL-00001-of-00003.gguf \
|
||
--ctx-size 16384 \
|
||
--seed 3407 \
|
||
--prio 2 \
|
||
--temp 0.6 \
|
||
--top-p 0.95
|
||
```
|
||
|
||
### llama-server Deployment
|
||
|
||
```bash
|
||
./llama.cpp/llama-server \
|
||
--model unsloth/NVIDIA-Nemotron-3-Super-120B-A12B-GGUF/UD-Q4_K_XL/NVIDIA-Nemotron-3-Super-120B-A12B-UD-Q4_K_XL-00001-of-00003.gguf \
|
||
--alias "unsloth/NVIDIA-Nemotron-3-Super-120B-A12B" \
|
||
--prio 3 \
|
||
--min_p 0.01 \
|
||
--temp 0.6 --top-p 0.95 \
|
||
--ctx-size 16384 \
|
||
--port 8001
|
||
```
|
||
|
||
### OpenAI-Compatible Client (with Reasoning Output)
|
||
|
||
```python
|
||
from openai import OpenAI
|
||
|
||
openai_client = OpenAI(
|
||
base_url = "http://127.0.0.1:8001/v1",
|
||
api_key = "sk-no-key-required",
|
||
)
|
||
|
||
completion = openai_client.chat.completions.create(
|
||
model = "unsloth/NVIDIA-Nemotron-3-Super-120B-A12B",
|
||
messages = [{"role": "user", "content": "What is 2+2?"}],
|
||
)
|
||
|
||
# Access both reasoning and final answer
|
||
print(completion.choices[0].message.reasoning_content)
|
||
print(completion.choices[0].message.content)
|
||
```
|
||
|
||
> **Note:** Unlike some models, Nemotron 3 Super exposes reasoning content separately via `reasoning_content` in the OpenAI-compatible API.
|
||
|
||
## Fine-Tuning Notes
|
||
|
||
- **bf16 LoRA** works on **256GB VRAM** (multi-GPU setup).
|
||
- Router-layer fine-tuning is **disabled by default** for stability.
|
||
- For multi-GPU, use `device_map = "balanced"` or follow Unsloth's multiGPU guide.
|
||
- To retain reasoning capabilities in fine-tuning, use at least **75% reasoning** and **25% non-reasoning** data.
|
||
|
||
## Available Formats
|
||
|
||
| Format | Link |
|
||
|---|---|
|
||
| GGUF (Dynamic quantization) | [unsloth/NVIDIA-Nemotron-3-Super-120B-A12B-GGUF](https://huggingface.co/unsloth/NVIDIA-Nemotron-3-Super-120B-A12B-GGUF) |
|
||
| NVFP4 | [unsloth/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4](https://huggingface.co/unsloth/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4) |
|
||
| FP8 | [unsloth/NVIDIA-Nemotron-3-Super-120B-A12B-FP8](https://huggingface.co/unsloth/NVIDIA-Nemotron-3-Super-120B-A12B-FP8) |
|
||
| BF16 | [unsloth/NVIDIA-Nemotron-3-Super-120B-A12B](https://huggingface.co/unsloth/NVIDIA-Nemotron-3-Super-120B-A12B) |
|
||
|
||
## Sources
|
||
|
||
- Unsloth NVIDIA Nemotron 3 Super Documentation (How to Run)
|
||
- GGUF: <https://huggingface.co/unsloth/NVIDIA-Nemotron-3-Super-120B-A12B-GGUF> |