---
created: '2026-04-09T18:05:28+00:00'
id: 4aec42e4-21ba-4279-b41b-58a8ebfefd73
keywords:
- Nemotron 3
- NVIDIA
- Nano-4B
- Nano-30B-A3B
- temperature
- top_p
- min_p
- GGUF
- quantization
- UD-Q4_K_XL
- Q8_0
- NoPE
- 1M context
- llama.cpp
- llama-server
- MoE
- tool calling
- reasoning tokens
links: []
modified: '2026-04-09T18:05:28+00:00'
summary: Optimal running parameters for NVIDIA's Nemotron 3 Nano family (4B and 30B-A3B),
  covering inference settings for general chat vs tool calling, hardware requirements,
  NoPE context extension, reasoning token handling, and llama.cpp example commands.
tags:
- reference
- nemotron-3
- nvidia
- llm
- inference
- llama-cpp
- local-ai
title: NVIDIA Nemotron 3 Nano — Optimal Running Parameters
---

# NVIDIA Nemotron 3 Nano — Optimal Running Parameters

A reference for optimally running NVIDIA's Nemotron 3 Nano family of hybrid MoE models locally.

## Model Variants

| Variant | Architecture | Context | Best Fit |
|---|---|---|---|
| **Nano-4B** | MoE | Up to **1M tokens** | Lightweight coding, math, agentic tasks; fits 5GB devices |
| **Nano-30B-A3B** | MoE (3B active) | Up to **1M tokens** | Best performing for size; fits 24GB devices |
| Super-120B-A12B | MoE (12B active) | Up to **1M tokens** | High-end workstation / server (separate guide) |

> All Nemotron 3 models support a **1M-token context window**.

## Recommended Inference Parameters

NVIDIA recommends two parameter profiles depending on use-case:

### General Chat / Instruction

| Parameter | Value |
|---|---|
| `temperature` | **1.0** |
| `top_p` | **1.0** |

### Tool Calling

| Parameter | Value |
|---|---|
| `temperature` | **0.6** |
| `top_p` | **0.95** |

### Additional Settings

| Setting | Value | Notes |
|---|---|---|
| `max_new_tokens` | **32,768 – 262,144** | Standard prompts; max 1M |
| `min_p` | **0.01** | Used in llama-server deployment |
| `--prio` | **2** or **3** | Priority scheduling in llama.cpp |
| `--seed` | **3407** | Reproducibility |
| `--special` | (flag) | Needed to see reasoning tokens (`<think)>` / `</think)>`) |
| `--verbose-prompt` | (flag) | Needed to see `<think)>` since it's prepended |

### Quick Reference — One-Liners

**General chat/instruction:**
```
temperature=1.0, top_p=1.0
```

**Tool calling:**
```
temperature=0.6, top_p=0.95
```

## Hardware Requirements (Total Memory: RAM + VRAM, or Unified Memory)

| Variant | 4-bit | 8-bit | Notes |
|---|---|---|---|
| **Nano-4B** | ~3 GB | ~5 GB | Fits virtually any device |
| **Nano-30B-A3B** | ~24 GB | ~36 GB | Fits 24GB Mac / RTX device |

> **Architecture note:** Some GGUF quants end up similar in size because the model architecture has dimensions not divisible by 128, so parts can't be quantized to lower bits.

### Quantization Recommendations

- **Nano-4B**: Use **Q8_0** (8-bit) for near full precision — only 5GB.
- **Nano-30B-A3B**: Use **UD-Q4_K_XL** (Dynamic 4-bit) — fits 24GB devices.

## Context Window Management

| Setting | Value |
|---|---|
| Native max context | **1M tokens** |
| Default context | **262,144** (256K) |
| Recommended starting context | **16,384** (16K) or **32,768** (32K) |

> ⚠️ Setting context to 1M may trigger **CUDA OOM and crash** — that's why the default is 262,144. Increase gradually as your RAM/VRAM allows.

### NoPE — No Positional Embeddings

Because Nemotron 3 was trained with **NoPE** (No Positional Embeddings):
- You only need to change `max_position_embeddings` to extend context.
- **YaRN is NOT needed** for context extension.
- The model doesn't use explicit positional embeddings.

## Chat Template Format

Nemotron 3 uses the standard ChatML-style format:

```
<|im_start|>system\n<|im_end|>\n
<|im_start|>user\n{message}<|im_end|>\n
<|im_start|>assistant\n<think)>{reasoning}</think)>{answer}<|im_end|>\n
```

### Reasoning Tokens

- `<think)>` = token ID **12**
- `</think)>` = token ID **13**
- Use `--special` in llama.cpp to see these tokens.
- Use `--verbose-prompt` to see `<think)>` since it's prepended to assistant responses.

## Critical Notes

- The model outputs a reasoning channel (`<think)>..._think)>`) before the final answer in most cases.
- Increase `max_new_tokens` for deep reasoning or long-form generation as RAM/VRAM allows.
- For fine-tuning: use at least **75% reasoning** and **25% non-reasoning** data to retain reasoning capabilities.
- The router layer is disabled by default in Unsloth fine-tuning for MoE models.

## Example llama.cpp Commands

### Nano-4B — General Instruction

```bash
./llama.cpp/llama-cli \
    -hf unsloth/NVIDIA-Nemotron-3-Nano-4B-GGUF:Q8_0 \
    --ctx-size 16384 \
    --temp 1.0 --top-p 1.0
```

### Nano-4B — Tool Calling

```bash
./llama.cpp/llama-cli \
    -hf unsloth/NVIDIA-Nemotron-3-Nano-4B-GGUF:Q8_0 \
    --ctx-size 32768 \
    --temp 0.6 --top-p 0.95
```

### Nano-4B — Conversation Mode (Local GGUF)

```bash
./llama.cpp/llama-cli \
    --model unsloth/NVIDIA-Nemotron-3-Nano-4B-GGUF/NVIDIA-Nemotron-3-Nano-4B-Q8_0.gguf \
    --ctx-size 16384 \
    --seed 3407 \
    --prio 2 \
    --temp 0.6 \
    --top-p 0.95
```

### Nano-30B-A3B — General Instruction

```bash
./llama.cpp/llama-cli \
    -hf unsloth/Nemotron-3-Nano-30B-A3B-GGUF:UD-Q4_K_XL \
    --ctx-size 32768 \
    --temp 1.0 --top-p 1.0
```

### Nano-30B-A3B — Tool Calling

```bash
./llama.cpp/llama-cli \
    -hf unsloth/Nemotron-3-Nano-30B-A3B-GGUF:UD-Q4_K_XL \
    --ctx-size 32768 \
    --temp 0.6 --top-p 0.95
```

### Nano-30B-A3B — Conversation Mode (Local GGUF)

```bash
./llama.cpp/llama-cli \
    --model unsloth/Nemotron-3-Nano-30B-A3B-GGUF/Nemotron-3-Nano-30B-A3B-UD-Q4_K_XL.gguf \
    --ctx-size 16384 \
    --seed 3407 \
    --prio 2 \
    --temp 0.6 \
    --top-p 0.95
```

### llama-server Deployment (Nano-30B-A3B)

```bash
./llama.cpp/llama-server \
    --model unsloth/Nemotron-3-Nano-30B-A3B-GGUF/Nemotron-3-Nano-30B-A3B-UD-Q4_K_XL.gguf \
    --alias "unsloth/Nemotron-3-Nano-30B-A3B" \
    --prio 3 \
    --min_p 0.01 \
    --temp 0.6 --top-p 0.95 \
    --ctx-size 16384 \
    --port 8001
```

### OpenAI-Compatible Client

After deploying via `llama-server`:

```python
from openai import OpenAI

openai_client = OpenAI(
    base_url = "http://127.0.0.1:8001/v1",
    api_key = "sk-no-key-required",
)

completion = openai_client.chat.completions.create(
    model = "unsloth/Nemotron-3-Nano-30B-A3B",
    messages = [{"role": "user", "content": "What is 2+2?"}],
)

print(completion.choices[0].message.content)
```

## Sources

- Unsloth NVIDIA Nemotron 3 Nano Documentation (How to Run)
- GGUF collections:
  - <https://huggingface.co/unsloth/NVIDIA-Nemotron-3-Nano-4B-GGUF>
  - <https://huggingface.co/unsloth/Nemotron-3-Nano-30B-A3B-GGUF>