Create: reference/nemotron-3-nano-running-parameters.md
This commit is contained in:
Atlas
@@ -0,0 +1,249 @@
|
||||
---
|
||||
created: '2026-04-09T18:05:28+00:00'
|
||||
id: 4aec42e4-21ba-4279-b41b-58a8ebfefd73
|
||||
keywords:
|
||||
- Nemotron 3
|
||||
- NVIDIA
|
||||
- Nano-4B
|
||||
- Nano-30B-A3B
|
||||
- temperature
|
||||
- top_p
|
||||
- min_p
|
||||
- GGUF
|
||||
- quantization
|
||||
- UD-Q4_K_XL
|
||||
- Q8_0
|
||||
- NoPE
|
||||
- 1M context
|
||||
- llama.cpp
|
||||
- llama-server
|
||||
- MoE
|
||||
- tool calling
|
||||
- reasoning tokens
|
||||
links: []
|
||||
modified: '2026-04-09T18:05:28+00:00'
|
||||
summary: Optimal running parameters for NVIDIA's Nemotron 3 Nano family (4B and 30B-A3B),
|
||||
covering inference settings for general chat vs tool calling, hardware requirements,
|
||||
NoPE context extension, reasoning token handling, and llama.cpp example commands.
|
||||
tags:
|
||||
- reference
|
||||
- nemotron-3
|
||||
- nvidia
|
||||
- llm
|
||||
- inference
|
||||
- llama-cpp
|
||||
- local-ai
|
||||
title: NVIDIA Nemotron 3 Nano — Optimal Running Parameters
|
||||
---
|
||||
|
||||
# NVIDIA Nemotron 3 Nano — Optimal Running Parameters
|
||||
|
||||
A reference for optimally running NVIDIA's Nemotron 3 Nano family of hybrid MoE models locally.
|
||||
|
||||
## Model Variants
|
||||
|
||||
| Variant | Architecture | Context | Best Fit |
|
||||
|---|---|---|---|
|
||||
| **Nano-4B** | MoE | Up to **1M tokens** | Lightweight coding, math, agentic tasks; fits 5GB devices |
|
||||
| **Nano-30B-A3B** | MoE (3B active) | Up to **1M tokens** | Best performing for size; fits 24GB devices |
|
||||
| Super-120B-A12B | MoE (12B active) | Up to **1M tokens** | High-end workstation / server (separate guide) |
|
||||
|
||||
> All Nemotron 3 models support a **1M-token context window**.
|
||||
|
||||
## Recommended Inference Parameters
|
||||
|
||||
NVIDIA recommends two parameter profiles depending on use-case:
|
||||
|
||||
### General Chat / Instruction
|
||||
|
||||
| Parameter | Value |
|
||||
|---|---|
|
||||
| `temperature` | **1.0** |
|
||||
| `top_p` | **1.0** |
|
||||
|
||||
### Tool Calling
|
||||
|
||||
| Parameter | Value |
|
||||
|---|---|
|
||||
| `temperature` | **0.6** |
|
||||
| `top_p` | **0.95** |
|
||||
|
||||
### Additional Settings
|
||||
|
||||
| Setting | Value | Notes |
|
||||
|---|---|---|
|
||||
| `max_new_tokens` | **32,768 – 262,144** | Standard prompts; max 1M |
|
||||
| `min_p` | **0.01** | Used in llama-server deployment |
|
||||
| `--prio` | **2** or **3** | Priority scheduling in llama.cpp |
|
||||
| `--seed` | **3407** | Reproducibility |
|
||||
| `--special` | (flag) | Needed to see reasoning tokens (`<think)>` / `</think)>`) |
|
||||
| `--verbose-prompt` | (flag) | Needed to see `<think)>` since it's prepended |
|
||||
|
||||
### Quick Reference — One-Liners
|
||||
|
||||
**General chat/instruction:**
|
||||
```
|
||||
temperature=1.0, top_p=1.0
|
||||
```
|
||||
|
||||
**Tool calling:**
|
||||
```
|
||||
temperature=0.6, top_p=0.95
|
||||
```
|
||||
|
||||
## Hardware Requirements (Total Memory: RAM + VRAM, or Unified Memory)
|
||||
|
||||
| Variant | 4-bit | 8-bit | Notes |
|
||||
|---|---|---|---|
|
||||
| **Nano-4B** | ~3 GB | ~5 GB | Fits virtually any device |
|
||||
| **Nano-30B-A3B** | ~24 GB | ~36 GB | Fits 24GB Mac / RTX device |
|
||||
|
||||
> **Architecture note:** Some GGUF quants end up similar in size because the model architecture has dimensions not divisible by 128, so parts can't be quantized to lower bits.
|
||||
|
||||
### Quantization Recommendations
|
||||
|
||||
- **Nano-4B**: Use **Q8_0** (8-bit) for near full precision — only 5GB.
|
||||
- **Nano-30B-A3B**: Use **UD-Q4_K_XL** (Dynamic 4-bit) — fits 24GB devices.
|
||||
|
||||
## Context Window Management
|
||||
|
||||
| Setting | Value |
|
||||
|---|---|
|
||||
| Native max context | **1M tokens** |
|
||||
| Default context | **262,144** (256K) |
|
||||
| Recommended starting context | **16,384** (16K) or **32,768** (32K) |
|
||||
|
||||
> ⚠️ Setting context to 1M may trigger **CUDA OOM and crash** — that's why the default is 262,144. Increase gradually as your RAM/VRAM allows.
|
||||
|
||||
### NoPE — No Positional Embeddings
|
||||
|
||||
Because Nemotron 3 was trained with **NoPE** (No Positional Embeddings):
|
||||
- You only need to change `max_position_embeddings` to extend context.
|
||||
- **YaRN is NOT needed** for context extension.
|
||||
- The model doesn't use explicit positional embeddings.
|
||||
|
||||
## Chat Template Format
|
||||
|
||||
Nemotron 3 uses the standard ChatML-style format:
|
||||
|
||||
```
|
||||
<|im_start|>system\n<|im_end|>\n
|
||||
<|im_start|>user\n{message}<|im_end|>\n
|
||||
<|im_start|>assistant\n<think)>{reasoning}</think)>{answer}<|im_end|>\n
|
||||
```
|
||||
|
||||
### Reasoning Tokens
|
||||
|
||||
- `<think)>` = token ID **12**
|
||||
- `</think)>` = token ID **13**
|
||||
- Use `--special` in llama.cpp to see these tokens.
|
||||
- Use `--verbose-prompt` to see `<think)>` since it's prepended to assistant responses.
|
||||
|
||||
## Critical Notes
|
||||
|
||||
- The model outputs a reasoning channel (`<think)>..._think)>`) before the final answer in most cases.
|
||||
- Increase `max_new_tokens` for deep reasoning or long-form generation as RAM/VRAM allows.
|
||||
- For fine-tuning: use at least **75% reasoning** and **25% non-reasoning** data to retain reasoning capabilities.
|
||||
- The router layer is disabled by default in Unsloth fine-tuning for MoE models.
|
||||
|
||||
## Example llama.cpp Commands
|
||||
|
||||
### Nano-4B — General Instruction
|
||||
|
||||
```bash
|
||||
./llama.cpp/llama-cli \
|
||||
-hf unsloth/NVIDIA-Nemotron-3-Nano-4B-GGUF:Q8_0 \
|
||||
--ctx-size 16384 \
|
||||
--temp 1.0 --top-p 1.0
|
||||
```
|
||||
|
||||
### Nano-4B — Tool Calling
|
||||
|
||||
```bash
|
||||
./llama.cpp/llama-cli \
|
||||
-hf unsloth/NVIDIA-Nemotron-3-Nano-4B-GGUF:Q8_0 \
|
||||
--ctx-size 32768 \
|
||||
--temp 0.6 --top-p 0.95
|
||||
```
|
||||
|
||||
### Nano-4B — Conversation Mode (Local GGUF)
|
||||
|
||||
```bash
|
||||
./llama.cpp/llama-cli \
|
||||
--model unsloth/NVIDIA-Nemotron-3-Nano-4B-GGUF/NVIDIA-Nemotron-3-Nano-4B-Q8_0.gguf \
|
||||
--ctx-size 16384 \
|
||||
--seed 3407 \
|
||||
--prio 2 \
|
||||
--temp 0.6 \
|
||||
--top-p 0.95
|
||||
```
|
||||
|
||||
### Nano-30B-A3B — General Instruction
|
||||
|
||||
```bash
|
||||
./llama.cpp/llama-cli \
|
||||
-hf unsloth/Nemotron-3-Nano-30B-A3B-GGUF:UD-Q4_K_XL \
|
||||
--ctx-size 32768 \
|
||||
--temp 1.0 --top-p 1.0
|
||||
```
|
||||
|
||||
### Nano-30B-A3B — Tool Calling
|
||||
|
||||
```bash
|
||||
./llama.cpp/llama-cli \
|
||||
-hf unsloth/Nemotron-3-Nano-30B-A3B-GGUF:UD-Q4_K_XL \
|
||||
--ctx-size 32768 \
|
||||
--temp 0.6 --top-p 0.95
|
||||
```
|
||||
|
||||
### Nano-30B-A3B — Conversation Mode (Local GGUF)
|
||||
|
||||
```bash
|
||||
./llama.cpp/llama-cli \
|
||||
--model unsloth/Nemotron-3-Nano-30B-A3B-GGUF/Nemotron-3-Nano-30B-A3B-UD-Q4_K_XL.gguf \
|
||||
--ctx-size 16384 \
|
||||
--seed 3407 \
|
||||
--prio 2 \
|
||||
--temp 0.6 \
|
||||
--top-p 0.95
|
||||
```
|
||||
|
||||
### llama-server Deployment (Nano-30B-A3B)
|
||||
|
||||
```bash
|
||||
./llama.cpp/llama-server \
|
||||
--model unsloth/Nemotron-3-Nano-30B-A3B-GGUF/Nemotron-3-Nano-30B-A3B-UD-Q4_K_XL.gguf \
|
||||
--alias "unsloth/Nemotron-3-Nano-30B-A3B" \
|
||||
--prio 3 \
|
||||
--min_p 0.01 \
|
||||
--temp 0.6 --top-p 0.95 \
|
||||
--ctx-size 16384 \
|
||||
--port 8001
|
||||
```
|
||||
|
||||
### OpenAI-Compatible Client
|
||||
|
||||
After deploying via `llama-server`:
|
||||
|
||||
```python
|
||||
from openai import OpenAI
|
||||
|
||||
openai_client = OpenAI(
|
||||
base_url = "http://127.0.0.1:8001/v1",
|
||||
api_key = "sk-no-key-required",
|
||||
)
|
||||
|
||||
completion = openai_client.chat.completions.create(
|
||||
model = "unsloth/Nemotron-3-Nano-30B-A3B",
|
||||
messages = [{"role": "user", "content": "What is 2+2?"}],
|
||||
)
|
||||
|
||||
print(completion.choices[0].message.content)
|
||||
```
|
||||
|
||||
## Sources
|
||||
|
||||
- Unsloth NVIDIA Nemotron 3 Nano Documentation (How to Run)
|
||||
- GGUF collections:
|
||||
- <https://huggingface.co/unsloth/NVIDIA-Nemotron-3-Nano-4B-GGUF>
|
||||
- <https://huggingface.co/unsloth/Nemotron-3-Nano-30B-A3B-GGUF>
|
||||
Reference in New Issue
Block a user