diff --git a/reference/nemotron-3-nano-running-parameters.md b/reference/nemotron-3-nano-running-parameters.md new file mode 100644 index 0000000..98a4537 --- /dev/null +++ b/reference/nemotron-3-nano-running-parameters.md @@ -0,0 +1,249 @@ +--- +created: '2026-04-09T18:05:28+00:00' +id: 4aec42e4-21ba-4279-b41b-58a8ebfefd73 +keywords: +- Nemotron 3 +- NVIDIA +- Nano-4B +- Nano-30B-A3B +- temperature +- top_p +- min_p +- GGUF +- quantization +- UD-Q4_K_XL +- Q8_0 +- NoPE +- 1M context +- llama.cpp +- llama-server +- MoE +- tool calling +- reasoning tokens +links: [] +modified: '2026-04-09T18:05:28+00:00' +summary: Optimal running parameters for NVIDIA's Nemotron 3 Nano family (4B and 30B-A3B), + covering inference settings for general chat vs tool calling, hardware requirements, + NoPE context extension, reasoning token handling, and llama.cpp example commands. +tags: +- reference +- nemotron-3 +- nvidia +- llm +- inference +- llama-cpp +- local-ai +title: NVIDIA Nemotron 3 Nano — Optimal Running Parameters +--- + +# NVIDIA Nemotron 3 Nano — Optimal Running Parameters + +A reference for optimally running NVIDIA's Nemotron 3 Nano family of hybrid MoE models locally. + +## Model Variants + +| Variant | Architecture | Context | Best Fit | +|---|---|---|---| +| **Nano-4B** | MoE | Up to **1M tokens** | Lightweight coding, math, agentic tasks; fits 5GB devices | +| **Nano-30B-A3B** | MoE (3B active) | Up to **1M tokens** | Best performing for size; fits 24GB devices | +| Super-120B-A12B | MoE (12B active) | Up to **1M tokens** | High-end workstation / server (separate guide) | + +> All Nemotron 3 models support a **1M-token context window**. + +## Recommended Inference Parameters + +NVIDIA recommends two parameter profiles depending on use-case: + +### General Chat / Instruction + +| Parameter | Value | +|---|---| +| `temperature` | **1.0** | +| `top_p` | **1.0** | + +### Tool Calling + +| Parameter | Value | +|---|---| +| `temperature` | **0.6** | +| `top_p` | **0.95** | + +### Additional Settings + +| Setting | Value | Notes | +|---|---|---| +| `max_new_tokens` | **32,768 – 262,144** | Standard prompts; max 1M | +| `min_p` | **0.01** | Used in llama-server deployment | +| `--prio` | **2** or **3** | Priority scheduling in llama.cpp | +| `--seed` | **3407** | Reproducibility | +| `--special` | (flag) | Needed to see reasoning tokens (`` / ``) | +| `--verbose-prompt` | (flag) | Needed to see `` since it's prepended | + +### Quick Reference — One-Liners + +**General chat/instruction:** +``` +temperature=1.0, top_p=1.0 +``` + +**Tool calling:** +``` +temperature=0.6, top_p=0.95 +``` + +## Hardware Requirements (Total Memory: RAM + VRAM, or Unified Memory) + +| Variant | 4-bit | 8-bit | Notes | +|---|---|---|---| +| **Nano-4B** | ~3 GB | ~5 GB | Fits virtually any device | +| **Nano-30B-A3B** | ~24 GB | ~36 GB | Fits 24GB Mac / RTX device | + +> **Architecture note:** Some GGUF quants end up similar in size because the model architecture has dimensions not divisible by 128, so parts can't be quantized to lower bits. + +### Quantization Recommendations + +- **Nano-4B**: Use **Q8_0** (8-bit) for near full precision — only 5GB. +- **Nano-30B-A3B**: Use **UD-Q4_K_XL** (Dynamic 4-bit) — fits 24GB devices. + +## Context Window Management + +| Setting | Value | +|---|---| +| Native max context | **1M tokens** | +| Default context | **262,144** (256K) | +| Recommended starting context | **16,384** (16K) or **32,768** (32K) | + +> ⚠️ Setting context to 1M may trigger **CUDA OOM and crash** — that's why the default is 262,144. Increase gradually as your RAM/VRAM allows. + +### NoPE — No Positional Embeddings + +Because Nemotron 3 was trained with **NoPE** (No Positional Embeddings): +- You only need to change `max_position_embeddings` to extend context. +- **YaRN is NOT needed** for context extension. +- The model doesn't use explicit positional embeddings. + +## Chat Template Format + +Nemotron 3 uses the standard ChatML-style format: + +``` +<|im_start|>system\n<|im_end|>\n +<|im_start|>user\n{message}<|im_end|>\n +<|im_start|>assistant\n{reasoning}{answer}<|im_end|>\n +``` + +### Reasoning Tokens + +- `` = token ID **12** +- `` = token ID **13** +- Use `--special` in llama.cpp to see these tokens. +- Use `--verbose-prompt` to see `` since it's prepended to assistant responses. + +## Critical Notes + +- The model outputs a reasoning channel (`..._think)>`) before the final answer in most cases. +- Increase `max_new_tokens` for deep reasoning or long-form generation as RAM/VRAM allows. +- For fine-tuning: use at least **75% reasoning** and **25% non-reasoning** data to retain reasoning capabilities. +- The router layer is disabled by default in Unsloth fine-tuning for MoE models. + +## Example llama.cpp Commands + +### Nano-4B — General Instruction + +```bash +./llama.cpp/llama-cli \ + -hf unsloth/NVIDIA-Nemotron-3-Nano-4B-GGUF:Q8_0 \ + --ctx-size 16384 \ + --temp 1.0 --top-p 1.0 +``` + +### Nano-4B — Tool Calling + +```bash +./llama.cpp/llama-cli \ + -hf unsloth/NVIDIA-Nemotron-3-Nano-4B-GGUF:Q8_0 \ + --ctx-size 32768 \ + --temp 0.6 --top-p 0.95 +``` + +### Nano-4B — Conversation Mode (Local GGUF) + +```bash +./llama.cpp/llama-cli \ + --model unsloth/NVIDIA-Nemotron-3-Nano-4B-GGUF/NVIDIA-Nemotron-3-Nano-4B-Q8_0.gguf \ + --ctx-size 16384 \ + --seed 3407 \ + --prio 2 \ + --temp 0.6 \ + --top-p 0.95 +``` + +### Nano-30B-A3B — General Instruction + +```bash +./llama.cpp/llama-cli \ + -hf unsloth/Nemotron-3-Nano-30B-A3B-GGUF:UD-Q4_K_XL \ + --ctx-size 32768 \ + --temp 1.0 --top-p 1.0 +``` + +### Nano-30B-A3B — Tool Calling + +```bash +./llama.cpp/llama-cli \ + -hf unsloth/Nemotron-3-Nano-30B-A3B-GGUF:UD-Q4_K_XL \ + --ctx-size 32768 \ + --temp 0.6 --top-p 0.95 +``` + +### Nano-30B-A3B — Conversation Mode (Local GGUF) + +```bash +./llama.cpp/llama-cli \ + --model unsloth/Nemotron-3-Nano-30B-A3B-GGUF/Nemotron-3-Nano-30B-A3B-UD-Q4_K_XL.gguf \ + --ctx-size 16384 \ + --seed 3407 \ + --prio 2 \ + --temp 0.6 \ + --top-p 0.95 +``` + +### llama-server Deployment (Nano-30B-A3B) + +```bash +./llama.cpp/llama-server \ + --model unsloth/Nemotron-3-Nano-30B-A3B-GGUF/Nemotron-3-Nano-30B-A3B-UD-Q4_K_XL.gguf \ + --alias "unsloth/Nemotron-3-Nano-30B-A3B" \ + --prio 3 \ + --min_p 0.01 \ + --temp 0.6 --top-p 0.95 \ + --ctx-size 16384 \ + --port 8001 +``` + +### OpenAI-Compatible Client + +After deploying via `llama-server`: + +```python +from openai import OpenAI + +openai_client = OpenAI( + base_url = "http://127.0.0.1:8001/v1", + api_key = "sk-no-key-required", +) + +completion = openai_client.chat.completions.create( + model = "unsloth/Nemotron-3-Nano-30B-A3B", + messages = [{"role": "user", "content": "What is 2+2?"}], +) + +print(completion.choices[0].message.content) +``` + +## Sources + +- Unsloth NVIDIA Nemotron 3 Nano Documentation (How to Run) +- GGUF collections: + - + - \ No newline at end of file