Create: reference/nemotron-3-nano-running-parameters.md

2026-04-09 11:05:28 -07:00
parent e21342084f
commit a5a550dc50
1 changed files with 249 additions and 0 deletions
@@ -0,0 +1,249 @@
 ---
 created: '2026-04-09T18:05:28+00:00'
 id: 4aec42e4-21ba-4279-b41b-58a8ebfefd73
 keywords:
 - Nemotron 3
 - NVIDIA
 - Nano-4B
 - Nano-30B-A3B
 - temperature
 - top_p
 - min_p
 - GGUF
 - quantization
 - UD-Q4_K_XL
 - Q8_0
 - NoPE
 - 1M context
 - llama.cpp
 - llama-server
 - MoE
 - tool calling
 - reasoning tokens
 links: []
 modified: '2026-04-09T18:05:28+00:00'
 summary: Optimal running parameters for NVIDIA's Nemotron 3 Nano family (4B and 30B-A3B),
  covering inference settings for general chat vs tool calling, hardware requirements,
  NoPE context extension, reasoning token handling, and llama.cpp example commands.
 tags:
 - reference
 - nemotron-3
 - nvidia
 - llm
 - inference
 - llama-cpp
 - local-ai
 title: NVIDIA Nemotron 3 Nano — Optimal Running Parameters
 ---
 # NVIDIA Nemotron 3 Nano — Optimal Running Parameters
 A reference for optimally running NVIDIA's Nemotron 3 Nano family of hybrid MoE models locally.
 ## Model Variants
 | Variant | Architecture | Context | Best Fit |
 |---|---|---|---|
 | **Nano-4B** | MoE | Up to **1M tokens** | Lightweight coding, math, agentic tasks; fits 5GB devices |
 | **Nano-30B-A3B** | MoE (3B active) | Up to **1M tokens** | Best performing for size; fits 24GB devices |
 | Super-120B-A12B | MoE (12B active) | Up to **1M tokens** | High-end workstation / server (separate guide) |
 > All Nemotron 3 models support a **1M-token context window**.
 ## Recommended Inference Parameters
 NVIDIA recommends two parameter profiles depending on use-case:
 ### General Chat / Instruction
 | Parameter | Value |
 |---|---|
 | `temperature` | **1.0** |
 | `top_p` | **1.0** |
 ### Tool Calling
 | Parameter | Value |
 |---|---|
 | `temperature` | **0.6** |
 | `top_p` | **0.95** |
 ### Additional Settings
 | Setting | Value | Notes |
 |---|---|---|
 | `max_new_tokens` | **32,768 – 262,144** | Standard prompts; max 1M |
 | `min_p` | **0.01** | Used in llama-server deployment |
 | `--prio` | **2** or **3** | Priority scheduling in llama.cpp |
 | `--seed` | **3407** | Reproducibility |
 | `--special` | (flag) | Needed to see reasoning tokens (`<think)>` / `</think)>`) |
 | `--verbose-prompt` | (flag) | Needed to see `<think)>` since it's prepended |
 ### Quick Reference — One-Liners
 **General chat/instruction:**
 ```
 temperature=1.0, top_p=1.0
 ```
 **Tool calling:**
 ```
 temperature=0.6, top_p=0.95
 ```
 ## Hardware Requirements (Total Memory: RAM + VRAM, or Unified Memory)
 | Variant | 4-bit | 8-bit | Notes |
 |---|---|---|---|
 | **Nano-4B** | ~3 GB | ~5 GB | Fits virtually any device |
 | **Nano-30B-A3B** | ~24 GB | ~36 GB | Fits 24GB Mac / RTX device |
 > **Architecture note:** Some GGUF quants end up similar in size because the model architecture has dimensions not divisible by 128, so parts can't be quantized to lower bits.
 ### Quantization Recommendations
 - **Nano-4B**: Use **Q8_0** (8-bit) for near full precision — only 5GB.
 - **Nano-30B-A3B**: Use **UD-Q4_K_XL** (Dynamic 4-bit) — fits 24GB devices.
 ## Context Window Management
 | Setting | Value |
 |---|---|
 | Native max context | **1M tokens** |
 | Default context | **262,144** (256K) |
 | Recommended starting context | **16,384** (16K) or **32,768** (32K) |
 > ⚠️ Setting context to 1M may trigger **CUDA OOM and crash** — that's why the default is 262,144. Increase gradually as your RAM/VRAM allows.
 ### NoPE — No Positional Embeddings
 Because Nemotron 3 was trained with **NoPE** (No Positional Embeddings):
 - You only need to change `max_position_embeddings` to extend context.
 - **YaRN is NOT needed** for context extension.
 - The model doesn't use explicit positional embeddings.
 ## Chat Template Format
 Nemotron 3 uses the standard ChatML-style format:
 ```
 <|im_start|>system\n<|im_end|>\n
 <|im_start|>user\n{message}<|im_end|>\n
 <|im_start|>assistant\n<think)>{reasoning}</think)>{answer}<|im_end|>\n
 ```
 ### Reasoning Tokens
 - `<think)>` = token ID **12**
 - `</think)>` = token ID **13**
 - Use `--special` in llama.cpp to see these tokens.
 - Use `--verbose-prompt` to see `<think)>` since it's prepended to assistant responses.
 ## Critical Notes
 - The model outputs a reasoning channel (`<think)>..._think)>`) before the final answer in most cases.
 - Increase `max_new_tokens` for deep reasoning or long-form generation as RAM/VRAM allows.
 - For fine-tuning: use at least **75% reasoning** and **25% non-reasoning** data to retain reasoning capabilities.
 - The router layer is disabled by default in Unsloth fine-tuning for MoE models.
 ## Example llama.cpp Commands
 ### Nano-4B — General Instruction
 ```bash
 ./llama.cpp/llama-cli \
    -hf unsloth/NVIDIA-Nemotron-3-Nano-4B-GGUF:Q8_0 \
    --ctx-size 16384 \
    --temp 1.0 --top-p 1.0
 ```
 ### Nano-4B — Tool Calling
 ```bash
 ./llama.cpp/llama-cli \
    -hf unsloth/NVIDIA-Nemotron-3-Nano-4B-GGUF:Q8_0 \
    --ctx-size 32768 \
    --temp 0.6 --top-p 0.95
 ```
 ### Nano-4B — Conversation Mode (Local GGUF)
 ```bash
 ./llama.cpp/llama-cli \
    --model unsloth/NVIDIA-Nemotron-3-Nano-4B-GGUF/NVIDIA-Nemotron-3-Nano-4B-Q8_0.gguf \
    --ctx-size 16384 \
    --seed 3407 \
    --prio 2 \
    --temp 0.6 \
    --top-p 0.95
 ```
 ### Nano-30B-A3B — General Instruction
 ```bash
 ./llama.cpp/llama-cli \
    -hf unsloth/Nemotron-3-Nano-30B-A3B-GGUF:UD-Q4_K_XL \
    --ctx-size 32768 \
    --temp 1.0 --top-p 1.0
 ```
 ### Nano-30B-A3B — Tool Calling
 ```bash
 ./llama.cpp/llama-cli \
    -hf unsloth/Nemotron-3-Nano-30B-A3B-GGUF:UD-Q4_K_XL \
    --ctx-size 32768 \
    --temp 0.6 --top-p 0.95
 ```
 ### Nano-30B-A3B — Conversation Mode (Local GGUF)
 ```bash
 ./llama.cpp/llama-cli \
    --model unsloth/Nemotron-3-Nano-30B-A3B-GGUF/Nemotron-3-Nano-30B-A3B-UD-Q4_K_XL.gguf \
    --ctx-size 16384 \
    --seed 3407 \
    --prio 2 \
    --temp 0.6 \
    --top-p 0.95
 ```
 ### llama-server Deployment (Nano-30B-A3B)
 ```bash
 ./llama.cpp/llama-server \
    --model unsloth/Nemotron-3-Nano-30B-A3B-GGUF/Nemotron-3-Nano-30B-A3B-UD-Q4_K_XL.gguf \
    --alias "unsloth/Nemotron-3-Nano-30B-A3B" \
    --prio 3 \
    --min_p 0.01 \
    --temp 0.6 --top-p 0.95 \
    --ctx-size 16384 \
    --port 8001
 ```
 ### OpenAI-Compatible Client
 After deploying via `llama-server`:
 ```python
 from openai import OpenAI
 openai_client = OpenAI(
    base_url = "http://127.0.0.1:8001/v1",
    api_key = "sk-no-key-required",
 )
 completion = openai_client.chat.completions.create(
    model = "unsloth/Nemotron-3-Nano-30B-A3B",
    messages = [{"role": "user", "content": "What is 2+2?"}],
 )
 print(completion.choices[0].message.content)
 ```
 ## Sources
 - Unsloth NVIDIA Nemotron 3 Nano Documentation (How to Run)
 - GGUF collections:
  - <https://huggingface.co/unsloth/NVIDIA-Nemotron-3-Nano-4B-GGUF>
  - <https://huggingface.co/unsloth/Nemotron-3-Nano-30B-A3B-GGUF>