VuongKB/reference/nemotron-3-nano-running-parameters.md

created, id, keywords, links, modified, summary, tags, title

created	id	keywords	links	modified	summary	tags	title
2026-04-09T18:05:28+00:00	4aec42e4-21ba-4279-b41b-58a8ebfefd73	Nemotron 3 NVIDIA Nano-4B Nano-30B-A3B temperature top_p min_p GGUF quantization UD-Q4_K_XL Q8_0 NoPE 1M context llama.cpp llama-server MoE tool calling reasoning tokens		2026-04-09T18:05:28+00:00	Optimal running parameters for NVIDIA's Nemotron 3 Nano family (4B and 30B-A3B), covering inference settings for general chat vs tool calling, hardware requirements, NoPE context extension, reasoning token handling, and llama.cpp example commands.	reference nemotron-3 nvidia llm inference llama-cpp local-ai	NVIDIA Nemotron 3 Nano — Optimal Running Parameters

NVIDIA Nemotron 3 Nano — Optimal Running Parameters

A reference for optimally running NVIDIA's Nemotron 3 Nano family of hybrid MoE models locally.

Model Variants

Variant	Architecture	Context	Best Fit
Nano-4B	MoE	Up to 1M tokens	Lightweight coding, math, agentic tasks; fits 5GB devices
Nano-30B-A3B	MoE (3B active)	Up to 1M tokens	Best performing for size; fits 24GB devices
Super-120B-A12B	MoE (12B active)	Up to 1M tokens	High-end workstation / server (separate guide)

All Nemotron 3 models support a 1M-token context window.

Recommended Inference Parameters

NVIDIA recommends two parameter profiles depending on use-case:

General Chat / Instruction

Parameter	Value
temperature	1.0
top_p	1.0

Tool Calling

Parameter	Value
temperature	0.6
top_p	0.95

Additional Settings

Setting	Value	Notes
max_new_tokens	32,768 – 262,144	Standard prompts; max 1M
min_p	0.01	Used in llama-server deployment
--prio	2 or 3	Priority scheduling in llama.cpp
--seed	3407	Reproducibility
--special	(flag)	Needed to see reasoning tokens (<think)> / </think)>)
--verbose-prompt	(flag)	Needed to see <think)> since it's prepended

Quick Reference — One-Liners

General chat/instruction:

temperature=1.0, top_p=1.0

Tool calling:

temperature=0.6, top_p=0.95

Hardware Requirements (Total Memory: RAM + VRAM, or Unified Memory)

Variant	4-bit	8-bit	Notes
Nano-4B	~3 GB	~5 GB	Fits virtually any device
Nano-30B-A3B	~24 GB	~36 GB	Fits 24GB Mac / RTX device

Architecture note: Some GGUF quants end up similar in size because the model architecture has dimensions not divisible by 128, so parts can't be quantized to lower bits.

Quantization Recommendations

• Nano-4B: Use Q8_0 (8-bit) for near full precision — only 5GB.
• Nano-30B-A3B: Use UD-Q4_K_XL (Dynamic 4-bit) — fits 24GB devices.

Context Window Management

Setting	Value
Native max context	1M tokens
Default context	262,144 (256K)
Recommended starting context	16,384 (16K) or 32,768 (32K)

⚠️ Setting context to 1M may trigger CUDA OOM and crash — that's why the default is 262,144. Increase gradually as your RAM/VRAM allows.

NoPE — No Positional Embeddings

Because Nemotron 3 was trained with NoPE (No Positional Embeddings):

• You only need to change max_position_embeddings to extend context.
• YaRN is NOT needed for context extension.
• The model doesn't use explicit positional embeddings.

Chat Template Format

Nemotron 3 uses the standard ChatML-style format:

<|im_start|>system\n<|im_end|>\n
<|im_start|>user\n{message}<|im_end|>\n
<|im_start|>assistant\n<think)>{reasoning}</think)>{answer}<|im_end|>\n

Reasoning Tokens

• <think)> = token ID 12
• </think)> = token ID 13
• Use --special in llama.cpp to see these tokens.
• Use --verbose-prompt to see <think)> since it's prepended to assistant responses.

Critical Notes

• The model outputs a reasoning channel (<think)>..._think)>) before the final answer in most cases.
• Increase max_new_tokens for deep reasoning or long-form generation as RAM/VRAM allows.
• For fine-tuning: use at least 75% reasoning and 25% non-reasoning data to retain reasoning capabilities.
• The router layer is disabled by default in Unsloth fine-tuning for MoE models.

Example llama.cpp Commands

Nano-4B — General Instruction

./llama.cpp/llama-cli \
    -hf unsloth/NVIDIA-Nemotron-3-Nano-4B-GGUF:Q8_0 \
    --ctx-size 16384 \
    --temp 1.0 --top-p 1.0

Nano-4B — Tool Calling

./llama.cpp/llama-cli \
    -hf unsloth/NVIDIA-Nemotron-3-Nano-4B-GGUF:Q8_0 \
    --ctx-size 32768 \
    --temp 0.6 --top-p 0.95

Nano-4B — Conversation Mode (Local GGUF)

./llama.cpp/llama-cli \
    --model unsloth/NVIDIA-Nemotron-3-Nano-4B-GGUF/NVIDIA-Nemotron-3-Nano-4B-Q8_0.gguf \
    --ctx-size 16384 \
    --seed 3407 \
    --prio 2 \
    --temp 0.6 \
    --top-p 0.95

Nano-30B-A3B — General Instruction

./llama.cpp/llama-cli \
    -hf unsloth/Nemotron-3-Nano-30B-A3B-GGUF:UD-Q4_K_XL \
    --ctx-size 32768 \
    --temp 1.0 --top-p 1.0

Nano-30B-A3B — Tool Calling

./llama.cpp/llama-cli \
    -hf unsloth/Nemotron-3-Nano-30B-A3B-GGUF:UD-Q4_K_XL \
    --ctx-size 32768 \
    --temp 0.6 --top-p 0.95

Nano-30B-A3B — Conversation Mode (Local GGUF)

./llama.cpp/llama-cli \
    --model unsloth/Nemotron-3-Nano-30B-A3B-GGUF/Nemotron-3-Nano-30B-A3B-UD-Q4_K_XL.gguf \
    --ctx-size 16384 \
    --seed 3407 \
    --prio 2 \
    --temp 0.6 \
    --top-p 0.95

llama-server Deployment (Nano-30B-A3B)

./llama.cpp/llama-server \
    --model unsloth/Nemotron-3-Nano-30B-A3B-GGUF/Nemotron-3-Nano-30B-A3B-UD-Q4_K_XL.gguf \
    --alias "unsloth/Nemotron-3-Nano-30B-A3B" \
    --prio 3 \
    --min_p 0.01 \
    --temp 0.6 --top-p 0.95 \
    --ctx-size 16384 \
    --port 8001

OpenAI-Compatible Client

After deploying via llama-server:

from openai import OpenAI

openai_client = OpenAI(
    base_url = "http://127.0.0.1:8001/v1",
    api_key = "sk-no-key-required",
)

completion = openai_client.chat.completions.create(
    model = "unsloth/Nemotron-3-Nano-30B-A3B",
    messages = [{"role": "user", "content": "What is 2+2?"}],
)

print(completion.choices[0].message.content)

Sources

• Unsloth NVIDIA Nemotron 3 Nano Documentation (How to Run)
• GGUF collections:
  • https://huggingface.co/unsloth/NVIDIA-Nemotron-3-Nano-4B-GGUF
  • https://huggingface.co/unsloth/Nemotron-3-Nano-30B-A3B-GGUF