VuongKB/reference/nemotron-3-super-running-parameters.md at 72f0fa67360fa1f4beb24960861843586092b9db

Files

T

Atlas f33a3a82df Update: reference/nemotron-3-super-running-parameters.md — Corrected EOS token leak fix: clarified that --stop flag does not exist in llama-server; stop sequences must be provided in the API payload's 'stop' array instead of as a command-line argument.

2026-04-09 18:48:14 -07:00

8.1 KiB

Raw Blame History

created, id, keywords, links, modified, summary, tags, title

created

keywords

links

modified

summary

Model Overview

Property	Value
Architecture	MoE — 120B total, 12B active parameters
Max Context	1M tokens (default 262,144)
Positional Embeddings	NoPE — no explicit positional embeddings; YaRN not needed
Reasoning Tokens	`<think)>` (token ID 12), `</think)>` (token ID 13)
Best For	Multi-agent AI, high-efficiency reasoning, coding, math, agentic workflows
Performance Tier	Leads its size class on AIME 2025, Terminal Bench, SWE-Bench Verified; highest throughput

Recommended Inference Parameters

NVIDIA recommends two parameter profiles:

General Chat / Instruction

Parameter	Value
`temperature`	1.0
`top_p`	1.0

Tool Calling

Parameter	Value
`temperature`	0.6
`top_p`	0.95

Additional Settings

Setting	Value	Notes
`max_new_tokens`	32,768 – 262,144	Standard prompts; max 1M
`min_p`	0.01	Used in llama-server deployment
`--prio`	2 or 3	Priority scheduling in llama.cpp
`--seed`	3407	Reproducibility
`--special`	(flag)	Needed to see reasoning tokens
`--verbose-prompt`	(flag)	Needed to see `<think)>` since it's prepended

Quick Reference — One-Liners

General chat/instruction:

temperature=1.0, top_p=1.0, min_p=0.01

Tool calling:

temperature=0.6, top_p=0.95, min_p=0.01

Hardware Requirements (Total Memory: RAM + VRAM, or Unified Memory)

Precision	Memory Required
2-bit (UD-Q2_K_XL)	~32–36 GB
4-bit (UD-Q4_K_XL)	~64–72 GB
8-bit	~128 GB
BF16	~240 GB

Rule of thumb: Total available memory should exceed the quantized model size. If it doesn't, llama.cpp can offload to SSD/HDD but inference will be slower.

Quantization Recommendations

Minimum recommended: UD-Q2_K_XL (Dynamic 2-bit) to balance size and accuracy.
Recommended starting point: UD-Q4_K_XL (Dynamic 4-bit) — fits 64–72GB devices.
Available in multiple formats: GGUF, NVFP4, FP8, BF16.

Context Window Management

Setting	Value
Native max context	1M tokens
Default context	262,144 (256K)
Recommended starting context	16,384 (16K) or 32,768 (32K)

⚠️ Setting context to 1M may trigger CUDA OOM and crash — increase gradually as RAM/VRAM allows.

NoPE — No Positional Embeddings

Because Nemotron 3 was trained with NoPE:

Extend context by simply changing max_position_embeddings.
YaRN is NOT needed for context extension.

Chat Template Format

Nemotron 3 uses ChatML-style format:

system\n</think>

user\n{message}
</think>{reasoning}</think){answer}

Reasoning Tokens

<think)> = token ID 12
</think)> = token ID 13
Use --special to see tokens, --verbose-prompt to see the prepended <think)>.

Deployment Issues: EOS Token Leak

Problem: NVIDIA Nemotron 3 models (Super 120B, Nano 30B) use ChatML-style chat templates. The EOS token </think> can leak into generated content when running with the --special flag in llama.cpp. This causes parse failures on subsequent turns, e.g.:

Error: Failed to parse input at pos 285: 
</think>

The --special flag is required for Nemotron reasoning tokens, so it cannot be removed.

Fix: Stop the generation when the </think> token appears by including it in the payload's stop array when calling the API. The --stop command-line flag does not exist in llama-server; stop sequences must be handled at the request level in the API payload.

Applies to:

nemotron-3-super-120b — the 120B-A12B flagship model
nemotron-3-nano-30b — the 30B-A3B compact model

Config file: configs/pfi-ana/docker/conf/llama-swap/config.yaml

Example fix in API payload (Python client):

completion = openai_client.chat.completions.create(
    model = "unsloth/NVIDIA-Nemotron-3-Super-120B-A12B",
    messages = [{"role": "user", "content": "What is 2+2?"}],
    stop=["</think>"]  # Stop sequences must be in the payload, not command-line
)

Example fix with cURL:

curl -X POST http://localhost:8001/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "unsloth/NVIDIA-Nemotron-3-Super-120B-A12B",
        "messages": [{"role": "user", "content": "What is 2+2?"}],
        "stop": ["</think>"]
    }'

Example llama.cpp Commands

General Instruction (4-bit)

./llama.cpp/llama-cli \
    -hf unsloth/NVIDIA-Nemotron-3-Super-120B-A12B-GGUF:UD-Q4_K_XL \
    --ctx-size 16384 \
    --temp 1.0 --top-p 1.0

Tool Calling (4-bit)

./llama.cpp/llama-cli \
    -hf unsloth/NVIDIA-Nemotron-3-Super-120B-A12B-GGUF:UD-Q4_K_XL \
    --ctx-size 32768 \
    --temp 0.6 --top-p 0.95

Conversation Mode (Local GGUF)

./llama.cpp/llama-cli \
    --model unsloth/NVIDIA-Nemotron-3-Super-120B-A12B-GGUF/UD-Q4_K_XL/NVIDIA-Nemotron-3-Super-120B-A12B-UD-Q4_K_XL-00001-of-00003.gguf \
    --ctx-size 16384 \
    --seed 3407 \
    --prio 2 \
    --temp 0.6 \
    --top-p 0.95

llama-server Deployment

./llama.cpp/llama-server \
    --model unsloth/NVIDIA-Nemotron-3-Super-120B-A12B-GGUF/UD-Q4_K_XL/NVIDIA-Nemotron-3-Super-120B-A12B-UD-Q4_K_XL-00001-of-00003.gguf \
    --alias "unsloth/NVIDIA-Nemotron-3-Super-120B-A12B" \
    --prio 3 \
    --min_p 0.01 \
    --temp 0.6 --top-p 0.95 \
    --ctx-size 16384 \
    --port 8001

OpenAI-Compatible Client (with Reasoning Output)

from openai import OpenAI

openai_client = OpenAI(
    base_url = "http://127.0.0.1:8001/v1",
    api_key = "sk-no-key-required",
)

completion = openai_client.chat.completions.create(
    model = "unsloth/NVIDIA-Nemotron-3-Super-120B-A12B",
    messages = [{"role": "user", "content": "What is 2+2?"}]
)

# Access both reasoning and final answer
print(completion.choices[0].message.reasoning_content)
print(completion.choices[0].message.content)

Note: Unlike some models, Nemotron 3 Super exposes reasoning content separately via reasoning_content in the OpenAI-compatible API.

Fine-Tuning Notes

bf16 LoRA works on 256GB VRAM (multi-GPU setup).
Router-layer fine-tuning is disabled by default for stability.
For multi-GPU, use device_map = "balanced" or follow Unsloth's multiGPU guide.
To retain reasoning capabilities in fine-tuning, use at least 75% reasoning and 25% non-reasoning data.

Available Formats

Format	Link
GGUF (Dynamic quantization)	unsloth/NVIDIA-Nemotron-3-Super-120B-A12B-GGUF
NVFP4	unsloth/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4
FP8	unsloth/NVIDIA-Nemotron-3-Super-120B-A12B-FP8
BF16	unsloth/NVIDIA-Nemotron-3-Super-120B-A12B

Sources

Unsloth NVIDIA Nemotron 3 Super Documentation (How to Run)
GGUF: https://huggingface.co/unsloth/NVIDIA-Nemotron-3-Super-120B-A12B-GGUF

8.1 KiB Raw Blame History Unescape Escape

NVIDIA Nemotron 3 Super — Optimal Running Parameters