From 938b35286da2a9107af9ae9b41762a0da01911e6 Mon Sep 17 00:00:00 2001 From: Atlas Date: Thu, 9 Apr 2026 11:07:12 -0700 Subject: [PATCH] Create: reference/nemotron-3-super-running-parameters.md --- .../nemotron-3-super-running-parameters.md | 233 ++++++++++++++++++ 1 file changed, 233 insertions(+) create mode 100644 reference/nemotron-3-super-running-parameters.md diff --git a/reference/nemotron-3-super-running-parameters.md b/reference/nemotron-3-super-running-parameters.md new file mode 100644 index 0000000..582cd02 --- /dev/null +++ b/reference/nemotron-3-super-running-parameters.md @@ -0,0 +1,233 @@ +--- +created: '2026-04-09T18:07:12+00:00' +id: 2a98c58e-a3e3-429b-87b0-dd9c199a6e91 +keywords: +- Nemotron 3 Super +- NVIDIA +- 120B-A12B +- temperature +- top_p +- min_p +- GGUF +- quantization +- UD-Q4_K_XL +- NoPE +- 1M context +- llama.cpp +- llama-server +- MoE +- 12B active +- tool calling +- reasoning tokens +- NVFP4 +- FP8 +- BF16 +links: [] +modified: '2026-04-09T18:07:12+00:00' +summary: Optimal running parameters for NVIDIA's Nemotron 3 Super (120B-A12B MoE), + covering inference settings for general chat vs tool calling, hardware requirements, + NoPE context extension, reasoning token handling, GGUF/NVFP4/FP8 formats, and llama.cpp + example commands. +tags: +- reference +- nemotron-3 +- nvidia +- llm +- inference +- llama-cpp +- local-ai +title: NVIDIA Nemotron 3 Super — Optimal Running Parameters +--- + +# NVIDIA Nemotron 3 Super — Optimal Running Parameters + +A reference for optimally running NVIDIA's Nemotron 3 Super (120B-A12B) hybrid reasoning MoE model locally. + +## Model Overview + +| Property | Value | +|---|---| +| Architecture | MoE — 120B total, **12B active parameters** | +| Max Context | **1M tokens** (default 262,144) | +| Positional Embeddings | **NoPE** — no explicit positional embeddings; YaRN not needed | +| Reasoning Tokens | `` (token ID 12), `` (token ID 13) | +| Best For | Multi-agent AI, high-efficiency reasoning, coding, math, agentic workflows | +| Performance Tier | Leads its size class on AIME 2025, Terminal Bench, SWE-Bench Verified; highest throughput | + +## Recommended Inference Parameters + +NVIDIA recommends two parameter profiles: + +### General Chat / Instruction + +| Parameter | Value | +|---|---| +| `temperature` | **1.0** | +| `top_p` | **1.0** | + +### Tool Calling + +| Parameter | Value | +|---|---| +| `temperature` | **0.6** | +| `top_p` | **0.95** | + +### Additional Settings + +| Setting | Value | Notes | +|---|---|---| +| `max_new_tokens` | **32,768 – 262,144** | Standard prompts; max 1M | +| `min_p` | **0.01** | Used in llama-server deployment | +| `--prio` | **2** or **3** | Priority scheduling in llama.cpp | +| `--seed` | **3407** | Reproducibility | +| `--special` | (flag) | Needed to see reasoning tokens | +| `--verbose-prompt` | (flag) | Needed to see `` since it's prepended | + +### Quick Reference — One-Liners + +**General chat/instruction:** +``` +temperature=1.0, top_p=1.0 +``` + +**Tool calling:** +``` +temperature=0.6, top_p=0.95 +``` + +## Hardware Requirements (Total Memory: RAM + VRAM, or Unified Memory) + +| Precision | Memory Required | +|---|---| +| 2-bit (UD-Q2_K_XL) | ~32–36 GB | +| **4-bit (UD-Q4_K_XL)** | **~64–72 GB** | +| 8-bit | ~128 GB | +| BF16 | ~240 GB | + +> **Rule of thumb:** Total available memory should exceed the quantized model size. If it doesn't, llama.cpp can offload to SSD/HDD but inference will be slower. + +### Quantization Recommendations + +- **Minimum recommended:** **UD-Q2_K_XL** (Dynamic 2-bit) to balance size and accuracy. +- **Recommended starting point:** **UD-Q4_K_XL** (Dynamic 4-bit) — fits 64–72GB devices. +- Available in multiple formats: GGUF, NVFP4, FP8, BF16. + +## Context Window Management + +| Setting | Value | +|---|---| +| Native max context | **1M tokens** | +| Default context | **262,144** (256K) | +| Recommended starting context | **16,384** (16K) or **32,768** (32K) | + +> ⚠️ Setting context to 1M may trigger **CUDA OOM and crash** — increase gradually as RAM/VRAM allows. + +### NoPE — No Positional Embeddings + +Because Nemotron 3 was trained with **NoPE**: +- Extend context by simply changing `max_position_embeddings`. +- **YaRN is NOT needed** for context extension. + +## Chat Template Format + +Nemotron 3 uses ChatML-style format: + +``` +<|im_start|>system\n<|im_end|>\n +<|im_start|>user\n{message}<|im_end|>\n +<|im_start|>assistant\n{reasoning}{answer}<|im_end|>\n +``` + +### Reasoning Tokens + +- `` = token ID **12** +- `` = token ID **13** +- Use `--special` to see tokens, `--verbose-prompt` to see the prepended ``. + +## Example llama.cpp Commands + +### General Instruction (4-bit) + +```bash +./llama.cpp/llama-cli \ + -hf unsloth/NVIDIA-Nemotron-3-Super-120B-A12B-GGUF:UD-Q4_K_XL \ + --ctx-size 16384 \ + --temp 1.0 --top-p 1.0 +``` + +### Tool Calling (4-bit) + +```bash +./llama.cpp/llama-cli \ + -hf unsloth/NVIDIA-Nemotron-3-Super-120B-A12B-GGUF:UD-Q4_K_XL \ + --ctx-size 32768 \ + --temp 0.6 --top-p 0.95 +``` + +### Conversation Mode (Local GGUF) + +```bash +./llama.cpp/llama-cli \ + --model unsloth/NVIDIA-Nemotron-3-Super-120B-A12B-GGUF/UD-Q4_K_XL/NVIDIA-Nemotron-3-Super-120B-A12B-UD-Q4_K_XL-00001-of-00003.gguf \ + --ctx-size 16384 \ + --seed 3407 \ + --prio 2 \ + --temp 0.6 \ + --top-p 0.95 +``` + +### llama-server Deployment + +```bash +./llama.cpp/llama-server \ + --model unsloth/NVIDIA-Nemotron-3-Super-120B-A12B-GGUF/UD-Q4_K_XL/NVIDIA-Nemotron-3-Super-120B-A12B-UD-Q4_K_XL-00001-of-00003.gguf \ + --alias "unsloth/NVIDIA-Nemotron-3-Super-120B-A12B" \ + --prio 3 \ + --min_p 0.01 \ + --temp 0.6 --top-p 0.95 \ + --ctx-size 16384 \ + --port 8001 +``` + +### OpenAI-Compatible Client (with Reasoning Output) + +```python +from openai import OpenAI + +openai_client = OpenAI( + base_url = "http://127.0.0.1:8001/v1", + api_key = "sk-no-key-required", +) + +completion = openai_client.chat.completions.create( + model = "unsloth/NVIDIA-Nemotron-3-Super-120B-A12B", + messages = [{"role": "user", "content": "What is 2+2?"}], +) + +# Access both reasoning and final answer +print(completion.choices[0].message.reasoning_content) +print(completion.choices[0].message.content) +``` + +> **Note:** Unlike some models, Nemotron 3 Super exposes reasoning content separately via `reasoning_content` in the OpenAI-compatible API. + +## Fine-Tuning Notes + +- **bf16 LoRA** works on **256GB VRAM** (multi-GPU setup). +- Router-layer fine-tuning is **disabled by default** for stability. +- For multi-GPU, use `device_map = "balanced"` or follow Unsloth's multiGPU guide. +- To retain reasoning capabilities in fine-tuning, use at least **75% reasoning** and **25% non-reasoning** data. + +## Available Formats + +| Format | Link | +|---|---| +| GGUF (Dynamic quantization) | [unsloth/NVIDIA-Nemotron-3-Super-120B-A12B-GGUF](https://huggingface.co/unsloth/NVIDIA-Nemotron-3-Super-120B-A12B-GGUF) | +| NVFP4 | [unsloth/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4](https://huggingface.co/unsloth/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4) | +| FP8 | [unsloth/NVIDIA-Nemotron-3-Super-120B-A12B-FP8](https://huggingface.co/unsloth/NVIDIA-Nemotron-3-Super-120B-A12B-FP8) | +| BF16 | [unsloth/NVIDIA-Nemotron-3-Super-120B-A12B](https://huggingface.co/unsloth/NVIDIA-Nemotron-3-Super-120B-A12B) | + +## Sources + +- Unsloth NVIDIA Nemotron 3 Super Documentation (How to Run) +- GGUF: \ No newline at end of file