--- created: '2026-04-09T17:56:14+00:00' id: fe1135ec-1e2a-4ece-bb16-7762c5854380 keywords: - Gemma 4 - Google DeepMind - temperature - top_p - top_k - GGUF - quantization - UD-Q4_K_XL - thinking mode - llama.cpp - llama-server - MoE - multimodal - visual token budget - CUDA - E2B - E4B - 26B-A4B - 31B links: [] modified: '2026-04-09T17:56:14+00:00' summary: Optimal running parameters for Google DeepMind's Gemma 4 family of models, covering inference settings, quantization choices, hardware requirements, thinking mode, multimodal configuration, and llama.cpp example commands. tags: - reference - gemma-4 - llm - inference - llama-cpp - local-ai title: Gemma 4 — Optimal Running Parameters --- # Gemma 4 — Optimal Running Parameters A reference for optimally running Google DeepMind's Gemma 4 family of models locally. ## Model Variants | Variant | Architecture | Context | Modalities | Best Fit | |---|---|---|---|---| | **E2B** | Dense + PLE | 128K | Text, Image, Audio | Phone / edge, ASR, speech translation | | **E4B** | Dense + PLE | 128K | Text, Image, Audio | Laptops, fast local multimodal | | **26B-A4B** | MoE (4B active) | 256K | Text, Image | Best speed/quality tradeoff | | **31B** | Dense | 256K | Text, Image | Strongest performance | ## Recommended Inference Parameters (All Variants) These are Google's default Gemma 4 parameters — use them unless you have a specific reason to change: | Parameter | Value | |---|---| | `temperature` | **1.0** | | `top_p` | **0.95** | | `top_k` | **64** | | Repetition / Presence Penalty | **Disabled** or 1.0 (only increase if you see looping) | | End of Sentence Token | `` | ### Context Length - **E2B / E4B**: max **128K** - **26B-A4B / 31B**: max **256K** - **Practical tip**: Start with **32K context** for responsiveness, then increase as needed. ## Quantization Recommendations | Variant | Recommended Quant | Alternative | Full Precision (BF16/FP16) | |---|---|---|---| | **E2B** | Q8_0 (8-bit) | Q4 (4 GB) | 10 GB | | **E4B** | Q8_0 (8-bit) | Q4 (5.5–6 GB) | 16 GB | | **26B-A4B** | **UD-Q4_K_XL** (Dynamic 4-bit) | 8-bit (28–30 GB) | 52 GB | | **31B** | **UD-Q4_K_XL** (Dynamic 4-bit) | 8-bit (34–38 GB) | 62 GB | - Small models (E2B, E4B): prefer **8-bit** for quality. - Large models (26B-A4B, 31B): prefer **Dynamic 4-bit** (`UD-Q4_K_XL`) as the starting point. ## Hardware Requirements (Total Memory: RAM + VRAM) | Variant | 4-bit | 8-bit | BF16/FP16 | |---|---|---|---| | **E2B** | 4 GB | 5–8 GB | 10 GB | | **E4B** | 5.5–6 GB | 9–12 GB | 16 GB | | **26B-A4B** | 16–18 GB | 28–30 GB | 52 GB | | **31B** | 17–20 GB | 34–38 GB | 62 GB | > **Rule of thumb:** Total available memory should at least exceed the size of the quantized model. If it doesn't, llama.cpp can use partial RAM/disk offload but generation will be slower. ## 26B-A4B vs 31B — When to Choose Which - **26B-A4B** — Choose when RAM is limited. MoE architecture with 4B active parameters makes it faster. Slight quality tradeoff for speed. - **31B** — Choose when you have enough memory (≥20 GB for 4-bit) and want maximum quality. Slower inference. ## Thinking Mode ### Enabling Thinking Add `<|think|>` at the **start of the system prompt**: ``` <|think|> You are a careful coding assistant. Explain your answer clearly. ``` When enabled, the model outputs an internal reasoning channel before the final answer: ``` <|channel>thought [internal reasoning] [final answer] ``` ### Disabling Thinking Use the `llama-server` flag (not `llama-cli`, which may be unreliable): ```bash --chat-template-kwargs '{"enable_thinking":false}' ``` On **Windows PowerShell**: ```bash --chat-template-kwargs "{\"enable_thinking\":false}" ``` > **Note:** Even with thinking disabled, larger models may still emit an empty thought block (`<|channel>thought`) before the answer. This is normal. ### Multi-Turn Chat Rule **Only keep the final visible answer** in chat history. Do **not** feed prior thought blocks back into subsequent turns. ## Multimodal Settings ### Image Placement Always put **images and/or audio before text** in prompts. For video, pass frames first, then the instruction. ### Visual Token Budgets | Budget | Use Case | |---|---| | **70 / 140** | Classification, captioning, fast video understanding | | **280 / 560** | General multimodal chat, charts, screens, UI reasoning | | **1120** | OCR, document parsing, handwriting, small text | ### Audio Limits - Audio is **only on E2B and E4B**. - Maximum audio length: **30 seconds**. - Maximum video length: **60 seconds** (at 1 frame/second). ## Critical Warnings - ⚠️ **Do NOT use CUDA 13.2 runtime** for any GGUF — it will cause poor outputs. - Use `llama-server` instead of `llama-cli` for thinking control (more reliable). ## Example llama.cpp Commands ### 26B-A4B (Dynamic 4-bit) ```bash ./llama.cpp/llama-cli \ -hf unsloth/gemma-4-26B-A4B-it-GGUF:UD-Q4_K_XL \ --temp 1.0 --top-p 0.95 --top-k 64 ``` ### 31B (Dynamic 4-bit) ```bash ./llama.cpp/llama-cli \ -hf unsloth/gemma-4-31B-it-GGUF:UD-Q4_K_XL \ --temp 1.0 --top-p 0.95 --top-k 64 ``` ### E4B (8-bit) ```bash ./llama.cpp/llama-cli \ -hf unsloth/gemma-4-E4B-it-GGUF:Q8_0 \ --temp 1.0 --top-p 0.95 --top-k 64 ``` ### E2B (8-bit) ```bash ./llama.cpp/llama-cli \ -hf unsloth/gemma-4-E2B-it-GGUF:Q8_0 \ --temp 1.0 --top-p 0.95 --top-k 64 ``` ### llama-server with Vision + Thinking ```bash ./llama.cpp/llama-server \ --model unsloth/gemma-4-26B-A4B-it-GGUF/gemma-4-26B-A4B-it-UD-Q4_K_XL.gguf \ --mmproj unsloth/gemma-4-26B-A4B-it-GGUF/mmproj-BF16.gguf \ --temp 1.0 --top-p 0.95 --top-k 64 \ --alias "unsloth/gemma-4-26B-A4B-it-GGUF" \ --port 8001 \ --chat-template-kwargs '{"enable_thinking":true}' ``` ## Sources - Unsloth Gemma 4 Documentation (How to Run Locally) - Gemma 4 GGUF collection: