diff --git a/reference/gemma-4-running-parameters.md b/reference/gemma-4-running-parameters.md new file mode 100644 index 0000000..1ad3854 --- /dev/null +++ b/reference/gemma-4-running-parameters.md @@ -0,0 +1,212 @@ +--- +created: '2026-04-09T17:56:14+00:00' +id: fe1135ec-1e2a-4ece-bb16-7762c5854380 +keywords: +- Gemma 4 +- Google DeepMind +- temperature +- top_p +- top_k +- GGUF +- quantization +- UD-Q4_K_XL +- thinking mode +- llama.cpp +- llama-server +- MoE +- multimodal +- visual token budget +- CUDA +- E2B +- E4B +- 26B-A4B +- 31B +links: [] +modified: '2026-04-09T17:56:14+00:00' +summary: Optimal running parameters for Google DeepMind's Gemma 4 family of models, + covering inference settings, quantization choices, hardware requirements, thinking + mode, multimodal configuration, and llama.cpp example commands. +tags: +- reference +- gemma-4 +- llm +- inference +- llama-cpp +- local-ai +title: Gemma 4 — Optimal Running Parameters +--- + +# Gemma 4 — Optimal Running Parameters + +A reference for optimally running Google DeepMind's Gemma 4 family of models locally. + +## Model Variants + +| Variant | Architecture | Context | Modalities | Best Fit | +|---|---|---|---|---| +| **E2B** | Dense + PLE | 128K | Text, Image, Audio | Phone / edge, ASR, speech translation | +| **E4B** | Dense + PLE | 128K | Text, Image, Audio | Laptops, fast local multimodal | +| **26B-A4B** | MoE (4B active) | 256K | Text, Image | Best speed/quality tradeoff | +| **31B** | Dense | 256K | Text, Image | Strongest performance | + +## Recommended Inference Parameters (All Variants) + +These are Google's default Gemma 4 parameters — use them unless you have a specific reason to change: + +| Parameter | Value | +|---|---| +| `temperature` | **1.0** | +| `top_p` | **0.95** | +| `top_k` | **64** | +| Repetition / Presence Penalty | **Disabled** or 1.0 (only increase if you see looping) | +| End of Sentence Token | `` | + +### Context Length + +- **E2B / E4B**: max **128K** +- **26B-A4B / 31B**: max **256K** +- **Practical tip**: Start with **32K context** for responsiveness, then increase as needed. + +## Quantization Recommendations + +| Variant | Recommended Quant | Alternative | Full Precision (BF16/FP16) | +|---|---|---|---| +| **E2B** | Q8_0 (8-bit) | Q4 (4 GB) | 10 GB | +| **E4B** | Q8_0 (8-bit) | Q4 (5.5–6 GB) | 16 GB | +| **26B-A4B** | **UD-Q4_K_XL** (Dynamic 4-bit) | 8-bit (28–30 GB) | 52 GB | +| **31B** | **UD-Q4_K_XL** (Dynamic 4-bit) | 8-bit (34–38 GB) | 62 GB | + +- Small models (E2B, E4B): prefer **8-bit** for quality. +- Large models (26B-A4B, 31B): prefer **Dynamic 4-bit** (`UD-Q4_K_XL`) as the starting point. + +## Hardware Requirements (Total Memory: RAM + VRAM) + +| Variant | 4-bit | 8-bit | BF16/FP16 | +|---|---|---|---| +| **E2B** | 4 GB | 5–8 GB | 10 GB | +| **E4B** | 5.5–6 GB | 9–12 GB | 16 GB | +| **26B-A4B** | 16–18 GB | 28–30 GB | 52 GB | +| **31B** | 17–20 GB | 34–38 GB | 62 GB | + +> **Rule of thumb:** Total available memory should at least exceed the size of the quantized model. If it doesn't, llama.cpp can use partial RAM/disk offload but generation will be slower. + +## 26B-A4B vs 31B — When to Choose Which + +- **26B-A4B** — Choose when RAM is limited. MoE architecture with 4B active parameters makes it faster. Slight quality tradeoff for speed. +- **31B** — Choose when you have enough memory (≥20 GB for 4-bit) and want maximum quality. Slower inference. + +## Thinking Mode + +### Enabling Thinking + +Add `<|think|>` at the **start of the system prompt**: + +``` +<|think|> +You are a careful coding assistant. Explain your answer clearly. +``` + +When enabled, the model outputs an internal reasoning channel before the final answer: + +``` +<|channel>thought +[internal reasoning] + +[final answer] +``` + +### Disabling Thinking + +Use the `llama-server` flag (not `llama-cli`, which may be unreliable): + +```bash +--chat-template-kwargs '{"enable_thinking":false}' +``` + +On **Windows PowerShell**: + +```bash +--chat-template-kwargs "{\"enable_thinking\":false}" +``` + +> **Note:** Even with thinking disabled, larger models may still emit an empty thought block (`<|channel>thought`) before the answer. This is normal. + +### Multi-Turn Chat Rule + +**Only keep the final visible answer** in chat history. Do **not** feed prior thought blocks back into subsequent turns. + +## Multimodal Settings + +### Image Placement + +Always put **images and/or audio before text** in prompts. For video, pass frames first, then the instruction. + +### Visual Token Budgets + +| Budget | Use Case | +|---|---| +| **70 / 140** | Classification, captioning, fast video understanding | +| **280 / 560** | General multimodal chat, charts, screens, UI reasoning | +| **1120** | OCR, document parsing, handwriting, small text | + +### Audio Limits + +- Audio is **only on E2B and E4B**. +- Maximum audio length: **30 seconds**. +- Maximum video length: **60 seconds** (at 1 frame/second). + +## Critical Warnings + +- ⚠️ **Do NOT use CUDA 13.2 runtime** for any GGUF — it will cause poor outputs. +- Use `llama-server` instead of `llama-cli` for thinking control (more reliable). + +## Example llama.cpp Commands + +### 26B-A4B (Dynamic 4-bit) + +```bash +./llama.cpp/llama-cli \ + -hf unsloth/gemma-4-26B-A4B-it-GGUF:UD-Q4_K_XL \ + --temp 1.0 --top-p 0.95 --top-k 64 +``` + +### 31B (Dynamic 4-bit) + +```bash +./llama.cpp/llama-cli \ + -hf unsloth/gemma-4-31B-it-GGUF:UD-Q4_K_XL \ + --temp 1.0 --top-p 0.95 --top-k 64 +``` + +### E4B (8-bit) + +```bash +./llama.cpp/llama-cli \ + -hf unsloth/gemma-4-E4B-it-GGUF:Q8_0 \ + --temp 1.0 --top-p 0.95 --top-k 64 +``` + +### E2B (8-bit) + +```bash +./llama.cpp/llama-cli \ + -hf unsloth/gemma-4-E2B-it-GGUF:Q8_0 \ + --temp 1.0 --top-p 0.95 --top-k 64 +``` + +### llama-server with Vision + Thinking + +```bash +./llama.cpp/llama-server \ + --model unsloth/gemma-4-26B-A4B-it-GGUF/gemma-4-26B-A4B-it-UD-Q4_K_XL.gguf \ + --mmproj unsloth/gemma-4-26B-A4B-it-GGUF/mmproj-BF16.gguf \ + --temp 1.0 --top-p 0.95 --top-k 64 \ + --alias "unsloth/gemma-4-26B-A4B-it-GGUF" \ + --port 8001 \ + --chat-template-kwargs '{"enable_thinking":true}' +``` + +## Sources + +- Unsloth Gemma 4 Documentation (How to Run Locally) +- Gemma 4 GGUF collection: \ No newline at end of file