212 lines
5.8 KiB
Markdown
212 lines
5.8 KiB
Markdown
---
|
||
created: '2026-04-09T17:56:14+00:00'
|
||
id: fe1135ec-1e2a-4ece-bb16-7762c5854380
|
||
keywords:
|
||
- Gemma 4
|
||
- Google DeepMind
|
||
- temperature
|
||
- top_p
|
||
- top_k
|
||
- GGUF
|
||
- quantization
|
||
- UD-Q4_K_XL
|
||
- thinking mode
|
||
- llama.cpp
|
||
- llama-server
|
||
- MoE
|
||
- multimodal
|
||
- visual token budget
|
||
- CUDA
|
||
- E2B
|
||
- E4B
|
||
- 26B-A4B
|
||
- 31B
|
||
links: []
|
||
modified: '2026-04-09T17:56:14+00:00'
|
||
summary: Optimal running parameters for Google DeepMind's Gemma 4 family of models,
|
||
covering inference settings, quantization choices, hardware requirements, thinking
|
||
mode, multimodal configuration, and llama.cpp example commands.
|
||
tags:
|
||
- reference
|
||
- gemma-4
|
||
- llm
|
||
- inference
|
||
- llama-cpp
|
||
- local-ai
|
||
title: Gemma 4 — Optimal Running Parameters
|
||
---
|
||
|
||
# Gemma 4 — Optimal Running Parameters
|
||
|
||
A reference for optimally running Google DeepMind's Gemma 4 family of models locally.
|
||
|
||
## Model Variants
|
||
|
||
| Variant | Architecture | Context | Modalities | Best Fit |
|
||
|---|---|---|---|---|
|
||
| **E2B** | Dense + PLE | 128K | Text, Image, Audio | Phone / edge, ASR, speech translation |
|
||
| **E4B** | Dense + PLE | 128K | Text, Image, Audio | Laptops, fast local multimodal |
|
||
| **26B-A4B** | MoE (4B active) | 256K | Text, Image | Best speed/quality tradeoff |
|
||
| **31B** | Dense | 256K | Text, Image | Strongest performance |
|
||
|
||
## Recommended Inference Parameters (All Variants)
|
||
|
||
These are Google's default Gemma 4 parameters — use them unless you have a specific reason to change:
|
||
|
||
| Parameter | Value |
|
||
|---|---|
|
||
| `temperature` | **1.0** |
|
||
| `top_p` | **0.95** |
|
||
| `top_k` | **64** |
|
||
| Repetition / Presence Penalty | **Disabled** or 1.0 (only increase if you see looping) |
|
||
| End of Sentence Token | `<turn\|>` |
|
||
|
||
### Context Length
|
||
|
||
- **E2B / E4B**: max **128K**
|
||
- **26B-A4B / 31B**: max **256K**
|
||
- **Practical tip**: Start with **32K context** for responsiveness, then increase as needed.
|
||
|
||
## Quantization Recommendations
|
||
|
||
| Variant | Recommended Quant | Alternative | Full Precision (BF16/FP16) |
|
||
|---|---|---|---|
|
||
| **E2B** | Q8_0 (8-bit) | Q4 (4 GB) | 10 GB |
|
||
| **E4B** | Q8_0 (8-bit) | Q4 (5.5–6 GB) | 16 GB |
|
||
| **26B-A4B** | **UD-Q4_K_XL** (Dynamic 4-bit) | 8-bit (28–30 GB) | 52 GB |
|
||
| **31B** | **UD-Q4_K_XL** (Dynamic 4-bit) | 8-bit (34–38 GB) | 62 GB |
|
||
|
||
- Small models (E2B, E4B): prefer **8-bit** for quality.
|
||
- Large models (26B-A4B, 31B): prefer **Dynamic 4-bit** (`UD-Q4_K_XL`) as the starting point.
|
||
|
||
## Hardware Requirements (Total Memory: RAM + VRAM)
|
||
|
||
| Variant | 4-bit | 8-bit | BF16/FP16 |
|
||
|---|---|---|---|
|
||
| **E2B** | 4 GB | 5–8 GB | 10 GB |
|
||
| **E4B** | 5.5–6 GB | 9–12 GB | 16 GB |
|
||
| **26B-A4B** | 16–18 GB | 28–30 GB | 52 GB |
|
||
| **31B** | 17–20 GB | 34–38 GB | 62 GB |
|
||
|
||
> **Rule of thumb:** Total available memory should at least exceed the size of the quantized model. If it doesn't, llama.cpp can use partial RAM/disk offload but generation will be slower.
|
||
|
||
## 26B-A4B vs 31B — When to Choose Which
|
||
|
||
- **26B-A4B** — Choose when RAM is limited. MoE architecture with 4B active parameters makes it faster. Slight quality tradeoff for speed.
|
||
- **31B** — Choose when you have enough memory (≥20 GB for 4-bit) and want maximum quality. Slower inference.
|
||
|
||
## Thinking Mode
|
||
|
||
### Enabling Thinking
|
||
|
||
Add `<|think|>` at the **start of the system prompt**:
|
||
|
||
```
|
||
<|think|>
|
||
You are a careful coding assistant. Explain your answer clearly.
|
||
```
|
||
|
||
When enabled, the model outputs an internal reasoning channel before the final answer:
|
||
|
||
```
|
||
<|channel>thought
|
||
[internal reasoning]
|
||
<channel|>
|
||
[final answer]
|
||
```
|
||
|
||
### Disabling Thinking
|
||
|
||
Use the `llama-server` flag (not `llama-cli`, which may be unreliable):
|
||
|
||
```bash
|
||
--chat-template-kwargs '{"enable_thinking":false}'
|
||
```
|
||
|
||
On **Windows PowerShell**:
|
||
|
||
```bash
|
||
--chat-template-kwargs "{\"enable_thinking\":false}"
|
||
```
|
||
|
||
> **Note:** Even with thinking disabled, larger models may still emit an empty thought block (`<|channel>thought<channel|>`) before the answer. This is normal.
|
||
|
||
### Multi-Turn Chat Rule
|
||
|
||
**Only keep the final visible answer** in chat history. Do **not** feed prior thought blocks back into subsequent turns.
|
||
|
||
## Multimodal Settings
|
||
|
||
### Image Placement
|
||
|
||
Always put **images and/or audio before text** in prompts. For video, pass frames first, then the instruction.
|
||
|
||
### Visual Token Budgets
|
||
|
||
| Budget | Use Case |
|
||
|---|---|
|
||
| **70 / 140** | Classification, captioning, fast video understanding |
|
||
| **280 / 560** | General multimodal chat, charts, screens, UI reasoning |
|
||
| **1120** | OCR, document parsing, handwriting, small text |
|
||
|
||
### Audio Limits
|
||
|
||
- Audio is **only on E2B and E4B**.
|
||
- Maximum audio length: **30 seconds**.
|
||
- Maximum video length: **60 seconds** (at 1 frame/second).
|
||
|
||
## Critical Warnings
|
||
|
||
- ⚠️ **Do NOT use CUDA 13.2 runtime** for any GGUF — it will cause poor outputs.
|
||
- Use `llama-server` instead of `llama-cli` for thinking control (more reliable).
|
||
|
||
## Example llama.cpp Commands
|
||
|
||
### 26B-A4B (Dynamic 4-bit)
|
||
|
||
```bash
|
||
./llama.cpp/llama-cli \
|
||
-hf unsloth/gemma-4-26B-A4B-it-GGUF:UD-Q4_K_XL \
|
||
--temp 1.0 --top-p 0.95 --top-k 64
|
||
```
|
||
|
||
### 31B (Dynamic 4-bit)
|
||
|
||
```bash
|
||
./llama.cpp/llama-cli \
|
||
-hf unsloth/gemma-4-31B-it-GGUF:UD-Q4_K_XL \
|
||
--temp 1.0 --top-p 0.95 --top-k 64
|
||
```
|
||
|
||
### E4B (8-bit)
|
||
|
||
```bash
|
||
./llama.cpp/llama-cli \
|
||
-hf unsloth/gemma-4-E4B-it-GGUF:Q8_0 \
|
||
--temp 1.0 --top-p 0.95 --top-k 64
|
||
```
|
||
|
||
### E2B (8-bit)
|
||
|
||
```bash
|
||
./llama.cpp/llama-cli \
|
||
-hf unsloth/gemma-4-E2B-it-GGUF:Q8_0 \
|
||
--temp 1.0 --top-p 0.95 --top-k 64
|
||
```
|
||
|
||
### llama-server with Vision + Thinking
|
||
|
||
```bash
|
||
./llama.cpp/llama-server \
|
||
--model unsloth/gemma-4-26B-A4B-it-GGUF/gemma-4-26B-A4B-it-UD-Q4_K_XL.gguf \
|
||
--mmproj unsloth/gemma-4-26B-A4B-it-GGUF/mmproj-BF16.gguf \
|
||
--temp 1.0 --top-p 0.95 --top-k 64 \
|
||
--alias "unsloth/gemma-4-26B-A4B-it-GGUF" \
|
||
--port 8001 \
|
||
--chat-template-kwargs '{"enable_thinking":true}'
|
||
```
|
||
|
||
## Sources
|
||
|
||
- Unsloth Gemma 4 Documentation (How to Run Locally)
|
||
- Gemma 4 GGUF collection: <https://huggingface.co/collections/unsloth/gemma-4> |