Create: reference/gemma-4-running-parameters.md
This commit is contained in:
Atlas
@@ -0,0 +1,212 @@
|
||||
---
|
||||
created: '2026-04-09T17:56:14+00:00'
|
||||
id: fe1135ec-1e2a-4ece-bb16-7762c5854380
|
||||
keywords:
|
||||
- Gemma 4
|
||||
- Google DeepMind
|
||||
- temperature
|
||||
- top_p
|
||||
- top_k
|
||||
- GGUF
|
||||
- quantization
|
||||
- UD-Q4_K_XL
|
||||
- thinking mode
|
||||
- llama.cpp
|
||||
- llama-server
|
||||
- MoE
|
||||
- multimodal
|
||||
- visual token budget
|
||||
- CUDA
|
||||
- E2B
|
||||
- E4B
|
||||
- 26B-A4B
|
||||
- 31B
|
||||
links: []
|
||||
modified: '2026-04-09T17:56:14+00:00'
|
||||
summary: Optimal running parameters for Google DeepMind's Gemma 4 family of models,
|
||||
covering inference settings, quantization choices, hardware requirements, thinking
|
||||
mode, multimodal configuration, and llama.cpp example commands.
|
||||
tags:
|
||||
- reference
|
||||
- gemma-4
|
||||
- llm
|
||||
- inference
|
||||
- llama-cpp
|
||||
- local-ai
|
||||
title: Gemma 4 — Optimal Running Parameters
|
||||
---
|
||||
|
||||
# Gemma 4 — Optimal Running Parameters
|
||||
|
||||
A reference for optimally running Google DeepMind's Gemma 4 family of models locally.
|
||||
|
||||
## Model Variants
|
||||
|
||||
| Variant | Architecture | Context | Modalities | Best Fit |
|
||||
|---|---|---|---|---|
|
||||
| **E2B** | Dense + PLE | 128K | Text, Image, Audio | Phone / edge, ASR, speech translation |
|
||||
| **E4B** | Dense + PLE | 128K | Text, Image, Audio | Laptops, fast local multimodal |
|
||||
| **26B-A4B** | MoE (4B active) | 256K | Text, Image | Best speed/quality tradeoff |
|
||||
| **31B** | Dense | 256K | Text, Image | Strongest performance |
|
||||
|
||||
## Recommended Inference Parameters (All Variants)
|
||||
|
||||
These are Google's default Gemma 4 parameters — use them unless you have a specific reason to change:
|
||||
|
||||
| Parameter | Value |
|
||||
|---|---|
|
||||
| `temperature` | **1.0** |
|
||||
| `top_p` | **0.95** |
|
||||
| `top_k` | **64** |
|
||||
| Repetition / Presence Penalty | **Disabled** or 1.0 (only increase if you see looping) |
|
||||
| End of Sentence Token | `<turn\|>` |
|
||||
|
||||
### Context Length
|
||||
|
||||
- **E2B / E4B**: max **128K**
|
||||
- **26B-A4B / 31B**: max **256K**
|
||||
- **Practical tip**: Start with **32K context** for responsiveness, then increase as needed.
|
||||
|
||||
## Quantization Recommendations
|
||||
|
||||
| Variant | Recommended Quant | Alternative | Full Precision (BF16/FP16) |
|
||||
|---|---|---|---|
|
||||
| **E2B** | Q8_0 (8-bit) | Q4 (4 GB) | 10 GB |
|
||||
| **E4B** | Q8_0 (8-bit) | Q4 (5.5–6 GB) | 16 GB |
|
||||
| **26B-A4B** | **UD-Q4_K_XL** (Dynamic 4-bit) | 8-bit (28–30 GB) | 52 GB |
|
||||
| **31B** | **UD-Q4_K_XL** (Dynamic 4-bit) | 8-bit (34–38 GB) | 62 GB |
|
||||
|
||||
- Small models (E2B, E4B): prefer **8-bit** for quality.
|
||||
- Large models (26B-A4B, 31B): prefer **Dynamic 4-bit** (`UD-Q4_K_XL`) as the starting point.
|
||||
|
||||
## Hardware Requirements (Total Memory: RAM + VRAM)
|
||||
|
||||
| Variant | 4-bit | 8-bit | BF16/FP16 |
|
||||
|---|---|---|---|
|
||||
| **E2B** | 4 GB | 5–8 GB | 10 GB |
|
||||
| **E4B** | 5.5–6 GB | 9–12 GB | 16 GB |
|
||||
| **26B-A4B** | 16–18 GB | 28–30 GB | 52 GB |
|
||||
| **31B** | 17–20 GB | 34–38 GB | 62 GB |
|
||||
|
||||
> **Rule of thumb:** Total available memory should at least exceed the size of the quantized model. If it doesn't, llama.cpp can use partial RAM/disk offload but generation will be slower.
|
||||
|
||||
## 26B-A4B vs 31B — When to Choose Which
|
||||
|
||||
- **26B-A4B** — Choose when RAM is limited. MoE architecture with 4B active parameters makes it faster. Slight quality tradeoff for speed.
|
||||
- **31B** — Choose when you have enough memory (≥20 GB for 4-bit) and want maximum quality. Slower inference.
|
||||
|
||||
## Thinking Mode
|
||||
|
||||
### Enabling Thinking
|
||||
|
||||
Add `<|think|>` at the **start of the system prompt**:
|
||||
|
||||
```
|
||||
<|think|>
|
||||
You are a careful coding assistant. Explain your answer clearly.
|
||||
```
|
||||
|
||||
When enabled, the model outputs an internal reasoning channel before the final answer:
|
||||
|
||||
```
|
||||
<|channel>thought
|
||||
[internal reasoning]
|
||||
<channel|>
|
||||
[final answer]
|
||||
```
|
||||
|
||||
### Disabling Thinking
|
||||
|
||||
Use the `llama-server` flag (not `llama-cli`, which may be unreliable):
|
||||
|
||||
```bash
|
||||
--chat-template-kwargs '{"enable_thinking":false}'
|
||||
```
|
||||
|
||||
On **Windows PowerShell**:
|
||||
|
||||
```bash
|
||||
--chat-template-kwargs "{\"enable_thinking\":false}"
|
||||
```
|
||||
|
||||
> **Note:** Even with thinking disabled, larger models may still emit an empty thought block (`<|channel>thought<channel|>`) before the answer. This is normal.
|
||||
|
||||
### Multi-Turn Chat Rule
|
||||
|
||||
**Only keep the final visible answer** in chat history. Do **not** feed prior thought blocks back into subsequent turns.
|
||||
|
||||
## Multimodal Settings
|
||||
|
||||
### Image Placement
|
||||
|
||||
Always put **images and/or audio before text** in prompts. For video, pass frames first, then the instruction.
|
||||
|
||||
### Visual Token Budgets
|
||||
|
||||
| Budget | Use Case |
|
||||
|---|---|
|
||||
| **70 / 140** | Classification, captioning, fast video understanding |
|
||||
| **280 / 560** | General multimodal chat, charts, screens, UI reasoning |
|
||||
| **1120** | OCR, document parsing, handwriting, small text |
|
||||
|
||||
### Audio Limits
|
||||
|
||||
- Audio is **only on E2B and E4B**.
|
||||
- Maximum audio length: **30 seconds**.
|
||||
- Maximum video length: **60 seconds** (at 1 frame/second).
|
||||
|
||||
## Critical Warnings
|
||||
|
||||
- ⚠️ **Do NOT use CUDA 13.2 runtime** for any GGUF — it will cause poor outputs.
|
||||
- Use `llama-server` instead of `llama-cli` for thinking control (more reliable).
|
||||
|
||||
## Example llama.cpp Commands
|
||||
|
||||
### 26B-A4B (Dynamic 4-bit)
|
||||
|
||||
```bash
|
||||
./llama.cpp/llama-cli \
|
||||
-hf unsloth/gemma-4-26B-A4B-it-GGUF:UD-Q4_K_XL \
|
||||
--temp 1.0 --top-p 0.95 --top-k 64
|
||||
```
|
||||
|
||||
### 31B (Dynamic 4-bit)
|
||||
|
||||
```bash
|
||||
./llama.cpp/llama-cli \
|
||||
-hf unsloth/gemma-4-31B-it-GGUF:UD-Q4_K_XL \
|
||||
--temp 1.0 --top-p 0.95 --top-k 64
|
||||
```
|
||||
|
||||
### E4B (8-bit)
|
||||
|
||||
```bash
|
||||
./llama.cpp/llama-cli \
|
||||
-hf unsloth/gemma-4-E4B-it-GGUF:Q8_0 \
|
||||
--temp 1.0 --top-p 0.95 --top-k 64
|
||||
```
|
||||
|
||||
### E2B (8-bit)
|
||||
|
||||
```bash
|
||||
./llama.cpp/llama-cli \
|
||||
-hf unsloth/gemma-4-E2B-it-GGUF:Q8_0 \
|
||||
--temp 1.0 --top-p 0.95 --top-k 64
|
||||
```
|
||||
|
||||
### llama-server with Vision + Thinking
|
||||
|
||||
```bash
|
||||
./llama.cpp/llama-server \
|
||||
--model unsloth/gemma-4-26B-A4B-it-GGUF/gemma-4-26B-A4B-it-UD-Q4_K_XL.gguf \
|
||||
--mmproj unsloth/gemma-4-26B-A4B-it-GGUF/mmproj-BF16.gguf \
|
||||
--temp 1.0 --top-p 0.95 --top-k 64 \
|
||||
--alias "unsloth/gemma-4-26B-A4B-it-GGUF" \
|
||||
--port 8001 \
|
||||
--chat-template-kwargs '{"enable_thinking":true}'
|
||||
```
|
||||
|
||||
## Sources
|
||||
|
||||
- Unsloth Gemma 4 Documentation (How to Run Locally)
|
||||
- Gemma 4 GGUF collection: <https://huggingface.co/collections/unsloth/gemma-4>
|
||||
Reference in New Issue
Block a user