VuongKB/reference/gemma-4-running-parameters.md

created, id, keywords, links, modified, summary, tags, title

created	id	keywords	links	modified	summary	tags	title
2026-04-09T17:56:14+00:00	fe1135ec-1e2a-4ece-bb16-7762c5854380	Gemma 4 Google DeepMind temperature top_p top_k GGUF quantization UD-Q4_K_XL thinking mode llama.cpp llama-server MoE multimodal visual token budget CUDA E2B E4B 26B-A4B 31B		2026-04-09T17:56:14+00:00	Optimal running parameters for Google DeepMind's Gemma 4 family of models, covering inference settings, quantization choices, hardware requirements, thinking mode, multimodal configuration, and llama.cpp example commands.	reference gemma-4 llm inference llama-cpp local-ai	Gemma 4 — Optimal Running Parameters

Gemma 4 — Optimal Running Parameters

A reference for optimally running Google DeepMind's Gemma 4 family of models locally.

Model Variants

Variant	Architecture	Context	Modalities	Best Fit
E2B	Dense + PLE	128K	Text, Image, Audio	Phone / edge, ASR, speech translation
E4B	Dense + PLE	128K	Text, Image, Audio	Laptops, fast local multimodal
26B-A4B	MoE (4B active)	256K	Text, Image	Best speed/quality tradeoff
31B	Dense	256K	Text, Image	Strongest performance

Recommended Inference Parameters (All Variants)

These are Google's default Gemma 4 parameters — use them unless you have a specific reason to change:

Parameter	Value
temperature	1.0
top_p	0.95
top_k	64
Repetition / Presence Penalty	Disabled or 1.0 (only increase if you see looping)
End of Sentence Token	<turn|>

Context Length

• E2B / E4B: max 128K
• 26B-A4B / 31B: max 256K
• Practical tip: Start with 32K context for responsiveness, then increase as needed.

Quantization Recommendations

Variant	Recommended Quant	Alternative	Full Precision (BF16/FP16)
E2B	Q8_0 (8-bit)	Q4 (4 GB)	10 GB
E4B	Q8_0 (8-bit)	Q4 (5.5–6 GB)	16 GB
26B-A4B	UD-Q4_K_XL (Dynamic 4-bit)	8-bit (28–30 GB)	52 GB
31B	UD-Q4_K_XL (Dynamic 4-bit)	8-bit (34–38 GB)	62 GB

• Small models (E2B, E4B): prefer 8-bit for quality.
• Large models (26B-A4B, 31B): prefer Dynamic 4-bit (UD-Q4_K_XL) as the starting point.

Hardware Requirements (Total Memory: RAM + VRAM)

Variant	4-bit	8-bit	BF16/FP16
E2B	4 GB	5–8 GB	10 GB
E4B	5.5–6 GB	9–12 GB	16 GB
26B-A4B	16–18 GB	28–30 GB	52 GB
31B	17–20 GB	34–38 GB	62 GB

Rule of thumb: Total available memory should at least exceed the size of the quantized model. If it doesn't, llama.cpp can use partial RAM/disk offload but generation will be slower.

26B-A4B vs 31B — When to Choose Which

• 26B-A4B — Choose when RAM is limited. MoE architecture with 4B active parameters makes it faster. Slight quality tradeoff for speed.
• 31B — Choose when you have enough memory (≥20 GB for 4-bit) and want maximum quality. Slower inference.

Thinking Mode

Enabling Thinking

Add <|think|> at the start of the system prompt:

<|think|>
You are a careful coding assistant. Explain your answer clearly.

When enabled, the model outputs an internal reasoning channel before the final answer:

<|channel>thought
[internal reasoning]
<channel|>
[final answer]

Disabling Thinking

Use the llama-server flag (not llama-cli, which may be unreliable):

--chat-template-kwargs '{"enable_thinking":false}'

On Windows PowerShell:

--chat-template-kwargs "{\"enable_thinking\":false}"

Note: Even with thinking disabled, larger models may still emit an empty thought block (<|channel>thought<channel|>) before the answer. This is normal.

Multi-Turn Chat Rule

Only keep the final visible answer in chat history. Do not feed prior thought blocks back into subsequent turns.

Multimodal Settings

Image Placement

Always put images and/or audio before text in prompts. For video, pass frames first, then the instruction.

Visual Token Budgets

Budget	Use Case
70 / 140	Classification, captioning, fast video understanding
280 / 560	General multimodal chat, charts, screens, UI reasoning
1120	OCR, document parsing, handwriting, small text

Audio Limits

• Audio is only on E2B and E4B.
• Maximum audio length: 30 seconds.
• Maximum video length: 60 seconds (at 1 frame/second).

Critical Warnings

• ⚠️ Do NOT use CUDA 13.2 runtime for any GGUF — it will cause poor outputs.
• Use llama-server instead of llama-cli for thinking control (more reliable).

Example llama.cpp Commands

26B-A4B (Dynamic 4-bit)

./llama.cpp/llama-cli \
    -hf unsloth/gemma-4-26B-A4B-it-GGUF:UD-Q4_K_XL \
    --temp 1.0 --top-p 0.95 --top-k 64

31B (Dynamic 4-bit)

./llama.cpp/llama-cli \
    -hf unsloth/gemma-4-31B-it-GGUF:UD-Q4_K_XL \
    --temp 1.0 --top-p 0.95 --top-k 64

E4B (8-bit)

./llama.cpp/llama-cli \
    -hf unsloth/gemma-4-E4B-it-GGUF:Q8_0 \
    --temp 1.0 --top-p 0.95 --top-k 64

E2B (8-bit)

./llama.cpp/llama-cli \
    -hf unsloth/gemma-4-E2B-it-GGUF:Q8_0 \
    --temp 1.0 --top-p 0.95 --top-k 64

llama-server with Vision + Thinking

./llama.cpp/llama-server \
    --model unsloth/gemma-4-26B-A4B-it-GGUF/gemma-4-26B-A4B-it-UD-Q4_K_XL.gguf \
    --mmproj unsloth/gemma-4-26B-A4B-it-GGUF/mmproj-BF16.gguf \
    --temp 1.0 --top-p 0.95 --top-k 64 \
    --alias "unsloth/gemma-4-26B-A4B-it-GGUF" \
    --port 8001 \
    --chat-template-kwargs '{"enable_thinking":true}'

Sources

• Unsloth Gemma 4 Documentation (How to Run Locally)
• Gemma 4 GGUF collection: https://huggingface.co/collections/unsloth/gemma-4