esh-pfi-infrastructure/stacks/gemma4-charrp

gemma4-charrp — the char-rp seat (ana-ml2 GPU0)

google/gemma-4-26B-A4B-it, NVFP4, serving both halves of the char-rp pair on :8016. Replaced the dense G4-MeroMero-v2-31B-NVFP4A16 seat on 2026-08-24.

char-rp            non-thinking   ->  http://10.250.50.54:8016/v1
char-rp-reasoning  thinking       ->  http://10.250.50.54:8016/v1

Two LiteLLM aliases, one backend. They are not two seats — this trips people up, and it cost a peer a mis-attributed benchmark before it was noticed.

Three model directories, and they are not interchangeable

path	size	what it is
gemma4-26b-a4b-it-bf16	49 GB	Stock BF16. Unquantized. Cannot be served here — 48.10 GiB of weights against ~49 GiB of free GPU0 leaves nothing for KV cache. Its chat_template.jinja is the canonical upstream one; see below.
gemma4-26b-a4b-it-heretic-bf16	49 GB	QLoRA trainee base (operator's pick, 2026-08-24). llmfan46, Heretic v1.2.0 ARA, KL 0.1237, refusals 3/100.
gemma4-26b-a4b-it-abliterated-bf16	49 GB	Trainee alternate. TrevorJS, KL 0.09, refusals 1/100 effective and 5/686 cross-dataset — lowest measured damage of the field.
gemma4-26b-a4b-it-nvfp4	16 GB	What is served. RedHatAI, compressed-tensors, W4A4.
gemma4-26b-a4b-it-nvfp4a16	17 GB	Activation-axis control, for benching only. prithivMLmods, compressed-tensors, W4A16.

All under /tank/aimodels/, pulled by the gemma4-26b-*-dl.py scripts beside them with revisions pinned.

⚠ Third-party Gemma-4 builds ship STALE CHAT TEMPLATES — this is endemic

Verified by hash on 2026-08-24 across every third-party derivative pulled here. Not one of them ships upstream's template:

build	lines	sha256 (normalised)
upstream google/gemma-4-26B-A4B-it	390	6a1015c47ccfcfa6
RedHatAI NVFP4 (what is served)	389	6a1015c47ccfcfa6 — the only match
llmfan46 heretic	365	0a52be69cda5ab8a
TrevorJS abliterated	266	58c66fdee4afa297
jenerallee78 abliterated	266	58c66fdee4afa297
prithivMLmods NVFP4A16	266	58c66fdee4afa297

Three independent repos carrying the identical stale 266-line file means it propagated through the ecosystem, not that one packager slipped.

Consequences differ by use and both are silent:

• Serving — a different template renders a different prompt. This is why the production compose pins the template explicitly.
• Training — if the harness renders examples through base/chat_template.jinja, you train on a different prompt format than production serves. Train/serve skew, no error, and it presents as a tuning failure.

For both, point at the upstream file: /tank/aimodels/gemma4-26b-a4b-it-bf16/chat_template.jinja.

Choosing an abliterated base — compare on published damage, not on names

"Low damage" has a measurable proxy and the field spreads widely on it:

build	method	KL	refusals
TrevorJS	ARA-family	0.09	1/100 effective, 5/686 cross-dataset, manually audited
llmfan46	Heretic v1.2.0 ARA	0.1237	3/100
jenerallee78	ARA 2-pass	0.1299	7.7% StrongREJECT
huihui-ai	remove-refusals-with-transformers	none published	none published
trohrbaugh/heretic-ara	—	0.2999	31.4%
coder3101/heretic	—	0.4118	15.8%

Fleet anchor for reading those numbers: our own abliteration work found Heretic at KL 0.12 preserved the MTP head at 83.7% acceptance. Both staged builds sit at or below that, so neither is an extrapolation past what has been measured here.

huihui-ai is rejected on this stack despite being the best-known abliteration house: no published metrics, its card describes the method as "a crude, proof-of-concept implementation", it states both thinking and non-thinking modes were "completely abliterated", and its parameter count is 26,544,131,376 against upstream's 25,805,936,206 — roughly 738M unexplained extra. The operator's independent read is the same ("huihui produces garbage").

GPU0 is shared and the budgets must sum under ~0.92

vllm-gen runs at --gpu-memory-utilization 0.43 but actually holds ~45.6 GiB of the 94.97 GiB card — that flag sizes the KV cache and does not cover CUDA context, graphs and non-torch overhead. The predecessor seat sat at 0.51, the pair summed to 0.94, and on 2026-08-24 it stopped fitting and crash-looped 13 times with torch.OutOfMemoryError: ... 195.19 MiB is free.

This seat runs 0.47. Raising it means lowering gen's in the same change, and check the real numbers, not the flags:

nvidia-smi --query-compute-apps=pid,used_memory --format=csv

There is no room for a second concurrent seat on this card. That is why the A16 control below is a swap rather than a parallel deployment.

Running the A16 activation-axis control

Why it was run, and what it actually settled: a 2026-08-24 battery appeared to show a large contradiction-detection deficit with CoT off, which had the exact shape 4-bit input activations would produce on the most reasoning-dense task. The control ran and found activation precision close to free — every other task identical across the two builds. That quantization result stands.

⚠ The deficit it was chasing does not — the benchmark item was ill-posed. See "Superseded claims" at the end of this file. The procedure below is kept because the flip is a genuinely useful capability, not because the original question was sound.

⚠ The A16 build ships a STALE CHAT TEMPLATE and the run MUST override it. Verified by hash 2026-08-24: upstream google/gemma-4-26B-A4B-it is 390 lines, the A4 build's is 389 and byte-identical to upstream once trailing newlines are normalised, the A16 build's is 266 and is not. Upstream and A4 open the thinking path with {%- set enable_thinking = enable_thinking | default(false) -%}; the A16 template has no such set. Its tokenizer_config.json response_schema also lacks the thinking property. Served with its own template the two arms render different prompts, and a score delta could be the template rather than the activations.

Overriding is safe: the tokenizers agree — vocab identical at 262,144 entries, added_tokens identical — so the same template over the same vocab renders the same token ids. brokkr independently diffed every non-quantization config field of both builds against the upstream BF16 and found only transformers_version differing, which is a save-time library version rather than a model property. Residual risk stated honestly on both sides: config identity is not weight identity, and nobody has done a dequantization pass.

The procedure, ~2 minutes each way:

1. Add the template flag to compose.yaml (a no-op for production — the A4 build ships this exact file, so pinning it explicitly changes nothing and guards against precisely the staleness above):

   - --chat-template
   - ${GEMMA4_CHAT_TEMPLATE:-/tank/aimodels/gemma4-26b-a4b-it-nvfp4/chat_template.jinja}
   

   ⚠ If GEMMA4_MODEL is ever pointed at a different checkpoint, this default must move with it. A pinned template is only correct for the checkpoint it came from — that is the mistake the outgoing MeroMero seat's hand-patched template was warning about, inverted.

2. GEMMA4_MODEL=/tank/aimodels/gemma4-26b-a4b-it-nvfp4a16 in the host .env, then docker compose up -d --force-recreate vllm-gemma4-charrp.

3. Tell brokkr; they fire one CoT-off arm, 96 items, under a minute of wall clock, and report back.

4. Revert GEMMA4_MODEL to the nvfp4 path and recreate.

Only the CoT-off arm is worth running. With thinking on the model already scores 100% on every task it completes, and a ceiling cannot move.

⚠ Displacing this seat is an operator decision, not a routine one. It is live.

Rollback to MeroMero-v2

stacks/meromero-charrp/ is retained stopped in created state, labelled AI - Dormant. Both stacks bind :8016, so rollback is stop-then-start:

cd /opt/docker/compose/gemma4-charrp  && docker compose stop vllm-gemma4-charrp
cd /opt/docker/compose/meromero-charrp && docker compose up -d vllm-meromero-rp

Gemma-4 flags that are load-bearing

Carried over from the MeroMero seat because they are architecture-level, not checkpoint-level:

• --tool-call-parser gemma4 + --enable-auto-tool-choice — Gemma-4 emits its own native tool syntax, not the qwen3_coder XML the other seats use. Without these, any tools-bearing request 400s outright.
• --reasoning-parser gemma4 — absorbs the <|channel> thought markers; without it they leak into content verbatim on the post-tool turn (vllm #45834).
• --default-chat-template-kwargs '{"enable_thinking": false}' — mandatory companion to the reasoning parser. The parser defaults enable_thinking to TRUE, which pre-initialises the engine to REASONING, so all plain prose lands in reasoning_content with a null content and every char-rp consumer breaks. Production runs thinking off; the thinking route reaches it per-request.

Known defect, not ours

With thinking on, the model does not reliably terminate on constraint-following. brokkr measured 32 of 96 calls truncating at a 12,000-token cap, with all 16 constraint items among them, and inspected the traces: the reasoning is sound right up to the point it fails to stop. A non-termination defect, not a wrong-answer one. VLLM_USE_V2_MODEL_RUNNER=0 enables thinking_token_budget and is the first lever to try if a usable thinking seat is ever wanted — deliberately not applied here, since it costs the faster model runner for a mode production does not serve.

Superseded claims

Following the repo convention for quant work: when a recorded claim turns out wrong, it gets a dated row rather than a silent edit, so older notes elsewhere stop misleading people.

date	claim, as recorded	status
2026-08-24	"Gemma-4-26B-A4B-it scores 12% on contradiction detection with CoT off, against gen's 81% — the model owns the deficit."	RETRACTED same day. The benchmark item was ill-posed: it presented two mutually contradicting statements and asked for "the contradicting statement", but contradiction is symmetric — neither was more the contradicting one, and the model consistently named the absolute claim, a defensible reading the labelling scored wrong every time.
2026-08-24	"Domain tuning costs 43 points of contradiction detection."	RETRACTED. Rested entirely on the same item. On the corrected instrument the effect does not shrink, it reverses.
2026-08-24	All pre-fix T2 (contradiction) numbers for Gemma-4, MeroMero-v2, sec and gen.	VOID.

The tell, worth internalising: the score was BELOW CHANCE. 12% on a five-option task is under the 20% floor. A below-chance score indicts the instrument before it indicts the model, and that should be the first reaction rather than a late one. Neither side caught it until the individual items were read.

A second defect surfaced while fixing the first: all generators shared one RNG, so rewriting one task reshuffled every task after it. Each task now seeds from its own name.

What survived, and it is not nothing: the A16 control result holds — activation precision is close to free on this workload, with every other task identical across the W4A4 and W4A16 builds. The two staging confounds caught before the run (the misnamed A16 repos, the stale chat template) were real and independent of the item defect. The corrected picture, CoT off, n=96:

arm	core	T2	T6	median latency
char-rp (Gemma-4)	92.5%	94%	69%	0.24s
sec	92.5%	81%	81%	1.11s
gen	86.2%	50%	81%	0.33s

Gemma leads on the very axis it was suspected of failing. Its actual weak axis with thinking off is T6, spatial composition at 69% — and an earlier run showed CoT-on takes T6 to 100%, which nothing else benched has managed.