Files

T

vh 8d6a9390de docs(gemma4-charrp): RETRACT the contradiction-deficit claim — the item was ill-posed

Supersedes what commit 3446367 recorded. That message stated the A16 control
"settles a question" and quoted 12% contradiction detection against gen's 81%.
The quantization half of it stands; the deficit it was measuring does not exist.

brokkr-smithy-dev retracted the finding after the operator asked to see the
individual items. The task presented two mutually contradicting statements and
asked for "the contradicting statement" — but CONTRADICTION IS SYMMETRIC.
Neither statement was more the contradicting one, the model had no way to know
which had been inserted, and it consistently named the absolute claim: a
defensible reading that the labelling scored wrong on every single item.

The tell was there and both of us walked past it: 12% on a five-option task is
BELOW THE 20% CHANCE FLOOR. A below-chance score indicts the instrument before
it indicts the model. That should be the first reaction to a below-chance
result, not a late one, and it is now written into the README as such.

Retracted: "the model owns the contradiction deficit"; "domain tuning costs 43
points of contradiction detection" (which on a sound instrument does not shrink
but REVERSES); and every pre-fix T2 number for Gemma-4, MeroMero-v2, sec and
gen. A second defect surfaced during the fix — all generators shared one RNG, so
rewriting one task reshuffled every task after it.

What survives is real and worth separating out: the A16 control result holds.
Activation precision is close to free on this workload, every other task
identical across the W4A4 and W4A16 builds. The two staging confounds caught
before the run — the two Hub repos named NVFP4A16 that declare 4-bit
activations, and the stale chat template — were independent of the item defect
and remain load-bearing. On the corrected instrument Gemma leads the very axis
it was suspected of failing (94% against sec's 81% and gen's 50%); its actual
weak axis with thinking off is spatial composition at 69%.

Recorded as a dated superseded-claims table rather than a silent edit, per the
repo's quant-work convention, so notes elsewhere stop misleading people.

2026-08-24 13:07:58 -07:00

9.0 KiB

Raw Blame History

gemma4-charrp — the char-rp seat (ana-ml2 GPU0)

google/gemma-4-26B-A4B-it, NVFP4, serving both halves of the char-rp pair on :8016. Replaced the dense G4-MeroMero-v2-31B-NVFP4A16 seat on 2026-08-24.

char-rp            non-thinking   ->  http://10.250.50.54:8016/v1
char-rp-reasoning  thinking       ->  http://10.250.50.54:8016/v1

Two LiteLLM aliases, one backend. They are not two seats — this trips people up, and it cost a peer a mis-attributed benchmark before it was noticed.

Three model directories, and they are not interchangeable

path	size	what it is
`gemma4-26b-a4b-it-bf16`	49 GB	QLoRA tuning base. BF16, unquantized. Cannot be served here — 48.10 GiB of weights against ~49 GiB of free GPU0 leaves nothing for KV cache.
`gemma4-26b-a4b-it-nvfp4`	16 GB	What is served. RedHatAI, compressed-tensors, W4A4.
`gemma4-26b-a4b-it-nvfp4a16`	17 GB	Activation-axis control, for benching only. prithivMLmods, compressed-tensors, W4A16.

All under /tank/aimodels/, pulled by the gemma4-26b-*-dl.py scripts beside them with revisions pinned.

GPU0 is shared and the budgets must sum under ~0.92

vllm-gen runs at --gpu-memory-utilization 0.43 but actually holds ~45.6 GiB of the 94.97 GiB card — that flag sizes the KV cache and does not cover CUDA context, graphs and non-torch overhead. The predecessor seat sat at 0.51, the pair summed to 0.94, and on 2026-08-24 it stopped fitting and crash-looped 13 times with torch.OutOfMemoryError: ... 195.19 MiB is free.

This seat runs 0.47. Raising it means lowering gen's in the same change, and check the real numbers, not the flags:

nvidia-smi --query-compute-apps=pid,used_memory --format=csv

There is no room for a second concurrent seat on this card. That is why the A16 control below is a swap rather than a parallel deployment.

Running the A16 activation-axis control

Why it was run, and what it actually settled: a 2026-08-24 battery appeared to show a large contradiction-detection deficit with CoT off, which had the exact shape 4-bit input activations would produce on the most reasoning-dense task. The control ran and found activation precision close to free — every other task identical across the two builds. That quantization result stands.

⚠ The deficit it was chasing does not — the benchmark item was ill-posed. See "Superseded claims" at the end of this file. The procedure below is kept because the flip is a genuinely useful capability, not because the original question was sound.

⚠ The A16 build ships a STALE CHAT TEMPLATE and the run MUST override it. Verified by hash 2026-08-24: upstream google/gemma-4-26B-A4B-it is 390 lines, the A4 build's is 389 and byte-identical to upstream once trailing newlines are normalised, the A16 build's is 266 and is not. Upstream and A4 open the thinking path with {%- set enable_thinking = enable_thinking | default(false) -%}; the A16 template has no such set. Its tokenizer_config.json response_schema also lacks the thinking property. Served with its own template the two arms render different prompts, and a score delta could be the template rather than the activations.

Overriding is safe: the tokenizers agree — vocab identical at 262,144 entries, added_tokens identical — so the same template over the same vocab renders the same token ids. brokkr independently diffed every non-quantization config field of both builds against the upstream BF16 and found only transformers_version differing, which is a save-time library version rather than a model property. Residual risk stated honestly on both sides: config identity is not weight identity, and nobody has done a dequantization pass.

The procedure, ~2 minutes each way:

Add the template flag to compose.yaml (a no-op for production — the A4 build ships this exact file, so pinning it explicitly changes nothing and guards against precisely the staleness above):
```
- --chat-template
- ${GEMMA4_CHAT_TEMPLATE:-/tank/aimodels/gemma4-26b-a4b-it-nvfp4/chat_template.jinja}
```
⚠ If GEMMA4_MODEL is ever pointed at a different checkpoint, this default must move with it. A pinned template is only correct for the checkpoint it came from — that is the mistake the outgoing MeroMero seat's hand-patched template was warning about, inverted.
GEMMA4_MODEL=/tank/aimodels/gemma4-26b-a4b-it-nvfp4a16 in the host .env, then docker compose up -d --force-recreate vllm-gemma4-charrp.
Tell brokkr; they fire one CoT-off arm, 96 items, under a minute of wall clock, and report back.
Revert GEMMA4_MODEL to the nvfp4 path and recreate.

Only the CoT-off arm is worth running. With thinking on the model already scores 100% on every task it completes, and a ceiling cannot move.

⚠ Displacing this seat is an operator decision, not a routine one. It is live.

Rollback to MeroMero-v2

stacks/meromero-charrp/ is retained stopped in created state, labelled AI - Dormant. Both stacks bind :8016, so rollback is stop-then-start:

cd /opt/docker/compose/gemma4-charrp  && docker compose stop vllm-gemma4-charrp
cd /opt/docker/compose/meromero-charrp && docker compose up -d vllm-meromero-rp

Gemma-4 flags that are load-bearing

Carried over from the MeroMero seat because they are architecture-level, not checkpoint-level:

--tool-call-parser gemma4 + --enable-auto-tool-choice — Gemma-4 emits its own native tool syntax, not the qwen3_coder XML the other seats use. Without these, any tools-bearing request 400s outright.
--reasoning-parser gemma4 — absorbs the <|channel> thought markers; without it they leak into content verbatim on the post-tool turn (vllm #45834).
--default-chat-template-kwargs '{"enable_thinking": false}' — mandatory companion to the reasoning parser. The parser defaults enable_thinking to TRUE, which pre-initialises the engine to REASONING, so all plain prose lands in reasoning_content with a null content and every char-rp consumer breaks. Production runs thinking off; the thinking route reaches it per-request.

Known defect, not ours

With thinking on, the model does not reliably terminate on constraint-following. brokkr measured 32 of 96 calls truncating at a 12,000-token cap, with all 16 constraint items among them, and inspected the traces: the reasoning is sound right up to the point it fails to stop. A non-termination defect, not a wrong-answer one. VLLM_USE_V2_MODEL_RUNNER=0 enables thinking_token_budget and is the first lever to try if a usable thinking seat is ever wanted — deliberately not applied here, since it costs the faster model runner for a mode production does not serve.

Superseded claims

Following the repo convention for quant work: when a recorded claim turns out wrong, it gets a dated row rather than a silent edit, so older notes elsewhere stop misleading people.

date	claim, as recorded	status
2026-08-24	"Gemma-4-26B-A4B-it scores 12% on contradiction detection with CoT off, against gen's 81% — the model owns the deficit."	RETRACTED same day. The benchmark item was ill-posed: it presented two mutually contradicting statements and asked for "the contradicting statement", but contradiction is symmetric — neither was more the contradicting one, and the model consistently named the absolute claim, a defensible reading the labelling scored wrong every time.
2026-08-24	"Domain tuning costs 43 points of contradiction detection."	RETRACTED. Rested entirely on the same item. On the corrected instrument the effect does not shrink, it reverses.
2026-08-24	All pre-fix T2 (contradiction) numbers for Gemma-4, MeroMero-v2, `sec` and `gen`.	VOID.

The tell, worth internalising: the score was BELOW CHANCE. 12% on a five-option task is under the 20% floor. A below-chance score indicts the instrument before it indicts the model, and that should be the first reaction rather than a late one. Neither side caught it until the individual items were read.

A second defect surfaced while fixing the first: all generators shared one RNG, so rewriting one task reshuffled every task after it. Each task now seeds from its own name.

What survived, and it is not nothing: the A16 control result holds — activation precision is close to free on this workload, with every other task identical across the W4A4 and W4A16 builds. The two staging confounds caught before the run (the misnamed A16 repos, the stale chat template) were real and independent of the item defect. The corrected picture, CoT off, n=96:

arm	core	T2	T6	median latency
`char-rp` (Gemma-4)	92.5%	94%	69%	0.24s
`sec`	92.5%	81%	81%	1.11s
`gen`	86.2%	50%	81%	0.33s

Gemma leads on the very axis it was suspected of failing. Its actual weak axis with thinking off is T6, spatial composition at 69% — and an earlier run showed CoT-on takes T6 to 100%, which nothing else benched has managed.

9.0 KiB Raw Blame History