The template is now passed explicitly, defaulting to the A4 build's
chat_template.jinja. That is a no-op for what is served — the A4 build ships
that exact file, byte-identical to upstream google/gemma-4-26B-A4B-it once
trailing newlines are normalised — and it permanently closes the class of bug
found while staging the control: the A16 build ships a stale 266-line template
against upstream's 390, with the thinking path built differently and no
`thinking` property in its tokenizer_config response_schema. Serving each build
with its own template would have moved a second axis.
The control ran on the operator's greenlight and has been reverted. Seat is back
on the W4A4 build, healthy, RestartCount 0, both aliases verified through the
gateway — char-rp returns content with reasoning_content empty,
char-rp-reasoning returns both.
Result, since it settles a question this repo's config now encodes: activation
precision does NOT explain the contradiction-detection deficit. Contradiction
detection moved 12% -> 19% between W4A4 and W4A16, which at n=16 is 2/16 -> 3/16
— one item — against gen's 81% on identical items. Every other task is identical
across the two builds and the core difference is 2.6 points carried almost
entirely by two single items. brokkr-smithy-dev pre-registered that a null
result would be the robust branch, because a hidden third axis would tend to
create a delta rather than suppress one, so the conclusion survives the residual
doubt neither side could close without a dequantization pass.
The practical upshot for future scheme choices: W4A4 costs less on this workload
than the caution warranted. The caution was still correct to have.
Displaced production for 3.7 seconds of measurement plus two container
recreates. The A16 build and the BF16 tuning base both stay on disk with the
runbook in the stack README, so re-running is a two-minute flip.
Straight-across replacement of the dense G4-MeroMero-v2-31B-NVFP4A16 seat with
google/gemma-4-26B-A4B-it on ana-ml2 GPU0. Port, served-model-names and every
gateway route are unchanged, so no consumer sees a difference in addressing:
`char-rp` -> hosted_vllm/char-rp and `char-rp-reasoning` ->
hosted_vllm/char-rp-thinking, both still :8016. The seat's requirements now
include chain-of-thought, which makes throughput more critical rather than less
— the user waits through the whole reasoning block before the first visible
token, and the MoE measures ~114 tok/s @32K against the dense 31B's ~40.7.
Both artifacts are on disk and they are NOT interchangeable. The BF16 weights
(/tank/aimodels/gemma4-26b-a4b-it-bf16, 49 GB) are the QLoRA tuning base, since
QLoRA does its own quantization. They CANNOT be served here: 48.10 GiB of
weights against ~49 GiB of free GPU0 leaves nothing for KV cache, and the
engine would die at allocation exactly the way the predecessor did this
afternoon. The serving copy is RedHatAI/gemma-4-26B-A4B-it-NVFP4 (16 GB),
chosen over the other -it quants because it is compressed-tensors
(nvfp4-pack-quantized) — the same loader path the outgoing seat used — from the
llm-compressor team at 357k downloads. The nvidia/ repo is the base rather than
-it, and the thinking channel lives in the instruction-tuned weights.
Smaller weights at the same 0.47 memory budget buy a much larger KV pool:
27.37 GiB and 1,724,110 tokens, against the predecessor's 371,023 at the same
budget. That is 6.5 full-length 262K sequences concurrent rather than 1.4.
The gemma4 tool-call parser, reasoning parser and the enable_thinking:false
default all carry over unchanged — they are architecture-level, not
checkpoint-level. The --chat-template override does NOT carry over: MeroMero
pointed at a jinja hand-patched against that checkpoint, and this model ships
its own. Verified that dropping it did not reintroduce the failure that flag
existed to prevent — non-thinking prose lands in content with reasoning_content
empty, and the thinking alias populates reasoning_content with content
carrying the answer.
⚠ Scheme differs from the incumbent and the bench should say so: this quant
declares 4-bit input activations (W4A4) where the outgoing seat was NVFP4A16.
Faster, and not like-for-like on the activation axis.
meromero-charrp is retained stopped in `created` state and relabelled to
AI - Dormant, per the house rollback pattern. Both stacks want :8016, so
rolling back means stopping the gemma4 seat first.
vh