Straight-across replacement of the dense G4-MeroMero-v2-31B-NVFP4A16 seat with
google/gemma-4-26B-A4B-it on ana-ml2 GPU0. Port, served-model-names and every
gateway route are unchanged, so no consumer sees a difference in addressing:
`char-rp` -> hosted_vllm/char-rp and `char-rp-reasoning` ->
hosted_vllm/char-rp-thinking, both still :8016. The seat's requirements now
include chain-of-thought, which makes throughput more critical rather than less
— the user waits through the whole reasoning block before the first visible
token, and the MoE measures ~114 tok/s @32K against the dense 31B's ~40.7.
Both artifacts are on disk and they are NOT interchangeable. The BF16 weights
(/tank/aimodels/gemma4-26b-a4b-it-bf16, 49 GB) are the QLoRA tuning base, since
QLoRA does its own quantization. They CANNOT be served here: 48.10 GiB of
weights against ~49 GiB of free GPU0 leaves nothing for KV cache, and the
engine would die at allocation exactly the way the predecessor did this
afternoon. The serving copy is RedHatAI/gemma-4-26B-A4B-it-NVFP4 (16 GB),
chosen over the other -it quants because it is compressed-tensors
(nvfp4-pack-quantized) — the same loader path the outgoing seat used — from the
llm-compressor team at 357k downloads. The nvidia/ repo is the base rather than
-it, and the thinking channel lives in the instruction-tuned weights.
Smaller weights at the same 0.47 memory budget buy a much larger KV pool:
27.37 GiB and 1,724,110 tokens, against the predecessor's 371,023 at the same
budget. That is 6.5 full-length 262K sequences concurrent rather than 1.4.
The gemma4 tool-call parser, reasoning parser and the enable_thinking:false
default all carry over unchanged — they are architecture-level, not
checkpoint-level. The --chat-template override does NOT carry over: MeroMero
pointed at a jinja hand-patched against that checkpoint, and this model ships
its own. Verified that dropping it did not reintroduce the failure that flag
existed to prevent — non-thinking prose lands in content with reasoning_content
empty, and the thinking alias populates reasoning_content with content
carrying the answer.
⚠ Scheme differs from the incumbent and the bench should say so: this quant
declares 4-bit input activations (W4A4) where the outgoing seat was NVFP4A16.
Faster, and not like-for-like on the activation axis.
meromero-charrp is retained stopped in `created` state and relabelled to
AI - Dormant, per the house rollback pattern. Both stacks want :8016, so
rolling back means stopping the gemma4 seat first.
The multi-turn Gemma-4 CoT problem is solved. One MeroMero-v2 seat, one weight set,
two aliases: char-rp (prose) + char-rp-reasoning (streaming chain-of-thought).
The winning stack, traced from vLLM source by the four-arm brokkr/dwarf panel:
- vllm/vllm-openai:v0.26.0 — ships transformers 5.14.1 natively, below the
head_dim guard, so Gemma-4-31B loads with no pin and no custom image. It also
carries the #48217 streaming pre-arm fix.
- A patched chat template whose enable_thinking:true branch force-opens a BARE
<|channel> (not <|channel>thought\n -- full-open defeats _preprocess_feed's
injection). --chat-template override, no re-quant.
- Two served-names char-rp / char-rp-thinking; --reasoning-parser gemma4;
default enable_thinking:false. LiteLLM char-rp -> prose, char-rp-reasoning ->
the thinking served-name with enable_thinking:true.
Verified: streaming CoT split 6/6 direct on :8016 and 3/3 through the gateway;
char-rp prose clean on both transports with no trailing-token leak.
Two hard-won facts recorded in persistent-memory:
- STREAMING ONLY. Non-streaming can't split -- extract_reasoning never receives
prompt_token_ids so the pre-arm can't fire (a vLLM one-shot bug unchanged
across v0.24-0.27). Fine here: Lobe/OWUI stream. Upstream PR #49797 fixes
non-streaming too, landing ~v0.28.0 -- then it's a clean image bump.
- KEY-NAME TRAP: vLLM streams reasoning in delta.reasoning; LiteLLM normalizes
to delta.reasoning_content. I lost two false-negative test rounds to this.
Canonical: stacks/meromero-charrp/ (compose + patched_chat_template.jinja) and
stacks/litellm/conf/config.yaml. Rollback is the .env image line + dropping
--chat-template.
The char-rp seat shipped with no tool-call parser at all, so every
tools-bearing request was rejected outright:
400 "auto" tool choice requires --enable-auto-tool-choice and
--tool-call-parser to be set
MeroMero-v2 is Gemma-4, which emits its own native
`<|tool_call>call:name{...}<tool_call|>` syntax rather than the
qwen3_coder XML the Qwen-family seats use. vLLM 0.24 ships a matching
`gemma4` parser whose TOOL_CALL_START/END, CHANNEL_START/END and escape
token constants line up with this tokenizer's etc/eoc/escape tokens
exactly.
Three flags, and they are a set:
- --tool-call-parser gemma4 + --enable-auto-tool-choice: the fix proper.
- --reasoning-parser gemma4: without it the post-tool-response turn
leaks a literal `<|channel>thought\n<channel|>` prefix into content
(upstream vllm #45834 — the chat template leaves the prompt inside an
open channel block).
- --default-chat-template-kwargs '{"enable_thinking": false}': MANDATORY
companion to the reasoning parser. The parser reads enable_thinking
from chat_template_kwargs and defaults it to True
(vllm/parser/gemma4.py:439); True makes is_reasoning_end() return
False at a new turn, pre-initialising the engine to REASONING, which
routes ALL plain RP prose into reasoning_content and returns a null
content — breaking every char-rp consumer. This template already
defaults enable_thinking to false (chat_template.jinja:350), so
passing it explicitly renders a byte-identical prompt (verified across
plain / tools / post-tool-response / system-prompt shapes). It changes
generation not at all; it only corrects the parser state machine.
Verified green on the live seat after deploy: tool call streaming and
non-streaming, tool-result round-trip (leak gone), plain prose in
content with reasoning null, vision unchanged.
vh