vh
3446367d5e
The template is now passed explicitly, defaulting to the A4 build's chat_template.jinja. That is a no-op for what is served — the A4 build ships that exact file, byte-identical to upstream google/gemma-4-26B-A4B-it once trailing newlines are normalised — and it permanently closes the class of bug found while staging the control: the A16 build ships a stale 266-line template against upstream's 390, with the thinking path built differently and no `thinking` property in its tokenizer_config response_schema. Serving each build with its own template would have moved a second axis. The control ran on the operator's greenlight and has been reverted. Seat is back on the W4A4 build, healthy, RestartCount 0, both aliases verified through the gateway — char-rp returns content with reasoning_content empty, char-rp-reasoning returns both. Result, since it settles a question this repo's config now encodes: activation precision does NOT explain the contradiction-detection deficit. Contradiction detection moved 12% -> 19% between W4A4 and W4A16, which at n=16 is 2/16 -> 3/16 — one item — against gen's 81% on identical items. Every other task is identical across the two builds and the core difference is 2.6 points carried almost entirely by two single items. brokkr-smithy-dev pre-registered that a null result would be the robust branch, because a hidden third axis would tend to create a delta rather than suppress one, so the conclusion survives the residual doubt neither side could close without a dequantization pass. The practical upshot for future scheme choices: W4A4 costs less on this workload than the caution warranted. The caution was still correct to have. Displaced production for 3.7 seconds of measurement plus two container recreates. The A16 build and the BF16 tuning base both stay on disk with the runbook in the stack README, so re-running is a two-minute flip.
138 lines
6.2 KiB
YAML
138 lines
6.2 KiB
YAML
# gemma4-charrp — google/gemma-4-26B-A4B-it NVFP4 char-rp seat on ana-ml2 GPU0.
|
|
#
|
|
# Straight-across replacement for `meromero-charrp` (operator, 2026-08-24): same
|
|
# port, same served-model-names, so every gateway route and consumer is
|
|
# unchanged. The outgoing G4-MeroMero-v2-31B-NVFP4A16 stack is retained stopped
|
|
# for rollback — see stacks/meromero-charrp/.
|
|
#
|
|
# WHY THE SWAP: the seat's requirements now include chain-of-thought, which makes
|
|
# throughput MORE critical rather than less — the user waits through the whole
|
|
# reasoning block before the first visible token. The incumbent was the DENSE
|
|
# 31B at ~40.7 tok/s @32K; this is the 26B-A4B MoE at ~114 tok/s @32K
|
|
# (brokkr-smithy-dev's measurements). Same Gemma-4 family, so the parser and
|
|
# chat-template machinery below carries over unchanged.
|
|
#
|
|
# ⚠ THIS IS THE QUANT, NOT THE TUNING BASE. The BF16 weights live at
|
|
# /tank/aimodels/gemma4-26b-a4b-it-bf16 and are for QLoRA — 48.10 GiB of BF16
|
|
# weights CANNOT be served here: GPU0 is shared with vllm-gen, which holds
|
|
# ~45.6 GiB of the 94.97 GiB card, leaving ~49 GiB. Weights alone would eat all
|
|
# of it and the engine would die at KV-cache allocation. Do not "simplify" this
|
|
# to the BF16 path.
|
|
#
|
|
# ⚠ SCHEME IS W4A4, NOT W4A16. The RedHatAI quant declares 4-bit
|
|
# input_activations (compressed-tensors, format nvfp4-pack-quantized). The
|
|
# incumbent was NVFP4**A16**. Faster, and NOT a like-for-like quality
|
|
# comparison on the activation axis — say so when benching the two.
|
|
#
|
|
# Tunables in .env.
|
|
|
|
name: gemma4-charrp
|
|
|
|
services:
|
|
vllm-gemma4-charrp:
|
|
image: ${GEMMA4_IMAGE:-vllm/vllm-openai:latest}
|
|
container_name: ${GEMMA4_CONTAINER:-vllm-gemma4-charrp}
|
|
restart: unless-stopped
|
|
ipc: host
|
|
ports:
|
|
- "${GEMMA4_PORT:-8016}:8000"
|
|
volumes:
|
|
- /tank/aimodels:/tank/aimodels
|
|
environment:
|
|
- PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
|
|
- VLLM_API_KEY=${API_KEY:-}
|
|
command:
|
|
- ${GEMMA4_MODEL:-/tank/aimodels/gemma4-26b-a4b-it-nvfp4}
|
|
- --quantization
|
|
- compressed-tensors
|
|
# UNCHANGED FROM THE OUTGOING SEAT, ON PURPOSE. Both names are live
|
|
# LiteLLM routes at http://10.250.50.54:8016/v1 — `char-rp` ->
|
|
# hosted_vllm/char-rp and `char-rp-reasoning` -> hosted_vllm/char-rp-thinking.
|
|
# They are two aliases onto ONE seat, not two seats. Renaming either
|
|
# breaks the gateway silently.
|
|
- --served-model-name
|
|
- char-rp
|
|
- char-rp-thinking
|
|
# Tool-calling: Gemma-4 emits its OWN native syntax
|
|
# (<|tool_call>call:name{...}<tool_call|>), NOT the qwen3_coder XML the
|
|
# other seats use. Without these two flags any tools-bearing request 400s
|
|
# outright. Architecture-level, so it carries over from MeroMero unchanged.
|
|
- --tool-call-parser
|
|
- gemma4
|
|
- --enable-auto-tool-choice
|
|
# The gemma4 REASONING parser absorbs the <|channel>...<channel|> thought
|
|
# markers; without it they leak into `content` verbatim on the post-tool
|
|
# turn (upstream vllm #45834).
|
|
- --reasoning-parser
|
|
- gemma4
|
|
# MANDATORY COMPANION TO THE REASONING PARSER. The parser reads
|
|
# enable_thinking from chat_template_kwargs and DEFAULTS IT TO TRUE. True
|
|
# makes is_reasoning_end() return False at a new turn, which pre-initialises
|
|
# the engine to REASONING — so ALL plain RP prose lands in
|
|
# reasoning_content with a NULL content, breaking every char-rp consumer.
|
|
# Do not remove. Thinking is still reachable per-request via
|
|
# chat_template_kwargs {"enable_thinking": true}, which is what the
|
|
# `char-rp-reasoning` alias exercises.
|
|
- --default-chat-template-kwargs
|
|
- '{"enable_thinking": false}'
|
|
# TEMPLATE PINNED EXPLICITLY, AND IT IS A NO-OP FOR THE DEFAULT MODEL.
|
|
# The A4 build ships this exact file — verified byte-identical to upstream
|
|
# google/gemma-4-26B-A4B-it once trailing newlines are normalised (390 vs
|
|
# 389 lines, same hash) — so pinning it changes nothing about what is
|
|
# served and permanently removes a real class of bug.
|
|
#
|
|
# ⚠ WHY IT IS PINNED AT ALL: the A16 control build at
|
|
# /tank/aimodels/gemma4-26b-a4b-it-nvfp4a16 ships a STALE 266-line
|
|
# template. Upstream and A4 open the thinking path with
|
|
# `{%- set enable_thinking = enable_thinking | default(false) -%}`; the
|
|
# A16 one has no such set, and its tokenizer_config response_schema lacks
|
|
# the `thinking` property. Serving it with its own template would render a
|
|
# DIFFERENT PROMPT, turning a one-axis activation-precision control into a
|
|
# two-axis comparison — a result that would look like a finding.
|
|
# Safe to force across both builds because the tokenizers are identical:
|
|
# vocab 262,144 entries, added_tokens identical.
|
|
#
|
|
# ⚠ IF GEMMA4_MODEL EVER POINTS AT A DIFFERENT CHECKPOINT, THIS DEFAULT
|
|
# MUST MOVE WITH IT. A pinned template is only correct for the checkpoint
|
|
# it came from. That is the inverse of the mistake the outgoing MeroMero
|
|
# seat warned about, where a hand-patched template was assumed to transfer.
|
|
- --chat-template
|
|
- ${GEMMA4_CHAT_TEMPLATE:-/tank/aimodels/gemma4-26b-a4b-it-nvfp4/chat_template.jinja}
|
|
- --max-model-len
|
|
- "${GEMMA4_MAX_MODEL_LEN:-262144}"
|
|
- --max-num-seqs
|
|
- "${GEMMA4_MAX_NUM_SEQS:-32}"
|
|
- --gpu-memory-utilization
|
|
- "${GEMMA4_GPU_MEM_UTIL:-0.47}"
|
|
- --kv-cache-dtype
|
|
- fp8
|
|
- --trust-remote-code
|
|
deploy:
|
|
resources:
|
|
reservations:
|
|
devices:
|
|
- driver: nvidia
|
|
device_ids:
|
|
- "${GEMMA4_GPU_ID:-0}"
|
|
capabilities:
|
|
- gpu
|
|
healthcheck:
|
|
test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
|
|
interval: 30s
|
|
timeout: 10s
|
|
retries: 3
|
|
start_period: 600s
|
|
networks:
|
|
- tnet
|
|
labels:
|
|
- homepage.group=AI - Inference
|
|
- homepage.name=char-rp (Gemma-4 26B-A4B NVFP4, MoE)
|
|
- homepage.icon=mdi-drama-masks
|
|
- homepage.description=gemma-4-26B-A4B-it NVFP4 MoE prose+CoT seat, 256K (ana-ml2 GPU0)
|
|
- homepage.href=http://10.250.50.54:${GEMMA4_PORT:-8016}/docs
|
|
|
|
networks:
|
|
tnet:
|
|
name: traefik-net
|
|
external: true
|