vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity

feat(char-rp): swap the seat to the Gemma-4 26B-A4B MoE, NVFP4, same port

Browse Source 
Straight-across replacement of the dense G4-MeroMero-v2-31B-NVFP4A16 seat with
google/gemma-4-26B-A4B-it on ana-ml2 GPU0. Port, served-model-names and every
gateway route are unchanged, so no consumer sees a difference in addressing:
`char-rp` -> hosted_vllm/char-rp and `char-rp-reasoning` ->
hosted_vllm/char-rp-thinking, both still :8016. The seat's requirements now
include chain-of-thought, which makes throughput more critical rather than less
— the user waits through the whole reasoning block before the first visible
token, and the MoE measures ~114 tok/s @32K against the dense 31B's ~40.7.

Both artifacts are on disk and they are NOT interchangeable. The BF16 weights
(/tank/aimodels/gemma4-26b-a4b-it-bf16, 49 GB) are the QLoRA tuning base, since
QLoRA does its own quantization. They CANNOT be served here: 48.10 GiB of
weights against ~49 GiB of free GPU0 leaves nothing for KV cache, and the
engine would die at allocation exactly the way the predecessor did this
afternoon. The serving copy is RedHatAI/gemma-4-26B-A4B-it-NVFP4 (16 GB),
chosen over the other -it quants because it is compressed-tensors
(nvfp4-pack-quantized) — the same loader path the outgoing seat used — from the
llm-compressor team at 357k downloads. The nvidia/ repo is the base rather than
-it, and the thinking channel lives in the instruction-tuned weights.

Smaller weights at the same 0.47 memory budget buy a much larger KV pool:
27.37 GiB and 1,724,110 tokens, against the predecessor's 371,023 at the same
budget. That is 6.5 full-length 262K sequences concurrent rather than 1.4.

The gemma4 tool-call parser, reasoning parser and the enable_thinking:false
default all carry over unchanged — they are architecture-level, not
checkpoint-level. The --chat-template override does NOT carry over: MeroMero
pointed at a jinja hand-patched against that checkpoint, and this model ships
its own. Verified that dropping it did not reintroduce the failure that flag
existed to prevent — non-thinking prose lands in content with reasoning_content
empty, and the thinking alias populates reasoning_content with content
carrying the answer.

⚠ Scheme differs from the incumbent and the bench should say so: this quant
declares 4-bit input activations (W4A4) where the outgoing seat was NVFP4A16.
Faster, and not like-for-like on the activation axis.

meromero-charrp is retained stopped in `created` state and relabelled to
AI - Dormant, per the house rollback pattern. Both stacks want :8016, so
rolling back means stopping the gemma4 seat first.
This commit is contained in:
vh
2026-08-24 12:02:59 -07:00
parent 850e0c3351
commit 27155c0f3b
3 changed files with 158 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files
stacks/gemma4-charrp/.env.example
+31
View File
@@ -0,0 +1,31 @@
# gemma4-charrp — char-rp seat on ana-ml2 GPU0. Real .env lives on the host.
# ⚠ GPU0 IS SHARED WITH `vllm-gen`. gen runs at --gpu-memory-utilization 0.43
# but actually holds ~45.6 GiB of the 94.97 GiB card — that flag sizes the KV
# cache and does NOT cover CUDA context, graphs and non-torch overhead. The
# predecessor seat sat at 0.51, the pair summed to 0.94, and on 2026-08-24 it
# stopped fitting and crash-looped 13 times with
# `torch.OutOfMemoryError: ... 195.19 MiB is free`.
#
# 0.47 keeps ~4.8 GiB of real margin. This model's weights are only ~15.3 GiB
# (NVFP4) against the predecessor's ~19.5 GiB, so the same budget buys MORE KV
# cache than before, not less. Raising this means lowering gen's in the same
# change — and check the real numbers, not the flags:
# nvidia-smi --query-compute-apps=pid,used_memory --format=csv
GEMMA4_GPU_MEM_UTIL=0.47
# Native context. config.json declares max_position_embeddings 262144, same as
# the outgoing seat, so this is a straight-across swap on context too.
GEMMA4_MAX_MODEL_LEN=262144
GEMMA4_MAX_NUM_SEQS=32
# ⚠ THE NVFP4 QUANT, NOT THE BF16. /tank/aimodels/gemma4-26b-a4b-it-bf16 is the
# QLoRA tuning base and is 48.10 GiB of weights — it does not fit beside gen.
GEMMA4_MODEL=/tank/aimodels/gemma4-26b-a4b-it-nvfp4
GEMMA4_PORT=8016
GEMMA4_GPU_ID=0
GEMMA4_CONTAINER=vllm-gemma4-charrp
# Same value as every other vLLM seat on this host — the gateway presents it.
API_KEY=
stacks/gemma4-charrp/compose.yaml
+120
View File
@@ -0,0 +1,120 @@
# gemma4-charrp — google/gemma-4-26B-A4B-it NVFP4 char-rp seat on ana-ml2 GPU0.
#
# Straight-across replacement for `meromero-charrp` (operator, 2026-08-24): same
# port, same served-model-names, so every gateway route and consumer is
# unchanged. The outgoing G4-MeroMero-v2-31B-NVFP4A16 stack is retained stopped
# for rollback — see stacks/meromero-charrp/.
#
# WHY THE SWAP: the seat's requirements now include chain-of-thought, which makes
# throughput MORE critical rather than less — the user waits through the whole
# reasoning block before the first visible token. The incumbent was the DENSE
# 31B at ~40.7 tok/s @32K; this is the 26B-A4B MoE at ~114 tok/s @32K
# (brokkr-smithy-dev's measurements). Same Gemma-4 family, so the parser and
# chat-template machinery below carries over unchanged.
#
# ⚠ THIS IS THE QUANT, NOT THE TUNING BASE. The BF16 weights live at
# /tank/aimodels/gemma4-26b-a4b-it-bf16 and are for QLoRA — 48.10 GiB of BF16
# weights CANNOT be served here: GPU0 is shared with vllm-gen, which holds
# ~45.6 GiB of the 94.97 GiB card, leaving ~49 GiB. Weights alone would eat all
# of it and the engine would die at KV-cache allocation. Do not "simplify" this
# to the BF16 path.
#
# ⚠ SCHEME IS W4A4, NOT W4A16. The RedHatAI quant declares 4-bit
# input_activations (compressed-tensors, format nvfp4-pack-quantized). The
# incumbent was NVFP4**A16**. Faster, and NOT a like-for-like quality
# comparison on the activation axis — say so when benching the two.
#
# Tunables in .env.
name: gemma4-charrp
services:
vllm-gemma4-charrp:
image: ${GEMMA4_IMAGE:-vllm/vllm-openai:latest}
container_name: ${GEMMA4_CONTAINER:-vllm-gemma4-charrp}
restart: unless-stopped
ipc: host
ports:
- "${GEMMA4_PORT:-8016}:8000"
volumes:
- /tank/aimodels:/tank/aimodels
environment:
- PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
- VLLM_API_KEY=${API_KEY:-}
command:
- ${GEMMA4_MODEL:-/tank/aimodels/gemma4-26b-a4b-it-nvfp4}
- --quantization
- compressed-tensors
# UNCHANGED FROM THE OUTGOING SEAT, ON PURPOSE. Both names are live
# LiteLLM routes at http://10.250.50.54:8016/v1 — `char-rp` ->
# hosted_vllm/char-rp and `char-rp-reasoning` -> hosted_vllm/char-rp-thinking.
# They are two aliases onto ONE seat, not two seats. Renaming either
# breaks the gateway silently.
- --served-model-name
- char-rp
- char-rp-thinking
# Tool-calling: Gemma-4 emits its OWN native syntax
# (<|tool_call>call:name{...}<tool_call|>), NOT the qwen3_coder XML the
# other seats use. Without these two flags any tools-bearing request 400s
# outright. Architecture-level, so it carries over from MeroMero unchanged.
- --tool-call-parser
- gemma4
- --enable-auto-tool-choice
# The gemma4 REASONING parser absorbs the <|channel>...<channel|> thought
# markers; without it they leak into `content` verbatim on the post-tool
# turn (upstream vllm #45834).
- --reasoning-parser
- gemma4
# MANDATORY COMPANION TO THE REASONING PARSER. The parser reads
# enable_thinking from chat_template_kwargs and DEFAULTS IT TO TRUE. True
# makes is_reasoning_end() return False at a new turn, which pre-initialises
# the engine to REASONING — so ALL plain RP prose lands in
# reasoning_content with a NULL content, breaking every char-rp consumer.
# Do not remove. Thinking is still reachable per-request via
# chat_template_kwargs {"enable_thinking": true}, which is what the
# `char-rp-reasoning` alias exercises.
- --default-chat-template-kwargs
- '{"enable_thinking": false}'
# ⚠ NO --chat-template. MeroMero pointed at a hand-patched jinja under
# meromero-v2-nvfp4-work/; that patch was cut against THAT checkpoint and
# does not describe this one. This model ships its own chat_template.jinja
# and vLLM picks it up. If thinking mode ever misbehaves, diff this
# model's template against the MeroMero patch before reaching for a
# copy — do not assume the patch transfers.
- --max-model-len
- "${GEMMA4_MAX_MODEL_LEN:-262144}"
- --max-num-seqs
- "${GEMMA4_MAX_NUM_SEQS:-32}"
- --gpu-memory-utilization
- "${GEMMA4_GPU_MEM_UTIL:-0.47}"
- --kv-cache-dtype
- fp8
- --trust-remote-code
deploy:
resources:
reservations:
devices:
- driver: nvidia
device_ids:
- "${GEMMA4_GPU_ID:-0}"
capabilities:
- gpu
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
interval: 30s
timeout: 10s
retries: 3
start_period: 600s
networks:
- tnet
labels:
- homepage.group=AI - Inference
- homepage.name=char-rp (Gemma-4 26B-A4B NVFP4, MoE)
- homepage.icon=mdi-drama-masks
- homepage.description=gemma-4-26B-A4B-it NVFP4 MoE prose+CoT seat, 256K (ana-ml2 GPU0)
- homepage.href=http://10.250.50.54:${GEMMA4_PORT:-8016}/docs
networks:
tnet:
name: traefik-net
external: true
stacks/meromero-charrp/compose.yaml
+7 -3
View File
@@ -86,10 +86,14 @@ services:
networks:
- tnet
labels:
- homepage.group=AI - Inference
- homepage.name=char-rp (MeroMero-v2 NVFP4, multimodal)
# STOPPED 2026-08-24 — superseded by stacks/gemma4-charrp (gemma-4-26B-A4B
# -it NVFP4 MoE) on the same port and the same served-model-names. Kept as
# the rollback seat: `docker compose up -d vllm-meromero-rp` here after
# stopping the gemma4 one, since both want :8016.
- homepage.group=AI - Dormant
- homepage.name=char-rp (MeroMero-v2 NVFP4) — rollback
- homepage.icon=mdi-drama-masks
- homepage.description=G4-MeroMero-v2-31B NVFP4A16 non-thinking prose seat, vision-enabled, 256K (ana-ml2 GPU0)
- homepage.description=Superseded 2026-08-24 by the Gemma-4 26B-A4B MoE seat; dense 31B NVFP4A16, kept for rollback (ana-ml2 GPU0)
- homepage.href=http://10.250.50.54:${MEROMERO_PORT:-8016}/docs
networks: