feat(char-rp): swap the seat to the Gemma-4 26B-A4B MoE, NVFP4, same port
Straight-across replacement of the dense G4-MeroMero-v2-31B-NVFP4A16 seat with google/gemma-4-26B-A4B-it on ana-ml2 GPU0. Port, served-model-names and every gateway route are unchanged, so no consumer sees a difference in addressing: `char-rp` -> hosted_vllm/char-rp and `char-rp-reasoning` -> hosted_vllm/char-rp-thinking, both still :8016. The seat's requirements now include chain-of-thought, which makes throughput more critical rather than less — the user waits through the whole reasoning block before the first visible token, and the MoE measures ~114 tok/s @32K against the dense 31B's ~40.7. Both artifacts are on disk and they are NOT interchangeable. The BF16 weights (/tank/aimodels/gemma4-26b-a4b-it-bf16, 49 GB) are the QLoRA tuning base, since QLoRA does its own quantization. They CANNOT be served here: 48.10 GiB of weights against ~49 GiB of free GPU0 leaves nothing for KV cache, and the engine would die at allocation exactly the way the predecessor did this afternoon. The serving copy is RedHatAI/gemma-4-26B-A4B-it-NVFP4 (16 GB), chosen over the other -it quants because it is compressed-tensors (nvfp4-pack-quantized) — the same loader path the outgoing seat used — from the llm-compressor team at 357k downloads. The nvidia/ repo is the base rather than -it, and the thinking channel lives in the instruction-tuned weights. Smaller weights at the same 0.47 memory budget buy a much larger KV pool: 27.37 GiB and 1,724,110 tokens, against the predecessor's 371,023 at the same budget. That is 6.5 full-length 262K sequences concurrent rather than 1.4. The gemma4 tool-call parser, reasoning parser and the enable_thinking:false default all carry over unchanged — they are architecture-level, not checkpoint-level. The --chat-template override does NOT carry over: MeroMero pointed at a jinja hand-patched against that checkpoint, and this model ships its own. Verified that dropping it did not reintroduce the failure that flag existed to prevent — non-thinking prose lands in content with reasoning_content empty, and the thinking alias populates reasoning_content with content carrying the answer. ⚠ Scheme differs from the incumbent and the bench should say so: this quant declares 4-bit input activations (W4A4) where the outgoing seat was NVFP4A16. Faster, and not like-for-like on the activation axis. meromero-charrp is retained stopped in `created` state and relabelled to AI - Dormant, per the house rollback pattern. Both stacks want :8016, so rolling back means stopping the gemma4 seat first.
This commit is contained in:
@@ -86,10 +86,14 @@ services:
|
||||
networks:
|
||||
- tnet
|
||||
labels:
|
||||
- homepage.group=AI - Inference
|
||||
- homepage.name=char-rp (MeroMero-v2 NVFP4, multimodal)
|
||||
# STOPPED 2026-08-24 — superseded by stacks/gemma4-charrp (gemma-4-26B-A4B
|
||||
# -it NVFP4 MoE) on the same port and the same served-model-names. Kept as
|
||||
# the rollback seat: `docker compose up -d vllm-meromero-rp` here after
|
||||
# stopping the gemma4 one, since both want :8016.
|
||||
- homepage.group=AI - Dormant
|
||||
- homepage.name=char-rp (MeroMero-v2 NVFP4) — rollback
|
||||
- homepage.icon=mdi-drama-masks
|
||||
- homepage.description=G4-MeroMero-v2-31B NVFP4A16 non-thinking prose seat, vision-enabled, 256K (ana-ml2 GPU0)
|
||||
- homepage.description=Superseded 2026-08-24 by the Gemma-4 26B-A4B MoE seat; dense 31B NVFP4A16, kept for rollback (ana-ml2 GPU0)
|
||||
- homepage.href=http://10.250.50.54:${MEROMERO_PORT:-8016}/docs
|
||||
|
||||
networks:
|
||||
|
||||
Reference in New Issue
Block a user