# erp-seat — the RP seat on fv-ml1 GPU1. Serves the **MeroMero A4B MoE** NVFP4A16 quant # (G4-MeroMero-26B-A4B-it-uncensored-heretic) behind the gateway alias `char-rp-fast`. # # ⚠ THE STACK NAME IS HISTORICAL. It served Pfish-6 (the run-6 ERP-tune LoRA merge) until # 2026-09-10, when the operator swapped the occupant: "replace that a4b moe over pfish-6 -- # remove the pfish-6 alias and create an alias for char-rp-fast." The compose PROJECT name is # deliberately NOT renamed: asset-engine derives seat liveness from it, so a rename reads as # OFFLINE. Pfish-6 remains on disk at /tank/aimodels/erp-tune-v6-nvfp4a16 and the pre-swap host # env is at /opt/docker/compose/erp-seat/.env.pfish6.bak-20260910 -- one cp plus `up -d` back. # # ⚠ THE FIRST A4B QUANT SERVED NaN AND LOOKED HEALTHY DOING IT. It was built with the DENSE # recipe, whose ignore list carries no router regex, so all 30 MoE routers were quantized to # 4 bits and expert selection was destroyed (playbook §3.15). The seat passed its healthcheck, # returned finish_reason=length with the full token count, and every response decoded to the # empty string; the give-away was NaN logprobs. Re-quantized with # services/erp-seat-quant/quant_nvfp4a16_gemma4_moe.py, whose target guard refuses exactly # that. Use the MoE recipe for anything in this family; the dense one is for the v2-31B. # # PFISH-6 PROVENANCE, kept because it is still the rollback target. Run 7 was retired by # operator ruling 2026-09-09 ("we're gonna stay on 6 for now"): its gate failure turned out to # be a DETECTOR BUG (the adjective "minor" in a HARD rule, fixed cc42d76 in brokkr-smithy), but # run 7 was independently a poor run (primary FLAT +2, both diversity families reduced, # long-context coherence 1.0 -> 0.875). Run 6 was the standing seat here until the 2026-09-10 # swap above. Routing aliases live in the gateway, not here. # # Serve recipe copied from stacks/gemma4-charrp (same architecture + quant format, proven on this # box): gemma4 tool + reasoning parsers, enable_thinking pinned false, model's own stock template. # It carries over to MeroMero A4B unchanged -- verified 2026-09-10 end to end: clean prose with # no channel-prefix leak, a solid-colour image read correctly (vision towers intact), and an # auto tool_choice call parsed. # GPU1 is SHARED (scriberr, embed, rerank, coder, reward, charrp live there): the live .env runs # ERP_GPU_MEM_UTIL=0.30, and the KV pool is pinned in bytes below regardless, so the ratio only # has to clear admission. # Quant pipeline: services/erp-seat-quant/ (MoE recipe -- NOT services/meromero-quant/, which is # the dense one). Tunables in .env. name: erp-seat services: vllm-erp-seat: image: ${ERP_IMAGE:-vllm/vllm-openai:nightly-311b3513af33bc29b4acb2fde2e9313e5e9966a0} container_name: ${ERP_CONTAINER:-vllm-erp-seat} restart: unless-stopped ipc: host ports: - "${ERP_PORT:-8021}:8000" volumes: - /tank/aimodels:/tank/aimodels environment: - VLLM_API_KEY=${API_KEY:-} command: - ${ERP_MODEL:-/tank/aimodels/G4-MeroMero-26B-A4B-it-uncensored-heretic-NVFP4A16} - --quantization - compressed-tensors - --served-model-name - ${ERP_SERVED_NAME:-G4-MeroMero-26B-A4B-it-uncensored-heretic-NVFP4A16} - --tool-call-parser - gemma4 - --enable-auto-tool-choice # --reasoning-parser gemma4 + the enable_thinking pin are BOTH required (measured 2026-09-08): # without them the post-tool-response turn leaks a literal "<|channel>thought\n" # prefix into content (3/3), exactly as stacks/gemma4-charrp/README.md warns. They do NOT # fix forced (named) tool_choice — that failure is independent of the reasoning parser # (measured 1/9 with it, 3/9 without): the model free-writes Gemma native call syntax # instead of grammar-constrained JSON. See README for the forced-tool_choice status. - --reasoning-parser - gemma4 - --default-chat-template-kwargs - '{"enable_thinking": false}' # tool_choice:"none" trap (measured 2026-09-08): without this flag vLLM still renders the # tools into the prompt, the model emits a tool call anyway, and because tool parsing is # off for tool_choice=none the reply comes back with content=null AND tool_calls=null — # an empty turn. The flag drops the tools from the prompt so the model answers in prose. - --exclude-tools-when-tool-choice-none - --chat-template - ${ERP_CHAT_TEMPLATE:-/tank/aimodels/G4-MeroMero-26B-A4B-it-uncensored-heretic-NVFP4A16/chat_template.jinja} - --max-model-len - "${ERP_MAX_MODEL_LEN:-262144}" # KV pool pinned in BYTES, not inferred from the utilization ratio. GPU1 is # SHARED with seven other containers (charrp, scriberr, embed, rerank, coder, # reward), so a ratio silently yields a different cache depending on what else # is resident at start time; an explicit figure is reproducible and is what the # concurrency ceiling is actually computed from. # # ⚠ KV-per-token is NOT a constant you may carry between context settings. # Measured on THIS build (2026-09-09): 63.2 KiB/token at max-model-len 32768, # 16.6 KiB/token at 262144 — a 3.8x difference, because Gemma-4 is hybrid # attention (sliding_window 1024 on most layers) and the sliding layers stop # scaling with the context while the full-attention layers keep going. # Extrapolating the 32k figure to 256k under-counted the achievable cache by # 3.8x and would have sized this seat wrong. Re-measure per configuration. # # 8.49 GiB -> 534,649 tokens -> 2.04x a full 262,144-token context, which is # the operator's sizing rule (KV = 2x max context, 2026-09-09). # # This figure SURVIVED the 2026-09-10 Pfish-6 -> MeroMero-A4B swap unchanged, and that # is not luck: the two are the same architecture field for field (30 layers, kv 8, # head_dim 256, sliding_window 1024, 25 sliding / 5 full, 128 experts top-8), so the # KV-per-token is the same number. Confirmed by reading 534,649 tokens / 2.04x back out # of the new engine's log rather than assuming the pinning carried. - --kv-cache-memory - "${ERP_KV_CACHE_MEMORY:-9114000000}" - --max-num-seqs - "${ERP_MAX_NUM_SEQS:-32}" - --gpu-memory-utilization - "${ERP_GPU_MEM_UTIL:-0.35}" - --kv-cache-dtype - fp8 # SPEED (2026-09-09). ⚠ NVFP4A16 is WEIGHT-ONLY: activations stay 16-bit, so # there is no FP4 GEMM to issue and the FlashInfer/CUTLASS NVFP4 MoE kernels # do not apply — vLLM's "your GPU does not have native support for FP4" # warning is about the SCHEME, not the card. fv-ml1 is sm_120 and DOES have # native FP4; a W4A4 build would use it, at a fidelity cost the operator # already settled against for this seat. Marlin is the correct kernel here. # PROVEN, not assumed: forcing moe_backend=flashinfer_cutedsl fails at engine # init — "does not support the deployment configuration since kernel does not # support current device cuda". Marlin is the only viable NVFP4 MoE kernel # here, so `auto` is already optimal and this knob exists to document that # rather than to be turned. # # THE REAL LEVER WAS CONCURRENCY. max-num-seqs was 8, which capped the seat # far below its KV pool (534,649 tokens). Measured on this build 2026-09-09, # 3 rounds each, 300-token generations: # n=1 219 tok/s (unchanged by the seq limit) # n=4 676 tok/s aggregate # n=8 1269 tok/s aggregate <- was NOT REACHABLE at max-num-seqs 8 # n=16 2170 tok/s aggregate <- 3.2x the old ceiling # Single-stream latency is identical, so the raise costs nothing and lifts # the ceiling ~3.2x. CUDA-graph capture sizes followed automatically to 64. - --kernel-config - '{"moe_backend": "${ERP_MOE_BACKEND:-auto}"}' - --trust-remote-code deploy: resources: reservations: devices: - driver: nvidia device_ids: - "${ERP_GPU_ID:-1}" capabilities: - gpu healthcheck: test: ["CMD", "curl", "-f", "http://localhost:8000/health"] interval: 30s timeout: 10s retries: 3 start_period: 600s networks: - tnet labels: - homepage.group=AI - Inference - homepage.name=char-rp-fast (MeroMero 26B-A4B, NVFP4A16 MoE) - homepage.icon=mdi-fire - homepage.description=MeroMero A4B abliterated RP seat, NVFP4A16 weight-only, vision intact (fv-ml1 GPU1) - homepage.href=http://10.251.50.54:${ERP_PORT:-8021}/docs networks: tnet: name: traefik-net external: true