vh
c35f6510bf
Layout: GPU 0 = cyberprev (KV 16 -> 15 GiB, 1.29x), erp-seat (moved from GPU 1, KV 8.49 -> 5.5 GiB, 1.32x), voices (ctx 8,192 -> 40,960, its native maximum, KV 6 GiB, 1.07x), coder (moved from GPU 1, ctx 16,384 -> 32,768, the card maximum, 2.35x), parakeet-nemo. GPU 1 = gen-small (moved from GPU 0, 5.5 GiB, 1.76x), meromero-rp (22 GiB, 1.32x), intern-decision. Sizing rule: KV >= 1.25x max context rounded up to 0.5 GiB, checked against 7 days of LiteLLM traffic and the engines' own KV-usage logs. Each card keeps spare memory for its bursty tenant (parakeet's cap on GPU 0, intern-decision's 32k call on GPU 1). YaRN extensions are not enabled: they do not fit, and the Qwen cards warn static YaRN degrades short prompts. gen-small gets --override-generation-config max_new_tokens 32768. It applies only to requests that omit max_tokens (vLLM get_max_tokens) and answers 9 runaway ~260k-token replies in 7 days; the longest normal reply was 6,357. Applied with playbooks/fv-ml1-rebalance-20261005.yaml (boot-check guard and health/KV verification per seat); all nine seats healthy, gateway smoke tests pass.
161 lines
8.7 KiB
YAML
161 lines
8.7 KiB
YAML
# erp-seat — the RP seat on fv-ml1 GPU1. Serves the **MeroMero A4B MoE** NVFP4A16 quant
|
|
# (G4-MeroMero-26B-A4B-it-uncensored-heretic) behind the gateway alias `char-rp-fast`.
|
|
#
|
|
# ⚠ THE STACK NAME IS HISTORICAL. It served Pfish-6 (the run-6 ERP-tune LoRA merge) until
|
|
# 2026-09-10, when the operator swapped the occupant: "replace that a4b moe over pfish-6 --
|
|
# remove the pfish-6 alias and create an alias for char-rp-fast." The compose PROJECT name is
|
|
# deliberately NOT renamed: asset-engine derives seat liveness from it, so a rename reads as
|
|
# OFFLINE. Pfish-6 remains on disk at /tank/aimodels/erp-tune-v6-nvfp4a16 and the pre-swap host
|
|
# env is at /opt/docker/compose/erp-seat/.env.pfish6.bak-20260910 -- one cp plus `up -d` back.
|
|
#
|
|
# ⚠ THE FIRST A4B QUANT SERVED NaN AND LOOKED HEALTHY DOING IT. It was built with the DENSE
|
|
# recipe, whose ignore list carries no router regex, so all 30 MoE routers were quantized to
|
|
# 4 bits and expert selection was destroyed (playbook §3.15). The seat passed its healthcheck,
|
|
# returned finish_reason=length with the full token count, and every response decoded to the
|
|
# empty string; the give-away was NaN logprobs. Re-quantized with
|
|
# services/erp-seat-quant/quant_nvfp4a16_gemma4_moe.py, whose target guard refuses exactly
|
|
# that. Use the MoE recipe for anything in this family; the dense one is for the v2-31B.
|
|
#
|
|
# PFISH-6 PROVENANCE, kept because it is still the rollback target. Run 7 was retired by
|
|
# operator ruling 2026-09-09 ("we're gonna stay on 6 for now"): its gate failure turned out to
|
|
# be a DETECTOR BUG (the adjective "minor" in a HARD rule, fixed cc42d76 in brokkr-smithy), but
|
|
# run 7 was independently a poor run (primary FLAT +2, both diversity families reduced,
|
|
# long-context coherence 1.0 -> 0.875). Run 6 was the standing seat here until the 2026-09-10
|
|
# swap above. Routing aliases live in the gateway, not here.
|
|
#
|
|
# Serve recipe copied from stacks/gemma4-charrp (same architecture + quant format, proven on this
|
|
# box): gemma4 tool + reasoning parsers, enable_thinking pinned false, model's own stock template.
|
|
# It carries over to MeroMero A4B unchanged -- verified 2026-09-10 end to end: clean prose with
|
|
# no channel-prefix leak, a solid-colour image read correctly (vision towers intact), and an
|
|
# auto tool_choice call parsed.
|
|
# GPU1 is SHARED (scriberr, embed, rerank, coder, reward, charrp live there): the live .env runs
|
|
# ERP_GPU_MEM_UTIL=0.30, and the KV pool is pinned in bytes below regardless, so the ratio only
|
|
# has to clear admission.
|
|
# Quant pipeline: services/erp-seat-quant/ (MoE recipe -- NOT services/meromero-quant/, which is
|
|
# the dense one). Tunables in .env.
|
|
|
|
name: erp-seat
|
|
|
|
services:
|
|
vllm-erp-seat:
|
|
image: ${ERP_IMAGE:-vllm/vllm-openai:nightly-311b3513af33bc29b4acb2fde2e9313e5e9966a0}
|
|
container_name: ${ERP_CONTAINER:-vllm-erp-seat}
|
|
restart: unless-stopped
|
|
ipc: host
|
|
ports:
|
|
- "${ERP_PORT:-8021}:8000"
|
|
volumes:
|
|
- /tank/aimodels:/tank/aimodels
|
|
environment:
|
|
- VLLM_API_KEY=${API_KEY:-}
|
|
command:
|
|
- ${ERP_MODEL:-/tank/aimodels/G4-MeroMero-26B-A4B-it-uncensored-heretic-NVFP4A16}
|
|
- --quantization
|
|
- compressed-tensors
|
|
- --served-model-name
|
|
- ${ERP_SERVED_NAME:-G4-MeroMero-26B-A4B-it-uncensored-heretic-NVFP4A16}
|
|
- --tool-call-parser
|
|
- gemma4
|
|
- --enable-auto-tool-choice
|
|
# --reasoning-parser gemma4 + the enable_thinking pin are BOTH required (measured 2026-09-08):
|
|
# without them the post-tool-response turn leaks a literal "<|channel>thought\n<channel|>"
|
|
# prefix into content (3/3), exactly as stacks/gemma4-charrp/README.md warns. They do NOT
|
|
# fix forced (named) tool_choice — that failure is independent of the reasoning parser
|
|
# (measured 1/9 with it, 3/9 without): the model free-writes Gemma native call syntax
|
|
# instead of grammar-constrained JSON. See README for the forced-tool_choice status.
|
|
- --reasoning-parser
|
|
- gemma4
|
|
- --default-chat-template-kwargs
|
|
- '{"enable_thinking": false}'
|
|
# tool_choice:"none" trap (measured 2026-09-08): without this flag vLLM still renders the
|
|
# tools into the prompt, the model emits a tool call anyway, and because tool parsing is
|
|
# off for tool_choice=none the reply comes back with content=null AND tool_calls=null —
|
|
# an empty turn. The flag drops the tools from the prompt so the model answers in prose.
|
|
- --exclude-tools-when-tool-choice-none
|
|
- --chat-template
|
|
- ${ERP_CHAT_TEMPLATE:-/tank/aimodels/G4-MeroMero-26B-A4B-it-uncensored-heretic-NVFP4A16/chat_template.jinja}
|
|
- --max-model-len
|
|
- "${ERP_MAX_MODEL_LEN:-262144}"
|
|
# KV pool pinned in BYTES, not inferred from the utilization ratio. GPU1 is
|
|
# SHARED with seven other containers (charrp, scriberr, embed, rerank, coder,
|
|
# reward), so a ratio silently yields a different cache depending on what else
|
|
# is resident at start time; an explicit figure is reproducible and is what the
|
|
# concurrency ceiling is actually computed from.
|
|
#
|
|
# ⚠ KV-per-token is NOT a constant you may carry between context settings.
|
|
# Measured on THIS build (2026-09-09): 63.2 KiB/token at max-model-len 32768,
|
|
# 16.6 KiB/token at 262144 — a 3.8x difference, because Gemma-4 is hybrid
|
|
# attention (sliding_window 1024 on most layers) and the sliding layers stop
|
|
# scaling with the context while the full-attention layers keep going.
|
|
# Extrapolating the 32k figure to 256k under-counted the achievable cache by
|
|
# 3.8x and would have sized this seat wrong. Re-measure per configuration.
|
|
#
|
|
# 8.49 GiB -> 534,649 tokens -> 2.04x a full 262,144-token context, which is
|
|
# the operator's sizing rule (KV = 2x max context, 2026-09-09).
|
|
#
|
|
# This figure SURVIVED the 2026-09-10 Pfish-6 -> MeroMero-A4B swap unchanged, and that
|
|
# is not luck: the two are the same architecture field for field (30 layers, kv 8,
|
|
# head_dim 256, sliding_window 1024, 25 sliding / 5 full, 128 experts top-8), so the
|
|
# KV-per-token is the same number. Confirmed by reading 534,649 tokens / 2.04x back out
|
|
# of the new engine's log rather than assuming the pinning carried.
|
|
- --kv-cache-memory
|
|
- "${ERP_KV_CACHE_MEMORY:-9114000000}"
|
|
- --max-num-seqs
|
|
- "${ERP_MAX_NUM_SEQS:-32}"
|
|
- --gpu-memory-utilization
|
|
- "${ERP_GPU_MEM_UTIL:-0.35}"
|
|
- --kv-cache-dtype
|
|
- fp8
|
|
# SPEED (2026-09-09). ⚠ NVFP4A16 is WEIGHT-ONLY: activations stay 16-bit, so
|
|
# there is no FP4 GEMM to issue and the FlashInfer/CUTLASS NVFP4 MoE kernels
|
|
# do not apply — vLLM's "your GPU does not have native support for FP4"
|
|
# warning is about the SCHEME, not the card. fv-ml1 is sm_120 and DOES have
|
|
# native FP4; a W4A4 build would use it, at a fidelity cost the operator
|
|
# already settled against for this seat. Marlin is the correct kernel here.
|
|
# PROVEN, not assumed: forcing moe_backend=flashinfer_cutedsl fails at engine
|
|
# init — "does not support the deployment configuration since kernel does not
|
|
# support current device cuda". Marlin is the only viable NVFP4 MoE kernel
|
|
# here, so `auto` is already optimal and this knob exists to document that
|
|
# rather than to be turned.
|
|
#
|
|
# THE REAL LEVER WAS CONCURRENCY. max-num-seqs was 8, which capped the seat
|
|
# far below its KV pool (534,649 tokens). Measured on this build 2026-09-09,
|
|
# 3 rounds each, 300-token generations:
|
|
# n=1 219 tok/s (unchanged by the seq limit)
|
|
# n=4 676 tok/s aggregate
|
|
# n=8 1269 tok/s aggregate <- was NOT REACHABLE at max-num-seqs 8
|
|
# n=16 2170 tok/s aggregate <- 3.2x the old ceiling
|
|
# Single-stream latency is identical, so the raise costs nothing and lifts
|
|
# the ceiling ~3.2x. CUDA-graph capture sizes followed automatically to 64.
|
|
- --kernel-config
|
|
- '{"moe_backend": "${ERP_MOE_BACKEND:-auto}"}'
|
|
- --trust-remote-code
|
|
deploy:
|
|
resources:
|
|
reservations:
|
|
devices:
|
|
- driver: nvidia
|
|
device_ids:
|
|
- "${ERP_GPU_ID:-1}"
|
|
capabilities:
|
|
- gpu
|
|
healthcheck:
|
|
test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
|
|
interval: 30s
|
|
timeout: 10s
|
|
retries: 3
|
|
start_period: 600s
|
|
networks:
|
|
- tnet
|
|
labels:
|
|
- homepage.group=AI - Inference
|
|
- homepage.name=char-rp-fast (MeroMero 26B-A4B, NVFP4A16 MoE)
|
|
- homepage.icon=mdi-fire
|
|
- homepage.description=MeroMero A4B abliterated RP seat, NVFP4A16 weight-only, vision intact (fv-ml1 GPU0)
|
|
- homepage.href=http://10.251.50.54:${ERP_PORT:-8021}/docs
|
|
|
|
networks:
|
|
tnet:
|
|
name: traefik-net
|
|
external: true
|