vh
5f049cb4ad
SGLang 0.5.13 confirmed to support our formats on Blackwell sm_120 (compressed-tensors NVFP4 W4A4, fp8, modelopt_fp4, petit_nvfp4, fp4_e2m1 KV), so the bench can be a real NVFP4 head-to-head. Parameterized compose (model/ quant/GPU via .env) + a common streaming load generator (bench.py: agg tok/s, TTFT p50/p99, TPOT) so both engines are driven identically on an exclusive GPU. Bench-oriented; promote to a real stack only if SGLang wins. Launch deferred until the NVFP4 eval frees a GPU.
34 lines
1.3 KiB
Bash
34 lines
1.3 KiB
Bash
# SGLang bench tunables. Copy to .env on ana-ml2 before deploying.
|
|
#
|
|
# Set SGLANG_MODEL + SGLANG_QUANT to match whatever vLLM config we're benching
|
|
# against (the format that wins Brokkr's eval is the production-relevant target).
|
|
|
|
SGLANG_VERSION=latest
|
|
SGLANG_PORT=30000
|
|
|
|
# EXCLUSIVE GPU for the bench (no co-tenants → fair numbers). Default 0; ensure
|
|
# the eval endpoints + any llama-swap hot-load are torn down / off this GPU first.
|
|
SGLANG_GPU_ID=0
|
|
|
|
# ── Model under test ── point at the SAME checkpoint vLLM serves ──────────────
|
|
# NVFP4 W4A4 (the Blackwell-relevant format):
|
|
SGLANG_MODEL=/aimodels/nvfp4/granite-4.1-8b-NVFP4-W4A4-test
|
|
SGLANG_QUANT=compressed-tensors
|
|
SGLANG_SERVED_NAME=granite-4.1-8b-nvfp4
|
|
#
|
|
# FP8 alternative (engine comparison on the current prod format):
|
|
# SGLANG_MODEL=ibm-granite/granite-4.1-8b-fp8
|
|
# SGLANG_QUANT=fp8
|
|
# SGLANG_SERVED_NAME=granite-4.1-8b
|
|
|
|
# KV cache dtype — match vLLM's fp8 for apples-to-apples (or fp4_e2m1 to test
|
|
# SGLang's FP4 KV, but then vLLM isn't comparable):
|
|
SGLANG_KV_DTYPE=fp8_e4m3
|
|
|
|
# Context length — match the vLLM endpoints (64k for the heavy-substrate bench):
|
|
SGLANG_CTX_LEN=65536
|
|
|
|
# Static memory fraction (SGLang's gpu-memory-utilization analog). On an
|
|
# exclusive 96GB card this can be high (0.85-0.90).
|
|
SGLANG_MEM_FRACTION=0.85
|