feat(sglang): stage vLLM-vs-SGLang bench stack on ana-ml2
SGLang 0.5.13 confirmed to support our formats on Blackwell sm_120 (compressed-tensors NVFP4 W4A4, fp8, modelopt_fp4, petit_nvfp4, fp4_e2m1 KV), so the bench can be a real NVFP4 head-to-head. Parameterized compose (model/ quant/GPU via .env) + a common streaming load generator (bench.py: agg tok/s, TTFT p50/p99, TPOT) so both engines are driven identically on an exclusive GPU. Bench-oriented; promote to a real stack only if SGLang wins. Launch deferred until the NVFP4 eval frees a GPU.
This commit is contained in:
@@ -0,0 +1,33 @@
|
||||
# SGLang bench tunables. Copy to .env on ana-ml2 before deploying.
|
||||
#
|
||||
# Set SGLANG_MODEL + SGLANG_QUANT to match whatever vLLM config we're benching
|
||||
# against (the format that wins Brokkr's eval is the production-relevant target).
|
||||
|
||||
SGLANG_VERSION=latest
|
||||
SGLANG_PORT=30000
|
||||
|
||||
# EXCLUSIVE GPU for the bench (no co-tenants → fair numbers). Default 0; ensure
|
||||
# the eval endpoints + any llama-swap hot-load are torn down / off this GPU first.
|
||||
SGLANG_GPU_ID=0
|
||||
|
||||
# ── Model under test ── point at the SAME checkpoint vLLM serves ──────────────
|
||||
# NVFP4 W4A4 (the Blackwell-relevant format):
|
||||
SGLANG_MODEL=/aimodels/nvfp4/granite-4.1-8b-NVFP4-W4A4-test
|
||||
SGLANG_QUANT=compressed-tensors
|
||||
SGLANG_SERVED_NAME=granite-4.1-8b-nvfp4
|
||||
#
|
||||
# FP8 alternative (engine comparison on the current prod format):
|
||||
# SGLANG_MODEL=ibm-granite/granite-4.1-8b-fp8
|
||||
# SGLANG_QUANT=fp8
|
||||
# SGLANG_SERVED_NAME=granite-4.1-8b
|
||||
|
||||
# KV cache dtype — match vLLM's fp8 for apples-to-apples (or fp4_e2m1 to test
|
||||
# SGLang's FP4 KV, but then vLLM isn't comparable):
|
||||
SGLANG_KV_DTYPE=fp8_e4m3
|
||||
|
||||
# Context length — match the vLLM endpoints (64k for the heavy-substrate bench):
|
||||
SGLANG_CTX_LEN=65536
|
||||
|
||||
# Static memory fraction (SGLang's gpu-memory-utilization analog). On an
|
||||
# exclusive 96GB card this can be high (0.85-0.90).
|
||||
SGLANG_MEM_FRACTION=0.85
|
||||
@@ -0,0 +1,60 @@
|
||||
# sglang — vLLM-vs-SGLang bench on ana-ml2
|
||||
|
||||
Stood up to benchmark **SGLang against vLLM** on the same model + hardware, to
|
||||
see whether SGLang's throughput/latency wins justify it as a serving option
|
||||
(or a replacement) for the granite path on the Blackwells.
|
||||
|
||||
**Bench-oriented, not a permanent service** (yet). If SGLang wins decisively →
|
||||
promote to a real stack + add a gateway entry. Otherwise tear it down after.
|
||||
|
||||
## Capability (checked 2026-06-13)
|
||||
|
||||
SGLang 0.5.13 (torch 2.11+cu130) supports our formats on Blackwell sm_120:
|
||||
`compressed-tensors` (the llm-compressor NVFP4 W4A4 output), `fp8`,
|
||||
`modelopt_fp4`, `petit_nvfp4`, `mxfp4`, and `fp4_e2m1` KV. So the bench can be a
|
||||
real **NVFP4 head-to-head**, not just FP8.
|
||||
|
||||
## The one rule for a fair bench
|
||||
|
||||
**Exclusive GPU, same everything.** Both engines must run on a card with NO
|
||||
co-tenants (no eval endpoints, no llama-swap hot-load), same model, same context
|
||||
length, same prompt profile, same concurrency sweep, driven by the SAME load
|
||||
generator (`bench.py`) — not each engine's self-flattering built-in benchmark.
|
||||
The contention that skewed the earlier vLLM throughput probe is exactly what to
|
||||
avoid here.
|
||||
|
||||
## Run
|
||||
|
||||
```bash
|
||||
# 1. On ana-ml2, after the eval frees a GPU: cp .env.example .env, set
|
||||
# SGLANG_MODEL / SGLANG_QUANT to match the vLLM config under test, and
|
||||
# SGLANG_GPU_ID to an EXCLUSIVE card.
|
||||
scripts/deploy-stack.sh ana-ml2 sglang
|
||||
# (or docker compose up -d on the host)
|
||||
|
||||
# 2. Bench SGLang:
|
||||
python3 stacks/sglang/bench.py --url http://10.250.50.54:30000/v1 \
|
||||
--model granite-4.1-8b-nvfp4 --concurrency 1 10 50 100 200 --in-tokens 2048 --out-tokens 256
|
||||
|
||||
# 3. Stop SGLang, bring up vLLM on the SAME GPU + model, bench identically:
|
||||
python3 stacks/sglang/bench.py --url http://10.250.50.54:8006/v1 \
|
||||
--model granite-4.1-8b-nvfp4 --concurrency 1 10 50 100 200 --in-tokens 2048 --out-tokens 256
|
||||
|
||||
# 4. Repeat the sweep at --in-tokens 30000 (the prefill-heavy agent-memory
|
||||
# regime, where the engines can diverge sharply).
|
||||
```
|
||||
|
||||
## Metrics (`bench.py` reports)
|
||||
|
||||
- **agg_tok/s** — aggregate output throughput at concurrency N (the headline)
|
||||
- **ttft_p50 / p99** — time-to-first-token (prefill latency; matters most at high in-tokens)
|
||||
- **tpot_ms** — time-per-output-token (decode latency; the per-stream UX number)
|
||||
|
||||
`mem-fraction-static` is SGLang's `gpu-memory-utilization` analog; set it high
|
||||
(0.85–0.90) on an exclusive 96 GB card.
|
||||
|
||||
## Bench target
|
||||
|
||||
Bench whichever format wins Brokkr's quality eval (the production-relevant one):
|
||||
8B-NVFP4-W4A4 if that's the path, else FP8. Benching a format we won't ship is
|
||||
academic. Optionally run both formats to see if the engine ranking flips.
|
||||
@@ -0,0 +1,73 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Common LLM-serving load generator — drives any OpenAI-compatible endpoint
|
||||
(vLLM or SGLang) identically, so the vLLM-vs-SGLang comparison is apples-to-apples.
|
||||
|
||||
Streams responses to measure TTFT (time-to-first-token) and TPOT (time-per-output-
|
||||
token), not just aggregate throughput. Run it against each engine in turn on the
|
||||
SAME exclusive GPU, same prompt profile, same concurrency sweep.
|
||||
|
||||
Usage:
|
||||
python3 bench.py --url http://localhost:8006/v1 --model granite-4.1-8b-nvfp4 \
|
||||
--concurrency 1 10 50 100 --in-tokens 2048 --out-tokens 256 --requests 200
|
||||
|
||||
For the prefill-heavy agent-memory regime, bump --in-tokens to ~30000.
|
||||
"""
|
||||
import argparse, json, time, urllib.request, statistics, concurrent.futures
|
||||
|
||||
def stream_one(url, model, prompt, out_tokens):
|
||||
body = json.dumps({"model": model, "messages": [{"role": "user", "content": prompt}],
|
||||
"max_tokens": out_tokens, "temperature": 0.7, "stream": True}).encode()
|
||||
req = urllib.request.Request(url + "/chat/completions", data=body,
|
||||
headers={"Content-Type": "application/json"})
|
||||
t0 = time.perf_counter(); ttft = None; n = 0
|
||||
with urllib.request.urlopen(req, timeout=600) as r:
|
||||
for line in r:
|
||||
line = line.strip()
|
||||
if not line or not line.startswith(b"data:"):
|
||||
continue
|
||||
if line == b"data: [DONE]":
|
||||
break
|
||||
try:
|
||||
delta = json.loads(line[5:])["choices"][0]["delta"].get("content")
|
||||
except Exception:
|
||||
delta = None
|
||||
if delta:
|
||||
if ttft is None:
|
||||
ttft = time.perf_counter() - t0
|
||||
n += 1
|
||||
dur = time.perf_counter() - t0
|
||||
return ttft, n, dur
|
||||
|
||||
def run(url, model, prompt, out_tokens, N):
|
||||
t0 = time.perf_counter()
|
||||
with concurrent.futures.ThreadPoolExecutor(max_workers=N) as ex:
|
||||
res = list(ex.map(lambda _: stream_one(url, model, prompt, out_tokens), range(N)))
|
||||
wall = time.perf_counter() - t0
|
||||
res = [r for r in res if r[1] > 0]
|
||||
ttfts = sorted(r[0] for r in res if r[0])
|
||||
tot_tok = sum(r[1] for r in res)
|
||||
# TPOT: per-request (dur - ttft) / (tokens - 1), averaged
|
||||
tpots = [(d - t) / max(1, n - 1) for (t, n, d) in res if t and n > 1]
|
||||
pc = lambda xs, p: xs[min(len(xs) - 1, int(len(xs) * p))] if xs else float("nan")
|
||||
return {
|
||||
"N": N, "ok": len(res),
|
||||
"agg_tok_s": tot_tok / wall,
|
||||
"ttft_p50": pc(ttfts, 0.50), "ttft_p99": pc(ttfts, 0.99),
|
||||
"tpot_ms": (statistics.mean(tpots) * 1000) if tpots else float("nan"),
|
||||
}
|
||||
|
||||
if __name__ == "__main__":
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("--url", required=True)
|
||||
ap.add_argument("--model", required=True)
|
||||
ap.add_argument("--concurrency", type=int, nargs="+", default=[1, 10, 50, 100])
|
||||
ap.add_argument("--in-tokens", type=int, default=2048)
|
||||
ap.add_argument("--out-tokens", type=int, default=256)
|
||||
a = ap.parse_args()
|
||||
# crude prompt sized to ~in-tokens (word ≈ 1.3 tokens)
|
||||
prompt = "Summarize and analyze the following text in detail. " + ("data point " * int(a.in_tokens / 1.3))
|
||||
print(f"# {a.url} model={a.model} in≈{a.in_tokens} out={a.out_tokens}")
|
||||
print(f"{'N':>4} {'ok':>4} {'agg_tok/s':>10} {'ttft_p50':>9} {'ttft_p99':>9} {'tpot_ms':>8}")
|
||||
for N in a.concurrency:
|
||||
r = run(a.url, a.model, prompt, a.out_tokens, N)
|
||||
print(f"{r['N']:>4} {r['ok']:>4} {r['agg_tok_s']:>10.0f} {r['ttft_p50']:>9.2f} {r['ttft_p99']:>9.2f} {r['tpot_ms']:>8.1f}")
|
||||
@@ -0,0 +1,77 @@
|
||||
# SGLang — alternative LLM serving engine, stood up on ana-ml2 to bench
|
||||
# head-to-head against vLLM on the same model + hardware.
|
||||
#
|
||||
# Bench-oriented (not yet a permanent service): point it at the SAME checkpoint
|
||||
# vLLM serves, give it an EXCLUSIVE GPU (fair benches need no co-tenants), and
|
||||
# drive both engines with one common load generator. If SGLang wins decisively,
|
||||
# promote this to a real stack; otherwise tear it down after the bench.
|
||||
#
|
||||
# SGLang 0.5.13 supports our formats on Blackwell sm_120: compressed-tensors
|
||||
# (the llm-compressor NVFP4 W4A4 output), fp8, modelopt_fp4, petit_nvfp4, and
|
||||
# fp4_e2m1 KV. So SGLANG_QUANT can be compressed-tensors (NVFP4) or fp8.
|
||||
#
|
||||
# All tunables in .env — edit that, not this file.
|
||||
|
||||
services:
|
||||
sglang:
|
||||
image: lmsysorg/sglang:${SGLANG_VERSION}
|
||||
container_name: sglang
|
||||
restart: unless-stopped
|
||||
ipc: host # SGLang needs large shared memory
|
||||
ports:
|
||||
- "${SGLANG_PORT}:30000"
|
||||
volumes:
|
||||
- /tank/aimodels:/aimodels:ro # NVFP4 checkpoints + HF cache live here
|
||||
environment:
|
||||
- HF_HOME=/aimodels/huggingface
|
||||
- HF_HUB_CACHE=/aimodels/huggingface/hub
|
||||
command:
|
||||
- python3
|
||||
- -m
|
||||
- sglang.launch_server
|
||||
- --model-path
|
||||
- ${SGLANG_MODEL} # e.g. /aimodels/nvfp4/granite-4.1-8b-NVFP4-W4A4-test
|
||||
- --served-model-name
|
||||
- ${SGLANG_SERVED_NAME}
|
||||
- --host
|
||||
- 0.0.0.0
|
||||
- --port
|
||||
- "30000"
|
||||
- --quantization
|
||||
- ${SGLANG_QUANT} # compressed-tensors (NVFP4) | fp8
|
||||
- --kv-cache-dtype
|
||||
- ${SGLANG_KV_DTYPE:-fp8_e4m3}
|
||||
- --context-length
|
||||
- ${SGLANG_CTX_LEN:-65536}
|
||||
- --mem-fraction-static
|
||||
- ${SGLANG_MEM_FRACTION:-0.85} # SGLang's analog of vLLM gpu-memory-utilization
|
||||
- --tp
|
||||
- "1"
|
||||
deploy:
|
||||
resources:
|
||||
reservations:
|
||||
devices:
|
||||
- driver: nvidia
|
||||
device_ids:
|
||||
- "${SGLANG_GPU_ID}" # exclusive GPU for a fair bench
|
||||
capabilities:
|
||||
- gpu
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "python3 -c \"import urllib.request,sys; sys.exit(0 if urllib.request.urlopen('http://localhost:30000/health').status==200 else 1)\""]
|
||||
interval: 30s
|
||||
timeout: 10s
|
||||
retries: 3
|
||||
start_period: 300s
|
||||
networks:
|
||||
- tnet
|
||||
labels:
|
||||
- homepage.group=AI Systems
|
||||
- homepage.name=SGLang (bench)
|
||||
- homepage.icon=mdi-speedometer
|
||||
- homepage.description=SGLang serving — vLLM bench comparison (ana-ml2)
|
||||
- homepage.href=http://10.250.50.54:${SGLANG_PORT}
|
||||
|
||||
networks:
|
||||
tnet:
|
||||
name: traefik-net
|
||||
external: true
|
||||
Reference in New Issue
Block a user