From 5f049cb4ad3eb762e466e92db14096b7469ff43c Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Fri, 12 Jun 2026 22:40:26 -0700 Subject: [PATCH] feat(sglang): stage vLLM-vs-SGLang bench stack on ana-ml2 SGLang 0.5.13 confirmed to support our formats on Blackwell sm_120 (compressed-tensors NVFP4 W4A4, fp8, modelopt_fp4, petit_nvfp4, fp4_e2m1 KV), so the bench can be a real NVFP4 head-to-head. Parameterized compose (model/ quant/GPU via .env) + a common streaming load generator (bench.py: agg tok/s, TTFT p50/p99, TPOT) so both engines are driven identically on an exclusive GPU. Bench-oriented; promote to a real stack only if SGLang wins. Launch deferred until the NVFP4 eval frees a GPU. --- stacks/sglang/.env.example | 33 ++++++++++++++++ stacks/sglang/README.md | 60 +++++++++++++++++++++++++++++ stacks/sglang/bench.py | 73 ++++++++++++++++++++++++++++++++++++ stacks/sglang/compose.yaml | 77 ++++++++++++++++++++++++++++++++++++++ 4 files changed, 243 insertions(+) create mode 100644 stacks/sglang/.env.example create mode 100644 stacks/sglang/README.md create mode 100644 stacks/sglang/bench.py create mode 100644 stacks/sglang/compose.yaml diff --git a/stacks/sglang/.env.example b/stacks/sglang/.env.example new file mode 100644 index 0000000..6d03e52 --- /dev/null +++ b/stacks/sglang/.env.example @@ -0,0 +1,33 @@ +# SGLang bench tunables. Copy to .env on ana-ml2 before deploying. +# +# Set SGLANG_MODEL + SGLANG_QUANT to match whatever vLLM config we're benching +# against (the format that wins Brokkr's eval is the production-relevant target). + +SGLANG_VERSION=latest +SGLANG_PORT=30000 + +# EXCLUSIVE GPU for the bench (no co-tenants → fair numbers). Default 0; ensure +# the eval endpoints + any llama-swap hot-load are torn down / off this GPU first. +SGLANG_GPU_ID=0 + +# ── Model under test ── point at the SAME checkpoint vLLM serves ────────────── +# NVFP4 W4A4 (the Blackwell-relevant format): +SGLANG_MODEL=/aimodels/nvfp4/granite-4.1-8b-NVFP4-W4A4-test +SGLANG_QUANT=compressed-tensors +SGLANG_SERVED_NAME=granite-4.1-8b-nvfp4 +# +# FP8 alternative (engine comparison on the current prod format): +# SGLANG_MODEL=ibm-granite/granite-4.1-8b-fp8 +# SGLANG_QUANT=fp8 +# SGLANG_SERVED_NAME=granite-4.1-8b + +# KV cache dtype — match vLLM's fp8 for apples-to-apples (or fp4_e2m1 to test +# SGLang's FP4 KV, but then vLLM isn't comparable): +SGLANG_KV_DTYPE=fp8_e4m3 + +# Context length — match the vLLM endpoints (64k for the heavy-substrate bench): +SGLANG_CTX_LEN=65536 + +# Static memory fraction (SGLang's gpu-memory-utilization analog). On an +# exclusive 96GB card this can be high (0.85-0.90). +SGLANG_MEM_FRACTION=0.85 diff --git a/stacks/sglang/README.md b/stacks/sglang/README.md new file mode 100644 index 0000000..b26178c --- /dev/null +++ b/stacks/sglang/README.md @@ -0,0 +1,60 @@ +# sglang — vLLM-vs-SGLang bench on ana-ml2 + +Stood up to benchmark **SGLang against vLLM** on the same model + hardware, to +see whether SGLang's throughput/latency wins justify it as a serving option +(or a replacement) for the granite path on the Blackwells. + +**Bench-oriented, not a permanent service** (yet). If SGLang wins decisively → +promote to a real stack + add a gateway entry. Otherwise tear it down after. + +## Capability (checked 2026-06-13) + +SGLang 0.5.13 (torch 2.11+cu130) supports our formats on Blackwell sm_120: +`compressed-tensors` (the llm-compressor NVFP4 W4A4 output), `fp8`, +`modelopt_fp4`, `petit_nvfp4`, `mxfp4`, and `fp4_e2m1` KV. So the bench can be a +real **NVFP4 head-to-head**, not just FP8. + +## The one rule for a fair bench + +**Exclusive GPU, same everything.** Both engines must run on a card with NO +co-tenants (no eval endpoints, no llama-swap hot-load), same model, same context +length, same prompt profile, same concurrency sweep, driven by the SAME load +generator (`bench.py`) — not each engine's self-flattering built-in benchmark. +The contention that skewed the earlier vLLM throughput probe is exactly what to +avoid here. + +## Run + +```bash +# 1. On ana-ml2, after the eval frees a GPU: cp .env.example .env, set +# SGLANG_MODEL / SGLANG_QUANT to match the vLLM config under test, and +# SGLANG_GPU_ID to an EXCLUSIVE card. +scripts/deploy-stack.sh ana-ml2 sglang +# (or docker compose up -d on the host) + +# 2. Bench SGLang: +python3 stacks/sglang/bench.py --url http://10.250.50.54:30000/v1 \ + --model granite-4.1-8b-nvfp4 --concurrency 1 10 50 100 200 --in-tokens 2048 --out-tokens 256 + +# 3. Stop SGLang, bring up vLLM on the SAME GPU + model, bench identically: +python3 stacks/sglang/bench.py --url http://10.250.50.54:8006/v1 \ + --model granite-4.1-8b-nvfp4 --concurrency 1 10 50 100 200 --in-tokens 2048 --out-tokens 256 + +# 4. Repeat the sweep at --in-tokens 30000 (the prefill-heavy agent-memory +# regime, where the engines can diverge sharply). +``` + +## Metrics (`bench.py` reports) + +- **agg_tok/s** — aggregate output throughput at concurrency N (the headline) +- **ttft_p50 / p99** — time-to-first-token (prefill latency; matters most at high in-tokens) +- **tpot_ms** — time-per-output-token (decode latency; the per-stream UX number) + +`mem-fraction-static` is SGLang's `gpu-memory-utilization` analog; set it high +(0.85–0.90) on an exclusive 96 GB card. + +## Bench target + +Bench whichever format wins Brokkr's quality eval (the production-relevant one): +8B-NVFP4-W4A4 if that's the path, else FP8. Benching a format we won't ship is +academic. Optionally run both formats to see if the engine ranking flips. diff --git a/stacks/sglang/bench.py b/stacks/sglang/bench.py new file mode 100644 index 0000000..913a939 --- /dev/null +++ b/stacks/sglang/bench.py @@ -0,0 +1,73 @@ +#!/usr/bin/env python3 +"""Common LLM-serving load generator — drives any OpenAI-compatible endpoint +(vLLM or SGLang) identically, so the vLLM-vs-SGLang comparison is apples-to-apples. + +Streams responses to measure TTFT (time-to-first-token) and TPOT (time-per-output- +token), not just aggregate throughput. Run it against each engine in turn on the +SAME exclusive GPU, same prompt profile, same concurrency sweep. + +Usage: + python3 bench.py --url http://localhost:8006/v1 --model granite-4.1-8b-nvfp4 \ + --concurrency 1 10 50 100 --in-tokens 2048 --out-tokens 256 --requests 200 + +For the prefill-heavy agent-memory regime, bump --in-tokens to ~30000. +""" +import argparse, json, time, urllib.request, statistics, concurrent.futures + +def stream_one(url, model, prompt, out_tokens): + body = json.dumps({"model": model, "messages": [{"role": "user", "content": prompt}], + "max_tokens": out_tokens, "temperature": 0.7, "stream": True}).encode() + req = urllib.request.Request(url + "/chat/completions", data=body, + headers={"Content-Type": "application/json"}) + t0 = time.perf_counter(); ttft = None; n = 0 + with urllib.request.urlopen(req, timeout=600) as r: + for line in r: + line = line.strip() + if not line or not line.startswith(b"data:"): + continue + if line == b"data: [DONE]": + break + try: + delta = json.loads(line[5:])["choices"][0]["delta"].get("content") + except Exception: + delta = None + if delta: + if ttft is None: + ttft = time.perf_counter() - t0 + n += 1 + dur = time.perf_counter() - t0 + return ttft, n, dur + +def run(url, model, prompt, out_tokens, N): + t0 = time.perf_counter() + with concurrent.futures.ThreadPoolExecutor(max_workers=N) as ex: + res = list(ex.map(lambda _: stream_one(url, model, prompt, out_tokens), range(N))) + wall = time.perf_counter() - t0 + res = [r for r in res if r[1] > 0] + ttfts = sorted(r[0] for r in res if r[0]) + tot_tok = sum(r[1] for r in res) + # TPOT: per-request (dur - ttft) / (tokens - 1), averaged + tpots = [(d - t) / max(1, n - 1) for (t, n, d) in res if t and n > 1] + pc = lambda xs, p: xs[min(len(xs) - 1, int(len(xs) * p))] if xs else float("nan") + return { + "N": N, "ok": len(res), + "agg_tok_s": tot_tok / wall, + "ttft_p50": pc(ttfts, 0.50), "ttft_p99": pc(ttfts, 0.99), + "tpot_ms": (statistics.mean(tpots) * 1000) if tpots else float("nan"), + } + +if __name__ == "__main__": + ap = argparse.ArgumentParser() + ap.add_argument("--url", required=True) + ap.add_argument("--model", required=True) + ap.add_argument("--concurrency", type=int, nargs="+", default=[1, 10, 50, 100]) + ap.add_argument("--in-tokens", type=int, default=2048) + ap.add_argument("--out-tokens", type=int, default=256) + a = ap.parse_args() + # crude prompt sized to ~in-tokens (word ≈ 1.3 tokens) + prompt = "Summarize and analyze the following text in detail. " + ("data point " * int(a.in_tokens / 1.3)) + print(f"# {a.url} model={a.model} in≈{a.in_tokens} out={a.out_tokens}") + print(f"{'N':>4} {'ok':>4} {'agg_tok/s':>10} {'ttft_p50':>9} {'ttft_p99':>9} {'tpot_ms':>8}") + for N in a.concurrency: + r = run(a.url, a.model, prompt, a.out_tokens, N) + print(f"{r['N']:>4} {r['ok']:>4} {r['agg_tok_s']:>10.0f} {r['ttft_p50']:>9.2f} {r['ttft_p99']:>9.2f} {r['tpot_ms']:>8.1f}") diff --git a/stacks/sglang/compose.yaml b/stacks/sglang/compose.yaml new file mode 100644 index 0000000..cb44d97 --- /dev/null +++ b/stacks/sglang/compose.yaml @@ -0,0 +1,77 @@ +# SGLang — alternative LLM serving engine, stood up on ana-ml2 to bench +# head-to-head against vLLM on the same model + hardware. +# +# Bench-oriented (not yet a permanent service): point it at the SAME checkpoint +# vLLM serves, give it an EXCLUSIVE GPU (fair benches need no co-tenants), and +# drive both engines with one common load generator. If SGLang wins decisively, +# promote this to a real stack; otherwise tear it down after the bench. +# +# SGLang 0.5.13 supports our formats on Blackwell sm_120: compressed-tensors +# (the llm-compressor NVFP4 W4A4 output), fp8, modelopt_fp4, petit_nvfp4, and +# fp4_e2m1 KV. So SGLANG_QUANT can be compressed-tensors (NVFP4) or fp8. +# +# All tunables in .env — edit that, not this file. + +services: + sglang: + image: lmsysorg/sglang:${SGLANG_VERSION} + container_name: sglang + restart: unless-stopped + ipc: host # SGLang needs large shared memory + ports: + - "${SGLANG_PORT}:30000" + volumes: + - /tank/aimodels:/aimodels:ro # NVFP4 checkpoints + HF cache live here + environment: + - HF_HOME=/aimodels/huggingface + - HF_HUB_CACHE=/aimodels/huggingface/hub + command: + - python3 + - -m + - sglang.launch_server + - --model-path + - ${SGLANG_MODEL} # e.g. /aimodels/nvfp4/granite-4.1-8b-NVFP4-W4A4-test + - --served-model-name + - ${SGLANG_SERVED_NAME} + - --host + - 0.0.0.0 + - --port + - "30000" + - --quantization + - ${SGLANG_QUANT} # compressed-tensors (NVFP4) | fp8 + - --kv-cache-dtype + - ${SGLANG_KV_DTYPE:-fp8_e4m3} + - --context-length + - ${SGLANG_CTX_LEN:-65536} + - --mem-fraction-static + - ${SGLANG_MEM_FRACTION:-0.85} # SGLang's analog of vLLM gpu-memory-utilization + - --tp + - "1" + deploy: + resources: + reservations: + devices: + - driver: nvidia + device_ids: + - "${SGLANG_GPU_ID}" # exclusive GPU for a fair bench + capabilities: + - gpu + healthcheck: + test: ["CMD-SHELL", "python3 -c \"import urllib.request,sys; sys.exit(0 if urllib.request.urlopen('http://localhost:30000/health').status==200 else 1)\""] + interval: 30s + timeout: 10s + retries: 3 + start_period: 300s + networks: + - tnet + labels: + - homepage.group=AI Systems + - homepage.name=SGLang (bench) + - homepage.icon=mdi-speedometer + - homepage.description=SGLang serving — vLLM bench comparison (ana-ml2) + - homepage.href=http://10.250.50.54:${SGLANG_PORT} + +networks: + tnet: + name: traefik-net + external: true