llamaperf

SGLang benchmarks

An inference engine for running open-weight LLMs locally.

38 community reports

SGLang results by GPU

Hardware with reported SGLang results, with each report's model, quant and speed, newest first. Runs on several cards, with speculative decoding, with batched requests or with part of the model in system RAM say so, since each describes a different setup. Prompt processing is how fast the input is read, not how long you wait for the first word. How to measure time to first token.

SGLang on NVIDIA RTX 30905 reports

Filter SGLang reports on the NVIDIA RTX 3090

SGLang, GPU not identified2 reports

This engine doesn't yet have an editorial profile on llamaperf. The community reports below show how it's been used in practice across different hardware.

Compare GPUs running SGLang

Every card is compared on one model, Qwen3.8 27B at 2-bit, the size and bit class that most cards here have plain runs of. A card with no run of that model shows none rather than a figure from a different model.

GPUVRAMReportsMedian speed, Qwen3.8 27B 2-bit
NVIDIA RTX Pro 6000 Blackwellnvidia96GB10no Qwen3.8 27B 2-bit run reported
NVIDIA DGX Sparknvidia128GB7no Qwen3.8 27B 2-bit run reported
NVIDIA RTX 5090nvidia32GB7no Qwen3.8 27B 2-bit run reported
NVIDIA RTX 3090nvidia24GB5no Qwen3.8 27B 2-bit run reported
NVIDIA RTX 4090nvidia24GB3Gen 67.0 t/s
NVIDIA RTX PRO 6000 Max-Qnvidia96GB3no Qwen3.8 27B 2-bit run reported
AMD Instinct MI300X 192GBamd192GB2no Qwen3.8 27B 2-bit run reported
NVIDIA B200 180GBnvidia180GB1no Qwen3.8 27B 2-bit run reported

SGLang against other engines

Pairs of plain runs on the same card, of the same model size at the same bit class: one device, one request, no speculative decoding, the whole model in memory. Context length and build still differ between the two sides, and each side shows its own.

Model and GPUSGLangOther engine
Qwen3.8 125B · 6B active 4-bitNVIDIA RTX Pro 6000 Blackwell-Prompt processing (prefill) 11352.0 tokens/s (1)no generation reports, 256K contextik_llama.cpp40.0 t/sPrompt processing (prefill) 2000.0 tokens/s (1)one generation report, 200K context
Qwen3.8 125B · 6B active 4-bitNVIDIA DGX Spark35.0 t/sone generation report, 256K contextllama.cpp24.5 t/sone generation report, 128K context
Qwen3.8 125B · 6B active 4-bitNVIDIA RTX Pro 6000 Blackwell-Prompt processing (prefill) 11352.0 tokens/s (1)no generation reports, 256K contextllama.cpp105.5 t/sPrompt processing (prefill) 1955.0 tokens/s (1)median of 2 generation reports, 2K to 240K context
Qwen3.8 125B · 6B active 4-bitNVIDIA RTX Pro 6000 Blackwell-Prompt processing (prefill) 11352.0 tokens/s (1)no generation reports, 256K contextNInfer172.0 t/sPrompt processing (prefill) 5905.0 tokens/s (1)one generation report, 512 context
Qwen3-Coder 30B · 3B active 16-bitAMD Instinct MI300X 192GB173.0 t/sone generation report, 128K contextvLLM196.0 t/sone generation report, context not stated

Top models on SGLang

Frequently asked

Is SGLang faster than other engines?

llamaperf has 2 matched pairs of SGLang and other-engine runs on the same card, model size and bit class, but none is comparable enough to say which engine is faster: the sides have too few reports or were run at different context lengths. A pair counts toward a direction only when each side has at least 3 generation reports with a stated context, and the two sides' contexts overlap. The measured figures and their conditions are in the table on this page.