SGLang benchmarks
An inference engine for running open-weight LLMs locally.
38 community reports
SGLang results by GPU
Hardware with reported SGLang results, with each report's model, quant and speed, newest first. Runs on several cards, with speculative decoding, with batched requests or with part of the model in system RAM say so, since each describes a different setup. Prompt processing is how fast the input is read, not how long you wait for the first word. How to measure time to first token.
SGLang on NVIDIA RTX Pro 6000 Blackwell9 reports
- GLM-5.3 320B · 18B active · W4A16-NVFP4-K32-Experts-FP8-WO164.8 t/s(2 cards, speculative)
- Minimax M3 428B · 23B active · NVFP488.5 t/s(4 cards)
- GLM-4.5-Air 106B · 12B active · FP8133.0 t/s(4 cards)
- DeepSeek V4.1 Flash 552B · 16B active · FP4200.9 t/s, 1,943 t/s prompt(4 cards, speculative, part in system RAM)
- Qwen3.8 27B · NVFP4210.0 t/s(speculative)
SGLang on NVIDIA RTX 50907 reports
- Qwen3.8 27B · NVFP4208.2 t/s(speculative)
- Qwen3.8 27B · NVFP4139.7 t/s, 9,081 t/s prompt(speculative)
- Qwen3.8 27B · NVFP4152.0 t/s(speculative)
- Qwen3.8 27B · NVFP4100.0 t/s(speculative)
- Qwen3.8 27B · NVFP4200.0 t/s
SGLang on NVIDIA DGX Spark6 reports
- DeepSeek V4.1 Flash 552B · 16B active60.8 t/s, 2,574 t/s prompt(4 cards, speculative, part in system RAM)
- DeepSeek V4.1 Flash 552B · 16B active · MXFP487.7 t/s(4 cards, speculative)
- Qwen3.8 27B · NVFP441.0 t/s(speculative)
- Qwen3.8 27B · NVFP434.0 t/s(speculative)
- Qwen3.8 125B · 6B active · NVFP435.0 t/s
SGLang on NVIDIA RTX 30905 reports
- Qwen3.8 27B · W8A83,843 t/s prompt(4 cards, speculative)
- Qwen3.8 27B · 3.00bpw904 t/s prompt(speculative, part in system RAM)
- Qwen3.8 27B · EXL3 3.0 bpw96.2 t/s(speculative)
- Qwen3.8 27B · AWQ-INT4136.8 t/s, 1,581 t/s prompt(4 cards, speculative)
- Qwen3.8 27B · 3.00bpw98.2 t/s, 914 t/s prompt(speculative)
SGLang on NVIDIA RTX 40903 reports
- Qwen3.6 35B · 3B active · AWQ 4-bit180.0 t/s(2 cards)
- Qwen3.8 27B · escha 2-bit67.0 t/s
- Qwen3.8 27B · EXL3 3bpw135.4 t/s
SGLang on NVIDIA RTX PRO 6000 Max-Q3 reports
- DeepSeek V4.1 Flash 552B · 16B active · FP81280.0 t/s(4 cards, part in system RAM)
- Qwen3.8 27B · BF16106.6 t/s(speculative)
- Qwen3.8 125B · 6B active · NVFP4240.0 t/s(speculative)
SGLang on AMD Instinct MI300X 192GB1 report
- Qwen3-Coder 30B · 3B active · BF16173.0 t/s
SGLang on NVIDIA B200 180GB1 report
- DeepSeek V4 Flash 284B · 13B active · NVFP44.8 t/s(4 cards)
SGLang on NVIDIA H100 80GB1 report
- GLM-4.7 30B168.0 t/s(speculative)
SGLang on NVIDIA RTX 4080 Super1 report
- Qwen3.8 27B · 2.469 bpw59.0 t/s
SGLang on NVIDIA RTX 5060 Ti 16GB1 report
- Qwen3.8 27B · NVFP4126.0 t/s(2 cards, speculative)
SGLang on NVIDIA RTX Pro 4000 Blackwell1 report
- Qwen2.5 32B · AWQ71.4 t/s(2 cards, speculative)
SGLang on NVIDIA V100 32GB1 report
- Qwen3.8 125B · 6B active · NVFP460.0 t/s(4 cards, part in system RAM)
SGLang, GPU not identified2 reports
- Qwen3.8 125B · 6B active · NVFP4221.0 t/s
- Qwen3.8 27B130.0 t/s, 4,000 t/s prompt(5 requests at once)
This engine doesn't yet have an editorial profile on llamaperf. The community reports below show how it's been used in practice across different hardware.
Compare GPUs running SGLang
Every card is compared on one model, Qwen3.8 27B at 2-bit, the size and bit class that most cards here have plain runs of. A card with no run of that model shows none rather than a figure from a different model.
| GPU | VRAM | Reports | Median speed, Qwen3.8 27B 2-bit |
|---|---|---|---|
| NVIDIA RTX Pro 6000 Blackwellnvidia | 96GB | 10 | no Qwen3.8 27B 2-bit run reported |
| NVIDIA DGX Sparknvidia | 128GB | 7 | no Qwen3.8 27B 2-bit run reported |
| NVIDIA RTX 5090nvidia | 32GB | 7 | no Qwen3.8 27B 2-bit run reported |
| NVIDIA RTX 3090nvidia | 24GB | 5 | no Qwen3.8 27B 2-bit run reported |
| NVIDIA RTX 4090nvidia | 24GB | 3 | Gen 67.0 t/s |
| NVIDIA RTX PRO 6000 Max-Qnvidia | 96GB | 3 | no Qwen3.8 27B 2-bit run reported |
| AMD Instinct MI300X 192GBamd | 192GB | 2 | no Qwen3.8 27B 2-bit run reported |
| NVIDIA B200 180GBnvidia | 180GB | 1 | no Qwen3.8 27B 2-bit run reported |
SGLang against other engines
Pairs of plain runs on the same card, of the same model size at the same bit class: one device, one request, no speculative decoding, the whole model in memory. Context length and build still differ between the two sides, and each side shows its own.
| Model and GPU | SGLang | Other engine |
|---|---|---|
| Qwen3.8 125B · 6B active 4-bitNVIDIA RTX Pro 6000 Blackwell | -Prompt processing (prefill) 11352.0 tokens/s (1)no generation reports, 256K context | ik_llama.cpp40.0 t/sPrompt processing (prefill) 2000.0 tokens/s (1)one generation report, 200K context |
| Qwen3.8 125B · 6B active 4-bitNVIDIA DGX Spark | 35.0 t/sone generation report, 256K context | llama.cpp24.5 t/sone generation report, 128K context |
| Qwen3.8 125B · 6B active 4-bitNVIDIA RTX Pro 6000 Blackwell | -Prompt processing (prefill) 11352.0 tokens/s (1)no generation reports, 256K context | llama.cpp105.5 t/sPrompt processing (prefill) 1955.0 tokens/s (1)median of 2 generation reports, 2K to 240K context |
| Qwen3.8 125B · 6B active 4-bitNVIDIA RTX Pro 6000 Blackwell | -Prompt processing (prefill) 11352.0 tokens/s (1)no generation reports, 256K context | NInfer172.0 t/sPrompt processing (prefill) 5905.0 tokens/s (1)one generation report, 512 context |
| Qwen3-Coder 30B · 3B active 16-bitAMD Instinct MI300X 192GB | 173.0 t/sone generation report, 128K context | vLLM196.0 t/sone generation report, context not stated |
Top models on SGLang
Frequently asked
Is SGLang faster than other engines?
llamaperf has 2 matched pairs of SGLang and other-engine runs on the same card, model size and bit class, but none is comparable enough to say which engine is faster: the sides have too few reports or were run at different context lengths. A pair counts toward a direction only when each side has at least 3 generation reports with a stated context, and the two sides' contexts overlap. The measured figures and their conditions are in the table on this page.