vLLM benchmarks
An inference server with PagedAttention, built for serving many requests at once.
147 community reports
vLLM results by GPU
Hardware with reported vLLM results, with each report's model, quant and speed, newest first. Runs on several cards, with speculative decoding, with batched requests or with part of the model in system RAM say so, since each describes a different setup.
vLLM on NVIDIA DGX Spark38 reports
- Qwen3.8 125B · 6B active · NVFP445.8 t/s, 1,717 t/s prompt(speculative)
- Qwen3.8 27B · NVFP442.2 t/s, 2,158 t/s prompt(speculative)
- Qwen3.8 125B · 6B active · NVFP486.0 t/s(2 cards, speculative)
- Qwen3.8 · NVFP4 / MXFP8 mixed118.6 t/s(2 cards, speculative)
- Qwen3.8 · NVFP4 / MXFP8 mixed78.6 t/s(speculative)
vLLM on NVIDIA RTX 309025 reports
- Qwen3.8 125B · 6B active · GPTQ-INT457.6 t/s, 6,574 t/s prompt(6 cards, part in system RAM)
- Qwen3.8 27B · W8A16100.0 t/s(2 cards, speculative)
- DeepSeek V4.1 Flash 552B · 16B active · EXL3 3.0 bpw19.3 t/s(part in system RAM)
- Gemma 4 12B · QAT82.7 t/s
- Qwen3.8 27B127.0 t/s, 1,440 t/s prompt(speculative)
vLLM on NVIDIA RTX Pro 6000 Blackwell15 reports
- Qwen3.8 125B · 6B active · FP4287.0 t/s(speculative)
- Gemma 3 4B · BF16132.5 t/s(speculative)
- Qwen3 8B86.0 t/s(32 requests at once)
- Minimax M3 428B · 23B active · MXFP4113.0 t/s, 2,800 t/s prompt(4 cards)
- GLM-5.1 744B · 40B active · NVFP490.6 t/s(8 cards, speculative)
vLLM on AMD Radeon AI PRO R9700 32GB13 reports
- Qwen3.8 125B · 6B active · MXFP4-FP8-GPTQ70.0 t/s(4 cards, speculative)
- Qwen3.8 27B · W3A4179.7 t/s(speculative)
- Qwen3.8 27B · MXFP4113.0 t/s(2 cards)
- Qwen3.8 27B · MXFP4184.9 t/s, 3,948 t/s prompt(2 cards)
- Qwen3.8 27B · NVFP4276.0 t/s, 5,809 t/s prompt
vLLM on NVIDIA RTX 509012 reports
- Qwen3.8 27B · NVFP473.2 t/s
- Qwen3.8 27B · NVFP4304.0 t/s, 12,500 t/s prompt(speculative)
- DeepSeek V4.1 Flash 552B · 16B active · MXFP497.2 t/s(8 cards, part in system RAM)
- Qwen3.8 27B · FP881.3 t/s, 3,160 t/s prompt(2 cards, speculative)
- Qwen3.8 27B · NVFP477.2 t/s
vLLM on NVIDIA CMP 170HX 64GB (unlocked)7 reports
- DeepSeek V4.1 Flash 552B · 16B active117.0 t/s, 6,066 t/s prompt(8 cards, speculative, part in system RAM)
- Qwen3.8 125B · 6B active · W4A16101.3 t/s, 3,191 t/s prompt(2 cards, speculative, part in system RAM)
- DeepSeek V4 Flash · FP895.0 t/s(4 cards, 2 requests at once, speculative)
- Qwen3-Next 80B · 3B active · W4A16116.0 t/s
- DeepSeek V4 Flash 284B · 13B active · MXFP4+FP898.1 t/s, 5,300 t/s prompt(4 cards, speculative)
vLLM on AMD Radeon Pro V620 32GB4 reports
- Qwen3.8 125B · 6B active · AWQ W4A1664.0 t/s, 1,200 t/s prompt(4 cards, part in system RAM)
- Qwen3.8 27B · Q442.0 t/s(2 cards)
- Qwen3.8 125B · 6B active · W4A1670.0 t/s, 1,300 t/s prompt(4 cards, speculative)
- Qwen3.8 125B · 6B active · W4A1670.0 t/s, 1,386 t/s prompt(4 cards)
vLLM on Intel Arc Pro B704 reports
- Qwen3.8 125B · 6B active · FP820.0 t/s, 450 t/s prompt(4 cards, speculative)
- Qwen3.8 125B · 6B active · W4A16-AutoRound70.0 t/s, 2,800 t/s prompt(4 cards, 2 requests at once)
- Qwen3.8 27B · EXL3 4.00bpw91.2 t/s, 2,421 t/s prompt(speculative)
- Qwen3.8 27B · INT452.2 t/s, 763 t/s prompt(speculative)
vLLM on NVIDIA RTX 3080 20GB4 reports
- Qwen3.8 27B · AWQ-INT42,080 t/s prompt(2 cards, speculative)
- Qwen3.8 27B · AWQ-INT41,168 t/s prompt(2 cards, speculative)
- Qwen3.8 27B · W4A16143.0 t/s, 889 t/s prompt(4 cards, speculative)
- Qwen3.8 27B · AWQ-INT457.7 t/s, 2,500 t/s prompt(2 cards)
vLLM on AMD RX 7900 XTX3 reports
- Qwen3.8 27B · BF1650.0 t/s(6 cards)
- Model not named100.0 t/s(4 cards)
- Gemma 4 8B · FP1658.0 t/s, 83 t/s prompt
vLLM on AMD Strix Halo 128GB3 reports
- Qwen3.8 125B · 6B active · AWQ W4A1643.0 t/s, 1,539 t/s prompt(speculative)
- Qwen3.6 35B · 3B active · bf1632.6 t/s, 3,581 t/s prompt
- Qwen3.5 0.8B · BF1686.6 t/s
vLLM on NVIDIA H100 80GB3 reports
- DeepSeek V4 Flash 284B · 13B active · W4A16-FP8193.0 t/s(speculative, part in system RAM)
- Gemma 4 31B125.3 t/s(speculative)
- Qwen3.6 27B45.0 t/s(2 cards, 20 requests at once)
vLLM on NVIDIA RTX 2080 Ti 22GB (modded)3 reports
- Qwen3.8 27B · NVFP4220.8 t/s(2 cards, speculative)
- Qwen3.6 35B · 3B active · AWQ 4-bit120.0 t/s(2 cards)
- Qwen3.6 27B · AWQ101.3 t/s, 1,842 t/s prompt(2 cards, speculative)
vLLM on NVIDIA RTX 5060 Ti 16GB3 reports
- Qwen3.8 27B27.0 t/s(2 cards)
- Qwen3.8 27B · NVFP4132.0 t/s, 4,000 t/s prompt(2 cards, speculative)
- Qwen3.8 27B · NVFP467.3 t/s, 953 t/s prompt(2 cards, speculative)
vLLM on AMD RX 7900 XT2 reports
- Qwen2.5 7B · GPTQ-Int4115.0 t/s
- Gemma 4 31B · w4a1643.0 t/s(2 cards)
vLLM on NVIDIA Jetson AGX Thor 128GB2 reports
- Qwen3.8 27B · NVFP425.0 t/s, 3,557 t/s prompt
- Qwen3.6 35B · 3B active · NVFP4139.1 t/s(speculative)
vLLM on NVIDIA RTX PRO 6000 Max-Q2 reports
- DeepSeek V4.1 Flash 552B · 16B active249.1 t/s, 17,982 t/s prompt(4 cards, speculative)
- Qwen3.8 125B · 6B active · NVFP4177.0 t/s
vLLM on AMD Instinct MI300X 192GB1 report
- Qwen3-Coder 30B · 3B active · BF16196.0 t/s
vLLM on NVIDIA B300 288GB1 report
- Kimi K3 2800B · 104B active · MXFP492.0 t/s(8 cards)
vLLM on NVIDIA CMP 170HX 40GB (unlocked)1 report
- Qwen3.8 27B · W4A16225.0 t/s
vLLM on NVIDIA H2001 report
- Qwen3.8 27B · FP8165.0 t/s(speculative)
vLLM on NVIDIA RTX 3060 Ti 8GB1 report
- Qwen3.8 27B · W4A16120.0 t/s(4 cards, speculative)
vLLM on NVIDIA RTX 3090 Ti1 report
- Qwen3.8 27B · W4A1634.0 t/s(4 requests at once)
vLLM on NVIDIA RTX 40901 report
- DeepSeek V4 Flash 284B · 13B active180.0 t/s, 5,000 t/s prompt(4 cards, speculative)
vLLM on NVIDIA RTX 5070 Ti1 report
- Qwen3.8 27B · FP882.2 t/s, 6,488 t/s prompt(4 cards)
vLLM on NVIDIA T4 16GB1 report
- Qwen2.5 7B · AWQ58.0 t/s(2 cards)
vLLM on NVIDIA V100 32GB1 report
- Qwen3.8 125B · 6B active · AWQ201.0 t/s(4 cards, speculative, part in system RAM)
vLLM, GPU not identified12 reports
- Qwen3.8 125B · 6B active · W4A1687.7 t/s(4 cards)
- Qwen3.8 27B · INT450.9 t/s(4 cards)
- Qwen3.8 27B · INT4105.9 t/s, 3,809 t/s prompt(4 cards, speculative)
- Nemotron 3.5 Lightning 30B · 3B active80.0 t/s
- Qwen3.5 35B · 3B active · GPTQ-Int431.4 t/s, 2,001 t/s prompt
vLLM is built for serving many requests together. PagedAttention keeps the KV cache in blocks, the way an operating system pages memory, so many requests can share GPU memory and throughput holds up as requests are added.
llamaperf's medians count one request at a time, which isn't the workload vLLM is designed around. When someone reports batched throughput, the figure stays on their report and never enters a median.
vLLM's GPU backends are CUDA for NVIDIA and ROCm for AMD. It has no Apple Silicon GPU backend of its own.
Compare GPUs running vLLM
Every card is compared on one model, Qwen3.8 27B at 4-bit, the size and bit class that most cards here have plain runs of. A card with no run of that model shows none rather than a figure from a different model.
| GPU | VRAM | Reports | Median speed, Qwen3.8 27B 4-bit |
|---|---|---|---|
| NVIDIA DGX Sparknvidia | 128GB | 43 | no Qwen3.8 27B 4-bit run reported |
| NVIDIA RTX 3090nvidia | 24GB | 30 | Gen 38.4 t/sPP 561.0 t/s · n=2 |
| NVIDIA RTX Pro 6000 Blackwellnvidia | 96GB | 21 | no Qwen3.8 27B 4-bit run reported |
| NVIDIA RTX 5090nvidia | 32GB | 19 | Gen 73.2 t/s |
| AMD Radeon AI PRO R9700 32GBamd | 32GB | 16 | no Qwen3.8 27B 4-bit run reported |
| NVIDIA CMP 170HX 64GB (unlocked)nvidia | 64GB | 7 | no Qwen3.8 27B 4-bit run reported |
| AMD Radeon Pro V620 32GBamd | 32GB | 6 | no Qwen3.8 27B 4-bit run reported |
| Intel Arc Pro B70intel | 32GB | 5 | no Qwen3.8 27B 4-bit run reported |
vLLM against other engines
Pairs of plain runs on the same card, of the same model size at the same bit class: one device, one request, no speculative decoding, the whole model in memory. Context length and build still differ between the two sides, and each side shows its own.
| Model and GPU | vLLM | Other engine |
|---|---|---|
| Gemma 4 26B · 4B active 4-bitNVIDIA DGX Spark | 48.2 t/sone generation report, context not stated | llama.cpp69.9 t/sPrompt processing (prefill) 2657.0 tokens/s (1)one generation report, 256K context |
| Qwen3.6 35B · 3B active 16-bitAMD Strix Halo 128GB | 32.6 t/sPrompt processing (prefill) 3581.0 tokens/s (1)one generation report, 2K context | llama.cpp21.5 t/sPrompt processing (prefill) 484.0 tokens/s (1)one generation report, context not stated |
| Qwen3.6 35B · 3B active 4-bitNVIDIA DGX Spark | 78.3 t/sone generation report, 128K context | llama.cpp-Prompt processing (prefill) 3556.7 tokens/s (1)no generation reports, context not stated |
| Qwen3.8 27B 4-bitNVIDIA RTX 3090 | 38.4 t/sPrompt processing (prefill) 561.0 tokens/s (2)one generation report, 144K to 200K context | llama.cpp38.7 t/smedian of 4 generation reports, 128K to 256K context, 1 not stated |
| Qwen3.8 27B 4-bitNVIDIA RTX 5090 | 73.2 t/sone generation report, context not stated | llama.cpp84.2 t/smedian of 3 generation reports, 40K to 120K context, 1 not stated |
| Qwen3-Coder 30B · 3B active 16-bitAMD Instinct MI300X 192GB | 196.0 t/sone generation report, context not stated | SGLang173.0 t/sone generation report, 128K context |
| Qwen3.8 27B 4-bitNVIDIA RTX 5090 | 73.2 t/sone generation report, context not stated | TensorSharp80.0 t/sone generation report, 40K context |
Top models on vLLM
Frequently asked
Is vLLM faster than other engines?
llamaperf has 6 matched pairs of vLLM and other-engine runs on the same card, model size and bit class, but none is comparable enough to say which engine is faster: the sides have too few reports or were run at different context lengths. A pair counts toward a direction only when each side has at least 3 generation reports with a stated context, and the two sides' contexts overlap. The measured figures and their conditions are in the table on this page.
Does vLLM run on Apple Silicon?
Not on the Mac's GPU. Its GPU backends are CUDA and ROCm. On a Mac, people run llama.cpp's Metal backend or MLX.
What models does vLLM support?
Most open-weight architectures, including Llama, Qwen, Mistral, DeepSeek, Phi and Gemma. It loads AWQ, GPTQ, FP8 and bitsandbytes quants among others.