vLLM on the NVIDIA RTX 3090
On the NVIDIA RTX 3090, vLLM runs Qwen3.8 27B at 4-bit at a median 38.4 tokens per second for generation, from 1 plain run (144K to 200K context).
Speed by model on the NVIDIA RTX 3090
Plain runs only: one device, one request, no speculative decoding, the whole model in memory. Each row is one model size at one bit class, with the other engines' plain runs of the same model on this card beside it. Context length and build still differ between runs, and each side shows its own.
| Model | vLLM | Other engines |
|---|---|---|
| Qwen3.8 27B 4-bit | 38.4 t/sone report, 144K to 200K context | llama.cpp 35.7 t/smedian of 2, 128K context, 1 not stated |
25 vLLM reports on the NVIDIA RTX 3090
Newest first. A run on several cards, with speculative decoding, with part of the model in system RAM or with batched requests says so, since each describes a different setup.
- Qwen3.8 125B · 6B active · GPTQ-INT457.6 t/s(6 cards, part in system RAM)
- Qwen3.8 27B · W8A16100.0 t/s(2 cards, speculative decoding)
- DeepSeek V4.1 Flash 552B · 16B active · EXL3 3.0 bpw19.3 t/s(part in system RAM)
- Gemma 4 12B · QAT82.7 t/s
- Qwen3.8 27B127.0 t/s(speculative decoding)
- Qwen3.8 27B · INT8 W8A16114.8 t/s(2 cards, speculative decoding, part in system RAM)
- Qwen3.8 125B · 6B active · W4A16193.3 t/s(4 cards, speculative decoding)
- Qwen3.8 27B127.0 t/s(speculative decoding)
- Qwen3.8 125B · 6B active · W4A16116.6 t/s(4 cards, speculative decoding)
- Qwen3.8 27B · GPTQ-int4138.0 t/s(speculative decoding)
- Qwen3.8 27B · INT470.0 t/s(2 cards, speculative decoding)
- Qwen3.8 27B · Q4PP 250.0 t/s
- Qwen3.8 125B · 6B active80.0 t/s(3 cards, part in system RAM)
- Qwen3.8 125B · 6B active · IQ4_XS40.0 t/s(2 cards)
- Qwen3.8 27B · Q4100.0 t/s
- Qwen3.8 27B · INT4 AutoRound38.4 t/s
- DeepSeek V4 Flash 285B · 13B active · FP460.0 t/s(10 cards, speculative decoding)
- Qwen3.8 27B124.0 t/s(speculative decoding)
- Qwen3.8 27B · W4A1682.0 t/s
- Qwen3.8 27B · INT8100.0 t/s(speculative decoding, part in system RAM)
- Qwen3.8 27B · INT4218.3 t/s(speculative decoding)
- Qwen3.8 27B382.0 t/s(speculative decoding)
- Qwen3.8 27B · INT4165.0 t/s(2 cards, speculative decoding)
- Qwen3.6 27B29.0 t/s(4 cards)
- Qwen3.6 27B66.0 t/s
Frequently asked
How fast is vLLM on the NVIDIA RTX 3090?
On the NVIDIA RTX 3090, vLLM runs Qwen3.8 27B at 4-bit at a median 38.4 tokens per second for generation, from 1 plain run (144K to 200K context).
Is vLLM faster than other engines on the NVIDIA RTX 3090?
Same card, same model size and bit class, plain runs only: Qwen3.8 27B at 4-bit gives vLLM 38.4 t/s (1 run) and llama.cpp 35.7 t/s (2). Context length and engine build differ between the runs, so treat it as a lead, not a benchmark.