Qwen2.5 7B
NVIDIA L4 · vLLM
- generation:
- 17.6 tokens/s
- quant:
- bf16 (safetensors)
User benchmarks vLLM against SGLang on one NVIDIA L4 with Qwen2.5-7B-Instruct in bf16, 128 output tokens per request with ignore_eos. Setup pins the model, GPU class, dtype, context length and memory fraction across both engines, driven through /v1/completions by a Go load generator that times TTFT at the client. Prefix caching is on except where noted. At one request each engine decodes about 17.5 t/s; at 32 concurrent short prompts vLLM reaches 459.3 t/s and SGLang 449.0 t/s. On long prompts at 32 concurrent requests vLLM gets 252.7 t/s and SGLang 310.2 t/s; turning prefix caching off drops vLLM to 147.1 t/s and raises p99 TTFT from 6.7 s to 18.6 s. SGLang reaches first token sooner under load (258 ms vs 639 ms at 32 concurrent short prompts) while vLLM is quicker at a single request (72 ms vs 120 ms). Cost per million output tokens falls from $12.60 at one request to $0.48 at 32 on the same card. Cold start was 122 s for SGLang and 149 s for vLLM. The author notes one run per cell and that throughput moved by more than the gap between engines on a re-measurement, so small differences are noise.