Qwen2.5 7B
2× NVIDIA T4 16GB · vLLM
- generation:
- 58.0 tokens/s
- quant:
- AWQ (AWQ)
User benchmarks Qwen2.5-7B-Instruct-AWQ on two Kaggle T4 GPUs, comparing single GPU, data parallel and tensor parallel serving with vLLM 0.30.0. Tensor parallel (TP=2) reached 58 tok/s for one user against 32 tok/s for single GPU and data parallel, with 276 ms time to first token. At 32 concurrent requests TP reached 437 tok/s aggregate output throughput, against 382 tok/s for data parallel and 215 tok/s for single GPU. Benchmarks used 512-token prompts and 128-token replies, run inside Kaggle through a load-balancing proxy. The two T4s are connected over PCIe PHB with no NVLink, and the user notes the result may differ on a host with more than 4 vCPUs.