Qwen3.5 9B
on NVIDIA RTX 3090 · Nano-vLLM
Summary
User reports Qwen3.5-9B BF16 at 41.4 tok/s decode on a single RTX 3090.
Setup is Nano-vLLM with BF16 weights, CUDA Graph decode, batch 1, greedy decoding, 24 GB card at 0.9 GPU memory utilization.
Decode is memory-bandwidth bound at 24.2 ms/token; eager mode gives about 17 tok/s and 4 concurrent requests give 138 tok/s aggregate. Prefill latency is 238 / 254 / 378 / 899 ms for 16 / 128 / 512 / 2048 tokens. The repo also reports Qwen3.6-27B-FP8 at about 41 tok/s on 4x RTX 4090 and Qwen3.5-9B at about 98 tok/s on 4x RTX 4090.