NInfer on the NVIDIA RTX 5090
llamaperf has 16 NInfer reports on the NVIDIA RTX 5090, none of them a plain single-card run with a model size and quant, so there is no median to quote. The reports are listed below with how each one ran.
16 NInfer reports on the NVIDIA RTX 5090
Newest first. A run on several cards, with speculative decoding, with part of the model in system RAM or with batched requests says so, since each describes a different setup.
- Qwen3.8210.5 t/s(speculative decoding)
- Qwen3.8 27B · NVFP4267.0 t/s(speculative decoding)
- Qwen3.8 27B · NVFP4287.0 t/s(speculative decoding)
- Qwen3.8 27B · NVFP4158.0 t/s(speculative decoding)
- Qwen3.8 27B · NVFP4160.8 t/s(speculative decoding)
- Qwen3.8 27B · NVFP4175.0 t/s(speculative decoding)
- Qwen3.8 27B · NVFP4178.0 t/s(speculative decoding)
- Qwen3.8 27B · NVFP4205.0 t/s(speculative decoding)
- Qwen3.8 27B · Q8162.1 t/s
- Qwen3.8 27B117.0 t/s(speculative decoding, 3 requests at once)
- Qwen3.8 27B200.0 t/s(speculative decoding)
- Qwen3.8 27B · NVFP4200.0 t/s(speculative decoding)
- Qwen3.8 27B169.7 t/s
- Qwen3.8 27B · NVFP4202.0 t/s(speculative decoding)
- Qwen3.8 27B · NVFP4200.5 t/s(speculative decoding)
- Qwen3.8 27B · NVFP4119.0 t/s(2 cards, speculative decoding)
Frequently asked
How fast is NInfer on the NVIDIA RTX 5090?
llamaperf has 16 NInfer reports on the NVIDIA RTX 5090, none of them a plain single-card run with a model size and quant, so there is no median to quote. The reports are listed below with how each one ran.