NInfer on the NVIDIA RTX 4090
llamaperf has 6 NInfer reports on the NVIDIA RTX 4090, none of them a plain single-card run with a model size and quant, so there is no median to quote. The reports are listed below with how each one ran.
6 NInfer reports on the NVIDIA RTX 4090
Newest first. A run on several cards, with speculative decoding, with part of the model in system RAM or with batched requests says so, since each describes a different setup.
- Qwen3.8 27B · INT8130.0 t/s
- Qwen3.8 27B · INT8149.0 t/s(speculative decoding, part in system RAM)
- Qwen3.8 27B · Q4/Q5107.0 t/s(speculative decoding)
- Bonsai 2 27B · Q4/Q5188.0 t/s(speculative decoding)
- Qwen3.8 27B · E8 4-bit148.6 t/s(speculative decoding)
- Qwen3.8 27B80.0 t/s
Frequently asked
How fast is NInfer on the NVIDIA RTX 4090?
llamaperf has 6 NInfer reports on the NVIDIA RTX 4090, none of them a plain single-card run with a model size and quant, so there is no median to quote. The reports are listed below with how each one ran.