NInfer benchmarks
An inference engine for running open-weight LLMs locally.
47 community reports
NInfer results by GPU
Hardware with reported NInfer results, with each report's model, quant and speed, newest first. Runs on several cards, with speculative decoding, with batched requests or with part of the model in system RAM say so, since each describes a different setup. Prompt processing is how fast the input is read, not how long you wait for the first word. How to measure time to first token.
NInfer on NVIDIA RTX 509018 reports
- Inkling 2 Mini 26B · 3.4B active · UD-IQ4_XS182.0 t/s, 6,664 t/s prompt(speculative)
- Qwen3.6 35B · 3B active389.0 t/s, 5,310 t/s prompt
- Qwen3.8210.5 t/s, 6,709 t/s prompt(speculative)
- Qwen3.8 27B · NVFP4267.0 t/s(speculative)
- Qwen3.8 27B · NVFP4287.0 t/s, 13,700 t/s prompt(speculative)
NInfer on NVIDIA RTX 40906 reports
- Qwen3.8 27B · INT8130.0 t/s
- Qwen3.8 27B · INT8149.0 t/s(speculative, part in system RAM)
- Qwen3.8 27B · Q4/Q5107.0 t/s, 5,008 t/s prompt(speculative)
- Bonsai 2 27B · Q4/Q5188.0 t/s(speculative)
- Qwen3.8 27B · E8 4-bit148.6 t/s, 1,849 t/s prompt(speculative)
NInfer on NVIDIA RTX Pro 6000 Blackwell4 reports
- Qwen3.6 35B · 3B active500.0 t/s(3 cards)
- Qwen3.8 125B · 6B active · NVFP4172.0 t/s, 5,905 t/s prompt
- Qwen3.6 35B · 3B active600.0 t/s
- Qwen3.8 27B · groupwise-int104.8 t/s, 12,400 t/s prompt
NInfer on NVIDIA V100 16GB3 reports
- Qwen3.8 27B · NVFP4353.0 t/s(4 cards, speculative)
- Qwen3.8 27B · NVFP4143.5 t/s, 1,310 t/s prompt(2 cards, speculative)
- Qwen3.8 27B · NVFP4180.0 t/s(2 cards, speculative)
NInfer on NVIDIA RTX 2080 Ti 22GB (modded)2 reports
- Qwen3.6 27B27.0 t/s, 616 t/s prompt
- Qwen3.8 27B · W8A1625.0 t/s(part in system RAM)
NInfer on NVIDIA RTX 30902 reports
- Qwen3.8 27B71.0 t/s, 862 t/s prompt(speculative)
- Qwen3.8 27B70.2 t/s(speculative)
NInfer on NVIDIA RTX 5060 Ti 16GB2 reports
- Qwen3.8 27B · NVFP475.0 t/s, 2,100 t/s prompt(2 cards, speculative)
- Qwen3.8 27B · NVFP476.7 t/s(2 cards, speculative)
NInfer on NVIDIA RTX 50802 reports
- Qwen3.8 27B96.5 t/s(speculative)
- Qwen3.8 27B · Q3G64_F16S/Q4G64_F16S/Q5G64_F16S71.6 t/s, 1,381 t/s prompt(speculative)
NInfer on NVIDIA RTX Pro 4000 Blackwell2 reports
- Qwen3.8 27B · QUASAR mixed NVFP4/FP8111.1 t/s, 5,634 t/s prompt(2 cards)
- Qwen3.8 27B67.0 t/s, 785 t/s prompt(speculative)
NInfer on NVIDIA RTX 40801 report
- Qwen3.8 27B · GSQ166.7 t/s, 2,720 t/s prompt(speculative)
NInfer on NVIDIA RTX 5070 Ti1 report
- Qwen3.8 27B · NVFP4220.0 t/s, 4,956 t/s prompt(2 cards, speculative)
NInfer on NVIDIA RTX PRO 6000 Max-Q1 report
- Qwen3.8 27B · groupwise-int96.8 t/s(speculative)
NInfer on NVIDIA V100 32GB1 report
- Qwen3.8 27B · NVFP4218.0 t/s(speculative)
NInfer, GPU not identified1 report
- Qwen3.6 35B · 3B active210.0 t/s, 4,000 t/s prompt(speculative)
This engine doesn't yet have an editorial profile on llamaperf. The community reports below show how it's been used in practice across different hardware.
Compare GPUs running NInfer
Every card is compared on one model, Qwen3.8 125B · 6B active at 4-bit, the size and bit class that most cards here have plain runs of. A card with no run of that model shows none rather than a figure from a different model.
| GPU | VRAM | Reports | Median speed, Qwen3.8 125B · 6B active 4-bit |
|---|---|---|---|
| NVIDIA RTX 5090nvidia | 32GB | 23 | no Qwen3.8 125B · 6B active 4-bit run reported |
| NVIDIA RTX 4090nvidia | 24GB | 7 | no Qwen3.8 125B · 6B active 4-bit run reported |
| NVIDIA RTX Pro 6000 Blackwellnvidia | 96GB | 4 | Gen 172.0 t/sPP 5905.0 t/s · n=1 |
| NVIDIA RTX 5060 Ti 16GBnvidia | 16GB | 3 | no Qwen3.8 125B · 6B active 4-bit run reported |
| NVIDIA RTX 5080nvidia | 16GB | 3 | no Qwen3.8 125B · 6B active 4-bit run reported |
| NVIDIA V100 16GBnvidia | 16GB | 3 | no Qwen3.8 125B · 6B active 4-bit run reported |
| NVIDIA RTX 2080 Ti 22GB (modded)nvidia | 22GB | 2 | no Qwen3.8 125B · 6B active 4-bit run reported |
| NVIDIA RTX 3090nvidia | 24GB | 2 | no Qwen3.8 125B · 6B active 4-bit run reported |
NInfer against other engines
Pairs of plain runs on the same card, of the same model size at the same bit class: one device, one request, no speculative decoding, the whole model in memory. Context length and build still differ between the two sides, and each side shows its own.
| Model and GPU | NInfer | Other engine |
|---|---|---|
| Qwen3.8 125B · 6B active 4-bitNVIDIA RTX Pro 6000 Blackwell | 172.0 t/sPrompt processing (prefill) 5905.0 tokens/s (1)one generation report, 512 context | ik_llama.cpp40.0 t/sPrompt processing (prefill) 2000.0 tokens/s (1)one generation report, 200K context |
| Qwen3.8 125B · 6B active 4-bitNVIDIA RTX Pro 6000 Blackwell | 172.0 t/sPrompt processing (prefill) 5905.0 tokens/s (1)one generation report, 512 context | llama.cpp105.5 t/sPrompt processing (prefill) 1955.0 tokens/s (1)median of 2 generation reports, 2K to 240K context |
| Qwen3.8 125B · 6B active 4-bitNVIDIA RTX Pro 6000 Blackwell | 172.0 t/sPrompt processing (prefill) 5905.0 tokens/s (1)one generation report, 512 context | SGLang-Prompt processing (prefill) 11352.0 tokens/s (1)no generation reports, 256K context |
Top models on NInfer
Frequently asked
Is NInfer faster than other engines?
llamaperf has 2 matched pairs of NInfer and other-engine runs on the same card, model size and bit class, but none is comparable enough to say which engine is faster: the sides have too few reports or were run at different context lengths. A pair counts toward a direction only when each side has at least 3 generation reports with a stated context, and the two sides' contexts overlap. The measured figures and their conditions are in the table on this page.