llamaperf

Local model performance on your hardware

Find which open-weight LLMs fit in your GPU or Mac and compare the speeds people report on setups like yours.

What runs on your hardware?

Pick your GPU or Mac, then read the speeds people reported on it, or estimate which models fit and how fast they run.

Free to use. No account needed. Memory estimates and community measurements are labelled separately.

Looking for a particular model?

Search for a model to see its reported speeds across GPUs and Macs.

Local model performance reports from the community

These are individual setups, not a controlled benchmark. Compare GPU count, quantization, context and offloading before comparing speeds. How to read a report →

GPU: NVIDIA RTX 5090
Compare setup details (1 active)

Exact recorded values. Context may be a configured limit; matching filters does not establish identical prompts, offloading or concurrency.

Tone: positiveNVIDIA hardware
generation:
182.0 tokens/s
prompt processing (prefill):
6664.0 tokens/s
quant:
UD-IQ4_XS (GGUF)
vision

User reports Inkling 2 Mini at 182 t/s decode and 6,664 t/s prefill on an RTX 5090, compared with 97 t/s decode and 2,318 t/s prefill for llama.cpp on the same GGUF. Setup is the user's NInfer-all fork with Unsloth's UD-IQ4_XS GGUF, a 10K-token prompt, and the DSpark drafter enabled; llama.cpp was Unsloth's build because mainline does not load this GGUF. The model does not fit in 16 GB or less, so on those cards experts sit in system RAM and the GPU caches the most-used ones; reading experts straight from disk is slow, about 9 t/s on the RTX 5060. The post also lists results for the RTX 4090, 3090, 5080, 3060 and 5060.

Nov 21, 2026
NVIDIA hardware
generation:
389.0 tokens/s
prompt processing (prefill):
5310.0 tokens/s

User reports Qwen3.6-35B-A3B at 389.0 t/s decode and 5,310 t/s prefill on a single RTX 5090. Setup is the NInfer engine (ninfer-ext fork) with no quant code stated, one model at a time, no Python in the inference path. Decode figures are aggregate rates for the stated number of lanes in flight, not per-request latency; the C=1 prefill figure is a true per-request rate while C=2 and above are point aggregates that include each request's decode time. MTP rows draft 3 tokens with the full proposal head and reach 650.3 t/s at C=1, 958.0 at C=2, 1,163.9 at C=4 and 1,368.0 at C=8. The same chart also covers Gemma 4 26B-A4B, Gemma 4 31B IT M1, Qwen3.6-35B-A3B nvfp4, Qwen3.8-27B groupwise-int, Qwen3.8-27B nvfp4 and Qwen3.8-Flash-Next nvfp4.

Oct 14, 2026
Tone: positiveNVIDIA hardware
generation:
210.5 tokens/s
prompt processing (prefill):
6708.5 tokens/s
coding

User reports Qwen3.8 at 210.5 t/s decode and 6708.5 t/s prefill on a single RTX 5090. Setup is NInfer with MTP speculative decoding, 1690 prompt tokens and 3029 generated tokens, 254 ms TTFT. The user runs a demoscene generator where the model writes Three.js effects and revises them from captured video; LM Studio is also supported for the regular workflow.

Oct 9, 2026
NVIDIA hardware
generation:
267.0 tokens/s
quant:
NVFP4 (NVFP4)
codinglong-context

User reports Qwen3.8-27B NVFP4 on an RTX 5090 decoding at 267-282 tok/s on a mixed short-prompt set with the DFlash2 block drafter, against 192 tok/s for the recipe's MTP-10 and 77 tok/s with speculation off. Setup is NInfer with NVFP4 weights and speculative decoding; the same campaign also measured llama.cpp, Ollama and vLLM, and a 5080 16 GB desktop for comparison. The 450 tok/s NVFP4 recipe figure reproduced at 426-457 tok/s on a recall workload that copies from the prompt, while ordinary prose decodes at 113-141 tok/s; 8 concurrent requests totalled 673 tok/s with MTP-3.

Oct 7, 2026
Tone: positiveNVIDIA hardware
generation:
287.0 tokens/s
prompt processing (prefill):
13700.0 tokens/s
quant:
NVFP4 (NVFP4)
agenticcoding

User compares NInfer's official Qwen3.8-27B quant (part NVFP4, part FP8) against QUASAR's QAT full-NVFP4 checkpoint with DFlash2 embedded, both on a single RTX 5090. QUASAR reaches 484 tok/s on JSON output, 204 tok/s on prose, and 13.7k tok/s prefill, using 27.0 GB VRAM. The official quant gets 400 tok/s JSON, 176 tok/s prose, 11.4k tok/s prefill, and 30.8 GB VRAM. On a real agentic task the QUASAR build averages 287 tok/s over 33k tokens. GPQA Diamond scores 89.9% versus 90.4% for the official quant, and perplexity is about 1.9% worse.

Oct 3, 2026

Qwen3.8 27B

NVIDIA RTX 5090 · NInfer · 240,000 ctx

Tone: positiveNVIDIA hardware
generation:
158.0 tokens/s
prompt processing (prefill):
7265.0 tokens/s
quant:
NVFP4 (NVFP4)
kv:
FP8
long-contextsummarizationtool-useagentic

User reports Qwen3.8-27B at 158 tok/s decode and 7,265 tok/s prefill on an RTX 5090 32GB (eGPU via OCuLink Gen4 x4). Setup is NInfer with NVFP4 weights, FP8 KV cache, 240K context, MTP3 speculative decoding at 76% acceptance, and 2 concurrent lanes. Decode rises to 213 tok/s at 32K and 202 tok/s at 128K; prefill is 6,892 tok/s at 32K and 3,904 tok/s at 128K. User compares against llama.cpp (Q5_K_M GGUF, q8_0 KV, 196K context, MTP on) at 114 tok/s decode and 1,545 tok/s prefill at 1K, and notes vLLM at about 70 tok/s decode at short context. Quality was statistically indistinguishable across engines on a 250+ item eval. NInfer does not support json_mode.

Sep 27, 2026

Qwen3.8 27B Swift-1.5

NVIDIA RTX 5090 · NInfer · 262,144 ctx

NVIDIA hardware
generation:
160.8 tokens/s
prompt processing (prefill):
3269.0 tokens/s
quant:
NVFP4 (NVFP4)
kv:
k8v4
codingagenticlong-contextvision

User reports Swift-1.5 Qwen3.8-27B at 160.8 t/s decode on a single RTX 5090 at 262,144 context. Setup is the NInfer v3 engine with an all-NVFP4 (W4A4 gs16) artifact plus a z-lab DFlash2 drafter at K=7, k8v4 KV cache, 18.0 GiB of weights in VRAM, 450 W power cap, and concurrency 4. Prefill at 200k context measured 3,269 t/s. IFBench prompt-strict 69.0, prompt-loose 72.7, instr-strict 70.4, instr-loose 73.6; GSM8K-200 95.0% (190/200); long-context needle at 250,031 tokens exact across 3 depths.

Sep 25, 2026

Qwen3.8 27B Uncensored

NVIDIA RTX 5090 · NInfer · 262,144 ctx

Tone: positiveNVIDIA hardware
generation:
175.0 tokens/s
quant:
NVFP4
kv:
Q4
mtp (multi-token prediction):
on
agenticcodinglong-context

User reports Qwen3.8 27B uncensored at 175 t/s on an RTX 5090 32GB. Setup is NInfer with NVFP4 / groupwise-int quantization and MTP enabled, running at 262,144 context with a Q4 KV cache. The user says the Q4 KV cache matched Q8 quality and that the longer context enabled long reasoning tasks in a Hermes harness.

Sep 24, 2026
Tone: positiveNVIDIA hardware
generation:
178.0 tokens/s
quant:
NVFP4

User reports Qwen3.8 27B NVFP4 at 178 t/s on an RTX 5090 after undervolting the GPU and capping CPU power at 50%. Setup uses the Ninfer inference engine. GPU memory was overclocked by 2400 MHz and the GPU was undervolted; CPU power was limited to 50% with no measurable TPS loss. A systemd service polls GPU usage and applies the CPU cap automatically during inference. Stock settings gave 172 t/s in synthetic tests; undervolting and VRAM overclocking raised this to 178 t/s, and DFlash2 speculative decoding currently yields 200 t/s. GPU power dropped from 600 W to under 450 W and GPU temperature from 75°C to 62°C.

Sep 18, 2026
Tone: positiveNVIDIA hardware
generation:
205.0 tokens/s
prompt processing (prefill):
9000.0 tokens/s
quant:
NVFP4 (NVFP4)
kv:
fp8
codingcreative-writinglong-contextvision

User reports Qwen3.8 27B Huihui abliterated NVFP4 at 205 t/s decode on a single RTX 5090 32GB, with prefill around 9,000 t/s at 8K context. Setup is NInfer on Windows 11 + WSL2 with fp8 KV cache at 196,608 context, MTP speculative decoding with 3 draft tokens and --lm-head-draft, vision enabled, weights about 19.7 GB. Decode varies with MTP acceptance: 253 t/s on predictable text, 205 t/s on coding, about 120 t/s on creative prose, and 60-80 t/s with MTP off. At 128K context decode is about 115 t/s and prefill about 4,500 t/s.

Sep 17, 2026

Qwen3.8 27B

NVIDIA RTX 5090 · NInfer · 200,000 ctx

Tone: positiveNVIDIA hardware
generation:
162.1 tokens/s
prompt processing (prefill):
4260.0 tokens/s
quant:
Q8
kv:
Q8

User reports Qwen3.8 27B at ~162.1 t/s generation and ~4.26k t/s prefill on an RTX 5090. Setup is the Ninfer Windows Edition with a Q8 KV cache at 200k context. Switching from LM Studio Q6 to Ninfer roughly doubled both decode and prompt speeds.

Sep 17, 2026

Qwen3.8 27B

NVIDIA RTX 5090 · NInfer · 555,000 ctx

Tone: positiveNVIDIA hardware
generation:
117.0 tokens/s
prompt processing (prefill):
1600.0 tokens/s
kv:
NVFP4
long-contextagenticcoding

User reports a fork of NInfer with a custom NVFP4 KV cache, YaRN context extension to 555k, multi-level prefix reuse with a host KV safety net, tool calling improvements, and monitoring. Benchmarks show decode at 117 tok/s at 400k+ context, cold prefill of 260s for 414k tokens at 1600 tok/s, H2D restore in 0.4s, and 30GB of host KV. Quality results show LongBench matching int8, AIME at 96.7%, and needle-in-haystack at 100%.

Sep 9, 2026
Tone: positiveNVIDIA hardware
generation:
200.0 tokens/s

User reports Qwen3.8-27B at about 200 t/s generation on a single RTX 5090 with speculative decoding, on day-0 support in NInfer. The setup uses a shared paged KV cache and ReplaySSM for GDN, with PDL in use. The user notes support for up to 8 concurrent requests.

Sep 7, 2026

Qwen3.8 27B

NVIDIA RTX 5090 · NInfer · 262,144 ctx

Tone: positiveNVIDIA hardware
generation:
200.0 tokens/s
prompt processing (prefill):
5950.0 tokens/s
quant:
NVFP4

User reports an aggregate 880 t/s at 6 parallel requests, peaking at 967 t/s, and over 200 t/s single stream with MTP speculative decoding. Prefill runs at about 5,950 t/s. Weights take 16.8 GiB, leaving about 13 GiB for KV cache. Benchmarks are HumanEval+ 152/164 and AIME25+AIME26 55/60, identical to int4 but 1.56x-1.98x faster. The run requires Blackwell FP4 cores and a 6-line patch not yet upstream.

Sep 7, 2026
NVIDIA hardware
generation:
169.7 tokens/s
kv:
Q8
coding

User reports median decode speeds by context size on an unspecified setup: 169.7 t/s under 50k, 167.6 t/s at 50-100k, 156.9 t/s at 100-150k, 149.0 t/s at 150-200k, and 143.7 t/s at 200k and above. Context length is set to maximum. Peak decode for a single request is 222.0 t/s, with 211.2 t/s sustained over 5s.

Sep 7, 2026

Qwen3.8 27B

NVIDIA RTX 5090 · NInfer · 240,000 ctx

Tone: positiveNVIDIA hardware
generation:
202.0 tokens/s
prompt processing (prefill):
3904.0 tokens/s
quant:
NVFP4
kv:
FP8
mtp (multi-token prediction):
on
long-contexttool-useagentic

User compares NInfer, llama.cpp and vLLM for Qwen3.8-27B on an RTX 5090. NInfer with NVFP4 and MTP3 reaches 158-213 t/s decode and 3904-7265 t/s prefill. NInfer outperforms llama.cpp with Q5_K_M by 1.4-2.8x decode and 2.6-4.7x prefill. Quality is statistically indistinguishable across engines. NInfer lacks json_mode support, and vLLM speed is not directly comparable due to wall-clock timing.

Sep 7, 2026
Tone: positiveNVIDIA hardware
generation:
200.5 tokens/s
prompt processing (prefill):
2587.0 tokens/s
quant:
NVFP4
kv:
Q8
mtp (multi-token prediction):
on
agenticcoding

User benchmarks 180K context with MTP5 enabled, reaching 2587 t/s prefill and 200.5 t/s decode. The run also covers 64K and 120K contexts. A synthetic corpus yields high MTP acceptance, while real agentic coding logs show about 51% accept and about 154 t/s.

Sep 7, 2026

Qwen3.8 27B

2× NVIDIA RTX 5090 · NInfer · 1,048,576 ctx

Tone: positiveNVIDIA hardware
generation:
119.0 tokens/s
quant:
NVFP4
kv:
Q8

User reports a fork of NInfer with tensor parallelism and YaRN scaling decoding at 119 t/s with MTP and 57 t/s without at 653k context, and 48 t/s decode at 1M context, around 100 t/s with MTP. Setup is a fork of NInfer with tensor parallelism and YaRN scaling. Against vLLM at 42 t/s at 653k, MTP acceptance drops to zero past 262k. Prefill is 1.2-1.3x slower than vLLM. Two GPUs reach 75 t/s versus 54 t/s on one at 250k.

Sep 7, 2026
Showing 1–18 of 18
Page 1 of 1

Community benchmarks snapshot

Records by GPU

AMD Strix Halo 128GB348NVIDIA RTX 3090271NVIDIA RTX 5090197NVIDIA DGX Spark99NVIDIA RTX 3060 12GB96NVIDIA RTX 5060 Ti 16GB88NVIDIA RTX Pro 6000 Blackwell81AMD Radeon AI PRO R9700 32GB81NVIDIA RTX 5070 Ti77NVIDIA RTX 409068

Records by model

3266 total
Qwen3.81567
Qwen3.6334
Gemma 4185
DeepSeek V4 Flash150
Qwen3.5145
DeepSeek V4.1 Flash103
other782

Records by engine

2308 total
llama.cpp1134
Strata234
vLLM227
MLX107
Ollama99
other507

Use cases

coding 937agentic 640long-context 442tool-use 281vision 174summarization 81math 77creative-writing 58multilingual 35rp 22text-generation 9reasoning 3
coding937agentic640long-context442tool-use281vision174summarization81math77creative-writing58

Median generation t/s by GPU

On Qwen3.8 27B at 4-bit, plain single-GPU runs. Full ranking

RTX 509080RTX Pro 600067RTX 5070 Ti55M5 Ultra 256GB50RTX 409040RX 7900 XTX40RTX 309038RX 7800 XT 16GB35RTX 5090 Laptop 24GB34RX 7900 XT32

Prompt processing, same model

M5 Ultra 256GB1800RTX 40901949RX 7900 XTX640RTX 3090561RX 7900 XT800M3 Ultra 512GB372V100 32GB941Radeon AI PRO R9700 32GB1287RTX 3080 20GB997Arc Pro B70512

Reports by model size

Qwen3.8 27B819Qwen3.8 125B · 6B active676Qwen3.6 35B · 3B active242DeepSeek V4 Flash 284B · 13B active133DeepSeek V4.1 Flash 552B · 16B active92Qwen3.6 27B81Gemma 4 26B · 4B active61GLM-5.3 320B · 18B active57

Quants

Q4_K_M395UD-Q4_K_XL189NVFP4174Q8_01194-bit106IQ4_XS86IQ3_XXS85IQ3_S84Q4_K_XL83Q480