llamaperf

Local model performance on your hardware

Find which open-weight LLMs fit in your GPU or Mac and compare the speeds people report on setups like yours.

What runs on your hardware?

Pick your GPU or Mac, then read the speeds people reported on it, or estimate which models fit and how fast they run.

Free to use. No account needed. Memory estimates and community measurements are labelled separately.

Looking for a particular model?

Search for a model to see its reported speeds across GPUs and Macs.

Local model performance reports from the community

These are individual setups, not a controlled benchmark. Compare GPU count, quantization, context and offloading before comparing speeds. How to read a report →

GPU: NVIDIA RTX 2080 Ti 22GB (modded)
Compare setup details

Exact recorded values. Context may be a configured limit; matching filters does not establish identical prompts, offloading or concurrency.

Tone: positiveNVIDIA hardware
generation:
60.1 tokens/s
prompt processing (prefill):
28.0 tokens/s
quant:
IQ3_S (GGUF)

User reports Qwen3.8-Flash-Next at 60.1 tok/s decode and 28 tok/s prefill on two modded RTX 2080 Ti 22GB cards, with a peak of 70.9 tok/s at 61K of 128K context. Setup is Strata with IQ3_S weights, custom tensor parallelism over an NVLink bridge on M.2 OCuLink PCIe Gen3 x4, on a Minisforum AR900i with a Core i9-13900HX and 64GB DDR5. The 60.1 tok/s figure comes from 41,288 tokens in 695 s at 41K context; the internal RTX 5070 was idle during these runs.

Nov 20, 2026
Tone: positiveNVIDIA hardware
generation:
27.0 tokens/s
prompt processing (prefill):
616.0 tokens/s
kv:
INT8
codingvision

User reports Qwen3.6/3.8-27B decoding at 27.0 tok/s without MTP on a single modded 22 GB RTX 2080 Ti under NInfer v0.3.0 on Windows 11, with prefill of 616 tok/s on a 512-token prompt. Setup uses a 64k context with an INT8 KV cache; the weights take 15.4 GiB and the engine runs entirely in VRAM with no system RAM offloading. MTP speculative decoding is on by default. With MTP, decode reaches 47.5 tok/s on a real code prompt (26.3 without) and 28.8 tok/s on a real prose prompt (26.0 without); prefill on a 64k-token prompt is 458 tok/s. The engine supports only Qwen3.6/3.8-27B, and dropping RTX 5090, FP8, FP4 and 35B-A3B code paths cut the binaries from 1.56 GB to 0.28 GB.

Oct 30, 2026
NVIDIA hardware
generation:
220.8 tokens/s
quant:
NVFP4 (NVFP4)
kv:
FP8

User reports 220.84 tok/s decode in their fork's README at a 4K prompt for Qwen3.8 27B NVFP4 on two RTX 2080 Ti 22GB cards, and 209.35 tok/s at 32K. Setup is 2x RTX 2080 Ti 22GB joined by NVLink with tensor parallel 2, NVFP4 weights, FP8 KV cache, a 256K window and DFlash2 speculative decoding (K=7), on a single request. The figures use synthetic text inputs with high speculative-hit rates, and the README says real-task throughput may not reach them. 4x Tesla T10 with Qwen3.8 27B FP8 reaches 191.89 and 189.38 tok/s on the same test.

Oct 8, 2026
Tone: positiveNVIDIA hardware
generation:
120.0 tokens/s
quant:
AWQ 4-bit (safetensors)
kv:
turboquant_k3v4_nc

User reports Qwen3.6-35B-A3B at AWQ 4-bit decoding at about 120 tokens/second under a single request on 2x modded 22GB RTX 2080 Ti with NVLink. Setup is vLLM with tensor parallelism across the two cards. The post's main benchmark ran a W8A8-INT8 build with a TurboQuant KV cache (about 395K tokens) and up to 16 concurrent requests, but those results appear only in screenshots; the author notes AWQ 4-bit is the better choice on Turing.

Oct 8, 2026
Tone: positiveNVIDIA hardware
generation:
69.0 tokens/s
quant:
UD-IQ4_XS (GGUF)
kv:
q8_0
coding

User reports Qwen3.6-35B-A3B at 4-bit (UD-IQ4_XS) scoring 89.63% pass@1 on HumanEval at 69 tok/s on a single RTX 2080 Ti 22GB. Setup is llama.cpp with UD-IQ4_XS dynamic 4-bit (4.25 bpw), q8_0 KV cache, 16384 context, flash-attn on, whole model resident in VRAM with no offload. A routing patch (MoE expansion, 20 experts instead of 8 on layers 25-39) reached 90.85% pass@1 at 56 tok/s, a 19% decode speed cost; the user calls the +2 problems within statistical noise and notes the tests are original HumanEval, not EvalPlus.

Oct 7, 2026
Tone: positiveNVIDIA hardware
generation:
59-68 tokens/s
quant:
GPTQ-Int4 (GPTQ)
kv:
turboquant_k3v4_nc
codingagentictool-usevisionlong-context

User reports Qwen3.8-27B at ~59-68 tok/s single-stream decode on 2x modded RTX 2080 Ti 22GB (SM75, TP=2, NVLink). Setup is a vLLM fork with GPTQ-Int4 self-quant, turboquant_k3v4_nc KV cache, 524,288 max-model-len (YaRN 2x), MTP K=2 speculative decoding, and vision enabled. Aggregate throughput is 364.7 tok/s at 16 lanes; 24 lanes regresses. The S4 scoped re-emission drafter measured 811.5 tok/s on copy-shaped spans but crashes at current HEAD, and v3 GPU-merge async is negative.

Oct 7, 2026
Tone: positiveNVIDIA hardware
generation:
95.5 tokens/s
prompt processing (prefill):
465.8 tokens/s
quant:
Q4_K_P (GGUF)
kv:
Q8_0
agenticlong-context

User reports Qwen3.6-35B-A3B at 95.5 t/s decode and 465.8 t/s prefill on a mixed three-GPU Turing setup of 2x CMP 50HX 10GB plus 1x RTX 2080 Ti 22GB. Setup is llama.cpp with a ported DP2A patch (PR #25834) and -fmad=false, Q4_K_P GGUF weights, Q8_0 KV cache, Flash Attention, 262144 context, MTP speculative decoding at --spec-draft-n-max 3, tensor split 1,1,2.5 with the RTX as tail stage, ubatch 448. Baseline DP4A gave 47.8 t/s decode and 372.7 t/s prefill; DP2A alone 54.0 t/s; adding -fmad=false 62.2 t/s. A repo-28k workload measured 405.4 t/s prefill and 82.3 t/s decode. An end-to-end rerun showed 91.4 t/s raw eval, 4% below the 95.5 reference. Context 368640 also works but quality beyond the trained window was not measured.

Oct 7, 2026
Tone: positiveNVIDIA hardware
generation:
36.6 tokens/s
quant:
IQ3_S (GGUF)
kv:
q8_0
long-contextagentictool-usevision

User reports Qwen3.8-27B at 36.6 tok/s decode on a single modded RTX 2080 Ti 22GB, with 262,144 tokens of context. Setup is KVMem (retrieval-based long context) with IQ3_S weights, q8_0 KV cache, MTP speculative decoding (58.6%/61.6% acceptance) and vision, using 16,552 of 22,528 MiB VRAM. The user compares against the upstream author's RTX 5060 Ti 16GB at 31.7 tok/s, and notes the lossless KV-streaming alternative runs 30-42 tok/s in the resident window but drops to ~9.6 tok/s past 135K tokens; a 260,096-token needle-in-a-haystack test hit exactly.

Oct 7, 2026
Tone: positiveNVIDIA hardware
generation:
101.3 tokens/s
prompt processing (prefill):
1841.7 tokens/s
quant:
AWQ (AWQ)
agenticlong-context

User reports Qwen3.6-27B-AWQ at 101.3 tok/s decode and 1841.7 tok/s prefill on dual modified RTX 2080 Ti 22GB cards with NVLink. Setup is vLLM 0.21.0 with AWQ Marlin, TP=2, MTP K=3, FlashInfer/FA2 attention, and FlashQLA SM70/SM75 legacy GDN prefill. The speed columns use the PP4096/TG128 repeat. The same rig reached a 735,084 token KV cache with turboquant_4bit_nc at max_model_len=262144, and passed a PP262000/TG1 gate at 785.26 tok/s prefill. A sequential 60-request Ragent6 run averaged 700.9 tok/s prefill and 35.2 tok/s generation. Gemma4 31B GPTQ reached 99.64 tok/s decode and 1655.65 tok/s prefill on the same runtime.

Oct 7, 2026
Tone: positiveNVIDIA hardware
generation:
44.1 tokens/s
quant:
IQ3_S (GGUF)
kv:
Q8_0
flash attention:
on
long-contextvision

User reports Qwen3.8-27B-GSQ-RCO at 44.07 t/s decode on a hardware-modded RTX 2080 Ti 22GB, with 38~44 t/s quoted as the overall range. Setup is a llama.cpp fork (TurboQuant 4-bit + Adaptive KV Streaming) with IQ3_S weights, 256K context (262,144 tokens), Q8_0 key cache and turbo4 value cache, a 2048 MiB GPU staging pool, and MTP speculative decoding with 2 draft tokens. Needle tests at 8K/16K/32K gave 41.23, 39.00 and 34.84 t/s decode with 81-83% draft acceptance and 100% recall at 82% depth; a sustained 256K run gave 40.55 t/s at 89.17% acceptance. Peak VRAM was 18,619 MiB and GPU power peaked at 266.4 W.

Sep 28, 2026
Tone: positiveNVIDIA hardware
generation:
25.0 tokens/s
quant:
W8A16
kv:
Q8

User reports Qwen3.8 27B at ~25 tok/s with standard autoregressive decoding (MTP0) on a modded RTX 2080 Ti 22GB. Setup is a Turing port of the NInfer engine with the official groupwise-int (W8A16) artifact and Q8 KV cache. With MTP3 speculative decoding (draft window = 3) the title gives 45 tok/s and the body ~456 tok/s, at ~65% acceptance rate. VRAM usage is ~17.5 GiB with the MTP draft weights loaded, leaving ~4.5-5.0 GiB free for KV cache.

Sep 7, 2026
Showing 1–11 of 11
Page 1 of 1

Community benchmarks snapshot

Records by GPU

AMD Strix Halo 128GB348NVIDIA RTX 3090271NVIDIA RTX 5090197NVIDIA DGX Spark99NVIDIA RTX 3060 12GB96NVIDIA RTX 5060 Ti 16GB88NVIDIA RTX Pro 6000 Blackwell81AMD Radeon AI PRO R9700 32GB81NVIDIA RTX 5070 Ti77NVIDIA RTX 409068

Records by model

3266 total
Qwen3.81567
Qwen3.6334
Gemma 4185
DeepSeek V4 Flash150
Qwen3.5145
DeepSeek V4.1 Flash103
other782

Records by engine

2308 total
llama.cpp1134
Strata234
vLLM227
MLX107
Ollama99
other507

Use cases

coding 937agentic 640long-context 442tool-use 281vision 174summarization 81math 77creative-writing 58multilingual 35rp 22text-generation 9reasoning 3
coding937agentic640long-context442tool-use281vision174summarization81math77creative-writing58

Median generation t/s by GPU

On Qwen3.8 27B at 4-bit, plain single-GPU runs. Full ranking

RTX 509080RTX Pro 600067RTX 5070 Ti55M5 Ultra 256GB50RTX 409040RX 7900 XTX40RTX 309038RX 7800 XT 16GB35RTX 5090 Laptop 24GB34RX 7900 XT32

Prompt processing, same model

M5 Ultra 256GB1800RTX 40901949RX 7900 XTX640RTX 3090561RX 7900 XT800M3 Ultra 512GB372V100 32GB941Radeon AI PRO R9700 32GB1287RTX 3080 20GB997Arc Pro B70512

Reports by model size

Qwen3.8 27B819Qwen3.8 125B · 6B active676Qwen3.6 35B · 3B active242DeepSeek V4 Flash 284B · 13B active133DeepSeek V4.1 Flash 552B · 16B active92Qwen3.6 27B81Gemma 4 26B · 4B active61GLM-5.3 320B · 18B active57

Quants

Q4_K_M395UD-Q4_K_XL189NVFP4174Q8_01194-bit106IQ4_XS86IQ3_XXS85IQ3_S84Q4_K_XL83Q480