llamaperf

Local model performance on your hardware

Find which open-weight LLMs fit in your GPU or Mac and compare the speeds people report on setups like yours.

What runs on your hardware?

Pick your GPU or Mac, then read the speeds people reported on it, or estimate which models fit and how fast they run.

Free to use. No account needed. Memory estimates and community measurements are labelled separately.

Looking for a particular model?

Search for a model to see its reported speeds across GPUs and Macs.

Local model performance reports from the community

These are individual setups, not a controlled benchmark. Compare GPU count, quantization, context and offloading before comparing speeds. How to read a report →

GPU: NVIDIA RTX Pro 4000 Blackwell
Compare setup details

Exact recorded values. Context may be a configured limit; matching filters does not establish identical prompts, offloading or concurrency.

Qwen3.8 27B

2× NVIDIA RTX Pro 4000 Blackwell · NInfer · 39,843 ctx

NVIDIA hardware
generation:
111.1 tokens/s
prompt processing (prefill):
5634.0 tokens/s
quant:
QUASAR mixed NVFP4/FP8
codingmath

User benchmarks Qwen3.8-27B at 155.9 t/s decode on a ~40k-token prompt using two RTX PRO 4000 Blackwell 24GB cards in tensor parallel 2. Setup is NInfer v0.9.0 (ninfer-serve, CUDA) with a QUASAR mixed NVFP4/FP8 artifact, int8 KV cache, 262144-token context and DFlash2 speculative decoding with 7 draft tokens; GPUs are power-capped to 145 W each. A second long-context run gave 163.8 t/s decode and 5590 t/s prefill; prose decode was 111.1 t/s and short reasoning tasks averaged 204.7 t/s with 10/10 correct. A ~9k-token prompt had a median TTFT of 1.31 s. Timings are client-side, one request at a time.

Nov 9, 2026

Qwen3.8 27B

2× NVIDIA RTX Pro 4000 Blackwell · llama.cpp · 256,000 ctx

Tone: positiveNVIDIA hardware
generation:
68.0 tokens/s
quant:
Q8 (GGUF)
agentictool-usecodinglong-contextvisionsummarization

User reports a fully local voice and chat assistant running Qwen3.8-27B (dense, 27.3B, Q8) at roughly 68 tok/s decode and ~1.5K tok/s prefill on 2x RTX PRO 4000 Blackwell 24GB, with an RTX 5060 Ti 16GB handling display and helper models. Setup is a patched llama.cpp build for Blackwell with speculative decoding (MTP, ~62% draft acceptance) and a vision projector, serving 256K context across 2 concurrent slots. The machine is an AMD Threadripper 9960X with 128 GB DDR5 ECC on Ubuntu 26.04, drawing ~820 W across GPUs and CPU. The user built a custom Python proxy (~22 modules) in front of llama-server with tool-calling, context compaction, a document workspace, and six MCP servers (filesystem, SearXNG web search, ChromaDB RAG with bge-m3, reminders, documents, sandboxed code editor). A voice pipeline with wake word, faster-whisper STT and Piper TTS is currently paused. The user asks for recommendations and comments.

Oct 16, 2026
NVIDIA hardware
generation:
13.6 tokens/s
prompt processing (prefill):
338.4 tokens/s
quant:
MXFP4 (GGUF)

User reports DeepSeek-V4-Flash at 13.61 t/s generation and 338.41 t/s prompt eval on 2 x RTX PRO 4000 with 48 GiB VRAM and 160 GiB system RAM. Setup is llama.cpp with an MXFP4/Q8_0 GGUF (about 146 GiB) at 32768 context, using --fit on and --no-mmap. A llama-batched-bench sweep from 8K to 1M context shows prompt throughput falling from 448.57 t/s to 65.97 t/s and generation from 19.20 t/s to 6.56 t/s; without hc-ops the same sweep was slower (322.50 t/s prompt, 14.62 t/s generation at 8K). KLD against the base logits is 0.422277 mean for the Q2 quant and 0.115089 for the Q4 quant.

Oct 8, 2026
Tone: positiveNVIDIA hardware
generation:
71.4 tokens/s
quant:
AWQ (AWQ)
kv:
fp8_e5m2
codinglong-context

User benchmarks Qwen2.5-32B-Instruct-AWQ at 71.37 tok/s median decode on 2x RTX PRO 4000 Blackwell. Setup is SGLang 0.5.19 with AWQ weights, NGRAM speculative decoding (K=5), tensor parallelism 2 over PCIe Gen4, and FP8 KV cache. NGRAM speculative decoding gives a 1.28x median speedup over the 55.60 tok/s baseline, with high inter-prompt variance (std dev 19.18 tok/s). Per-domain decode ranges are 65-107 tok/s for JSON, 60-226 tok/s for code, and 58-82 tok/s for prose. FP8 KV cache alone measured 54.77 tok/s, a 1.5% regression.

Sep 26, 2026
Tone: positiveNVIDIA hardware
generation:
67.0 tokens/s
prompt processing (prefill):
785.0 tokens/s
kv:
Q8

User reports 67 t/s with MTP3 speculative decoding enabled, against 24.4 tok/s without MTP. Setup uses an INT8 KV cache with group-64. The figure comes from a 128K NIAH benchmark with 130,048 prompt tokens, where MTP acceptance was 100% on a deterministic answer. Roughly 727 MiB of VRAM was left.

Sep 7, 2026
Tone: positiveNVIDIA hardware
generation:
33.9 tokens/s

User reports Qwen3.8:27b at 33.91 t/s on 2x RTX PRO 4000s and an RTX PRO 2000, up from 12.2 t/s. Splitting the GPUs across VMs produced the gain. User also reports muse-glimmer:30B at 22.61 t/s on the same setup, up from 14.3 t/s.

Sep 7, 2026
Showing 1–6 of 6
Page 1 of 1

Community benchmarks snapshot

Records by GPU

AMD Strix Halo 128GB348NVIDIA RTX 3090271NVIDIA RTX 5090197NVIDIA DGX Spark99NVIDIA RTX 3060 12GB96NVIDIA RTX 5060 Ti 16GB88NVIDIA RTX Pro 6000 Blackwell81AMD Radeon AI PRO R9700 32GB81NVIDIA RTX 5070 Ti77NVIDIA RTX 409068

Records by model

3266 total
Qwen3.81567
Qwen3.6334
Gemma 4185
DeepSeek V4 Flash150
Qwen3.5145
DeepSeek V4.1 Flash103
other782

Records by engine

2308 total
llama.cpp1134
Strata234
vLLM227
MLX107
Ollama99
other507

Use cases

coding 937agentic 640long-context 442tool-use 281vision 174summarization 81math 77creative-writing 58multilingual 35rp 22text-generation 9reasoning 3
coding937agentic640long-context442tool-use281vision174summarization81math77creative-writing58

Median generation t/s by GPU

On Qwen3.8 27B at 4-bit, plain single-GPU runs. Full ranking

RTX 509080RTX Pro 600067RTX 5070 Ti55M5 Ultra 256GB50RTX 409040RX 7900 XTX40RTX 309038RX 7800 XT 16GB35RTX 5090 Laptop 24GB34RX 7900 XT32

Prompt processing, same model

M5 Ultra 256GB1800RTX 40901949RX 7900 XTX640RTX 3090561RX 7900 XT800M3 Ultra 512GB372V100 32GB941Radeon AI PRO R9700 32GB1287RTX 3080 20GB997Arc Pro B70512

Reports by model size

Qwen3.8 27B819Qwen3.8 125B · 6B active676Qwen3.6 35B · 3B active242DeepSeek V4 Flash 284B · 13B active133DeepSeek V4.1 Flash 552B · 16B active92Qwen3.6 27B81Gemma 4 26B · 4B active61GLM-5.3 320B · 18B active57

Quants

Q4_K_M395UD-Q4_K_XL189NVFP4174Q8_01194-bit106IQ4_XS86IQ3_XXS85IQ3_S84Q4_K_XL83Q480