llamaperf

Local model performance on your hardware

Find which open-weight LLMs fit in your GPU or Mac and compare the speeds people report on setups like yours.

What runs on your hardware?

Pick your GPU or Mac, then read the speeds people reported on it, or estimate which models fit and how fast they run.

Free to use. No account needed. Memory estimates and community measurements are labelled separately.

Looking for a particular model?

Search for a model to see its reported speeds across GPUs and Macs.

Local model performance reports from the community

These are individual setups, not a controlled benchmark. Compare GPU count, quantization, context and offloading before comparing speeds. How to read a report →

GPU: NVIDIA RTX 3080 10GB
Compare setup details

Exact recorded values. Context may be a configured limit; matching filters does not establish identical prompts, offloading or concurrency.

Tone: mixedNVIDIA hardware
generation:
85.0 tokens/s
prompt processing (prefill):
2900.0 tokens/s
quant:
i1-IQ3_XXS (GGUF)
codingagentictool-use

User reports Qwen3.6 35B-A3B (Opus distill, abliterated, i1-IQ3_XXS) at about 85 t/s generation and about 2900 t/s prompt processing on an RTX 3080 10GB plus RTX 2070 Super 8GB, with 124k context. Setup is llama.cpp (CUDA) with opencode; the model fits fully in VRAM. Generation drops to about 68 t/s at 30k context. User finds the speed fine but the output quality poor and suspects the 3-bit quant. A Qwen3-Coder 30B-A3B abliterated at i1-IQ4_XS ran about 33 t/s at 64k, but at 128k the KV cache pushed experts to CPU and it fell to about 17 t/s. Qwen3.5 35B abliterated on Ollama was fast but the code quality was not great. User asks for a middle ground with at least 90k context for agentic coding in C++ and Python.

Nov 19, 2026
Tone: positiveNVIDIA hardware
generation:
145-185 tokens/s
prompt processing (prefill):
3957.0 tokens/s
quant:
UD-IQ4_XS (GGUF)
codingagentic

User reports Qwen3.8-Flash-Next on 2x RTX 3080 10GB with experts in system RAM, reaching 3,957 t/s prefill at a 200K cold prompt and decode of ~145-185 t/s with UD-IQ4_XS. Setup is the Speedy fork of Strata with UD-IQ4_XS, a 200K-token cold prompt, two pipeline windows and speculative decoding; the box is a Threadripper PRO 5975WX with 128 GB DDR4-3200. Decode varies with reply length and context, sitting at the top of the range for short replies and the bottom after a 200K prompt; a 200K cold prompt took 51 s. The 1x RTX 3080 table gives ~113-128 t/s decode and 1,792 t/s prefill at 200K for the same quant.

Nov 10, 2026

Qwen3.5 4B

NVIDIA RTX 3080 10GB · llama.cpp · 704,000 ctx

NVIDIA hardware
prompt processing (prefill):
662.0 tokens/s
quant:
Q4_K_M (GGUF)
long-context

User tests Qwen3.5-4B Q4_K_M on an RTX 3080 10GB with llama.cpp built from upstream PR 29827, flash attention on and ubatch 64, on a RULER-style multikey needle task. Prefill at a 709,596-token prompt (704K, YaRN x2.75) was about 662 tok/s, 18.4 minutes per run, peaking at 9,784 MiB VRAM. Stating the question before the document as well as after fixed all five configs that failed with question-last, at 256K and at 512K/704K with YaRN. One run per config, so the user calls it a strong signal on a small sample rather than a rate. In 10 paired trials at 256K with question last, q8_0 KV cache passed 7/10 at ~7,730 MiB peak and q4_0 passed 6/10 at ~5,680 MiB; 9 of 10 pairs gave identical answers and McNemar exact p = 1.0. The user notes 114-170 MiB showed as shared memory, f16 cache does not fit at 256K on this card, and the model rambles after answering.

Oct 29, 2026

Bonsai 2 27B

NVIDIA RTX 3080 10GB · llama.cpp · 32,768 ctx

NVIDIA hardware
generation:
52.2 tokens/s
prompt processing (prefill):
1210.6 tokens/s
quant:
PTQ1_0 (GGUF)
kv:
q8_0
coding

User reports Bonsai 2 27B PTQ1_0 at 52.18 t/s generation and 1210.6 t/s prefill on an RTX 3080 10GB. Setup is llama.cpp (fork build prism-b10735-842b188, CUDA 12.8) with PTQ1_0 quant and q8_0 KV cache at 32K context, single card, batch 1, depth 0, -ngl 99 -fa on. User also benchmarks PQ2_0 at 61.38 t/s and Qwen3.8-27B-UD-IQ2_XXS at 44.46 t/s on the same card, and reports a context ladder up to 160K q4_0 at 9627 MiB peak VRAM. A LiveCodeBench v6 comparison gives PTQ1_0 32/50 pass@1 versus 16/50 for IQ2_XXS and 28/50 for PQ2_0.

Sep 28, 2026
Showing 1–4 of 4
Page 1 of 1

Community benchmarks snapshot

Records by GPU

AMD Strix Halo 128GB348NVIDIA RTX 3090271NVIDIA RTX 5090197NVIDIA DGX Spark99NVIDIA RTX 3060 12GB96NVIDIA RTX 5060 Ti 16GB88NVIDIA RTX Pro 6000 Blackwell81AMD Radeon AI PRO R9700 32GB81NVIDIA RTX 5070 Ti77NVIDIA RTX 409068

Records by model

3266 total
Qwen3.81567
Qwen3.6334
Gemma 4185
DeepSeek V4 Flash150
Qwen3.5145
DeepSeek V4.1 Flash103
other782

Records by engine

2308 total
llama.cpp1134
Strata234
vLLM227
MLX107
Ollama99
other507

Use cases

coding 937agentic 640long-context 442tool-use 281vision 174summarization 81math 77creative-writing 58multilingual 35rp 22text-generation 9reasoning 3
coding937agentic640long-context442tool-use281vision174summarization81math77creative-writing58

Median generation t/s by GPU

On Qwen3.8 27B at 4-bit, plain single-GPU runs. Full ranking

RTX 509080RTX Pro 600067RTX 5070 Ti55M5 Ultra 256GB50RTX 409040RX 7900 XTX40RTX 309038RX 7800 XT 16GB35RTX 5090 Laptop 24GB34RX 7900 XT32

Prompt processing, same model

M5 Ultra 256GB1800RTX 40901949RX 7900 XTX640RTX 3090561RX 7900 XT800M3 Ultra 512GB372V100 32GB941Radeon AI PRO R9700 32GB1287RTX 3080 20GB997Arc Pro B70512

Reports by model size

Qwen3.8 27B819Qwen3.8 125B · 6B active676Qwen3.6 35B · 3B active242DeepSeek V4 Flash 284B · 13B active133DeepSeek V4.1 Flash 552B · 16B active92Qwen3.6 27B81Gemma 4 26B · 4B active61GLM-5.3 320B · 18B active57

Quants

Q4_K_M395UD-Q4_K_XL189NVFP4174Q8_01194-bit106IQ4_XS86IQ3_XXS85IQ3_S84Q4_K_XL83Q480