llamaperf

Local model performance on your hardware

Find which open-weight LLMs fit in your GPU or Mac and compare the speeds people report on setups like yours.

What runs on your hardware?

Pick your GPU or Mac, then read the speeds people reported on it, or estimate which models fit and how fast they run.

Free to use. No account needed. Memory estimates and community measurements are labelled separately.

Looking for a particular model?

Search for a model to see its reported speeds across GPUs and Macs.

Local model performance reports from the community

These are individual setups, not a controlled benchmark. Compare GPU count, quantization, context and offloading before comparing speeds. How to read a report →

GPU: NVIDIA RTX 5060 Laptop 8GB
Compare setup details

Exact recorded values. Context may be a configured limit; matching filters does not establish identical prompts, offloading or concurrency.

Tone: positiveNVIDIA hardware
generation:
30-40 tokens/s
prompt processing (prefill):
800-970 tokens/s
quant:
TQ1_0 (GGUF)
codingagenticvision

User reports running Qwen3.8 Flash-Next (125B-A6B) with the Gyro S TQ1_0 GGUF on an RTX 5060 Laptop with 8 GB VRAM and 32 GB RAM, getting PP 800-970 t/s and TG 30-40 t/s at 262K context with the Chat configuration. Setup is Strata with experts mmapped to SSD, KV cache split between VRAM and RAM, and Vision run on the CPU; MTP is disabled. At 131K context the Chat configuration gives PP 930-1,040 t/s and TG 30-45 t/s, while the Agentic configuration gives PP 520-740 t/s and TG 30-40 t/s at 262K and PP 670-890 t/s and TG 30-45 t/s at 131K. The user compares this with Qwen 3.6 35B A3B in llama.cpp at PP 630-680 t/s and TG 27-35 t/s, and says the model replaced all their coding agents.

Oct 26, 2026
Tone: positiveNVIDIA hardware
generation:
43.0 tokens/s

User reports Qwen3.6-35B-A3B at a little over 43 tokens/s on an RTX 5060 Laptop 8GB in a Lenovo Legion 5 with a Ryzen AI 7 350 and 32GB DDR5-5600. Setup is llama.cpp with the model offloaded across the 8GB VRAM and system RAM; around 30K context looks promising. User is also building PolyStrata to try for more performance, with no numbers yet.

Oct 14, 2026
NVIDIA hardware
generation:
29.3 tokens/s
quant:
PTQ1_0 (GGUF)
math

User reports Ternary Bonsai 2 27B at 29.27 t/s on an RTX 5060 Laptop 8 GB under Windows, using a Prism llama.cpp build. Setup is llama.cpp with PTQ1_0 (5.53 GiB), temp 0, seed 42, a 2048 reasoning budget and a 3072 token cap on 50 MATH-500 questions. A −2 logit bias on "wait", "maybe" and "perhaps" (nine token ids) dropped accuracy from 44/50 to 43/50 and raised average tokens from 845.2 to 872.3; the biased run measured 29.28 t/s. The user calls it one deterministic data point, not proof.

Oct 3, 2026
Tone: positiveNVIDIA hardware
prompt processing (prefill):
300-400 tokens/s
codingagentic

User reports Ornith 1.5 35B-A3B at 30-40 t/s generation and 300-400 t/s prompt processing on an RTX 5060 Laptop 8GB with 32GB DDR5 RAM. Setup uses llama.cpp via OpenCode, with generation at 30 t/s for 100k-140k context and up to 40 t/s below 100k context. User also ran Qwen 3.8 27B on the same hardware and compares the two models for coding tasks, noting Ornith handles contained tasks well but struggles with complex multi-file changes.

Sep 26, 2026

Gemma 4 12B

NVIDIA RTX 5060 Laptop 8GB · llama.cpp · 4,096 ctx

NVIDIA hardware
generation:
40.0 tokens/s
prompt processing (prefill):
700-900 tokens/s
quant:
Q6_K (GGUF)
kv:
F16

User reports Gemma 4 12B at 40.0 t/s on an RTX 5060 Laptop GPU with 8 GB VRAM, using MTP speculative decoding with a Q6_K draft head and F16 KV cache at 4096 context. Setup is llama.cpp with mainline draft-mtp support (commit b9193 or later), an imatrix-guided per-tensor fit-to-VRAM quant at 6.18 GB, 48 layers, parallel 1. Prefill is 700 to 900 t/s. Without MTP the same setup reaches 27.4 t/s. With a Q8 KV cache it reaches 40.4 t/s. In real chat use the user sees 42 or more t/s on fresh context, dropping to 33 to 37 t/s with 16k of context filled.

Sep 24, 2026
Showing 1–5 of 5
Page 1 of 1

Community benchmarks snapshot

Records by GPU

AMD Strix Halo 128GB348NVIDIA RTX 3090271NVIDIA RTX 5090197NVIDIA DGX Spark99NVIDIA RTX 3060 12GB96NVIDIA RTX 5060 Ti 16GB89NVIDIA RTX Pro 6000 Blackwell81AMD Radeon AI PRO R9700 32GB81NVIDIA RTX 5070 Ti77NVIDIA RTX 409068

Records by model

3270 total
Qwen3.81569
Qwen3.6334
Gemma 4185
DeepSeek V4 Flash150
Qwen3.5145
DeepSeek V4.1 Flash103
other784

Records by engine

2312 total
llama.cpp1136
Strata236
vLLM227
MLX107
Ollama99
other507

Use cases

coding 938agentic 640long-context 442tool-use 281vision 174summarization 81math 77creative-writing 58multilingual 35rp 22text-generation 9reasoning 3
coding938agentic640long-context442tool-use281vision174summarization81math77creative-writing58

Median generation t/s by GPU

On Qwen3.8 27B at 4-bit, plain single-GPU runs. Full ranking

RTX 509080RTX Pro 600067RTX 5070 Ti55M5 Ultra 256GB50RTX 409040RX 7900 XTX40RTX 309038RX 7800 XT 16GB35RTX 5090 Laptop 24GB34RX 7900 XT32

Prompt processing, same model

M5 Ultra 256GB1800RTX 40901949RX 7900 XTX640RTX 3090561RX 7900 XT800M3 Ultra 512GB372V100 32GB941Radeon AI PRO R9700 32GB1287RTX 3080 20GB997Arc Pro B70512

Reports by model size

Qwen3.8 27B819Qwen3.8 125B · 6B active678Qwen3.6 35B · 3B active242DeepSeek V4 Flash 284B · 13B active133DeepSeek V4.1 Flash 552B · 16B active92Qwen3.6 27B81Gemma 4 26B · 4B active61GLM-5.3 320B · 18B active57

Quants

Q4_K_M397UD-Q4_K_XL189NVFP4174Q8_01194-bit106IQ4_XS86IQ3_XXS85IQ3_S84Q4_K_XL83Q480