llamaperf

Local model performance on your hardware

Find which open-weight LLMs fit in your GPU or Mac and compare the speeds people report on setups like yours.

What runs on your hardware?

Pick your GPU or Mac, then read the speeds people reported on it, or estimate which models fit and how fast they run.

Free to use. No account needed. Memory estimates and community measurements are labelled separately.

Looking for a particular model?

Search for a model to see its reported speeds across GPUs and Macs.

Local model performance reports from the community

These are individual setups, not a controlled benchmark. Compare GPU count, quantization, context and offloading before comparing speeds. How to read a report →

GPU: NVIDIA RTX 4060 Laptop 8GB
Compare setup details

Exact recorded values. Context may be a configured limit; matching filters does not establish identical prompts, offloading or concurrency.

NVIDIA hardware
generation:
28.0 tokens/s
prompt processing (prefill):
31.2 tokens/s
quant:
PTQ1_0 (GGUF)

User reports Bonsai 2 27B at 28.0 tok/s on an RTX 4060 Laptop 8GB, with prompt reading at 31.2 tokens/s. Setup is the Kindi 0.3.0 image with a PTQ1_0 GGUF at 24,576 context, leaving 1,367 MiB free. Decode falls from 28.0 tok/s at position 100 to 25.2 tok/s at 24,000; a verify run passed token-exact against llama.cpp at 29.3 tokens/s.

Oct 31, 2026
Tone: mixedNVIDIA hardware
generation:
1.0 tokens/s
kv:
q8_0
agentic

User asks how to run LFM3 1.7B at high concurrency locally, using llama.cpp with q8_0 KV cache and --parallel 16 --kv-unified on an RTX 4060 Laptop with 8 GB VRAM and 16 GB system RAM. User reports that beyond about 6 concurrent sequences throughput stops scaling, and that starting a new request drops every other instance to about 1 tok/s until that request finishes prefill. No single-stream throughput figure is given; the 1 tok/s is a per-stream rate under concurrent load.

Oct 28, 2026
Tone: positiveNVIDIA hardware
generation:
60.0 tokens/s

User reports Gemma 4 26B MoE at 60+ tok/s on an RTX 4060 Laptop, up from 40 tok/s with standard llama.cpp. Setup is llama.cpp with MTP speculative decoding plus a custom DeepSeek-inspired hot-expert caching scheme that keeps frequently used experts on the GPU. The user asks for further performance ideas without changing the model.

Oct 14, 2026
NVIDIA hardware
generation:
37.7 tokens/s
prompt processing (prefill):
1828.8 tokens/s
quant:
Q4_K_M (GGUF)

User reports Qwen3.5 9B at 37.7 t/s generation and 1828.8 t/s prompt processing on an NVIDIA RTX 4060 Laptop 8GB. Setup is llama.cpp with Q4_K_M and full GPU offload via -ngl 99, run through native llama-bench CUDA on a Lenovo LOQ 15IRX9 with 24GB system RAM. The figures come from separate synthetic tests, pp512 and tg128, not a single 512-token request followed by a 128-token completion; context allocation and KV precision are not given.

Oct 12, 2026

Cyber-Tiel 35B (3B active) Coder

NVIDIA RTX 4060 Laptop 8GB · llama.cpp · 131,072 ctx

NVIDIA hardware
generation:
23.0 tokens/s
prompt processing (prefill):
35.0 tokens/s
quant:
UD-IQ3_XXS (GGUF)
kv:
q4_0
coding

User reports Cyber-Tiel Coder 35B-A3B at 23 t/s decode and 35 t/s prefill on an RTX 4060 Laptop 8GB with 16 GB system RAM, at 131,072 context. Setup is llama.cpp with the UD-IQ3_XXS GGUF, q4_0 KV cache, flash attention, 30 MoE layers offloaded to CPU, and MTP speculative decoding with 2 draft tokens. Figures are at 4K context and depend on free RAM for mmap caching; a 13-prompt benchmark table is promised later.

Sep 23, 2026
Tone: positiveNVIDIA hardware
generation:
39.0 tokens/s
agentic

User reports Qwen3.6 35B at 39 t/s on an 8 GB RTX 4060 laptop using the FreeToken engine. User also reports DeepSeek-V4-Flash 284B at 22-25 t/s on an RTX 5090 and GLM-5.2 753B at 15 t/s on an RTX PRO 6000.

Sep 7, 2026
Showing 1–6 of 6
Page 1 of 1

Community benchmarks snapshot

Records by GPU

AMD Strix Halo 128GB348NVIDIA RTX 3090271NVIDIA RTX 5090197NVIDIA DGX Spark99NVIDIA RTX 3060 12GB96NVIDIA RTX 5060 Ti 16GB88NVIDIA RTX Pro 6000 Blackwell81AMD Radeon AI PRO R9700 32GB81NVIDIA RTX 5070 Ti77NVIDIA RTX 409068

Records by model

3266 total
Qwen3.81567
Qwen3.6334
Gemma 4185
DeepSeek V4 Flash150
Qwen3.5145
DeepSeek V4.1 Flash103
other782

Records by engine

2308 total
llama.cpp1134
Strata234
vLLM227
MLX107
Ollama99
other507

Use cases

coding 937agentic 640long-context 442tool-use 281vision 174summarization 81math 77creative-writing 58multilingual 35rp 22text-generation 9reasoning 3
coding937agentic640long-context442tool-use281vision174summarization81math77creative-writing58

Median generation t/s by GPU

On Qwen3.8 27B at 4-bit, plain single-GPU runs. Full ranking

RTX 509080RTX Pro 600067RTX 5070 Ti55M5 Ultra 256GB50RTX 409040RX 7900 XTX40RTX 309038RX 7800 XT 16GB35RTX 5090 Laptop 24GB34RX 7900 XT32

Prompt processing, same model

M5 Ultra 256GB1800RTX 40901949RX 7900 XTX640RTX 3090561RX 7900 XT800M3 Ultra 512GB372V100 32GB941Radeon AI PRO R9700 32GB1287RTX 3080 20GB997Arc Pro B70512

Reports by model size

Qwen3.8 27B819Qwen3.8 125B · 6B active676Qwen3.6 35B · 3B active242DeepSeek V4 Flash 284B · 13B active133DeepSeek V4.1 Flash 552B · 16B active92Qwen3.6 27B81Gemma 4 26B · 4B active61GLM-5.3 320B · 18B active57

Quants

Q4_K_M395UD-Q4_K_XL189NVFP4174Q8_01194-bit106IQ4_XS86IQ3_XXS85IQ3_S84Q4_K_XL83Q480