llamaperf

Local model performance on your hardware

Find which open-weight LLMs fit in your GPU or Mac and compare the speeds people report on setups like yours.

What runs on your hardware?

Pick your GPU or Mac, then read the speeds people reported on it, or estimate which models fit and how fast they run.

Free to use. No account needed. Memory estimates and community measurements are labelled separately.

Looking for a particular model?

Search for a model to see its reported speeds across GPUs and Macs.

Local model performance reports from the community

These are individual setups, not a controlled benchmark. Compare GPU count, quantization, context and offloading before comparing speeds. How to read a report →

GPU: M5 Pro 24GB
Compare setup details

Exact recorded values. Context may be a configured limit; matching filters does not establish identical prompts, offloading or concurrency.

Tone: positiveApple hardware
generation:
61.8 tokens/s
quant:
4bit (MLX)
visionlong-contextagentic

User reports Qwen3.8-27B at 61.8 tok/s on an M5 Pro 24GB Mac, up from 57.3 tok/s with draft-tree speculative decoding off. Setup is the native Rust/Metal JustFit 2.6 engine with stock MLX 4-bit weights and Inco AI's DFlash2 draft, on a 24 GB Apple Silicon machine. The 7-prompt greedy suite also gave 40.9 tok/s on M6 (35.6 chain) and 50.9-51.3 tok/s on M4 Pro with a fixed 8-row tree. On NVIDIA SPEED-Bench at 16K prompts the M5 Pro median was 59.5 tok/s, M6 38.5 tok/s and M4 Pro 48.3 tok/s. Dual 128K+32K requests peaked at 21,563 MiB on M5 Pro.

Nov 22, 2026
Tone: positiveApple hardware
generation:
14.7 tokens/s
prompt processing (prefill):
374.1 tokens/s
quant:
IQ3_XXS (GGUF)
agentic

User benchmarks Qwen3.8-Flash-Next 125B at 14.69 tok/s decode on an M5 Pro 24GB, with 374.15 tok/s prefill on a 6,060-token prompt. Setup is the MoEShift backend with an IQ3_XXS quant (76GB on disk) and SSD offloading, peak process memory 8.5GB. Medians of 3 runs with prompt caching disabled. On the same setup an M4 Pro 24GB reached 9.84 tok/s short-prompt decode, 199.09 tok/s prefill and 9.57 tok/s long-prompt decode; long-prompt inference took 55.1s on the M5 Pro versus 83.9s on the M4 Pro.

Nov 9, 2026

Qwen3.8 27B

M5 Pro 24GB · Inco Splash · 120,000 ctx

Tone: positiveApple hardware
generation:
55.0 tokens/s
prompt processing (prefill):
448.0 tokens/s
quant:
UD-IQ3_XXS (GGUF)
agentictool-use

User reports Qwen3.8-27B at about 55 t/s generation and 448 t/s prefill on an M5 Pro MacBook Pro with 24GB unified memory. Setup is the Inco Splash engine (v1.4.1) with the UD-IQ3_XXS GGUF (10.9GB), capped at 18GB memory for around 120k context. User's own KLD measurements give 89.5% top-1 agreement with mean KLD 0.0542, and they say Safari and Discord still run comfortably alongside it.

Oct 27, 2026
Tone: positiveApple hardware
generation:
73.9 tokens/s
quant:
IQ3_XXS (GGUF)

User reports decode speed improvements on Qwen3-30B-A3B with IQ3_XXS quantization on an M5 MacBook Pro 24GB. The decode speed increased from 65.6 to 73.9 tok/s using a llama.cpp PR for Metal optimization. Setup uses llama.cpp with IQ3_XXS quantization. The user also tested on an M1 Pro 32GB. The user is looking for testers for Qwen3.8-Flash-Next and provides benchmark commands.

Oct 6, 2026
Tone: positiveApple hardware
generation:
125.8 tokens/s
prompt processing (prefill):
363.9 tokens/s
quant:
4bit (MLX)
codingagentic

User benchmarks DeepSeek-Coder-V2-Lite-Instruct at 125.8 tok/s generation and 363.9 tok/s prompt processing on an Apple M5 Pro with 24GB unified memory. Setup is MLX with a 4bit quant (8.2GB) on a single machine, running a fixed coding task capped at 1500 output tokens. The model produced correct working code; the user notes that 24GB has a hard ceiling around 20GB models and that background downloads measurably affect tok/s.

Oct 3, 2026
Tone: positiveApple hardware
generation:
4.8 tokens/s
quant:
2-bit dynamic

User reports DeepSeek-V4-Flash 284B-A13B at 4.8 tok/s on a 24 GB M5 Pro with about 5.3 GB of memory in use. Setup is Mference, an open-source engine. The user also runs Gemma 4 26B-A4B and Qwen 3.6 35B-A3B.

Sep 7, 2026
Showing 1–6 of 6
Page 1 of 1

Community benchmarks snapshot

Records by GPU

AMD Strix Halo 128GB348NVIDIA RTX 3090271NVIDIA RTX 5090197NVIDIA DGX Spark99NVIDIA RTX 3060 12GB96NVIDIA RTX 5060 Ti 16GB88NVIDIA RTX Pro 6000 Blackwell81AMD Radeon AI PRO R9700 32GB81NVIDIA RTX 5070 Ti77NVIDIA RTX 409068

Records by model

3266 total
Qwen3.81567
Qwen3.6334
Gemma 4185
DeepSeek V4 Flash150
Qwen3.5145
DeepSeek V4.1 Flash103
other782

Records by engine

2308 total
llama.cpp1134
Strata234
vLLM227
MLX107
Ollama99
other507

Use cases

coding 937agentic 640long-context 442tool-use 281vision 174summarization 81math 77creative-writing 58multilingual 35rp 22text-generation 9reasoning 3
coding937agentic640long-context442tool-use281vision174summarization81math77creative-writing58

Median generation t/s by GPU

On Qwen3.8 27B at 4-bit, plain single-GPU runs. Full ranking

RTX 509080RTX Pro 600067RTX 5070 Ti55M5 Ultra 256GB50RTX 409040RX 7900 XTX40RTX 309038RX 7800 XT 16GB35RTX 5090 Laptop 24GB34RX 7900 XT32

Prompt processing, same model

M5 Ultra 256GB1800RTX 40901949RX 7900 XTX640RTX 3090561RX 7900 XT800M3 Ultra 512GB372V100 32GB941Radeon AI PRO R9700 32GB1287RTX 3080 20GB997Arc Pro B70512

Reports by model size

Qwen3.8 27B819Qwen3.8 125B · 6B active676Qwen3.6 35B · 3B active242DeepSeek V4 Flash 284B · 13B active133DeepSeek V4.1 Flash 552B · 16B active92Qwen3.6 27B81Gemma 4 26B · 4B active61GLM-5.3 320B · 18B active57

Quants

Q4_K_M395UD-Q4_K_XL189NVFP4174Q8_01194-bit106IQ4_XS86IQ3_XXS85IQ3_S84Q4_K_XL83Q480