llamaperf

Local model performance on your hardware

Find which open-weight LLMs fit in your GPU or Mac and compare the speeds people report on setups like yours.

What runs on your hardware?

Pick your GPU or Mac, then read the speeds people reported on it, or estimate which models fit and how fast they run.

Free to use. No account needed. Memory estimates and community measurements are labelled separately.

Looking for a particular model?

Search for a model to see its reported speeds across GPUs and Macs.

Local model performance reports from the community

These are individual setups, not a controlled benchmark. Compare GPU count, quantization, context and offloading before comparing speeds. How to read a report →

GPU: M5 Ultra 96GB
Compare setup details (1 active)

Exact recorded values. Context may be a configured limit; matching filters does not establish identical prompts, offloading or concurrency.

Apple hardware
prompt processing (prefill):
1354.0 tokens/s
quant:
iQ-MLX 3.3 bpw (MLX)
codingagenticlong-context

User reports Qwen3.8-Flash-Next on an M5 Ultra 96GB (64-core GPU) prefilling a cold 3.7k-token prompt at 1,354 tok/s. Setup is mlx-serve 26.10.1 with iQ-MLX 3.3 bpw, 131,072 context, native MTP adaptive decoding, PLD and a hot prefix cache, one request at a time. Decode is given per workload only: 227.5 tok/s on copy, 172.8 on new code and 106.4 on novel prose, with speculation on. Cold prefill rose to 4,083 tok/s at 25.2k. An agent replay ran at 125.8 tok/s text-round decode.

Oct 8, 2026

Qwen3.8 27B Swift-1.5

M5 Ultra 96GB · mlx-serve · 25,000 ctx

Tone: positiveApple hardware
generation:
113.7 tokens/s
prompt processing (prefill):
3191.0 tokens/s
quant:
4.7bpw (MLX)

User reports Qwen3.8 Swift-1.5 at 113.7 tok/s decode and 3191 tok/s prefill on an M5 Ultra 96GB Mac Studio. Setup is mlx-serve 26.10.1 with a 4.7bpw MLX quantization, 107GB download, text-only, tested to 179,200 tokens of context. Decode falls to 81.1 tok/s after a 95k prompt, where prefill is 2928 tok/s. User compares against a llama.cpp IQ3_XXS build of the same model, which reaches 62.7 tok/s decode after 4k and 1427 tok/s prefill at 25k. Top-1 agreement with Swift BF16 is 91.0% across 680 held-out positions, against 84.1% for the llama.cpp build.

Oct 6, 2026
Tone: positiveApple hardware
generation:
42.7 tokens/s
prompt processing (prefill):
828.2 tokens/s
quant:
4-bit-8bit mix (MLX)
kv:
8-bit
codingagenticlong-context

User reports Qwen3.8 Flash-Next on a base M5 Ultra 96GB (64-core GPU) at a median 828.2 t/s prefill and 42.7 t/s decode per stream, with a single-stream max of 3,328 t/s prefill and 152 t/s decode. Setup is a custom mlx-serve build with continuous batching at 4-way concurrency, a 4-bit/8-bit mixed MLX quant with MTP, 4 x 128k context (512k total) and 8-bit KV cache using 90GB of unified memory. Aggregate throughput was ~3,200 t/s prefill and ~170.8 t/s decode at 4-way concurrency. The run covered 112M tokens (109M prompt, 3M generated) with an 89% cache hit rate across ~1,700 sub-agent calls.

Sep 23, 2026
Showing 1–3 of 3
Page 1 of 1

Community benchmarks snapshot

Records by GPU

AMD Strix Halo 128GB348NVIDIA RTX 3090271NVIDIA RTX 5090197NVIDIA DGX Spark99NVIDIA RTX 3060 12GB96NVIDIA RTX 5060 Ti 16GB88NVIDIA RTX Pro 6000 Blackwell81AMD Radeon AI PRO R9700 32GB81NVIDIA RTX 5070 Ti77NVIDIA RTX 409068

Records by model

3265 total
Qwen3.81567
Qwen3.6334
Gemma 4185
DeepSeek V4 Flash150
Qwen3.5145
DeepSeek V4.1 Flash103
other781

Records by engine

2307 total
llama.cpp1133
Strata234
vLLM227
MLX107
Ollama99
other507

Use cases

coding 937agentic 640long-context 442tool-use 280vision 173summarization 81math 77creative-writing 58multilingual 35rp 22text-generation 9reasoning 3
coding937agentic640long-context442tool-use280vision173summarization81math77creative-writing58

Median generation t/s by GPU

On Qwen3.8 27B at 4-bit, plain single-GPU runs. Full ranking

RTX 509080RTX Pro 600067RTX 5070 Ti55M5 Ultra 256GB50RTX 409040RX 7900 XTX40RTX 309038RX 7800 XT 16GB35RTX 5090 Laptop 24GB34RX 7900 XT32

Prompt processing, same model

M5 Ultra 256GB1800RTX 40901949RX 7900 XTX640RTX 3090561RX 7900 XT800M3 Ultra 512GB372V100 32GB941Radeon AI PRO R9700 32GB1287RTX 3080 20GB997Arc Pro B70512

Reports by model size

Qwen3.8 27B819Qwen3.8 125B · 6B active676Qwen3.6 35B · 3B active242DeepSeek V4 Flash 284B · 13B active133DeepSeek V4.1 Flash 552B · 16B active92Qwen3.6 27B81Gemma 4 26B · 4B active61GLM-5.3 320B · 18B active57

Quants

Q4_K_M395UD-Q4_K_XL189NVFP4174Q8_01194-bit106IQ4_XS86IQ3_XXS85IQ3_S84Q4_K_XL83Q480