llamaperf

Local model performance on your hardware

Find which open-weight LLMs fit in your GPU or Mac and compare the speeds people report on setups like yours.

What runs on your hardware?

Pick your GPU or Mac, then read the speeds people reported on it, or estimate which models fit and how fast they run.

Free to use. No account needed. Memory estimates and community measurements are labelled separately.

Looking for a particular model?

Search for a model to see its reported speeds across GPUs and Macs.

Local model performance reports from the community

These are individual setups, not a controlled benchmark. Compare GPU count, quantization, context and offloading before comparing speeds. How to read a report →

GPU: NVIDIA RTX 5060 8GB
Compare setup details

Exact recorded values. Context may be a configured limit; matching filters does not establish identical prompts, offloading or concurrency.

Tone: positiveNVIDIA hardware
generation:
57.0 tokens/s
quant:
Q4_K_M (GGUF)
codingagentictool-use

User reports Qwen3-8B at about 57 tok/s on an RTX 5060, the only machine where the full verification was run. Setup is llama.cpp with Q4_K_M weights; the tool picked the model, quant, context size and KV cache for the hardware. The figure comes from smoke tests, not a quality benchmark: 9 of 9 checks passed and tool calling works. AMD, Apple and the other supported runtimes have had much less testing.

Nov 9, 2026
NVIDIA hardware
generation:
20.0 tokens/s
agenticcoding

User asks for advice on upgrading a local stack with a €1000 budget, currently a Ryzen 5 9600, RTX 5060 8GB and 32GB DDR5. They report running Ornith 1.5 35B A3B with offload at an average of 20 tk/s, alongside Ornith 1.5 9B, and embeddings, TTS and STT on system RAM. Options considered are a used RTX 3090 at 600-700, a second GPU such as an RTX 3060, or a 16GB Mac Mini M6 at 850; no benchmark of those parts is given.

Oct 19, 2026
Tone: mixedNVIDIA hardware
generation:
30.0 tokens/s
quant:
Q4_K_XL (GGUF)
kv:
Q8_0
codingagentic

User reports Qwen3.6 35B-A3B at close to 30 t/s on an RTX 5060 8GB with a Ryzen 5 3600 and 32GB RAM. Setup is LM Studio with Q4_K_XL weights and a Q8_0 KV cache, reaching 98k context by moving some layers out of VRAM; total VRAM use stays below 7.5GB and RAM around 28-30GB. The user is asking for a better agentic coding model for this hardware and doubts Qwen3.8 27B would fit, and has not yet tested the 2-bit Ternary Bonsai 2.

Oct 18, 2026

Qwen3.8 27B

NVIDIA RTX 5060 8GB · llama.cpp · 8,192 ctx

NVIDIA hardware
generation:
30.1 tokens/s
quant:
UD-IQ2_XXS (GGUF)
kv:
q4_0

User reports Qwen3.8-27B at 30.10 tok/s median decode on an RTX 5060 8GB, over five runs at an occupied 8192-token prompt. Setup is llama.cpp with UD-IQ2_XXS weights and q4_0 KV cache, 65/65 layers on CUDA, peak VRAM 7767 MiB. A short-context FULL_GPU decode of 31.39 tok/s and a ctx512 control of 18.11 tok/s are also given; quality gate and uncensored checkpoint are not done.

Oct 7, 2026
NVIDIA hardware
generation:
2.4 tokens/s
quant:
FP8 (safetensors)

User reports DeepSeek-V4.1-Flash (552B backbone + 196B Engram, 384 experts top-6, FP8 + FP4) running on a single RTX 5060 8GB at ~1.6 tokens/s from disk only and ~2.4 tokens/s with a 16 GB RAM cache. Setup streams experts and Engram rows from NVMe using DeepSeek's reference code with a modified storage layer; only the dense part lives on the GPU (cap --vram_gb 7.3), with 240 experts (4.2 GiB) read per token. Batch size 1, text only, context limit 8192, prefill max 700 tokens. Live chat via the OpenAI-compatible server with a 16 GB RAM cache reaches 2.43 tokens/s; peak VRAM 7.06 GiB. A 311-token prompt takes 17.9 s. The user notes the speed-ups changed nothing in output (136/136 tokens identical) and that DeepSeek's untouched reference could not be run side by side because it cannot load the model in 8 GB.

Oct 5, 2026
NVIDIA hardware
generation:
40.0 tokens/s
prompt processing (prefill):
500.0 tokens/s
quant:
Q4_K_XL (GGUF)
kv:
Q8

User asks whether speculative decoding (MTP) is still worth enabling for a 35B A3B MoE model offloaded across an 8GB RTX 5060 and 32GB of system RAM. Current llama-server setup uses a Q4_K_XL GGUF with a Q8 KV cache, flash attention on, 4096 batch and ubatch, 16 CPU cores, 40 MoE layers on CPU and 99 GPU layers, yielding about 40 t/s generation and 500 t/s prompt processing without MTP. User recalls earlier reports that MTP hurt prompt processing and wants to know if that is still the case and how others configure llama-server.

Sep 18, 2026
Showing 1–6 of 6
Page 1 of 1

Community benchmarks snapshot

Records by GPU

AMD Strix Halo 128GB348NVIDIA RTX 3090271NVIDIA RTX 5090197NVIDIA DGX Spark99NVIDIA RTX 3060 12GB96NVIDIA RTX 5060 Ti 16GB88NVIDIA RTX Pro 6000 Blackwell81AMD Radeon AI PRO R9700 32GB81NVIDIA RTX 5070 Ti77NVIDIA RTX 409068

Records by model

3266 total
Qwen3.81567
Qwen3.6334
Gemma 4185
DeepSeek V4 Flash150
Qwen3.5145
DeepSeek V4.1 Flash103
other782

Records by engine

2308 total
llama.cpp1134
Strata234
vLLM227
MLX107
Ollama99
other507

Use cases

coding 937agentic 640long-context 442tool-use 281vision 174summarization 81math 77creative-writing 58multilingual 35rp 22text-generation 9reasoning 3
coding937agentic640long-context442tool-use281vision174summarization81math77creative-writing58

Median generation t/s by GPU

On Qwen3.8 27B at 4-bit, plain single-GPU runs. Full ranking

RTX 509080RTX Pro 600067RTX 5070 Ti55M5 Ultra 256GB50RTX 409040RX 7900 XTX40RTX 309038RX 7800 XT 16GB35RTX 5090 Laptop 24GB34RX 7900 XT32

Prompt processing, same model

M5 Ultra 256GB1800RTX 40901949RX 7900 XTX640RTX 3090561RX 7900 XT800M3 Ultra 512GB372V100 32GB941Radeon AI PRO R9700 32GB1287RTX 3080 20GB997Arc Pro B70512

Reports by model size

Qwen3.8 27B819Qwen3.8 125B · 6B active676Qwen3.6 35B · 3B active242DeepSeek V4 Flash 284B · 13B active133DeepSeek V4.1 Flash 552B · 16B active92Qwen3.6 27B81Gemma 4 26B · 4B active61GLM-5.3 320B · 18B active57

Quants

Q4_K_M395UD-Q4_K_XL189NVFP4174Q8_01194-bit106IQ4_XS86IQ3_XXS85IQ3_S84Q4_K_XL83Q480