llamaperf

Local model performance on your hardware

Find which open-weight LLMs fit in your GPU or Mac and compare the speeds people report on setups like yours.

What runs on your hardware?

Pick your GPU or Mac, then read the speeds people reported on it, or estimate which models fit and how fast they run.

Free to use. No account needed. Memory estimates and community measurements are labelled separately.

Looking for a particular model?

Search for a model to see its reported speeds across GPUs and Macs.

Local model performance reports from the community

These are individual setups, not a controlled benchmark. Compare GPU count, quantization, context and offloading before comparing speeds. How to read a report →

GPU: NVIDIA RTX 5060 Ti 16GB
Compare setup details (1 active)

Exact recorded values. Context may be a configured limit; matching filters does not establish identical prompts, offloading or concurrency.

Inkling 2 Mini

NVIDIA RTX 5060 Ti 16GB · llama.cpp · 128,000 ctx

Tone: positiveNVIDIA hardware
generation:
44.4 tokens/s
quant:
IQ4_NL (GGUF)
kv:
q4_0

User reports Inkling 2 Mini at 44.35 t/s on an RTX 5060 Ti 16GB with 32 GB system RAM. Setup is llama.cpp (build 19bc41dbb) with the IQ4_NL GGUF from ggml-org, q4_0 KV cache, 128k context, no vision, and ngram-mod speculation. The same setup with q8_0 KV cache reached 43.34 t/s; the user started at 12 t/s before tuning.

Nov 22, 2026

Inkling-2 Mini

NVIDIA RTX 5060 Ti 16GB · llama.cpp · 150,000 ctx

Tone: positiveNVIDIA hardware
generation:
18-23 tokens/s
prompt processing (prefill):
283.0 tokens/s
quant:
UD-Q4_K_M (GGUF)
codingagentic

User reports Inkling-2-Mini completing a Godot collision-shape task in 23 minutes on an RTX 5060 Ti, with prefill at 283 tokens/s and generation at 18-23 tokens/s. Setup is llama.cpp with UD-Q4_K_M GGUF, 150000 context, 14 MoE layers on CPU (--n-cpu-moe 14), and all GPU layers offloaded. The user notes Qwen3.8 Next Flash on an R9700 ran at 30-40 tokens/s and GLM 5.3 Flash Maya-S at 7-10 tokens/s, but neither finished the task in 1-2 hours. They also recall 50-60 tokens/s on Qwen3.6 35B with a Q8 KV cache, and wish Inkling-2-Mini generated faster.

Nov 21, 2026
NVIDIA hardware
generation:
40.6 tokens/s
quant:
IQ1_S (GGUF)

User reports Qwen3.8 Flash-Next at a mean decode speed of 40.6 t/s on an RTX 5060 Ti 16GB, up from 24 t/s before tuning. Setup is llama.cpp with Unsloth's IQ1_S quant at a 128k context window, on a Ryzen 7 7700X with 32 GB DDR5-6000. The 70 t/s peak came from a cached prompt.

Nov 19, 2026
Tone: mixedNVIDIA hardware
generation:
14.0 tokens/s
quant:
Q4 (GGUF)

User reports Qwen3.8 27B Q4 at about 14 t/s over llama.cpp RPC between an Ubuntu server with an RTX 5060 Ti 16GB and a Bazzite gaming PC with an RX 9060 XT 16GB, connected over home wifi. The Bazzite machine was built with the ROCm backend under distrobox; the Vulkan backend produced repeated answers and looping phrases once VRAM use passed 7GB. User asks whether the Vulkan behaviour is a bug or a setup mistake.

Nov 12, 2026
Tone: mixedNVIDIA hardware
generation:
15.0 tokens/s
quant:
Q4_K_XL (GGUF)

User reports Qwen3.8-Flash-Next Q4_K_XL (unsloth) at 15 t/s generation on dual RTX 5060 Ti 16GB cards. Setup is llama.cpp v0.8.0 with tensor parallel over PCIe 5.0 x8x8, no MTP, alongside 96GB of DDR5-6000 system RAM. The figure is offered as a hypothetical ("might run") rather than a measured benchmark, and the user says they stay on llama.cpp for quality despite the slower speed of newer engines.

Nov 4, 2026
NVIDIA hardware
generation:
40.0 tokens/s
quant:
IQ3_S (GGUF)
codingagentic

User reports about 40 tokens per second running a Swift-1.5 Qwen3.8 27B IQ3_S GGUF on an RTX 5060 Ti 16GB. Setup is llama.cpp with MTP speculative decoding at 81920 context size, on a Ryzen 7 9800X3D with 32GB DDR5-6000. User asks whether this is the best they can do for general use, coding and agentic tasks.

Oct 26, 2026
NVIDIA hardware
generation:
15.0 tokens/s
quant:
IQ3_S (GGUF)

User reports Qwen3.8-Flash-Next IQ3_S (83.6 GB) decoding at ~15 tok/s (~67 ms/token) in steady state on an RTX 5060 Ti 16 GB with 15.38 GiB RAM and a Gen3-class NVMe. Setup is a patched llama.cpp (Sabah-MoE-Accelerator) that streams the expert bank from disk into a 9.8 GB VRAM slab with unbuffered I/O; the 28.8 GB per-layer token embedding is left on CPU via -ot, peak VRAM is 15.36 of 16.31 GB, and host RAM stays flat at 4.78 GiB. Time to first token is 11.2 s for a 53-token prompt (12.68 GB cold fill at 1.11 GB/s), so a 256-token answer is 7.1 tok/s end to end and 10.4 tok/s over the decodes alone; warm-up runs 79-204 ms/token. The author notes speculative decoding and larger batches were 2.8-7.7x slower on this rig, and a second GTX 1080 was measured as no faster than disk.

Oct 18, 2026
Tone: positiveNVIDIA hardware
generation:
60.0 tokens/s
prompt processing (prefill):
1100.0 tokens/s
quant:
QAT
rpcreative-writingcoding

User reports Gemma 4 31B IT QAT at over 60 t/s generation and 1100 t/s prompt processing on dual RTX 5060 Ti 16GB cards. Setup is llama.cpp 0.7.0 with QAT weights, tensor split and MTP enabled across the two cards. Earlier runs on the same setup gave 40 t/s gen and 980 t/s pp, and 17 t/s gen with 970 t/s pp at 32K for Q6_K. Qwen3.8 27B runs at around 70 t/s with MTP and 100 t/s when generating code. The user uses the models for roleplay in SillyTavern.

Oct 17, 2026
NVIDIA hardware
generation:
300.0 tokens/s

User reports Bonsai 2 at 300 tok/s on an RTX 5060 Ti 16GB using a fork of llama.cpp. The user states the fork previously ran Bonsai 2 at 140 tok/s on an RTX 4070 Ti Super 16GB, and that after further optimization it reaches 140 tok/s on the 5060 Ti 16GB. The 300 tok/s figure is given as a correction for running without full context. The fork was also modified to run more than one session in parallel, as it previously ran only one.

Oct 16, 2026
Tone: positiveNVIDIA hardware
generation:
50.0 tokens/s
prompt processing (prefill):
900.0 tokens/s
quant:
IQ3_XXS (GGUF)
kv:
Q8_0
agentic

User reports Qwen3.8 27B at just over 50 t/s decode and just under 900 t/s prompt processing on a single 16GB RTX 5060 Ti. Setup is llama.cpp with an IQ3_XXS GGUF pruned to 9.3 GB via vocabulary pruning, Q8_0 KV cache, MTP speculative decoding, and everything in VRAM. The user gets just over 100k context with Q8 KV cache and a little over 170k with Q4 KV cache. They previously ran Gemma 4 26B A4B QAT at 90 t/s generation and just under 3000 t/s prompt processing, and Qwen 3.6 35B A3B, but found their intelligence insufficient for agentic use cases.

Oct 14, 2026
NVIDIA hardware
generation:
77.0 tokens/s
quant:
Q4 (GGUF)

User benchmarks llama.cpp, LM Studio, and Ollama on an RTX 5060 Ti 16GB and an Apple M3 Max 36GB, serving Qwen2.5-Coder-7B Q4 GGUF. On the RTX 5060 Ti, llama.cpp reaches 77.0 tok/s, LM Studio 76.8 tok/s (0.3% behind), and Ollama 69.1 tok/s (10.3% slower). On the M3 Max, llama.cpp reaches 53.5 tok/s, Ollama 46.2 tok/s (14% slower), and LM Studio 38.2 tok/s (29% slower). The metric is wall-clock tokens per second over a 12-prompt suite with 256 tokens generated per prompt and greedy decoding. The author attributes the gap to wrapper overhead rather than the shared llama.cpp engine.

Oct 12, 2026

Qwen3.8 27B

NVIDIA RTX 5060 Ti 16GB · llama.cpp · 14,644 ctx

Tone: positiveNVIDIA hardware
generation:
28.5 tokens/s
prompt processing (prefill):
878-892 tokens/s
quant:
IQ3_S (GGUF)
kv:
q8_0

User reports Qwen3.8 27B at 28.5 t/s decode on a 14,644-token prompt on an RTX 5060 Ti 16GB, with prefill at 878-892 t/s. Setup is an adaptive KV-streaming llama.cpp fork on one RTX 5060 Ti with GSQ-RCO IQ3_S weights, q8_0 KV cache, a 1536 MiB KV window and 131K context allocated, the whole model on the GPU, no speculative decoding; the host is a Ryzen 9600X with 32 GB single-channel DDR5. Decode falls as the prompt grows: 25.2 t/s at 35K, 20.6 at 55K, 18.7 at 69K and 12.6 at 94K with the 1536 MiB window, and about 9 t/s near 131K. A smaller 512 MiB window drops faster (8.7 t/s at 94K). Prefill falls from 878-892 to 630-635 t/s. MTP is not usable with the fork; it gained about 5 t/s on stock llama.cpp.

Oct 8, 2026
Tone: positiveNVIDIA hardware
generation:
95.2 tokens/s
prompt processing (prefill):
3476.0 tokens/s
quant:
MXFP4-MOE (GGUF)
kv:
q4_0

User reports Gemma 4 26B-A4B-it at 95.22 tok/s decode and 3,476 tok/s prompt processing on an RTX 5060 Ti 16 GB. Setup is a custom llama.cpp build (commit 0b484ab2b plus 12 commits) with MXFP4-MOE quantization (experts MXFP4, dense Q8_0, 4.66 BPW) and q4_0 KV cache, 1 slot, flash attention enabled, 65,536 token context. The MXFP4-MOE quant brings the model to 13.70 GiB, fitting the 16 GB card where Q4_K_M at 15.85 GiB does not. On an RTX 5090 the same quant gives 10,733 tok/s prompt and 196.6 tok/s decode versus 8,744 and 219.9 for Q4_K_M, with perplexity 3,864 vs 3,615.

Oct 6, 2026
NVIDIA hardware
generation:
30.5 tokens/s
prompt processing (prefill):
250.4 tokens/s
quant:
IQ1_M (GGUF)
kv:
q8_0
coding

User reports Qwen3.8-Flash-Next at 30.5 t/s generation and 250.4 t/s prompt processing on 2x RTX 5060 Ti with 32GB system RAM. Setup is llama.cpp with an IQ1_M GGUF (27.58 GB), 98304 context, q8_0 KV cache, tensor split 1,1, and 8 MoE layers offloaded to CPU. User asks whether their settings are correct and what config others run for this model on this hardware.

Oct 5, 2026

Qwen3.8 27B

NVIDIA RTX 5060 Ti 16GB · llama.cpp · 4,096 ctx

NVIDIA hardware
generation:
29.2 tokens/s
quant:
UD-IQ3_XXS (GGUF)
kv:
q4_0
coding

User reports Qwen 3.8 27B at 29.18 tok/s decode on an RTX 5060 Ti 16 GB at 4K context. Setup is llama.cpp build ad1de39e0 with UD-IQ3_XXS quant and q4_0 K/V cache, full GPU offload, flash attention, one parallel slot. Baseline decode without speculation is 29.18 tok/s at 4K, 23.62 at 32K and 19.77 at 64K; MTP n=2 gives 56.91/42.53/34.95 and MTP n=3 gives 63.89/45.48/39.06 tok/s at the same contexts. N-gram speculation added only 1.7% (29.66 tok/s). A 96K prompt measured 617.73 prompt tok/s and 36.74 decode tok/s.

Oct 3, 2026
Tone: mixedNVIDIA hardware
generation:
35.0 tokens/s
quant:
IQ3_XXS (GGUF)
kv:
q4_0

User reports about 35 t/s or more with Qwen3.8 27B GSQ-RCO-Uncensored on an RTX 5060 Ti 16GB. Setup is llama.cpp with IQ3_XXS quant, q4_0 KV cache, 131072 context, and MTP draft plus ngram speculative decoding. The user is on Fedora 44 with an AMD Ryzen 9600x and 16 GB system RAM, and had struggled to get reasonable speed before finding this model.

Oct 1, 2026
NVIDIA hardware
generation:
47-51 tokens/s
prompt processing (prefill):
1168.0 tokens/s
quant:
UD-IQ3_XXS (GGUF)
kv:
q4_0

User reports Qwen3.5-35B-A3B at 47-51 tok/s generation on an RTX 5060 Ti 16GB over OCuLink in a Proxmox VM. Setup is llama.cpp with UD-IQ3_XXS quant and q4_0 KV cache, fully on the GPU with 348 MiB headroom, at 160K context. Prompt eval of 75K tokens took 64.8 s (1,168 tok/s). The 9B variant reached 40-50 tok/s with UD-Q4_K_XL and about 47-51 tok/s with IQ3_XXS.

Sep 30, 2026
Tone: positiveNVIDIA hardware
generation:
80.0 tokens/s
prompt processing (prefill):
4737.5 tokens/s
quant:
NVFP4 (GGUF)
kv:
q8_0

User reports Nemotron 3.5 Lightning 30B-A3B at 79.98 t/s generation and 4737.46 t/s prompt processing on 2x RTX 5060 Ti 16GB. Setup is llama.cpp with NVFP4 GGUF weights and q8_0 KV cache, 1048576 token context, flash attention on, no MTP layers. The run processed a 54025-token prompt and generated 2156 tokens; the user notes it fits in 32 GB VRAM with no expert-layer offloading to RAM.

Sep 28, 2026
NVIDIA hardware
generation:
41.6 tokens/s
prompt processing (prefill):
1718.3 tokens/s
quant:
Q8_0 (GGUF)
kv:
q4_0
coding

User reports MiMo 2.6 Distill Qwen 9B at 41.63 t/s generation and 1718.29 t/s prompt processing on an RTX 5060 Ti 16GB. Setup is llama.cpp (Llama UI) with GGUF Q8_0 weights, q4_0 KV cache, 122880-token context, and Flash Attention enabled. The run produced 11979 output tokens over 4 min 47 s from a 1929-token prompt; the first generated version failed and a second review pass by the model produced the final working demo.

Sep 27, 2026

Qwen3.8 27B Swift-Genesis

2× NVIDIA RTX 5060 Ti 16GB · llama.cpp · 262,144 ctx

Tone: mixedNVIDIA hardware
generation:
76.0 tokens/s
mtp (multi-token prediction):
on
visionagenticlong-context

User reports Qwen3.8 27B Swift-Genesis at 76 t/s in their benchmark on 2x RTX 5060 Ti 16GB, with a range of 40-113 t/s and 45-65 t/s under normal agentic work. Setup is llama.cpp with GGUF weights at 262k context, MTP4 speculative decoding, and vision offloaded to system RAM. The user notes the model fits under 17GB with vision, allowing full 262k context on 32GB VRAM, and asks what the tradeoff of this model is compared to other Swift NVFP4 models.

Sep 27, 2026
Showing 1–20 of 34
Page 1 of 2

Community benchmarks snapshot

Records by GPU

AMD Strix Halo 128GB348NVIDIA RTX 3090271NVIDIA RTX 5090197NVIDIA DGX Spark99NVIDIA RTX 3060 12GB96NVIDIA RTX 5060 Ti 16GB89NVIDIA RTX Pro 6000 Blackwell81AMD Radeon AI PRO R9700 32GB81NVIDIA RTX 5070 Ti77NVIDIA RTX 409068

Records by model

3270 total
Qwen3.81569
Qwen3.6334
Gemma 4185
DeepSeek V4 Flash150
Qwen3.5145
DeepSeek V4.1 Flash103
other784

Records by engine

2312 total
llama.cpp1136
Strata236
vLLM227
MLX107
Ollama99
other507

Use cases

coding 938agentic 640long-context 442tool-use 281vision 174summarization 81math 77creative-writing 58multilingual 35rp 22text-generation 9reasoning 3
coding938agentic640long-context442tool-use281vision174summarization81math77creative-writing58

Median generation t/s by GPU

On Qwen3.8 27B at 4-bit, plain single-GPU runs. Full ranking

RTX 509080RTX Pro 600067RTX 5070 Ti55M5 Ultra 256GB50RTX 409040RX 7900 XTX40RTX 309038RX 7800 XT 16GB35RTX 5090 Laptop 24GB34RX 7900 XT32

Prompt processing, same model

M5 Ultra 256GB1800RTX 40901949RX 7900 XTX640RTX 3090561RX 7900 XT800M3 Ultra 512GB372V100 32GB941Radeon AI PRO R9700 32GB1287RTX 3080 20GB997Arc Pro B70512

Reports by model size

Qwen3.8 27B819Qwen3.8 125B · 6B active678Qwen3.6 35B · 3B active242DeepSeek V4 Flash 284B · 13B active133DeepSeek V4.1 Flash 552B · 16B active92Qwen3.6 27B81Gemma 4 26B · 4B active61GLM-5.3 320B · 18B active57

Quants

Q4_K_M397UD-Q4_K_XL189NVFP4174Q8_01194-bit106IQ4_XS86IQ3_XXS85IQ3_S84Q4_K_XL83Q480