llamaperf

Local model performance on your hardware

Find which open-weight LLMs fit in your GPU or Mac and compare the speeds people report on setups like yours.

What runs on your hardware?

Pick your GPU or Mac, then read the speeds people reported on it, or estimate which models fit and how fast they run.

Free to use. No account needed. Memory estimates and community measurements are labelled separately.

Looking for a particular model?

Search for a model to see its reported speeds across GPUs and Macs.

Local model performance reports from the community

These are individual setups, not a controlled benchmark. Compare GPU count, quantization, context and offloading before comparing speeds. How to read a report →

GPU: NVIDIA RTX 4070 Ti
Compare setup details

Exact recorded values. Context may be a configured limit; matching filters does not establish identical prompts, offloading or concurrency.

Inkling 2 Mini

NVIDIA RTX 4070 Ti · llama.cpp · 65,536 ctx

Tone: positiveNVIDIA hardware
generation:
51-64 tokens/s
quant:
mixB (GGUF)
kv:
q8_0
tool-usevision

User reports Inkling 2 Mini at 51-64 tok/s decode and ~1,500 tok/s prefill at 64K context on a single RTX 4070 Ti 12 GB. Setup is a forked llama.cpp (inkling-fast) with a custom 2-3 bit mixB GGUF (10.3 GiB) and q8_0 KV cache, fitting in 11.6 GB VRAM including the Windows desktop; layers 22-23 run on CPU via -ot. Decode is 61-64 tok/s on short prompts and 51-53 tok/s at 22K; llama-bench pp512/tg128 gives 63.5-63.8 tok/s decode and 2,183-2,191 tok/s prefill. Quality is 90.9% top-1 vs Q8_0, KLD 0.048, PPL ratio 1.034.

Nov 22, 2026

Qwen3.5 4B Fonzie0.2

NVIDIA RTX 4070 Ti · llama.cpp · 65,536 ctx

Tone: mixedNVIDIA hardware
generation:
118.0 tokens/s
quant:
Q4_K_M (GGUF)
kv:
q8_0
tool-useagenticmultilingualvision

User reports Fonzie0.2-4B, a LoRA fine-tune of Qwen3.5-4B, decoding about 118 tok/s single-user on an RTX 4070 Ti with llama-server, and about 157 tok/s with MTP 2. Setup is llama.cpp's llama-server with the Q4_K_M GGUF (2.8 GB plus a 0.7 GB vision projector) and a q8_0 KV cache; at 64k context the whole server takes about 5.4 GiB. The model was trained on one DGX Spark with Unsloth on 1,390 conversations in 97 minutes. The user reports accuracy gains in three agent apps but flat accuracy on everyday questions, more invented links (5.9% vs 0.3%), and that GRPO gains in their own harness did not transfer to real apps.

Oct 31, 2026
Tone: mixedNVIDIA hardware
generation:
19.2 tokens/s
quant:
UD-IQ3_XXS (GGUF)

User benchmarks MoMoE, an MIT-licensed fork of Strata, against llama.cpp on Ornith-1.5-122B-A10B UD-IQ3_XXS on an RTX 4070 Ti 12 GB, reporting 19.2 t/s versus 4.6 t/s. MoMoE runs mixture-of-experts models on one GPU plus system RAM, so weights are partly offloaded. The post also lists Qwen3.8-Flash-Next at 65-77 vs 23.1 t/s and Ornith-1.5-35B-A3B at 132-154 vs 77.8 t/s on the same card, both with MTP drafting. User notes llama.cpp ties or wins when a model fits on the GPU, citing LFM2.5-24B at 105 vs 76 t/s and Flint at 103 vs 90 t/s on the PC, and says the fork is early and tested only on their own machines.

Oct 24, 2026
NVIDIA hardware
generation:
69.7 tokens/s
prompt processing (prefill):
6757.3 tokens/s
quant:
Q8_0 (GGUF)

User reports Gemma 4 E4B at 69.66 t/s generation and 6757.3 t/s prompt processing on an NVIDIA RTX 4070 Ti. Setup is llama.cpp with Q8_0 and requested GPU layers 999, a direct full-offload run on a 12GB card. The figures come from a synthetic pp512/tg128 test at zero context depth, so they do not measure generation after a filled context.

Oct 14, 2026

Qwen3.8 27B

NVIDIA RTX 4070 Ti · llama.cpp · 4,096 ctx

NVIDIA hardware
generation:
43.6 tokens/s
prompt processing (prefill):
943.3 tokens/s
quant:
UD-IQ2_XXS (GGUF)
kv:
Q8

User benchmarks Qwen3.8-27B at 43.643 generation tok/s and 943.278 prompt tok/s on an RTX 4070 Ti 12GB. Setup is llama.cpp b10448 with UD-IQ2_XXS GGUF, Q8 KV cache, 4096-token context, all 66 layers on GPU, one slot, 256-token outputs. A second quant UD-Q2_K_XL decoded 38.030 tok/s and 995.095 prompt tok/s with 1,583 MiB more peak VRAM. An IQ2 context ladder gave 41.124 tok/s at 4K, 40.522 at 8K and 39.201 at 16K (12,831 prompt tokens, 11.119 s TTFT). An isolated IQ2 draft-mtp experiment raised throughput 47.284% on prose and 92.651% on Python code, but prose output diverged at token 16, so MTP stays off by default. In a 24-task quality pass, Q2 scored 10/24 and IQ2 9/24 (McNemar p = 1.0).

Sep 28, 2026
Showing 1–5 of 5
Page 1 of 1

Community benchmarks snapshot

Records by GPU

AMD Strix Halo 128GB348NVIDIA RTX 3090271NVIDIA RTX 5090197NVIDIA DGX Spark99NVIDIA RTX 3060 12GB96NVIDIA RTX 5060 Ti 16GB89NVIDIA RTX Pro 6000 Blackwell81AMD Radeon AI PRO R9700 32GB81NVIDIA RTX 5070 Ti77NVIDIA RTX 409068

Records by model

3270 total
Qwen3.81569
Qwen3.6334
Gemma 4185
DeepSeek V4 Flash150
Qwen3.5145
DeepSeek V4.1 Flash103
other784

Records by engine

2312 total
llama.cpp1136
Strata236
vLLM227
MLX107
Ollama99
other507

Use cases

coding 938agentic 640long-context 442tool-use 281vision 174summarization 81math 77creative-writing 58multilingual 35rp 22text-generation 9reasoning 3
coding938agentic640long-context442tool-use281vision174summarization81math77creative-writing58

Median generation t/s by GPU

On Qwen3.8 27B at 4-bit, plain single-GPU runs. Full ranking

RTX 509080RTX Pro 600067RTX 5070 Ti55M5 Ultra 256GB50RTX 409040RX 7900 XTX40RTX 309038RX 7800 XT 16GB35RTX 5090 Laptop 24GB34RX 7900 XT32

Prompt processing, same model

M5 Ultra 256GB1800RTX 40901949RX 7900 XTX640RTX 3090561RX 7900 XT800M3 Ultra 512GB372V100 32GB941Radeon AI PRO R9700 32GB1287RTX 3080 20GB997Arc Pro B70512

Reports by model size

Qwen3.8 27B819Qwen3.8 125B · 6B active678Qwen3.6 35B · 3B active242DeepSeek V4 Flash 284B · 13B active133DeepSeek V4.1 Flash 552B · 16B active92Qwen3.6 27B81Gemma 4 26B · 4B active61GLM-5.3 320B · 18B active57

Quants

Q4_K_M397UD-Q4_K_XL189NVFP4174Q8_01194-bit106IQ4_XS86IQ3_XXS85IQ3_S84Q4_K_XL83Q480