llamaperf

Local model performance on your hardware

Find which open-weight LLMs fit in your GPU or Mac and compare the speeds people report on setups like yours.

Join the community

Follow along:XLinkedIn

What runs on your hardware?

Pick your GPU or Mac, then read the speeds people reported on it, or estimate which models fit and how fast they run.

Free to use. No account needed. Memory estimates and community measurements are labelled separately.

Looking for a particular model?

Search for a model to see its reported speeds across GPUs and Macs.

Local model performance reports from the community

These are individual setups, not a controlled benchmark. Compare GPU count, quantization, context and offloading before comparing speeds. How to read a report →

Compare setup details

Exact recorded values. Context may be a configured limit; matching filters does not establish identical prompts, offloading or concurrency.

Tone: positiveNVIDIA hardware
generation:
140.0 tokens/s
quant:
NVFP4 (NVFP4)

User reports Edge0-145B-NVFP4, a 145B latent MoE with about 10B active built on Qwen3.8-Flash-Next, at 140 tok/s single-stream on one RTX 5090 32 GB. Setup is a vLLM plugin on the NGC vLLM 0.33 image with NVFP4 routed experts, Q8_0 dense weights, a DFlash2 drafter and hot/cold expert placement: 12.8-16.2 GiB of experts in VRAM, the rest in pinned host RAM, with the CPU computing 55% of cold experts. The 51 GB PLE n-gram table is read from NVMe every step. At c4, c8 and c32 the same config reaches 213, 214 and 214 tok/s. Strata with MTP gets 121, 117 and 113 tok/s at c1, c4 and c8; like for like at c1 the user measures 140.5 vs 122.0 tok/s. Top-1 matches Strata in 257 of 261 teacher-forced positions.

Nov 21, 2026
Tone: mixedAMD hardware
generation:
30.0 tokens/s
quant:
Q8_0 (GGUF)

User reports Inkling-2-Mini at about 30 tok/s on a single Radeon AI PRO R9700, which they call pretty slow. Setup is llama.cpp built from source with the Q8_0 GGUF from ggml-org; the model only runs on one card. The unsloth Q8_K_XL build did not work with the latest llama.cpp, and its Mainline_Support variant produced gibberish.

Nov 21, 2026

Inkling-2 Mini

NVIDIA RTX 5060 Ti 16GB · llama.cpp · 150,000 ctx

Tone: positiveNVIDIA hardware
generation:
18-23 tokens/s
prompt processing (prefill):
283.0 tokens/s
quant:
UD-Q4_K_M (GGUF)
codingagentic

User reports Inkling-2-Mini completing a Godot collision-shape task in 23 minutes on an RTX 5060 Ti, with prefill at 283 tokens/s and generation at 18-23 tokens/s. Setup is llama.cpp with UD-Q4_K_M GGUF, 150000 context, 14 MoE layers on CPU (--n-cpu-moe 14), and all GPU layers offloaded. The user notes Qwen3.8 Next Flash on an R9700 ran at 30-40 tokens/s and GLM 5.3 Flash Maya-S at 7-10 tokens/s, but neither finished the task in 1-2 hours. They also recall 50-60 tokens/s on Qwen3.6 35B with a Q8 KV cache, and wish Inkling-2-Mini generated faster.

Nov 21, 2026
Tone: positiveNVIDIA hardware
generation:
59.7 tokens/s
prompt processing (prefill):
1162.0 tokens/s
quant:
XQ (GGUF)

User reports Inkling 2 Mini at 59.7 t/s decode and 1,162 t/s prefill at 8K on a single RTX 4090. Setup is a custom engine written in Xolotl with a 13.9 GB GGUF quant of the abliterated model, which fits on the 24 GB card. The model is a 26B MoE with about 3B active parameters and a looped middle of 12 layers run 3 times per token. Held-out KL against the parent is 0.013 with 96.3% top-1 agreement; refusals were 4/100 on a harmful prompt set and 1/100 on harmless ones. The model's MTP drafter is an optional separate file and was not used for these figures. User expects 100+ t/s after fusing the many small per-token kernels.

Nov 21, 2026

Maya-S

2× NVIDIA RTX 3090

Tone: positiveNVIDIA hardware
generation:
35.0 tokens/s

User reports running Maya-S DeepSeek at 35 t/s on 2x RTX 3090 after tuning, on a WRX80 Sage mainboard with a Threadripper 3955X and 256GB DDR4 3200 ECC. User asks about Qengine, which they say requires 3 GPUs and so cannot run on their two-card setup. User is enthusiastic about the 90-100 t/s figure quoted for Qengine and asks the author to support 2-GPU configurations.

Nov 21, 2026
Tone: positiveApple hardware
generation:
36.4 tokens/s
prompt processing (prefill):
968.0 tokens/s
quant:
4-bit (MLX)

User reports Hy4-Flash at 36.4 t/s decode and 968 t/s prefill on a MacBook Pro M5 Max with 128 GB, using their own Kapitsa engine. Setup is Kapitsa with a 4-bit MLX quant (8-bit trunk, 4-bit routed experts, group size 64) and the fast profile; the 134 GB pack keeps every expert in RAM with an 88 GB peak, Engram tables memory-mapped on the CPU. Decode falls from 36.4 t/s at short context to 34.6 t/s at 32K; prefill is 657 t/s at 4K on the default profile and 968-971 t/s with the fast profile. MLX 3.1-bit is compared at 20.8 t/s decode and 228 t/s prefill at 4K. A 48 GB M5 Pro streaming experts from SSD decodes at about 3.5 t/s.

Nov 20, 2026
Apple hardware
generation:
178.0 tokens/s
quant:
4-bit (MLX)

User benchmarks eight MLX inference servers across 11 models on an M5 Max 36GB, M4 Pro 48GB and M2 Pro 32GB. On the M5 Max, Qwen3.6 35B-A3B reaches 178 t/s in mlx-serve, the fastest of the engines tested. Runs use each engine's default flags, one request at a time, a ~540-token prompt and 256 output tokens, median of 3 reps, with 4-bit MLX builds except the ternary Bonsai models and 8-bit LFM3. mlx-serve has the best average at 98.2% of the row best but failed to load 6 cells; Rapid-MLX loads everything and averages 98.0%, with MTP on by default where a checkpoint has an MTP head. Qwen3.6 35B-A3B spans 107 to 178 t/s on the M5 Max depending on engine, and Rapid-MLX with MTP off gives 171.6 t/s there and 98.9 t/s on the M4 Pro.

Nov 20, 2026
Tone: positiveApple hardware
generation:
36.4 tokens/s
quant:
4-bit (MLX)

User reports Hy4-Flash at 36.4 t/s decode on a MacBook Pro M5 Max (40-core GPU, 128 GB), with prefill of 657 tok/s at 4K context in the default profile and 968 tok/s with --profile fast. Setup is the Kapitsa engine with a 4-bit MLX quant (8-bit trunk, 4-bit group-64 routed experts, plus Engram tables), 134 GB on disk and 88 GB peak RAM; the Engram tables are memory-mapped and read on the CPU. Decode falls from 36.4 t/s at short context to 34.6 t/s at 32K, while MLX (mlx-lm 0.32.2, 3.1-bit) drops from 20.8 to 4.4 t/s over the same contexts. Warm turns in a cached conversation start in 0.15-0.21 s at 1K to 32K. A 48 GB M5 Pro streams experts from SSD and decodes at about 3.5 tok/s.

Nov 20, 2026
Apple hardware
generation:
41.5 tokens/s
quant:
4bit (MLX)

User benchmarks Qwen3.6-35B-A3B at 41.5 tok/s single-stream decode on an Apple M3 Max 36GB. Setup is mlx_lm.server with 4-bit MLX weights and continuous batching; prefill at 1k input is 919 tok/s. Decode peaks at 95 tok/s at 8 concurrent requests. Schema following scored 1.00, GSM8K 0.95 and IFEval 0.82 over 40 items.

Nov 20, 2026
Tone: positiveNVIDIA hardware
generation:
225.0 tokens/s
quant:
Q3_K_M (GGUF)

User reports a pruned one-subject Gemma 4 26B-A4B pack at 225 tok/s on an RTX 5080, against 223 tok/s for the full model. Setup is unmodified llama.cpp (b12549) with Q3_K_M GGUF; the pack keeps 64 of 128 experts per layer and is 7.4 GB versus 13.3 GB for the full model. The full model at Q2_K produced broken output. On a Ryzen 7 9700X with no GPU the pack runs 20.5 tok/s and the full model 20.0 tok/s. Against the full model on 300 blind-judged questions a pack wins or ties 53% at 64 experts per layer and 77% at 96, and is much weaker outside its subject.

Nov 20, 2026
NVIDIA hardware
generation:
272.5 tokens/s

User reports Qwen3.6-35B-A3B at 272.5 tok/s on a single RTX 4090 using their own VEIZIK runtime. Setup is VEIZIK with an approximately 4-bit-class quant at a 512-token context, single stream, with 21.4 GiB of GPU memory resident. llama.cpp with Q4_K_M on the same pod measured 184.5 tok/s; the user notes slightly higher KL divergence for VEIZIK.

Nov 20, 2026

Gemma 4 8B E4B

NVIDIA RTX 3060 12GB · LM Studio · 4,096 ctx

NVIDIA hardware
generation:
74.0 tokens/s
quant:
Q4_K_M (GGUF)

User reports Gemma 4 E4B at 74 tok/s on an RTX 3060 12GB, measured with LM Studio (llama.cpp CUDA) at Q4_K_M with full GPU offload. VRAM use was 4.51 GB at 4K context, 4.72 GB at 16K and 5.01 GB at 32K. The user notes E4B is 8B parameters on paper but about 2.8B of that is per-layer embeddings llama.cpp keeps in system RAM. Qwen3.5 9B ran at 54 tok/s using 6.66 GB at 4K, 7.07 GB at 16K and 7.63 GB at 32K. Qwen3.8 27B did not fit, with 9.5 GB on GPU and the rest in RAM, at 5.4 tok/s. The user also built a browser-based VRAM calculator around these measurements.

Nov 20, 2026
Tone: positiveNVIDIA hardware
generation:
60.1 tokens/s
prompt processing (prefill):
28.0 tokens/s
quant:
IQ3_S (GGUF)

User reports Qwen3.8-Flash-Next at 60.1 tok/s decode and 28 tok/s prefill on two modded RTX 2080 Ti 22GB cards, with a peak of 70.9 tok/s at 61K of 128K context. Setup is Strata with IQ3_S weights, custom tensor parallelism over an NVLink bridge on M.2 OCuLink PCIe Gen3 x4, on a Minisforum AR900i with a Core i9-13900HX and 64GB DDR5. The 60.1 tok/s figure comes from 41,288 tokens in 695 s at 41K context; the internal RTX 5070 was idle during these runs.

Nov 20, 2026
NVIDIA hardware
generation:
40.0 tokens/s
prompt processing (prefill):
1000.0 tokens/s
quant:
IQ3_S (GGUF)

User compares two IQ3_S quant builds of Qwen3.8 Flash-Next on an RTX 4070 12GB with a Ryzen 5600X and 64GB DDR4, running Strata. The IQ3_S build decodes at 40 t/s with 1000 t/s prefill, which the user says is 20% slower than the Gyro-M build. The 125B-A6B model cannot fit in 12GB of VRAM, so weights ran partly from system RAM.

Nov 20, 2026

SenseNova-Loop

NVIDIA RTX 4060 · 100,000 ctx

Tone: positiveNVIDIA hardware
generation:
22.0 tokens/s

User reports SenseNova-Loop at 22 t/s on an RTX 4060 with 8GB VRAM, using the 100k context flavor. No engine, quant or KV cache setting is stated. User says the model is very concise in reasoning and answers, roughly half the tokens of Qwen3.8, and is happy after a day of testing.

Nov 20, 2026

Qwen3.8 27B

AMD RX 7900 XTX · llama.cpp · 196,608 ctx

AMD hardware
generation:
140.0 tokens/s
prompt processing (prefill):
862.3 tokens/s
quant:
Q4_K_M (GGUF)
kv:
q8_0
flash attention:
on
codingagentic

User reports Qwen3.8 27B at 140.01 t/s generation and 862.29 t/s prompt eval on an RX 7900 XTX 24GB. Setup is an RDNA3-optimized llama.cpp fork (Vulkan) with Q4_K_M GGUF weights, q8_0 KV cache, 196608 context, and DFlash2 speculative decoding using a 1.14 GB draft sidecar. Whole model and draft on the GPU, single request. Three cold-prompt runs of ~2600 prompt tokens and 128 generated tokens: decode 140.01, 134.10 and 140.31 t/s; prefill 862.29, 863.38 and 852.28 t/s. Nothing averaged. Use case is a daily-driver local coding agent.

Nov 20, 2026
Tone: positiveNVIDIA hardware
generation:
193.2 tokens/s
prompt processing (prefill):
4589.0 tokens/s
quant:
IQ2_XS (GGUF)
kv:
int8
coding

User benchmarks Qwen3.8-Flash-Next at 193.2 t/s decode and 4,589 tok/s prefill on an RTX 5080 with Basalt 0.5.0, against 155.6 t/s and 3,785 tok/s on Strata 0.1.55. Setup is IQ2_XS weights with a drafter, a 65,536-token prompt and an int8 KV cache, median of 3 runs with no prompt cache reuse and a 1,024-token output cap. At 120,071 tokens Basalt reached 205.6 t/s decode and 4,661 tok/s prefill versus Strata's 175.9 t/s and 3,706 tok/s; Basalt finished 18-20% sooner overall. The user is switching their daily engine to Basalt.

Nov 20, 2026
Intel hardware
generation:
51.1 tokens/s
prompt processing (prefill):
1456.2 tokens/s
quant:
IQ3_XXS (GGUF)

User reports Qwen3.8-Flash-Next at 51.09 t/s generation and 1456.24 t/s prefill on a single Intel Arc Pro B70, with a 4096-token prompt. Setup is Strata with IQ3_XXS weights, a 262144-token context, MTP and drafting off, and a 20000 MiB expert cache with 32768 KV tokens resident. The launch uses all 32GB of the card's VRAM. A sweep from 4096 to 250000 prompt tokens gives generation between 43.61 and 51.09 t/s and prefill between 1053.93 and 2250.96 t/s. Peak prefill was 2529 t/s. The host is a Ryzen 7 5700X3D with 48GB RAM, with a B580 for display.

Nov 20, 2026
Tone: negativeApple hardware
generation:
5.5 tokens/s
quant:
oQ6e (MLX)

User reports Qwen3.8 at about 5.5 tokens/sec on a Mac Studio M5 Ultra with 256 GB unified memory when the GPU clock stays at 338 MHz during MLX inference. Setup is oMLX 0.7.3 with MLX 0.32.4, oQ6e weights and MTP speculation; GPU power sat at roughly 1.5-7.4 W in the degraded state. Restarting oMLX did not help; a full reboot restored the GPU to 1620 MHz and 60-70 W. The user says previous normal Qwen3.8 oQ6e-MTP runs on this machine were about 120-143 tokens/sec, and asks whether FileVault, SSH or a LaunchDaemon is involved.

Nov 20, 2026
Tone: positiveAMD hardware
generation:
94.3 tokens/s
quant:
HT4
codingagentic

User reports Qwen3.8 Flash-Next at 94.3 tok/s on one short coding request on a 128 GB Beelink Strix Halo. Setup is Halogen 0.20.7 with HT4 weights; a separate run on 0.20.1 prefilled a fresh 32K prompt at 1,933 tok/s. Combined throughput across four requests was 130 tok/s, and cached coding follow-up first-token latency fell from 620 ms to 131 ms. The user notes some figures beat published single-DGX Spark runs, with different prompts and quants.

Nov 20, 2026
Showing 1–20 of 2469
Page 1 of 124

Local model performance: measured examples

Local model performance covers how fast a model generates a reply and how much memory its setup needs. These community measurements show generation and prompt processing speed for specific setups. To estimate memory fit on your hardware, use the VRAM calculator.

Recent examples, one report per row and one example per GPU. Each report records one GPU or Mac. Different workloads and possible CPU offloading mean these speeds do not rank the hardware.
HardwareModel and setupGenerationPrompt processingEvidence
NVIDIA RTX 3060 12GBGemma 4 8B

E4B

Q4_K_M · LM Studio

4,096 reported context

74 tokens/s-Full setupOriginal source ↗
Intel Arc Pro B70Qwen3.8 125B

Flash-Next

IQ3_XXS · Strata

262,144 reported context

51.09 tokens/s1,456.24 tokens/sFull setupOriginal source ↗
AMD Radeon 890MKolibri-1 78B

78B-A3.5B

IQ2_M · llama.cpp

262,000 reported context

27.28 tokens/s430 tokens/sFull setupOriginal source ↗
AMD RX 7900 XTXQwen3.8 27B

27B

Q4_K_M · llama.cpp

32,768 reported context

40 tokens/s-Full setupOriginal source ↗
NVIDIA RTX 5080Gemma 4 12B

12B

Q4_0 · llama.cpp

8,192 reported context

78 tokens/s3,100 tokens/sFull setupOriginal source ↗
NVIDIA RTX 3080 TiQwen3.6 35B

35B-A3B

Q4_K_M · llama.cpp

131,072 reported context

50.47 tokens/s303.7 tokens/sFull setupOriginal source ↗

Reported context may be a configured limit rather than the prompt length used for timing. Check the full setup for KV cache, offloading, batching and speculative decoding; missing details are unknown. What affects local model performance? →

Community benchmarks snapshot

Records by GPU

AMD Strix Halo 128GB347NVIDIA RTX 3090263NVIDIA RTX 5090196NVIDIA DGX Spark97NVIDIA RTX 3060 12GB93NVIDIA RTX 5060 Ti 16GB86NVIDIA RTX Pro 6000 Blackwell81AMD Radeon AI PRO R9700 32GB77NVIDIA RTX 5070 Ti76M5 Max 48GB67

Records by model

3210 total
Qwen3.81553
Qwen3.6331
Gemma 4182
DeepSeek V4 Flash150
Qwen3.5143
DeepSeek V4.1 Flash102
other749

Records by engine

2275 total
llama.cpp1119
Strata231
vLLM221
MLX107
Ollama96
other501

Use cases

coding 923agentic 624long-context 428tool-use 269vision 165summarization 78math 75creative-writing 57multilingual 34rp 20text-generation 9reasoning 3
coding923agentic624long-context428tool-use269vision165summarization78math75creative-writing57

Median generation t/s by GPU

On Qwen3.8 27B at 4-bit, plain single-GPU runs. Full ranking

RTX 509080RTX Pro 600067RTX 5070 Ti55M5 Ultra 256GB50RTX 409040RX 7900 XTX40RTX 309037RX 7800 XT 16GB35RTX 5090 Laptop 24GB34RX 7900 XT32

Prompt processing, same model

M5 Ultra 256GB1800RTX 40901949RX 7900 XTX640RTX 3090561RX 7900 XT800M3 Ultra 512GB372V100 32GB941Radeon AI PRO R9700 32GB1287RTX 3080 20GB997Arc Pro B70512

Reports by model size

Qwen3.8 27B814Qwen3.8 125B · 6B active667Qwen3.6 35B · 3B active239DeepSeek V4 Flash 284B · 13B active133DeepSeek V4.1 Flash 552B · 16B active91Qwen3.6 27B81Gemma 4 26B · 4B active61GLM-5.3 320B · 18B active57

Quants

Q4_K_M390UD-Q4_K_XL185NVFP4170Q8_01194-bit106IQ4_XS85IQ3_S84Q4_K_XL83IQ3_XXS82Q480