llamaperf

Local model performance on your hardware

Find which open-weight LLMs fit in your GPU or Mac and compare the speeds people report on setups like yours.

Join the community

Follow along:XLinkedIn

What runs on your hardware?

Pick your GPU or Mac, then read the speeds people reported on it, or estimate which models fit and how fast they run.

Free to use. No account needed. Memory estimates and community measurements are labelled separately.

Looking for a particular model?

Search for a model to see its reported speeds across GPUs and Macs.

Local model performance reports from the community

These are individual setups, not a controlled benchmark. Compare GPU count, quantization, context and offloading before comparing speeds. How to read a report →

Compare setup details

Exact recorded values. Context may be a configured limit; matching filters does not establish identical prompts, offloading or concurrency.

Tone: positiveNVIDIA hardware
generation:
140.0 tokens/s
quant:
NVFP4 (NVFP4)

User reports Edge0-145B-NVFP4, a 145B latent MoE with about 10B active built on Qwen3.8-Flash-Next, at 140 tok/s single-stream on one RTX 5090 32 GB. Setup is a vLLM plugin on the NGC vLLM 0.33 image with NVFP4 routed experts, Q8_0 dense weights, a DFlash2 drafter and hot/cold expert placement: 12.8-16.2 GiB of experts in VRAM, the rest in pinned host RAM, with the CPU computing 55% of cold experts. The 51 GB PLE n-gram table is read from NVMe every step. At c4, c8 and c32 the same config reaches 213, 214 and 214 tok/s. Strata with MTP gets 121, 117 and 113 tok/s at c1, c4 and c8; like for like at c1 the user measures 140.5 vs 122.0 tok/s. Top-1 matches Strata in 257 of 261 teacher-forced positions.

Nov 21, 2026
Tone: mixedAMD hardware
generation:
30.0 tokens/s
quant:
Q8_0 (GGUF)

User reports Inkling-2-Mini at about 30 tok/s on a single Radeon AI PRO R9700, which they call pretty slow. Setup is llama.cpp built from source with the Q8_0 GGUF from ggml-org; the model only runs on one card. The unsloth Q8_K_XL build did not work with the latest llama.cpp, and its Mainline_Support variant produced gibberish.

Nov 21, 2026

Inkling-2 Mini

Strix Halo 8060S 128GB · Loom

Tone: positive
generation:
44.8 tokens/s
prompt processing (prefill):
1427.0 tokens/s
quant:
UD-Q4_K_XL (GGUF)

User reports Inkling-2-Mini at 44.8 t/s decode on a Strix Halo 8060S with 128 GB, against 23.8 t/s for llama.cpp ROCm and 20.8 t/s for llama.cpp Vulkan on the same GGUF weights. Setup is Loom with the unsloth UD-Q4_K_XL GGUF, single stream; prefill at ~1k is 1,427 t/s versus 356 t/s for llama.cpp ROCm. Decode is said to reach ~80% of the memory-bandwidth roofline (228 GB/s ÷ ~4 GB read per token ≈ 56 tok/s). At ~2k context decode is 40.8 t/s and prefill 1,496 t/s; prefill at ~9k is 1,550 t/s and at ~18k 1,465 t/s. Four streams aggregate 100.8 t/s. The package was produced by Gemini 4 Neon driving the loom-skills playbook in about 4.3 hours and ~490 tool calls, gated against llama.cpp with token-level goldens up to 37k tokens. Speculative decoding with dspark is still a work in progress.

Nov 21, 2026

Inkling-2 Mini

NVIDIA RTX 5060 Ti 16GB · llama.cpp · 150,000 ctx

Tone: positiveNVIDIA hardware
generation:
18-23 tokens/s
prompt processing (prefill):
283.0 tokens/s
quant:
UD-Q4_K_M (GGUF)
codingagentic

User reports Inkling-2-Mini completing a Godot collision-shape task in 23 minutes on an RTX 5060 Ti, with prefill at 283 tokens/s and generation at 18-23 tokens/s. Setup is llama.cpp with UD-Q4_K_M GGUF, 150000 context, 14 MoE layers on CPU (--n-cpu-moe 14), and all GPU layers offloaded. The user notes Qwen3.8 Next Flash on an R9700 ran at 30-40 tokens/s and GLM 5.3 Flash Maya-S at 7-10 tokens/s, but neither finished the task in 1-2 hours. They also recall 50-60 tokens/s on Qwen3.6 35B with a Q8 KV cache, and wish Inkling-2-Mini generated faster.

Nov 21, 2026
Tone: positiveNVIDIA hardware
generation:
59.7 tokens/s
prompt processing (prefill):
1162.0 tokens/s
quant:
XQ (GGUF)

User reports Inkling 2 Mini at 59.7 t/s decode and 1,162 t/s prefill at 8K on a single RTX 4090. Setup is a custom engine written in Xolotl with a 13.9 GB GGUF quant of the abliterated model, which fits on the 24 GB card. The model is a 26B MoE with about 3B active parameters and a looped middle of 12 layers run 3 times per token. Held-out KL against the parent is 0.013 with 96.3% top-1 agreement; refusals were 4/100 on a harmful prompt set and 1/100 on harmless ones. The model's MTP drafter is an optional separate file and was not used for these figures. User expects 100+ t/s after fusing the many small per-token kernels.

Nov 21, 2026
Tone: negativeNVIDIA hardware
generation:
3-4 tokens/s
quant:
IQ3_S (GGUF)

User reports Qwen3.8 27B at IQ3_S producing 3-4 tk/s on an RTX 4060 laptop with 16GB DDR5 system RAM. Setup is LM Studio on Pop!_OS, with the model partly offloaded given 8GB VRAM against a 27B model. User asks whether another approach such as polystrata for MoE models could help, and does not want to drop to Q1 just for speed.

Nov 21, 2026

Maya-S

2× NVIDIA RTX 3090

Tone: positiveNVIDIA hardware
generation:
35.0 tokens/s

User reports running Maya-S DeepSeek at 35 t/s on 2x RTX 3090 after tuning, on a WRX80 Sage mainboard with a Threadripper 3955X and 256GB DDR4 3200 ECC. User asks about Qengine, which they say requires 3 GPUs and so cannot run on their two-card setup. User is enthusiastic about the 90-100 t/s figure quoted for Qengine and asks the author to support 2-GPU configurations.

Nov 21, 2026

Inkling-2 Mini

NVIDIA RTX 4090 · vLLM · 35,000 ctx

Tone: positiveNVIDIA hardware
generation:
190-220 tokens/s
quant:
NVFP4 experts (W4A16) + FP8 W8A16 attention/head/embeddings (NVFP4)
kv:
FP8

User reports Inkling 2 Mini running on a single RTX 4090 at about 190-220 tokens/s with vLLM, up from roughly 117 out of the box. Setup uses NVFP4 experts (W4A16) plus FP8 W8A16 attention, head and embeddings, with FP8 KV cache, fitting in 24 GB at a 35k context. The run uses the model's built-in drafter and required small patches to vLLM, which does not yet support 4090s for this model. An initial quality check on sample prompts reported about a 0.4% loss in perplexity; the user plans further testing.

Nov 21, 2026
Tone: positiveApple hardware
generation:
36.4 tokens/s
prompt processing (prefill):
968.0 tokens/s
quant:
4-bit (MLX)

User reports Hy4-Flash at 36.4 t/s decode and 968 t/s prefill on a MacBook Pro M5 Max with 128 GB, using their own Kapitsa engine. Setup is Kapitsa with a 4-bit MLX quant (8-bit trunk, 4-bit routed experts, group size 64) and the fast profile; the 134 GB pack keeps every expert in RAM with an 88 GB peak, Engram tables memory-mapped on the CPU. Decode falls from 36.4 t/s at short context to 34.6 t/s at 32K; prefill is 657 t/s at 4K on the default profile and 968-971 t/s with the fast profile. MLX 3.1-bit is compared at 20.8 t/s decode and 228 t/s prefill at 4K. A 48 GB M5 Pro streaming experts from SSD decodes at about 3.5 t/s.

Nov 20, 2026
Apple hardware
generation:
178.0 tokens/s
quant:
4-bit (MLX)

User benchmarks eight MLX inference servers across 11 models on an M5 Max 36GB, M4 Pro 48GB and M2 Pro 32GB. On the M5 Max, Qwen3.6 35B-A3B reaches 178 t/s in mlx-serve, the fastest of the engines tested. Runs use each engine's default flags, one request at a time, a ~540-token prompt and 256 output tokens, median of 3 reps, with 4-bit MLX builds except the ternary Bonsai models and 8-bit LFM3. mlx-serve has the best average at 98.2% of the row best but failed to load 6 cells; Rapid-MLX loads everything and averages 98.0%, with MTP on by default where a checkpoint has an MTP head. Qwen3.6 35B-A3B spans 107 to 178 t/s on the M5 Max depending on engine, and Rapid-MLX with MTP off gives 171.6 t/s there and 98.9 t/s on the M4 Pro.

Nov 20, 2026
Tone: positiveApple hardware
generation:
36.4 tokens/s
quant:
4-bit (MLX)

User reports Hy4-Flash at 36.4 t/s decode on a MacBook Pro M5 Max (40-core GPU, 128 GB), with prefill of 657 tok/s at 4K context in the default profile and 968 tok/s with --profile fast. Setup is the Kapitsa engine with a 4-bit MLX quant (8-bit trunk, 4-bit group-64 routed experts, plus Engram tables), 134 GB on disk and 88 GB peak RAM; the Engram tables are memory-mapped and read on the CPU. Decode falls from 36.4 t/s at short context to 34.6 t/s at 32K, while MLX (mlx-lm 0.32.2, 3.1-bit) drops from 20.8 to 4.4 t/s over the same contexts. Warm turns in a cached conversation start in 0.15-0.21 s at 1K to 32K. A 48 GB M5 Pro streams experts from SSD and decodes at about 3.5 tok/s.

Nov 20, 2026
Apple hardware
generation:
41.5 tokens/s
quant:
4bit (MLX)

User benchmarks Qwen3.6-35B-A3B at 41.5 tok/s single-stream decode on an Apple M3 Max 36GB. Setup is mlx_lm.server with 4-bit MLX weights and continuous batching; prefill at 1k input is 919 tok/s. Decode peaks at 95 tok/s at 8 concurrent requests. Schema following scored 1.00, GSM8K 0.95 and IFEval 0.82 over 40 items.

Nov 20, 2026

Qwen2.5-Coder 1.5B FFCraft

RX 6400 · llama.cpp

Tone: positive
generation:
57.0 tokens/s
quant:
Q8_0 (GGUF)
coding

User reports FFCraft-1.5B, a fine-tune of Qwen2.5-Coder-1.5B, running at about 57 tok/s on an AMD RX 6400 via Vulkan. Setup is llama.cpp with a Q8_0 GGUF (1.6GB) and no system prompt; the model was trained with full bf16 LoRA on an L4. On 60 held-out FFmpeg tasks the fine-tune passed 95.0% with zero system prompt versus 28.3% for the base model; the user notes exotic multi-input filter_complex pipelines remain outside its training distribution.

Nov 20, 2026
Tone: positiveNVIDIA hardware
generation:
72-86 tokens/s
quant:
Q6_K_XL (GGUF)
codingagenticlong-contexttool-use

User compares Qwen3.8 Flash-Next Q6_K_XL on Strata against several DeepSeek V4.1 Flash quants on 2x RTX 3090s capped at 300W, in a 3975WX/256GB DDR4 box. Qwen decodes at 72-86 tok/s and prefills at about 3,950 tok/s at 120K context; the DeepSeek runs decode at 23-62 tok/s depending on quant. On a 242-question accuracy benchmark Qwen Q6 scored 93.0%, DeepSeek EXL3 3.5bpw 90.1%, Maya-XS 89.7% and Maya-S 88.4%. Qwen Q6 finished six agent tasks in 14 minutes against 32 for Maya-XS and 81 for EXL3, and answered 5/4 of the 100K/180K long-document questions where Maya-XS got 1/0 and EXL3 0/0. DeepSeek models often exhausted the reply budget on thinking tokens; a thinking budget below the reply limit fixed it. The user notes one run per agent task and only 16 long-document questions, and that the DeepSeek setups drew 700-800W to finish 2-6x slower.

Nov 20, 2026
Tone: positiveNVIDIA hardware
generation:
225.0 tokens/s
quant:
Q3_K_M (GGUF)

User reports a pruned one-subject Gemma 4 26B-A4B pack at 225 tok/s on an RTX 5080, against 223 tok/s for the full model. Setup is unmodified llama.cpp (b12549) with Q3_K_M GGUF; the pack keeps 64 of 128 experts per layer and is 7.4 GB versus 13.3 GB for the full model. The full model at Q2_K produced broken output. On a Ryzen 7 9700X with no GPU the pack runs 20.5 tok/s and the full model 20.0 tok/s. Against the full model on 300 blind-judged questions a pack wins or ties 53% at 64 experts per layer and 77% at 96, and is much weaker outside its subject.

Nov 20, 2026
NVIDIA hardware
generation:
272.5 tokens/s

User reports Qwen3.6-35B-A3B at 272.5 tok/s on a single RTX 4090 using their own VEIZIK runtime. Setup is VEIZIK with an approximately 4-bit-class quant at a 512-token context, single stream, with 21.4 GiB of GPU memory resident. llama.cpp with Q4_K_M on the same pod measured 184.5 tok/s; the user notes slightly higher KL divergence for VEIZIK.

Nov 20, 2026

GLM-5.3

M3 Ultra · Open WebUI

Tone: mixed
generation:
12-15 tokens/s
quant:
4-bit dynamic
long-context

User compares GLM-5.3 on an M3 Ultra against ChatGPT's browser harness on a research prompt about democracy and labor power. GLM-5.3 ran through Open WebUI at a 4-bit dynamic quant and took about 27 minutes to answer, against about 7 minutes for ChatGPT. User puts GLM-5.3 at roughly 12-15 t/s and Astra at a reported 40 t/s, and rates Astra's historical examples better while GLM found better high-level theoretical sources.

Nov 20, 2026

Gemma 4 8B E4B

NVIDIA RTX 3060 12GB · LM Studio · 4,096 ctx

NVIDIA hardware
generation:
74.0 tokens/s
quant:
Q4_K_M (GGUF)

User reports Gemma 4 E4B at 74 tok/s on an RTX 3060 12GB, measured with LM Studio (llama.cpp CUDA) at Q4_K_M with full GPU offload. VRAM use was 4.51 GB at 4K context, 4.72 GB at 16K and 5.01 GB at 32K. The user notes E4B is 8B parameters on paper but about 2.8B of that is per-layer embeddings llama.cpp keeps in system RAM. Qwen3.5 9B ran at 54 tok/s using 6.66 GB at 4K, 7.07 GB at 16K and 7.63 GB at 32K. Qwen3.8 27B did not fit, with 9.5 GB on GPU and the rest in RAM, at 5.4 tok/s. The user also built a browser-based VRAM calculator around these measurements.

Nov 20, 2026

Qwen3.8 27B

2× 2x Tesla M10

Tone: negative
generation:
1.3 tokens/s
quant:
Q4
summarizationtool-use

User reports Qwen3.8 27B at 1.3 t/s on 2x Tesla M10s, running a Q4 quant for a shift-report dashboard with multi-turn follow-ups. No engine, context length or KV cache is stated. The user finds the speed too slow for the CEO to use interactively and is asking whether 4x P6000s or 2x RTX 5060 Ti 16GB would be better.

Nov 20, 2026
Tone: positiveNVIDIA hardware
generation:
60.1 tokens/s
prompt processing (prefill):
28.0 tokens/s
quant:
IQ3_S (GGUF)

User reports Qwen3.8-Flash-Next at 60.1 tok/s decode and 28 tok/s prefill on two modded RTX 2080 Ti 22GB cards, with a peak of 70.9 tok/s at 61K of 128K context. Setup is Strata with IQ3_S weights, custom tensor parallelism over an NVLink bridge on M.2 OCuLink PCIe Gen3 x4, on a Minisforum AR900i with a Core i9-13900HX and 64GB DDR5. The 60.1 tok/s figure comes from 41,288 tokens in 695 s at 41K context; the internal RTX 5070 was idle during these runs.

Nov 20, 2026
Showing 1–20 of 3382
Page 1 of 170

Community benchmarks snapshot

Records by GPU

AMD Strix Halo 128GB347NVIDIA RTX 3090263NVIDIA RTX 5090196NVIDIA DGX Spark97NVIDIA RTX 3060 12GB93NVIDIA RTX 5060 Ti 16GB86NVIDIA RTX Pro 6000 Blackwell81AMD Radeon AI PRO R9700 32GB77NVIDIA RTX 5070 Ti76M5 Max 48GB67

Records by model

3210 total
Qwen3.81553
Qwen3.6331
Gemma 4182
DeepSeek V4 Flash150
Qwen3.5143
DeepSeek V4.1 Flash102
other749

Records by engine

2275 total
llama.cpp1119
Strata231
vLLM221
MLX107
Ollama96
other501

Use cases

coding 923agentic 624long-context 428tool-use 269vision 165summarization 78math 75creative-writing 57multilingual 34rp 20text-generation 9reasoning 3
coding923agentic624long-context428tool-use269vision165summarization78math75creative-writing57

Median generation t/s by GPU

On Qwen3.8 27B at 4-bit, plain single-GPU runs. Full ranking

RTX 509080RTX Pro 600067RTX 5070 Ti55M5 Ultra 256GB50RTX 409040RX 7900 XTX40RTX 309037RX 7800 XT 16GB35RTX 5090 Laptop 24GB34RX 7900 XT32

Prompt processing, same model

M5 Ultra 256GB1800RTX 40901949RX 7900 XTX640RTX 3090561RX 7900 XT800M3 Ultra 512GB372V100 32GB941Radeon AI PRO R9700 32GB1287RTX 3080 20GB997Arc Pro B70512

Reports by model size

Qwen3.8 27B814Qwen3.8 125B · 6B active667Qwen3.6 35B · 3B active239DeepSeek V4 Flash 284B · 13B active133DeepSeek V4.1 Flash 552B · 16B active91Qwen3.6 27B81Gemma 4 26B · 4B active61GLM-5.3 320B · 18B active57

Quants

Q4_K_M390UD-Q4_K_XL185NVFP4170Q8_01194-bit106IQ4_XS85IQ3_S84Q4_K_XL83IQ3_XXS82Q480