Find which open-weight LLMs fit in your GPU or Mac and compare the speeds people report on setups like yours.
Pick your GPU or Mac, then read the speeds people reported on it, or estimate which models fit and how fast they run.
Free to use. No account needed. Memory estimates and community measurements are labelled separately.
Search for a model to see its reported speeds across GPUs and Macs.
These are individual setups, not a controlled benchmark. Compare GPU count, quantization, context and offloading before comparing speeds. How to read a report →
Exact recorded values. Context may be a configured limit; matching filters does not establish identical prompts, offloading or concurrency.
User reports Qwen2.5 32B at 37 tokens per second on an RTX 5090. No engine, quant, context length or KV cache settings are stated. The user is an amateur and has not collected full benchmarks yet, and may post more results later.
NVIDIA RTX 4060 · llama.cpp
User reports Qwen2.5-1.5B-MoE-A0.9B at 253 t/s text generation on an RTX 4060 8 GB, up 32% over the dense Qwen2.5-1.5B at 191 t/s. Setup is llama.cpp with Q4_K_M GGUF on both models; CPU generation was 77.5 t/s for the MoE versus 54.9 t/s for the dense model, a 41% gain. Perplexity is about 40% higher for the MoE (WikiText-2 14.63 vs 10.41, C4 22.88 vs 15.76) and zero-shot mean acc_norm drops from 0.565 to 0.462, with the largest losses on ARC. The user calls this a proof of concept, noting quality is roughly in Qwen2.5-0.5B range and that the GGUF is larger (1.1 GB vs 0.9 GB) because llama.cpp's MoE architecture needs an untied output head.
User reports Qwen2.5-72B-Instruct-4bit at 14.3 t/s on a Mac Studio with an M3 Ultra and 512 GB of unified memory. Setup is MLX 0.31.1 with mlx-lm 0.31.2, 4-bit group_size=64 quantization, no speculative decoding or batching, at a 4132-token prompt with 128 output tokens. The figure is one observation, not a repeated-run mean; the source's more precise value is 14.331110981167 t/s, and the same sweep drops to 6.056028636 t/s at 131108 prompt tokens.
M4 16GB · llama.cpp
User reports Qwen2.5 3B at 47.4 t/s generation and 556.9 t/s prompt processing on a base Apple M4 Mac mini with 16 GB unified memory. Setup is llama.cpp with Q4_K_M, all layers on the GPU via Metal, measured with llama-bench PP512 and TG128 passes. The figures are means of three repetitions, with 10.42 GiB baseline system RAM in use during the run, so it is not an idle-machine result.
M2 Max 64GB · MLX
User reports Qwen2.5 7B at 76.8 t/s generation and 602.8 t/s prompt processing on an Apple M2 Max with 64GB unified memory. Setup is MLX via python -m mlx_lm benchmark with the mlx-community/Qwen2.5-7B-Instruct-4bit artifact, batch size 1, 512-token prompt and 128-token generation cap. The 76.8 t/s and 602.8 t/s are three-trial means after discarding Trial 1; retained generation values were 76.773, 76.747 and 76.821 t/s.
NVIDIA RTX 5060 Ti 16GB · llama.cpp
User benchmarks llama.cpp, LM Studio, and Ollama on an RTX 5060 Ti 16GB and an Apple M3 Max 36GB, serving Qwen2.5-Coder-7B Q4 GGUF. On the RTX 5060 Ti, llama.cpp reaches 77.0 tok/s, LM Studio 76.8 tok/s (0.3% behind), and Ollama 69.1 tok/s (10.3% slower). On the M3 Max, llama.cpp reaches 53.5 tok/s, Ollama 46.2 tok/s (14% slower), and LM Studio 38.2 tok/s (29% slower). The metric is wall-clock tokens per second over a 12-prompt suite with 256 tokens generated per prompt and greedy decoding. The author attributes the gap to wrapper overhead rather than the shared llama.cpp engine.
NVIDIA RTX 5080 · Ollama · 16,384 ctx
User reports Qwen2.5 Coder at 93.4 t/s generation and 4842.2 t/s prompt processing on an NVIDIA RTX 5080. Setup is Ollama with Q4_K_M at 16384 tokens of configured context, on one 16GB card. The recorded prompt was 1368 tokens, not a 16384-token prefill, and the figures are medians of 3 fresh-prefill runs at temperature 0 with a 256-token target.
NVIDIA RTX 5080 · Ollama · 16,384 ctx
User reports Qwen2.5 Coder-7B at 171.4 t/s generation on one NVIDIA RTX 5080 16GB. Setup is Ollama with Q4_K_M at 16384 tokens of configured context, on a Ryzen 7 7800X3D under Windows 11. The figure is the median of 3 fresh-prefill runs with a 1368-token prompt, not a 16384-token prefill; prompt processing was 9825.5 t/s.
NVIDIA RTX 5080 · Ollama · 16,384 ctx
User reports Qwen2.5 7B at 176.6 t/s generation and 9984.4 t/s prompt processing on an NVIDIA RTX 5080. Setup is Ollama with Q4_K_M at a configured 16384-token context, on one 16GB card. The recorded prompt was 1674 tokens, not a full 16384-token prefill, and the figures are medians of 3 fresh-prefill runs.
NVIDIA RTX 5080 · Ollama · 16,384 ctx
User reports Qwen2.5 14B at 94.6 t/s generation and 4864.7 t/s prompt processing on one NVIDIA RTX 5080 16GB. Setup is Ollama with Q4_K_M at a configured 16384-token context; the recorded prompt was 1674 tokens, not a full 16384-token prefill. The figures are medians of 3 fresh-prefill runs, with 256 target output tokens and a unique nonce per repetition to avoid prompt-cache hits.
AMD MI50 32GB · vLLM · 8,192 ctx
User reports Qwen2.5-1.5B-Instruct at 32.14 tok/s single-stream on an AMD Instinct MI50 32GB with vLLM. Setup is vLLM with unquantized FP16 weights on one MI50, one request, no speculative decoding, enforce-eager, ROCm flash attention, max model length 8192, in an unprivileged Proxmox LXC container. Under continuous batching aggregate throughput rises to 63.8 tok/s at 2 streams, 238.9 at 8 and 422.8 at 32. llama.cpp with 4-bit GGUF weights on the same card gave 31.25 tok/s single-stream and stayed near 31 at every concurrency. Qwen2.5-0.5B reached 648.2 tok/s aggregate.
M1 16GB · llama.cpp
User benchmarks Qwen2.5-0.5B Q4 at 95.6 t/s decode with 18 ms TTFT on an Apple M1 16GB, averaged across defaults and normalized modes on the chat_turn workload. Setup is llama.cpp (llama-server) with Q4 GGUF weights, Metal backend, 1 warmup plus 4 measured reps per cell. LlamaStash leads raw llama-server (91.9 t/s / 20 ms), LM Studio (88.4 / 68) and Ollama 0.24.0 (79.6 / 102); in defaults mode LlamaStash reaches 99.0 t/s / 15 ms vs raw llama-server's 92.3 / 19, and normalized mode collapses the gap to under 1% (92.2 vs 91.5).
NVIDIA L4 · vLLM
User benchmarks vLLM against SGLang on one NVIDIA L4 with Qwen2.5-7B-Instruct in bf16, 128 output tokens per request with ignore_eos. Setup pins the model, GPU class, dtype, context length and memory fraction across both engines, driven through /v1/completions by a Go load generator that times TTFT at the client. Prefix caching is on except where noted. At one request each engine decodes about 17.5 t/s; at 32 concurrent short prompts vLLM reaches 459.3 t/s and SGLang 449.0 t/s. On long prompts at 32 concurrent requests vLLM gets 252.7 t/s and SGLang 310.2 t/s; turning prefix caching off drops vLLM to 147.1 t/s and raises p99 TTFT from 6.7 s to 18.6 s. SGLang reaches first token sooner under load (258 ms vs 639 ms at 32 concurrent short prompts) while vLLM is quicker at a single request (72 ms vs 120 ms). Cost per million output tokens falls from $12.60 at one request to $0.48 at 32 on the same card. Cold start was 122 s for SGLang and 149 s for vLLM. The author notes one run per cell and that throughput moved by more than the gap between engines on a re-measurement, so small differences are noise.
AMD RX 7900 XT · vLLM
User reports Qwen2.5-7B-Instruct-GPTQ-Int4 at 115 tok/s single-stream decode on a Radeon RX 7900 XT (gfx1100) under native Windows with AMD ROCm. Setup is vLLM v0.19.1 with GPTQ Int4 weights, fp16 KV cache, native exllama W4A16 GEMM and hipGraph decode capture (FULL_DECODE_ONLY), gpu_memory_utilization=0.9, verified spill-free. Aggregate throughput scales with concurrency: ~73 tok/s at batch 4, ~232 at batch 16, ~358 at batch 32. KVarN KV-cache quantization (K 4-bit / V 2-bit) gives 4.74x KV capacity (999,296 vs 210,784 tokens) at 74.7 tok/s, about 35% slower. A community-validated RX 7800 XT (gfx1101) reached 81.9 tok/s on the same model.
2× NVIDIA T4 16GB · vLLM
User benchmarks Qwen2.5-7B-Instruct-AWQ on two Kaggle T4 GPUs, comparing single GPU, data parallel and tensor parallel serving with vLLM 0.30.0. Tensor parallel (TP=2) reached 58 tok/s for one user against 32 tok/s for single GPU and data parallel, with 276 ms time to first token. At 32 concurrent requests TP reached 437 tok/s aggregate output throughput, against 382 tok/s for data parallel and 215 tok/s for single GPU. Benchmarks used 512-token prompts and 128-token replies, run inside Kaggle through a load-balancing proxy. The two T4s are connected over PCIe PHB with no NVLink, and the user notes the result may differ on a host with more than 4 vCPUs.
M2 Ultra 192GB · LM Studio
User compares GGUF and MLX inference speeds on a Mac Studio M2 Ultra with 192GB, using LM Studio 0.3.10 with MLX 0.6. For Qwen2.5 72B 8-bit, GGUF reaches 7.68 tokens/sec while MLX drops to 0.42 tokens/sec. Smaller models show the opposite: Qwen2.5 0.5B 4-bit runs at 79 tokens/sec in GGUF versus 317 tokens/sec in MLX, and Mistral Small 22B 8-bit is roughly equal at 25.01 versus 25.96 tokens/sec. User reports MLX is 4-5x faster for very small models but about 10x slower for large ones, with the inflection point around 22B parameters, and notes MLX does not use much GPU for larger models while LM Studio lacks a layer-offload option for MLX.
NVIDIA RTX 4090 · llama.cpp
User benchmarks Qwen2.5-Coder-7B-Instruct Q4_K_M on an RTX 4090, comparing llama.cpp at 158.90 tok/s decode against apr 0.64.0 at 103.26 tok/s. Setup is llama.cpp build 39173bcac and apr 0.64.0 CUDA with the same GGUF, profile medium, 15s warmup, 3 x 30s runs, 10s cooldown, concurrency 1, streaming. Decode is 0.650x of llama.cpp, with ITL p50 6.29 ms vs 9.68 ms and throughput 1.24 vs 0.77 req/s. The user notes apr's figure is 2.6x the ~40 tok/s 7B reference and that an earlier 11.8 tok/s report was an end-to-end rate including ~9.7s model load.
AMD RX 9060 XT 16GB · llama.cpp
User benchmarks Qwen 2.5-Coder 14B Instruct Q4_K_M at 33.51 t/s generation and 1210.29 t/s prompt processing on an AMD RX 9060 XT 16GB via Vulkan. Setup is llama.cpp build c4ae9a88f8 with -ngl 99 and -fa 1; the same model on ROCm reaches 30.79 t/s generation and 1211.32 t/s prompt processing. A Qwen 3.5 9B Q4_K_M run reaches 50.15 t/s generation and 1904.16 t/s prompt processing on Vulkan, and a 14B quant sweep gives about 32, 28 and 20 tok/s for Q4_K_M, Q5_K_M and Q8_0.
2× NVIDIA RTX Pro 4000 Blackwell · SGLang
User benchmarks Qwen2.5-32B-Instruct-AWQ at 71.37 tok/s median decode on 2x RTX PRO 4000 Blackwell. Setup is SGLang 0.5.19 with AWQ weights, NGRAM speculative decoding (K=5), tensor parallelism 2 over PCIe Gen4, and FP8 KV cache. NGRAM speculative decoding gives a 1.28x median speedup over the 55.60 tok/s baseline, with high inter-prompt variance (std dev 19.18 tok/s). Per-domain decode ranges are 65-107 tok/s for JSON, 60-226 tok/s for code, and 58-82 tok/s for prose. FP8 KV cache alone measured 54.77 tok/s, a 1.5% regression.
NVIDIA RTX 3060 12GB · Ollama
User reports Qwen2.5 7B Instruct at 49.0 tok/s on an RTX 3060 12GB, and 6.4 tok/s CPU-only on the same machine. Setup is Ollama 0.11.4 with Q4_K_M weights, single-stream, warm speed as the median of three runs with 200-token answers. About 1.2 GB of the card was already in use by the desktop and a Stable Diffusion web UI. The same model measured 48.9 tok/s on the GPU in a separate CPU-vs-GPU comparison, with a cold first answer of 5.6 s on GPU against 31.3 s CPU-only, a 7.6x speed-up. The user notes the CPU is a 2012 Xeon E5-2665 without AVX2, and that a modern CPU with AVX2 or AVX-512 will beat these CPU numbers.
On Qwen3.8 27B at 4-bit, plain single-GPU runs. Full ranking