Find which open-weight LLMs fit in your GPU or Mac and compare the speeds people report on setups like yours.
Pick your GPU or Mac, then read the speeds people reported on it, or estimate which models fit and how fast they run.
Free to use. No account needed. Memory estimates and community measurements are labelled separately.
Search for a model to see its reported speeds across GPUs and Macs.
These are individual setups, not a controlled benchmark. Compare GPU count, quantization, context and offloading before comparing speeds. How to read a report →
Exact recorded values. Context may be a configured limit; matching filters does not establish identical prompts, offloading or concurrency.
User reports Llama 3.1 70B at 14.9 t/s on a Mac Studio with an M3 Ultra and 512 GB of unified memory. Setup is MLX with 4-bit quantization, group_size 64, no speculative decoding or batching, at a 4k prompt (4138 actual tokens) with 128 output tokens. The figure is one observation, not a repeated-run mean, from a single timed generation.
User reports Llama 3.1 8B at 72.9 t/s generation and 566.4 t/s prompt processing on a single Apple M2 Max with 64 GB unified memory. Setup is MLX via a direct python -m mlx_lm benchmark, 4-bit quantized, batch size 1, with a requested 512-token prompt and 128-token generation cap. The figures are three-trial means after discarding Trial 1; peak memory was 5.116 in engine output units, and KV cache precision and prefill chunk size are unreported.
User reports Llama 3.1 8B at 101.1 t/s generation and 2758.3 t/s prompt processing on an M5 Max with 48GB unified memory. Setup is llama.cpp build 11146 with Metal, Q4_K_M weights, n_gpu_layers=99 and flash attention enabled. The figures are separate llama-bench microtests, tg128 and pp512, not one 512-input/128-output request; context length is unspecified and KV cache precision was not captured.
User reports Llama 3.1 8B at 115.4 t/s generation and 3225.7 t/s prompt processing on an M5 Max with 48GB unified RAM. Setup is MLX with 4-bit weights and unquantized KV cache, batch size 1, one model per process. The headline uses pp512 and tg128 from a five-trial mean, not a best-of sweep; the source also records pp128=1932.3, pp256=2487.89 and tg256=115.52 t/s.
M4 16GB · llama.cpp
User reports Llama 3.1 8B at 21 t/s generation and 229.2 t/s prompt processing on an M4 16GB Mac mini. Setup is llama.cpp 0.4.0 with Q4_K_M and Metal, all layers on GPU; context, KV precision and Flash Attention are unknown. The figures come from separate llama-bench tests, pp512 at 229.23 ±0.08 t/s and tg128 at 20.95 ±0.02 t/s over three runs, not a combined 512-input/128-output completion, with 10.29 GiB RAM already in use.
M4 16GB · Evrmind custom Metal binary / llama.cpp CLI
User reports Llama 3.1 8B at 8.2 t/s generation and 9.1 t/s prompt processing on an M4 16GB Mac mini. Setup is Evrmind's custom Metal binary with EVR-1 compression at roughly 3 bits per weight and full GPU offload. The figures come from separate pp512 and tg128 phase tests, not a paired 512-input/128-output run; the source table lists 9.11 and 8.16 t/s.
M3 Ultra 512GB · MLX · 1,024 ctx
User benchmarks Llama 3.1 405B at 2.99 t/s on a single M3 Ultra 512GB node. Setup is MLX with Q4 quantisation at 1K context, single node. This is part of a 200+ configuration campaign across 5 models and 9 topologies on a 5-node M3 Ultra cluster; the report also covers Qwen 2.5 32B, Mixtral 8x7B, DeepSeek V3.2 and Kimi K2.5, with TP and PP scaling results.
AMD RX 7800 XT 16GB · Ollama
User reports Llama 3.1 8B at 1.5 t/s on an AMD RX 7800 XT under Ollama, compared with 65 t/s in LM Studio on the same machine. Setup is Ollama 0.4.7 on Windows with the Q4_K_M quant, Adrenalin Edition 24.10.1 drivers. The user asks whether the gap is their setup or an Ollama limitation.
On Qwen3.8 27B at 4-bit, plain single-GPU runs. Full ranking