Find which open-weight LLMs fit in your GPU or Mac and compare the speeds people report on setups like yours.
Pick your GPU or Mac, then read the speeds people reported on it, or estimate which models fit and how fast they run.
Free to use. No account needed. Memory estimates and community measurements are labelled separately.
Search for a model to see its reported speeds across GPUs and Macs.
These are individual setups, not a controlled benchmark. Compare GPU count, quantization, context and offloading before comparing speeds. How to read a report →
Exact recorded values. Context may be a configured limit; matching filters does not establish identical prompts, offloading or concurrency.
User reports Gemma 3 4B at 132.52 t/s on an RTX 6000 Pro Blackwell, a 3.34x speedup over Ollama. Setup is vLLM (dev build) with BF16 weights and MTP speculative decoding using 3 draft tokens, single request, ~512-token prompt and 64 output tokens, averaged over 3 runs. The user also tested Qwen3-VL 8B (Abliterated) with 1 MTP layer, noting Gemma 3's 3 MTP layers explain its larger gain; going beyond the MTP layer count as draft tokens does not help.
NVIDIA RTX 3090 · llama.cpp
User reports 294.1 t/s generation. Source evidence (verbatim): Source: https://dev.to/sikamikanikobg/vllm-vs-llamacpp-vs-ollama-what-happens-when-your-model-doesnt-fit-in-24gb-vram-56eb # vLLM vs llama.cpp vs Ollama: What Happens When Your Model Doesn't Fit in 24GB VRAM Hardware: NVIDIA RTX 3090 24GB, 128GB system RAM. Concurrency 1 decode tok/s (llama.cpp / vLLM): Gemma 3 1B 294.1 / 235.5. Qwen3-Coder 30B-A3B 157.2 / 172.0 (vLLM AWQ). Gemma 4 26B-A4B 118.8 / 133.8. Ollama at concurrency 1: 125.6, 129.3, 84.5. Single-stream steady-state decode on the longest prompt tier with models that exceed VRAM (llama.cpp -ngl 12 for gpt-oss-120b): GPT-OSS-120B 7.65 tok/s llama.cpp, 7.6 Ollama. GLM-4.5-Air 4.58 llama.cpp, 4.59 Ollama. vLLM ran out of memory on both. Engine versions and context length not stated.
M1 Max 64GB · LM Studio
User reports 33 t/s generation. Source evidence (verbatim): Source: https://famstack.dev/guides/mlx-vs-gguf-part-2-isolating-variables/ # Mac Studio M1Max64GB native generation after Gemma3QAT MLXdtype conversion Original author article March20,2026. Measured Mac Studio AppleM1 Max 64GB unified RAM,24GPUcores; CPUcorecount unknown. Gemma3-12B-it QAT4-bit MLX in LMStudio: author states generation increases from 28 to 33 tok/s after converting nonquantized dtype frombf16 tofloat16. No averaging or calculatedspeed used. Tested sourcequant repository verified https://huggingface.co/mlx-community/gemma-3-12b-it-qat-4bit ; convertedartifact generated locally with mlx_lm.convert --dtype float16, not a separate verifiedHFartifact. Testedrevision/runtimeversion/prompt/outputcounts/power/KVquant/speculation/residentRAM unknown. Separate reportedprefilltimes at655tokens6.2s→3.7s;8K114.4s→68.9s; do not derive nativeprefillT/s from approximate8K. Other charts report effectiveoutputtokens/totalrequesttime includingprefill, not generation; keep separate. Source: https://famstack.dev/guides/mlx-vs-gguf-part-2-isolating-variables/ . Rawlogs linkGitHub and were not fetched under sourceaccessstop. CurrentHFcard contains unrelated-looking multimodalmetadata; no testedhistoricalrevision established, so preserve artifact caveat. Selected run: locally converted float16 nonquantized dtype artifact, native generation 33 tok/s. Baseline before conversion is a separate run at 28 tok/s. Prompt processing speed unknown; do not derive it from prefill times.
M6 32GB · MLX-LM 0.31.3 / MLX 0.32.2 / Metal
User reports Gemma 3 27B at 8.9 t/s generation and 11.2 t/s prompt processing on a Mac mini M6 with 32GB unified memory. Setup is MLX-LM with the mlx-community 4-bit quantized model, run through the mlx_lm.generate CLI with a 20-token prompt and 500 output tokens. Peak memory was 16.296 GB. The rates are native MLX-LM phase timings from a single displayed run, not a benchmark average.
AMD Strix Halo 128GB · llama.cpp / Vulkan KHR cooperative matrices / build unknown
User reports Gemma 3 27B at 3.9 t/s generation and 104 t/s prompt processing on an AMD Strix Halo 128GB machine. Setup is llama.cpp with Vulkan on the Radeon 8060S iGPU, BF16 weights, 6 threads and maximum GPU offload (-ngl 999). The figures come from separate synthetic llama-bench phase tests, PP512 and TG128, not an end-to-end rate; the source gives rounded rates without raw durations or repeat counts.
AMD Strix Halo 128GB · llama.cpp
User reports Gemma 3 27B IT QAT at 13 t/s generation and 382.5 t/s prompt processing on an AMD Strix Halo 128GB machine. Setup is llama.cpp with Q4_0 and f16 K and V cache, flash attention on, batch 2048, one GPU with full layer offload. The stored figures are the quiet baseline cell, PP 382.490359 ± 1.929762 and TG 12.962264 ± 0.046976 t/s; other campaign cells reach up to 437.558764 t/s PP and 13.436544 t/s TG.
M5 Max 48GB · llama.cpp
User reports Gemma 3 4B at 150.5 t/s generation and 5760.7 t/s prompt processing on an M5 Max 48GB. Setup is llama.cpp build 11146 with Metal, Q4_K_M weights, n_gpu_layers=99 and flash_attention=true. The figures are separate llama-bench microtests, tg128 and pp512, not one 512-input/128-output request or server concurrency throughput; context length is unspecified.
M5 Max 48GB · llama.cpp
User reports Gemma 3 27B at 29.9 t/s generation and 751 t/s prompt processing on an M5 Max with 48GB unified memory. Setup is llama.cpp build 11146 with Metal, Q4_K_M weights, n_gpu_layers=99 and flash attention enabled. The figures are separate llama-bench microtests, tg128 and pp512, not one 512-input/128-output request or server concurrency throughput; context length is unspecified.
M5 Max 48GB · llama.cpp
User reports Gemma 3 1B at 294.9 t/s generation and 16750.2 t/s prompt processing on an M5 Max 48GB. Setup is llama.cpp build 11146 with Metal, Q4_K_M weights, n_gpu_layers=99, flash_attention=true and threads=0. The figures are separate llama-bench microtests, tg128 and pp512, not one 512-input/128-output request or server concurrency throughput; context length is unspecified.
M5 Max 48GB · llama.cpp
User reports Gemma 3 12B at 61.5 t/s generation and 1721.5 t/s prompt processing on an M5 Max 48GB. Setup is llama.cpp build 11146 with Metal, Q4_K_M weights, n_gpu_layers=99 and flash attention enabled. The figures are separate llama-bench microtests, tg128 and pp512, not one 512-input/128-output request or server concurrency throughput; context length is unspecified.
M5 Max 48GB · MLX
User reports Gemma 3 4B 4-bit at 176.2 t/s generation and 5550.3 t/s prompt processing on an M5 Max 48GB. Setup is MLX with 4-bit weights and unquantized KV cache on one Mac Studio, batch size 1, no speculative decoding. The headline figures are pp512 and tg128, not a best-of sweep or an average across prompt lengths; the source also records pp128=2589.02, pp256=3725.55 and tg256=178.13 t/s.
M5 Max 48GB · MLX
User reports Gemma 3 27B at 31.8 t/s generation and 854.4 t/s prompt processing on an M5 Max with 48GB unified RAM. Setup is MLX with 4-bit weights, unquantized KV cache, batch size 1, and a 2048-token prefill step. The figures are arithmetic means of five trials from mlx-lm 0.31.3, with prompt processing including first-token work; peak MLX allocator memory was 16.69 GB.
M5 Max 48GB · MLX
User reports Gemma 3 1B at 374.4 t/s generation and 17261.9 t/s prompt processing on an M5 Max 48GB. Setup is MLX with 4-bit weights and unquantized KV cache, batch size 1, one model invocation per process. The headline figures are pp512 and tg128 from a five-trial mean, not a best-of sweep or average across prompt lengths; the benchmark is synthetic text-token throughput, not end-to-end request throughput.
M5 Max 48GB · MLX
User reports Gemma 3 12B 4-bit at 66.9 t/s generation and 1834.9 t/s prompt processing on an M5 Max 48GB. Setup is MLX with 4-bit weights and unquantized KV cache on one Mac Studio, batch size 1, greedy sampling, no speculative decoding. The figures are native engine phase rates: prompt speed includes first-token work, and the retained tg128 and pp512 come from the (512,128) run, not a best-of sweep or average.
On Qwen3.8 27B at 4-bit, plain single-GPU runs. Full ranking