Find which open-weight LLMs fit in your GPU or Mac and compare the speeds people report on setups like yours.
Pick your GPU or Mac, then read the speeds people reported on it, or estimate which models fit and how fast they run.
Free to use. No account needed. Memory estimates and community measurements are labelled separately.
Search for a model to see its reported speeds across GPUs and Macs.
These are individual setups, not a controlled benchmark. Compare GPU count, quantization, context and offloading before comparing speeds. How to read a report →
Exact recorded values. Context may be a configured limit; matching filters does not establish identical prompts, offloading or concurrency.
Unknown GPU · llama.cpp
User reports Gemma 3 1B at 4.47 t/s on a 2015 Nexus 5X phone, running on its CPU with the model held in 2 GB of RAM. Setup is llama.cpp baked into a trimmed Linux 3.10 kernel and tiny rootfs booted straight into RAM via fastboot, with no Android. The user also mentions running Qwen 2.5 and Llama 3.2 on the same device.
Unknown GPU
User reports a paper replacing Gemma 3 270M's embedding table with a frozen 16 MB 512-bit code table, with a ternary BitNet b1.58 body giving a 46 MB model that generates 615 to 740 tokens per second on a laptop CPU in 62 MB of memory. The paper states this is five to six times the speed of the float model's 8-bit GGUF in llama.cpp at a seventh of the memory. The frozen table wins by 1.7 to 2.3 nats on rare words and accepts 10,000 new words after training at a 3% cost where the learned table collapses by 43%.
User reports Gemma 3 4B at 132.52 t/s on an RTX 6000 Pro Blackwell, a 3.34x speedup over Ollama. Setup is vLLM (dev build) with BF16 weights and MTP speculative decoding using 3 draft tokens, single request, ~512-token prompt and 64 output tokens, averaged over 3 runs. The user also tested Qwen3-VL 8B (Abliterated) with 1 MTP layer, noting Gemma 3's 3 MTP layers explain its larger gain; going beyond the MTP layer count as draft tokens does not help.
NVIDIA RTX 3090 · llama.cpp
User reports 294.1 t/s generation. Source evidence (verbatim): Source: https://dev.to/sikamikanikobg/vllm-vs-llamacpp-vs-ollama-what-happens-when-your-model-doesnt-fit-in-24gb-vram-56eb # vLLM vs llama.cpp vs Ollama: What Happens When Your Model Doesn't Fit in 24GB VRAM Hardware: NVIDIA RTX 3090 24GB, 128GB system RAM. Concurrency 1 decode tok/s (llama.cpp / vLLM): Gemma 3 1B 294.1 / 235.5. Qwen3-Coder 30B-A3B 157.2 / 172.0 (vLLM AWQ). Gemma 4 26B-A4B 118.8 / 133.8. Ollama at concurrency 1: 125.6, 129.3, 84.5. Single-stream steady-state decode on the longest prompt tier with models that exceed VRAM (llama.cpp -ngl 12 for gpt-oss-120b): GPT-OSS-120B 7.65 tok/s llama.cpp, 7.6 Ollama. GLM-4.5-Air 4.58 llama.cpp, 4.59 Ollama. vLLM ran out of memory on both. Engine versions and context length not stated.
M1 Max 64GB · LM Studio
User reports 33 t/s generation. Source evidence (verbatim): Source: https://famstack.dev/guides/mlx-vs-gguf-part-2-isolating-variables/ # Mac Studio M1Max64GB native generation after Gemma3QAT MLXdtype conversion Original author article March20,2026. Measured Mac Studio AppleM1 Max 64GB unified RAM,24GPUcores; CPUcorecount unknown. Gemma3-12B-it QAT4-bit MLX in LMStudio: author states generation increases from 28 to 33 tok/s after converting nonquantized dtype frombf16 tofloat16. No averaging or calculatedspeed used. Tested sourcequant repository verified https://huggingface.co/mlx-community/gemma-3-12b-it-qat-4bit ; convertedartifact generated locally with mlx_lm.convert --dtype float16, not a separate verifiedHFartifact. Testedrevision/runtimeversion/prompt/outputcounts/power/KVquant/speculation/residentRAM unknown. Separate reportedprefilltimes at655tokens6.2s→3.7s;8K114.4s→68.9s; do not derive nativeprefillT/s from approximate8K. Other charts report effectiveoutputtokens/totalrequesttime includingprefill, not generation; keep separate. Source: https://famstack.dev/guides/mlx-vs-gguf-part-2-isolating-variables/ . Rawlogs linkGitHub and were not fetched under sourceaccessstop. CurrentHFcard contains unrelated-looking multimodalmetadata; no testedhistoricalrevision established, so preserve artifact caveat. Selected run: locally converted float16 nonquantized dtype artifact, native generation 33 tok/s. Baseline before conversion is a separate run at 28 tok/s. Prompt processing speed unknown; do not derive it from prefill times.
M6 32GB · MLX-LM 0.31.3 / MLX 0.32.2 / Metal
User reports Gemma 3 27B at 8.9 t/s generation and 11.2 t/s prompt processing on a Mac mini M6 with 32GB unified memory. Setup is MLX-LM with the mlx-community 4-bit quantized model, run through the mlx_lm.generate CLI with a 20-token prompt and 500 output tokens. Peak memory was 16.296 GB. The rates are native MLX-LM phase timings from a single displayed run, not a benchmark average.
AMD Strix Halo 128GB · llama.cpp / Vulkan KHR cooperative matrices / build unknown
User reports Gemma 3 27B at 3.9 t/s generation and 104 t/s prompt processing on an AMD Strix Halo 128GB machine. Setup is llama.cpp with Vulkan on the Radeon 8060S iGPU, BF16 weights, 6 threads and maximum GPU offload (-ngl 999). The figures come from separate synthetic llama-bench phase tests, PP512 and TG128, not an end-to-end rate; the source gives rounded rates without raw durations or repeat counts.
AMD Strix Halo 128GB · llama.cpp
User reports Gemma 3 27B IT QAT at 13 t/s generation and 382.5 t/s prompt processing on an AMD Strix Halo 128GB machine. Setup is llama.cpp with Q4_0 and f16 K and V cache, flash attention on, batch 2048, one GPU with full layer offload. The stored figures are the quiet baseline cell, PP 382.490359 ± 1.929762 and TG 12.962264 ± 0.046976 t/s; other campaign cells reach up to 437.558764 t/s PP and 13.436544 t/s TG.
M5 Max 48GB · llama.cpp
User reports Gemma 3 4B at 150.5 t/s generation and 5760.7 t/s prompt processing on an M5 Max 48GB. Setup is llama.cpp build 11146 with Metal, Q4_K_M weights, n_gpu_layers=99 and flash_attention=true. The figures are separate llama-bench microtests, tg128 and pp512, not one 512-input/128-output request or server concurrency throughput; context length is unspecified.
M5 Max 48GB · llama.cpp
User reports Gemma 3 27B at 29.9 t/s generation and 751 t/s prompt processing on an M5 Max with 48GB unified memory. Setup is llama.cpp build 11146 with Metal, Q4_K_M weights, n_gpu_layers=99 and flash attention enabled. The figures are separate llama-bench microtests, tg128 and pp512, not one 512-input/128-output request or server concurrency throughput; context length is unspecified.
M5 Max 48GB · llama.cpp
User reports Gemma 3 1B at 294.9 t/s generation and 16750.2 t/s prompt processing on an M5 Max 48GB. Setup is llama.cpp build 11146 with Metal, Q4_K_M weights, n_gpu_layers=99, flash_attention=true and threads=0. The figures are separate llama-bench microtests, tg128 and pp512, not one 512-input/128-output request or server concurrency throughput; context length is unspecified.
M5 Max 48GB · llama.cpp
User reports Gemma 3 12B at 61.5 t/s generation and 1721.5 t/s prompt processing on an M5 Max 48GB. Setup is llama.cpp build 11146 with Metal, Q4_K_M weights, n_gpu_layers=99 and flash attention enabled. The figures are separate llama-bench microtests, tg128 and pp512, not one 512-input/128-output request or server concurrency throughput; context length is unspecified.
M5 Max 48GB · MLX
User reports Gemma 3 4B 4-bit at 176.2 t/s generation and 5550.3 t/s prompt processing on an M5 Max 48GB. Setup is MLX with 4-bit weights and unquantized KV cache on one Mac Studio, batch size 1, no speculative decoding. The headline figures are pp512 and tg128, not a best-of sweep or an average across prompt lengths; the source also records pp128=2589.02, pp256=3725.55 and tg256=178.13 t/s.
M5 Max 48GB · MLX
User reports Gemma 3 27B at 31.8 t/s generation and 854.4 t/s prompt processing on an M5 Max with 48GB unified RAM. Setup is MLX with 4-bit weights, unquantized KV cache, batch size 1, and a 2048-token prefill step. The figures are arithmetic means of five trials from mlx-lm 0.31.3, with prompt processing including first-token work; peak MLX allocator memory was 16.69 GB.
M5 Max 48GB · MLX
User reports Gemma 3 1B at 374.4 t/s generation and 17261.9 t/s prompt processing on an M5 Max 48GB. Setup is MLX with 4-bit weights and unquantized KV cache, batch size 1, one model invocation per process. The headline figures are pp512 and tg128 from a five-trial mean, not a best-of sweep or average across prompt lengths; the benchmark is synthetic text-token throughput, not end-to-end request throughput.
M5 Max 48GB · MLX
User reports Gemma 3 12B 4-bit at 66.9 t/s generation and 1834.9 t/s prompt processing on an M5 Max 48GB. Setup is MLX with 4-bit weights and unquantized KV cache on one Mac Studio, batch size 1, greedy sampling, no speculative decoding. The figures are native engine phase rates: prompt speed includes first-token work, and the retained tg128 and pp512 come from the (512,128) run, not a best-of sweep or average.
Unknown GPU
User reports Gemma 3 270M at 321 tok/s on a Ryzen 7 7700 CPU, up from 132 tok/s, by replacing the output head with a two-level prefix head that sorts the vocabulary by byte order. Setup is a custom prefix head with 2,048 groups of 128 tokens, AVX2 only, one token at a time. The same grouping gives 27-33% lower KL to the full head than Google's Gemma 4 token ordering across Gemma 3 270M/1B, Gemma 4 E2B, Qwen3-0.6B and gpt-oss-20b. Caveats: one text domain, no downstream evals, no speculative-decoding acceptance numbers.
On Qwen3.8 27B at 4-bit, plain single-GPU runs. Full ranking