Find which open-weight LLMs fit in your GPU or Mac and compare the speeds people report on setups like yours.
Pick your GPU or Mac, then read the speeds people reported on it, or estimate which models fit and how fast they run.
Free to use. No account needed. Memory estimates and community measurements are labelled separately.
Search for a model to see its reported speeds across GPUs and Macs.
These are individual setups, not a controlled benchmark. Compare GPU count, quantization, context and offloading before comparing speeds. How to read a report →
Exact recorded values. Context may be a configured limit; matching filters does not establish identical prompts, offloading or concurrency.
AMD RX 7900 XT · LM Studio
User reports running Gemma 4 26B (MoE) locally at around 75-90 tok/s on an AMD Radeon RX 7900 XT with 20 GB VRAM. Setup is LM Studio with ROCm on Windows 10 Pro, with a Ryzen 7 7700X and 32 GB DDR5; the phone UI connects over Tailscale. The speed is given only as a range, and the post is mainly about the mobile-first agentic web UI with tool calling for file edits, PowerShell commands and web search.
2× NVIDIA RTX 3060 12GB · llama.cpp
User reports Gemma 4 31B Q5 at 20 t/s on two RTX 3060 12GB cards in llama.cpp, up from 14 t/s with layer splitting, a 42% gain. One card runs on PCIe x16 and the other on a PCIe 3.0 x1 riser; the user notes the fit and context are tight and that testing was limited while awaiting a PSU upgrade. On a rented 2x RTX 5060 Ti 16GB machine, L3.1 Nemotron Sunfall 71B IQ3_XS at 2k context with flash attention gave 20.7 t/s generation and 464.4 t/s prompt processing under tensor split, versus 11.7 t/s and 591.8 t/s under layer split. PCIe traffic during decode measured roughly 90-110 MB/s per direction, with load spikes above 4,000 MB/s.
Intel Arc A770 16GB · llama.cpp · 128,000 ctx
User reports Gemma 4 12B QAT at 20 t/s decode and 1100 t/s prefill on a single Intel Arc A770 16GB. Setup is llama.cpp with QAT weights, Q8 KV cache and a Q4 MTP head, giving 128k context with about 5GB VRAM headroom; the card is in a 13-year-old Haswell desktop with under 6GB RAM headroom. Four models run together at 14-15GB VRAM via llama-swap, with LiteLLM routing prompts through Laya to thinking and non-thinking aliases of the 12B. The user notes SYCL/llama.cpp lacks Alchemist optimizations, making the A770 the bottleneck, and that GPU-resident models still use 4-5GB of system RAM as compute buffer.
Unknown GPU · LiteRT
User reports Gemma 4 E2B running at a consistent 5.5 t/s on an Android phone, with peaks up to 8.8 t/s, in Italian. Setup is LiteRT on the phone's mobile processor, after switching from llama.cpp and updating the device from Android 15 to 16. The user also reports Qwen2.5 1.5B at a consistent 8 t/s on the same device, and says the setup shows local AI on Android is doable without expensive hardware.
User reports gemma-4-26B-A4B Q8_0 (26.9 GB) on a Radeon AI PRO R9700 32GB in a Minisforum DEG2 eGPU over USB4, at about 3,200 t/s prefill (pp512) and 66-71 t/s decode, versus 1,387 t/s and 46.6 t/s on the Strix Halo iGPU. Setup is llama.cpp Vulkan/RADV with -ngl 99 -fa 1, a 256 MB BAR, the card pinned to profile_peak, and both GGML_VK_DISABLE_HOST_VISIBLE_VIDMEM=1 and GGML_VK_MMID_F16B=0 set. A 10 h soak ran 6,724/6,724 requests with decode flat from 66.2 to 63.7 tok/s between the first and last quarter. Big model loads wedge the card at the default power level unless profile_peak is set, which idles at 137 W; a wrapper switching it around jobs passed 22/22 image jobs. A full 32 GB BAR works only with the dock powered on at boot, where 27B Q8_0 decode is 17.9 tok/s at a 1k context.
NVIDIA RTX 3060 12GB · Ollama
User asks about a budget upgrade to run Gemma 4 31B Q5_K_M at 16K context and at least 15 t/s, currently getting 2.3 t/s generation (2.7 t/s with MTP) on an RTX 3060 12GB with 47% GPU / 53% CPU offload. Setup is Ollama/llama.cpp with Q5_K_M weights and Q8 KV cache on an i5-14400F with 32GB DDR4; the 243-token prompt was processed with the model partly offloaded to system RAM. User says IQ3_M drops quality without adding speed, and is weighing an RTX 5060 Ti 16GB plus the 3060 on a PCIe 3.0 x1 riser, dual Tesla P100s on an X79/X99 Xeon system, or a cheaper AMD 16GB card.
NVIDIA RTX 5070 Ti · llama.cpp
User reports 137.7 t/s generation and 5951 t/s prompt processing. Source evidence (verbatim): Source: https://algollabs.com/blog/gemma4-bfcl # Gemma 4 26B on consumer-grade GPU: Benchmarks, Build Pain, and a Week of Daily Use Hardware: NVIDIA RTX 5070 Ti 16 GB GDDR7, AMD Ryzen 9 9950X3D, 64 GB DDR5, Fedora 43, kernel 6.18. Engine: llama.cpp build 9024 built from source with GGML_CUDA_FA_ALL_QUANTS=ON. Model: Gemma 4 26B A4B Instruct (26B total, 3.8B active), unsloth/gemma-4-26B-A4B-it-GGUF UD-IQ4_XS, 12.65 GiB, full GPU offload (31 of 31 layers), flash attention on, asymmetric KV cache (K q8_0, V q4_0). llama-bench, no speculative decoding or MTP, single stream: 5,951 tokens/s prompt processing (pp2048) and 137.7 tokens/s token generation (tg64).
NVIDIA RTX A2000 12GB · Ollama
User reports Gemma 4 12B at 28.22 t/s on the RAID 1 explanation task, run on an RTX A2000 12GB in a Lenovo ThinkCentre M920x Tiny with an i5-9500T and 32 GB RAM. Setup is Ollama on Ubuntu 26.04.1 with Open WebUI in Docker; the model sat fully on the GPU and drew up to 82 W during the run. The Python task was not verified for this model. The same machine ran seven other models: DeepSeek Coder V2 16B was fastest at 45.86 and 46.03 t/s, Qwen3 8B at 30.80 and 31.25 t/s, Qwen3.5 9B at 23.23 and 22.94 t/s, Gemma 3 12B at 20.58 and 20.33 t/s, Ministral 3 14B at 19.82 and 20.29 t/s, Qwen3 14B at 16.60 and 18.53 t/s, and a Qwen of about 27B fell to 1.45 and 1.99 t/s with CPU offload. Idle draw was about 15 W.
AMD Strix Halo 128GB · LM Studio · 131,072 ctx
User benchmarks 12 local models on a Ryzen AI MAX+ 395 with Radeon 8060S and 128 GB unified memory, scoring four tasks (reasoning, management analysis, coding, MCP tool use) out of 100. Setup is LM Studio with GGUF files at a configured 131,072-token context; speeds are observed generation rates, not end-to-end latency. Gemma 4 26B-A4B QAT Q4_0 and Qwen3.8 27B Q6_K tie at 90/100, at 52-54 t/s and 15-17 t/s respectively; Qwen3.8 was the only model that checked an existing file before overwriting it. GPT-OSS 120B MXFP4 scored 87 at 29-30 t/s with partial GPU offloading. Each model was run once per task and scores are subjective rubric evaluations.
2× NVIDIA RTX 5060 Ti 16GB · llama.cpp
User reports Gemma 4 31B IT QAT at over 60 t/s generation and 1100 t/s prompt processing on dual RTX 5060 Ti 16GB cards. Setup is llama.cpp 0.7.0 with QAT weights, tensor split and MTP enabled across the two cards. Earlier runs on the same setup gave 40 t/s gen and 980 t/s pp, and 17 t/s gen with 970 t/s pp at 32K for Q6_K. Qwen3.8 27B runs at around 70 t/s with MTP and 100 t/s when generating code. The user uses the models for roleplay in SillyTavern.
NVIDIA RTX 4090 · Glyd · 32,768 ctx
User reports Gemma 4 31B QAT at 46 tok/s on an RTX 4090. Setup is Glyd 0.31 with Google's QAT 4-bit checkpoint (4.5 bits per weight) at 32K context, using 21.8 of 24.6 GB. With Google's draft model, chat prompts reach 78-80 tok/s and predictable text up to 176 tok/s. A 30K-token prompt took 20 s. On an RTX 3090 the same model runs at 26 tok/s; a 12B QAT on an RTX 4080 SUPER runs at 82.5 tok/s plain and 109 tok/s with the draft model. Quality vs Google's QAT weights: mean KLD 0.018, median 0.003, top-1 96%, GSM8K 72/75, MMLU-Pro 70/75.
AMD Strix Halo 128GB · llama.cpp
User reports Gemma 4 26B-A4B QAT at 102/102/127 tok/s (Swedish prose / English prose / JSON) on a Strix Halo 128GB box, about 45% faster than the regular UD-Q4_K_M at 70/71/90 tok/s. Setup is llama.cpp Vulkan/RADV build 11235 with QAT UD-Q4_K_XL and the matching QAT MTP drafter, single request, -ctxcp 0, using 17.7 GB versus 20.9 GB for Q4_K_M. Lower K-quants (Q3_K_XL, IQ3_S) gave no speed gain. Nemotron 3.5 Lightning 30B-A3B and Qwen3.6-35B-A3B made factual errors in 50-80% of summaries versus about 10% for Gemma. Strata v0.1.44.1 with Qwen3.8-Flash-Next IQ3_S decodes at 50.9 tok/s shallow context and passed 256k needle tests up to 242k tokens with prefill flat near 1,200 tok/s.
Unknown GPU
User reports running Gemma 4 26B A4B on an HP laptop with an AMD Ryzen 3 3200U and 24GB DDR4 RAM, no GPU, at around 8 t/s. No engine, quant, or context length is stated. The user is surprised the laptop can run the model and asks what breakthroughs to expect in the next 2-5 years.
Unknown GPU · L-AI
User reports running Gemma 4 E4B IT Q8 on a OnePlus Pad 3 with 16GB RAM using L-AI, getting 6-7 tok/s. Qwen2.5-1.5B reached 9-10 tok/s and Qwen3-0.6B-F16 reached 37 tok/s on the same device. The runs are CPU-only on the tablet. User asks whether anyone is working on a proper interface for running these models that supports uploading files and photos.
NVIDIA RTX 4060 Laptop 8GB · llama.cpp
User reports Gemma 4 26B MoE at 60+ tok/s on an RTX 4060 Laptop, up from 40 tok/s with standard llama.cpp. Setup is llama.cpp with MTP speculative decoding plus a custom DeepSeek-inspired hot-expert caching scheme that keeps frequently used experts on the GPU. The user asks for further performance ideas without changing the model.
Unknown GPU · llama.cpp · 4,096 ctx
User benchmarks Gemma 4 E2B QAT GGUF on a Pixel 8 Pro (Tensor G3) with llama.cpp, reporting 58 prompt tok/s and 5.0 gen tok/s with 4 threads pinned to medium cores 4-7. Setup is llama.cpp via the Rust llama-cpp-2 crate, QAT GGUF (2.6 GB), 4096 context, q8_0 KV cache, CPU-only. The fastest setup was 5 threads pinned to cores 4-8 at 70 prompt tok/s and 4.9 gen tok/s, while 4 medium cores were within 8% and used about 10% less battery. Vulkan on Mali was 3x slower with all layers on GPU and 8x slower with half the layers. The user notes that Android background apps only get the small cores, around 14 s per item.
User reports Gemma 4 26B-A4B QAT at about 170 tokens a second on one RTX 4090, with a full two-call pass taking 5 seconds. Setup is Ollama with QAT weights, one card, 96 GB system RAM; vLLM was faster at 3.5 seconds per pass but held 94% of the card against Ollama's 70%. Sending both calls of a pass together cut it to 3.8 seconds on Ollama and 2.8 seconds on vLLM. MedGemma 27B Q4_K_M took 18 seconds per pass. The user asks about speculative decoding, other engines, other models, and CPU offload.
RTX 3050 8GB · SonicBoom
User reports Gemma 4 E4B-it at 51.65 tok/s decode on an RTX 3050 8GB laptop, compared to 45.20 tok/s with llama.cpp on the same machine and GGUF file. Setup is SonicBoom, a C++/CUDA runtime, with Q3_K_M quant and greedy sampling. The decode lead comes from a graph compiler that fuses quantization and captures each step as a CUDA Graph; the quantized matmul kernels are ported from ggml. Prefill is about half of llama.cpp at 1051 tok/s versus ~1944 tok/s. Decode at depth 2048 is 47.13 tok/s. TTFT for a 4000-token prefix plus 100 new tokens is 503 ms with prefix reuse. Only sm_86 is tested and only Q3_K, Q4_K and Q5_K formats load.
NVIDIA DGX Spark · llama.cpp
User reports 59.42 t/s generation and 3633.84 t/s prompt processing. Source evidence (verbatim): Source: https://blog.majid.info/llama-bench/dgx1.txt # NVIDIA DGX Spark 128GB / GB10: original Gemma4 E4B Q4_K_M llama-bench log First-hand author log linked from https://blog.majid.info/llama-bench/ (April 13, 2026 article). Measured hardware: NVIDIA DGX Spark 128GB / GB10. Gemma4 E4B Q4_K_M GGUF, llama.cpp ff5ef8278 (8763), backend CUDA. pp512 3633.84 ±201.59 tokens/s; tg128 59.42 ±0.96 tokens/s. One GB10, ngl99; 122502 MiB runtime device VRAM visibility. 128GB is unified system capacity, not dedicated VRAM or resident use. Command names --hf-repo unsloth/gemma-4-E4B-it-GGUF; exact tested quantization repository https://huggingface.co/unsloth/gemma-4-E4B-it-GGUF ; tested revision unknown. Log reports 4.62 GiB model size and 7.52B counted parameters; size is not resident memory. Context depth, KV quantization, measured power, driver version and explicit speculation/concurrency declarations unreported. Raw source: https://blog.majid.info/llama-bench/dgx1.txt .
Radeon 860M · llama.cpp
User reports 6.65 t/s generation and 291.48 t/s prompt processing. Source evidence (verbatim): Source: https://blog.majid.info/llama-bench/xhystos.txt # AMD Ryzen AI 7 350 / Radeon 860M, 32GB system RAM: original Gemma4 E4B Q4_K_M llama-bench log First-hand author log linked from https://blog.majid.info/llama-bench/ (April 13, 2026 article). Measured hardware: AMD Ryzen AI 7 350 / Radeon 860M, 32GB system RAM. Gemma4 E4B Q4_K_M GGUF, llama.cpp 82764d8f4 (8770), backend ROCm. pp512 291.48 ±49.07 tokens/s; tg128 6.65 ±0.48 tokens/s. gfx1150; one device, ngl99; 15609 MiB runtime device VRAM visibility, not dedicated VRAM or resident memory. Command names --hf-repo unsloth/gemma-4-E4B-it-GGUF; exact tested quantization repository https://huggingface.co/unsloth/gemma-4-E4B-it-GGUF ; tested revision unknown. Log reports 4.62 GiB model size and 7.52B counted parameters; size is not resident memory. Context depth, KV quantization, measured power, driver version and explicit speculation/concurrency declarations unreported. Raw source: https://blog.majid.info/llama-bench/xhystos.txt .
On Qwen3.8 27B at 4-bit, plain single-GPU runs. Full ranking