Gemma 4

reported speed:

30.1 tokens/s generation · 422.3 tokens/s prompt processing

quant:

Q4_K_XL (GGUF)

kv:

Q4_0

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User benchmarks Gemma 4 31B on an RX 7900 XTX, reporting 30.13 t/s generation and 422.27 t/s prompt processing with ROCm. Setup is llama.cpp on Windows 10 with the UD-Q4_K_XL GGUF, flash attention, 32K context, all layers on GPU, and a Q4_0 KV cache. Vulkan single-GPU gave 24.81 t/s generation and 117.29 t/s prompt, while Vulkan multi-GPU with an added RTX 3080 gave 26.18 t/s generation and 74.69 t/s prompt. User notes ROCm started at maximum generation speed and declined with context, whereas Vulkan briefly sped up before declining.

reported speed:

25.9 tokens/s generation · 639.0 tokens/s prompt processing

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports a custom C runtime for CPU inference with int8 weights, OpenMP, AVX2 and AVX-512 VNNI, running faster than llama.cpp on a Ryzen 7 7700.

reported speed:

50.0 tokens/s generation

quant:

Q5_K_XL (gguf)

kv:

Q8

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports a switch from Q4_K_XL at 61 t/s to Q5_K_XL at 50 t/s for better syntax accuracy. The model file is about 8.6 GB, with total VRAM usage around 15.7 GB at 32k context and a Q8 KV cache. The user finds it compares favorably to Qwen 3.6 27B and 35B A3B for ease of use and tool call compatibility.

reported speed:

73.9 tokens/s generation

quant:

Q4_K_M (gguf)

kv:

Q4

rating:

3/5

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports that adding Mempalace as an MCP to LM Studio did not work properly, fetching memory only sometimes. The same setup used via Copilot Chat in VS Code Insiders may stop responding or modify only a small part of the code.

reported speed:

51.9 tokens/s generation · 322.0 tokens/s prompt processing

quant:

Q4_K_M (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports a 36% improvement in generation speed with a dual GPU setup of an RX 7900 GRE 16GB and an RX 480 8GB. The user tested medgemma-27b-it-UD-Q6_K_XL, Qwen3.8-27B-Q6_K, Qwen3.8-27B-OBLITERATED-Q5_K_M, and Gemma4-26B-A4B-QAT-Uncensored-HauhauCS-Balanced-Q4_K_M, with and without Flash Attention.

reported speed:

51.7 tokens/s generation · 2488.0 tokens/s prompt processing

quant:

Q8_0

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User benchmarks TensorSharp multi-GPU tensor parallelism on 2x RTX 2000 Ada 16GB. Gemma 4 E4B Q8_0 is the primary model among several tested. TP=2 raises decode speed from 37.3 to 51.7 tok/s.

User benchmarks Gemma4-12B on an RTX 5090, comparing vLLM stock, vLLM+Emmy, and llama.cpp. The runs report throughput and latency across various token lengths and concurrency levels. The comparison also covers speculative decoding (MTP) tests and kernel optimization details.

reported speed:

10.0 tokens/s generation

quant:

Q4_K_XL (gguf)

kv:

Q4

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports Gemma4 31B at about 10 t/s, compared with Qwen3.8 27B at 40-70 t/s and a 120B MoE at 10-20 t/s. Setup is llama.cpp with speculative decoding using an MTP draft model and a Q4 KV cache. The user notes memory allocation on a second GPU despite free VRAM on CUDA0.

quant:

UD_Q4_K_XL (gguf)

User benchmarks Gemma 4 IT 12B QAT on a 12GB VRAM card, scoring 87/93, against Qwen 3.8 27B Q8 on ~30GB VRAM, scoring 90/93. The 12B model is recommended as the default. User highlights a validation feature to prevent fabricated data.

reported speed:

5.5 tokens/s generation

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports 31-35 t/s on an M5 MacBook Pro with a custom Swift/Metal engine. Setup is an OpenAI-compatible server with streaming and tool-call support.

reported speed:

3029.0 tokens/s prompt processing

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports custom w8a8 kernels on an M5 MacBook Air, with prefill improving from 2193 t/s to 3029 t/s for 130k tokens. User also mentions llama.cpp for Macs.

User benchmarks Gemma 4 QAT against regular quants on an AMD 7900 XTX. No token/s figures are reported; wall clock times show speedups, including 45% faster and an 83% throughput increase for the 12B QAT. Models tested are 12B, 26B, 31B and E4B. The user reports quality as identical.

reported speed:

138.0 tokens/s generation

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User benchmarks Gemma 4 26B-A4B against Gemma 4 12B on an RTX 4090. The 26B-A4B runs at 138 tok/s using 15 GB of VRAM, while the 12B runs at 80 tok/s using 9 GB. The 26B-A4B wins every scene and runs about 1.7x faster. The user notes the 12B is ideal for a 16 GB laptop.

reported speed:

243.9 tokens/s generation · 13809.2 tokens/s prompt processing

quant:

Q4_K_M (gguf)

kv:

Q8

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User benchmarks an FWHT CUDA implementation for KV cache quantization on Gemma 4 26B.A4B Q4_K_M. Setup is llama.cpp with -ctk q8_0 -ctv q8_0, reporting pp2048 and tg128 values. The implementation gives a 1-2% pp boost and a 7-9% tg boost, with the highest t/s taken from the cuda-fwt column.

reported speed:

125.3 tokens/s generation

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User benchmarks Gemma 4 31B dense with MTP and DFlash speculative decoding, alongside Gemma 4 26B-A4B MoE (25.2B total, 3.8B active). At concurrency 1 the dense baseline runs 40.3 tok/s, MTP 125.3 tok/s and DFlash 122.1 tok/s, making MTP 3.11x and DFlash 3.03x faster than baseline. At concurrency 16 the dense baseline reaches 375 tok/s, MTP 953 tok/s and DFlash 725 tok/s. For the MoE at concurrency 1 the baseline runs 177.1 tok/s, MTP 264.2 tok/s and DFlash 306.4 tok/s. At concurrency 16 the MoE baseline reaches 975 tok/s, MTP 1808 tok/s and DFlash 1957 tok/s. Coding, math, STEM and reasoning benefited more.

reported speed:

578.0 tokens/s generation

quant:

AWQ-4bit

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports 228 t/s baseline and 578 t/s with DFlash speculative decoding, a 2.56x speedup, on vLLM 0.19.2rc1. Setup uses the z-lab/gemma-4-26B-A4B-it-DFlash draft model, with 256 input tokens and 1024 output tokens.

reported speed:

97.0 tokens/s generation

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports a Multi-Token Prediction (MTP) implementation yielding a 40% speedup, reaching 138 t/s with MTP.

reported speed:

7.5 tokens/s generation

flash attention:

on

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports Gemma 4 at 7.5 t/s and Qwen3.6-27B locking up, while Qwen3.6-35B-A3 is fast. User suspects a bug affecting dense models.

reported speed:

7.5 tokens/s generation

flash attention:

on

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports dense models running poorly on an M5 Max 128GB, with Gemma4-31B at about 7.5 t/s and Qwen3.6-27B locking up, while Qwen3.6-35B-A3B MoE is fast. The user mentions using DFLASH, likely flash attention.

reported speed:

40.0 tokens/s generation

quant:

Q4_K_M (gguf)

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports Gemma 4 E4B at 35-45 t/s on an RX 7900 XTX. Setup is Ollama with the Q4_K_M quant. User calls it the best consumer AMD option.

reported speed:

58.0 tokens/s generation · 83.0 tokens/s prompt processing

quant:

FP16 (safetensors)

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports Gemma 4 E4B at 57.96 gen tok/s and 82.96 prompt tok/s on an RX 7900 XTX. Setup is vLLM with ROCm on the default path. Source is flexinfer.ai.

reported speed:

7.5 tokens/s generation

flash attention:

on

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports Gemma4-31B at 7.5 t/s and Qwen3.6-27B locking up on an M5 Max with 128 GB, while Qwen3.6-35B-A3 is fast. Setup uses DFLASH.

reported speed:

7.5 tokens/s generation

quant:

Q4 (gguf)

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports Gemma 4 E2B at 5-10 t/s on CPU. The run is CPU-only, with no GPU involved. User considers E2B usable in this CPU-only configuration.

reported speed:

55.0 tokens/s generation

quant:

Q4_K_M (gguf)

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports about 55 t/s on an RTX 4070 12GB. The figure is estimated from compute-market tiers rather than a measured run.

reported speed:

60.0 tokens/s generation

quant:

Q4_K_M (gguf)

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports E2B at about 60 t/s on an RTX 3060 12GB. The figure is estimated from compute-market tiers rather than measured.

reported speed:

45.0 tokens/s generation

quant:

Q4_K_M (gguf)

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports about 45 t/s on an RTX 3060 12GB. The E4B model fits easily.

quant:

NVFP4 (safetensors)

User reports nvidia/Gemma-4-26B-A4B-NVFP4 running on an RTX 5090 with 80% of the 32 GB allocated, reaching around 50k context. The model is 18.8 GB. Benchmarks compare baseline against NVFP4: GPQA Diamond 80.30% vs 79.90%, AIME 2025 88.95% vs 90.00%, MMLU Pro 85.00% vs 84.80%, LiveCodeBench pass@1 80.50% vs 79.80%, IFBench 77.77% vs 78.1%, and IFEval 96.60% vs 96.40%.

reported speed:

149.6 tokens/s generation · 15.6 tokens/s prompt processing

quant:

Q4_K_M (gguf)

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports about 150 t/s generation. The model is described as a star performer.

reported speed:

16.9 tokens/s generation

quant:

bf16 (safetensors)

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports bf16 with no quantization at 10.25 GB VRAM and 61 ms TTFT. Source is dev.to Gaurav Vij.

reported speed:

17.5 tokens/s generation

quant:

Q4 (gguf)

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports Gemma 4 at 15-20 t/s, a range the user calls usable for simple tasks.

reported speed:

68.8 tokens/s generation · 204.1 tokens/s prompt processing

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports 204.07 t/s prompt eval and 68.76 t/s generation on an M5 Pro with 48 GB RAM.