Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
codingmath
User benchmarks Gemma 4 E4B with MTP speculative decoding on a MacBook Pro M4 Pro 24GB, reporting 82.8 tok/s versus 44.6 tok/s baseline (1.86x mean, 2.03x best speedup).
Setup is LiteRT-LM Metal GPU with int4 weights at batch=1, temperature=0, using real prompts; the E2B model reaches 144.3 tok/s with MTP versus 84.2 tok/s baseline. HF transformers + MPS with bf16 weights achieves only 12.1 to 15.1 tok/s (1.24x), and llama.cpp Metal Q8_0 cannot engage MTP for Gemma 4.
The user notes that synthetic-token benchmarks show MTP as a regression (0.93x) because random sequences yield near-zero drafter acceptance, and warns that E4B bf16 on 24GB Macs can trigger kernel panics from Metal wired-memory pressure.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User benchmarks the Gemma 4 family on an NVIDIA DGX Spark (GB10 Grace Blackwell SoC) with 128 GB unified memory, served via llama.cpp in a Docker container.
Setup is llama.cpp build 5d3a4a7 with GGUF weights, flash attention enabled, batch size 2048, and a 256K context window. The 26B-A4B MoE model uses Q4_K_M and reaches 69.9 t/s generation and 2657 t/s prefill at 512 tokens.
The 31B dense model reaches 11.0 t/s generation and 743 t/s prefill. At 32K context depth, the 26B drops to 52.1 t/s generation and 1877 t/s prefill. Batched concurrency on the 26B reaches 537 t/s aggregate generation at 32 concurrent requests and 3015 t/s aggregate prefill at batch 16.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Gemma 4 E4B at 25.6–28.3 tok/s on an M4 alone, with TTFT of about 0.31 s on a 71-token prompt.
Setup is Ollama with 4-bit weights; a distributed run across the M4 and an RTX 3060 6 GB over Wi-Fi reached about 8 tok/s with layers 0–14 on the M4 and 14–42 on the RTX, and about 10.2 tok/s with layers 0–4 on the M4 and 4–42 on the RTX.
Gemma 4 E2B runs at 42.8 tok/s on the M4 alone, up from about 11.5 tok/s before optimising the decode path. The user notes that adding a second machine does not automatically speed up inference and is exploring task-level rather than layer-level distribution.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Gemma 4 E4B at about 11 tok/s steady-state decode on a Pixel 10 Pro XL (Tensor G5, 16 GB shared memory).
Setup is an Android OpenAI-compatible server app (PixelUnlockGPU) running the model on-device via LiteRT-LM with a GPU bundle of 2.97 GB and context capped at 32k; follow-up turns take about 1.7 s because the KV prefix is reused across turns, and the engine warm build takes about 12 s once per model change.
The user notes the NPU path aborts on stock Tensor G5 firmware so GPU is the shipping backend, short replies read slower than 11 tok/s because warm and prefill dominate, and token counts are estimated at about 4 chars/token since LiteRT-LM exposes none.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Gemma 4 26B at 75 t/s generation and 1500 t/s prompt processing on 2x RTX 4060 8GB.
Setup is LM Studio with an IQ quant, serving to Hermes.
User credits mradermacher's IQ quants for getting the model running.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Gemma 4 26B at 65-68 tok/s on an AMD Instinct MI50 32GB.
Setup is Ollama v0.32.14 with Q4_K_M and q8_0 KV cache, flash attention on, all layers offloaded, MTP speculative decoding enabled automatically.
The figure is a range; short answers reach ~95 tok/s prompt eval. A QAT Q4_0 build with MTP reaches ~78 tok/s, and Qwen 3.5 35B Q6_K runs at ~52 tok/s with ~541 tok/s prompt eval. Power overdrive to 160W adds another +8-9%.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Gemma 4 31B IT NVFP4 Turbo at 51 tok/s single-stream decode and 15359 tok/s prefill on an RTX PRO 6000.
Setup is vLLM with NVFP4 weights, FP8 KV cache, and 16384 tokens of context, using 18.5 GiB of GPU memory.
Batched decode reaches 1244 tok/s and concurrency 6.22 req/s at 1K input / 200 output tokens; the same benchmark on an RTX 5090 performed identically because at 16k context performance is not limited by GPU memory.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Gemma 4 at 6.97 t/s generation on an Intel Arc GPU (Core Ultra 7 155U) with the SYCL backend, compared to 20.09 t/s on the Vulkan backend under identical conditions.
Setup is llama.cpp with Q4_K_M, flash attention on, 64k context, and MTP speculative decoding with a draft model.
Prompt processing was 25.59 t/s on SYCL versus 21.47 t/s on Vulkan, and draft acceptance was 51.07% on SYCL versus 59.61% on Vulkan.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
visionmultilingual
User reports Gemma 4 26B at 182 t/s decode and 5660 t/s prefill on an RTX 5080 Laptop 16GB GPU.
Setup is the custom Gem16 engine with an EXL3-like custom quantisation, fitting 220k context; the engine is single-user and uses MTP speculative decoding.
The user also reports Gemma 4 12B at 87 t/s decode and 5800 t/s prefill, and notes the 12B can serve 2 sessions. The 12B's native audio understanding degrades past around 8k context, which the user attributes to the model.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Gemma4 26B Q4_K_M at ~25.00 t/s generation and ~208.72 t/s prompt processing on an AMD Radeon 780M iGPU.
Setup is llama.cpp with Vulkan backend, Q4_K_M GGUF, -ngl 99, on a MINISFORUM UM890 Pro mini PC running Ubuntu 24.04.
A CLI smoke test gave ~23.4 t/s generation and ~37.3 t/s prompt; a no-reasoning run gave ~24.4 t/s generation and ~117.6 t/s prompt. Ollama on the same box was around 4.5 t/s generation, roughly a 5x-6x uplift. Ollama's installed Gemma4 blob could not be loaded directly in upstream llama.cpp due to a tensor count mismatch.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
codingvisionagenticlong-context
User reports Gemma 4 26B-A4B at ~129 tok/s single-request generation on a single RTX 4090, with ~5,600 tok/s prefill.
Setup is a TurboQuant fork of llama.cpp with Q5_K_M weights, turbo3 3-bit KV cache, Flash Attention, and a multimodal projector, using 22.3 GB of 24 GB VRAM at 262,144 tokens of context.
Concurrent generation is reported at ~83-85 tok/s. A 15-test agentic coding benchmark through Claude Code passed 15/15 with an average quality of 4.2/5.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Gemma 4-26b-a4b at 80~90 t/s at 2048 context and 40+ t/s at 96K context on an RTX 5060 Ti 16GB.
Setup is IQ4-XS quantization on an Intel i7-4790 with DDR3 1600MHz RAM.
User compares it favorably to Qwen 3.5-35b-a3b at 45 t/s and notes the older DDR3 platform limits memory bandwidth.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Gemma 4 12B at 40.0 t/s on an RTX 5060 Laptop GPU with 8 GB VRAM, using MTP speculative decoding with a Q6_K draft head and F16 KV cache at 4096 context.
Setup is llama.cpp with mainline draft-mtp support (commit b9193 or later), an imatrix-guided per-tensor fit-to-VRAM quant at 6.18 GB, 48 layers, parallel 1. Prefill is 700 to 900 t/s.
Without MTP the same setup reaches 27.4 t/s. With a Q8 KV cache it reaches 40.4 t/s. In real chat use the user sees 42 or more t/s on fresh context, dropping to 33 to 37 t/s with 16k of context filled.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Gemma 4 26B-A4B at 142.76 t/s generation and 4331.21 t/s prompt processing on an RX 7900 XTX after a llama.cpp Vulkan int8 coopmat1 matmul patch.
Setup is llama.cpp with the Q4_0 GGUF, Vulkan backend, flash attention enabled, n_ubatch 1024, and all layers offloaded.
Before the patch the same run measured 135.64 t/s generation and 3410.53 t/s prompt processing. At 8192 context the figures are 125.29 t/s generation and 2885.67 t/s prompt processing; at 16384 context they are 120.82 t/s and 2402.13 t/s.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Gemma 4 26B A4B at 40-50 t/s on an RTX 4070 Laptop 8GB with 32GB system RAM.
Setup is Unsloth Studio with the QAT GGUF, 65,536 context, q8 KV cache, MTP speculative decoding with 2 draft tokens, and 31 GPU layers.
Initial throughput was around 20 t/s; dropping context length and shifting more layers to the GPU raised it to 40-50 t/s.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
agentictool-usecodinglong-context
User reports Gemma 4 e4b-it-qat at 33 tok/s on a 16 GB M1 Pro, up from 27 tok/s with the non-QAT build.
Setup is Ollama with a 6.4 GiB footprint; the 12B deep model cannot stay co-resident, so each swap costs a 4 to 7 s cold load. Moving per-message content out of the system prompt cut follow-up time to first token from 1.8-2.0 s to 0.4-0.5 s.
Tool-calling accuracy on 9 prompts was 8/9 bare, 5/9 with a 700-token system prompt, and 9/9 with a one-line nudge placed after the question. Memory injection of 109 facts cost 2,300 tokens per turn; recalling the nearest 10 costs about 150 tokens.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Gemma 4 26B-A4B at 52.03 t/s generation and 340.86 t/s prompt processing on a dual-GPU setup of GTX 1080 Ti and Radeon MI50 16GB, totaling 27GB VRAM.
Setup is llama.cpp Vulkan pre-built binary build 851cb34f2 (11055) with the UD-Q6_K_XL GGUF, flash attention on, and 99 GPU layers offloaded.
Single-GPU GTX 1080 Ti runs of the same model reached 11.76 t/s generation and 162.12 t/s prompt processing. The user also benchmarked Qwen3.6-35B-A3B MXFP4 MoE, Nemotron 31B-A3.5B Q5_K_M, Qwen3.8 27B Q6_K, and medgemma 27B Q6_K_XL, noting dense models benefited most from the second GPU.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports a 26B-A4B Gemma 4 uncensored MoE running at about 13 tok/s on an RTX 4060 Ti 8GB, with experts offloaded to CPU.
Setup is LM Studio with a Jev decision-model router that picks between a 3B, a 4B, a 12B and the 26B MoE; only one model fits in VRAM at a time, so a misroute costs a 15 to 45 second model swap.
The router agreed with the user's labels 16 out of 16 on labelled prompts, with median latency 530ms and p95 about 643ms; a post-reply judge caught refusals 15 out of 15 on synthetic pairs and 5 of 6 on real replies.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
multilingual
User reports Gemma 4 26B-A4B at over 10 t/s on an RTX 3050 6GB with 16GB DDR4 system RAM.
Setup uses the Ultra-Uncensored-Heretic IQ4_XS GGUF.
User is using it for translation and finds the results decent.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User benchmarks Gemma 4 31B on an RX 7900 XTX, reporting 30.13 t/s generation and 422.27 t/s prompt processing with ROCm.
Setup is llama.cpp on Windows 10 with the UD-Q4_K_XL GGUF, flash attention, 32K context, all layers on GPU, and a Q4_0 KV cache.
Vulkan single-GPU gave 24.81 t/s generation and 117.29 t/s prompt, while Vulkan multi-GPU with an added RTX 3080 gave 26.18 t/s generation and 74.69 t/s prompt. User notes ROCm started at maximum generation speed and declined with context, whereas Vulkan briefly sped up before declining.