Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
vision
User benchmarks Gemma 4 E2B at 205 t/s decode at 4k context on an Apple M5 Max with 128 GB unified memory.
Setup is mlx-vlm 0.4.4 with MLX 0.31.1, 4bit weights, and TurboQuant 2.5-bit KV cache quantization; the model uses 4.7 GB at 4k.
Decode falls to 78 t/s at 256k context. TurboQuant gives +15-19% at 128-256k on larger models but is negative on E2B. The same report also covers Gemma 4 E4B, Gemma 4 26B-A4B, Gemma 4 31B, Qwen 3.5 2B and Qwen 3.5 4B.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
mathtool-usemultilingualvisionlong-context
User reports Gemma 4 26B-A4B-it NVFP4 at 48.2 t/s for a 1000-token generation on an NVIDIA DGX Spark (GB10 Blackwell, 128GB unified memory), against 23.3 t/s for the BF16 original.
Setup is vLLM with an NVFP4 (W4A4) modelopt checkpoint and FP8 KV cache, single DGX Spark, Marlin MoE backend, 256K max context; the model loads at 15.7 GiB.
TTFT is 53 ms versus 97 ms for BF16, and the checkpoint is ~16.5 GB on disk versus ~49 GB. Quality retention averages 97.6% across GSM8K and IFEval, with GSM8K down about 4 percentage points.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Gemma 4 31B IT NVFP4 at 66.80 tok/s aggregate effective batch throughput on an RTX 5070 Ti 16GB.
Setup is llama.cpp with NVFP4 GGUF weights, a Gemma 4 31B Assistant/MTP draft model, --spec-type draft-mtp, --spec-draft-n-max 5, flash attention on, and all layers offloaded to GPU. The benchmark used mtp-bench across 13 mixed tasks with 2,016 predicted tokens in 30.18 s and a 60.2% draft acceptance rate.
Aggregate throughput rose from 53.10 tok/s at n_max 2 to 66.80 tok/s at n_max 5, while acceptance fell from 78.2% to 60.2%. Per-task speeds ranged from 35.8 tok/s on creative writing to 104.6 tok/s on repeated patterns. The author notes these are aggregate figures and that no non-speculative baseline was measured.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User benchmarks Gemma 4 12B at 82.7 tok/s decode on vLLM and 85.0 tok/s on Ollama on one RTX 3090 24 GB capped at 300 W.
Setup is vLLM 0.30.0 with Google's W4A16 QAT checkpoint and Ollama 0.34.4 with gemma4:12b-it-qat, temperature 0, 256 tokens out, one request at a time.
At sixteen concurrent requests vLLM reached 506.8 tok/s against Ollama's 142.3 tok/s; first token was 213.0 ms on vLLM against 441.9 ms on Ollama. Board energy at sixteen requests was 590 joules per 1,000 tokens on vLLM and 1,384 on Ollama. On 108 questions the two agreed on 63 strictly and 98 leniently.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Gemma 4 26B-A4B UD-Q4_K_XL at 15.8-16.4 tok/s generation and ~245 tok/s prefill without MTP on a GTX 1060 6GB with 16GB of single-channel DDR4.
Setup is a single GTX 1060 6GB, UD-Q4_K_XL weights, llama.cpp with 23 MoE layers' experts on the CPU (--n-cpu-moe 23), no speculative decoding, 16K context.
With the MTP Q8_0 drafter (--spec-draft-n-max 3) generation rises to 19-21.5 tok/s at 55-65% draft acceptance; a DFlash drafter managed only 13 tok/s at 25% acceptance.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
agentictool-use
User benchmarks 18 tool-calling models on 33 recorded DevOps failure scenarios on a single RTX 5090 32GB, reporting gemma4:31b at 64 tokens/s with a 94% pass rate and 23.1 GiB measured VRAM.
Setup is Ollama 0.35.0 with each model's default build (Q4_K_M, except gpt-oss:20b at MXFP4), 32k context, temperature 0, one run each; VRAM is what nvidia-smi showed for the Ollama process.
The top six models tie between 88% and 94%, separated by speed and memory; the fastest model on the card (nemotron-3.5-lightning:30b at 427 tokens/s) passes 52%, and gemma4:12b passes 88% in 9.2 GiB. Ollama's own ollama ps size under-reported real GPU usage by 0.6 to 4.2 GiB for 17 of 18 models. north-mini-code-1.0 could not run due to a CUDA illegal memory access error. Nothing spilled to CPU.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Gemma 4 E4B at approximately 29.5 tokens/s on a Mac mini M4 with 24GB unified memory.
Setup is LM Studio 0.4.12 with llama.cpp engine 2.13.0 (Metal), GGUF Q4_K_M, 4096-token context, automatic GPU offload, temperature 0, 200 generated tokens per request; two trials gave 29.62 and 29.46 tokens/s with about 0.3s time to first token.
A separate run at 32768 context and 12961 prompt tokens gave 25.95 tokens/s with 39.44s cold TTFT and 0.12s repeat TTFT from prefix caching. Four simultaneous requests with parallel=4 gave 9.66-9.69 tokens/s per request and 37.5 tokens/s aggregate. Source-reported measurements, not independently reproduced; prefill throughput, measured memory, KV-cache quantization, Flash Attention and speculative-decoding settings unspecified, and the 6.29GiB memory figure is an estimate.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Gemma 4 26B-A4B at 6.86 t/s generation and 2181.18 t/s prompt processing on an RTX 5070 Ti with ik_llama.cpp.
Setup is ik_llama.cpp with Q4_K_XL GGUF and q8_0 KV cache, 30 MoE layers offloaded to CPU, 8192-token prompt and 128-token generation.
Upstream llama.cpp on the same hardware and model reaches 30.10 t/s generation and 577.00 t/s prompt processing, making ik_llama.cpp about 4.4x slower for generation despite being 3.8x faster for prompt processing. Disabling CUDA graphs, using q4_0 KV cache, or keeping the last 4 MoE layers on GPU did not resolve the generation slowdown.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports a ~3.8x generation speed regression on Apple M4 Mac Mini (16 GB unified memory) with Gemma 4 26B A4B IQ4_XS. Master build (d12cc3d) achieves 8.14 tok/s versus 27.18 tok/s on tag b8680, both built from source with llama-server and identical flags. Setup uses Metal backend, --flash-attn on, q4_0 KV cache, 65536 context, and -ngl 99. The regression is reproducible and not explained by memory or swap pressure; prefill on b8680 was 55.7 tok/s.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
agentictool-uselong-context
User reports Gemma 4 26B-A4B QAT at 36-38 tok/s generation on an M4 Pro 24GB MacBook Pro.
Setup is Ollama (llama.cpp) with QAT 4-bit weights, q8_0 KV cache, flash attention on, and a 64K context window; the model is 14 GB and 100% GPU-resident.
Prefill is 95-125 tok/s on short prompts, cold load ~16 s, warm reload 0.4 s. The user notes the model squeezes the rest of the OS (9% memory free, swap nearly full) and that a dense 31B would spill to CPU.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
long-context
User benchmarks Gemma 4 12B (IQ4_NL, 6.24 GiB) on an AMD RX 6700 XT under llama.cpp, comparing the ROCm and Vulkan backends at 8192-token prefill and 512-token generation with q8_0 KV cache and flash-attention on.
ROCm averages 653.9 t/s prefill and 34.60 t/s decode over 3 runs; Vulkan averages 354.4 t/s prefill and 40.92 t/s decode. ROCm is 84.5% faster on prefill but 15.4% slower on decode, giving a net wall-clock win of about 23% for a full 8192-prefill plus 512-generate cycle, with a crossover near 1760 prompt tokens.
ROCm required two workarounds on gfx1031: building for gfx1030 with HSA_OVERRIDE_GFX_VERSION=10.3.0, and patching a flash-attention assert in fattn-common.cuh. A separate TOP_K sampler gap on ROCm is noted as under investigation.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Gemma 4 26B-A4B at 23.6 tok/s decode and 55 tok/s prompt processing in a browser tab on an M4 Pro 24GB Mac.
Setup is LocalMind (WebGPU) with Google's QAT Q4_0 GGUF (14.4 GB), a 4 GB expert cache, and routed experts streamed from disk; the Chrome GPU process sits at 6.9 GB with about 8.6 GB of experts on disk.
llama.cpp Metal does 70.6 and 204 tok/s on the same Mac, so the tab is about 3x slower at decode. The user also ran Qwen3.6 35B-A3B (unsloth Q8_0, 36.9 GB) at 9.9 tok/s decode with 2.2 s to first token, a file larger than the machine's memory. Output matched llama.cpp on 9/9 Gemma conversations and 8/8 Qwen cuts.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Gemma 4 26B-A4B-it at 95.22 tok/s decode and 3,476 tok/s prompt processing on an RTX 5060 Ti 16 GB.
Setup is a custom llama.cpp build (commit 0b484ab2b plus 12 commits) with MXFP4-MOE quantization (experts MXFP4, dense Q8_0, 4.66 BPW) and q4_0 KV cache, 1 slot, flash attention enabled, 65,536 token context.
The MXFP4-MOE quant brings the model to 13.70 GiB, fitting the 16 GB card where Q4_K_M at 15.85 GiB does not. On an RTX 5090 the same quant gives 10,733 tok/s prompt and 196.6 tok/s decode versus 8,744 and 219.9 for Q4_K_M, with perplexity 3,864 vs 3,615.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
rpcreative-writing
User reports Gemma 4 E4B at ~33–34 tok/s on a GTX 1070 8GB.
Setup is LM Studio with 4096 context, thinking off, temperature 1.0, full GPU offload, Flash Attention and KV cache on GPU, using around 5 GB VRAM.
User settled on Gemma 4 E4B for short NPC dialogue and roleplay after testing several other models that produced incoherent replies or poor quality.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
long-contextcoding
User reports Gemma-4-31B at 36.5 tok/s on a single RTX 3090 with full big context, using mainline llama.cpp built from source with -DGGML_CUDA_FA_ALL_QUANTS=ON.
Setup is llama.cpp with Q4_K_S weights and q5_0/q4_1 KV cache on one 24 GB card; the stock prebuilt image only reaches ~12 tok/s because the flag is off by default.
The post compares engines: beellama.cpp with DFlash speculative decoding reaches 54-193 tok/s depending on workload (131-200K context), vLLM gets 86-145 tok/s on dual cards at 262K, and ik_llama is fast but context-capped near 24K. Quality on an 8-pack was 109/150 with reasoning off and 114/150 with reasoning on.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Gemma-4-26B-A4B 4-bit at 52.2 tok/s decode on a base Mac mini M6 with 32 GB unified memory.
Setup is SwiftLM with MLX 4-bit weights, full GPU offload, peak GPU memory 19.5 GB; the longest prompt that passed was 80.7K tokens at 622 tok/s prefill and 24.3 tok/s decode.
Also benchmarks Qwen3.6-35B-A3B 4-bit at 46.7 tok/s (GPU) and 13.2 tok/s (--stream-experts), Qwen3.8-27B 4-bit dense at 9.3 tok/s with 200 tok/s prefill, and Gemma-4-26B-A4B 8-bit at 8.8 tok/s with --stream-experts. An A/B against the prior revision shows 998 tok/s prefill at ~2.3K tokens versus 508 tok/s with swap, and 914 tok/s at ~9.5K tokens where the earlier build aborted on swap.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
codingagenticsummarization
User reports Gemma 4 E4B at around 8.2 tok/s on an M1 MacBook Air with 16GB unified memory.
Setup is Unsloth Studio with a 4-bit MLX model and a 131.1k context window.
The user says this speed is fine for chatbot use and asks for suggestions of 7B or 8B MLX models for coding and background tasks.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports gemma-4-31B at 43 tok/s decode on 2× RX 7900 XT with tensor parallelism.
Setup is vLLM with w4a16 weights, TP=2, both cards drawing 265 W at the same time.
The machine is a VFIO virtual machine with no P2P and cross-die PCIe 3.0, measured with no PCIe atomics. A 26B MoE reaches 108 tok/s at short context on the same pair.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
summarization
User reports Gemma 4 E2B at ~45 tokens/s single-task on three M4 Mac minis with 16GB RAM, each handling around 10 parallel tasks at ~6 tokens/s each.
Setup is local inference on Apple Silicon; no engine, quant, or context length stated.
The user asks whether one Mac Studio or more Mac minis would give better throughput for parallel labeling and categorization.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
math
User reports Gemma 4 12B at about 14 tokens/second on an M2 MacBook Pro with 32GB of memory.
Setup is the Nativ app with an MLX 4-bit quantized model; the user says the prompt takes 50 seconds to read.
The user finds the answers excellent but the model dog-slow, and asks for a faster model for textbook math questions.