Local model performance on your hardware

generation:

20.0 tokens/s

quant:

Q5 (GGUF)

User reports Gemma 4 31B Q5 at 20 t/s on two RTX 3060 12GB cards in llama.cpp, up from 14 t/s with layer splitting, a 42% gain. One card runs on PCIe x16 and the other on a PCIe 3.0 x1 riser; the user notes the fit and context are tight and that testing was limited while awaiting a PSU upgrade. On a rented 2x RTX 5060 Ti 16GB machine, L3.1 Nemotron Sunfall 71B IQ3_XS at 2k context with flash attention gave 20.7 t/s generation and 464.4 t/s prompt processing under tensor split, versus 11.7 t/s and 591.8 t/s under layer split. PCIe traffic during decode measured roughly 90-110 MB/s per direction, with load spikes above 4,000 MB/s.

generation:

20.0 tokens/s

prompt processing (prefill):

1100.0 tokens/s

quant:

QAT

kv:

Q8

User reports Gemma 4 12B QAT at 20 t/s decode and 1100 t/s prefill on a single Intel Arc A770 16GB. Setup is llama.cpp with QAT weights, Q8 KV cache and a Q4 MTP head, giving 128k context with about 5GB VRAM headroom; the card is in a 13-year-old Haswell desktop with under 6GB RAM headroom. Four models run together at 14-15GB VRAM via llama-swap, with LiteLLM routing prompts through Laya to thinking and non-thinking aliases of the 12B. The user notes SYCL/llama.cpp lacks Alchemist optimizations, making the A770 the bottleneck, and that GPU-resident models still use 4-5GB of system RAM as compute buffer.

generation:

66-71 tokens/s

prompt processing (prefill):

3200.0 tokens/s

quant:

Q8_0 (GGUF)

User reports gemma-4-26B-A4B Q8_0 (26.9 GB) on a Radeon AI PRO R9700 32GB in a Minisforum DEG2 eGPU over USB4, at about 3,200 t/s prefill (pp512) and 66-71 t/s decode, versus 1,387 t/s and 46.6 t/s on the Strix Halo iGPU. Setup is llama.cpp Vulkan/RADV with -ngl 99 -fa 1, a 256 MB BAR, the card pinned to profile_peak, and both GGML_VK_DISABLE_HOST_VISIBLE_VIDMEM=1 and GGML_VK_MMID_F16B=0 set. A 10 h soak ran 6,724/6,724 requests with decode flat from 66.2 to 63.7 tok/s between the first and last quarter. Big model loads wedge the card at the default power level unless profile_peak is set, which idles at 137 W; a wrapper switching it around jobs passed 22/22 image jobs. A full 32 GB BAR works only with the dock powered on at boot, where 27B Q8_0 decode is 17.9 tok/s at a 1k context.

generation:

2.3 tokens/s

quant:

Q5_K_M (GGUF)

kv:

Q8

User asks about a budget upgrade to run Gemma 4 31B Q5_K_M at 16K context and at least 15 t/s, currently getting 2.3 t/s generation (2.7 t/s with MTP) on an RTX 3060 12GB with 47% GPU / 53% CPU offload. Setup is Ollama/llama.cpp with Q5_K_M weights and Q8 KV cache on an i5-14400F with 32GB DDR4; the 243-token prompt was processed with the model partly offloaded to system RAM. User says IQ3_M drops quality without adding speed, and is weighing an RTX 5060 Ti 16GB plus the 3060 on a PCIe 3.0 x1 riser, dual Tesla P100s on an X79/X99 Xeon system, or a cheaper AMD 16GB card.

generation:

137.7 tokens/s

prompt processing (prefill):

5951.0 tokens/s

quant:

UD-IQ4_XS (GGUF)

kv:

K q8_0, V q4_0

User reports 137.7 t/s generation and 5951 t/s prompt processing. Source evidence (verbatim): Source: https://algollabs.com/blog/gemma4-bfcl # Gemma 4 26B on consumer-grade GPU: Benchmarks, Build Pain, and a Week of Daily Use Hardware: NVIDIA RTX 5070 Ti 16 GB GDDR7, AMD Ryzen 9 9950X3D, 64 GB DDR5, Fedora 43, kernel 6.18. Engine: llama.cpp build 9024 built from source with GGML_CUDA_FA_ALL_QUANTS=ON. Model: Gemma 4 26B A4B Instruct (26B total, 3.8B active), unsloth/gemma-4-26B-A4B-it-GGUF UD-IQ4_XS, 12.65 GiB, full GPU offload (31 of 31 layers), flash attention on, asymmetric KV cache (K q8_0, V q4_0). llama-bench, no speculative decoding or MTP, single stream: 5,951 tokens/s prompt processing (pp2048) and 137.7 tokens/s token generation (tg64).

generation:

28.2 tokens/s

User reports Gemma 4 12B at 28.22 t/s on the RAID 1 explanation task, run on an RTX A2000 12GB in a Lenovo ThinkCentre M920x Tiny with an i5-9500T and 32 GB RAM. Setup is Ollama on Ubuntu 26.04.1 with Open WebUI in Docker; the model sat fully on the GPU and drew up to 82 W during the run. The Python task was not verified for this model. The same machine ran seven other models: DeepSeek Coder V2 16B was fastest at 45.86 and 46.03 t/s, Qwen3 8B at 30.80 and 31.25 t/s, Qwen3.5 9B at 23.23 and 22.94 t/s, Gemma 3 12B at 20.58 and 20.33 t/s, Ministral 3 14B at 19.82 and 20.29 t/s, Qwen3 14B at 16.60 and 18.53 t/s, and a Qwen of about 27B fell to 1.45 and 1.99 t/s with CPU offload. Idle draw was about 15 W.

generation:

60.0 tokens/s

prompt processing (prefill):

1100.0 tokens/s

quant:

QAT

User reports Gemma 4 31B IT QAT at over 60 t/s generation and 1100 t/s prompt processing on dual RTX 5060 Ti 16GB cards. Setup is llama.cpp 0.7.0 with QAT weights, tensor split and MTP enabled across the two cards. Earlier runs on the same setup gave 40 t/s gen and 980 t/s pp, and 17 t/s gen with 970 t/s pp at 32K for Q6_K. Qwen3.8 27B runs at around 70 t/s with MTP and 100 t/s when generating code. The user uses the models for roleplay in SillyTavern.

generation:

46.0 tokens/s

quant:

QAT

User reports Gemma 4 31B QAT at 46 tok/s on an RTX 4090. Setup is Glyd 0.31 with Google's QAT 4-bit checkpoint (4.5 bits per weight) at 32K context, using 21.8 of 24.6 GB. With Google's draft model, chat prompts reach 78-80 tok/s and predictable text up to 176 tok/s. A 30K-token prompt took 20 s. On an RTX 3090 the same model runs at 26 tok/s; a 12B QAT on an RTX 4080 SUPER runs at 82.5 tok/s plain and 109 tok/s with the draft model. Quality vs Google's QAT weights: mean KLD 0.018, median 0.003, top-1 96%, GSM8K 72/75, MMLU-Pro 70/75.

generation:

102.0 tokens/s

quant:

QAT UD-Q4_K_XL (GGUF)

User reports Gemma 4 26B-A4B QAT at 102/102/127 tok/s (Swedish prose / English prose / JSON) on a Strix Halo 128GB box, about 45% faster than the regular UD-Q4_K_M at 70/71/90 tok/s. Setup is llama.cpp Vulkan/RADV build 11235 with QAT UD-Q4_K_XL and the matching QAT MTP drafter, single request, -ctxcp 0, using 17.7 GB versus 20.9 GB for Q4_K_M. Lower K-quants (Q3_K_XL, IQ3_S) gave no speed gain. Nemotron 3.5 Lightning 30B-A3B and Qwen3.6-35B-A3B made factual errors in 50-80% of summaries versus about 10% for Gemma. Strata v0.1.44.1 with Qwen3.8-Flash-Next IQ3_S decodes at 50.9 tok/s shallow context and passed 256k needle tests up to 242k tokens with prefill flat near 1,200 tok/s.

generation:

60.0 tokens/s

User reports Gemma 4 26B MoE at 60+ tok/s on an RTX 4060 Laptop, up from 40 tok/s with standard llama.cpp. Setup is llama.cpp with MTP speculative decoding plus a custom DeepSeek-inspired hot-expert caching scheme that keeps frequently used experts on the GPU. The user asks for further performance ideas without changing the model.

generation:

170.0 tokens/s

quant:

QAT

User reports Gemma 4 26B-A4B QAT at about 170 tokens a second on one RTX 4090, with a full two-call pass taking 5 seconds. Setup is Ollama with QAT weights, one card, 96 GB system RAM; vLLM was faster at 3.5 seconds per pass but held 94% of the card against Ollama's 70%. Sending both calls of a pass together cut it to 3.8 seconds on Ollama and 2.8 seconds on vLLM. MedGemma 27B Q4_K_M took 18 seconds per pass. The user asks about speculative decoding, other engines, other models, and CPU offload.

generation:

59.4 tokens/s

prompt processing (prefill):

3633.8 tokens/s

quant:

Q4_K_M (GGUF)

User reports 59.42 t/s generation and 3633.84 t/s prompt processing. Source evidence (verbatim): Source: https://blog.majid.info/llama-bench/dgx1.txt # NVIDIA DGX Spark 128GB / GB10: original Gemma4 E4B Q4_K_M llama-bench log First-hand author log linked from https://blog.majid.info/llama-bench/ (April 13, 2026 article). Measured hardware: NVIDIA DGX Spark 128GB / GB10. Gemma4 E4B Q4_K_M GGUF, llama.cpp ff5ef8278 (8763), backend CUDA. pp512 3633.84 ±201.59 tokens/s; tg128 59.42 ±0.96 tokens/s. One GB10, ngl99; 122502 MiB runtime device VRAM visibility. 128GB is unified system capacity, not dedicated VRAM or resident use. Command names --hf-repo unsloth/gemma-4-E4B-it-GGUF; exact tested quantization repository https://huggingface.co/unsloth/gemma-4-E4B-it-GGUF ; tested revision unknown. Log reports 4.62 GiB model size and 7.52B counted parameters; size is not resident memory. Context depth, KV quantization, measured power, driver version and explicit speculation/concurrency declarations unreported. Raw source: https://blog.majid.info/llama-bench/dgx1.txt .

generation:

165.6 tokens/s

prompt processing (prefill):

9555.9 tokens/s

quant:

Q4_K_M (GGUF)

User reports 165.64 t/s generation and 9555.89 t/s prompt processing. Source evidence (verbatim): Source: https://blog.majid.info/llama-bench/golgotha.txt # NVIDIA GeForce RTX 5090 Laptop GPU 24GB: original Gemma4 E4B Q4_K_M llama-bench log First-hand author log linked from https://blog.majid.info/llama-bench/ (April 13, 2026 article). Measured hardware: NVIDIA GeForce RTX 5090 Laptop GPU 24GB. Gemma4 E4B Q4_K_M GGUF, llama.cpp e3f542d8 (4236), backend CUDA. pp512 9555.89 ±101.42 tokens/s; tg128 165.64 ±0.19 tokens/s. One device, ngl -1; 23983 MiB visible VRAM, not resident use. Laptop GPU, not desktop RTX5090. CPU and system RAM unknown. Command names --hf-repo unsloth/gemma-4-E4B-it-GGUF; exact tested quantization repository https://huggingface.co/unsloth/gemma-4-E4B-it-GGUF ; tested revision unknown. Log reports 4.62 GiB model size and 7.52B counted parameters; size is not resident memory. Context depth, KV quantization, measured power, driver version and explicit speculation/concurrency declarations unreported. Raw source: https://blog.majid.info/llama-bench/golgotha.txt .

generation:

51.9 tokens/s

prompt processing (prefill):

1831.8 tokens/s

quant:

Q4_K_M (GGUF)

User reports 51.92 t/s generation and 1831.78 t/s prompt processing. Source evidence (verbatim): Source: https://blog.majid.info/llama-bench/zanzibar.txt # NVIDIA RTX A2000 12GB: original Gemma4 E4B Q4_K_M llama-bench log First-hand author log linked from https://blog.majid.info/llama-bench/ (April 13, 2026 article). Measured hardware: NVIDIA RTX A2000 12GB. Gemma4 E4B Q4_K_M GGUF, llama.cpp 82764d8f4 (8770), backend CUDA. pp512 1831.78 ±13.79 tokens/s; tg128 51.92 ±0.39 tokens/s. One device, ngl99; 11906 MiB visible VRAM, not resident use. CPU and system RAM unknown. Command names --hf-repo unsloth/gemma-4-E4B-it-GGUF; exact tested quantization repository https://huggingface.co/unsloth/gemma-4-E4B-it-GGUF ; tested revision unknown. Log reports 4.62 GiB model size and 7.52B counted parameters; size is not resident memory. Context depth, KV quantization, measured power, driver version and explicit speculation/concurrency declarations unreported. Raw source: https://blog.majid.info/llama-bench/zanzibar.txt .

generation:

21.2 tokens/s

prompt processing (prefill):

568.5 tokens/s

quant:

Q4_K_M (GGUF)

User reports 21.16 t/s generation and 568.54 t/s prompt processing. Source evidence (verbatim): Source: https://blog.majid.info/llama-bench/ai-x1-pro.txt # AMD Ryzen AI 9 HX 370 / Radeon 890M, 96GB system RAM: original Gemma4 E4B Q4_K_M llama-bench log First-hand author log linked from https://blog.majid.info/llama-bench/ (April 13, 2026 article). Measured hardware: AMD Ryzen AI 9 HX 370 / Radeon 890M, 96GB system RAM. Gemma4 E4B Q4_K_M GGUF, llama.cpp 69c28f1 (1), backend ROCm. pp512 568.54 ±1.96 tokens/s; tg128 21.16 ±0.03 tokens/s. gfx1150; ngl99; runtime reports 47072 MiB device VRAM visibility; integrated unified memory, not dedicated VRAM or measured resident use. Command names --hf-repo unsloth/gemma-4-E4B-it-GGUF; exact tested quantization repository https://huggingface.co/unsloth/gemma-4-E4B-it-GGUF ; tested revision unknown. Log reports 4.62 GiB model size and 7.52B counted parameters; size is not resident memory. Context depth, KV quantization, measured power, driver version and explicit speculation/concurrency declarations unreported. Raw source: https://blog.majid.info/llama-bench/ai-x1-pro.txt .

generation:

69.7 tokens/s

prompt processing (prefill):

1172.9 tokens/s

quant:

Q4_K_M (GGUF)

User reports 69.73 t/s generation and 1172.93 t/s prompt processing. Source evidence (verbatim): Source: https://blog.majid.info/llama-bench/utumno.txt # Mac Studio M1 Ultra 128GB: original Gemma4 E4B Q4_K_M llama-bench log First-hand author log linked from https://blog.majid.info/llama-bench/ (April 13, 2026 article). Measured hardware: Mac Studio M1 Ultra 128GB. Gemma4 E4B Q4_K_M GGUF, llama.cpp ff5ef8278 (8763), backend Metal/BLAS. pp512 1172.93 ±0.36 tokens/s; tg128 69.73 ±0.19 tokens/s. 16 CPU threads; recommendedMaxWorkingSetSize 103079.22 MB is a runtime recommendation, not resident memory. Unified system RAM 128GB, GPU core count unknown. Command names --hf-repo unsloth/gemma-4-E4B-it-GGUF; exact tested quantization repository https://huggingface.co/unsloth/gemma-4-E4B-it-GGUF ; tested revision unknown. Log reports 4.62 GiB model size and 7.52B counted parameters; size is not resident memory. Context depth, KV quantization, measured power, driver version and explicit speculation/concurrency declarations unreported. Raw source: https://blog.majid.info/llama-bench/utumno.txt .

generation:

69.7 tokens/s

prompt processing (prefill):

6757.3 tokens/s

quant:

Q8_0 (GGUF)

User reports Gemma 4 E4B at 69.66 t/s generation and 6757.3 t/s prompt processing on an NVIDIA RTX 4070 Ti. Setup is llama.cpp with Q8_0 and requested GPU layers 999, a direct full-offload run on a 12GB card. The figures come from a synthetic pp512/tg128 test at zero context depth, so they do not measure generation after a filled context.

generation:

17.2 tokens/s

prompt processing (prefill):

63.7 tokens/s

quant:

4bit (MLX)

User reports Gemma 4 31B at 17.2 t/s generation and 63.7 t/s prompt processing on a Mac Studio M2 Max with 64 GB unified memory. Setup is mlx-vlm 0.4.4 with 4-bit affine group-64 weights and MLX 0.31.1 on one GPU, running the text-only long-text request with 55 prompt tokens and 500 generated tokens. The 29.9-second whole generate call is not TTFT, and the 63.7 t/s prompt figure is a single request rather than a llama-bench pairing.

generation:

87.8 tokens/s

prompt processing (prefill):

2595.3 tokens/s

quant:

MXFP4_MOE (GGUF)

flash attention:

on

mtp (multi-token prediction):

off

User reports Gemma 4 26B-A4B at 87.8 t/s generation and 2595.3 t/s prompt processing on an M3 Ultra Mac Studio with 512 GB of unified memory. Setup is llama.cpp build b8783 with the MXFP4_MOE quantization, Metal backend, F16 K and V cache, and -ngl 999, running one GPU. The figures come from separate llama-bench phases: PP2048 at 2595.28 t/s and TG32 at 87.81 t/s, each the mean of five repetitions at context depth 0.

generation:

84.9 tokens/s

prompt processing (prefill):

2634.8 tokens/s

quant:

UD-Q8_K_XL (GGUF)

flash attention:

on

mtp (multi-token prediction):

off

User reports Gemma 4 26B-A4B at 84.9 t/s generation and 2634.8 t/s prompt processing on an M3 Ultra Mac Studio with 512 GB of unified memory. Setup is llama.cpp build b8783 with the UD-Q8_K_XL quant, Metal backend, F16 K and V cache, and -ngl 999 offloading to one GPU. The figures come from llama-bench with -p 2048 -n 32 -ub 2048 -fa 1, five repetitions, batch 2048 and context depth 0; the article's llama-server, quality, VLM and long-context tests are separate.