Gemma 3 27B at 4-bit generates a median of 13.0 tokens per second on the AMD Strix Halo 128GB, the hardware it is most run on, from one plain run (community reports, as of 30 Oct 2026).
As of 30 Oct 2026, Gemma 3 27B at 4-bit on the hardware it is most run on, with the median of plain runs (one device, one request, no speculative decoding, the whole model in its memory):
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Gemma 3 1B at 4.47 t/s on a 2015 Nexus 5X phone, running on its CPU with the model held in 2 GB of RAM.
Setup is llama.cpp baked into a trimmed Linux 3.10 kernel and tiny rootfs booted straight into RAM via fastboot, with no Android.
The user also mentions running Qwen 2.5 and Llama 3.2 on the same device.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports a paper replacing Gemma 3 270M's embedding table with a frozen 16 MB 512-bit code table, with a ternary BitNet b1.58 body giving a 46 MB model that generates 615 to 740 tokens per second on a laptop CPU in 62 MB of memory.
The paper states this is five to six times the speed of the float model's 8-bit GGUF in llama.cpp at a seventh of the memory.
The frozen table wins by 1.7 to 2.3 nats on rare words and accepts 10,000 new words after training at a 3% cost where the learned table collapses by 43%.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Gemma 3 4B at 132.52 t/s on an RTX 6000 Pro Blackwell, a 3.34x speedup over Ollama.
Setup is vLLM (dev build) with BF16 weights and MTP speculative decoding using 3 draft tokens, single request, ~512-token prompt and 64 output tokens, averaged over 3 runs.
The user also tested Qwen3-VL 8B (Abliterated) with 1 MTP layer, noting Gemma 3's 3 MTP layers explain its larger gain; going beyond the MTP layer count as draft tokens does not help.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports 294.1 t/s generation.
Source evidence (verbatim):
Source: https://dev.to/sikamikanikobg/vllm-vs-llamacpp-vs-ollama-what-happens-when-your-model-doesnt-fit-in-24gb-vram-56eb
# vLLM vs llama.cpp vs Ollama: What Happens When Your Model Doesn't Fit in 24GB VRAM
Hardware: NVIDIA RTX 3090 24GB, 128GB system RAM. Concurrency 1 decode tok/s (llama.cpp / vLLM): Gemma 3 1B 294.1 / 235.5. Qwen3-Coder 30B-A3B 157.2 / 172.0 (vLLM AWQ). Gemma 4 26B-A4B 118.8 / 133.8. Ollama at concurrency 1: 125.6, 129.3, 84.5. Single-stream steady-state decode on the longest prompt tier with models that exceed VRAM (llama.cpp -ngl 12 for gpt-oss-120b): GPT-OSS-120B 7.65 tok/s llama.cpp, 7.6 Ollama. GLM-4.5-Air 4.58 llama.cpp, 4.59 Ollama. vLLM ran out of memory on both. Engine versions and context length not stated.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports 33 t/s generation.
Source evidence (verbatim):
Source: https://famstack.dev/guides/mlx-vs-gguf-part-2-isolating-variables/
# Mac Studio M1Max64GB native generation after Gemma3QAT MLXdtype conversion
Original author article March20,2026. Measured Mac Studio AppleM1 Max 64GB unified RAM,24GPUcores; CPUcorecount unknown. Gemma3-12B-it QAT4-bit MLX in LMStudio: author states generation increases from 28 to 33 tok/s after converting nonquantized dtype frombf16 tofloat16. No averaging or calculatedspeed used. Tested sourcequant repository verified https://huggingface.co/mlx-community/gemma-3-12b-it-qat-4bit ; convertedartifact generated locally with mlx_lm.convert --dtype float16, not a separate verifiedHFartifact. Testedrevision/runtimeversion/prompt/outputcounts/power/KVquant/speculation/residentRAM unknown. Separate reportedprefilltimes at655tokens6.2s→3.7s;8K114.4s→68.9s; do not derive nativeprefillT/s from approximate8K. Other charts report effectiveoutputtokens/totalrequesttime includingprefill, not generation; keep separate. Source: https://famstack.dev/guides/mlx-vs-gguf-part-2-isolating-variables/ . Rawlogs linkGitHub and were not fetched under sourceaccessstop. CurrentHFcard contains unrelated-looking multimodalmetadata; no testedhistoricalrevision established, so preserve artifact caveat. Selected run: locally converted float16 nonquantized dtype artifact, native generation 33 tok/s. Baseline before conversion is a separate run at 28 tok/s. Prompt processing speed unknown; do not derive it from prefill times.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Gemma 3 27B at 8.9 t/s generation and 11.2 t/s prompt processing on a Mac mini M6 with 32GB unified memory.
Setup is MLX-LM with the mlx-community 4-bit quantized model, run through the mlx_lm.generate CLI with a 20-token prompt and 500 output tokens.
Peak memory was 16.296 GB. The rates are native MLX-LM phase timings from a single displayed run, not a benchmark average.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Gemma 3 27B at 3.9 t/s generation and 104 t/s prompt processing on an AMD Strix Halo 128GB machine.
Setup is llama.cpp with Vulkan on the Radeon 8060S iGPU, BF16 weights, 6 threads and maximum GPU offload (-ngl 999).
The figures come from separate synthetic llama-bench phase tests, PP512 and TG128, not an end-to-end rate; the source gives rounded rates without raw durations or repeat counts.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Gemma 3 27B IT QAT at 13 t/s generation and 382.5 t/s prompt processing on an AMD Strix Halo 128GB machine.
Setup is llama.cpp with Q4_0 and f16 K and V cache, flash attention on, batch 2048, one GPU with full layer offload.
The stored figures are the quiet baseline cell, PP 382.490359 ± 1.929762 and TG 12.962264 ± 0.046976 t/s; other campaign cells reach up to 437.558764 t/s PP and 13.436544 t/s TG.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Gemma 3 4B at 150.5 t/s generation and 5760.7 t/s prompt processing on an M5 Max 48GB.
Setup is llama.cpp build 11146 with Metal, Q4_K_M weights, n_gpu_layers=99 and flash_attention=true.
The figures are separate llama-bench microtests, tg128 and pp512, not one 512-input/128-output request or server concurrency throughput; context length is unspecified.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Gemma 3 27B at 29.9 t/s generation and 751 t/s prompt processing on an M5 Max with 48GB unified memory.
Setup is llama.cpp build 11146 with Metal, Q4_K_M weights, n_gpu_layers=99 and flash attention enabled.
The figures are separate llama-bench microtests, tg128 and pp512, not one 512-input/128-output request or server concurrency throughput; context length is unspecified.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Gemma 3 1B at 294.9 t/s generation and 16750.2 t/s prompt processing on an M5 Max 48GB.
Setup is llama.cpp build 11146 with Metal, Q4_K_M weights, n_gpu_layers=99, flash_attention=true and threads=0.
The figures are separate llama-bench microtests, tg128 and pp512, not one 512-input/128-output request or server concurrency throughput; context length is unspecified.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Gemma 3 12B at 61.5 t/s generation and 1721.5 t/s prompt processing on an M5 Max 48GB.
Setup is llama.cpp build 11146 with Metal, Q4_K_M weights, n_gpu_layers=99 and flash attention enabled.
The figures are separate llama-bench microtests, tg128 and pp512, not one 512-input/128-output request or server concurrency throughput; context length is unspecified.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Gemma 3 4B 4-bit at 176.2 t/s generation and 5550.3 t/s prompt processing on an M5 Max 48GB.
Setup is MLX with 4-bit weights and unquantized KV cache on one Mac Studio, batch size 1, no speculative decoding.
The headline figures are pp512 and tg128, not a best-of sweep or an average across prompt lengths; the source also records pp128=2589.02, pp256=3725.55 and tg256=178.13 t/s.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Gemma 3 27B at 31.8 t/s generation and 854.4 t/s prompt processing on an M5 Max with 48GB unified RAM.
Setup is MLX with 4-bit weights, unquantized KV cache, batch size 1, and a 2048-token prefill step.
The figures are arithmetic means of five trials from mlx-lm 0.31.3, with prompt processing including first-token work; peak MLX allocator memory was 16.69 GB.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Gemma 3 1B at 374.4 t/s generation and 17261.9 t/s prompt processing on an M5 Max 48GB.
Setup is MLX with 4-bit weights and unquantized KV cache, batch size 1, one model invocation per process.
The headline figures are pp512 and tg128 from a five-trial mean, not a best-of sweep or average across prompt lengths; the benchmark is synthetic text-token throughput, not end-to-end request throughput.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Gemma 3 12B 4-bit at 66.9 t/s generation and 1834.9 t/s prompt processing on an M5 Max 48GB.
Setup is MLX with 4-bit weights and unquantized KV cache on one Mac Studio, batch size 1, greedy sampling, no speculative decoding.
The figures are native engine phase rates: prompt speed includes first-token work, and the retained tg128 and pp512 come from the (512,128) run, not a best-of sweep or average.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Gemma 3 270M at 321 tok/s on a Ryzen 7 7700 CPU, up from 132 tok/s, by replacing the output head with a two-level prefix head that sorts the vocabulary by byte order.
Setup is a custom prefix head with 2,048 groups of 128 tokens, AVX2 only, one token at a time.
The same grouping gives 27-33% lower KL to the full head than Google's Gemma 4 token ordering across Gemma 3 270M/1B, Gemma 4 E2B, Qwen3-0.6B and gpt-oss-20b. Caveats: one text domain, no downstream evals, no speculative-decoding acceptance numbers.
Individual observations from this page, not expected speeds or a ranking. Select a model to open its report, with the source and offloading conditions. Reported context may be a limit rather than actual input length.
The hardware Gemma 3 27B at 4-bit is most run on: AMD Strix Halo 128GB (13.0 t/s median, one run); M5 Max 48GB (29.9 t/s median, one run); M6 32GB (8.9 t/s median, one run). Medians of plain runs, as of 30 Oct 2026.
Which inference engines do people use for Gemma 3?
Reports of Gemma 3 name llama.cpp (7 reports), MLX (4 reports), LM Studio (1 report), vLLM (1 report), as of 30 Oct 2026.
Speed by hardware on Gemma 3 27B at 4-bit
The size and quant level the most devices have plain runs of: one device, one request, no speculative decoding, and the whole model in the device's memory. Every speed in this table is the same model.