Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Gemma 4 26B-A4B at 32 t/s generation and 657 t/s prompt processing on an M4 32GB Mac mini.
Setup is Ollama 0.30.5 with GGUF Q4_K_M and num_ctx=4096, 100% GPU offload, 17.37 GB loaded memory including KV cache.
Both figures are three-run averages; the 400-character prompt instruction is not an output-token count, and 4096 is the configured context, not measured input length.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Gemma 4 12B at 12.5 t/s generation and 236 t/s prompt processing on an Apple Mac mini M4 with 32 GB unified memory.
Setup is Ollama 0.30.5 with GGUF Q4_K_M and num_ctx=4096, reporting 100% GPU offload and 8.05 GB loaded memory including KV cache.
Both figures are three-run averages; the 4096 is the configured context, not measured input length, and the prompt asks for at least 400 Chinese characters.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Gemma 4 E4B at 29.9 t/s generation and 566 t/s prompt processing on an Apple Mac mini M4 with 32 GB unified memory.
Setup is Ollama 0.30.5 with GGUF Q4_K_M at num_ctx=4096, with 100% GPU offload and 9.55 GB loaded memory including KV cache.
The figures are source-reported three-run averages, with decode from eval_count/eval_duration and prompt processing from prompt_eval_count/prompt_eval_duration, excluding client and model-loading overhead.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Gemma 4 E2B at 55 t/s generation and 1089 t/s prompt processing on an Apple Mac mini M4 with 32 GB unified memory.
Setup is Ollama 0.30.5 with GGUF Q4_K_M at num_ctx=4096 and 100% GPU offload, using the Ollama library gemma4:e2b tag.
The figures are a three-run average (54.97 t/s decode, 1089 t/s prompt) from a fixed Chinese long-form prompt; reported loaded memory is 7.06 GB including KV cache.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Gemma 4 12B at 8.7 t/s generation and 110.4 t/s prompt processing on an M4 32GB Mac mini.
Setup is Ollama 0.30.5 with Q4_K_M and q8_0 KV cache, num_ctx=8192, flash attention enabled, one card.
The row is the first long-context tier: 5713 actual input tokens and 512 actual output tokens, not a full 8192-token prompt or an average. The source TTFT estimate is 64.5 s and request wall time 123.4 s.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Gemma 4 E4B at 16.7 t/s generation and 245.5 t/s prompt processing on an M4 32GB Mac mini.
Setup is Ollama 0.30.5 with Q4_K_M and q8_0 KV cache at 8192 context, one Mac mini M4 with 32 GB unified memory.
The figure is a single long-context row with 5709 actual input and 505 actual output tokens, not an average or a full 8192-token prompt; the source TTFT estimate is 23.53 s.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Edge0 at 28 t/s on a single Apple Mac mini M4 Pro with 24 GB unified memory.
Setup is the Edge0 MLX backend with int4 affine group-64 weights, prediction-as-routing staged decode, whole-layer prefill and SSD-streamed experts, using a shared LRU of 64 expert bundles.
The figure is the paper's single-device warm decode result for the released Edge0 8B K=8 low-memory profile, a median over three paired rounds; a separate LRU1024 variant reaches 31.8 t/s with about 0.9 GiB more memory.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Edge0 at 20.4 t/s on a single Apple Mac mini M4 Pro with 24 GB unified memory.
Setup is the Edge0 MLX pipeline with int4 affine group-64 weights, a K=4 production profile, and SSD-streamed experts; allocated context is not reported.
The figure is the paper's single-device warm decode result, a median over three paired rounds, and the timed output count is absent. The same paper reports prefill at 113 t/s cold and 140 t/s warm.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
visionmultilingual
User reports MiniCPM-V 4.7 35B-A3B at 17+ tok/s on an iPad Pro M4 16GB, with 13+ tok/s on an iPhone 17 Pro Max and about 3GB memory usage.
Setup details are not stated; the post highlights the model's LES (Lookahead Expert Scheduling) technique for MoE models, claimed to give up to 5x speedup with identical outputs.
The post also cites 90.4 OCRBench with 16x visual compression, 34 multimodal benchmarks, a new 1B lightweight VLM, Self-Replay Fine-Tuning, and support for Apple, Intel, AMD, Arm and Rockchip.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Mellum2.1 at 64 tokens/s at 1K context on an M4 16GB Mac.
Setup is MLX with an NVFP4 conversion, 6.84 GB on disk.
The user converted JetBrains' Mellum2.1 to MLX NVFP4 and asks for feedback on real tasks.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8-Flash-Next at 12 t/s generation on an M4 Max 128GB, with the source's 128K benchmark point given as 11.98 tokens/s.
Setup is llama.cpp with UD-Q4_K_XL and f16 K/V cache, all layers offloaded, flash attention on, batch 1024 and microbatch 256.
The figure is generation after cache prefilling, not prompt processing or TTFT; the source's f16 KV series is 16.62 t/s at 64K, 14.93 at 81K and 11.98 at 128K.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.5-122B-A10B at 15.6 t/s generation on an M4 Max 128GB.
Setup is llama.cpp with Q4_K_M and f16 K/V cache, all layers offloaded, batch 1024 and flash attention on.
The figure is the 128K benchmark point after cache prefilling, not newly processed input tokens; the source's f16 KV series is 24.51 t/s at 64K, 22.25 at 81K and 15.59 at 128K, with a reported Metal/VRAM peak of 75.16 GB at 128K.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8 27B at 11.4 t/s generation and 96.8 t/s prompt on a Mac mini M4 Pro with 24GB unified memory.
Setup is llama.cpp b10488 with Metal, Flash Attention on, all layers offloaded, and Q4_K_M quant.
The figures come from separate llama-bench pp2048 and tg128 tests, mean of three repetitions, not one request; run-to-run variance is claimed under 3%.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Llama 3.2 3B at 46.7 t/s on an M4 16GB Mac mini running macOS 15.3.1.
Setup is Ollama 0.31.2 with llama.cpp and Metal, Q4_K_M, default settings, fresh runtime launch.
The figure is the median of three requests after one discarded warm-up, with a source-reported max-minus-min spread of 0.1 t/s; configured context, KV precision and speculative decoding are unknown.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Llama 3.1 8B at 21 t/s generation and 229.2 t/s prompt processing on an M4 16GB Mac mini.
Setup is llama.cpp 0.4.0 with Q4_K_M and Metal, all layers on GPU; context, KV precision and Flash Attention are unknown.
The figures come from separate llama-bench tests, pp512 at 229.23 ±0.08 t/s and tg128 at 20.95 ±0.02 t/s over three runs, not a combined 512-input/128-output completion, with 10.29 GiB RAM already in use.
M4 16GB · Evrmind custom Metal binary / llama.cpp CLI
generation:
8.2 tokens/s
prompt processing (prefill):
9.1 tokens/s
quant:
EVR-1 custom compression (~3-bit weights) (GGUF)
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Llama 3.1 8B at 8.2 t/s generation and 9.1 t/s prompt processing on an M4 16GB Mac mini.
Setup is Evrmind's custom Metal binary with EVR-1 compression at roughly 3 bits per weight and full GPU offload.
The figures come from separate pp512 and tg128 phase tests, not a paired 512-input/128-output run; the source table lists 9.11 and 8.16 t/s.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Grug-12B at 50.9 t/s generation and 511.9 t/s prompt processing on an M4 Max with 64 GB unified memory.
Setup is oMLX with MLX affine 4-bit, group 64, at a pp8192/tg128 workload, with 8.26 GB peak memory.
The same row lists TTFT 16003.6 ms, TPOT 19.80 ms and E2E 18.540 s; the 8192-token prompt is a workload label, not a configured maximum context.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Grug-35B-A3B at 103.9 t/s generation and 1807.2 t/s prompt processing on an M4 Max 64GB.
Setup is oMLX with MLX affine 4-bit group 64, gate modules at 8-bit, at 1024 prompt and 128 generated tokens.
The run is single-request, with 566.6 ms TTFT, 9.70 ms TPOT, 1.805 s end-to-end and 19.35 GB peak memory; warm-up, repetition count and KV precision are not stated.
M4 Max 128GB · Apple Core AI (coreai-pipelined GPU)
generation:
26.7 tokens/s
prompt processing (prefill):
269.0 tokens/s
quant:
int4hu block32 symmetric
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Muse at 26.7 t/s generation and 269 t/s prompt processing on an M4 Max 128GB.
Setup is Apple's coreai-pipelined GPU engine with int4hu block32 symmetric quantization, text decoder only, 16.35 GB bundle size.
The source table pairs 269.0 tok/s prompt with 26.69 tok/s decode under llm-benchmark at 512 prompt / 1024 generation, 3 trials; aggregation, cache state and batch configuration for that row are not stated.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Gemma 4 E4B at 78.3 t/s generation and 1236 t/s prompt processing on a single M4 Pro Mac mini with 64 GB unified memory.
Setup is MLX 0.32.0 with MLX-LM 0.31.3, an affine mixed 3.40 bpw text-only conversion, batch 1, greedy temperature 0, thinking enabled.
Source medians are 1,236.00 PP and 78.32 TG over 2,048 prompt tokens and 235 generated tokens; ranges are 1,233.55-1,236.60 PP and 78.22-78.52 TG. The author reports quality losses versus its bf16 comparator, including MMLU-Pro-400 -9 percentage points.