Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Gemma 4 E4B at approximately 29.5 tokens/s on a Mac mini M4 with 24GB unified memory.
Setup is LM Studio 0.4.12 with llama.cpp engine 2.13.0 (Metal), GGUF Q4_K_M, 4096-token context, automatic GPU offload, temperature 0, 200 generated tokens per request; two trials gave 29.62 and 29.46 tokens/s with about 0.3s time to first token.
A separate run at 32768 context and 12961 prompt tokens gave 25.95 tokens/s with 39.44s cold TTFT and 0.12s repeat TTFT from prefix caching. Four simultaneous requests with parallel=4 gave 9.66-9.69 tokens/s per request and 37.5 tokens/s aggregate. Source-reported measurements, not independently reproduced; prefill throughput, measured memory, KV-cache quantization, Flash Attention and speculative-decoding settings unspecified, and the 6.29GiB memory figure is an estimate.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.6-35B-A3B at 104.5 tokens/s single-stream decode on an Apple Mac mini M4 Pro with 48 GB unified memory.
Setup is Rapid-MLX 0.15.4 with MLX 4-bit weights (17.5 GB on disk). The figure is the median of three runs at temperature 0.
Separately, the source reports 106.5 tokens/s aggregate with four concurrent streams and a 2.4 s TTFT for a 2K-token prompt. The 23.2 GB Metal peak covers the whole sweep, not single-stream use. These are developer-produced measurements; the linked raw-data directory returned a 404 when checked.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.5-9B at 82.53 tokens/second generation on a Mac Studio M4 Max with 64GB unified memory.
Setup is vllm-mlx with a 4-bit MLX quant, measured by llama-benchy over 2048 prompt tokens and 32 generated tokens across three runs. A generation peak of 87.99 tokens/second and a TTFT of 2.41 seconds are also reported.
Exact quantization scheme, model artifact, runtime versions, configured context limit, concurrency, speculative decoding status and resident memory usage are not stated. The 64GB is system memory capacity, not measured memory use.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
creative-writing
User reports Qwen3.5-9B at 60.1 tokens/s median decode on an Apple Mac mini M4 Pro with 48 GB unified memory.
Setup is Rapid-MLX 0.15.6 with mlx-community/Qwen3.5-9B-4bit, greedy decoding, thinking off, MTP plus prompt lookup speculation, and a short-story task. Baseline mlx-lm 0.32.0 server with speculation off reaches 47.1 tokens/s.
Three-run medians with fresh server and isolated cache per run. The whole-file rename task reaches 198.4 tokens/s versus 46.4 for the baseline, but that peak is task-specific and not a general generation rate.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
codingvisionlong-context
User reports Qwen3.8-Flash-Next at ~60 tok/s serial decode and ~730 tok/s prefill on an M4 Max 128GB.
Setup is mlx-serve 26.8.11 with a mixed 4-bit/8-bit MLX pack, ~75 GB resident, prefix cache on, images included. With MTP enabled decode reaches 78 tok/s (+41% on code, a few percent slower on prose). A needle at 24.8k tokens was recovered with sparse attention engaged.
The pack includes the MTP head and vision tower; the 51B n-gram table ships as a 32 GB mmapped file that is not resident. The author recommends a re-quantized iQ-MLX-4.7bpw version instead.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
codingagentic
User reports Qwen3.8-Flash-Next at 57.4 tok/s decode and 586 tok/s prefill on an M4 Max 128GB, where the model fits fully in memory.
Setup is oMLX with the Jundot oQ4e+MTP MLX checkpoint; the 36GB paging configuration uses oMLX 0.7.0 expert offload with resident fraction 0.28 (143/512 experts per layer), PLE on SSD and MTP off.
The same model on an M4 Max Studio 36GB measures 15.55 tok/s steady (n>=1024) and 13.0 tok/s short-run (n=128); exact replay on repeated prompts adds about 40% (14.3 to 20.1 tok/s). Other memory tiers are simulated: 48GB at 192 experts ~15.9 tok/s, 48GB at 128 experts ~11.8 tok/s, M4 Pro 32GB ~9.7 tok/s, M4 24GB ~4.9 tok/s.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
codinglong-contextvision
User reports Qwen3.8 Flash-Next at 73.8 tok/s on an M4 Max with 128 GB unified memory.
Setup is mlx-serve with Q4 affine routed experts, BF16 core and verification head, native MTP at depth 4, and the full 262,144-token context; the 51.2 GB n-gram lookup table streams from SSD.
Matched warm workloads measured 71-74 tok/s for code and technical prose, with serial decode at 33.2 tok/s and a 2.22x MTP speedup. Long-context prefill reached 280.7 prompt tok/s at 248,445 tokens.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports a ~3.8x generation speed regression on Apple M4 Mac Mini (16 GB unified memory) with Gemma 4 26B A4B IQ4_XS. Master build (d12cc3d) achieves 8.14 tok/s versus 27.18 tok/s on tag b8680, both built from source with llama-server and identical flags. Setup uses Metal backend, --flash-attn on, q4_0 KV cache, 65536 context, and -ngl 99. The regression is reproducible and not explained by memory or swap pressure; prefill on b8680 was 55.7 tok/s.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
agentictool-uselong-context
User reports Gemma 4 26B-A4B QAT at 36-38 tok/s generation on an M4 Pro 24GB MacBook Pro.
Setup is Ollama (llama.cpp) with QAT 4-bit weights, q8_0 KV cache, flash attention on, and a 64K context window; the model is 14 GB and 100% GPU-resident.
Prefill is 95-125 tok/s on short prompts, cold load ~16 s, warm reload 0.4 s. The user notes the model squeezes the rest of the OS (9% memory free, swap nearly full) and that a dense 31B would spill to CPU.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports running the full 2.78T-parameter Kimi K3 on a 64 GB MacBook Pro at around 0.32–0.34 tokens per second.
Setup is the WASTE (Weight-Aware Streaming Tensor Engine) with a 982 GiB NVMe-optimized container, streaming routed experts from disk and caching them in RAM, at a 4K context with a measured minimum memory requirement of approximately 29 GB.
The engine needs roughly 17 GB of expert data per token and more than half of decode time is spent reading experts from disk. The same engine runs Kimi-Linear 48B from a 19 GB container at 8.92 tokens per second with an 8 GB memory budget.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Gemma 4 26B-A4B at 23.6 tok/s decode and 55 tok/s prompt processing in a browser tab on an M4 Pro 24GB Mac.
Setup is LocalMind (WebGPU) with Google's QAT Q4_0 GGUF (14.4 GB), a 4 GB expert cache, and routed experts streamed from disk; the Chrome GPU process sits at 6.9 GB with about 8.6 GB of experts on disk.
llama.cpp Metal does 70.6 and 204 tok/s on the same Mac, so the tab is about 3x slower at decode. The user also ran Qwen3.6 35B-A3B (unsloth Q8_0, 36.9 GB) at 9.9 tok/s decode with 2.2 s to first token, a file larger than the machine's memory. Output matched llama.cpp on 9/9 Gemma conversations and 8/8 Qwen cuts.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
codingagentic
User reports Qwen3.5-4B think at ~150 t/s on an M4 Pro 48GB.
Setup is llama.cpp with Q4_K_M at 32k context, using 2.5 GB of RAM.
The model scores 6/7 on the CC-Agent harness and 14/14 on the V3.1 text suite, matching the 35B-A3B model at 1/8 the memory.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8-Flash-Next at 50~70 tg/s at 128k context on an M4 Max 128GB.
Setup is oMLX 0.7.0 with an oQ4e MLX quant and MTP speculative decoding enabled; memory use is 75~80 GB at that context length.
User says older versions dropped from 60 tg/s to 35 tg/s once context passed 30k, and that the speed now holds as context grows.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
summarization
User reports Gemma 4 E2B at ~45 tokens/s single-task on three M4 Mac minis with 16GB RAM, each handling around 10 parallel tasks at ~6 tokens/s each.
Setup is local inference on Apple Silicon; no engine, quant, or context length stated.
The user asks whether one Mac Studio or more Mac minis would give better throughput for parallel labeling and categorization.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
codingmath
User benchmarks Gemma 4 E4B with MTP speculative decoding on a MacBook Pro M4 Pro 24GB, reporting 82.8 tok/s versus 44.6 tok/s baseline (1.86x mean, 2.03x best speedup).
Setup is LiteRT-LM Metal GPU with int4 weights at batch=1, temperature=0, using real prompts; the E2B model reaches 144.3 tok/s with MTP versus 84.2 tok/s baseline. HF transformers + MPS with bf16 weights achieves only 12.1 to 15.1 tok/s (1.24x), and llama.cpp Metal Q8_0 cannot engage MTP for Gemma 4.
The user notes that synthetic-token benchmarks show MTP as a regression (0.93x) because random sequences yield near-zero drafter acceptance, and warns that E4B bf16 on 24GB Macs can trigger kernel panics from Metal wired-memory pressure.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports a decode throughput cliff on a Mac mini M4 Pro 48GB running Qwen3.8-Flash-Next 4-bit via Slotstream. Below 32,768 active tokens decode is about 11 tok/s; above it drops to 3.8-4.0 tok/s. The transition is sharp: 32,320 prompt tokens gives 9.49 tok/s, while 32,770 gives 3.84 tok/s. Setup uses a 26 GB expert pool, 65,536 max context, and MTP off. The user notes MTP on this machine is slower (about 3.7 tok/s vs 12 tok/s off) and disabled it for these tests.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User benchmarks Qwen 3 14B at 11.5 tok/s on a Mac Mini M4 with 24 GB unified memory.
Setup is MLX with 4-bit quantization, using 8.4 GB peak memory and leaving 74% of system memory free with no swap.
The study also reports Ollama q4_K_M results: Llama 3.2 3B at 42.0 tok/s, Gemma 2 9B at 15.2 tok/s, Qwen 2.5 14B at 10.8 tok/s, and DeepSeek-R1 32B at 0.28 tok/s. MLX Qwen 3 30B-A3B MoE reaches 44.8 tok/s but uses 16.3 GB and causes 2.74 GB swap; Qwen 3.5 35B-A3B crashes with Metal OOM at 20.4 GB. Tests ran under normal workstation load with ~12-13 GB already committed.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Gemma 4 E4B at 25.6–28.3 tok/s on an M4 alone, with TTFT of about 0.31 s on a 71-token prompt.
Setup is Ollama with 4-bit weights; a distributed run across the M4 and an RTX 3060 6 GB over Wi-Fi reached about 8 tok/s with layers 0–14 on the M4 and 14–42 on the RTX, and about 10.2 tok/s with layers 0–4 on the M4 and 4–42 on the RTX.
Gemma 4 E2B runs at 42.8 tok/s on the M4 alone, up from about 11.5 tok/s before optimising the decode path. The user notes that adding a second machine does not automatically speed up inference and is exploring task-level rather than layer-level distribution.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
agenticsecurity-researchlearning
User reports about 25 tokens/s on short prompts running a 35B MoE model at 4-bit with 131K context on two base M4 Mac minis (16 GB each) over Thunderbolt 4 using llama.cpp RPC mode.
Setup splits the model across the two M4s with a 50/50 memory split; a 40/60 split OOMed one machine. An M1 Mac mini acts as gateway with an OpenAI-compatible API and HTTP proxy.
A 64K-token prompt takes almost five minutes to first token, and prompt cache hits arrive roughly 70x sooner. User notes reasoning was on by default and that the setup is good for background agents and learning but cloud models are still used for daily work.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
codingagenticlong-context
User reports Qwen3.8 27B at 48 tok/s decode and 198 tok/s prompt processing at 4K context on an M4 Max with 36GB unified memory.
Setup is Ollama 0.34.4 with NVFP4/MLX weights, q8_0 KV cache, flash attention enabled, and a 128K configured context. The GGUF Q4_K_M version of the same model reached only about 10.7 tok/s decode.
Decode falls to 44 tok/s at 8K, 25 tok/s at 16K and 32K, and 14 tok/s at 115K, where time to first token reaches 21 minutes. The user notes that maximum context is not the same as comfortable interactive context and that memory pressure and swap grew during the stress runs.