Local model performance on your hardware

generation:

83.8 tokens/s

quant:

4bit (MLX)

User reports Qwen3.6-35B-A3B at 83.75 t/s on an Apple M2 Max 64GB, the middle of three sequential runs (82.28, 83.75, 83.09 t/s). Setup is LM Studio with the MLX runtime (mlx-llm-mac-arm64-apple-metal-advsimd 1.13.1), 4-bit MLX weights, 262144-token context, parallel 4, whole model in unified memory. One request at a time with no speculative decoding; 705 prompt tokens and 255 completion tokens, all reasoning tokens. Run 1 was cold with 6.973 s to first token, warm runs 1.156 s and 0.897 s.

generation:

17.2 tokens/s

prompt processing (prefill):

63.7 tokens/s

quant:

4bit (MLX)

User reports Gemma 4 31B at 17.2 t/s generation and 63.7 t/s prompt processing on a Mac Studio M2 Max with 64 GB unified memory. Setup is mlx-vlm 0.4.4 with 4-bit affine group-64 weights and MLX 0.31.1 on one GPU, running the text-only long-text request with 55 prompt tokens and 500 generated tokens. The 29.9-second whole generate call is not TTFT, and the 63.7 t/s prompt figure is a single request rather than a llama-bench pairing.

generation:

93.0 tokens/s

prompt processing (prefill):

468.6 tokens/s

quant:

4-bit (source config) (MLX)

User reports Nemotron 3 at 93 t/s generation and 468.6 t/s prompt processing on an Apple M2 Max with 64GB unified memory. Setup is MLX with 4-bit quantization, a requested prompt length of 512 and generation cap of 128 at batch size 1, with no HTTP server involved. The figures are the mean of three retained trials after discarding Trial 1; peak memory was 18.974 in engine output units, and KV cache precision and prefill chunk size are unreported.

generation:

18.0 tokens/s

prompt processing (prefill):

126.8 tokens/s

quant:

4-bit (source config) (MLX)

User reports Qwen 2.5 Coder 32B at 18 t/s generation and 126.8 t/s prompt processing on an M2 Max 64GB. Setup is MLX with 4-bit quantization, batch size 1, a requested 512-token prompt and 128-token generation cap, with peak memory 19.12 in engine output units. The stored figures are the three-trial means (TG 17.96 t/s, PP 126.82 t/s) rounded to 0.1 t/s; the source's CLI average includes a discarded fourth trial.

generation:

88.9 tokens/s

prompt processing (prefill):

521.5 tokens/s

quant:

4-bit (source config) (MLX)

User reports Qwen3.5 35B-A3B at 88.9 t/s generation and 521.5 t/s prompt processing on a single Apple M2 Max with 64 GB unified memory. Setup is MLX with 4-bit quantization, a requested 512-token prompt and 128-token generation cap at batch size 1, with no drafter passed. The figures are three-trial means after discarding Trial 1; the source means are 88.85 t/s generation and 521.47 t/s prompt, and peak memory was 20.218 in engine output units.

generation:

20.7 tokens/s

prompt processing (prefill):

91.4 tokens/s

quant:

4-bit (source config) (MLX)

User reports Qwen3.5 27B at 20.7 t/s generation and 91.4 t/s prompt processing on an Apple M2 Max with 64GB unified memory. Setup is MLX with 4-bit quantization, batch size 1, a requested prompt length of 512 and generation cap of 128, run directly via python -m mlx_lm benchmark with no HTTP server. The figures are the mean of three retained trials after discarding Trial 1; the source reports 20.71 t/s TG and 91.36 t/s PP, and peak memory of 16.302 in engine output units.

generation:

52.6 tokens/s

prompt processing (prefill):

413.7 tokens/s

quant:

4-bit (source config) (MLX)

User reports GLM-4.7 at 52.6 t/s generation and 413.7 t/s prompt processing on a single Apple M2 Max with 64 GB unified memory. Setup is MLX with a 4-bit artifact, batch size 1, requested 512-token prompt and 128-token generation; KV cache precision and prefill chunk size are unreported. The figures are three-trial means after discarding Trial 1, with sample SD 0.04 for TG and 0.38 for PP; peak memory was 17.497 in engine output units.

generation:

61.6 tokens/s

prompt processing (prefill):

465.2 tokens/s

quant:

4-bit mixed (config) (MLX)

User reports Gemma 4 26B-A4B at 61.6 t/s generation and 465.2 t/s prompt processing on an Apple M2 Max with 64GB of shared unified memory. Setup is MLX via the mlx_vlm text-only helper with a 4-bit mixed config, batch_size=1, requested 512-token prompt and 128-token generation cap, no HTTP server. The figures are three-trial means after discarding Trial 1; retained generation values were 61.389, 61.732 and 61.742 t/s, and peak memory was 15.255 GiB.

generation:

117.8 tokens/s

prompt processing (prefill):

616.4 tokens/s

quant:

4-bit mixed (config) (MLX)

User reports LFM2 24B-A2B at 117.8 t/s generation and 616.4 t/s prompt processing on an M2 Max 64GB. Setup is MLX with a 4-bit mixed config, run directly via python -m mlx_lm benchmark with batch_size=1, a 512-token prompt and 128-token generation cap. The figures are three-trial means after discarding Trial 1; peak memory was 13.822 in engine output units, and KV cache precision, prefill chunk size and sampling settings are unreported.

generation:

24.4 tokens/s

prompt processing (prefill):

104.2 tokens/s

quant:

4-bit (source config) (MLX)

User reports Devstral-Small-2505-4bit at 24.4 t/s generation and 104.2 t/s prompt processing on a single Apple M2 Max with 64 GB unified memory. Setup is MLX via python -m mlx_lm benchmark with 4-bit group 64 weights, a requested 512-token prompt and 128-token generation cap at batch size 1; KV cache precision is unreported. The figures are three-trial means after discarding Trial 1, with retained TG values of 24.434, 24.417 and 24.424 t/s; peak memory was 13.763 in engine units.

generation:

85.5 tokens/s

prompt processing (prefill):

501.5 tokens/s

quant:

MXFP4-Q4 mixed (config) (MLX)

User reports gpt-oss 21B-A3.6B at 85.5 t/s generation and 501.5 t/s prompt processing on an M2 Max with 64GB unified memory. Setup is MLX with MXFP4-Q4 mixed quantization, run directly via python -m mlx_lm benchmark with batch_size=1, a requested prompt length of 512 and generation cap of 128. The figures are three-trial means after discarding Trial 1; peak memory was 11.713 in engine output units, and KV cache precision, prefill chunk size and sampling settings are unreported.

generation:

62.7 tokens/s

prompt processing (prefill):

321.2 tokens/s

quant:

4-bit (source config) (MLX)

User reports Qwen3.5 9B at 62.7 t/s generation and 321.2 t/s prompt processing on an Apple M2 Max with 64GB unified memory. Setup is MLX with 4-bit quantization, a requested prompt length of 512 and generation cap of 128, batch size 1, no HTTP server. The figures are three-trial means after discarding Trial 1; retained generation trials were 62.651, 62.747 and 62.627 t/s, and peak memory was 5.87 in engine output units.

generation:

72.9 tokens/s

prompt processing (prefill):

566.4 tokens/s

quant:

4-bit (source config) (MLX)

User reports Llama 3.1 8B at 72.9 t/s generation and 566.4 t/s prompt processing on a single Apple M2 Max with 64 GB unified memory. Setup is MLX via a direct python -m mlx_lm benchmark, 4-bit quantized, batch size 1, with a requested 512-token prompt and 128-token generation cap. The figures are three-trial means after discarding Trial 1; peak memory was 5.116 in engine output units, and KV cache precision and prefill chunk size are unreported.

generation:

64.4 tokens/s

prompt processing (prefill):

491.1 tokens/s

quant:

4-bit (source config) (MLX)

User reports Granite 3.2 at 64.4 t/s generation and 491.1 t/s prompt processing on an M2 Max 64GB. Setup is MLX with 4-bit quantization, batch size 1, a requested 512-token prompt and 128-token generation cap, with no HTTP server involved. The figures are three-trial means after discarding Trial 1; peak memory was 5.218 in engine output units, and KV cache precision and prefill chunk size are unreported.

generation:

76.8 tokens/s

prompt processing (prefill):

602.8 tokens/s

quant:

4-bit (source config) (MLX)

User reports Qwen2.5 7B at 76.8 t/s generation and 602.8 t/s prompt processing on an Apple M2 Max with 64GB unified memory. Setup is MLX via python -m mlx_lm benchmark with the mlx-community/Qwen2.5-7B-Instruct-4bit artifact, batch size 1, 512-token prompt and 128-token generation cap. The 76.8 t/s and 602.8 t/s are three-trial means after discarding Trial 1; retained generation values were 76.773, 76.747 and 76.821 t/s.

generation:

128.3 tokens/s

prompt processing (prefill):

743.4 tokens/s

quant:

4-bit (source config) (MLX)

User reports Ministral 3 3B at 128.3 t/s generation and 743.4 t/s prompt processing on an Apple M2 Max with 64 GB unified memory. Setup is MLX via a direct python -m mlx_lm benchmark, 4-bit quantized, with a requested prompt length of 512 and generation cap of 128 at batch_size=1. The figures are three-trial means after discarding Trial 1; peak memory was 2.523 in engine output units, and KV cache precision and prefill chunk size are unreported.

generation:

71.3 tokens/s

quant:

NVFP4 (MLX)

kv:

q8_0

User reports Qwen3.6-35B-A3B at 71.3 t/s decode on short prompts on a Mac Studio M2 Max 64GB. Setup is Ollama with NVFP4 MLX weights, q8_0 KV cache, 32768 context, flash attention on, single concurrent request, think disabled. Decode stays near-flat with prompt length (69.3 t/s medium, 67.8 t/s long) and the 5-turn multiturn total is 23.7 s; the model is the recommended default coding model, replacing Qwen3-Coder-30B.

generation:

35.9 tokens/s

prompt processing (prefill):

428.5 tokens/s

quant:

Q4_0 (GGUF)

User benchmarks llama.cpp in a GPU-accelerated Podman container on an M2 Max Mac Studio with 64GB RAM, reporting 428.47 t/s prompt processing and 35.88 t/s token generation for llama 7B Q4_0. Setup uses llama.cpp built with Vulkan and GPU-remoting via krunkit, with -ngl 99, in a Fedora 40 container. The container is about 40% slower than native Metal (746.06 t/s pp512, 71.36 t/s tg128), but 3x faster than a CPU-only container in prompt processing and 25% faster in token generation. Enabling -fa in the Vulkan container drops performance to 312.92 t/s pp512 and 14.10 t/s tg128.

generation:

70.4 tokens/s

quant:

Q4_K_M (GGUF)

User reports Qwen3.5 4B at 70.4 t/s on an M2 Max using native GGUF support in transformers. Setup is the Q4_K_M GGUF from unsloth, reusing ggml kernels on Apple Silicon. llama.cpp reached 71.8 t/s on the same model. Two other models were also tested: Qwen3.8 27B UD-Q4_K_M at 15.9 t/s (vs 13.4 t/s with llama.cpp) and Qwen3.5 35B-A3B UD-IQ4_XS at 60.2 t/s (vs 61.3 t/s).

generation:

2.0 tokens/s

User reports DeepSeek V4.1 Flash at 1.8-2.2 t/s on an M2 Max 64GB. Setup is the Whallm app with MLX, loading parts of the model from SSD, around 33 GiB peak MLX memory, with 1K-16K input tokens. The app also adds built-in throughput benchmarks and an OpenAI-compatible API.