Local model performance on your hardware

generation:

21.4 tokens/s

prompt processing (prefill):

235.1 tokens/s

quant:

4bit MLX (MLX)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Gemma 4 31B at 21.4 t/s generation and 235.1 t/s prompt processing on a single Mac Studio with an M3 Ultra and 512 GB unified memory. Setup is mlx-vlm with 4bit MLX, 36,777 actual input tokens and 128 generated tokens per run, three runs, peak memory 29.7 GB. The figures are medians; prompt processing p10/p90 are 234.8 and 236.1 t/s, and KV quantization, flash attention and speculative decoding are not exposed.

generation:

33.0 tokens/s

prompt processing (prefill):

386.4 tokens/s

quant:

4bit MLX (MLX)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.5 397B-A17B at 33 t/s generation and 386.4 t/s prompt processing on a single Mac Studio with an M3 Ultra and 512 GB unified memory. Setup is mlx-vlm with 4bit MLX, run at long context with 34,448 actual input tokens and 128 generated tokens per run, three runs, peak memory 234.9 GB. The figures are medians; prompt processing p10/p90 are 359.4 and 428.1 t/s, and median wall time was 105 seconds. KV quantization, flash attention and speculative decoding are not exposed, and cold-cache prefill is not established.

generation:

67.6 tokens/s

prompt processing (prefill):

1645.0 tokens/s

quant:

4bit MLX (MLX)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Gemma 4 26B-A4B at 67.6 t/s generation and 1,645.0 t/s prompt processing on a Mac Studio with an M3 Ultra and 512 GB unified memory. Setup is mlx-vlm with 4bit MLX, run on a 36,777-token prompt with 128 generated tokens per run over three runs; both figures are medians. Peak memory was 20.0 GB and median wall time 27 seconds; the configured context cap and KV quantization are not stated.

generation:

13.2 tokens/s

quant:

BF16 unquantized (MLX)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8 at 13.2 t/s on an M3 Ultra with 512GB unified memory. Setup is mlx-vlm 0.6.13 with the unquantized bf16 MLX VLM build, peak memory 55.8 GB. The figure comes from one 470-token image-plus-text prompt, one run per build, presented as an order-of-magnitude guide rather than repeated benchmarking.