Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Llama 3.1 8B at 115.4 t/s generation and 3225.7 t/s prompt processing on an M5 Max with 48GB unified RAM.
Setup is MLX with 4-bit weights and unquantized KV cache, batch size 1, one model per process.
The headline uses pp512 and tg128 from a five-trial mean, not a best-of sweep; the source also records pp128=1932.3, pp256=2487.89 and tg256=115.52 t/s.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Llama 3.2 3B Instruct 4-bit at 237.9 t/s generation and 7307.2 t/s prompt processing on an M5 Max with 48GB unified RAM.
Setup is MLX via mlx-lm 0.31.3 with 4-bit Safetensors weights, batch size 1, unquantized KV cache and no speculative decoding.
The figures are arithmetic means of five trials at prompt 512 and generation 128; the benchmark is synthetic token throughput, not end-to-end request throughput.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Llama 3.2 1B Instruct 4-bit at 527.7 t/s generation and 17477 t/s prompt processing on an M5 Max 48GB.
Setup is MLX with 4-bit weights and unquantized KV cache, batch size 1, one warmup plus five measured trials.
The headline uses pp512 and tg128, not a best-of sweep; the source also lists pp128 7439.85, pp256 13408.64 and tg256 542.87 t/s.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports DeepSeek R1 Distill at 121.3 t/s generation and 3351.9 t/s prompt processing on an M5 Max 48GB.
Setup is MLX with 4-bit weights, batch size 1, unquantized KV cache, one Mac Studio with 18 CPU cores and 40 GPU cores.
The headline uses pp512 and tg128 from a five-trial mean, not a best-of sweep; the source also lists pp128=2139.57, pp256=2956.62, tg256=121.99 t/s.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Gemma 4 31B at 13.9 t/s generation and 1237 t/s prompt processing on an M5 Ultra 256GB.
Setup is MLX with BF16 weights at a 32768-token input and 128-token output, batch 1 on one device, with KV capacity unconfigured and blank.
The 13.9 t/s is the median of five trials, with a range of 13.821 to 13.917 t/s; peak allocator memory was 71.978GB.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.6 35B-A3B at 130.4 t/s generation and 7154.6 t/s prompt processing on an M5 Ultra 256GB.
Setup is MLX with a 4-bit quant, batch 1 on one device, at a 32768-token input and 128-token output.
The figures are separate five-trial medians, not pooled across lengths; the 32768-input run peaked at 22.079GB of allocator high-water memory.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.5 122B-A10B at 74.7 t/s generation and 2710.5 t/s prompt processing on an M5 Ultra 256GB.
Setup is MLX with a 4-bit quant, batch 1 on one device, at a 32768-token input and 128-token output.
The figures are separate five-trial medians, not pooled across lengths; the same run peaked at 72.259GB of allocator high-water memory.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3-8B-4bit at 110.6 t/s generation and 3150.6 t/s prompt processing on an M5 Max with 48GB unified RAM.
Setup is MLX via mlx-lm 0.31.3 with 4-bit weights and unquantized KV cache, batch size 1, one model per process.
The headline figures are pp512 and tg128 from a five-trial mean, not a best-of sweep; the source also records pp128=1876.59, pp256=2405.12 and tg256=111.03 t/s.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports DeepSeek R1 Distill at 28.3 t/s generation and 731.4 t/s prompt processing on an M5 Max with 48GB unified RAM.
Setup is MLX with 4-bit weights and unquantized KV cache, one model invocation per process, batch size 1, no speculative decoding.
The headline figures are pp512 and tg128, not a best-of sweep or average; the source also lists pp128=493.38, pp256=616.57 and tg256=28.44 t/s. Peak MLX allocator memory was 19.119 GB.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports DeepSeek R1 Distill at 60.7 t/s generation and 1714.8 t/s prompt processing on an M5 Max 48GB.
Setup is MLX with 4-bit weights, batch size 1, unquantized KV cache, one model invocation per process.
The headline uses pp512 and tg128, not a best-of sweep or average; the benchmark is synthetic text-token throughput, not end-to-end request throughput.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3-14B-4bit at 67 t/s generation and 2529.8 t/s prompt processing on a Mac Studio M5 Ultra with 256 GB unified memory.
Setup is MLX via mlx-lm 0.31.3 with 4-bit MLX Safetensors, batch 1, no prompt cache, no draft model, and no KV or activation quantization.
The figures are medians across five trials at a native 32768-token input and 128-token generation cell, with peak memory 14.332 GB; the 32768 context is the actual input length, not a supported maximum.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8 oQ4e-mtp restored-prefix B3 / PR1 61bd8a06 at 8.4 t/s on an M5 Max 128GB.
Setup is oMLX with oQ4e 4-bit weights and TurboQuant 4-bit KV cache, three greedy requests batched at roughly 20k / 45k / 135k prompt tokens with SSD prefix cache enabled.
The 8.4 t/s is the per-row rate at the 20k row, measured at 8.35 t/s; the other rows ran 8.83 and 7.84 t/s per row, and the batch phase took 66 seconds.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports DeepSeek V4.1 Flash at ~30 tok/s decode at 2K context on an M5 Max 128GB, streaming a 341 GB Q2 GGUF from SSD.
Setup is a fork of antirez's ds4 engine (sf-ds4-1flash) with Metal backend, one machine, weights streamed from disk.
Decode is ~26 tok/s at 32K context and settles around 24 tok/s after an hour of heavy use. First token after an 8K prompt dropped from 1.23 s to 0.15 s. Prefill improved 45-99% and decode 62-78% versus upstream ds4. Output is bit-identical to ds4.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3 30B-A3B at 98.2 t/s on an M5 Pro with 64GB unified memory.
Setup is MLX with 4-bit affine quantization, group size 64, standard 8 routed experts per token, batch 1, 944-token prompt and 512 output tokens.
The figure is the arithmetic mean of six fresh-process runs, ranging 97.97 to 98.37 t/s; an altered 6-expert run reached 107.79 t/s but is not a standard-model result.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3 30B-A3B at 144.4 t/s on an M5 Max with 128GB unified memory.
Setup is MLX with 4-bit affine quantization, group size 64, standard 8 routed experts per token, batch 1 with a 944-token prompt and 512 output tokens.
The figure is the arithmetic mean of six fresh-process runs, ranging 144.10 to 145.01 t/s; an altered 6-expert routing gave 155.61 t/s but is not a standard-model result.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8 27B at 18.5 t/s generation and 381.6 t/s prompt processing on an M5 Pro with 64GB unified RAM.
Setup is llama.cpp with the Unsloth UD-IQ4_XS quant, run via llama-bench at -p 512 -n 128.
The source gives 18.45 +/- 0.03 tok/s and 381.62 +/- 0.18 tok/s; the generation figure is not a post-512-prompt end-to-end run, and context limit, KV precision and offload settings are unstated.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8-Flash-Next at 100.2 tok/s on an M5 Ultra Mac Studio with 256GB unified memory.
Setup is llama.cpp b11538 with IQ4_NL weights, f16 KV cache, 262K context, and MTP speculative decoding with probabilistic drafting at n-max 5.
Without speculation the model decodes at 57.0 tok/s; prompt processing reaches 2150 tok/s with ubatch 4096. The full 262K context fits with about 103 GB of GPU memory free. A second copy of the model loaded by another app caused swapping and dropped speed to 12 tok/s.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User benchmarks Qwen3.8-Flash-Next oQ6e at 135.2 t/s decode and 2,458 t/s prefill on an M5 Ultra 256GB, comparing oMLX 0.7.0 and 0.7.1.dev1.
Setup is oMLX with MLX oQ6e weights, 8-bit KV cache, Lightning MTP on, 1K context, single request.
Across 4K-200K contexts, average decode is 120.9 t/s and prefill 4,761 t/s; 2 parallel requests give 74.2 t/s per request and 4 parallel 41.0 t/s per request. GLM-5.3-Flash oQ4 reaches 97.9 t/s decode and 2,264 t/s prefill, and Flash-Next oQ4e 121.6 t/s decode and 5,450 t/s prefill.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8 27B running at 10-20 tok/s on an M5 Pro Mac Mini with 48GB RAM, below the 30-50 tok/s they expected from community optimizations.
Setup is oMLX 0.70 with both 4-bit and 8-bit models; the 4-bit run took 40 minutes on a lengthy command and the 8-bit run progressed only 5% in over 10 minutes before being cancelled.
The user asks whether this performance is expected for their hardware.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8-Flash-Next (FP8) at 15.0–17.2 tok/s decode and 500–600 tok/s prefill at 16K context on an M5 Pro 64GB Mac.
Setup is Whallm 1.1.11 with MTP on by default, streaming experts from SSD; peak MLX memory stays under 19 GiB at 16K context.
A second row for Swift1.5-Qwen3.8 (MXFP4) gives 14.6–17.5 tok/s decode and 571 tok/s prefill at 18.8 GiB peak. Previous v1.1.7 numbers were ~9 tok/s decode and ~146 tok/s prefill.