- generation:
- 31.8 tokens/s
- prompt processing (prefill):
- 854.4 tokens/s
- quant:
- 4-bit (MLX)
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Gemma 3 27B at 31.8 t/s generation and 854.4 t/s prompt processing on an M5 Max with 48GB unified RAM.
Setup is MLX with 4-bit weights, unquantized KV cache, batch size 1, and a 2048-token prefill step.
The figures are arithmetic means of five trials from mlx-lm 0.31.3, with prompt processing including first-token work; peak MLX allocator memory was 16.69 GB.
- generation:
- 374.4 tokens/s
- prompt processing (prefill):
- 17261.9 tokens/s
- quant:
- 4-bit (MLX)
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Gemma 3 1B at 374.4 t/s generation and 17261.9 t/s prompt processing on an M5 Max 48GB.
Setup is MLX with 4-bit weights and unquantized KV cache, batch size 1, one model invocation per process.
The headline figures are pp512 and tg128 from a five-trial mean, not a best-of sweep or average across prompt lengths; the benchmark is synthetic text-token throughput, not end-to-end request throughput.
- generation:
- 66.9 tokens/s
- prompt processing (prefill):
- 1834.9 tokens/s
- quant:
- 4-bit (MLX)
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Gemma 3 12B 4-bit at 66.9 t/s generation and 1834.9 t/s prompt processing on an M5 Max 48GB.
Setup is MLX with 4-bit weights and unquantized KV cache on one Mac Studio, batch size 1, greedy sampling, no speculative decoding.
The figures are native engine phase rates: prompt speed includes first-token work, and the retained tg128 and pp512 come from the (512,128) run, not a best-of sweep or average.
- generation:
- 151.1 tokens/s
- prompt processing (prefill):
- 3333.4 tokens/s
- quant:
- 4-bit (MLX)
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.6 35B-A3B at 151.1 t/s generation and 3333.4 t/s prompt processing on an M5 Max 48GB.
Setup is MLX with 4-bit weights, batch size 1, unquantized KV cache and no speculative decoding, on one Mac Studio with 18 CPU cores and 40 GPU cores.
The figures are arithmetic means of five trials from mlx-lm 0.31.3; the prompt rate is a native engine phase rate with first-token work included, not client TTFT.
- generation:
- 186.0 tokens/s
- prompt processing (prefill):
- 5505.7 tokens/s
- quant:
- 4-bit (MLX)
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen 3 4B 4-bit at 186 t/s generation and 5505.7 t/s prompt processing on an M5 Max 48GB.
Setup is MLX with 4-bit weights and unquantized KV cache on one Mac Studio with 48 GB unified RAM.
The headline figures are pp512=5505.67 and tg128=185.98 tokens/s, not a best-of sweep or an average across prompt lengths; the benchmark is synthetic text-token throughput with batch size 1.
- generation:
- 28.4 tokens/s
- prompt processing (prefill):
- 730.9 tokens/s
- quant:
- 4-bit (MLX)
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3-32B-4bit at 28.4 t/s generation and 730.9 t/s prompt processing on an M5 Max with 48GB unified RAM.
Setup is MLX via mlx-lm 0.31.3 with 4-bit weights and unquantized KV cache, batch size 1, one warmup plus five measured trials.
The headline figures are pp512 and tg128, not a best-of sweep or average; peak MLX allocator memory was 19.151 GB.
- generation:
- 147.2 tokens/s
- prompt processing (prefill):
- 3458.2 tokens/s
- quant:
- 4-bit (MLX)
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen 3 30B-A3B 4-bit at 147.2 t/s generation and 3458.2 t/s prompt processing on an M5 Max 48GB.
Setup is MLX with 4-bit weights and unquantized KV cache on one Mac Studio with 48 GB unified RAM.
The figures are single-batch synthetic token rates, not end-to-end request throughput; peak MLX allocator memory was 17.755 GB.
- generation:
- 62.6 tokens/s
- prompt processing (prefill):
- 1738.6 tokens/s
- quant:
- 4-bit (MLX)
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3-14B-4bit at 62.6 t/s generation and 1738.6 t/s prompt processing on an M5 Max with 48GB unified RAM.
Setup is MLX with 4-bit weights and unquantized KV cache, batch size 1, one model per process.
The stored figures are the pp512 and tg128 values from a five-trial mean; the source also records pp128=1145.62, pp256=1393.0, tg256=62.71 t/s.
- generation:
- 366.0 tokens/s
- prompt processing (prefill):
- 12362.2 tokens/s
- quant:
- 4-bit (MLX)
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen 3 1.7B at 366 t/s generation and 12362.2 t/s prompt processing on an M5 Max 48GB.
Setup is MLX with 4-bit weights and unquantized KV cache on one Mac Studio with 18 CPU cores and 40 GPU cores.
The figures are native engine phase rates at batch size 1, not end-to-end request throughput; the source lists tg128=365.99 and pp512=12362.19.
- generation:
- 570.6 tokens/s
- prompt processing (prefill):
- 22693.8 tokens/s
- quant:
- 4-bit (MLX)
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen 3 0.6B 4-bit at 570.6 t/s generation and 22693.8 t/s prompt processing on an M5 Max 48GB.
Setup is MLX with 4-bit weights and unquantized KV cache on one Mac Studio, batch size 1, no speculative decoding.
The headline figures are the pp512 and tg128 rates, not a best-of sweep or average; the source also lists pp128=9042.7, pp256=12175.94 and tg256=594.11 t/s.
- generation:
- 121.8 tokens/s
- prompt processing (prefill):
- 3362.9 tokens/s
- quant:
- 4-bit (MLX)
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen 2.5 Coder 7B at 121.8 t/s generation and 3362.9 t/s prompt processing on an M5 Max 48GB.
Setup is MLX with 4-bit weights, batch size 1, unquantized KV cache and no speculative decoding, on one Mac Studio with 18 CPU and 40 GPU cores.
The headline uses pp512 and tg128, not a best-of sweep; the source also records tg256 at 122.5 t/s and pp128 at 2139.32 t/s.
- generation:
- 28.4 tokens/s
- prompt processing (prefill):
- 734.4 tokens/s
- quant:
- 4-bit (MLX)
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen 2.5 Coder 32B at 28.4 t/s generation and 734.4 t/s prompt processing on an M5 Max 48GB.
Setup is MLX with 4-bit weights, batch size 1, unquantized KV cache, one Mac Studio with 18 CPU cores and 40 GPU cores.
The headline uses pp512 and tg128 from a five-trial mean, not a best-of sweep; peak MLX allocator memory was 19.119 GB.
- generation:
- 60.8 tokens/s
- prompt processing (prefill):
- 1719.3 tokens/s
- quant:
- 4-bit (MLX)
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen 2.5 Coder 14B at 60.8 t/s generation and 1719.3 t/s prompt processing on an M5 Max 48GB.
Setup is MLX with 4-bit weights, batch size 1, unquantized KV cache and no speculative decoding, on one Mac Studio with 48 GB unified RAM.
The headline uses pp512=1719.28 and tg128=60.76 from a five-trial mean; the source also records pp128=1079.54, pp256=1358.9 and tg256=61.35.
QwQ 32B 4-bit; native pp512/tg128 summary
M5 Max 48GB · MLX
- generation:
- 28.4 tokens/s
- prompt processing (prefill):
- 734.8 tokens/s
- quant:
- 4-bit (MLX)
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports QwQ-32B-4bit at 28.4 t/s generation and 734.8 t/s prompt processing on an M5 Max with 48GB unified RAM.
Setup is MLX with 4-bit weights and unquantized KV cache, batch size 1, one warmup plus five measured trials per configuration.
The headline figures are the pp512 and tg128 rates, not a best-of sweep or an average across prompt lengths; peak MLX allocator memory was 19.119 GB.
Mistral Small-3.1-24B-Instruct-2503 4-bit; native pp512/tg128 summary
M5 Max 48GB · MLX
- generation:
- 41.1 tokens/s
- prompt processing (prefill):
- 1051.0 tokens/s
- quant:
- 4-bit (MLX)
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Mistral-Small-3.1-24B-Instruct-2503-4bit at 41.1 t/s generation and 1051 t/s prompt processing on an M5 Max 48GB.
Setup is MLX with 4-bit weights and unquantized KV cache on one machine, batch size 1, no speculative decoding.
The stored 41.1 t/s is the tg256 figure; the headline pair is pp512=1050.95 and tg128=41.06, with pp128=723.28 and pp256=882.13. Peak MLX allocator memory was 13.762 GB.
Mistral Nemo-Instruct-2407 4-bit; native pp512/tg128 summary
M5 Max 48GB · MLX
- generation:
- 75.6 tokens/s
- prompt processing (prefill):
- 2087.2 tokens/s
- quant:
- 4-bit (MLX)
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
Source-reported historical campaign by emihiggins, measured 2026-09-27T20:00:30Z; recovered 2026-10-13. No new run or local reproduction. Raw source: https://github.com/emihiggins/m5-max-llm-bench/blob/main/data/mac-llm-bench/m5-max_18c-40g_48gb/mlx/Mistral-Nemo-Instruct-2407-4bit_4bit_ngl99.json . Companion provenance: https://github.com/enescingoz/mac-llm-bench/pull/12 . The source states these are unedited copies of its upstream submission. Data files show public-release commit 34fee99b6df6129fb3c64255c1bfb179a54ca044 (September 27, 2026); the measurement predates discovery. One report groups this model, precision, runtime and hardware; all retained lengths are below.
Hardware: one Mac Studio (source PR identifies Mac17,14), Apple M5 Max, 18 CPU cores and 40 GPU cores, 48 GB unified RAM, macOS 27.0 (PR build 26A428), AC power and High Power energy mode. Source CPU JSON labels performance_cores=6 and efficiency_cores=12; author clarifies these are 6 Super plus 12 Performance cores. Reported 614 GB/s is theoretical bandwidth. Installed unified RAM is not dedicated VRAM. The raw available_ram_gb=30.7 is a system-availability snapshot, not installed capacity or model allocation. The source records a 32768 MB iogpu wired limit, swap <=2 MB checked every 60 seconds, no system thermal/performance warning, ambient about 20.6 C, external NVMe model storage, and idle Ollama with no model loaded. Per-model temperature, clock and wall power are unknown.
Model: mlx-community/Mistral-Nemo-Instruct-2407-4bit, source quant=4bit, MLX weights in Safetensors. Recorded fetched HF snapshot: 647ca0751669b21a364c86ccc5df54c4d7e4e91c; manifest: https://github.com/emihiggins/m5-max-llm-bench/blob/main/data/mac-llm-bench/m5-max_18c-40g_48gb/mlx-revisions.tsv . Identity verified individually at https://huggingface.co/mlx-community/Mistral-Nemo-Instruct-2407-4bit . Current card identifies conversion from mistralai/Mistral-Nemo-Instruct-2407. The snapshot is source-reported; model files were not downloaded or hashed, and current main must not be substituted for the recorded revision. Source file_size_gb=0 is a serializer placeholder for MLX repo strings, not a zero-byte model. Conversion-library versions on HF describe conversion, not the measured runtime.
Runtime: source JSON mlx-lm 0.31.3; source PR additionally reports mlx 0.32.2 and Python 3.12. One model invocation per process using ./bench_mlx.sh --repo <repo> --cleanup. The author reports cleanup path bugs; this does not establish a deleted model cache or cold load. Source's generic n_gpu_layers=99, flash_attention=true, threads=0 fields are hard-coded by the MLX serializer and do not verify actual MLX offload layers, Flash Attention setting or thread count. These feature fields stay unknown rather than borrowing llama.cpp flags. Configured context capacity is not recorded; prompt length is not context-window capacity.
All retained source speed values, tokens/s: pp128=1317.72; pp256=1659.43; pp512=2087.16; tg128=75.57; tg256=75.82. Headline uses pp512=2087.16 and tg128=75.57, not a best-of sweep or an average across prompt lengths. Linkage is inferred from the published wrapper's overwrite order, not independently retained per-trial logs. The current wrapper runs (prompt,generation)=(128,128),(128,256),(256,128),(256,256),(512,128), skipping (512,256); it overwrites each pp and tg key on every success. Therefore if all documented calls completed, the retained pp128 is from (128,256), pp256 and tg256 from (256,256), and pp512 and tg128 from (512,128). The exported dictionary loses the other paired rates, trial-level variance and failure details; do not reconstruct them. The historical executable/harness commit is unpinned. The source claims documented commands without deviations; current upstream code establishes method and interpretation, not byte-for-byte historical reproduction.
Timing method: mlx-lm v0.31.3 benchmark.py uses seeded random token IDs, batch size 1 by default, disables EOS stopping, performs one warmup then defaults to five measured trials for each configuration, and prints arithmetic means of native response rates. The wrapper defaults to five trials, parses those means and rounds speed to two decimals. The benchmark is synthetic text-token throughput, not conversational quality or end-to-end request throughput. Published versioned stream_generate measures prompt.size divided by elapsed time to its first generated token, then resets the timer and reports (n+1)/elapsed for generation. Thus PP is a native engine phase rate with first-token work included; it is not input tokens divided by client/network TTFT, load time or total request wall time. TG follows the library's own numerator/timer convention, not a newly recomputed rate. Each generate call makes a new prompt cache when none is supplied, while the model is already loaded and warmed. The source logs do not independently validate cache state; avoid a claim of proven cold prefill. No chat template, --no-think, speculative draft, MTP or quantized KV option is passed by this published benchmark. Default greedy sampler, unquantized KV and prefill step 2048 are code-derived defaults, not separately recorded run flags; corresponding form fields remain unknown.
Memory: source memory.peak_rss_gb=7.48. Despite that field name, the MLX exporter passes the highest native peak_memory mean among configurations. v0.31.3 stream_generate uses mx.get_peak_memory()/1e9, an MLX allocator high-water measurement in decimal GB, not operating-system process RSS, installed RAM, dedicated VRAM or necessarily a reset per-trial peak. No RSS/model-size claim is made.
Method sources: https://github.com/enescingoz/mac-llm-bench/blob/main/lib/run_bench_mlx.sh ; https://github.com/enescingoz/mac-llm-bench/blob/main/bench_mlx.sh ; https://github.com/enescingoz/mac-llm-bench/blob/main/lib/collect_results.sh ; https://github.com/ml-explore/mlx-lm/blob/v0.31.3/mlx_lm/benchmark.py ; https://github.com/ml-explore/mlx-lm/blob/v0.31.3/mlx_lm/generate.py . No HumanEval+ or other model-quality score is included for this campaign.
Mistral 7B-Instruct-v0.3 4-bit; native pp512/tg128 summary
M5 Max 48GB · MLX
- generation:
- 120.7 tokens/s
- prompt processing (prefill):
- 3210.4 tokens/s
- quant:
- 4-bit (MLX)
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Mistral-7B-Instruct-v0.3-4bit at 120.7 t/s generation and 3210.4 t/s prompt processing on an M5 Max 48GB.
Setup is MLX with 4-bit weights and unquantized KV cache on one Mac Studio with 18 CPU cores and 40 GPU cores.
The figures are arithmetic means of five trials at batch size 1, with pp512 and tg128 taken from the (512,128) configuration; the benchmark is synthetic token throughput, not end-to-end request throughput.