Local model performance on your hardware

generation:

105.6 tokens/s

prompt processing (prefill):

2899.7 tokens/s

quant:

Q4_K_M (GGUF)

flash attention:

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

Source-reported historical campaign by emihiggins, saved 2026-09-26T23:31:28Z; recovered 2026-10-13. No local model run or reproduction performed. Primary raw result: https://github.com/emihiggins/m5-max-llm-bench/blob/main/data/mac-llm-bench/m5-max_18c-40g_48gb/gguf/deepseek-r1-distill-7b_Q4_K_M_ngl99.json . Source publication commit is 34fee99b6df6129fb3c64255c1bfb179a54ca044. The author's upstream submission, including corrections and limitations, is https://github.com/enescingoz/mac-llm-bench/pull/12 . Hardware: one Apple Mac Studio Mac17,14, M5 Max, 18 CPU cores and 40 GPU cores, 48 GB unified memory; macOS 27.0 build 26A428, bare metal, AC power, High Power energy mode. Source corrects its JSON core labels: the 6/12 split denotes 6 Super and 12 Performance cores, not 12 Efficiency cores. iogpu.wired_limit_mb=32768 is a session limit, not installed RAM or dedicated VRAM. PR reports ambient about 20.6 C, external WD_BLACK SN7100 NVMe in OWC Express 1M2 over USB4, idle Ollama with no model loaded and no other heavy apps. Swap was at most 2 MB when checked every 60 s. No macOS thermal/performance warnings were recorded, but per-model thermal state was not captured. Do not import zero-swap/per-cell thermal claims from the author's separate bandwidth harness. Exact model: bartowski/DeepSeek-R1-Distill-Qwen-7B-GGUF, file DeepSeek-R1-Distill-Qwen-7B-Q4_K_M.gguf, GGUF Q4_K_M. Manifest: https://github.com/emihiggins/m5-max-llm-bench/blob/main/data/mac-llm-bench/m5-max_18c-40g_48gb/gguf-revisions.tsv . It records measured repository revision 361004151d4f4f6b446dc5e6d46fbf4422a80d5f and SHA256 731ece8d06dc7eda6f6572997feb9ee1258db0784827e642909d9b565641937b. The exact revision's HF file page was independently read and its displayed SHA256 matched: https://huggingface.co/bartowski/DeepSeek-R1-Distill-Qwen-7B-GGUF/blob/361004151d4f4f6b446dc5e6d46fbf4422a80d5f/DeepSeek-R1-Distill-Qwen-7B-Q4_K_M.gguf . This binds model identity and quantized artifact; today's main is not substituted for the measured revision. Source nominal model size label is 7B. Runtime: llama.cpp Homebrew build 11146, commit 7fe450e19, ggml 0.25.3, Metal. The raw runtime.version accidentally stores a load_backend BLAS log line; the author explicitly corrects it in PR 12 and reports Metal 4 tensor API active. This is not a CPU-only BLAS benchmark. Each model was run in a separate ./bench_gguf.sh --model process because --auto stopped after the first model. Source parameters are n_gpu_layers=99, flash_attention=true, threads=0. The helper omits -t when threads=0; this means runtime default rather than zero running threads. Requested 99 GPU layers is not a measured layer-offload count. Quantization above describes weights, not KV cache. Headline generation is tg128=105.57 tokens/s; prompt processing is pp512=2899.69 tokens/s. These are separate native llama-bench microtests from the same campaign, not one 512-input/128-output request, an average across lengths, or server concurrency throughput. All retained native rates in tokens/s: pp128=1905.93; pp256=2513.99; pp512=2899.69; tg128=105.57; tg256=103.15. The pp tests process 128/256/512 synthetic token IDs; the tg tests perform 128/256 one-token decode steps without a preceding 512-token prompt. The context field is intentionally unspecified: these test lengths are not an advertised context window or a long-context setting. Metric method: published mac-llm-bench bench_gguf.sh entrypoint passes -p 128,256,512 -n 128,256 -ngl 99 -fa 1 to llama-bench. It first runs human-readable output, then a separate -o json invocation for saved speed results; those two invocations are not combined. This entrypoint has inline speed/memory logic; the separate lib/run_bench_gguf.sh helper is not the saved-speed entrypoint, so its different display/capture ordering must not be substituted. lib/collect_results.sh maps native n_prompt/n_gen and avg_ts into these speed keys and rounds to 2 decimals. Native llama.cpp code at the reported 7fe450e19 revision times synchronized inference, computes each repetition's token count divided by elapsed nanoseconds, then takes the arithmetic mean of those rates. This is not input divided by TTFT, total wall time, or a pooled count/total-duration calculation. The pinned llama-bench defaults are 5 repetitions, warmup enabled, depth 0, with model memory cleared before each timed repetition; PP uses synthetic random IDs and TG sequential random IDs, without sampling meaningful chat output. This establishes native benchmark semantics. The exact executed shell-harness commit and full raw llama-bench samples, durations, dispersion and command log were not preserved in this result, so the 5-repetition/default-cache interpretation is code/protocol-derived rather than independently counted from raw samples. Current upstream shell code is supporting method evidence, not proof of byte-for-byte historical reproduction. KV precision, actual CPU thread count, realized batch settings, speculation/MTP and quality scores are not separately captured in this JSON; no quality validation or conversational thinking-mode conclusion follows. Memory: file_size_gb=4.36 is the on-disk file size divided by 2^30, despite the GB key. Source peak_rss_gb=4.5 is separately measured process maximum RSS. The published entrypoint obtains RSS with /usr/bin/time -l on a separate llama-bench -p 32 -n 1 invocation, then divides bytes by 2^30. It is not the timed speed test's peak, dedicated VRAM, model-only allocation or installed capacity. The per-record available_ram_gb=20.9 is available system memory at collection, not model usage. Method sources: https://github.com/enescingoz/mac-llm-bench/blob/main/bench_gguf.sh ; https://github.com/enescingoz/mac-llm-bench/blob/main/lib/collect_results.sh ; https://github.com/ggml-org/llama.cpp/blob/7fe450e19/tools/llama-bench/llama-bench.cpp . Source data/documentation license is CC BY 4.0. All values remain source-reported single-machine results; these short synthetic tests do not establish long-context, multi-user or application end-to-end performance.

generation:

26.4 tokens/s

prompt processing (prefill):

685.5 tokens/s

quant:

Q4_K_M (GGUF)

flash attention:

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

Source-reported historical campaign by emihiggins, saved 2026-09-26T23:29:11Z; recovered 2026-10-13. No local model run or reproduction performed. Primary raw result: https://github.com/emihiggins/m5-max-llm-bench/blob/main/data/mac-llm-bench/m5-max_18c-40g_48gb/gguf/deepseek-r1-distill-32b_Q4_K_M_ngl99.json . Source publication commit is 34fee99b6df6129fb3c64255c1bfb179a54ca044. The author's upstream submission, including corrections and limitations, is https://github.com/enescingoz/mac-llm-bench/pull/12 . Hardware: one Apple Mac Studio Mac17,14, M5 Max, 18 CPU cores and 40 GPU cores, 48 GB unified memory; macOS 27.0 build 26A428, bare metal, AC power, High Power energy mode. Source corrects its JSON core labels: the 6/12 split denotes 6 Super and 12 Performance cores, not 12 Efficiency cores. iogpu.wired_limit_mb=32768 is a session limit, not installed RAM or dedicated VRAM. PR reports ambient about 20.6 C, external WD_BLACK SN7100 NVMe in OWC Express 1M2 over USB4, idle Ollama with no model loaded and no other heavy apps. Swap was at most 2 MB when checked every 60 s. No macOS thermal/performance warnings were recorded, but per-model thermal state was not captured. Do not import zero-swap/per-cell thermal claims from the author's separate bandwidth harness. Exact model: bartowski/DeepSeek-R1-Distill-Qwen-32B-GGUF, file DeepSeek-R1-Distill-Qwen-32B-Q4_K_M.gguf, GGUF Q4_K_M. Manifest: https://github.com/emihiggins/m5-max-llm-bench/blob/main/data/mac-llm-bench/m5-max_18c-40g_48gb/gguf-revisions.tsv . It records measured repository revision 1dc8cf9ffa5dd333057ea1b09ccf4772d8726dec and SHA256 bed9b0f551f5b95bf9da5888a48f0f87c37ad6b72519c4cbd775f54ac0b9fc62. The exact revision's HF file page was independently read and its displayed SHA256 matched: https://huggingface.co/bartowski/DeepSeek-R1-Distill-Qwen-32B-GGUF/blob/1dc8cf9ffa5dd333057ea1b09ccf4772d8726dec/DeepSeek-R1-Distill-Qwen-32B-Q4_K_M.gguf . This binds model identity and quantized artifact; today's main is not substituted for the measured revision. Source nominal model size label is 32B. Runtime: llama.cpp Homebrew build 11146, commit 7fe450e19, ggml 0.25.3, Metal. The raw runtime.version accidentally stores a load_backend BLAS log line; the author explicitly corrects it in PR 12 and reports Metal 4 tensor API active. This is not a CPU-only BLAS benchmark. Each model was run in a separate ./bench_gguf.sh --model process because --auto stopped after the first model. Source parameters are n_gpu_layers=99, flash_attention=true, threads=0. The helper omits -t when threads=0; this means runtime default rather than zero running threads. Requested 99 GPU layers is not a measured layer-offload count. Quantization above describes weights, not KV cache. Headline generation is tg128=26.38 tokens/s; prompt processing is pp512=685.53 tokens/s. These are separate native llama-bench microtests from the same campaign, not one 512-input/128-output request, an average across lengths, or server concurrency throughput. All retained native rates in tokens/s: pp128=467.26; pp256=640.43; pp512=685.53; tg128=26.38; tg256=26.32. The pp tests process 128/256/512 synthetic token IDs; the tg tests perform 128/256 one-token decode steps without a preceding 512-token prompt. The context field is intentionally unspecified: these test lengths are not an advertised context window or a long-context setting. Metric method: published mac-llm-bench bench_gguf.sh entrypoint passes -p 128,256,512 -n 128,256 -ngl 99 -fa 1 to llama-bench. It first runs human-readable output, then a separate -o json invocation for saved speed results; those two invocations are not combined. This entrypoint has inline speed/memory logic; the separate lib/run_bench_gguf.sh helper is not the saved-speed entrypoint, so its different display/capture ordering must not be substituted. lib/collect_results.sh maps native n_prompt/n_gen and avg_ts into these speed keys and rounds to 2 decimals. Native llama.cpp code at the reported 7fe450e19 revision times synchronized inference, computes each repetition's token count divided by elapsed nanoseconds, then takes the arithmetic mean of those rates. This is not input divided by TTFT, total wall time, or a pooled count/total-duration calculation. The pinned llama-bench defaults are 5 repetitions, warmup enabled, depth 0, with model memory cleared before each timed repetition; PP uses synthetic random IDs and TG sequential random IDs, without sampling meaningful chat output. This establishes native benchmark semantics. The exact executed shell-harness commit and full raw llama-bench samples, durations, dispersion and command log were not preserved in this result, so the 5-repetition/default-cache interpretation is code/protocol-derived rather than independently counted from raw samples. Current upstream shell code is supporting method evidence, not proof of byte-for-byte historical reproduction. KV precision, actual CPU thread count, realized batch settings, speculation/MTP and quality scores are not separately captured in this JSON; no quality validation or conversational thinking-mode conclusion follows. Memory: file_size_gb=18.49 is the on-disk file size divided by 2^30, despite the GB key. Source peak_rss_gb=18.68 is separately measured process maximum RSS. The published entrypoint obtains RSS with /usr/bin/time -l on a separate llama-bench -p 32 -n 1 invocation, then divides bytes by 2^30. It is not the timed speed test's peak, dedicated VRAM, model-only allocation or installed capacity. The per-record available_ram_gb=13.2 is available system memory at collection, not model usage. Method sources: https://github.com/enescingoz/mac-llm-bench/blob/main/bench_gguf.sh ; https://github.com/enescingoz/mac-llm-bench/blob/main/lib/collect_results.sh ; https://github.com/ggml-org/llama.cpp/blob/7fe450e19/tools/llama-bench/llama-bench.cpp . Source data/documentation license is CC BY 4.0. All values remain source-reported single-machine results; these short synthetic tests do not establish long-context, multi-user or application end-to-end performance.

generation:

56.5 tokens/s

prompt processing (prefill):

1547.5 tokens/s

quant:

Q4_K_M (GGUF)

flash attention:

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

Source-reported historical campaign by emihiggins, saved 2026-09-26T23:16:38Z; recovered 2026-10-13. No local model run or reproduction performed. Primary raw result: https://github.com/emihiggins/m5-max-llm-bench/blob/main/data/mac-llm-bench/m5-max_18c-40g_48gb/gguf/deepseek-r1-distill-14b_Q4_K_M_ngl99.json . Source publication commit is 34fee99b6df6129fb3c64255c1bfb179a54ca044. The author's upstream submission, including corrections and limitations, is https://github.com/enescingoz/mac-llm-bench/pull/12 . Hardware: one Apple Mac Studio Mac17,14, M5 Max, 18 CPU cores and 40 GPU cores, 48 GB unified memory; macOS 27.0 build 26A428, bare metal, AC power, High Power energy mode. Source corrects its JSON core labels: the 6/12 split denotes 6 Super and 12 Performance cores, not 12 Efficiency cores. iogpu.wired_limit_mb=32768 is a session limit, not installed RAM or dedicated VRAM. PR reports ambient about 20.6 C, external WD_BLACK SN7100 NVMe in OWC Express 1M2 over USB4, idle Ollama with no model loaded and no other heavy apps. Swap was at most 2 MB when checked every 60 s. No macOS thermal/performance warnings were recorded, but per-model thermal state was not captured. Do not import zero-swap/per-cell thermal claims from the author's separate bandwidth harness. Exact model: bartowski/DeepSeek-R1-Distill-Qwen-14B-GGUF, file DeepSeek-R1-Distill-Qwen-14B-Q4_K_M.gguf, GGUF Q4_K_M. Manifest: https://github.com/emihiggins/m5-max-llm-bench/blob/main/data/mac-llm-bench/m5-max_18c-40g_48gb/gguf-revisions.tsv . It records measured repository revision 9f5d77d401799416e0702290a691038b44012e0c and SHA256 0b319bd0572f2730bfe11cc751defe82045fad5085b4e60591ac2cd2d9633181. The exact revision's HF file page was independently read and its displayed SHA256 matched: https://huggingface.co/bartowski/DeepSeek-R1-Distill-Qwen-14B-GGUF/blob/9f5d77d401799416e0702290a691038b44012e0c/DeepSeek-R1-Distill-Qwen-14B-Q4_K_M.gguf . This binds model identity and quantized artifact; today's main is not substituted for the measured revision. Source nominal model size label is 14B. Runtime: llama.cpp Homebrew build 11146, commit 7fe450e19, ggml 0.25.3, Metal. The raw runtime.version accidentally stores a load_backend BLAS log line; the author explicitly corrects it in PR 12 and reports Metal 4 tensor API active. This is not a CPU-only BLAS benchmark. Each model was run in a separate ./bench_gguf.sh --model process because --auto stopped after the first model. Source parameters are n_gpu_layers=99, flash_attention=true, threads=0. The helper omits -t when threads=0; this means runtime default rather than zero running threads. Requested 99 GPU layers is not a measured layer-offload count. Quantization above describes weights, not KV cache. Headline generation is tg128=56.48 tokens/s; prompt processing is pp512=1547.46 tokens/s. These are separate native llama-bench microtests from the same campaign, not one 512-input/128-output request, an average across lengths, or server concurrency throughput. All retained native rates in tokens/s: pp128=998.57; pp256=1401.46; pp512=1547.46; tg128=56.48; tg256=56.26. The pp tests process 128/256/512 synthetic token IDs; the tg tests perform 128/256 one-token decode steps without a preceding 512-token prompt. The context field is intentionally unspecified: these test lengths are not an advertised context window or a long-context setting. Metric method: published mac-llm-bench bench_gguf.sh entrypoint passes -p 128,256,512 -n 128,256 -ngl 99 -fa 1 to llama-bench. It first runs human-readable output, then a separate -o json invocation for saved speed results; those two invocations are not combined. This entrypoint has inline speed/memory logic; the separate lib/run_bench_gguf.sh helper is not the saved-speed entrypoint, so its different display/capture ordering must not be substituted. lib/collect_results.sh maps native n_prompt/n_gen and avg_ts into these speed keys and rounds to 2 decimals. Native llama.cpp code at the reported 7fe450e19 revision times synchronized inference, computes each repetition's token count divided by elapsed nanoseconds, then takes the arithmetic mean of those rates. This is not input divided by TTFT, total wall time, or a pooled count/total-duration calculation. The pinned llama-bench defaults are 5 repetitions, warmup enabled, depth 0, with model memory cleared before each timed repetition; PP uses synthetic random IDs and TG sequential random IDs, without sampling meaningful chat output. This establishes native benchmark semantics. The exact executed shell-harness commit and full raw llama-bench samples, durations, dispersion and command log were not preserved in this result, so the 5-repetition/default-cache interpretation is code/protocol-derived rather than independently counted from raw samples. Current upstream shell code is supporting method evidence, not proof of byte-for-byte historical reproduction. KV precision, actual CPU thread count, realized batch settings, speculation/MTP and quality scores are not separately captured in this JSON; no quality validation or conversational thinking-mode conclusion follows. Memory: file_size_gb=8.37 is the on-disk file size divided by 2^30, despite the GB key. Source peak_rss_gb=8.54 is separately measured process maximum RSS. The published entrypoint obtains RSS with /usr/bin/time -l on a separate llama-bench -p 32 -n 1 invocation, then divides bytes by 2^30. It is not the timed speed test's peak, dedicated VRAM, model-only allocation or installed capacity. The per-record available_ram_gb=19.4 is available system memory at collection, not model usage. Method sources: https://github.com/enescingoz/mac-llm-bench/blob/main/bench_gguf.sh ; https://github.com/enescingoz/mac-llm-bench/blob/main/lib/collect_results.sh ; https://github.com/ggml-org/llama.cpp/blob/7fe450e19/tools/llama-bench/llama-bench.cpp . Source data/documentation license is CC BY 4.0. All values remain source-reported single-machine results; these short synthetic tests do not establish long-context, multi-user or application end-to-end performance.

generation:

121.3 tokens/s

prompt processing (prefill):

3351.9 tokens/s

quant:

4-bit (MLX)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

Source-reported historical campaign by emihiggins, measured 2026-09-27T19:32:00Z; recovered 2026-10-13. No new run or local reproduction. Raw source: https://github.com/emihiggins/m5-max-llm-bench/blob/main/data/mac-llm-bench/m5-max_18c-40g_48gb/mlx/DeepSeek-R1-Distill-Qwen-7B-4bit_4bit_ngl99.json . Companion provenance: https://github.com/enescingoz/mac-llm-bench/pull/12 . The source states these are unedited copies of its upstream submission. Data files show public-release commit 34fee99b6df6129fb3c64255c1bfb179a54ca044 (September 27, 2026); the measurement predates discovery. One report groups this model, precision, runtime and hardware; all retained lengths are below. Hardware: one Mac Studio (source PR identifies Mac17,14), Apple M5 Max, 18 CPU cores and 40 GPU cores, 48 GB unified RAM, macOS 27.0 (PR build 26A428), AC power and High Power energy mode. Source CPU JSON labels performance_cores=6 and efficiency_cores=12; author clarifies these are 6 Super plus 12 Performance cores. Reported 614 GB/s is theoretical bandwidth. Installed unified RAM is not dedicated VRAM. The raw available_ram_gb=30.2 is a system-availability snapshot, not installed capacity or model allocation. The source records a 32768 MB iogpu wired limit, swap <=2 MB checked every 60 seconds, no system thermal/performance warning, ambient about 20.6 C, external NVMe model storage, and idle Ollama with no model loaded. Per-model temperature, clock and wall power are unknown. Model: mlx-community/DeepSeek-R1-Distill-Qwen-7B-4bit, source quant=4bit, MLX weights in Safetensors. Recorded fetched HF snapshot: 21848dbf533d2518a1ef895104820d5ee51317ea; manifest: https://github.com/emihiggins/m5-max-llm-bench/blob/main/data/mac-llm-bench/m5-max_18c-40g_48gb/mlx-revisions.tsv . Identity verified individually at https://huggingface.co/mlx-community/DeepSeek-R1-Distill-Qwen-7B-4bit . Current card identifies conversion from deepseek-ai/DeepSeek-R1-Distill-Qwen-7B. The snapshot is source-reported; model files were not downloaded or hashed, and current main must not be substituted for the recorded revision. Source file_size_gb=0 is a serializer placeholder for MLX repo strings, not a zero-byte model. Conversion-library versions on HF describe conversion, not the measured runtime. Runtime: source JSON mlx-lm 0.31.3; source PR additionally reports mlx 0.32.2 and Python 3.12. One model invocation per process using ./bench_mlx.sh --repo <repo> --cleanup. The author reports cleanup path bugs; this does not establish a deleted model cache or cold load. Source's generic n_gpu_layers=99, flash_attention=true, threads=0 fields are hard-coded by the MLX serializer and do not verify actual MLX offload layers, Flash Attention setting or thread count. These feature fields stay unknown rather than borrowing llama.cpp flags. Configured context capacity is not recorded; prompt length is not context-window capacity. All retained source speed values, tokens/s: pp128=2139.57; pp256=2956.62; pp512=3351.85; tg128=121.31; tg256=121.99. Headline uses pp512=3351.85 and tg128=121.31, not a best-of sweep or an average across prompt lengths. Linkage is inferred from the published wrapper's overwrite order, not independently retained per-trial logs. The current wrapper runs (prompt,generation)=(128,128),(128,256),(256,128),(256,256),(512,128), skipping (512,256); it overwrites each pp and tg key on every success. Therefore if all documented calls completed, the retained pp128 is from (128,256), pp256 and tg256 from (256,256), and pp512 and tg128 from (512,128). The exported dictionary loses the other paired rates, trial-level variance and failure details; do not reconstruct them. The historical executable/harness commit is unpinned. The source claims documented commands without deviations; current upstream code establishes method and interpretation, not byte-for-byte historical reproduction. Timing method: mlx-lm v0.31.3 benchmark.py uses seeded random token IDs, batch size 1 by default, disables EOS stopping, performs one warmup then defaults to five measured trials for each configuration, and prints arithmetic means of native response rates. The wrapper defaults to five trials, parses those means and rounds speed to two decimals. The benchmark is synthetic text-token throughput, not conversational quality or end-to-end request throughput. Published versioned stream_generate measures prompt.size divided by elapsed time to its first generated token, then resets the timer and reports (n+1)/elapsed for generation. Thus PP is a native engine phase rate with first-token work included; it is not input tokens divided by client/network TTFT, load time or total request wall time. TG follows the library's own numerator/timer convention, not a newly recomputed rate. Each generate call makes a new prompt cache when none is supplied, while the model is already loaded and warmed. The source logs do not independently validate cache state; avoid a claim of proven cold prefill. No chat template, --no-think, speculative draft, MTP or quantized KV option is passed by this published benchmark. Default greedy sampler, unquantized KV and prefill step 2048 are code-derived defaults, not separately recorded run flags; corresponding form fields remain unknown. Memory: source memory.peak_rss_gb=4.849. Despite that field name, the MLX exporter passes the highest native peak_memory mean among configurations. v0.31.3 stream_generate uses mx.get_peak_memory()/1e9, an MLX allocator high-water measurement in decimal GB, not operating-system process RSS, installed RAM, dedicated VRAM or necessarily a reset per-trial peak. No RSS/model-size claim is made. Method sources: https://github.com/enescingoz/mac-llm-bench/blob/main/lib/run_bench_mlx.sh ; https://github.com/enescingoz/mac-llm-bench/blob/main/bench_mlx.sh ; https://github.com/enescingoz/mac-llm-bench/blob/main/lib/collect_results.sh ; https://github.com/ml-explore/mlx-lm/blob/v0.31.3/mlx_lm/benchmark.py ; https://github.com/ml-explore/mlx-lm/blob/v0.31.3/mlx_lm/generate.py . No HumanEval+ or other model-quality score is included for this campaign.

generation:

28.3 tokens/s

prompt processing (prefill):

731.4 tokens/s

quant:

4-bit (MLX)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

Source-reported historical campaign by emihiggins, measured 2026-09-27T19:29:13Z; recovered 2026-10-13. No new run or local reproduction. Raw source: https://github.com/emihiggins/m5-max-llm-bench/blob/main/data/mac-llm-bench/m5-max_18c-40g_48gb/mlx/DeepSeek-R1-Distill-Qwen-32B-4bit_4bit_ngl99.json . Companion provenance: https://github.com/enescingoz/mac-llm-bench/pull/12 . The source states these are unedited copies of its upstream submission. Data files show public-release commit 34fee99b6df6129fb3c64255c1bfb179a54ca044 (September 27, 2026); the measurement predates discovery. One report groups this model, precision, runtime and hardware; all retained lengths are below. Hardware: one Mac Studio (source PR identifies Mac17,14), Apple M5 Max, 18 CPU cores and 40 GPU cores, 48 GB unified RAM, macOS 27.0 (PR build 26A428), AC power and High Power energy mode. Source CPU JSON labels performance_cores=6 and efficiency_cores=12; author clarifies these are 6 Super plus 12 Performance cores. Reported 614 GB/s is theoretical bandwidth. Installed unified RAM is not dedicated VRAM. The raw available_ram_gb=30.9 is a system-availability snapshot, not installed capacity or model allocation. The source records a 32768 MB iogpu wired limit, swap <=2 MB checked every 60 seconds, no system thermal/performance warning, ambient about 20.6 C, external NVMe model storage, and idle Ollama with no model loaded. Per-model temperature, clock and wall power are unknown. Model: mlx-community/DeepSeek-R1-Distill-Qwen-32B-4bit, source quant=4bit, MLX weights in Safetensors. Recorded fetched HF snapshot: 4e0d3848a0ad8f9fb54638891e4928f04fcca978; manifest: https://github.com/emihiggins/m5-max-llm-bench/blob/main/data/mac-llm-bench/m5-max_18c-40g_48gb/mlx-revisions.tsv . Identity verified individually at https://huggingface.co/mlx-community/DeepSeek-R1-Distill-Qwen-32B-4bit . Current card identifies conversion from deepseek-ai/DeepSeek-R1-Distill-Qwen-32B. The snapshot is source-reported; model files were not downloaded or hashed, and current main must not be substituted for the recorded revision. Source file_size_gb=0 is a serializer placeholder for MLX repo strings, not a zero-byte model. Conversion-library versions on HF describe conversion, not the measured runtime. Runtime: source JSON mlx-lm 0.31.3; source PR additionally reports mlx 0.32.2 and Python 3.12. One model invocation per process using ./bench_mlx.sh --repo <repo> --cleanup. The author reports cleanup path bugs; this does not establish a deleted model cache or cold load. Source's generic n_gpu_layers=99, flash_attention=true, threads=0 fields are hard-coded by the MLX serializer and do not verify actual MLX offload layers, Flash Attention setting or thread count. These feature fields stay unknown rather than borrowing llama.cpp flags. Configured context capacity is not recorded; prompt length is not context-window capacity. All retained source speed values, tokens/s: pp128=493.38; pp256=616.57; pp512=731.39; tg128=28.3; tg256=28.44. Headline uses pp512=731.39 and tg128=28.3, not a best-of sweep or an average across prompt lengths. Linkage is inferred from the published wrapper's overwrite order, not independently retained per-trial logs. The current wrapper runs (prompt,generation)=(128,128),(128,256),(256,128),(256,256),(512,128), skipping (512,256); it overwrites each pp and tg key on every success. Therefore if all documented calls completed, the retained pp128 is from (128,256), pp256 and tg256 from (256,256), and pp512 and tg128 from (512,128). The exported dictionary loses the other paired rates, trial-level variance and failure details; do not reconstruct them. The historical executable/harness commit is unpinned. The source claims documented commands without deviations; current upstream code establishes method and interpretation, not byte-for-byte historical reproduction. Timing method: mlx-lm v0.31.3 benchmark.py uses seeded random token IDs, batch size 1 by default, disables EOS stopping, performs one warmup then defaults to five measured trials for each configuration, and prints arithmetic means of native response rates. The wrapper defaults to five trials, parses those means and rounds speed to two decimals. The benchmark is synthetic text-token throughput, not conversational quality or end-to-end request throughput. Published versioned stream_generate measures prompt.size divided by elapsed time to its first generated token, then resets the timer and reports (n+1)/elapsed for generation. Thus PP is a native engine phase rate with first-token work included; it is not input tokens divided by client/network TTFT, load time or total request wall time. TG follows the library's own numerator/timer convention, not a newly recomputed rate. Each generate call makes a new prompt cache when none is supplied, while the model is already loaded and warmed. The source logs do not independently validate cache state; avoid a claim of proven cold prefill. No chat template, --no-think, speculative draft, MTP or quantized KV option is passed by this published benchmark. Default greedy sampler, unquantized KV and prefill step 2048 are code-derived defaults, not separately recorded run flags; corresponding form fields remain unknown. Memory: source memory.peak_rss_gb=19.119. Despite that field name, the MLX exporter passes the highest native peak_memory mean among configurations. v0.31.3 stream_generate uses mx.get_peak_memory()/1e9, an MLX allocator high-water measurement in decimal GB, not operating-system process RSS, installed RAM, dedicated VRAM or necessarily a reset per-trial peak. No RSS/model-size claim is made. Method sources: https://github.com/enescingoz/mac-llm-bench/blob/main/lib/run_bench_mlx.sh ; https://github.com/enescingoz/mac-llm-bench/blob/main/bench_mlx.sh ; https://github.com/enescingoz/mac-llm-bench/blob/main/lib/collect_results.sh ; https://github.com/ml-explore/mlx-lm/blob/v0.31.3/mlx_lm/benchmark.py ; https://github.com/ml-explore/mlx-lm/blob/v0.31.3/mlx_lm/generate.py . No HumanEval+ or other model-quality score is included for this campaign.

generation:

60.7 tokens/s

prompt processing (prefill):

1714.8 tokens/s

quant:

4-bit (MLX)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

Source-reported historical campaign by emihiggins, measured 2026-09-27T19:18:25Z; recovered 2026-10-13. No new run or local reproduction. Raw source: https://github.com/emihiggins/m5-max-llm-bench/blob/main/data/mac-llm-bench/m5-max_18c-40g_48gb/mlx/DeepSeek-R1-Distill-Qwen-14B-4bit_4bit_ngl99.json . Companion provenance: https://github.com/enescingoz/mac-llm-bench/pull/12 . The source states these are unedited copies of its upstream submission. Data files show public-release commit 34fee99b6df6129fb3c64255c1bfb179a54ca044 (September 27, 2026); the measurement predates discovery. One report groups this model, precision, runtime and hardware; all retained lengths are below. Hardware: one Mac Studio (source PR identifies Mac17,14), Apple M5 Max, 18 CPU cores and 40 GPU cores, 48 GB unified RAM, macOS 27.0 (PR build 26A428), AC power and High Power energy mode. Source CPU JSON labels performance_cores=6 and efficiency_cores=12; author clarifies these are 6 Super plus 12 Performance cores. Reported 614 GB/s is theoretical bandwidth. Installed unified RAM is not dedicated VRAM. The raw available_ram_gb=27.7 is a system-availability snapshot, not installed capacity or model allocation. The source records a 32768 MB iogpu wired limit, swap <=2 MB checked every 60 seconds, no system thermal/performance warning, ambient about 20.6 C, external NVMe model storage, and idle Ollama with no model loaded. Per-model temperature, clock and wall power are unknown. Model: mlx-community/DeepSeek-R1-Distill-Qwen-14B-4bit, source quant=4bit, MLX weights in Safetensors. Recorded fetched HF snapshot: b39611728ed604538a2a18212ce33b02f99d8de2; manifest: https://github.com/emihiggins/m5-max-llm-bench/blob/main/data/mac-llm-bench/m5-max_18c-40g_48gb/mlx-revisions.tsv . Identity verified individually at https://huggingface.co/mlx-community/DeepSeek-R1-Distill-Qwen-14B-4bit . Current card identifies conversion from deepseek-ai/DeepSeek-R1-Distill-Qwen-14B. The snapshot is source-reported; model files were not downloaded or hashed, and current main must not be substituted for the recorded revision. Source file_size_gb=0 is a serializer placeholder for MLX repo strings, not a zero-byte model. Conversion-library versions on HF describe conversion, not the measured runtime. Runtime: source JSON mlx-lm 0.31.3; source PR additionally reports mlx 0.32.2 and Python 3.12. One model invocation per process using ./bench_mlx.sh --repo <repo> --cleanup. The author reports cleanup path bugs; this does not establish a deleted model cache or cold load. Source's generic n_gpu_layers=99, flash_attention=true, threads=0 fields are hard-coded by the MLX serializer and do not verify actual MLX offload layers, Flash Attention setting or thread count. These feature fields stay unknown rather than borrowing llama.cpp flags. Configured context capacity is not recorded; prompt length is not context-window capacity. All retained source speed values, tokens/s: pp128=1079.85; pp256=1355.64; pp512=1714.77; tg128=60.69; tg256=61.23. Headline uses pp512=1714.77 and tg128=60.69, not a best-of sweep or an average across prompt lengths. Linkage is inferred from the published wrapper's overwrite order, not independently retained per-trial logs. The current wrapper runs (prompt,generation)=(128,128),(128,256),(256,128),(256,256),(512,128), skipping (512,256); it overwrites each pp and tg key on every success. Therefore if all documented calls completed, the retained pp128 is from (128,256), pp256 and tg256 from (256,256), and pp512 and tg128 from (512,128). The exported dictionary loses the other paired rates, trial-level variance and failure details; do not reconstruct them. The historical executable/harness commit is unpinned. The source claims documented commands without deviations; current upstream code establishes method and interpretation, not byte-for-byte historical reproduction. Timing method: mlx-lm v0.31.3 benchmark.py uses seeded random token IDs, batch size 1 by default, disables EOS stopping, performs one warmup then defaults to five measured trials for each configuration, and prints arithmetic means of native response rates. The wrapper defaults to five trials, parses those means and rounds speed to two decimals. The benchmark is synthetic text-token throughput, not conversational quality or end-to-end request throughput. Published versioned stream_generate measures prompt.size divided by elapsed time to its first generated token, then resets the timer and reports (n+1)/elapsed for generation. Thus PP is a native engine phase rate with first-token work included; it is not input tokens divided by client/network TTFT, load time or total request wall time. TG follows the library's own numerator/timer convention, not a newly recomputed rate. Each generate call makes a new prompt cache when none is supplied, while the model is already loaded and warmed. The source logs do not independently validate cache state; avoid a claim of proven cold prefill. No chat template, --no-think, speculative draft, MTP or quantized KV option is passed by this published benchmark. Default greedy sampler, unquantized KV and prefill step 2048 are code-derived defaults, not separately recorded run flags; corresponding form fields remain unknown. Memory: source memory.peak_rss_gb=8.929. Despite that field name, the MLX exporter passes the highest native peak_memory mean among configurations. v0.31.3 stream_generate uses mx.get_peak_memory()/1e9, an MLX allocator high-water measurement in decimal GB, not operating-system process RSS, installed RAM, dedicated VRAM or necessarily a reset per-trial peak. No RSS/model-size claim is made. Method sources: https://github.com/enescingoz/mac-llm-bench/blob/main/lib/run_bench_mlx.sh ; https://github.com/enescingoz/mac-llm-bench/blob/main/bench_mlx.sh ; https://github.com/enescingoz/mac-llm-bench/blob/main/lib/collect_results.sh ; https://github.com/ml-explore/mlx-lm/blob/v0.31.3/mlx_lm/benchmark.py ; https://github.com/ml-explore/mlx-lm/blob/v0.31.3/mlx_lm/generate.py . No HumanEval+ or other model-quality score is included for this campaign.