Local model performance on your hardware

generation:

93.0 tokens/s

prompt processing (prefill):

468.6 tokens/s

quant:

4-bit (source config) (MLX)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

First-party historical Patrick Whelan benchmark, inspected 2026-10-13. Source snapshot 11f3f8f931dc97a71499e7b9c2583d010a3c160b verified by archive PAX comment and GitHub currentOid; pinned runner/config/hardware/VLM files byte-match archive. This pins the published evidence, not the executable used in April. No inference rerun or model-weight download. Hardware: one participating Apple M2 Max integrated GPU, 30 GPU cores, 12 CPU cores and 64 GB shared unified memory. Raw system data confirms those values; hardware profile explicitly says no discrete GPU and all inference on the M2 Max GPU. Shared 64 GB is system capacity, not dedicated VRAM or resident model use. Raw environment: Darwin 25.3.0 / ARM64_T6020 and Python 3.14.3. Do not borrow stale hardware-profile Sequoia 15.x or ai-gateway/vllm-mlx labels. Power, thermals, clocks, swap, wired-memory limit, GPU utilization and competing workload are unknown. Dates: raw system_info.timestamp=2026-04-02T23:30:07.406931+00:00; this is collected once before a run list, so it is a campaign/system-capture timestamp, not exact model start. Result filename nemotron-3-nano_default_20260403_011601.json uses local completion time with no timezone recorded. README updated 2026-04-04 is a publication label, not every measurement date. Configuration: direct python -m mlx_lm benchmark on Apple MLX/Metal, no HTTP server involved. MLX/MLX-LM/MLX-VLM installed versions and exact runtime commit are not retained in JSON. Requested prompt length 512 and generation length/cap 128, printed batch_size=1; three measured trials retained after discarding Trial 1, plus an internal/explicit warmup. Achieved token-count telemetry is not stored, so actual input/output fields remain unknown. Source context_size=4096, threads=8 and gpu_layers=-1 are explicitly unused for MLX and are not recorded here as active allocation, CPU threads or layer placement. KV/cache precision, prefill chunk size, sampling settings, thinking behavior and memory-cache reuse are unreported for the historical runtime; no drafter is passed by the wrapper. Native rates: PP 468.64 tok/s (sample SD 0.11; retained [468.515, 468.663, 468.735]); TG 93.03 tok/s (sample SD 0.15; retained [93.197, 93.003, 92.9]). Source aggregation rounds the three-trial arithmetic mean/sample SD to two decimals after parsing three-decimal CLI values. All values, means and SD were checked against raw Trial 2-4. CLI Averages includes all four trials and therefore is not the published three-trial metric. No pooling across engines/models. Timing exclusions: raw time_to_first_token_s mean 1.09 is explicitly computed as 512/prompt_tps by runner lines 718-719, not independently measured client or request TTFT. TTFT field is intentionally empty. Overall wall_time_s=2754.08 measures the entire subprocess and can include HF downloads, model load and warmup; it is not inference-only latency or end-to-end response timing. As a reference audit only, official MLX-LM tag v0.31.2 benchmark.py uses batch 1 random token IDs, internal warmup, disables EOS, and forwards stream_generate rates. generate.py starts the prompt timer before generation and closes it on the first generated token; PP is prompt.size/prompt_time. TG counts n+1 tokens over elapsed time after that boundary, so its numerator includes the first token while its first-token time is in PP. These are engine-reported native rates, not transport/client rates. The actual installed MLX-LM revision is absent; v0.31.2 is corroborating implementation, not a tested-run version. Artifact: mlx-community/NVIDIA-Nemotron-3-Nano-30B-A3B-4bit. Source-configured HF artifact joined by model key; raw JSON omits repository/digest; historical tested revision unknown. Raw quant is "N/A" because the MLX run dictionary lacks a quant key; quantization identification comes from source config/profile plus HF metadata, not that raw field. No weight digest is logged. Current artifact metadata inspected 2026-10-13 at HF revision 832f602eba5d22436c258c1462bdedc5afddb42b: MLX 4-bit group 64 affine. This revision is a current metadata reference, not a historical run pin. Model-card conversion-tool versions are not installed benchmark engine versions. Model context capacities from current HF metadata are not active benchmark allocations. Memory: raw retained peak_memory values [18.973, 18.973, 18.974], maximum 18.974 in engine output units. Wrapper overwrites peak_vram_mb with this value times 1024 (19429.4), but leaves a stale unified_memory_system_wide note. It is MLX allocator/process peak reporting, not dedicated VRAM or system-wide resident usage. Reference MLX-LM v0.31.2 reports decimal GB (bytes/1e9), so wrapper times-1024 conversion may be unit-inconsistent; actual installed version is unknown. Dedicated-VRAM and memory-used fields remain empty. Config size_gb is an estimate, not measured disk bytes, and is not substituted for residency. No Mac power measurements. Sources: https://github.com/patrickwhelan-uk/llm-inference-benchmarks/blob/11f3f8f931dc97a71499e7b9c2583d010a3c160b/results/m2-mac-studio/nemotron-3-nano_default_20260403_011601.json https://github.com/patrickwhelan-uk/llm-inference-benchmarks/blob/11f3f8f931dc97a71499e7b9c2583d010a3c160b/config-m2-mac-studio.yaml https://github.com/patrickwhelan-uk/llm-inference-benchmarks/blob/11f3f8f931dc97a71499e7b9c2583d010a3c160b/run_benchmarks.py#L624-L738 https://github.com/patrickwhelan-uk/llm-inference-benchmarks/blob/11f3f8f931dc97a71499e7b9c2583d010a3c160b/hardware/m2-mac-studio.md https://huggingface.co/mlx-community/NVIDIA-Nemotron-3-Nano-30B-A3B-4bit/blob/832f602eba5d22436c258c1462bdedc5afddb42b/config.json https://github.com/ml-explore/mlx-lm/blob/v0.31.2/mlx_lm/benchmark.py https://github.com/ml-explore/mlx-lm/blob/v0.31.2/mlx_lm/generate.py#L658-L753 Structured PP/TG fields are rounded to 0.1 tokens/s; the source means, sample deviations and all three retained trial values above preserve the published precision.

generation:

18.0 tokens/s

prompt processing (prefill):

126.8 tokens/s

quant:

4-bit (source config) (MLX)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

First-party historical Patrick Whelan benchmark, inspected 2026-10-13. Source snapshot 11f3f8f931dc97a71499e7b9c2583d010a3c160b verified by archive PAX comment and GitHub currentOid; pinned runner/config/hardware/VLM files byte-match archive. This pins the published evidence, not the executable used in April. No inference rerun or model-weight download. Hardware: one participating Apple M2 Max integrated GPU, 30 GPU cores, 12 CPU cores and 64 GB shared unified memory. Raw system data confirms those values; hardware profile explicitly says no discrete GPU and all inference on the M2 Max GPU. Shared 64 GB is system capacity, not dedicated VRAM or resident model use. Raw environment: Darwin 25.3.0 / ARM64_T6020 and Python 3.14.3. Do not borrow stale hardware-profile Sequoia 15.x or ai-gateway/vllm-mlx labels. Power, thermals, clocks, swap, wired-memory limit, GPU utilization and competing workload are unknown. Dates: raw system_info.timestamp=2026-04-02T21:00:05.046210+00:00; this is collected once before a run list, so it is a campaign/system-capture timestamp, not exact model start. Result filename qwen2.5-coder-32b_default_20260402_235731.json uses local completion time with no timezone recorded. README updated 2026-04-04 is a publication label, not every measurement date. Configuration: direct python -m mlx_lm benchmark on Apple MLX/Metal, no HTTP server involved. MLX/MLX-LM/MLX-VLM installed versions and exact runtime commit are not retained in JSON. Requested prompt length 512 and generation length/cap 128, printed batch_size=1; three measured trials retained after discarding Trial 1, plus an internal/explicit warmup. Achieved token-count telemetry is not stored, so actual input/output fields remain unknown. Source context_size=4096, threads=8 and gpu_layers=-1 are explicitly unused for MLX and are not recorded here as active allocation, CPU threads or layer placement. KV/cache precision, prefill chunk size, sampling settings, thinking behavior and memory-cache reuse are unreported for the historical runtime; no drafter is passed by the wrapper. Native rates: PP 126.82 tok/s (sample SD 0.01; retained [126.832, 126.812, 126.803]); TG 17.96 tok/s (sample SD 0.01; retained [17.957, 17.965, 17.951]). Source aggregation rounds the three-trial arithmetic mean/sample SD to two decimals after parsing three-decimal CLI values. All values, means and SD were checked against raw Trial 2-4. CLI Averages includes all four trials and therefore is not the published three-trial metric. No pooling across engines/models. Timing exclusions: raw time_to_first_token_s mean 4.04 is explicitly computed as 512/prompt_tps by runner lines 718-719, not independently measured client or request TTFT. TTFT field is intentionally empty. Overall wall_time_s=66.21 measures the entire subprocess and can include HF downloads, model load and warmup; it is not inference-only latency or end-to-end response timing. As a reference audit only, official MLX-LM tag v0.31.2 benchmark.py uses batch 1 random token IDs, internal warmup, disables EOS, and forwards stream_generate rates. generate.py starts the prompt timer before generation and closes it on the first generated token; PP is prompt.size/prompt_time. TG counts n+1 tokens over elapsed time after that boundary, so its numerator includes the first token while its first-token time is in PP. These are engine-reported native rates, not transport/client rates. The actual installed MLX-LM revision is absent; v0.31.2 is corroborating implementation, not a tested-run version. Artifact: mlx-community/Qwen2.5-Coder-32B-Instruct-4bit. Source-configured HF artifact joined by model key; raw JSON omits repository/digest; historical tested revision unknown. Raw quant is "N/A" because the MLX run dictionary lacks a quant key; quantization identification comes from source config/profile plus HF metadata, not that raw field. No weight digest is logged. Current artifact metadata inspected 2026-10-13 at HF revision d1e3b690c8e225d7795bccddf971ca6be68b2012: MLX 4-bit group 64 (mode not explicitly named in config). This revision is a current metadata reference, not a historical run pin. Model-card conversion-tool versions are not installed benchmark engine versions. Model context capacities from current HF metadata are not active benchmark allocations. Memory: raw retained peak_memory values [19.119, 19.119, 19.12], maximum 19.12 in engine output units. Wrapper overwrites peak_vram_mb with this value times 1024 (19578.9), but leaves a stale unified_memory_system_wide note. It is MLX allocator/process peak reporting, not dedicated VRAM or system-wide resident usage. Reference MLX-LM v0.31.2 reports decimal GB (bytes/1e9), so wrapper times-1024 conversion may be unit-inconsistent; actual installed version is unknown. Dedicated-VRAM and memory-used fields remain empty. Config size_gb is an estimate, not measured disk bytes, and is not substituted for residency. No Mac power measurements. Sources: https://github.com/patrickwhelan-uk/llm-inference-benchmarks/blob/11f3f8f931dc97a71499e7b9c2583d010a3c160b/results/m2-mac-studio/qwen2.5-coder-32b_default_20260402_235731.json https://github.com/patrickwhelan-uk/llm-inference-benchmarks/blob/11f3f8f931dc97a71499e7b9c2583d010a3c160b/config-m2-mac-studio.yaml https://github.com/patrickwhelan-uk/llm-inference-benchmarks/blob/11f3f8f931dc97a71499e7b9c2583d010a3c160b/run_benchmarks.py#L624-L738 https://github.com/patrickwhelan-uk/llm-inference-benchmarks/blob/11f3f8f931dc97a71499e7b9c2583d010a3c160b/hardware/m2-mac-studio.md https://huggingface.co/mlx-community/Qwen2.5-Coder-32B-Instruct-4bit/blob/d1e3b690c8e225d7795bccddf971ca6be68b2012/config.json https://github.com/ml-explore/mlx-lm/blob/v0.31.2/mlx_lm/benchmark.py https://github.com/ml-explore/mlx-lm/blob/v0.31.2/mlx_lm/generate.py#L658-L753 Structured PP/TG fields are rounded to 0.1 tokens/s; the source means, sample deviations and all three retained trial values above preserve the published precision.

generation:

88.9 tokens/s

prompt processing (prefill):

521.5 tokens/s

quant:

4-bit (source config) (MLX)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

First-party historical Patrick Whelan benchmark, inspected 2026-10-13. Source snapshot 11f3f8f931dc97a71499e7b9c2583d010a3c160b verified by archive PAX comment and GitHub currentOid; pinned runner/config/hardware/VLM files byte-match archive. This pins the published evidence, not the executable used in April. No inference rerun or model-weight download. Hardware: one participating Apple M2 Max integrated GPU, 30 GPU cores, 12 CPU cores and 64 GB shared unified memory. Raw system data confirms those values; hardware profile explicitly says no discrete GPU and all inference on the M2 Max GPU. Shared 64 GB is system capacity, not dedicated VRAM or resident model use. Raw environment: Darwin 25.3.0 / ARM64_T6020 and Python 3.14.3. Do not borrow stale hardware-profile Sequoia 15.x or ai-gateway/vllm-mlx labels. Power, thermals, clocks, swap, wired-memory limit, GPU utilization and competing workload are unknown. Dates: raw system_info.timestamp=2026-04-04T00:11:47.349025+00:00; this is collected once before a run list, so it is a campaign/system-capture timestamp, not exact model start. Result filename qwen3.5-35b-moe_default_20260404_011210.json uses local completion time with no timezone recorded. README updated 2026-04-04 is a publication label, not every measurement date. Configuration: direct python -m mlx_lm benchmark on Apple MLX/Metal, no HTTP server involved. MLX/MLX-LM/MLX-VLM installed versions and exact runtime commit are not retained in JSON. Requested prompt length 512 and generation length/cap 128, printed batch_size=1; three measured trials retained after discarding Trial 1, plus an internal/explicit warmup. Achieved token-count telemetry is not stored, so actual input/output fields remain unknown. Source context_size=4096, threads=8 and gpu_layers=-1 are explicitly unused for MLX and are not recorded here as active allocation, CPU threads or layer placement. KV/cache precision, prefill chunk size, sampling settings, thinking behavior and memory-cache reuse are unreported for the historical runtime; no drafter is passed by the wrapper. Native rates: PP 521.47 tok/s (sample SD 3.71; retained [524.95, 517.56, 521.899]); TG 88.85 tok/s (sample SD 0.52; retained [89.255, 88.27, 89.028]). Source aggregation rounds the three-trial arithmetic mean/sample SD to two decimals after parsing three-decimal CLI values. All values, means and SD were checked against raw Trial 2-4. CLI Averages includes all four trials and therefore is not the published three-trial metric. No pooling across engines/models. Timing exclusions: raw time_to_first_token_s mean 0.98 is explicitly computed as 512/prompt_tps by runner lines 718-719, not independently measured client or request TTFT. TTFT field is intentionally empty. Overall wall_time_s=23.07 measures the entire subprocess and can include HF downloads, model load and warmup; it is not inference-only latency or end-to-end response timing. As a reference audit only, official MLX-LM tag v0.31.2 benchmark.py uses batch 1 random token IDs, internal warmup, disables EOS, and forwards stream_generate rates. generate.py starts the prompt timer before generation and closes it on the first generated token; PP is prompt.size/prompt_time. TG counts n+1 tokens over elapsed time after that boundary, so its numerator includes the first token while its first-token time is in PP. These are engine-reported native rates, not transport/client rates. The actual installed MLX-LM revision is absent; v0.31.2 is corroborating implementation, not a tested-run version. Artifact: mlx-community/Qwen3.5-35B-A3B-4bit. Source-configured HF artifact joined by model key; raw JSON omits repository/digest; historical tested revision unknown. Raw quant is "N/A" because the MLX run dictionary lacks a quant key; quantization identification comes from source config/profile plus HF metadata, not that raw field. No weight digest is logged. Current artifact metadata inspected 2026-10-13 at HF revision 1e20fd8d42056f870933bf98ca6211024744f7ec: MLX 4-bit group 64 affine. This revision is a current metadata reference, not a historical run pin. Model-card conversion-tool versions are not installed benchmark engine versions. Model context capacities from current HF metadata are not active benchmark allocations. Memory: raw retained peak_memory values [20.217, 20.217, 20.218], maximum 20.218 in engine output units. Wrapper overwrites peak_vram_mb with this value times 1024 (20703.2), but leaves a stale unified_memory_system_wide note. It is MLX allocator/process peak reporting, not dedicated VRAM or system-wide resident usage. Reference MLX-LM v0.31.2 reports decimal GB (bytes/1e9), so wrapper times-1024 conversion may be unit-inconsistent; actual installed version is unknown. Dedicated-VRAM and memory-used fields remain empty. Config size_gb is an estimate, not measured disk bytes, and is not substituted for residency. No Mac power measurements. Sources: https://github.com/patrickwhelan-uk/llm-inference-benchmarks/blob/11f3f8f931dc97a71499e7b9c2583d010a3c160b/results/m2-mac-studio/qwen3.5-35b-moe_default_20260404_011210.json https://github.com/patrickwhelan-uk/llm-inference-benchmarks/blob/11f3f8f931dc97a71499e7b9c2583d010a3c160b/config-m2-mac-studio.yaml https://github.com/patrickwhelan-uk/llm-inference-benchmarks/blob/11f3f8f931dc97a71499e7b9c2583d010a3c160b/run_benchmarks.py#L624-L738 https://github.com/patrickwhelan-uk/llm-inference-benchmarks/blob/11f3f8f931dc97a71499e7b9c2583d010a3c160b/hardware/m2-mac-studio.md https://huggingface.co/mlx-community/Qwen3.5-35B-A3B-4bit/blob/1e20fd8d42056f870933bf98ca6211024744f7ec/config.json https://github.com/ml-explore/mlx-lm/blob/v0.31.2/mlx_lm/benchmark.py https://github.com/ml-explore/mlx-lm/blob/v0.31.2/mlx_lm/generate.py#L658-L753 Structured PP/TG fields are rounded to 0.1 tokens/s; the source means, sample deviations and all three retained trial values above preserve the published precision.

generation:

20.7 tokens/s

prompt processing (prefill):

91.4 tokens/s

quant:

4-bit (source config) (MLX)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

First-party historical Patrick Whelan benchmark, inspected 2026-10-13. Source snapshot 11f3f8f931dc97a71499e7b9c2583d010a3c160b verified by archive PAX comment and GitHub currentOid; pinned runner/config/hardware/VLM files byte-match archive. This pins the published evidence, not the executable used in April. No inference rerun or model-weight download. Hardware: one participating Apple M2 Max integrated GPU, 30 GPU cores, 12 CPU cores and 64 GB shared unified memory. Raw system data confirms those values; hardware profile explicitly says no discrete GPU and all inference on the M2 Max GPU. Shared 64 GB is system capacity, not dedicated VRAM or resident model use. Raw environment: Darwin 25.3.0 / ARM64_T6020 and Python 3.14.3. Do not borrow stale hardware-profile Sequoia 15.x or ai-gateway/vllm-mlx labels. Power, thermals, clocks, swap, wired-memory limit, GPU utilization and competing workload are unknown. Dates: raw system_info.timestamp=2026-04-03T23:29:40.612040+00:00; this is collected once before a run list, so it is a campaign/system-capture timestamp, not exact model start. Result filename qwen3.5-27b_default_20260404_011138.json uses local completion time with no timezone recorded. README updated 2026-04-04 is a publication label, not every measurement date. Configuration: direct python -m mlx_lm benchmark on Apple MLX/Metal, no HTTP server involved. MLX/MLX-LM/MLX-VLM installed versions and exact runtime commit are not retained in JSON. Requested prompt length 512 and generation length/cap 128, printed batch_size=1; three measured trials retained after discarding Trial 1, plus an internal/explicit warmup. Achieved token-count telemetry is not stored, so actual input/output fields remain unknown. Source context_size=4096, threads=8 and gpu_layers=-1 are explicitly unused for MLX and are not recorded here as active allocation, CPU threads or layer placement. KV/cache precision, prefill chunk size, sampling settings, thinking behavior and memory-cache reuse are unreported for the historical runtime; no drafter is passed by the wrapper. Native rates: PP 91.36 tok/s (sample SD 0.03; retained [91.33, 91.348, 91.388]); TG 20.71 tok/s (sample SD 0.0; retained [20.707, 20.703, 20.708]). Source aggregation rounds the three-trial arithmetic mean/sample SD to two decimals after parsing three-decimal CLI values. All values, means and SD were checked against raw Trial 2-4. CLI Averages includes all four trials and therefore is not the published three-trial metric. No pooling across engines/models. Timing exclusions: raw time_to_first_token_s mean 5.6 is explicitly computed as 512/prompt_tps by runner lines 718-719, not independently measured client or request TTFT. TTFT field is intentionally empty. Overall wall_time_s=2517.99 measures the entire subprocess and can include HF downloads, model load and warmup; it is not inference-only latency or end-to-end response timing. As a reference audit only, official MLX-LM tag v0.31.2 benchmark.py uses batch 1 random token IDs, internal warmup, disables EOS, and forwards stream_generate rates. generate.py starts the prompt timer before generation and closes it on the first generated token; PP is prompt.size/prompt_time. TG counts n+1 tokens over elapsed time after that boundary, so its numerator includes the first token while its first-token time is in PP. These are engine-reported native rates, not transport/client rates. The actual installed MLX-LM revision is absent; v0.31.2 is corroborating implementation, not a tested-run version. Artifact: mlx-community/Qwen3.5-27B-4bit. Source-configured HF artifact joined by model key; raw JSON omits repository/digest; historical tested revision unknown. Raw quant is "N/A" because the MLX run dictionary lacks a quant key; quantization identification comes from source config/profile plus HF metadata, not that raw field. No weight digest is logged. Current artifact metadata inspected 2026-10-13 at HF revision 45797d2985a12c55e6473686e9ea91b95e959553: MLX 4-bit group 64 affine. This revision is a current metadata reference, not a historical run pin. Model-card conversion-tool versions are not installed benchmark engine versions. Model context capacities from current HF metadata are not active benchmark allocations. Memory: raw retained peak_memory values [16.301, 16.301, 16.302], maximum 16.302 in engine output units. Wrapper overwrites peak_vram_mb with this value times 1024 (16693.2), but leaves a stale unified_memory_system_wide note. It is MLX allocator/process peak reporting, not dedicated VRAM or system-wide resident usage. Reference MLX-LM v0.31.2 reports decimal GB (bytes/1e9), so wrapper times-1024 conversion may be unit-inconsistent; actual installed version is unknown. Dedicated-VRAM and memory-used fields remain empty. Config size_gb is an estimate, not measured disk bytes, and is not substituted for residency. No Mac power measurements. Sources: https://github.com/patrickwhelan-uk/llm-inference-benchmarks/blob/11f3f8f931dc97a71499e7b9c2583d010a3c160b/results/m2-mac-studio/qwen3.5-27b_default_20260404_011138.json https://github.com/patrickwhelan-uk/llm-inference-benchmarks/blob/11f3f8f931dc97a71499e7b9c2583d010a3c160b/config-m2-mac-studio.yaml https://github.com/patrickwhelan-uk/llm-inference-benchmarks/blob/11f3f8f931dc97a71499e7b9c2583d010a3c160b/run_benchmarks.py#L624-L738 https://github.com/patrickwhelan-uk/llm-inference-benchmarks/blob/11f3f8f931dc97a71499e7b9c2583d010a3c160b/hardware/m2-mac-studio.md https://huggingface.co/mlx-community/Qwen3.5-27B-4bit/blob/45797d2985a12c55e6473686e9ea91b95e959553/config.json https://github.com/ml-explore/mlx-lm/blob/v0.31.2/mlx_lm/benchmark.py https://github.com/ml-explore/mlx-lm/blob/v0.31.2/mlx_lm/generate.py#L658-L753 Structured PP/TG fields are rounded to 0.1 tokens/s; the source means, sample deviations and all three retained trial values above preserve the published precision.

generation:

52.6 tokens/s

prompt processing (prefill):

413.7 tokens/s

quant:

4-bit (source config) (MLX)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

First-party historical Patrick Whelan benchmark, inspected 2026-10-13. Source snapshot 11f3f8f931dc97a71499e7b9c2583d010a3c160b verified by archive PAX comment and GitHub currentOid; pinned runner/config/hardware/VLM files byte-match archive. This pins the published evidence, not the executable used in April. No inference rerun or model-weight download. Hardware: one participating Apple M2 Max integrated GPU, 30 GPU cores, 12 CPU cores and 64 GB shared unified memory. Raw system data confirms those values; hardware profile explicitly says no discrete GPU and all inference on the M2 Max GPU. Shared 64 GB is system capacity, not dedicated VRAM or resident model use. Raw environment: Darwin 25.3.0 / ARM64_T6020 and Python 3.14.3. Do not borrow stale hardware-profile Sequoia 15.x or ai-gateway/vllm-mlx labels. Power, thermals, clocks, swap, wired-memory limit, GPU utilization and competing workload are unknown. Dates: raw system_info.timestamp=2026-04-02T21:00:05.046210+00:00; this is collected once before a run list, so it is a campaign/system-capture timestamp, not exact model start. Result filename glm-4.7-flash_default_20260402_235624.json uses local completion time with no timezone recorded. README updated 2026-04-04 is a publication label, not every measurement date. Configuration: direct python -m mlx_lm benchmark on Apple MLX/Metal, no HTTP server involved. MLX/MLX-LM/MLX-VLM installed versions and exact runtime commit are not retained in JSON. Requested prompt length 512 and generation length/cap 128, printed batch_size=1; three measured trials retained after discarding Trial 1, plus an internal/explicit warmup. Achieved token-count telemetry is not stored, so actual input/output fields remain unknown. Source context_size=4096, threads=8 and gpu_layers=-1 are explicitly unused for MLX and are not recorded here as active allocation, CPU threads or layer placement. KV/cache precision, prefill chunk size, sampling settings, thinking behavior and memory-cache reuse are unreported for the historical runtime; no drafter is passed by the wrapper. Native rates: PP 413.66 tok/s (sample SD 0.38; retained [413.314, 413.603, 414.072]); TG 52.6 tok/s (sample SD 0.04; retained [52.551, 52.627, 52.618]). Source aggregation rounds the three-trial arithmetic mean/sample SD to two decimals after parsing three-decimal CLI values. All values, means and SD were checked against raw Trial 2-4. CLI Averages includes all four trials and therefore is not the published three-trial metric. No pooling across engines/models. Timing exclusions: raw time_to_first_token_s mean 1.24 is explicitly computed as 512/prompt_tps by runner lines 718-719, not independently measured client or request TTFT. TTFT field is intentionally empty. Overall wall_time_s=2629.67 measures the entire subprocess and can include HF downloads, model load and warmup; it is not inference-only latency or end-to-end response timing. As a reference audit only, official MLX-LM tag v0.31.2 benchmark.py uses batch 1 random token IDs, internal warmup, disables EOS, and forwards stream_generate rates. generate.py starts the prompt timer before generation and closes it on the first generated token; PP is prompt.size/prompt_time. TG counts n+1 tokens over elapsed time after that boundary, so its numerator includes the first token while its first-token time is in PP. These are engine-reported native rates, not transport/client rates. The actual installed MLX-LM revision is absent; v0.31.2 is corroborating implementation, not a tested-run version. Artifact: mlx-community/GLM-4.7-Flash-4bit. Source-configured HF artifact joined by model key; raw JSON omits repository/digest; historical tested revision unknown. Raw quant is "N/A" because the MLX run dictionary lacks a quant key; quantization identification comes from source config/profile plus HF metadata, not that raw field. No weight digest is logged. Current artifact metadata inspected 2026-10-13 at HF revision 1454cffb1a21737e162f508e5bc70be9def89276: MLX 4-bit group 64 affine. This revision is a current metadata reference, not a historical run pin. Model-card conversion-tool versions are not installed benchmark engine versions. Model context capacities from current HF metadata are not active benchmark allocations. Memory: raw retained peak_memory values [17.496, 17.496, 17.497], maximum 17.497 in engine output units. Wrapper overwrites peak_vram_mb with this value times 1024 (17916.9), but leaves a stale unified_memory_system_wide note. It is MLX allocator/process peak reporting, not dedicated VRAM or system-wide resident usage. Reference MLX-LM v0.31.2 reports decimal GB (bytes/1e9), so wrapper times-1024 conversion may be unit-inconsistent; actual installed version is unknown. Dedicated-VRAM and memory-used fields remain empty. Config size_gb is an estimate, not measured disk bytes, and is not substituted for residency. No Mac power measurements. Sources: https://github.com/patrickwhelan-uk/llm-inference-benchmarks/blob/11f3f8f931dc97a71499e7b9c2583d010a3c160b/results/m2-mac-studio/glm-4.7-flash_default_20260402_235624.json https://github.com/patrickwhelan-uk/llm-inference-benchmarks/blob/11f3f8f931dc97a71499e7b9c2583d010a3c160b/config-m2-mac-studio.yaml https://github.com/patrickwhelan-uk/llm-inference-benchmarks/blob/11f3f8f931dc97a71499e7b9c2583d010a3c160b/run_benchmarks.py#L624-L738 https://github.com/patrickwhelan-uk/llm-inference-benchmarks/blob/11f3f8f931dc97a71499e7b9c2583d010a3c160b/hardware/m2-mac-studio.md https://huggingface.co/mlx-community/GLM-4.7-Flash-4bit/blob/1454cffb1a21737e162f508e5bc70be9def89276/config.json https://github.com/ml-explore/mlx-lm/blob/v0.31.2/mlx_lm/benchmark.py https://github.com/ml-explore/mlx-lm/blob/v0.31.2/mlx_lm/generate.py#L658-L753 Structured PP/TG fields are rounded to 0.1 tokens/s; the source means, sample deviations and all three retained trial values above preserve the published precision.

generation:

61.6 tokens/s

prompt processing (prefill):

465.2 tokens/s

quant:

4-bit mixed (config) (MLX)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

First-party historical Patrick Whelan benchmark, inspected 2026-10-13. Source snapshot 11f3f8f931dc97a71499e7b9c2583d010a3c160b verified by archive PAX comment and GitHub currentOid; pinned runner/config/hardware/VLM files byte-match archive. This pins the published evidence, not the executable used in April. No inference rerun or model-weight download. Hardware: one participating Apple M2 Max integrated GPU, 30 GPU cores, 12 CPU cores and 64 GB shared unified memory. Raw system data confirms those values; hardware profile explicitly says no discrete GPU and all inference on the M2 Max GPU. Shared 64 GB is system capacity, not dedicated VRAM or resident model use. Raw environment: Darwin 25.3.0 / ARM64_T6020 and Python 3.14.3. Do not borrow stale hardware-profile Sequoia 15.x or ai-gateway/vllm-mlx labels. Power, thermals, clocks, swap, wired-memory limit, GPU utilization and competing workload are unknown. Dates: raw system_info.timestamp=2026-04-03T22:21:25.804547+00:00; this is collected once before a run list, so it is a campaign/system-capture timestamp, not exact model start. Result filename gemma-4-26b-moe_default_20260403_232148.json uses local completion time with no timezone recorded. README updated 2026-04-04 is a publication label, not every measurement date. Configuration: direct mlx_vlm text-only helper on Apple MLX/Metal, no HTTP server involved. MLX/MLX-LM/MLX-VLM installed versions and exact runtime commit are not retained in JSON. Requested prompt length 512 and generation length/cap 128, printed batch_size=1; three measured trials retained after discarding Trial 1, plus an internal/explicit warmup. Achieved token-count telemetry is not stored, so actual input/output fields remain unknown. Source context_size=4096, threads=8 and gpu_layers=-1 are explicitly unused for MLX and are not recorded here as active allocation, CPU threads or layer placement. KV/cache precision, prefill chunk size, sampling settings, thinking behavior and memory-cache reuse are unreported for the historical runtime; no drafter is passed by the wrapper. Native rates: PP 465.17 tok/s (sample SD 0.58; retained [464.722, 464.972, 465.83]); TG 61.62 tok/s (sample SD 0.2; retained [61.389, 61.732, 61.742]). Source aggregation rounds the three-trial arithmetic mean/sample SD to two decimals after parsing three-decimal CLI values. All values, means and SD were checked against raw Trial 2-4. The raw helper prints four trials and no CLI average; the wrapper computes the retained three-trial metric. No pooling across engines/models. Timing exclusions: raw time_to_first_token_s mean 1.1 is explicitly computed as 512/prompt_tps by runner lines 718-719, not independently measured client or request TTFT. TTFT field is intentionally empty. Overall wall_time_s=22.43 measures the entire subprocess and can include HF downloads, model load and warmup; it is not inference-only latency or end-to-end response timing. Pinned local VLM helper shows the native result.prompt_tps/result.generation_tps rate forwarding described below; exact installed mlx-vlm generation implementation/version is not logged. Artifact: mlx-community/gemma-4-26b-a4b-it-4bit. Source-configured HF artifact, additionally named in raw output; historical tested revision unknown. Raw quant is "N/A" because the MLX run dictionary lacks a quant key; quantization identification comes from source config/profile plus HF metadata, not that raw field. No weight digest is logged. Current artifact metadata inspected 2026-10-13 at HF revision 0d77464eeb233a2da68ebf9d7dc4edaac7db956d: MLX nominal 4-bit affine group 64; 30 8-bit group-64 module overrides. This revision is a current metadata reference, not a historical run pin. Model-card conversion-tool versions are not installed benchmark engine versions. Model context capacities from current HF metadata are not active benchmark allocations. Distinct VLM text-only harness: source config sets vlm:true and raw output names this HF repository. Local mlx_vlm_benchmark.py constructs repeated "hello " text, encodes and truncates to 512 tokens, decodes it back to text, then calls mlx_vlm.generate(image=None,max_tokens=128). It performs one explicit warmup and four trials, forwarding result.prompt_tps/result.generation_tps. Actual re-tokenized input length and returned output token counts are not logged; 512/128 are requested workload and output cap, not verified achieved counts. EOS is not explicitly disabled here. Do not pool this with the random-token MLX-LM runs as an identical workload. VLM helper memory is mx.metal.get_peak_memory()/1024^3 after resetting the peak each trial, so its raw memory unit is GiB. The HF card identifies google/gemma-4-26B-A4B-it base revision 20da991ab4afab98e8f910c4a2e8f4fbefc404ad, a conversion-provenance reference, not the tested MLX revision. Memory: raw retained peak_memory values [15.255, 15.255, 15.255], maximum 15.255 in engine output units. Wrapper overwrites peak_vram_mb with this value times 1024 (15621.1), but leaves a stale unified_memory_system_wide note. It is MLX allocator/process peak reporting, not dedicated VRAM or system-wide resident usage. VLM helper explicitly uses GiB; Dedicated-VRAM and memory-used fields remain empty. Config size_gb is an estimate, not measured disk bytes, and is not substituted for residency. No Mac power measurements. Sources: https://github.com/patrickwhelan-uk/llm-inference-benchmarks/blob/11f3f8f931dc97a71499e7b9c2583d010a3c160b/results/m2-mac-studio/gemma-4-26b-moe_default_20260403_232148.json https://github.com/patrickwhelan-uk/llm-inference-benchmarks/blob/11f3f8f931dc97a71499e7b9c2583d010a3c160b/config-m2-mac-studio.yaml https://github.com/patrickwhelan-uk/llm-inference-benchmarks/blob/11f3f8f931dc97a71499e7b9c2583d010a3c160b/run_benchmarks.py#L624-L738 https://github.com/patrickwhelan-uk/llm-inference-benchmarks/blob/11f3f8f931dc97a71499e7b9c2583d010a3c160b/hardware/m2-mac-studio.md https://huggingface.co/mlx-community/gemma-4-26b-a4b-it-4bit/blob/0d77464eeb233a2da68ebf9d7dc4edaac7db956d/config.json https://github.com/patrickwhelan-uk/llm-inference-benchmarks/blob/11f3f8f931dc97a71499e7b9c2583d010a3c160b/mlx_vlm_benchmark.py Structured PP/TG fields are rounded to 0.1 tokens/s; the source means, sample deviations and all three retained trial values above preserve the published precision.

generation:

117.8 tokens/s

prompt processing (prefill):

616.4 tokens/s

quant:

4-bit mixed (config) (MLX)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

First-party historical Patrick Whelan benchmark, inspected 2026-10-13. Source snapshot 11f3f8f931dc97a71499e7b9c2583d010a3c160b verified by archive PAX comment and GitHub currentOid; pinned runner/config/hardware/VLM files byte-match archive. This pins the published evidence, not the executable used in April. No inference rerun or model-weight download. Hardware: one participating Apple M2 Max integrated GPU, 30 GPU cores, 12 CPU cores and 64 GB shared unified memory. Raw system data confirms those values; hardware profile explicitly says no discrete GPU and all inference on the M2 Max GPU. Shared 64 GB is system capacity, not dedicated VRAM or resident model use. Raw environment: Darwin 25.3.0 / ARM64_T6020 and Python 3.14.3. Do not borrow stale hardware-profile Sequoia 15.x or ai-gateway/vllm-mlx labels. Power, thermals, clocks, swap, wired-memory limit, GPU utilization and competing workload are unknown. Dates: raw system_info.timestamp=2026-04-03T22:55:02.397768+00:00; this is collected once before a run list, so it is a campaign/system-capture timestamp, not exact model start. Result filename lfm2-24b-moe_default_20260404_002932.json uses local completion time with no timezone recorded. README updated 2026-04-04 is a publication label, not every measurement date. Configuration: direct python -m mlx_lm benchmark on Apple MLX/Metal, no HTTP server involved. MLX/MLX-LM/MLX-VLM installed versions and exact runtime commit are not retained in JSON. Requested prompt length 512 and generation length/cap 128, printed batch_size=1; three measured trials retained after discarding Trial 1, plus an internal/explicit warmup. Achieved token-count telemetry is not stored, so actual input/output fields remain unknown. Source context_size=4096, threads=8 and gpu_layers=-1 are explicitly unused for MLX and are not recorded here as active allocation, CPU threads or layer placement. KV/cache precision, prefill chunk size, sampling settings, thinking behavior and memory-cache reuse are unreported for the historical runtime; no drafter is passed by the wrapper. Native rates: PP 616.43 tok/s (sample SD 1.03; retained [615.274, 617.245, 616.765]); TG 117.78 tok/s (sample SD 0.29; retained [117.461, 117.841, 118.03]). Source aggregation rounds the three-trial arithmetic mean/sample SD to two decimals after parsing three-decimal CLI values. All values, means and SD were checked against raw Trial 2-4. CLI Averages includes all four trials and therefore is not the published three-trial metric. No pooling across engines/models. Timing exclusions: raw time_to_first_token_s mean 0.83 is explicitly computed as 512/prompt_tps by runner lines 718-719, not independently measured client or request TTFT. TTFT field is intentionally empty. Overall wall_time_s=2069.96 measures the entire subprocess and can include HF downloads, model load and warmup; it is not inference-only latency or end-to-end response timing. As a reference audit only, official MLX-LM tag v0.31.2 benchmark.py uses batch 1 random token IDs, internal warmup, disables EOS, and forwards stream_generate rates. generate.py starts the prompt timer before generation and closes it on the first generated token; PP is prompt.size/prompt_time. TG counts n+1 tokens over elapsed time after that boundary, so its numerator includes the first token while its first-token time is in PP. These are engine-reported native rates, not transport/client rates. The actual installed MLX-LM revision is absent; v0.31.2 is corroborating implementation, not a tested-run version. Artifact: mlx-community/LFM2-24B-A2B-4bit. Source-configured HF artifact joined by model key; raw JSON omits repository/digest; historical tested revision unknown. Raw quant is "N/A" because the MLX run dictionary lacks a quant key; quantization identification comes from source config/profile plus HF metadata, not that raw field. No weight digest is logged. Current artifact metadata inspected 2026-10-13 at HF revision eede03bcde763b314fe70e0f5269b90aa0e4fd02: MLX nominal 4-bit affine group 64; 38 8-bit group-64 module overrides. This revision is a current metadata reference, not a historical run pin. Model-card conversion-tool versions are not installed benchmark engine versions. Model context capacities from current HF metadata are not active benchmark allocations. Identity correction: source params field says "64x1.3B MoE", an unsupported/malformed size label. The explicit source repository and its current model card identify LiquidAI/LFM2-24B-A2B. This packet uses the artifact name LFM2-24B-A2B and nominal 24B label, without deriving total parameters by multiplying experts. Current config has 64 experts and 4 selected per token. These are current artifact metadata, not independent proof of tested weights. Memory: raw retained peak_memory values [13.821, 13.821, 13.822], maximum 13.822 in engine output units. Wrapper overwrites peak_vram_mb with this value times 1024 (14153.7), but leaves a stale unified_memory_system_wide note. It is MLX allocator/process peak reporting, not dedicated VRAM or system-wide resident usage. Reference MLX-LM v0.31.2 reports decimal GB (bytes/1e9), so wrapper times-1024 conversion may be unit-inconsistent; actual installed version is unknown. Dedicated-VRAM and memory-used fields remain empty. Config size_gb is an estimate, not measured disk bytes, and is not substituted for residency. No Mac power measurements. Sources: https://github.com/patrickwhelan-uk/llm-inference-benchmarks/blob/11f3f8f931dc97a71499e7b9c2583d010a3c160b/results/m2-mac-studio/lfm2-24b-moe_default_20260404_002932.json https://github.com/patrickwhelan-uk/llm-inference-benchmarks/blob/11f3f8f931dc97a71499e7b9c2583d010a3c160b/config-m2-mac-studio.yaml https://github.com/patrickwhelan-uk/llm-inference-benchmarks/blob/11f3f8f931dc97a71499e7b9c2583d010a3c160b/run_benchmarks.py#L624-L738 https://github.com/patrickwhelan-uk/llm-inference-benchmarks/blob/11f3f8f931dc97a71499e7b9c2583d010a3c160b/hardware/m2-mac-studio.md https://huggingface.co/mlx-community/LFM2-24B-A2B-4bit/blob/eede03bcde763b314fe70e0f5269b90aa0e4fd02/config.json https://github.com/ml-explore/mlx-lm/blob/v0.31.2/mlx_lm/benchmark.py https://github.com/ml-explore/mlx-lm/blob/v0.31.2/mlx_lm/generate.py#L658-L753 Structured PP/TG fields are rounded to 0.1 tokens/s; the source means, sample deviations and all three retained trial values above preserve the published precision.

generation:

24.4 tokens/s

prompt processing (prefill):

104.2 tokens/s

quant:

4-bit (source config) (MLX)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

First-party historical Patrick Whelan benchmark, inspected 2026-10-13. Source snapshot 11f3f8f931dc97a71499e7b9c2583d010a3c160b verified by archive PAX comment and GitHub currentOid; pinned runner/config/hardware/VLM files byte-match archive. This pins the published evidence, not the executable used in April. No inference rerun or model-weight download. Hardware: one participating Apple M2 Max integrated GPU, 30 GPU cores, 12 CPU cores and 64 GB shared unified memory. Raw system data confirms those values; hardware profile explicitly says no discrete GPU and all inference on the M2 Max GPU. Shared 64 GB is system capacity, not dedicated VRAM or resident model use. Raw environment: Darwin 25.3.0 / ARM64_T6020 and Python 3.14.3. Do not borrow stale hardware-profile Sequoia 15.x or ai-gateway/vllm-mlx labels. Power, thermals, clocks, swap, wired-memory limit, GPU utilization and competing workload are unknown. Dates: raw system_info.timestamp=2026-04-02T21:00:05.046210+00:00; this is collected once before a run list, so it is a campaign/system-capture timestamp, not exact model start. Result filename devstral-small-2-24b_default_20260402_231235.json uses local completion time with no timezone recorded. README updated 2026-04-04 is a publication label, not every measurement date. Configuration: direct python -m mlx_lm benchmark on Apple MLX/Metal, no HTTP server involved. MLX/MLX-LM/MLX-VLM installed versions and exact runtime commit are not retained in JSON. Requested prompt length 512 and generation length/cap 128, printed batch_size=1; three measured trials retained after discarding Trial 1, plus an internal/explicit warmup. Achieved token-count telemetry is not stored, so actual input/output fields remain unknown. Source context_size=4096, threads=8 and gpu_layers=-1 are explicitly unused for MLX and are not recorded here as active allocation, CPU threads or layer placement. KV/cache precision, prefill chunk size, sampling settings, thinking behavior and memory-cache reuse are unreported for the historical runtime; no drafter is passed by the wrapper. Native rates: PP 104.15 tok/s (sample SD 0.02; retained [104.13, 104.167, 104.154]); TG 24.43 tok/s (sample SD 0.01; retained [24.434, 24.417, 24.424]). Source aggregation rounds the three-trial arithmetic mean/sample SD to two decimals after parsing three-decimal CLI values. All values, means and SD were checked against raw Trial 2-4. CLI Averages includes all four trials and therefore is not the published three-trial metric. No pooling across engines/models. Timing exclusions: raw time_to_first_token_s mean 4.92 is explicitly computed as 512/prompt_tps by runner lines 718-719, not independently measured client or request TTFT. TTFT field is intentionally empty. Overall wall_time_s=2130.12 measures the entire subprocess and can include HF downloads, model load and warmup; it is not inference-only latency or end-to-end response timing. As a reference audit only, official MLX-LM tag v0.31.2 benchmark.py uses batch 1 random token IDs, internal warmup, disables EOS, and forwards stream_generate rates. generate.py starts the prompt timer before generation and closes it on the first generated token; PP is prompt.size/prompt_time. TG counts n+1 tokens over elapsed time after that boundary, so its numerator includes the first token while its first-token time is in PP. These are engine-reported native rates, not transport/client rates. The actual installed MLX-LM revision is absent; v0.31.2 is corroborating implementation, not a tested-run version. Artifact: mlx-community/Devstral-Small-2505-4bit. Source-configured HF artifact, with raw tokenizer-cache snapshot revision. Raw quant is "N/A" because the MLX run dictionary lacks a quant key; quantization identification comes from source config/profile plus HF metadata, not that raw field. No weight digest is logged. Current artifact metadata inspected 2026-10-13 at HF revision 91ab74727385430dac575c8b0a27235367870cb6: MLX 4-bit group 64 (mode not explicitly named in config). This revision is a current metadata reference, not a historical run pin (although it matches the raw Devstral tokenizer snapshot). Model-card conversion-tool versions are not installed benchmark engine versions. Model context capacities from current HF metadata are not active benchmark allocations. Identity correction: published row name devstral-small-2-24b is misleading. Both source configuration and the raw tokenizer-cache warning identify mlx-community/Devstral-Small-2505-4bit, with snapshot 91ab74727385430dac575c8b0a27235367870cb6; its pinned HF card gives base_model mistralai/Devstral-Small-2505. This packet is Devstral Small 2505, not Devstral Small 2. The raw run warns of an incorrect tokenizer regex and recommends fix_mistral_regex=True. The source does not show that fix applied. Preserve warning as a quality/reproducibility caveat; no quality claim. The logged snapshot ties the tokenizer-cache path to a repository revision, not an independently verified weight checksum. Memory: raw retained peak_memory values [13.762, 13.762, 13.763], maximum 13.763 in engine output units. Wrapper overwrites peak_vram_mb with this value times 1024 (14093.3), but leaves a stale unified_memory_system_wide note. It is MLX allocator/process peak reporting, not dedicated VRAM or system-wide resident usage. Reference MLX-LM v0.31.2 reports decimal GB (bytes/1e9), so wrapper times-1024 conversion may be unit-inconsistent; actual installed version is unknown. Dedicated-VRAM and memory-used fields remain empty. Config size_gb is an estimate, not measured disk bytes, and is not substituted for residency. No Mac power measurements. Sources: https://github.com/patrickwhelan-uk/llm-inference-benchmarks/blob/11f3f8f931dc97a71499e7b9c2583d010a3c160b/results/m2-mac-studio/devstral-small-2-24b_default_20260402_231235.json https://github.com/patrickwhelan-uk/llm-inference-benchmarks/blob/11f3f8f931dc97a71499e7b9c2583d010a3c160b/config-m2-mac-studio.yaml https://github.com/patrickwhelan-uk/llm-inference-benchmarks/blob/11f3f8f931dc97a71499e7b9c2583d010a3c160b/run_benchmarks.py#L624-L738 https://github.com/patrickwhelan-uk/llm-inference-benchmarks/blob/11f3f8f931dc97a71499e7b9c2583d010a3c160b/hardware/m2-mac-studio.md https://huggingface.co/mlx-community/Devstral-Small-2505-4bit/blob/91ab74727385430dac575c8b0a27235367870cb6/config.json https://github.com/ml-explore/mlx-lm/blob/v0.31.2/mlx_lm/benchmark.py https://github.com/ml-explore/mlx-lm/blob/v0.31.2/mlx_lm/generate.py#L658-L753 Structured PP/TG fields are rounded to 0.1 tokens/s; the source means, sample deviations and all three retained trial values above preserve the published precision.

generation:

85.5 tokens/s

prompt processing (prefill):

501.5 tokens/s

quant:

MXFP4-Q4 mixed (config) (MLX)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

First-party historical Patrick Whelan benchmark, inspected 2026-10-13. Source snapshot 11f3f8f931dc97a71499e7b9c2583d010a3c160b verified by archive PAX comment and GitHub currentOid; pinned runner/config/hardware/VLM files byte-match archive. This pins the published evidence, not the executable used in April. No inference rerun or model-weight download. Hardware: one participating Apple M2 Max integrated GPU, 30 GPU cores, 12 CPU cores and 64 GB shared unified memory. Raw system data confirms those values; hardware profile explicitly says no discrete GPU and all inference on the M2 Max GPU. Shared 64 GB is system capacity, not dedicated VRAM or resident model use. Raw environment: Darwin 25.3.0 / ARM64_T6020 and Python 3.14.3. Do not borrow stale hardware-profile Sequoia 15.x or ai-gateway/vllm-mlx labels. Power, thermals, clocks, swap, wired-memory limit, GPU utilization and competing workload are unknown. Dates: raw system_info.timestamp=2026-04-02T23:01:42.288839+00:00; this is collected once before a run list, so it is a campaign/system-capture timestamp, not exact model start. Result filename gpt-oss-20b_default_20260403_002959.json uses local completion time with no timezone recorded. README updated 2026-04-04 is a publication label, not every measurement date. Configuration: direct python -m mlx_lm benchmark on Apple MLX/Metal, no HTTP server involved. MLX/MLX-LM/MLX-VLM installed versions and exact runtime commit are not retained in JSON. Requested prompt length 512 and generation length/cap 128, printed batch_size=1; three measured trials retained after discarding Trial 1, plus an internal/explicit warmup. Achieved token-count telemetry is not stored, so actual input/output fields remain unknown. Source context_size=4096, threads=8 and gpu_layers=-1 are explicitly unused for MLX and are not recorded here as active allocation, CPU threads or layer placement. KV/cache precision, prefill chunk size, sampling settings, thinking behavior and memory-cache reuse are unreported for the historical runtime; no drafter is passed by the wrapper. Native rates: PP 501.47 tok/s (sample SD 0.18; retained [501.591, 501.565, 501.263]); TG 85.49 tok/s (sample SD 0.02; retained [85.466, 85.49, 85.509]). Source aggregation rounds the three-trial arithmetic mean/sample SD to two decimals after parsing three-decimal CLI values. All values, means and SD were checked against raw Trial 2-4. CLI Averages includes all four trials and therefore is not the published three-trial metric. No pooling across engines/models. Timing exclusions: raw time_to_first_token_s mean 1.02 is explicitly computed as 512/prompt_tps by runner lines 718-719, not independently measured client or request TTFT. TTFT field is intentionally empty. Overall wall_time_s=1696.97 measures the entire subprocess and can include HF downloads, model load and warmup; it is not inference-only latency or end-to-end response timing. As a reference audit only, official MLX-LM tag v0.31.2 benchmark.py uses batch 1 random token IDs, internal warmup, disables EOS, and forwards stream_generate rates. generate.py starts the prompt timer before generation and closes it on the first generated token; PP is prompt.size/prompt_time. TG counts n+1 tokens over elapsed time after that boundary, so its numerator includes the first token while its first-token time is in PP. These are engine-reported native rates, not transport/client rates. The actual installed MLX-LM revision is absent; v0.31.2 is corroborating implementation, not a tested-run version. Artifact: mlx-community/gpt-oss-20b-MXFP4-Q4. Source-configured HF artifact joined by model key; raw JSON omits repository/digest; historical tested revision unknown. Raw quant is "N/A" because the MLX run dictionary lacks a quant key; quantization identification comes from source config/profile plus HF metadata, not that raw field. No weight digest is logged. Current artifact metadata inspected 2026-10-13 at HF revision f356f2747216d7e98fee755df25987459fc19089: MXFP4-Q4 mixed MLX: default 4-bit MXFP4 group 32; 98 affine 4-bit group-64 overrides and 24 8-bit group-64 overrides. This revision is a current metadata reference, not a historical run pin. Model-card conversion-tool versions are not installed benchmark engine versions. Model context capacities from current HF metadata are not active benchmark allocations. Memory: raw retained peak_memory values [11.712, 11.713, 11.713], maximum 11.713 in engine output units. Wrapper overwrites peak_vram_mb with this value times 1024 (11994.1), but leaves a stale unified_memory_system_wide note. It is MLX allocator/process peak reporting, not dedicated VRAM or system-wide resident usage. Reference MLX-LM v0.31.2 reports decimal GB (bytes/1e9), so wrapper times-1024 conversion may be unit-inconsistent; actual installed version is unknown. Dedicated-VRAM and memory-used fields remain empty. Config size_gb is an estimate, not measured disk bytes, and is not substituted for residency. No Mac power measurements. Sources: https://github.com/patrickwhelan-uk/llm-inference-benchmarks/blob/11f3f8f931dc97a71499e7b9c2583d010a3c160b/results/m2-mac-studio/gpt-oss-20b_default_20260403_002959.json https://github.com/patrickwhelan-uk/llm-inference-benchmarks/blob/11f3f8f931dc97a71499e7b9c2583d010a3c160b/config-m2-mac-studio.yaml https://github.com/patrickwhelan-uk/llm-inference-benchmarks/blob/11f3f8f931dc97a71499e7b9c2583d010a3c160b/run_benchmarks.py#L624-L738 https://github.com/patrickwhelan-uk/llm-inference-benchmarks/blob/11f3f8f931dc97a71499e7b9c2583d010a3c160b/hardware/m2-mac-studio.md https://huggingface.co/mlx-community/gpt-oss-20b-MXFP4-Q4/blob/f356f2747216d7e98fee755df25987459fc19089/config.json https://github.com/ml-explore/mlx-lm/blob/v0.31.2/mlx_lm/benchmark.py https://github.com/ml-explore/mlx-lm/blob/v0.31.2/mlx_lm/generate.py#L658-L753 Structured PP/TG fields are rounded to 0.1 tokens/s; the source means, sample deviations and all three retained trial values above preserve the published precision.

generation:

62.7 tokens/s

prompt processing (prefill):

321.2 tokens/s

quant:

4-bit (source config) (MLX)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

First-party historical Patrick Whelan benchmark, inspected 2026-10-13. Source snapshot 11f3f8f931dc97a71499e7b9c2583d010a3c160b verified by archive PAX comment and GitHub currentOid; pinned runner/config/hardware/VLM files byte-match archive. This pins the published evidence, not the executable used in April. No inference rerun or model-weight download. Hardware: one participating Apple M2 Max integrated GPU, 30 GPU cores, 12 CPU cores and 64 GB shared unified memory. Raw system data confirms those values; hardware profile explicitly says no discrete GPU and all inference on the M2 Max GPU. Shared 64 GB is system capacity, not dedicated VRAM or resident model use. Raw environment: Darwin 25.3.0 / ARM64_T6020 and Python 3.14.3. Do not borrow stale hardware-profile Sequoia 15.x or ai-gateway/vllm-mlx labels. Power, thermals, clocks, swap, wired-memory limit, GPU utilization and competing workload are unknown. Dates: raw system_info.timestamp=2026-04-03T22:54:28.016300+00:00; this is collected once before a run list, so it is a campaign/system-capture timestamp, not exact model start. Result filename qwen3.5-9b_default_20260403_235452.json uses local completion time with no timezone recorded. README updated 2026-04-04 is a publication label, not every measurement date. Configuration: direct python -m mlx_lm benchmark on Apple MLX/Metal, no HTTP server involved. MLX/MLX-LM/MLX-VLM installed versions and exact runtime commit are not retained in JSON. Requested prompt length 512 and generation length/cap 128, printed batch_size=1; three measured trials retained after discarding Trial 1, plus an internal/explicit warmup. Achieved token-count telemetry is not stored, so actual input/output fields remain unknown. Source context_size=4096, threads=8 and gpu_layers=-1 are explicitly unused for MLX and are not recorded here as active allocation, CPU threads or layer placement. KV/cache precision, prefill chunk size, sampling settings, thinking behavior and memory-cache reuse are unreported for the historical runtime; no drafter is passed by the wrapper. Native rates: PP 321.17 tok/s (sample SD 0.27; retained [321.212, 321.419, 320.884]); TG 62.68 tok/s (sample SD 0.06; retained [62.651, 62.747, 62.627]). Source aggregation rounds the three-trial arithmetic mean/sample SD to two decimals after parsing three-decimal CLI values. All values, means and SD were checked against raw Trial 2-4. CLI Averages includes all four trials and therefore is not the published three-trial metric. No pooling across engines/models. Timing exclusions: raw time_to_first_token_s mean 1.59 is explicitly computed as 512/prompt_tps by runner lines 718-719, not independently measured client or request TTFT. TTFT field is intentionally empty. Overall wall_time_s=24.7 measures the entire subprocess and can include HF downloads, model load and warmup; it is not inference-only latency or end-to-end response timing. As a reference audit only, official MLX-LM tag v0.31.2 benchmark.py uses batch 1 random token IDs, internal warmup, disables EOS, and forwards stream_generate rates. generate.py starts the prompt timer before generation and closes it on the first generated token; PP is prompt.size/prompt_time. TG counts n+1 tokens over elapsed time after that boundary, so its numerator includes the first token while its first-token time is in PP. These are engine-reported native rates, not transport/client rates. The actual installed MLX-LM revision is absent; v0.31.2 is corroborating implementation, not a tested-run version. Artifact: mlx-community/Qwen3.5-9B-MLX-4bit. Source-configured HF artifact joined by model key; raw JSON omits repository/digest; historical tested revision unknown. Raw quant is "N/A" because the MLX run dictionary lacks a quant key; quantization identification comes from source config/profile plus HF metadata, not that raw field. No weight digest is logged. Current artifact metadata inspected 2026-10-13 at HF revision 938d8919941c6e7efd3c7150eff7fe9d12afa631: MLX 4-bit group 64 affine. This revision is a current metadata reference, not a historical run pin. Model-card conversion-tool versions are not installed benchmark engine versions. Model context capacities from current HF metadata are not active benchmark allocations. Memory: raw retained peak_memory values [5.869, 5.87, 5.87], maximum 5.87 in engine output units. Wrapper overwrites peak_vram_mb with this value times 1024 (6010.9), but leaves a stale unified_memory_system_wide note. It is MLX allocator/process peak reporting, not dedicated VRAM or system-wide resident usage. Reference MLX-LM v0.31.2 reports decimal GB (bytes/1e9), so wrapper times-1024 conversion may be unit-inconsistent; actual installed version is unknown. Dedicated-VRAM and memory-used fields remain empty. Config size_gb is an estimate, not measured disk bytes, and is not substituted for residency. No Mac power measurements. Sources: https://github.com/patrickwhelan-uk/llm-inference-benchmarks/blob/11f3f8f931dc97a71499e7b9c2583d010a3c160b/results/m2-mac-studio/qwen3.5-9b_default_20260403_235452.json https://github.com/patrickwhelan-uk/llm-inference-benchmarks/blob/11f3f8f931dc97a71499e7b9c2583d010a3c160b/config-m2-mac-studio.yaml https://github.com/patrickwhelan-uk/llm-inference-benchmarks/blob/11f3f8f931dc97a71499e7b9c2583d010a3c160b/run_benchmarks.py#L624-L738 https://github.com/patrickwhelan-uk/llm-inference-benchmarks/blob/11f3f8f931dc97a71499e7b9c2583d010a3c160b/hardware/m2-mac-studio.md https://huggingface.co/mlx-community/Qwen3.5-9B-MLX-4bit/blob/938d8919941c6e7efd3c7150eff7fe9d12afa631/config.json https://github.com/ml-explore/mlx-lm/blob/v0.31.2/mlx_lm/benchmark.py https://github.com/ml-explore/mlx-lm/blob/v0.31.2/mlx_lm/generate.py#L658-L753 Structured PP/TG fields are rounded to 0.1 tokens/s; the source means, sample deviations and all three retained trial values above preserve the published precision.

generation:

72.9 tokens/s

prompt processing (prefill):

566.4 tokens/s

quant:

4-bit (source config) (MLX)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

First-party historical Patrick Whelan benchmark, inspected 2026-10-13. Source snapshot 11f3f8f931dc97a71499e7b9c2583d010a3c160b verified by archive PAX comment and GitHub currentOid; pinned runner/config/hardware/VLM files byte-match archive. This pins the published evidence, not the executable used in April. No inference rerun or model-weight download. Hardware: one participating Apple M2 Max integrated GPU, 30 GPU cores, 12 CPU cores and 64 GB shared unified memory. Raw system data confirms those values; hardware profile explicitly says no discrete GPU and all inference on the M2 Max GPU. Shared 64 GB is system capacity, not dedicated VRAM or resident model use. Raw environment: Darwin 25.3.0 / ARM64_T6020 and Python 3.14.3. Do not borrow stale hardware-profile Sequoia 15.x or ai-gateway/vllm-mlx labels. Power, thermals, clocks, swap, wired-memory limit, GPU utilization and competing workload are unknown. Dates: raw system_info.timestamp=2026-04-02T21:00:05.046210+00:00; this is collected once before a run list, so it is a campaign/system-capture timestamp, not exact model start. Result filename llama-3.1-8b_default_20260402_223701.json uses local completion time with no timezone recorded. README updated 2026-04-04 is a publication label, not every measurement date. Configuration: direct python -m mlx_lm benchmark on Apple MLX/Metal, no HTTP server involved. MLX/MLX-LM/MLX-VLM installed versions and exact runtime commit are not retained in JSON. Requested prompt length 512 and generation length/cap 128, printed batch_size=1; three measured trials retained after discarding Trial 1, plus an internal/explicit warmup. Achieved token-count telemetry is not stored, so actual input/output fields remain unknown. Source context_size=4096, threads=8 and gpu_layers=-1 are explicitly unused for MLX and are not recorded here as active allocation, CPU threads or layer placement. KV/cache precision, prefill chunk size, sampling settings, thinking behavior and memory-cache reuse are unreported for the historical runtime; no drafter is passed by the wrapper. Native rates: PP 566.4 tok/s (sample SD 0.57; retained [566.388, 565.835, 566.968]); TG 72.94 tok/s (sample SD 0.17; retained [73.024, 73.052, 72.745]). Source aggregation rounds the three-trial arithmetic mean/sample SD to two decimals after parsing three-decimal CLI values. All values, means and SD were checked against raw Trial 2-4. CLI Averages includes all four trials and therefore is not the published three-trial metric. No pooling across engines/models. Timing exclusions: raw time_to_first_token_s mean 0.9 is explicitly computed as 512/prompt_tps by runner lines 718-719, not independently measured client or request TTFT. TTFT field is intentionally empty. Overall wall_time_s=721.82 measures the entire subprocess and can include HF downloads, model load and warmup; it is not inference-only latency or end-to-end response timing. As a reference audit only, official MLX-LM tag v0.31.2 benchmark.py uses batch 1 random token IDs, internal warmup, disables EOS, and forwards stream_generate rates. generate.py starts the prompt timer before generation and closes it on the first generated token; PP is prompt.size/prompt_time. TG counts n+1 tokens over elapsed time after that boundary, so its numerator includes the first token while its first-token time is in PP. These are engine-reported native rates, not transport/client rates. The actual installed MLX-LM revision is absent; v0.31.2 is corroborating implementation, not a tested-run version. Artifact: mlx-community/Meta-Llama-3.1-8B-Instruct-4bit. Source-configured HF artifact joined by model key; raw JSON omits repository/digest; historical tested revision unknown. Raw quant is "N/A" because the MLX run dictionary lacks a quant key; quantization identification comes from source config/profile plus HF metadata, not that raw field. No weight digest is logged. Current artifact metadata inspected 2026-10-13 at HF revision 241a666dad6cb93c8ff213d39a7f34a36bf26db4: MLX 4-bit group 64 (mode not explicitly named in config). This revision is a current metadata reference, not a historical run pin. Model-card conversion-tool versions are not installed benchmark engine versions. Model context capacities from current HF metadata are not active benchmark allocations. Memory: raw retained peak_memory values [5.115, 5.115, 5.116], maximum 5.116 in engine output units. Wrapper overwrites peak_vram_mb with this value times 1024 (5238.8), but leaves a stale unified_memory_system_wide note. It is MLX allocator/process peak reporting, not dedicated VRAM or system-wide resident usage. Reference MLX-LM v0.31.2 reports decimal GB (bytes/1e9), so wrapper times-1024 conversion may be unit-inconsistent; actual installed version is unknown. Dedicated-VRAM and memory-used fields remain empty. Config size_gb is an estimate, not measured disk bytes, and is not substituted for residency. No Mac power measurements. Sources: https://github.com/patrickwhelan-uk/llm-inference-benchmarks/blob/11f3f8f931dc97a71499e7b9c2583d010a3c160b/results/m2-mac-studio/llama-3.1-8b_default_20260402_223701.json https://github.com/patrickwhelan-uk/llm-inference-benchmarks/blob/11f3f8f931dc97a71499e7b9c2583d010a3c160b/config-m2-mac-studio.yaml https://github.com/patrickwhelan-uk/llm-inference-benchmarks/blob/11f3f8f931dc97a71499e7b9c2583d010a3c160b/run_benchmarks.py#L624-L738 https://github.com/patrickwhelan-uk/llm-inference-benchmarks/blob/11f3f8f931dc97a71499e7b9c2583d010a3c160b/hardware/m2-mac-studio.md https://huggingface.co/mlx-community/Meta-Llama-3.1-8B-Instruct-4bit/blob/241a666dad6cb93c8ff213d39a7f34a36bf26db4/config.json https://github.com/ml-explore/mlx-lm/blob/v0.31.2/mlx_lm/benchmark.py https://github.com/ml-explore/mlx-lm/blob/v0.31.2/mlx_lm/generate.py#L658-L753 Structured PP/TG fields are rounded to 0.1 tokens/s; the source means, sample deviations and all three retained trial values above preserve the published precision.

generation:

64.4 tokens/s

prompt processing (prefill):

491.1 tokens/s

quant:

4-bit (source config) (MLX)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

First-party historical Patrick Whelan benchmark, inspected 2026-10-13. Source snapshot 11f3f8f931dc97a71499e7b9c2583d010a3c160b verified by archive PAX comment and GitHub currentOid; pinned runner/config/hardware/VLM files byte-match archive. This pins the published evidence, not the executable used in April. No inference rerun or model-weight download. Hardware: one participating Apple M2 Max integrated GPU, 30 GPU cores, 12 CPU cores and 64 GB shared unified memory. Raw system data confirms those values; hardware profile explicitly says no discrete GPU and all inference on the M2 Max GPU. Shared 64 GB is system capacity, not dedicated VRAM or resident model use. Raw environment: Darwin 25.3.0 / ARM64_T6020 and Python 3.14.3. Do not borrow stale hardware-profile Sequoia 15.x or ai-gateway/vllm-mlx labels. Power, thermals, clocks, swap, wired-memory limit, GPU utilization and competing workload are unknown. Dates: raw system_info.timestamp=2026-04-03T22:42:14.957326+00:00; this is collected once before a run list, so it is a campaign/system-capture timestamp, not exact model start. Result filename granite-3.2-8b_default_20260403_235419.json uses local completion time with no timezone recorded. README updated 2026-04-04 is a publication label, not every measurement date. Configuration: direct python -m mlx_lm benchmark on Apple MLX/Metal, no HTTP server involved. MLX/MLX-LM/MLX-VLM installed versions and exact runtime commit are not retained in JSON. Requested prompt length 512 and generation length/cap 128, printed batch_size=1; three measured trials retained after discarding Trial 1, plus an internal/explicit warmup. Achieved token-count telemetry is not stored, so actual input/output fields remain unknown. Source context_size=4096, threads=8 and gpu_layers=-1 are explicitly unused for MLX and are not recorded here as active allocation, CPU threads or layer placement. KV/cache precision, prefill chunk size, sampling settings, thinking behavior and memory-cache reuse are unreported for the historical runtime; no drafter is passed by the wrapper. Native rates: PP 491.05 tok/s (sample SD 1.12; retained [491.449, 491.918, 489.785]); TG 64.43 tok/s (sample SD 0.02; retained [64.428, 64.408, 64.441]). Source aggregation rounds the three-trial arithmetic mean/sample SD to two decimals after parsing three-decimal CLI values. All values, means and SD were checked against raw Trial 2-4. CLI Averages includes all four trials and therefore is not the published three-trial metric. No pooling across engines/models. Timing exclusions: raw time_to_first_token_s mean 1.04 is explicitly computed as 512/prompt_tps by runner lines 718-719, not independently measured client or request TTFT. TTFT field is intentionally empty. Overall wall_time_s=724.05 measures the entire subprocess and can include HF downloads, model load and warmup; it is not inference-only latency or end-to-end response timing. As a reference audit only, official MLX-LM tag v0.31.2 benchmark.py uses batch 1 random token IDs, internal warmup, disables EOS, and forwards stream_generate rates. generate.py starts the prompt timer before generation and closes it on the first generated token; PP is prompt.size/prompt_time. TG counts n+1 tokens over elapsed time after that boundary, so its numerator includes the first token while its first-token time is in PP. These are engine-reported native rates, not transport/client rates. The actual installed MLX-LM revision is absent; v0.31.2 is corroborating implementation, not a tested-run version. Artifact: mlx-community/granite-3.2-8b-instruct-preview-4bit. Source-configured HF artifact joined by model key; raw JSON omits repository/digest; historical tested revision unknown. Raw quant is "N/A" because the MLX run dictionary lacks a quant key; quantization identification comes from source config/profile plus HF metadata, not that raw field. No weight digest is logged. Current artifact metadata inspected 2026-10-13 at HF revision c11bf25d3ac421d693a76b6f77f784b2e6ce1e11: MLX 4-bit group 64 (mode not explicitly named in config). This revision is a current metadata reference, not a historical run pin. Model-card conversion-tool versions are not installed benchmark engine versions. Model context capacities from current HF metadata are not active benchmark allocations. Memory: raw retained peak_memory values [5.218, 5.218, 5.218], maximum 5.218 in engine output units. Wrapper overwrites peak_vram_mb with this value times 1024 (5343.2), but leaves a stale unified_memory_system_wide note. It is MLX allocator/process peak reporting, not dedicated VRAM or system-wide resident usage. Reference MLX-LM v0.31.2 reports decimal GB (bytes/1e9), so wrapper times-1024 conversion may be unit-inconsistent; actual installed version is unknown. Dedicated-VRAM and memory-used fields remain empty. Config size_gb is an estimate, not measured disk bytes, and is not substituted for residency. No Mac power measurements. Sources: https://github.com/patrickwhelan-uk/llm-inference-benchmarks/blob/11f3f8f931dc97a71499e7b9c2583d010a3c160b/results/m2-mac-studio/granite-3.2-8b_default_20260403_235419.json https://github.com/patrickwhelan-uk/llm-inference-benchmarks/blob/11f3f8f931dc97a71499e7b9c2583d010a3c160b/config-m2-mac-studio.yaml https://github.com/patrickwhelan-uk/llm-inference-benchmarks/blob/11f3f8f931dc97a71499e7b9c2583d010a3c160b/run_benchmarks.py#L624-L738 https://github.com/patrickwhelan-uk/llm-inference-benchmarks/blob/11f3f8f931dc97a71499e7b9c2583d010a3c160b/hardware/m2-mac-studio.md https://huggingface.co/mlx-community/granite-3.2-8b-instruct-preview-4bit/blob/c11bf25d3ac421d693a76b6f77f784b2e6ce1e11/config.json https://github.com/ml-explore/mlx-lm/blob/v0.31.2/mlx_lm/benchmark.py https://github.com/ml-explore/mlx-lm/blob/v0.31.2/mlx_lm/generate.py#L658-L753 Structured PP/TG fields are rounded to 0.1 tokens/s; the source means, sample deviations and all three retained trial values above preserve the published precision.

generation:

76.8 tokens/s

prompt processing (prefill):

602.8 tokens/s

quant:

4-bit (source config) (MLX)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

First-party historical Patrick Whelan benchmark, inspected 2026-10-13. Source snapshot 11f3f8f931dc97a71499e7b9c2583d010a3c160b verified by archive PAX comment and GitHub currentOid; pinned runner/config/hardware/VLM files byte-match archive. This pins the published evidence, not the executable used in April. No inference rerun or model-weight download. Hardware: one participating Apple M2 Max integrated GPU, 30 GPU cores, 12 CPU cores and 64 GB shared unified memory. Raw system data confirms those values; hardware profile explicitly says no discrete GPU and all inference on the M2 Max GPU. Shared 64 GB is system capacity, not dedicated VRAM or resident model use. Raw environment: Darwin 25.3.0 / ARM64_T6020 and Python 3.14.3. Do not borrow stale hardware-profile Sequoia 15.x or ai-gateway/vllm-mlx labels. Power, thermals, clocks, swap, wired-memory limit, GPU utilization and competing workload are unknown. Dates: raw system_info.timestamp=2026-04-02T21:00:05.046210+00:00; this is collected once before a run list, so it is a campaign/system-capture timestamp, not exact model start. Result filename qwen2.5-7b_default_20260402_222500.json uses local completion time with no timezone recorded. README updated 2026-04-04 is a publication label, not every measurement date. Configuration: direct python -m mlx_lm benchmark on Apple MLX/Metal, no HTTP server involved. MLX/MLX-LM/MLX-VLM installed versions and exact runtime commit are not retained in JSON. Requested prompt length 512 and generation length/cap 128, printed batch_size=1; three measured trials retained after discarding Trial 1, plus an internal/explicit warmup. Achieved token-count telemetry is not stored, so actual input/output fields remain unknown. Source context_size=4096, threads=8 and gpu_layers=-1 are explicitly unused for MLX and are not recorded here as active allocation, CPU threads or layer placement. KV/cache precision, prefill chunk size, sampling settings, thinking behavior and memory-cache reuse are unreported for the historical runtime; no drafter is passed by the wrapper. Native rates: PP 602.8 tok/s (sample SD 0.35; retained [602.881, 602.411, 603.095]); TG 76.78 tok/s (sample SD 0.04; retained [76.773, 76.747, 76.821]). Source aggregation rounds the three-trial arithmetic mean/sample SD to two decimals after parsing three-decimal CLI values. All values, means and SD were checked against raw Trial 2-4. CLI Averages includes all four trials and therefore is not the published three-trial metric. No pooling across engines/models. Timing exclusions: raw time_to_first_token_s mean 0.85 is explicitly computed as 512/prompt_tps by runner lines 718-719, not independently measured client or request TTFT. TTFT field is intentionally empty. Overall wall_time_s=680.92 measures the entire subprocess and can include HF downloads, model load and warmup; it is not inference-only latency or end-to-end response timing. As a reference audit only, official MLX-LM tag v0.31.2 benchmark.py uses batch 1 random token IDs, internal warmup, disables EOS, and forwards stream_generate rates. generate.py starts the prompt timer before generation and closes it on the first generated token; PP is prompt.size/prompt_time. TG counts n+1 tokens over elapsed time after that boundary, so its numerator includes the first token while its first-token time is in PP. These are engine-reported native rates, not transport/client rates. The actual installed MLX-LM revision is absent; v0.31.2 is corroborating implementation, not a tested-run version. Artifact: mlx-community/Qwen2.5-7B-Instruct-4bit. Source-configured HF artifact joined by model key; raw JSON omits repository/digest; historical tested revision unknown. Raw quant is "N/A" because the MLX run dictionary lacks a quant key; quantization identification comes from source config/profile plus HF metadata, not that raw field. No weight digest is logged. Current artifact metadata inspected 2026-10-13 at HF revision c26a38f6a37d0a51b4e9a1eb3026530fa35d9fed: MLX 4-bit group 64 (mode not explicitly named in config). This revision is a current metadata reference, not a historical run pin. Model-card conversion-tool versions are not installed benchmark engine versions. Model context capacities from current HF metadata are not active benchmark allocations. Memory: raw retained peak_memory values [4.833, 4.833, 4.834], maximum 4.834 in engine output units. Wrapper overwrites peak_vram_mb with this value times 1024 (4950.0), but leaves a stale unified_memory_system_wide note. It is MLX allocator/process peak reporting, not dedicated VRAM or system-wide resident usage. Reference MLX-LM v0.31.2 reports decimal GB (bytes/1e9), so wrapper times-1024 conversion may be unit-inconsistent; actual installed version is unknown. Dedicated-VRAM and memory-used fields remain empty. Config size_gb is an estimate, not measured disk bytes, and is not substituted for residency. No Mac power measurements. Sources: https://github.com/patrickwhelan-uk/llm-inference-benchmarks/blob/11f3f8f931dc97a71499e7b9c2583d010a3c160b/results/m2-mac-studio/qwen2.5-7b_default_20260402_222500.json https://github.com/patrickwhelan-uk/llm-inference-benchmarks/blob/11f3f8f931dc97a71499e7b9c2583d010a3c160b/config-m2-mac-studio.yaml https://github.com/patrickwhelan-uk/llm-inference-benchmarks/blob/11f3f8f931dc97a71499e7b9c2583d010a3c160b/run_benchmarks.py#L624-L738 https://github.com/patrickwhelan-uk/llm-inference-benchmarks/blob/11f3f8f931dc97a71499e7b9c2583d010a3c160b/hardware/m2-mac-studio.md https://huggingface.co/mlx-community/Qwen2.5-7B-Instruct-4bit/blob/c26a38f6a37d0a51b4e9a1eb3026530fa35d9fed/config.json https://github.com/ml-explore/mlx-lm/blob/v0.31.2/mlx_lm/benchmark.py https://github.com/ml-explore/mlx-lm/blob/v0.31.2/mlx_lm/generate.py#L658-L753 Structured PP/TG fields are rounded to 0.1 tokens/s; the source means, sample deviations and all three retained trial values above preserve the published precision.

generation:

128.3 tokens/s

prompt processing (prefill):

743.4 tokens/s

quant:

4-bit (source config) (MLX)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

First-party historical Patrick Whelan benchmark, inspected 2026-10-13. Source snapshot 11f3f8f931dc97a71499e7b9c2583d010a3c160b verified by archive PAX comment and GitHub currentOid; pinned runner/config/hardware/VLM files byte-match archive. This pins the published evidence, not the executable used in April. No inference rerun or model-weight download. Hardware: one participating Apple M2 Max integrated GPU, 30 GPU cores, 12 CPU cores and 64 GB shared unified memory. Raw system data confirms those values; hardware profile explicitly says no discrete GPU and all inference on the M2 Max GPU. Shared 64 GB is system capacity, not dedicated VRAM or resident model use. Raw environment: Darwin 25.3.0 / ARM64_T6020 and Python 3.14.3. Do not borrow stale hardware-profile Sequoia 15.x or ai-gateway/vllm-mlx labels. Power, thermals, clocks, swap, wired-memory limit, GPU utilization and competing workload are unknown. Dates: raw system_info.timestamp=2026-04-03T22:35:10.427177+00:00; this is collected once before a run list, so it is a campaign/system-capture timestamp, not exact model start. Result filename ministral-3-3b_default_20260403_234205.json uses local completion time with no timezone recorded. README updated 2026-04-04 is a publication label, not every measurement date. Configuration: direct python -m mlx_lm benchmark on Apple MLX/Metal, no HTTP server involved. MLX/MLX-LM/MLX-VLM installed versions and exact runtime commit are not retained in JSON. Requested prompt length 512 and generation length/cap 128, printed batch_size=1; three measured trials retained after discarding Trial 1, plus an internal/explicit warmup. Achieved token-count telemetry is not stored, so actual input/output fields remain unknown. Source context_size=4096, threads=8 and gpu_layers=-1 are explicitly unused for MLX and are not recorded here as active allocation, CPU threads or layer placement. KV/cache precision, prefill chunk size, sampling settings, thinking behavior and memory-cache reuse are unreported for the historical runtime; no drafter is passed by the wrapper. Native rates: PP 743.44 tok/s (sample SD 2.43; retained [746.147, 742.742, 741.442]); TG 128.31 tok/s (sample SD 0.15; retained [128.17, 128.467, 128.289]). Source aggregation rounds the three-trial arithmetic mean/sample SD to two decimals after parsing three-decimal CLI values. All values, means and SD were checked against raw Trial 2-4. CLI Averages includes all four trials and therefore is not the published three-trial metric. No pooling across engines/models. Timing exclusions: raw time_to_first_token_s mean 0.69 is explicitly computed as 512/prompt_tps by runner lines 718-719, not independently measured client or request TTFT. TTFT field is intentionally empty. Overall wall_time_s=414.99 measures the entire subprocess and can include HF downloads, model load and warmup; it is not inference-only latency or end-to-end response timing. As a reference audit only, official MLX-LM tag v0.31.2 benchmark.py uses batch 1 random token IDs, internal warmup, disables EOS, and forwards stream_generate rates. generate.py starts the prompt timer before generation and closes it on the first generated token; PP is prompt.size/prompt_time. TG counts n+1 tokens over elapsed time after that boundary, so its numerator includes the first token while its first-token time is in PP. These are engine-reported native rates, not transport/client rates. The actual installed MLX-LM revision is absent; v0.31.2 is corroborating implementation, not a tested-run version. Artifact: mlx-community/Ministral-3-3B-Instruct-2512-4bit. Source-configured HF artifact joined by model key; raw JSON omits repository/digest; historical tested revision unknown. Raw quant is "N/A" because the MLX run dictionary lacks a quant key; quantization identification comes from source config/profile plus HF metadata, not that raw field. No weight digest is logged. Current artifact metadata inspected 2026-10-13 at HF revision a962dcb09eee4169c890e544c9eb938f1113fdee: MLX 4-bit group 64 affine. This revision is a current metadata reference, not a historical run pin. Model-card conversion-tool versions are not installed benchmark engine versions. Model context capacities from current HF metadata are not active benchmark allocations. Memory: raw retained peak_memory values [2.522, 2.523, 2.523], maximum 2.523 in engine output units. Wrapper overwrites peak_vram_mb with this value times 1024 (2583.6), but leaves a stale unified_memory_system_wide note. It is MLX allocator/process peak reporting, not dedicated VRAM or system-wide resident usage. Reference MLX-LM v0.31.2 reports decimal GB (bytes/1e9), so wrapper times-1024 conversion may be unit-inconsistent; actual installed version is unknown. Dedicated-VRAM and memory-used fields remain empty. Config size_gb is an estimate, not measured disk bytes, and is not substituted for residency. No Mac power measurements. Sources: https://github.com/patrickwhelan-uk/llm-inference-benchmarks/blob/11f3f8f931dc97a71499e7b9c2583d010a3c160b/results/m2-mac-studio/ministral-3-3b_default_20260403_234205.json https://github.com/patrickwhelan-uk/llm-inference-benchmarks/blob/11f3f8f931dc97a71499e7b9c2583d010a3c160b/config-m2-mac-studio.yaml https://github.com/patrickwhelan-uk/llm-inference-benchmarks/blob/11f3f8f931dc97a71499e7b9c2583d010a3c160b/run_benchmarks.py#L624-L738 https://github.com/patrickwhelan-uk/llm-inference-benchmarks/blob/11f3f8f931dc97a71499e7b9c2583d010a3c160b/hardware/m2-mac-studio.md https://huggingface.co/mlx-community/Ministral-3-3B-Instruct-2512-4bit/blob/a962dcb09eee4169c890e544c9eb938f1113fdee/config.json https://github.com/ml-explore/mlx-lm/blob/v0.31.2/mlx_lm/benchmark.py https://github.com/ml-explore/mlx-lm/blob/v0.31.2/mlx_lm/generate.py#L658-L753 Structured PP/TG fields are rounded to 0.1 tokens/s; the source means, sample deviations and all three retained trial values above preserve the published precision.