Local model performance on your hardware

generation:

29.2 tokens/s

quant:

6-bit (MLX)

User reports 29.2 t/s generation. Source evidence (verbatim): Source: https://alooftwaffle.substack.com/p/96-hours-with-dual-dgx-sparks-and # First-hand Mac Studio M3 Ultra 512GB Qwen3.5-397B-A17B 6-bit mlx-vlm generation measurement Original first-hand report by AlooftWaffle, published March 27, 2026: https://alooftwaffle.substack.com/p/96-hours-with-dual-dgx-sparks-and . Historical source-reported measurements, not a new benchmark or independent reproduction. Selected configuration: one Mac Studio M3 Ultra with 512GB unified memory, running Qwen3.5-397B-A17B at publisher-described 6-bit precision using mlx-vlm. The shortest reported context row gives generation 29.2 tokens/s. The source labels this row 4K context; whether that label denotes actual occupied input or configured capacity is not established. Actual fresh input count, cached prefix, output count and runtime version are unknown. Leave structured context length unknown. Do not replace it with 4096 or infer zero cache. CPU/GPU core counts, sampling, thinking mode, speculative decoding, request concurrency, warmup, repeats, statistical aggregation, exact measurement date, OS, KV precision, Flash Attention, offload, temperatures, power and peak resident memory are unreported. The source's longer Mac rows report generation 28.6 tokens/s at the 8K label and 26.7 tokens/s at 32K; preserve these as separate source rows, not an average. The author also calls values 317/537/476 tokens/s 'prefill throughput (time to first token)'; native timing boundaries and counters are absent, so leave structured prompt throughput and TTFT unknown rather than converting or equating these metrics. Verified upstream model-family reference: https://huggingface.co/Qwen/Qwen3.5-397B-A17B . This is a base/reference repository, not the tested 6-bit artifact. The tested quantization repository, revision, file names, hashes and quantization recipe are unknown; do not assign a community pack or GGUF Q6 code. The source describes nearly 300GB of downloaded model files; that is download/storage volume, not measured resident RAM or a peak-memory measurement. The same article's dual-Spark INT4-AutoRound/vLLM measurements and embedding sentences/s are different hardware, quantization and workloads; none is the selected Mac generation speed or a controlled comparison isolating hardware. Full original source remains the URL above; no author contact or comments were made.

generation:

81.2 tokens/s

prompt processing (prefill):

2324.6 tokens/s

quant:

Q8_0 (GGUF)

kv:

F16

flash attention:

on

User reports Qwen3.6-35B-A3B Q8_0 at 81.2 t/s generation and 2324.6 t/s prompt processing on a Mac Studio with an Apple M3 Ultra and 512 GiB unified memory. Setup is llama.cpp with Metal GPU offload, Flash Attention on, F16 KV cache, 8 threads, and batch/ubatch 512/512; the Mac-only control arm uses one GPU. PP4096 and TG128 are separate synthetic tests, with TG128 starting from an empty context rather than generation after prefill. The article also reports a continuous 4096-input plus 128-output workload at 3.409 seconds, and a separate tuning pass at the same batch/ubatch gives PP4096 2323.37 and TG128 80.68 t/s.

generation:

80.9 tokens/s

prompt processing (prefill):

2779.0 tokens/s

quant:

MXFP4_MOE (GGUF)

flash attention:

on

mtp (multi-token prediction):

off

User reports Qwen3.5-35B-A3B at 80.9 t/s generation and 2779 t/s prompt processing on a Mac Studio M3 Ultra with 512 GB unified memory. Setup is llama.cpp b8783 with the MXFP4_MOE quantization, Metal backend, F16 K and V cache, and -ngl 999 offloading to one GPU. The figures come from llama-bench with -p 2048 -n 32 -ub 2048 -fa 1, five repetitions, batch 2048 and context depth 0; the source rates are 80.93 t/s and 2778.95 t/s.

generation:

87.8 tokens/s

prompt processing (prefill):

2595.3 tokens/s

quant:

MXFP4_MOE (GGUF)

flash attention:

on

mtp (multi-token prediction):

off

User reports Gemma 4 26B-A4B at 87.8 t/s generation and 2595.3 t/s prompt processing on an M3 Ultra Mac Studio with 512 GB of unified memory. Setup is llama.cpp build b8783 with the MXFP4_MOE quantization, Metal backend, F16 K and V cache, and -ngl 999, running one GPU. The figures come from separate llama-bench phases: PP2048 at 2595.28 t/s and TG32 at 87.81 t/s, each the mean of five repetitions at context depth 0.

generation:

84.9 tokens/s

prompt processing (prefill):

2634.8 tokens/s

quant:

UD-Q8_K_XL (GGUF)

flash attention:

on

mtp (multi-token prediction):

off

User reports Gemma 4 26B-A4B at 84.9 t/s generation and 2634.8 t/s prompt processing on an M3 Ultra Mac Studio with 512 GB of unified memory. Setup is llama.cpp build b8783 with the UD-Q8_K_XL quant, Metal backend, F16 K and V cache, and -ngl 999 offloading to one GPU. The figures come from llama-bench with -p 2048 -n 32 -ub 2048 -fa 1, five repetitions, batch 2048 and context depth 0; the article's llama-server, quality, VLM and long-context tests are separate.

generation:

88.1 tokens/s

prompt processing (prefill):

2554.2 tokens/s

quant:

UD-Q5_K_M (GGUF)

flash attention:

on

mtp (multi-token prediction):

off

User reports Gemma 4 26B-A4B at 88.1 t/s generation and 2554.2 t/s prompt processing on an M3 Ultra Mac Studio with 512 GB of unified memory. Setup is llama.cpp build b8783 with UD-Q5_K_M and F16 K and V cache, Metal backend, one GPU, -ngl 999, batch 2048, five repetitions. The figures are separate llama-bench phases: PP2048 at 2554.16 t/s and TG32 at 88.08 t/s, not a paired 2048-token prompt followed by 32 generated tokens.

generation:

62.4 tokens/s

prompt processing (prefill):

2649.3 tokens/s

quant:

BF16 (GGUF)

flash attention:

on

mtp (multi-token prediction):

off

User reports Gemma 4 26B-A4B at 62.4 t/s generation and 2649.3 t/s prompt processing on an M3 Ultra Mac Studio with 512 GB of unified memory. Setup is llama.cpp build b8783 with BF16 weights, F16 K and V cache, Metal backend, and -ngl 999, using -p 2048 -n 32 -ub 2048 -fa 1. The figures are separate llama-bench phases: PP evaluates 2048 synthetic tokens and TG 32 sequential single tokens, with five repetitions and no speculative decoding.

generation:

14.9 tokens/s

quant:

4-bit (MLX)

mtp (multi-token prediction):

off

User reports Llama 3.1 70B at 14.9 t/s on a Mac Studio with an M3 Ultra and 512 GB of unified memory. Setup is MLX with 4-bit quantization, group_size 64, no speculative decoding or batching, at a 4k prompt (4138 actual tokens) with 128 output tokens. The figure is one observation, not a repeated-run mean, from a single timed generation.

generation:

14.3 tokens/s

quant:

4-bit (MLX)

mtp (multi-token prediction):

off

User reports Qwen2.5-72B-Instruct-4bit at 14.3 t/s on a Mac Studio with an M3 Ultra and 512 GB of unified memory. Setup is MLX 0.31.1 with mlx-lm 0.31.2, 4-bit group_size=64 quantization, no speculative decoding or batching, at a 4132-token prompt with 128 output tokens. The figure is one observation, not a repeated-run mean; the source's more precise value is 14.331110981167 t/s, and the same sweep drops to 6.056028636 t/s at 131108 prompt tokens.

generation:

218.8 tokens/s

quant:

AXQ MXFP4 mixed (MLX)

mtp (multi-token prediction):

on

User reports Cyber-Tiel-Coder-35B-A3B at 218.8 t/s on an M3 Ultra with 512GB unified memory. Setup is the AX Engine with an AXQ MXFP4 trunk and protected MTP sidecar, one Metal GPU, expert streaming off, greedy decoding at MTP depth 3. The figure is the median decode rate over six samples on a 219-input/256-output python-lru workload, with a range of 217.75-220.11 t/s; completion including TTFT was 170.65 t/s.

generation:

6.8 tokens/s

quant:

Q8 (MLX)

User reports Llama 3.3 70B at 6.8 t/s on a Mac Studio M3 Ultra with 512GB unified memory. Setup is MLX with an 8-bit quantized mlx-community conversion, served over LAN by mlx_lm.server with temperature 0 and a 128-token output cap. The 6.8 t/s is the median of five measured requests at a nominal 32K context, with actual prompt median 32396 tokens; it is client-derived decode throughput, not native generation speed.

generation:

9.2 tokens/s

prompt processing (prefill):

295.9 tokens/s

quant:

BF16 (MLX)

User reports Gemma 4 31B at 9.2 t/s generation and 295.9 t/s prompt processing on an M3 Ultra 512GB. Setup is MLX with BF16 weights at 32768 input tokens and 128 output tokens, batch 1 on one device, with a fresh KV cache per call and no draft or KV-quant flags. The 32768-token row is a five-trial median, not pooled across lengths, with a peak allocator high-water of 71.714GB; a stuck indexer consumed one CPU core during the tests and its effect was not measured.

generation:

31.8 tokens/s

prompt processing (prefill):

391.2 tokens/s

quant:

Q4 (MLX)

User reports Qwen3.8 27B at 31.8 t/s generation and 391.2 t/s prompt processing on an M3 Ultra 512GB. Setup is MLX with Q4 and a fresh KV cache per call, batch 1 on one device, at 32768 input tokens and 128 output tokens. The figures are separate five-trial medians, not pooled across lengths; the 32768-token row measured 31.768 t/s generation and 391.157 t/s prompt processing with a 23.246GB peak.

generation:

89.3 tokens/s

prompt processing (prefill):

2258.1 tokens/s

quant:

Q4 (MLX)

User reports Qwen3.6 35B-A3B at 89.3 t/s generation and 2258.1 t/s prompt processing on an M3 Ultra 512GB. Setup is MLX with a 4-bit quant, batch 1 on one device, at 32768 input tokens and 128 output tokens. The figures are separate five-trial medians, not pooled across lengths; the 32768-token run peaked at 23.999GB of allocator memory.

generation:

49.5 tokens/s

prompt processing (prefill):

784.5 tokens/s

quant:

Q4 (MLX)

User reports Qwen3.5 122B-A10B at 49.5 t/s generation and 784.5 t/s prompt processing on an M3 Ultra 512GB. Setup is MLX with a 4-bit quant, batch 1 on one device, at a 32768-token input and 128-token output. The figures are separate five-trial medians, not pooled across lengths; the same run peaked at 76.32GB of allocator high-water memory.

generation:

35.8 tokens/s

prompt processing (prefill):

570.1 tokens/s

quant:

Q4 (MLX)

NetworkChuck campaign measured 2026-09-20, recovered 2026-10-13; historical source evidence, no new/local run. Merged window 2026-09-20T14:41:54.718Z–2026-09-20T16:55:25.045Z; native-cell timestamps unavailable. Credit NetworkChuck, CC BY 4.0 data/docs. Repository: https://github.com/theNetworkChuck/mac-studio-m5-ultra-local-ai/ . Native data: https://github.com/theNetworkChuck/mac-studio-m5-ultra-local-ai/blob/main/data/language-models/engine.json . Client summary: https://github.com/theNetworkChuck/mac-studio-m5-ultra-local-ai/blob/main/data/language-models/summary.json . Client requests: https://github.com/theNetworkChuck/mac-studio-m5-ultra-local-ai/blob/main/data/language-models/runs.jsonl . Method: https://github.com/theNetworkChuck/mac-studio-m5-ultra-local-ai/blob/main/docs/method.md . Headline: native 32768-input/128-generation cell, PP median 570.075, TG median 35.801 tokens/s across five trials; not pooled-duration or client rates. Context 32768 is actual native input length, not supported maximum or configured KV capacity. Batch 1. Hardware: one Apple Mac Studio M3 Ultra (32-core CPU, 80-core GPU), CPU core composition 24 performance+8 efficiency, 512 GB installed unified memory, macOS 26.5.1, GPU architecture applegpu_g15d. Both review units had 8 TB storage. Installed unified memory is not dedicated VRAM. A stuck background indexing process occupied 100% of one of this M3 Ultra's 32 CPU cores throughout this campaign, until 2026-09-21 10:10 CT. The effect was not measured by rerunning; do not silently correct the rates. This was Chuck's own machine. Power, energy, thermals, clock rates, exact storage part, explicit GPU-offload count and swap state are not reported for these rows. Model: mlx-community/Qwen3-14B-4bit, 4-bit MLX Safetensors; public identity verified 2026-10-13: https://huggingface.co/mlx-community/Qwen3-14B-4bit . Source calls it dense, nominal 14B total/active, local folder ~/bench-models/Qwen3-14B-4bit. Source disk label 7.8 GB is not runtime memory. Tested HF revision/hash unrecorded; author says weights were not hash-verified. Current HF main, conversion version and download size do not identify the historical snapshot/runtime. Quant group size/mixed-precision exceptions unverified. Runtime: mlx-lm 0.31.3 and MLX 0.32.2; server fingerprint 0.31.3-0.32.2-macOS-26.5.1-arm64-arm-64bit-applegpu_g15d. Native harness command is mlx_lm.benchmark --model <local-model-folder> -p <prompt-count> -g 128 -n 5, verified in https://github.com/theNetworkChuck/mac-studio-m5-ultra-local-ai/blob/main/harness/src/bench/ssh.ts . Model loaded before timing; one discarded warm-up then five trials. Harness median/min/max uses three-decimal printed trials; raw output retained. Warm-loaded, excludes cold loading. Native method: versioned https://github.com/ml-explore/mlx-lm/blob/v0.31.3/mlx_lm/benchmark.py seeds RNG 0, uses random token IDs, disables EOS stopping and calls stream_generate; no HTTP essay/chat workload. Published command leaves batch 1, prefill step 2048 and delay 0 defaults. https://github.com/ml-explore/mlx-lm/blob/v0.31.3/mlx_lm/generate.py defines prompt_tps=prompt.size/runtime time to first generated token, including that boundary rather than pure matrix-kernel time. Generation timing restarts there; generation_tps=generated count/runtime generation-window time, including final detokenizer handling, final count 128. This differs from client (128−1)/first-to-last text. No prompt_cache passed, so generate_step creates fresh KV. No draft model, KV-quantization or activation-quantization flag supplied. Flash/kernel selection and runtime build commit unrecorded; hardware alone does not establish a toggle. All native rows, input/output; PP median [min–max]; TG median [min–max], tokens/s; source peak-memory GB; trial count: 512/128; PP 813.779 [812.584–814.936]; TG 69.365 [69.32–69.391]; peak 8.826 GB; n=5 4096/128; PP 814.254 [813.998–815.156]; TG 62.896 [62.85–62.918]; peak 9.701 GB; n=5 16384/128; PP 690.643 [690.481–691.238]; TG 47.455 [47.42–47.465]; peak 11.602 GB; n=5 32768/128; PP 570.075 [569.782–570.199]; TG 35.801 [35.795–35.812]; peak 14.332 GB; n=5 Memory semantics: source peakMemoryGb is the maximum of printed native peak_memory values across five trials. Upstream peak_memory is mx.get_peak_memory()/1e9, an MLX allocator high-water mark in decimal GB; it is not RSS, installed RAM, dedicated VRAM, or a separately reset per-trial allocation. Warm-up and earlier allocation in that benchmark process can contribute to the high-water mark. No model-load timing or energy number is supplied by these rows. Separate HTTP arm, same model/Mac: serial streaming /v1/chat/completions from LAN controller to mlx_lm.server; --prompt-cache-size 0 and --chat-template-args '{"enable_thinking":false}' , temperature 0, max_tokens 128. Prompt repeats prompts/sample.txt with unique run-ID prefix and summary instruction. One discarded warm-up, five measured requests/cell, 30 s between cells. Single 100K requests, if listed, are not five-repeat cells. Actual counts from usage. All checked raw requests in this group: cachedTokens 0, outputTokens 128, firstWordField content. HTTP cache evidence does not mean cold model loading. Controller monotonic timing begins before request; first text is first nonempty delta. Reading=input tokens/client first-text seconds, including LAN/HTTP. Writing=(output−1)/(last-text−first-text seconds). Total ends at stream completion. These differ from native metrics. Implementation: https://github.com/theNetworkChuck/mac-studio-m5-ultra-local-ai/blob/main/harness/src/lane.ts and https://github.com/theNetworkChuck/mac-studio-m5-ultra-local-ai/blob/main/harness/src/timing.ts . All HTTP rows, each metric shown as median [min–max]; nominal length; actual token counts; client reading/writing tokens/s; first-text and total seconds; repetitions: 512 (512 nominal); actual input 507 [507–508], output 128 [128–128]; TTFT-derived reading 681.304886619 [680.15894523–683.748234999]; client writing 61.7620769176 [61.5733692754–61.8337956262]; first text 0.744678 [0.741501–0.745628] s; total 2.801364 [2.797872–2.808342] s; n=5 4K (4096 nominal); actual input 4097 [4095–4097], output 128 [128–128]; TTFT-derived reading 776.541288513 [774.603443497–777.677154077]; client writing 55.7032446921 [55.5113601595–55.8309246173]; first text 5.275959 [5.266477–5.289158] s; total 7.555642 [7.54761–7.577066] s; n=5 16K (16384 nominal); actual input 16280 [16279–16281], output 128 [128–128]; TTFT-derived reading 669.146411504 [668.529108244–669.423496892]; client writing 41.7188837078 [41.4444331315–41.7777862576]; first text 24.330998 [24.320927–24.350473] s; total 27.382021 [27.368085–27.394455] s; n=5 32K (32768 nominal); actual input 32521 [32520–32521], output 128 [128–128]; TTFT-derived reading 550.950577802 [550.6959282–551.18283725]; client writing 31.5317764581 [31.4994611608–31.5856519565]; first text 59.025261 [59.0022–59.054368] s; total 63.05821 [63.031217–63.090287] s; n=5 Independent M3/M5 invocations were merged with shared settings checked. Different OS, M3 indexing load and unverified weight snapshots confound silicon-only comparisons. No independent model-quality evaluation.

generation:

21.4 tokens/s

prompt processing (prefill):

235.1 tokens/s

quant:

4bit MLX (MLX)

User reports Gemma 4 31B at 21.4 t/s generation and 235.1 t/s prompt processing on a single Mac Studio with an M3 Ultra and 512 GB unified memory. Setup is mlx-vlm with 4bit MLX, 36,777 actual input tokens and 128 generated tokens per run, three runs, peak memory 29.7 GB. The figures are medians; prompt processing p10/p90 are 234.8 and 236.1 t/s, and KV quantization, flash attention and speculative decoding are not exposed.

generation:

62.6 tokens/s

prompt processing (prefill):

1077.3 tokens/s

quant:

MXFP4/BF16 MLX (MLX)

User reports gpt-oss 117B-A5.1B at 62.6 t/s generation and 1,077.3 t/s prompt processing on an M3 Ultra 512GB. Setup is MLX with MXFP4/BF16 MLX weights, 32,006 actual input tokens and 128 generated tokens per run, three runs, peak memory 67.2 GB. The figures are medians; prompt processing p10/p90 are 1,074.7 and 1,078.3 t/s, and the configured context cap is unknown.

generation:

33.0 tokens/s

prompt processing (prefill):

386.4 tokens/s

quant:

4bit MLX (MLX)

User reports Qwen3.5 397B-A17B at 33 t/s generation and 386.4 t/s prompt processing on a single Mac Studio with an M3 Ultra and 512 GB unified memory. Setup is mlx-vlm with 4bit MLX, run at long context with 34,448 actual input tokens and 128 generated tokens per run, three runs, peak memory 234.9 GB. The figures are medians; prompt processing p10/p90 are 359.4 and 428.1 t/s, and median wall time was 105 seconds. KV quantization, flash attention and speculative decoding are not exposed, and cold-cache prefill is not established.

generation:

67.6 tokens/s

prompt processing (prefill):

1645.0 tokens/s

quant:

4bit MLX (MLX)

User reports Gemma 4 26B-A4B at 67.6 t/s generation and 1,645.0 t/s prompt processing on a Mac Studio with an M3 Ultra and 512 GB unified memory. Setup is mlx-vlm with 4bit MLX, run on a 36,777-token prompt with 128 generated tokens per run over three runs; both figures are medians. Peak memory was 20.0 GB and median wall time 27 seconds; the configured context cap and KV quantization are not stated.