Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8 27B at 31.8 t/s generation and 391.2 t/s prompt processing on an M3 Ultra 512GB.
Setup is MLX with Q4 and a fresh KV cache per call, batch 1 on one device, at 32768 input tokens and 128 output tokens.
The figures are separate five-trial medians, not pooled across lengths; the 32768-token row measured 31.768 t/s generation and 391.157 t/s prompt processing with a 23.246GB peak.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.6 35B-A3B at 89.3 t/s generation and 2258.1 t/s prompt processing on an M3 Ultra 512GB.
Setup is MLX with a 4-bit quant, batch 1 on one device, at 32768 input tokens and 128 output tokens.
The figures are separate five-trial medians, not pooled across lengths; the 32768-token run peaked at 23.999GB of allocator memory.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.5 122B-A10B at 49.5 t/s generation and 784.5 t/s prompt processing on an M3 Ultra 512GB.
Setup is MLX with a 4-bit quant, batch 1 on one device, at a 32768-token input and 128-token output.
The figures are separate five-trial medians, not pooled across lengths; the same run peaked at 76.32GB of allocator high-water memory.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
NetworkChuck campaign measured 2026-09-20, recovered 2026-10-13; historical source evidence, no new/local run. Merged window 2026-09-20T14:41:54.718Z–2026-09-20T16:55:25.045Z; native-cell timestamps unavailable. Credit NetworkChuck, CC BY 4.0 data/docs. Repository: https://github.com/theNetworkChuck/mac-studio-m5-ultra-local-ai/ . Native data: https://github.com/theNetworkChuck/mac-studio-m5-ultra-local-ai/blob/main/data/language-models/engine.json . Client summary: https://github.com/theNetworkChuck/mac-studio-m5-ultra-local-ai/blob/main/data/language-models/summary.json . Client requests: https://github.com/theNetworkChuck/mac-studio-m5-ultra-local-ai/blob/main/data/language-models/runs.jsonl . Method: https://github.com/theNetworkChuck/mac-studio-m5-ultra-local-ai/blob/main/docs/method.md .
Headline: native 32768-input/128-generation cell, PP median 570.075, TG median 35.801 tokens/s across five trials; not pooled-duration or client rates. Context 32768 is actual native input length, not supported maximum or configured KV capacity. Batch 1.
Hardware: one Apple Mac Studio M3 Ultra (32-core CPU, 80-core GPU), CPU core composition 24 performance+8 efficiency, 512 GB installed unified memory, macOS 26.5.1, GPU architecture applegpu_g15d. Both review units had 8 TB storage. Installed unified memory is not dedicated VRAM. A stuck background indexing process occupied 100% of one of this M3 Ultra's 32 CPU cores throughout this campaign, until 2026-09-21 10:10 CT. The effect was not measured by rerunning; do not silently correct the rates. This was Chuck's own machine. Power, energy, thermals, clock rates, exact storage part, explicit GPU-offload count and swap state are not reported for these rows.
Model: mlx-community/Qwen3-14B-4bit, 4-bit MLX Safetensors; public identity verified 2026-10-13: https://huggingface.co/mlx-community/Qwen3-14B-4bit . Source calls it dense, nominal 14B total/active, local folder ~/bench-models/Qwen3-14B-4bit. Source disk label 7.8 GB is not runtime memory. Tested HF revision/hash unrecorded; author says weights were not hash-verified. Current HF main, conversion version and download size do not identify the historical snapshot/runtime. Quant group size/mixed-precision exceptions unverified.
Runtime: mlx-lm 0.31.3 and MLX 0.32.2; server fingerprint 0.31.3-0.32.2-macOS-26.5.1-arm64-arm-64bit-applegpu_g15d. Native harness command is mlx_lm.benchmark --model <local-model-folder> -p <prompt-count> -g 128 -n 5, verified in https://github.com/theNetworkChuck/mac-studio-m5-ultra-local-ai/blob/main/harness/src/bench/ssh.ts . Model loaded before timing; one discarded warm-up then five trials. Harness median/min/max uses three-decimal printed trials; raw output retained. Warm-loaded, excludes cold loading.
Native method: versioned https://github.com/ml-explore/mlx-lm/blob/v0.31.3/mlx_lm/benchmark.py seeds RNG 0, uses random token IDs, disables EOS stopping and calls stream_generate; no HTTP essay/chat workload. Published command leaves batch 1, prefill step 2048 and delay 0 defaults. https://github.com/ml-explore/mlx-lm/blob/v0.31.3/mlx_lm/generate.py defines prompt_tps=prompt.size/runtime time to first generated token, including that boundary rather than pure matrix-kernel time. Generation timing restarts there; generation_tps=generated count/runtime generation-window time, including final detokenizer handling, final count 128. This differs from client (128−1)/first-to-last text. No prompt_cache passed, so generate_step creates fresh KV. No draft model, KV-quantization or activation-quantization flag supplied. Flash/kernel selection and runtime build commit unrecorded; hardware alone does not establish a toggle.
All native rows, input/output; PP median [min–max]; TG median [min–max], tokens/s; source peak-memory GB; trial count:
512/128; PP 813.779 [812.584–814.936]; TG 69.365 [69.32–69.391]; peak 8.826 GB; n=5
4096/128; PP 814.254 [813.998–815.156]; TG 62.896 [62.85–62.918]; peak 9.701 GB; n=5
16384/128; PP 690.643 [690.481–691.238]; TG 47.455 [47.42–47.465]; peak 11.602 GB; n=5
32768/128; PP 570.075 [569.782–570.199]; TG 35.801 [35.795–35.812]; peak 14.332 GB; n=5
Memory semantics: source peakMemoryGb is the maximum of printed native peak_memory values across five trials. Upstream peak_memory is mx.get_peak_memory()/1e9, an MLX allocator high-water mark in decimal GB; it is not RSS, installed RAM, dedicated VRAM, or a separately reset per-trial allocation. Warm-up and earlier allocation in that benchmark process can contribute to the high-water mark. No model-load timing or energy number is supplied by these rows.
Separate HTTP arm, same model/Mac: serial streaming /v1/chat/completions from LAN controller to mlx_lm.server; --prompt-cache-size 0 and --chat-template-args '{"enable_thinking":false}' , temperature 0, max_tokens 128. Prompt repeats prompts/sample.txt with unique run-ID prefix and summary instruction. One discarded warm-up, five measured requests/cell, 30 s between cells. Single 100K requests, if listed, are not five-repeat cells. Actual counts from usage. All checked raw requests in this group: cachedTokens 0, outputTokens 128, firstWordField content. HTTP cache evidence does not mean cold model loading.
Controller monotonic timing begins before request; first text is first nonempty delta. Reading=input tokens/client first-text seconds, including LAN/HTTP. Writing=(output−1)/(last-text−first-text seconds). Total ends at stream completion. These differ from native metrics. Implementation: https://github.com/theNetworkChuck/mac-studio-m5-ultra-local-ai/blob/main/harness/src/lane.ts and https://github.com/theNetworkChuck/mac-studio-m5-ultra-local-ai/blob/main/harness/src/timing.ts .
All HTTP rows, each metric shown as median [min–max]; nominal length; actual token counts; client reading/writing tokens/s; first-text and total seconds; repetitions:
512 (512 nominal); actual input 507 [507–508], output 128 [128–128]; TTFT-derived reading 681.304886619 [680.15894523–683.748234999]; client writing 61.7620769176 [61.5733692754–61.8337956262]; first text 0.744678 [0.741501–0.745628] s; total 2.801364 [2.797872–2.808342] s; n=5
4K (4096 nominal); actual input 4097 [4095–4097], output 128 [128–128]; TTFT-derived reading 776.541288513 [774.603443497–777.677154077]; client writing 55.7032446921 [55.5113601595–55.8309246173]; first text 5.275959 [5.266477–5.289158] s; total 7.555642 [7.54761–7.577066] s; n=5
16K (16384 nominal); actual input 16280 [16279–16281], output 128 [128–128]; TTFT-derived reading 669.146411504 [668.529108244–669.423496892]; client writing 41.7188837078 [41.4444331315–41.7777862576]; first text 24.330998 [24.320927–24.350473] s; total 27.382021 [27.368085–27.394455] s; n=5
32K (32768 nominal); actual input 32521 [32520–32521], output 128 [128–128]; TTFT-derived reading 550.950577802 [550.6959282–551.18283725]; client writing 31.5317764581 [31.4994611608–31.5856519565]; first text 59.025261 [59.0022–59.054368] s; total 63.05821 [63.031217–63.090287] s; n=5
Independent M3/M5 invocations were merged with shared settings checked. Different OS, M3 indexing load and unverified weight snapshots confound silicon-only comparisons. No independent model-quality evaluation.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8-27B at 29 t/s generation and 274 t/s prompt processing on an M3 Ultra 96GB.
Setup is llama.cpp 0.4.0 with the Q4_K_M GGUF, run natively through llama-bench on one Apple GPU via Metal.
The native benchmark gives 28.97 +/- 0.03 tok/s generation and 273.98 +/- 0.03 tok/s prompt processing; context length, batch settings and offload flags are unreported.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
long-context
User reports Gemma 4 31B at 21.4 t/s generation and 235.1 t/s prompt processing on a single Mac Studio with an M3 Ultra and 512 GB unified memory.
Setup is mlx-vlm with 4bit MLX, 36,777 actual input tokens and 128 generated tokens per run, three runs, peak memory 29.7 GB.
The figures are medians; prompt processing p10/p90 are 234.8 and 236.1 t/s, and KV quantization, flash attention and speculative decoding are not exposed.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
long-context
User reports gpt-oss 117B-A5.1B at 62.6 t/s generation and 1,077.3 t/s prompt processing on an M3 Ultra 512GB.
Setup is MLX with MXFP4/BF16 MLX weights, 32,006 actual input tokens and 128 generated tokens per run, three runs, peak memory 67.2 GB.
The figures are medians; prompt processing p10/p90 are 1,074.7 and 1,078.3 t/s, and the configured context cap is unknown.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
long-context
User reports Qwen3.5 397B-A17B at 33 t/s generation and 386.4 t/s prompt processing on a single Mac Studio with an M3 Ultra and 512 GB unified memory.
Setup is mlx-vlm with 4bit MLX, run at long context with 34,448 actual input tokens and 128 generated tokens per run, three runs, peak memory 234.9 GB.
The figures are medians; prompt processing p10/p90 are 359.4 and 428.1 t/s, and median wall time was 105 seconds. KV quantization, flash attention and speculative decoding are not exposed, and cold-cache prefill is not established.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
long-context
User reports Gemma 4 26B-A4B at 67.6 t/s generation and 1,645.0 t/s prompt processing on a Mac Studio with an M3 Ultra and 512 GB unified memory.
Setup is mlx-vlm with 4bit MLX, run on a 36,777-token prompt with 128 generated tokens per run over three runs; both figures are medians.
Peak memory was 20.0 GB and median wall time 27 seconds; the configured context cap and KV quantization are not stated.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.5 9B at 77.8 t/s generation and 1010.3 t/s prompt processing on an M3 Ultra 256GB.
Setup is llama.cpp with Q4_K_M, run through native llama-bench with -p 512 -n 128 -r 3.
The 77.83 ± 0.11 t/s and 1010.32 ± 1.28 tokens/s are native benchmark rates for 512/128 test tokens, not configured maximum context; KV precision, flash attention and GPU-offload count are unreported.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Ling-3.0 Flash at 73.1 t/s on an M3 Ultra 256GB.
Setup is oMLX with oQ5 affine mixed precision, group 64, on a 124B total / 5.1B active model.
The figure is the median of three greedy 256-token runs (73.23, 73.10, 73.09) after a 256-token warm-up, with a 39-token prompt; peak memory is reported at 87.44 GB.
MLX affine 4-bit, group 64, 4.501 effective bpw (MLX)
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8 at 37.9 t/s on an M3 Ultra 512GB, from a single run of one 68-token prompt and 120 generated tokens.
Setup is MLX with affine 4-bit quantization, group size 64 and 4.501 effective bpw, peaking at 15.5 GB of memory.
The author describes the figure as a rough guide rather than a repeated controlled benchmark; no prompt-processing throughput or TTFT is given.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
vision
User reports Qwen3.8 at 13.2 t/s on an M3 Ultra with 512GB unified memory.
Setup is mlx-vlm 0.6.13 with the unquantized bf16 MLX VLM build, peak memory 55.8 GB.
The figure comes from one 470-token image-plus-text prompt, one run per build, presented as an order-of-magnitude guide rather than repeated benchmarking.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
coding
User reports Tiel-Coder 35B-A3B at 201.3 t/s median native decode on an Apple Mac Studio M3 Ultra with 512 GiB unified memory.
Setup is AX Engine with the AXQ-MXFP4 MLX export, explicit throughput-MTP at maximum draft depth three, expert streaming off, and cold request KV with no prefix reuse.
The figure is native decode only on a python-lru prompt with 194 input tokens and 256 total output tokens; completion throughput including TTFT is 160.35 t/s and median native TTFT is 329.74 ms.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports GLM-5.3-Flash Abliterated at 48.8 tok/s decode on a Mac Studio M3 Ultra 256 GB.
Setup is mlx-serve 26.10.1 with MLX 4-bit weights and 131072 context, MTP on with draft rerank, resident size about 172–181 GB.
Real sessions at 50k–85k tokens decode about 45–63 tok/s with prefill 150–300 tok/s on cache hits. A short probe at 8192 context measured 49.9 tok/s. TensorFold 1.0.5 on the same Mac left the MTP head on disk and one short reply measured 15.7 tok/s.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
summarizationvisionlong-context
User reports Qwen3.8 27B at 14.0 tok/s generation and 93.1 tok/s prompt processing on a Mac Studio M3 Ultra with 256GB unified memory.
Setup is Ollama with Q4_K_M (17GB) and five timed runs averaged; the predecessor qwen3.6:27b reached 28.6 tok/s on the same machine, and the 1-bit UD-IQ1_M quant (6.7GB) ran at 27.2 tok/s generation and 309 tok/s prompt processing in llama.cpp.
User notes the new hybrid attention architecture is slower per token but uses roughly a third the tokens per answer, so wall-clock per answer is close to a tie; community figures cited for Strix Halo (20.4 tok/s), dual RTX 3090 (~60 tok/s) and RTX 5090 (75-140 tok/s) are not the author's own measurements.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8-Flash-Next at 15.8 tok/s generation and 140.0 tok/s prompt processing on an M3 Max 64GB.
Setup is oMLX with an oQ4e quant, Lightning MTP on, MoE expert offload on and 50% resident experts, running a 99GB model at up to 130k context.
Figures come from a 13-request session with PI: 321,309 total prefill tokens, 292,209 cached (90.9% cache efficiency), so the prompt rate excludes cached tokens.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8 27B at 43.38 t/s generation on a Mac Studio M3 Ultra 256GB, the median of three requests with an 8,156-token prompt.
Setup is Rapid-MLX 0.13.4 with the rapid-mlx MLX 4-bit MTP checkpoint, one request, with the engine's adaptive MTP on by default for this checkpoint.
The source's 330.8 t/s prefill is the prompt divided by time to first token, so no prompt speed is stored. The figures come from the Rapid-MLX benchmark docs, not an independent rerun.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports a performance issue with Qwen3.8 27B on an M3 Max 48GB, comparing mlx-serve against LM Studio. The mlx-serve run used the ddalcu/Qwen3.8-27B-MLX-Serve-4bit model at 128k context and achieved 59.6 tok/s decode (54.9-60.2) and 196.4 tok/s prefill, while LM Studio with the Swift-1.5-Qwen3.8-27B-Splash model reached 96.4 tok/s decode (89.1-96.8) and 210.5 tok/s prefill.
Both runs used speculative decoding (MTP/draft active). The mlx-serve run showed a 42.1% slowdown under sustained load (59.6 to 34.5 tok/s over 8m 4s), and context scaling was worse than LM Studio at longer contexts (37.2 vs 76.5 tok/s at ~4.2k, 35.6 vs 60.5 at ~8.2k, 34.6 vs 57.6 at ~16.3k).
The user reports mlx-serve is 38% slower on decode overall and up to 51% slower at ~4.2k context, while being 18% faster on TTFT.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User benchmarks Llama 3.1 405B at 2.99 t/s on a single M3 Ultra 512GB node.
Setup is MLX with Q4 quantisation at 1K context, single node.
This is part of a 200+ configuration campaign across 5 models and 9 topologies on a 5-node M3 Ultra cluster; the report also covers Qwen 2.5 32B, Mixtral 8x7B, DeepSeek V3.2 and Kimi K2.5, with TP and PP scaling results.