Local model performance on your hardware

generation:

13.9 tokens/s

prompt processing (prefill):

1237.0 tokens/s

quant:

BF16 (MLX)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

NetworkChuck, measured 2026-09-20; recovered 2026-10-13, not locally rerun. Measurements/docs: CC BY 4.0. Native data: https://github.com/theNetworkChuck/mac-studio-m5-ultra-local-ai/blob/main/data/language-models/engine.json ; HTTP summary/raw: https://github.com/theNetworkChuck/mac-studio-m5-ultra-local-ai/blob/main/data/language-models/summary.json ; https://github.com/theNetworkChuck/mac-studio-m5-ultra-local-ai/blob/main/data/language-models/runs.jsonl . Method: https://github.com/theNetworkChuck/mac-studio-m5-ultra-local-ai/blob/main/docs/method.md . Merged suite window 14:41:54.718–16:55:25.045 UTC; native cells have no individual timestamps. Native 32768-input/128-output headline: PP=1236.992, TG=13.91 tokens/s, separate five-trial medians, not pooled across lengths. Input is exactly 32768 synthetic tokens. Configured KV capacity unknown and blank. Batch 1; one device. Hardware: Apple Mac Studio M5 Ultra (36-core CPU, 80-core GPU); 256GB installed unified memory; macOS27.0; applegpu_g17d; CPU composition 12 super+24 performance; 8TB storage. Unified RAM and allocator peaks are not dedicated VRAM. M5 was an Apple review loan. Comparator M3 used macOS26.5.1 and had a stuck indexer consuming one CPU core. OS/background effects were not isolated. Power, thermals, clocks, swap and offload count unknown. Model: mlx-community/gemma-4-31b-it-bf16; 16-bit (bf16); MLX Safetensors. Public identity/precision verified 2026-10-13: https://huggingface.co/mlx-community/gemma-4-31b-it-bf16 . Source labels dense, nominal total/active 31/31B; rounded labels, not exact architecture counts. Tested HF revision/hash and group/mixed-precision details unknown; weights not hash-checked. Current HF does not identify the historical snapshot/runtime. Runtime: mlx-lm0.31.3/MLX0.32.2; fingerprint 0.31.3-0.32.2-macOS-27.0-arm64-arm-64bit-applegpu_g17d. Harness command: mlx_lm.benchmark --model <folder> -p <input> -g 128 -n 5. Command/parser: https://github.com/theNetworkChuck/mac-studio-m5-ultra-local-ai/blob/main/harness/src/bench/ssh.ts ; https://github.com/theNetworkChuck/mac-studio-m5-ultra-local-ai/blob/main/harness/src/bench/engine.ts . Model already loaded; one discarded warm-up, five trials. Rates printed to 3 decimals; harness uses their median/min/max, not tool means. Versioned runtime semantics: https://github.com/ml-explore/mlx-lm/blob/v0.31.3/mlx_lm/benchmark.py and https://github.com/ml-explore/mlx-lm/blob/v0.31.3/mlx_lm/generate.py . Seed 0, random IDs rather than HTTP essay, EOS disabled, batch 1, default prefill step 2048, no trial delay. Native prompt_tps=prompt.size/time through first generated token; includes first-token work, not pure kernels. generation_tps=count/time after that boundary through final detokenization, count 128. Fresh KV each call: no prompt_cache passed. No draft/KV-quant/activation-quant flags; flash selection and build/source commit unknown. Native rows: input/output; PP/TG median[min,max] tokens/s; peak decimal GB; n trials: 512/128: PP=1128.726[1125.249,1129.814]; TG=15.797[15.773,15.808]; peak=62.369GB; n=5 4096/128: PP=1467.158[1466.061,1468.062]; TG=15.413[15.409,15.418]; peak=65.722GB; n=5 16384/128: PP=1381.849[1381.649,1383.683]; TG=14.784[14.747,14.791]; peak=68.424GB; n=5 32768/128: PP=1236.992[1235.416,1237.321]; TG=13.91[13.821,13.917]; peak=71.978GB; n=5 Peak=max printed MLX peak_memory, mx.get_peak_memory()/1e9 allocator high-water, not RSS/disk/RAM/VRAM. No trial reset shown; warm-up/earlier allocations may contribute. Load-time/energy unknown. HTTP: serial LAN streaming from separate controller; mlx_lm.server prompt-cache-size 0, thinking disabled, temperature 0, cap 128. Repeated sample.txt essay+summary instruction, unique run-ID prefix. One discarded warm-up+five measured requests/cell, 30-second rests; 100K has one measured run. Raw: cachedTokens 0, output 128, firstWordField=content. Uncached input, warm-loaded model; not cold loading. Controller monotonic timing starts before request; first nonempty text ends TTFT. Client read=input/TTFT; write=(output−1)/(last−first text time); total ends at stream completion. Network/HTTP included; not native PP/TG. Code: https://github.com/theNetworkChuck/mac-studio-m5-ultra-local-ai/blob/main/harness/src/lane.ts . HTTP rows: nominal; actual in/out; read/write tokens/s; first/total seconds; median[min,max], n. Shown to 10 significant digits; raw retains full precision. 512: in=509[508,510]; out=128[128,128]; read=727.3992766[726.0931339,727.7645023]; write=15.37374854[15.36803727,15.41293747]; first=0.701012[0.698028,0.701734]; total=8.961933[8.937977,8.964848]; n=5 4K: in=4097[4096,4098]; out=128[128,128]; read=1332.060865[1293.979757,1335.664659]; write=15.08621355[15.06871631,15.1194364]; first=3.076436[3.066638,3.166201]; total=11.492559[11.466502,11.594359]; n=5 16K: in=16259[16258,16259]; out=128[128,128]; read=1335.717216[1334.615825,1335.943596]; write=14.47164975[14.46916012,14.47692367]; first=12.171738[12.170424,12.182532]; total=20.949115[20.943445,20.956574]; n=5 32K: in=32480[32480,32482]; out=128[128,128]; read=1216.509244[1214.977339,1216.804155]; write=13.58385964[13.57196771,13.59763423]; first=26.699345[26.692874,26.734655]; total=36.04754[36.039337,36.081001]; n=5 Current HF card names upstream google/gemma-4-31B-it revision3548789868c5356dbf307c98e6f609007b82b3eb; this is not the measured MLX snapshot revision. No independent model-quality validation.

generation:

130.4 tokens/s

prompt processing (prefill):

7154.6 tokens/s

quant:

Q4 (MLX)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

NetworkChuck, measured 2026-09-20; recovered 2026-10-13, not locally rerun. Measurements/docs: CC BY 4.0. Native data: https://github.com/theNetworkChuck/mac-studio-m5-ultra-local-ai/blob/main/data/language-models/engine.json ; HTTP summary/raw: https://github.com/theNetworkChuck/mac-studio-m5-ultra-local-ai/blob/main/data/language-models/summary.json ; https://github.com/theNetworkChuck/mac-studio-m5-ultra-local-ai/blob/main/data/language-models/runs.jsonl . Method: https://github.com/theNetworkChuck/mac-studio-m5-ultra-local-ai/blob/main/docs/method.md . Merged suite window 14:41:54.718–16:55:25.045 UTC; native cells have no individual timestamps. Native 32768-input/128-output headline: PP=7154.599, TG=130.372 tokens/s, separate five-trial medians, not pooled across lengths. Input is exactly 32768 synthetic tokens. Configured KV capacity unknown and blank. Batch 1; one device. Hardware: Apple Mac Studio M5 Ultra (36-core CPU, 80-core GPU); 256GB installed unified memory; macOS27.0; applegpu_g17d; CPU composition 12 super+24 performance; 8TB storage. Unified RAM and allocator peaks are not dedicated VRAM. M5 was an Apple review loan. Comparator M3 used macOS26.5.1 and had a stuck indexer consuming one CPU core. OS/background effects were not isolated. Power, thermals, clocks, swap and offload count unknown. Model: mlx-community/Qwen3.6-35B-A3B-4bit; 4-bit; MLX Safetensors. Public identity/precision verified 2026-10-13: https://huggingface.co/mlx-community/Qwen3.6-35B-A3B-4bit . Source labels moe, nominal total/active 35/3B; rounded labels, not exact architecture counts. Tested HF revision/hash and group/mixed-precision details unknown; weights not hash-checked. Current HF does not identify the historical snapshot/runtime. Runtime: mlx-lm0.31.3/MLX0.32.2; fingerprint 0.31.3-0.32.2-macOS-27.0-arm64-arm-64bit-applegpu_g17d. Harness command: mlx_lm.benchmark --model <folder> -p <input> -g 128 -n 5. Command/parser: https://github.com/theNetworkChuck/mac-studio-m5-ultra-local-ai/blob/main/harness/src/bench/ssh.ts ; https://github.com/theNetworkChuck/mac-studio-m5-ultra-local-ai/blob/main/harness/src/bench/engine.ts . Model already loaded; one discarded warm-up, five trials. Rates printed to 3 decimals; harness uses their median/min/max, not tool means. Versioned runtime semantics: https://github.com/ml-explore/mlx-lm/blob/v0.31.3/mlx_lm/benchmark.py and https://github.com/ml-explore/mlx-lm/blob/v0.31.3/mlx_lm/generate.py . Seed 0, random IDs rather than HTTP essay, EOS disabled, batch 1, default prefill step 2048, no trial delay. Native prompt_tps=prompt.size/time through first generated token; includes first-token work, not pure kernels. generation_tps=count/time after that boundary through final detokenization, count 128. Fresh KV each call: no prompt_cache passed. No draft/KV-quant/activation-quant flags; flash selection and build/source commit unknown. Native rows: input/output; PP/TG median[min,max] tokens/s; peak decimal GB; n trials: 512/128: PP=4513.449[4473.286,4522.985]; TG=153.368[152.664,153.712]; peak=20.178GB; n=5 4096/128: PP=8799.022[8753.889,8804.838]; TG=150.757[150.312,150.997]; peak=21.493GB; n=5 16384/128: PP=8155.308[8137.31,8159.113]; TG=138.215[137.678,138.355]; peak=21.744GB; n=5 32768/128: PP=7154.599[7148.451,7161.732]; TG=130.372[129.996,130.819]; peak=22.079GB; n=5 Peak=max printed MLX peak_memory, mx.get_peak_memory()/1e9 allocator high-water, not RSS/disk/RAM/VRAM. No trial reset shown; warm-up/earlier allocations may contribute. Load-time/energy unknown. HTTP: serial LAN streaming from separate controller; mlx_lm.server prompt-cache-size 0, thinking disabled, temperature 0, cap 128. Repeated sample.txt essay+summary instruction, unique run-ID prefix. One discarded warm-up+five measured requests/cell, 30-second rests; 100K has one measured run. Raw: cachedTokens 0, output 128, firstWordField=content. Uncached input, warm-loaded model; not cold loading. Controller monotonic timing starts before request; first nonempty text ends TTFT. Client read=input/TTFT; write=(output−1)/(last−first text time); total ends at stream completion. Network/HTTP included; not native PP/TG. Code: https://github.com/theNetworkChuck/mac-studio-m5-ultra-local-ai/blob/main/harness/src/lane.ts . HTTP rows: nominal; actual in/out; read/write tokens/s; first/total seconds; median[min,max], n. Shown to 10 significant digits; raw retains full precision. 512: in=511[510,512]; out=128[128,128]; read=2237.988876[1882.034656,2263.866738]; write=125.9938134[125.9231056,126.4199851]; first=0.22833[0.22572,0.272046]; total=1.236513[1.23457,1.279022]; n=5 4K: in=4096[4096,4098]; out=128[128,128]; read=6640.327996[6538.700876,6674.488333]; write=123.358698[123.2088633,123.4919638]; first=0.616837[0.61368,0.62673]; total=1.645587[1.642196,1.657597]; n=5 16K: in=16262[16261,16262]; out=128[128,128]; read=6980.699409[6977.065488,6989.111907]; write=112.0657782[110.8986282,112.308578]; first=2.329566[2.326762,2.330636]; total=3.463995[3.459357,3.474895]; n=5 32K: in=32493[32492,32495]; out=128[128,128]; read=6011.956933[6003.173684,6017.592193]; write=109.1465684[109.0974378,109.33205]; first=5.404563[5.399668,5.412637]; total=6.568463[6.56141,6.576532]; n=5 No independent model-quality validation.

generation:

74.7 tokens/s

prompt processing (prefill):

2710.5 tokens/s

quant:

Q4 (MLX)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

NetworkChuck, measured 2026-09-20; recovered 2026-10-13, not locally rerun. Measurements/docs: CC BY 4.0. Native data: https://github.com/theNetworkChuck/mac-studio-m5-ultra-local-ai/blob/main/data/language-models/engine.json ; HTTP summary/raw: https://github.com/theNetworkChuck/mac-studio-m5-ultra-local-ai/blob/main/data/language-models/summary.json ; https://github.com/theNetworkChuck/mac-studio-m5-ultra-local-ai/blob/main/data/language-models/runs.jsonl . Method: https://github.com/theNetworkChuck/mac-studio-m5-ultra-local-ai/blob/main/docs/method.md . Merged suite window 14:41:54.718–16:55:25.045 UTC; native cells have no individual timestamps. Native 32768-input/128-output headline: PP=2710.513, TG=74.694 tokens/s, separate five-trial medians, not pooled across lengths. Input is exactly 32768 synthetic tokens. Configured KV capacity unknown and blank. Batch 1; one device. Hardware: Apple Mac Studio M5 Ultra (36-core CPU, 80-core GPU); 256GB installed unified memory; macOS27.0; applegpu_g17d; CPU composition 12 super+24 performance; 8TB storage. Unified RAM and allocator peaks are not dedicated VRAM. M5 was an Apple review loan. Comparator M3 used macOS26.5.1 and had a stuck indexer consuming one CPU core. OS/background effects were not isolated. Power, thermals, clocks, swap and offload count unknown. Model: mlx-community/Qwen3.5-122B-A10B-4bit; 4-bit; MLX Safetensors. Public identity/precision verified 2026-10-13: https://huggingface.co/mlx-community/Qwen3.5-122B-A10B-4bit . Source labels moe, nominal total/active 122/10B; rounded labels, not exact architecture counts. Tested HF revision/hash and group/mixed-precision details unknown; weights not hash-checked. Current HF does not identify the historical snapshot/runtime. Runtime: mlx-lm0.31.3/MLX0.32.2; fingerprint 0.31.3-0.32.2-macOS-27.0-arm64-arm-64bit-applegpu_g17d. Harness command: mlx_lm.benchmark --model <folder> -p <input> -g 128 -n 5. Command/parser: https://github.com/theNetworkChuck/mac-studio-m5-ultra-local-ai/blob/main/harness/src/bench/ssh.ts ; https://github.com/theNetworkChuck/mac-studio-m5-ultra-local-ai/blob/main/harness/src/bench/engine.ts . Model already loaded; one discarded warm-up, five trials. Rates printed to 3 decimals; harness uses their median/min/max, not tool means. Versioned runtime semantics: https://github.com/ml-explore/mlx-lm/blob/v0.31.3/mlx_lm/benchmark.py and https://github.com/ml-explore/mlx-lm/blob/v0.31.3/mlx_lm/generate.py . Seed 0, random IDs rather than HTTP essay, EOS disabled, batch 1, default prefill step 2048, no trial delay. Native prompt_tps=prompt.size/time through first generated token; includes first-token work, not pure kernels. generation_tps=count/time after that boundary through final detokenization, count 128. Fresh KV each call: no prompt_cache passed. No draft/KV-quant/activation-quant flags; flash selection and build/source commit unknown. Native rows: input/output; PP/TG median[min,max] tokens/s; peak decimal GB; n trials: 512/128: PP=1870.512[1865.216,1873.479]; TG=87.709[87.688,87.905]; peak=69.846GB; n=5 4096/128: PP=3200.297[3193.664,3203.373]; TG=85.742[85.451,85.787]; peak=71.554GB; n=5 16384/128: PP=3010.206[3008.737,3011.547]; TG=80.1[79.851,80.134]; peak=71.856GB; n=5 32768/128: PP=2710.513[2709.622,2711.209]; TG=74.694[74.651,74.787]; peak=72.259GB; n=5 Peak=max printed MLX peak_memory, mx.get_peak_memory()/1e9 allocator high-water, not RSS/disk/RAM/VRAM. No trial reset shown; warm-up/earlier allocations may contribute. Load-time/energy unknown. HTTP: serial LAN streaming from separate controller; mlx_lm.server prompt-cache-size 0, thinking disabled, temperature 0, cap 128. Repeated sample.txt essay+summary instruction, unique run-ID prefix. One discarded warm-up+five measured requests/cell, 30-second rests; 100K has one measured run. Raw: cachedTokens 0, output 128, firstWordField=content. Uncached input, warm-loaded model; not cold loading. Controller monotonic timing starts before request; first nonempty text ends TTFT. Client read=input/TTFT; write=(output−1)/(last−first text time); total ends at stream completion. Network/HTTP included; not native PP/TG. Code: https://github.com/theNetworkChuck/mac-studio-m5-ultra-local-ai/blob/main/harness/src/lane.ts . HTTP rows: nominal; actual in/out; read/write tokens/s; first/total seconds; median[min,max], n. Shown to 10 significant digits; raw retains full precision. 512: in=511[510,511]; out=128[128,128]; read=1256.096122[1224.29579,1257.790424]; write=77.53911604[76.90463099,77.73018079]; first=0.406622[0.406268,0.416566]; total=2.044738[2.042052,2.058322]; n=5 4K: in=4096[4095,4096]; out=128[128,128]; read=2739.515301[2729.305653,2741.341845]; write=75.92279071[75.83669672,75.95430302]; first=1.495155[1.493794,1.500748]; total=3.168134[3.166987,3.173735]; n=5 16K: in=16269[16268,16270]; out=128[128,128]; read=2650.536339[2643.423045,2655.044617]; write=68.87392222[68.5605576,69.23724004]; first=6.138003[6.127204,6.15452]; total=7.982103[7.968434,7.992452]; n=5 32K: in=32501[32500,32502]; out=128[128,128]; read=2300.210565[2297.96314,2302.14058]; write=65.26630193[65.11885473,65.32323686]; first=14.129576[14.117296,14.14383]; total=16.076793[16.067721,16.089006]; n=5 100K: in=99120[99120,99120]; out=128[128,128]; read=1329.419949[1329.419949,1329.419949]; write=51.20472212[51.20472212,51.20472212]; first=74.558833[74.558833,74.558833]; total=77.047687[77.047687,77.047687]; n=1 No independent model-quality validation.

generation:

67.0 tokens/s

prompt processing (prefill):

2529.8 tokens/s

quant:

Q4 (MLX)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

NetworkChuck campaign measured 2026-09-20, recovered 2026-10-13; historical source evidence, no new/local run. Merged window 2026-09-20T14:41:54.718Z–2026-09-20T16:55:25.045Z; native-cell timestamps unavailable. Credit NetworkChuck, CC BY 4.0 data/docs. Repository: https://github.com/theNetworkChuck/mac-studio-m5-ultra-local-ai/ . Native data: https://github.com/theNetworkChuck/mac-studio-m5-ultra-local-ai/blob/main/data/language-models/engine.json . Client summary: https://github.com/theNetworkChuck/mac-studio-m5-ultra-local-ai/blob/main/data/language-models/summary.json . Client requests: https://github.com/theNetworkChuck/mac-studio-m5-ultra-local-ai/blob/main/data/language-models/runs.jsonl . Method: https://github.com/theNetworkChuck/mac-studio-m5-ultra-local-ai/blob/main/docs/method.md . Headline: native 32768-input/128-generation cell, PP median 2529.82, TG median 66.962 tokens/s across five trials; not pooled-duration or client rates. Context 32768 is actual native input length, not supported maximum or configured KV capacity. Batch 1. Hardware: one Apple Mac Studio M5 Ultra (36-core CPU, 80-core GPU), CPU core composition 12 super+24 performance, 256 GB installed unified memory, macOS 27.0, GPU architecture applegpu_g17d. Both review units had 8 TB storage. Installed unified memory is not dedicated VRAM. This M5 Ultra was loaned by Apple for the review. The comparison M3 Ultra used macOS 26.5.1 and had an indexing process occupying one CPU core; neither OS nor background-load effects were isolated. Power, energy, thermals, clock rates, exact storage part, explicit GPU-offload count and swap state are not reported for these rows. Model: mlx-community/Qwen3-14B-4bit, 4-bit MLX Safetensors; public identity verified 2026-10-13: https://huggingface.co/mlx-community/Qwen3-14B-4bit . Source calls it dense, nominal 14B total/active, local folder ~/bench-models/Qwen3-14B-4bit. Source disk label 7.8 GB is not runtime memory. Tested HF revision/hash unrecorded; author says weights were not hash-verified. Current HF main, conversion version and download size do not identify the historical snapshot/runtime. Quant group size/mixed-precision exceptions unverified. Runtime: mlx-lm 0.31.3 and MLX 0.32.2; server fingerprint 0.31.3-0.32.2-macOS-27.0-arm64-arm-64bit-applegpu_g17d. Native harness command is mlx_lm.benchmark --model <local-model-folder> -p <prompt-count> -g 128 -n 5, verified in https://github.com/theNetworkChuck/mac-studio-m5-ultra-local-ai/blob/main/harness/src/bench/ssh.ts . Model loaded before timing; one discarded warm-up then five trials. Harness median/min/max uses three-decimal printed trials; raw output retained. Warm-loaded, excludes cold loading. Native method: versioned https://github.com/ml-explore/mlx-lm/blob/v0.31.3/mlx_lm/benchmark.py seeds RNG 0, uses random token IDs, disables EOS stopping and calls stream_generate; no HTTP essay/chat workload. Published command leaves batch 1, prefill step 2048 and delay 0 defaults. https://github.com/ml-explore/mlx-lm/blob/v0.31.3/mlx_lm/generate.py defines prompt_tps=prompt.size/runtime time to first generated token, including that boundary rather than pure matrix-kernel time. Generation timing restarts there; generation_tps=generated count/runtime generation-window time, including final detokenizer handling, final count 128. This differs from client (128−1)/first-to-last text. No prompt_cache passed, so generate_step creates fresh KV. No draft model, KV-quantization or activation-quantization flag supplied. Flash/kernel selection and runtime build commit unrecorded; hardware alone does not establish a toggle. All native rows, input/output; PP median [min–max]; TG median [min–max], tokens/s; source peak-memory GB; trial count: 512/128; PP 2885.105 [2879.185–2887.773]; TG 102.517 [102.496–102.535]; peak 8.826 GB; n=5 4096/128; PP 3542.723 [3365.751–3587.111]; TG 96.019 [95.364–96.031]; peak 9.701 GB; n=5 16384/128; PP 3073.941 [3041.921–3074.689]; TG 80.093 [80.066–80.115]; peak 11.602 GB; n=5 32768/128; PP 2529.82 [2529.664–2530.48]; TG 66.962 [66.938–66.972]; peak 14.332 GB; n=5 Memory semantics: source peakMemoryGb is the maximum of printed native peak_memory values across five trials. Upstream peak_memory is mx.get_peak_memory()/1e9, an MLX allocator high-water mark in decimal GB; it is not RSS, installed RAM, dedicated VRAM, or a separately reset per-trial allocation. Warm-up and earlier allocation in that benchmark process can contribute to the high-water mark. No model-load timing or energy number is supplied by these rows. Separate HTTP arm, same model/Mac: serial streaming /v1/chat/completions from LAN controller to mlx_lm.server; --prompt-cache-size 0 and --chat-template-args '{"enable_thinking":false}' , temperature 0, max_tokens 128. Prompt repeats prompts/sample.txt with unique run-ID prefix and summary instruction. One discarded warm-up, five measured requests/cell, 30 s between cells. Single 100K requests, if listed, are not five-repeat cells. Actual counts from usage. All checked raw requests in this group: cachedTokens 0, outputTokens 128, firstWordField content. HTTP cache evidence does not mean cold model loading. Controller monotonic timing begins before request; first text is first nonempty delta. Reading=input tokens/client first-text seconds, including LAN/HTTP. Writing=(output−1)/(last-text−first-text seconds). Total ends at stream completion. These differ from native metrics. Implementation: https://github.com/theNetworkChuck/mac-studio-m5-ultra-local-ai/blob/main/harness/src/lane.ts and https://github.com/theNetworkChuck/mac-studio-m5-ultra-local-ai/blob/main/harness/src/timing.ts . All HTTP rows, each metric shown as median [min–max]; nominal length; actual token counts; client reading/writing tokens/s; first-text and total seconds; repetitions: 512 (512 nominal); actual input 507 [506–509], output 128 [128–128]; TTFT-derived reading 1902.61750633 [1856.64104818–1909.08816516]; client writing 92.2522627228 [92.1225767047–92.3128810632]; first text 0.266475 [0.265048–0.274151] s; total 1.644405 [1.64122–1.650881] s; n=5 4K (4096 nominal); actual input 4100 [4098–4101], output 128 [128–128]; TTFT-derived reading 3170.20057838 [3160.40663539–3203.87090119]; client writing 86.3066072805 [86.3005665258–86.3724065325]; first text 1.293609 [1.279702–1.296985] s; total 2.76521 [2.751289–2.767449] s; n=5 16K (16384 nominal); actual input 16282 [16280–16283], output 128 [128–128]; TTFT-derived reading 2799.66424665 [2796.76769856–2800.17191081]; client writing 68.2793129273 [68.0394929076–68.5252661289]; first text 5.815585 [5.814643–5.821363] s; total 7.681837 [7.671321–7.683826] s; n=5 32K (32768 nominal); actual input 32527 [32526–32528], output 128 [128–128]; TTFT-derived reading 2277.80207751 [2276.97725949–2278.9217499]; client writing 55.420495372 [55.3967563672–55.5510842522]; first text 14.279994 [14.272978–14.285606] s; total 16.57192 [16.567309–16.57732] s; n=5 Independent M3/M5 invocations were merged with shared settings checked. Different OS, M3 indexing load and unverified weight snapshots confound silicon-only comparisons. No independent model-quality evaluation.

generation:

47.6 tokens/s

prompt processing (prefill):

1672.8 tokens/s

quant:

4bit (MLX)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8 27B at 47.597 t/s generation and 1672.768 t/s prompt processing on an M5 Ultra 256GB Mac Studio. Setup is native mlx_lm.benchmark (mlx-lm 0.31.3 on MLX 0.32.2) with 4-bit MLX weights, 32768 prompt tokens and 128 generated tokens over 5 trials, peak memory 20.143 GB. Figures are medians from a third-party review unit; generation ranged 47.558-47.610 t/s and prompt processing 1671.556-1673.415 t/s. Speed only, no quality evaluation, and no hardware speedup ratio is claimed.

generation:

108.0 tokens/s

prompt processing (prefill):

2057.0 tokens/s

quant:

oQ4e (MLX)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User shares a MacStories review of the M5 Ultra Mac Studio 256GB and a DeepSeek V4.1 Flash comparison against 2x DGX Spark. Qwen3.8-Flash-Next runs at 108 tok/s on a short prompt and 143 tok/s for code generation on the Mac in MLX oQ4e, versus 42.6 and 39.0 tok/s on 2x DGX Spark in NVFP4 TP2. Prefill is 2,057-2,887 tok/s with TTFT of 24.4 s at 64K, 50.0 s at 128K, and 104 s at 256K. Decode decays from 90.7 tok/s at 4K to 74.7 tok/s at 256K, and combined throughput rises from 66.2 tok/s at 1 stream to 81.5 tok/s at 3 streams. The comparison is not a controlled A/B: the Mac runs oQ4e on MLX/oMLX while the Sparks run NVFP4/FP8 on vLLM/SGLang.