Local model performance on your hardware

generation:

14.9 tokens/s

quant:

4-bit (MLX)

mtp (multi-token prediction):

off

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

SOURCE: https://github.com/abhiram304/mlx-dense-bench/blob/b06ff117ab8c2ffc5bf4581cf06e671742563761/results/matrix.jsonl ; configs: https://github.com/abhiram304/mlx-dense-bench/blob/b06ff117ab8c2ffc5bf4581cf06e671742563761/configs/matrix.json . First-party authors Abhi Ram Salammagari and Koushik Salammagari. Original explanation: https://www.abhiramsalammagari.com/writing/dense-70b-mlx-m3-ultra . Historical recorded runs on 2026-05-03, recovered without rerunning benchmarks. HARDWARE/RUNTIME: One Mac Studio, Apple M3 Ultra, source-labelled 512 GB unified memory. Raw environment total_ram_gb=549.755813888 decimal GB (=512 GiB); this is installed capacity, not dedicated GPU VRAM or used memory. CPU/GPU core counts and chip bin were not published. macOS 26.2 arm64, Python 3.12.13, MLX 0.31.1, mlx-lm 0.31.2, memory API mx; raw mlx_lm_git_sha=null. Dependency lock also pins mlx-metal 0.31.1. One physical participating Metal GPU is implementation-supported, not a per-run device listing: exact MLX v0.31.1 commit ce45c52505c8158ea48d2a54e8caae05efd86bfe selects CopyAllDevices()[0] (default-device fallback) and returns one static Metal Device. mlx-lm v0.31.2 dcbf6e33d135a1b7c6767ca0fe7ebbd23df814a7 creates its generation stream on mx.default_device(); MLX defaults to GPU when available. This is a local non-distributed harness with no device override. Core quantity is distinct from GPU quantity. PROTOCOL: One priming generation followed by one timed generation per model/prompt/target-length cell, temperature 0, seed 42, maximum output 128. Lorem is a Moby-Dick summarization prompt; niah is single-needle retrieval. Chat-template-aware inputs have actual counts exceeding nominal targets. Actual outputs below are source generation_tokens counts, including terminal EOS in stopped rows; max_tokens=128 is not assumed achieved. No speculation, request batching or network server is used. No context-capacity claim is made from the maximum tested prompt. TIMING: Custom bench.py starts perf_counter after clear_cache/reset_peak_memory, before stream_generate. There is no explicit pre-start mx.synchronize. TTFT ends when the first GenerationResponse reaches the Python harness; it includes tokenizer/library setup, prompt work, first-token sampling and event delivery. This is measured local time to first library token event, not HTTP/client/display TTFT or model-load time. Both headline rows record first_text='The'. Some other token events may have empty detokenized text; first-EOS Llama rows deliver a terminal event. Load and warmup are outside the timer. Source prefill_time_s equals this TTFT interval and source prefill_tps=P/TTFT is a combined prompt-to-first-event rate, not isolated native PP; the numeric PP field is deliberately unset. Decode is exactly max(N-1,0)/(timer after generator exhaustion minus first-event timer), including final event/finalization overhead. It is not N/time, whole-request throughput, or a guessed output budget. All 24 raw rows pass these arithmetic identities. Library internal prompt_tps also includes the first generated-token boundary, so it is not isolated PP either. Library internal rates are separate cross-checks, not mixed into headline values. SELECTED HEADLINE: llama70b_lorem_4k, config_hash 9181be8c0a1f86de; actual prompt 4138, actual output count 128, timed decode numerator 127, TTFT 24.093307125149s, decode interval 8.512535624905s, TG 14.919173980129 t/s. Timestamp 2026-05-03T09:22:34.492652+00:00. No error/OOM/suspect_swap flag. This is one observation, not a repeated-run mean. FULL MODEL SWEEP (source values rounded here only; raw JSON linked above retains precision): lorem_4k: P=4138, N=128, timed decode numerator=127; TTFT=24.093307125s, decode interval=8.512535625s, TG=14.919173980 t/s. niah_4k: P=4157, N=14, timed decode numerator=13; TTFT=24.095836417s, decode interval=0.879073167s, TG=14.788302599 t/s [early EOS]. lorem_8k: P=8234, N=128, timed decode numerator=127; TTFT=49.453326292s, decode interval=9.064785625s, TG=14.010259619 t/s. niah_8k: P=8253, N=14, timed decode numerator=13; TTFT=49.469442542s, decode interval=0.935589458s, TG=13.894983415 t/s [early EOS]. lorem_16k: P=16426, N=128, timed decode numerator=127; TTFT=105.872540250s, decode interval=9.808121500s, TG=12.948452973 t/s. niah_16k: P=16445, N=14, timed decode numerator=13; TTFT=105.885152042s, decode interval=1.010276250s, TG=12.867767602 t/s [early EOS]. lorem_32k: P=32810, N=128, timed decode numerator=127; TTFT=241.514821250s, decode interval=11.329528791s, TG=11.209645374 t/s. niah_32k: P=32829, N=14, timed decode numerator=13; TTFT=241.538686208s, decode interval=1.166795584s, TG=11.141625985 t/s [early EOS]. lorem_64k: P=65578, N=128, timed decode numerator=127; TTFT=604.677038417s, decode interval=14.617659750s, TG=8.688121229 t/s. niah_64k: P=65597, N=128, timed decode numerator=127; TTFT=604.581685083s, decode interval=14.611410375s, TG=8.691837183 t/s. lorem_128k: P=131114, N=1, timed decode numerator=0; TTFT=1809.205743792s, decode interval=0.008948083s, TG=0.000000000 t/s [first EOS; decode invalid]. niah_128k: P=131133, N=1, timed decode numerator=0; TTFT=1806.923823916s, decode interval=0.009040625s, TG=0.000000000 t/s [first EOS; decode invalid]. MEMORY: Headline MLX allocation high-water mark 82.790731804 decimal GB and active-at-end 80.58895156 GB are process allocator measurements, not weight bytes or physical residency. The harness model cache retains previously loaded models; especially Llama's peak can include the earlier Qwen model. Do not enter these as isolated model residency or VRAM. Swap guard is heuristic, not a measured no-swap proof. ARTIFACT: Source config explicitly names mlx-community/Meta-Llama-3.1-70B-Instruct-4bit. Current immutable reference https://huggingface.co/mlx-community/Meta-Llama-3.1-70B-Instruct-4bit/tree/7772c93cf077b642f5503dd8d763a4176d7d406c; config has 4-bit, group_size=64 MLX quantization. Current eight safetensors shards total 39688567605 bytes on disk per HF metadata; hashes and sizes retained in audit manifest. No weight bytes downloaded. Historical tested revision/hash is unpublished, so the pinned snapshot is a same-publisher/model reference, not proof of historical byte identity. Source snapshot is also not a tested runtime/weight hash. CAVEATS: Full historical harness commit, exact installed runtime build hash, CPU thread count, GPU-core bin, actual allocated context capacity and historical model bytes remain unknown. Current harness was inspected at the pinned source snapshot; raw arithmetic confirms the stated timing convention. No new measurements were made. Code proof: https://github.com/ml-explore/mlx/blob/ce45c52505c8158ea48d2a54e8caae05efd86bfe/mlx/backend/metal/device.cpp ; https://github.com/ml-explore/mlx-lm/blob/dcbf6e33d135a1b7c6767ca0fe7ebbd23df814a7/mlx_lm/generate.py ; timer https://github.com/abhiram304/mlx-dense-bench/blob/b06ff117ab8c2ffc5bf4581cf06e671742563761/bench.py . Both 128K Llama inputs (131114/131133 actual tokens) exceed the current reference config limit 131072 and immediately emit EOS. Their source-reported first-event latency is retained only as an EOS-only diagnostic, not real first-output TTFT; zero decode throughput is invalid, not a usable TG result. LlamaPerf structured generation rate is rounded to 0.1 tokens/s. The measured local first-token-event latency and full timing definitions are retained above; the detailed form has no dedicated TTFT field.

generation:

14.3 tokens/s

quant:

4-bit (MLX)

mtp (multi-token prediction):

off

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

SOURCE: https://github.com/abhiram304/mlx-dense-bench/blob/b06ff117ab8c2ffc5bf4581cf06e671742563761/results/matrix.jsonl ; configs: https://github.com/abhiram304/mlx-dense-bench/blob/b06ff117ab8c2ffc5bf4581cf06e671742563761/configs/matrix.json . First-party authors Abhi Ram Salammagari and Koushik Salammagari. Original explanation: https://www.abhiramsalammagari.com/writing/dense-70b-mlx-m3-ultra . Historical recorded runs on 2026-05-03, recovered without rerunning benchmarks. HARDWARE/RUNTIME: One Mac Studio, Apple M3 Ultra, source-labelled 512 GB unified memory. Raw environment total_ram_gb=549.755813888 decimal GB (=512 GiB); this is installed capacity, not dedicated GPU VRAM or used memory. CPU/GPU core counts and chip bin were not published. macOS 26.2 arm64, Python 3.12.13, MLX 0.31.1, mlx-lm 0.31.2, memory API mx; raw mlx_lm_git_sha=null. Dependency lock also pins mlx-metal 0.31.1. One physical participating Metal GPU is implementation-supported, not a per-run device listing: exact MLX v0.31.1 commit ce45c52505c8158ea48d2a54e8caae05efd86bfe selects CopyAllDevices()[0] (default-device fallback) and returns one static Metal Device. mlx-lm v0.31.2 dcbf6e33d135a1b7c6767ca0fe7ebbd23df814a7 creates its generation stream on mx.default_device(); MLX defaults to GPU when available. This is a local non-distributed harness with no device override. Core quantity is distinct from GPU quantity. PROTOCOL: One priming generation followed by one timed generation per model/prompt/target-length cell, temperature 0, seed 42, maximum output 128. Lorem is a Moby-Dick summarization prompt; niah is single-needle retrieval. Chat-template-aware inputs have actual counts exceeding nominal targets. Actual outputs below are source generation_tokens counts, including terminal EOS in stopped rows; max_tokens=128 is not assumed achieved. No speculation, request batching or network server is used. No context-capacity claim is made from the maximum tested prompt. TIMING: Custom bench.py starts perf_counter after clear_cache/reset_peak_memory, before stream_generate. There is no explicit pre-start mx.synchronize. TTFT ends when the first GenerationResponse reaches the Python harness; it includes tokenizer/library setup, prompt work, first-token sampling and event delivery. This is measured local time to first library token event, not HTTP/client/display TTFT or model-load time. Both headline rows record first_text='The'. Some other token events may have empty detokenized text; first-EOS Llama rows deliver a terminal event. Load and warmup are outside the timer. Source prefill_time_s equals this TTFT interval and source prefill_tps=P/TTFT is a combined prompt-to-first-event rate, not isolated native PP; the numeric PP field is deliberately unset. Decode is exactly max(N-1,0)/(timer after generator exhaustion minus first-event timer), including final event/finalization overhead. It is not N/time, whole-request throughput, or a guessed output budget. All 24 raw rows pass these arithmetic identities. Library internal prompt_tps also includes the first generated-token boundary, so it is not isolated PP either. Library internal rates are separate cross-checks, not mixed into headline values. SELECTED HEADLINE: qwen72b_lorem_4k, config_hash 7d703b8e092c14de; actual prompt 4132, actual output count 128, timed decode numerator 127, TTFT 24.861239207909s, decode interval 8.861839125166s, TG 14.331110981167 t/s. Timestamp 2026-05-03T07:27:58.017497+00:00. No error/OOM/suspect_swap flag. This is one observation, not a repeated-run mean. FULL MODEL SWEEP (source values rounded here only; raw JSON linked above retains precision): lorem_4k: P=4132, N=128, timed decode numerator=127; TTFT=24.861239208s, decode interval=8.861839125s, TG=14.331110981 t/s. niah_4k: P=4154, N=95, timed decode numerator=94; TTFT=24.751058917s, decode interval=6.551007458s, TG=14.348938023 t/s [early EOS]. lorem_8k: P=8228, N=128, timed decode numerator=127; TTFT=50.718213750s, decode interval=9.398965250s, TG=13.512125710 t/s. niah_8k: P=8250, N=75, timed decode numerator=74; TTFT=50.722647250s, decode interval=5.488790917s, TG=13.482022019 t/s [early EOS]. lorem_16k: P=16420, N=128, timed decode numerator=127; TTFT=108.380474084s, decode interval=10.164079291s, TG=12.494983201 t/s. niah_16k: P=16442, N=56, timed decode numerator=55; TTFT=108.391879959s, decode interval=4.393770958s, TG=12.517721230 t/s [early EOS]. lorem_32k: P=32804, N=128, timed decode numerator=127; TTFT=248.996037833s, decode interval=11.657598750s, TG=10.894181788 t/s. niah_32k: P=32826, N=24, timed decode numerator=23; TTFT=246.696812208s, decode interval=2.121898834s, TG=10.839348056 t/s [early EOS]. lorem_64k: P=65572, N=128, timed decode numerator=127; TTFT=614.634077833s, decode interval=14.946048458s, TG=8.497229241 t/s. niah_64k: P=65594, N=66, timed decode numerator=65; TTFT=614.760152500s, decode interval=7.664965333s, TG=8.480142724 t/s [early EOS]. lorem_128k: P=131108, N=128, timed decode numerator=127; TTFT=1729.998797334s, decode interval=20.970838750s, TG=6.056028636 t/s. niah_128k: P=131130, N=99, timed decode numerator=98; TTFT=1732.439958042s, decode interval=16.178016917s, TG=6.057602764 t/s [early EOS]. MEMORY: Headline MLX allocation high-water mark 43.117035548 decimal GB and active-at-end 40.900624392 GB are process allocator measurements, not weight bytes or physical residency. The harness model cache retains previously loaded models; especially Llama's peak can include the earlier Qwen model. Do not enter these as isolated model residency or VRAM. Swap guard is heuristic, not a measured no-swap proof. ARTIFACT: Source config explicitly names mlx-community/Qwen2.5-72B-Instruct-4bit. Current immutable reference https://huggingface.co/mlx-community/Qwen2.5-72B-Instruct-4bit/tree/36a74b07390031bb18c9f12fb7c06699bc6273c4; config has 4-bit, group_size=64 MLX quantization. Current eight safetensors shards total 40900550929 bytes on disk per HF metadata; hashes and sizes retained in audit manifest. No weight bytes downloaded. Historical tested revision/hash is unpublished, so the pinned snapshot is a same-publisher/model reference, not proof of historical byte identity. Source snapshot is also not a tested runtime/weight hash. CAVEATS: Full historical harness commit, exact installed runtime build hash, CPU thread count, GPU-core bin, actual allocated context capacity and historical model bytes remain unknown. Current harness was inspected at the pinned source snapshot; raw arithmetic confirms the stated timing convention. No new measurements were made. Code proof: https://github.com/ml-explore/mlx/blob/ce45c52505c8158ea48d2a54e8caae05efd86bfe/mlx/backend/metal/device.cpp ; https://github.com/ml-explore/mlx-lm/blob/dcbf6e33d135a1b7c6767ca0fe7ebbd23df814a7/mlx_lm/generate.py ; timer https://github.com/abhiram304/mlx-dense-bench/blob/b06ff117ab8c2ffc5bf4581cf06e671742563761/bench.py . Current pinned Qwen config max_position_embeddings=32768, sliding_window=131072 with use_sliding_window=false; this does not support the blog’s unqualified native-128K claim. Raw 128K output counts are lorem=128 and niah=99; the prose swaps these labels, so raw rows take precedence. LlamaPerf structured generation rate is rounded to 0.1 tokens/s. The measured local first-token-event latency and full timing definitions are retained above; the detailed form has no dedicated TTFT field.

generation:

6.8 tokens/s

quant:

Q8 (MLX)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

NetworkChuck source-reported historical M3-only benchmark, 2026-09-19T17:03:53.712Z–17:51:07.654Z (raw timestamps), recovered2026-10-13, not locally rerun. Credit NetworkChuck; measurements/docs CC BY4.0. Primary summary: https://github.com/theNetworkChuck/mac-studio-m5-ultra-local-ai/blob/main/data/language-models/dropped.json . Original requests: https://github.com/theNetworkChuck/mac-studio-m5-ultra-local-ai/blob/main/data/language-models/baseline-m3-ultra/runs.jsonl . Method: https://github.com/theNetworkChuck/mac-studio-m5-ultra-local-ai/blob/main/docs/method.md . Source dropped Llama from the comparison on Sept20 as an older model, not due to failed measurements. M5 never ran it. CLIENT-ONLY headline: write=6.763430949 tokens/s median of five measured requests at nominal32K; actual prompt median32396 (range32396–32398), output128 each. promptTps deliberately blank: no native PP phase was measured for this model. The input/TTFT reading rate127.05078209328913 is retained only below as client-derived throughput. Actual median prompt count is32396, distinct from nominal32768. Configured context/KV capacity is unreported; capacity field is blank. Decode is measured from first-to-last streamed text, not native generation_tps, not total-request throughput and not cross-context maximum. Hardware: one Mac Studio M3 Ultra, 32-core CPU(24P+8E), 80-core GPU, 512GB unified memory, macOS26.5.1, applegpu_g15d, 8TB storage, Chuck's own unit. Unified RAM is not dedicated VRAM. A stuck indexer occupied one CPU core; no impact-adjustment rerun exists. Allocator peak, RSS, model-load time, power, thermals, clocks, swap and GPU-offload count are unknown for these Llama rows. Artifact: mlx-community/Llama-3.3-70B-Instruct-8bit, MLX Safetensors8-bit. Exact HF identity/precision verified2026-10-13 at https://huggingface.co/mlx-community/Llama-3.3-70B-Instruct-8bit . The card identifies a conversion from meta-llama/Llama-3.3-70B-Instruct using mlx-lm0.20.1; conversion runtime is not measurement runtime. Historical tested HF revision/weight hashes and exact group/mixed-precision details unrecorded; author did not hash-verify weights. Source nominal70B dense label is not an asserted exact architecture parameter count. Measurement runtime is mlx-lm0.31.3 with MLX0.31.2, NOT main-comparison MLX0.32.2. Each raw request fingerprint: 0.31.3-0.31.2-macOS-26.5.1-arm64-arm-64bit-Mach-O-applegpu_g15d. Earlier baseline manifest is dated Sept20 and lists the later lineup, so it must not be used to claim Llama ran then. dropped.json and raw Llama timestamps are the appropriate campaign evidence. Workload: serial streaming HTTP chat requests to mlx_lm.server over LAN from a separate controller; prompts/sample.txt repeated to nominal length after unique run-ID prefix and summary instruction. Source settings: temperature0, thinking disabled, output cap128; one discarded warm-up then five measured requests for each of512/4K/16K/32K. Raw check finds24 requests,4 warmups,20 measured, all cachedTokens0/outputTokens128/firstWordField=content, no errors or invalid flags. This confirms uncached input and warm-up protocol, not cold model-load timings. Exact historical server launch flags and intercell rest are not independently logged for this older Llama session, so do not silently copy the later manifest's30-second rest into it. Runtime build/source revision, explicit flash-attention, KV quantization and speculative settings are not established. Client clock is monotonic in controller, starts immediately before writing request. First text is first nonempty text delta, excluding empty opening frames. read=usage.prompt_tokens/firstWordSec; write=(usage.completion_tokens−1)/(lastWordSec−firstWordSec); total through stream completion. Network/HTTP overhead remains. These are not native PP/TG. Published timing implementation: https://github.com/theNetworkChuck/mac-studio-m5-ultra-local-ai/blob/main/harness/src/lane.ts . Summary medians/ranges were independently recomputed from raw rows and match dropped.json; this model had no cells.json entry and no native engine result. All rows: nominal; actual in/out; client read/write tokens/s; first/total seconds; values median[min,max], n measured. Rates/times shown to10 significant digits; full precision in raw/source packet: 512: in=510[509,510]; out=128[128,128]; read=154.0893621[153.4980397,154.2952624]; write=8.724754202[8.722782087,8.737563764]; first=3.305909[3.303278,3.322518]; total=17.863348[17.83836,17.87311]; n=5 4K: in=4095[4093,4095]; out=128[128,128]; read=162.6170391[162.49048,162.6614415]; write=8.337744569[8.329403101,8.339250152]; first=25.181125[25.174989,25.189168]; total=40.418484[40.40705,40.423623]; n=5 16K: in=16223[16222,16224]; out=128[128,128]; read=146.2999658[144.8342954,146.3908233]; write=7.331800698[7.329965859,7.332916608]; first=110.885968[110.826619,112.003859]; total=128.207912[128.153024,129.323906]; n=5 32K: in=32396[32396,32398]; out=128[128,128]; read=127.0507821[127.0372148,127.0934549]; write=6.763430949[6.757327657,6.768535795]; first=255.000398[254.906911,255.011888]; total=273.764986[273.678006,273.812481]; n=5 The 32K first-text median is255.000398s and full-request median273.764986s. Neither includes a separately measured model-load phase. No independent quality claim follows from these throughput results.

generation:

9.2 tokens/s

prompt processing (prefill):

295.9 tokens/s

quant:

BF16 (MLX)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

NetworkChuck, measured 2026-09-20; recovered 2026-10-13, not locally rerun. Measurements/docs: CC BY 4.0. Native data: https://github.com/theNetworkChuck/mac-studio-m5-ultra-local-ai/blob/main/data/language-models/engine.json ; HTTP summary/raw: https://github.com/theNetworkChuck/mac-studio-m5-ultra-local-ai/blob/main/data/language-models/summary.json ; https://github.com/theNetworkChuck/mac-studio-m5-ultra-local-ai/blob/main/data/language-models/runs.jsonl . Method: https://github.com/theNetworkChuck/mac-studio-m5-ultra-local-ai/blob/main/docs/method.md . Merged suite window 14:41:54.718–16:55:25.045 UTC; native cells have no individual timestamps. Native 32768-input/128-output headline: PP=295.894, TG=9.212 tokens/s, separate five-trial medians, not pooled across lengths. Input is exactly 32768 synthetic tokens. Configured KV capacity unknown and blank. Batch 1; one device. Hardware: Apple Mac Studio M3 Ultra (32-core CPU, 80-core GPU); 512GB installed unified memory; macOS26.5.1; applegpu_g15d; CPU composition 24 performance+8 efficiency; 8TB storage. Unified RAM and allocator peaks are not dedicated VRAM. M3 was Chuck's own; a stuck indexer consumed one CPU core during these tests. Effect unmeasured, no corrective rerun. OS/background effects were not isolated. Power, thermals, clocks, swap and offload count unknown. Model: mlx-community/gemma-4-31b-it-bf16; 16-bit (bf16); MLX Safetensors. Public identity/precision verified 2026-10-13: https://huggingface.co/mlx-community/gemma-4-31b-it-bf16 . Source labels dense, nominal total/active 31/31B; rounded labels, not exact architecture counts. Tested HF revision/hash and group/mixed-precision details unknown; weights not hash-checked. Current HF does not identify the historical snapshot/runtime. Runtime: mlx-lm0.31.3/MLX0.32.2; fingerprint 0.31.3-0.32.2-macOS-26.5.1-arm64-arm-64bit-applegpu_g15d. Harness command: mlx_lm.benchmark --model <folder> -p <input> -g 128 -n 5. Command/parser: https://github.com/theNetworkChuck/mac-studio-m5-ultra-local-ai/blob/main/harness/src/bench/ssh.ts ; https://github.com/theNetworkChuck/mac-studio-m5-ultra-local-ai/blob/main/harness/src/bench/engine.ts . Model already loaded; one discarded warm-up, five trials. Rates printed to 3 decimals; harness uses their median/min/max, not tool means. Versioned runtime semantics: https://github.com/ml-explore/mlx-lm/blob/v0.31.3/mlx_lm/benchmark.py and https://github.com/ml-explore/mlx-lm/blob/v0.31.3/mlx_lm/generate.py . Seed 0, random IDs rather than HTTP essay, EOS disabled, batch 1, default prefill step 2048, no trial delay. Native prompt_tps=prompt.size/time through first generated token; includes first-token work, not pure kernels. generation_tps=count/time after that boundary through final detokenization, count 128. Fresh KV each call: no prompt_cache passed. No draft/KV-quant/activation-quant flags; flash selection and build/source commit unknown. Native rows: input/output; PP/TG median[min,max] tokens/s; peak decimal GB; n trials: 512/128: PP=308.02[307.627,308.462]; TG=10.3[10.297,10.3]; peak=62.193GB; n=5 4096/128: PP=340.678[340.53,340.983]; TG=10.095[10.091,10.098]; peak=65.364GB; n=5 16384/128: PP=321.324[321.243,321.448]; TG=9.702[9.701,9.703]; peak=68.079GB; n=5 32768/128: PP=295.894[295.434,296.123]; TG=9.212[9.209,9.216]; peak=71.714GB; n=5 Peak=max printed MLX peak_memory, mx.get_peak_memory()/1e9 allocator high-water, not RSS/disk/RAM/VRAM. No trial reset shown; warm-up/earlier allocations may contribute. Load-time/energy unknown. HTTP: serial LAN streaming from separate controller; mlx_lm.server prompt-cache-size 0, thinking disabled, temperature 0, cap 128. Repeated sample.txt essay+summary instruction, unique run-ID prefix. One discarded warm-up+five measured requests/cell, 30-second rests; 100K has one measured run. Raw: cachedTokens 0, output 128, firstWordField=content. Uncached input, warm-loaded model; not cold loading. Controller monotonic timing starts before request; first nonempty text ends TTFT. Client read=input/TTFT; write=(output−1)/(last−first text time); total ends at stream completion. Network/HTTP included; not native PP/TG. Code: https://github.com/theNetworkChuck/mac-studio-m5-ultra-local-ai/blob/main/harness/src/lane.ts . HTTP rows: nominal; actual in/out; read/write tokens/s; first/total seconds; median[min,max], n. Shown to 10 significant digits; raw retains full precision. 512: in=512[512,513]; out=128[128,128]; read=253.6292968[253.47664,253.7269868]; write=9.950550465[9.943366655,9.950901313]; first=2.019176[2.017917,2.022637]; total=14.783122[14.780995,14.791931]; n=5 4K: in=4097[4096,4099]; out=128[128,128]; read=329.8423873[329.5933841,330.3287307]; write=9.815066374[9.787594555,9.817879115]; first=12.421084[12.408851,12.433502]; total=25.358448[25.350521,25.409205]; n=5 16K: in=16256[16254,16257]; out=128[128,128]; read=318.583264[318.5137296,318.6403799]; write=9.446346648[9.434043556,9.453159298]; first=51.025691[51.019899,51.040186]; total=64.466356[64.460458,64.501646]; n=5 32K: in=32479[32479,32479]; out=128[128,128]; read=294.2110642[294.0358735,294.5408545]; write=8.960484335[8.955359437,8.961596525]; first=110.393537[110.269932,110.459311]; total=124.567014[124.451506,124.632582]; n=5 Current HF card names upstream google/gemma-4-31B-it revision3548789868c5356dbf307c98e6f609007b82b3eb; this is not the measured MLX snapshot revision. Separate earlier M3 cohort: MLX 0.31.2 versus main 0.32.2; mlx-lm 0.31.3; timestamps 2026-09-20T12:40:26.693Z to 2026-09-20T13:04:39.847Z. Source: https://github.com/theNetworkChuck/mac-studio-m5-ultra-local-ai/blob/main/data/language-models/baseline-m3-ultra/runs.jsonl and https://github.com/theNetworkChuck/mac-studio-m5-ultra-local-ai/blob/main/data/language-models/baseline-m3-ultra/engine.json . Later Sept20 baseline manifest does not cover all earlier timestamps. Earlier results excluded from headline. HTTP cachedTokens 0/output 128; one warm-up+five repeats, except 100K one measured. Same formulas; medians/ranges recomputed from raw times/counts. 512: in=512[511,513]; out=128[128,128]; read=245.1500275[242.273689,245.5957379]; write=9.944198171[9.927911636,9.945870968]; first=2.088517[2.080655,2.11744]; total=14.863785[14.853517,14.887785]; n=5 4K: in=4097[4095,4098]; out=128[128,128]; read=328.0047178[327.5073216,328.347874]; write=9.803344145[9.797806449,9.808081334]; first=12.490674[12.47762,12.509644]; total=25.450789[25.42626,25.464178]; n=5 16K: in=16251[16251,16253]; out=128[128,128]; read=317.8314348[317.710641,317.9204948]; write=9.455167208[9.443309471,9.456347156]; first=51.130877[51.119699,51.150317]; total=64.562643[64.551698,64.599118]; n=5 32K: in=32470[32467,32471]; out=128[128,128]; read=293.8402701[293.705526,294.0009899]; write=8.981949746[8.977686138,8.984867714]; first=110.492003[110.441805,110.552908]; total=124.631643[124.584001,124.687886]; n=5 Earlier native MLX 0.31.2, same method, exact input/output; PP/TG median[min,max]; allocator peak GB; n: 512/128: PP=307.831[307.443,308.187]; TG=10.293[10.291,10.293]; peak=62.193GB; n=5 4096/128: PP=340.54[340.425,340.738]; TG=10.092[10.089,10.092]; peak=65.364GB; n=5 16384/128: PP=321.169[321.052,321.367]; TG=9.714[9.712,9.715]; peak=68.079GB; n=5 32768/128: PP=295.488[295.188,295.734]; TG=9.249[9.248,9.25]; peak=71.714GB; n=5 No independent model-quality validation.

generation:

31.8 tokens/s

prompt processing (prefill):

391.2 tokens/s

quant:

Q4 (MLX)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

NetworkChuck, measured 2026-09-20; recovered 2026-10-13, not locally rerun. Measurements/docs: CC BY 4.0. Native data: https://github.com/theNetworkChuck/mac-studio-m5-ultra-local-ai/blob/main/data/language-models/engine.json ; HTTP summary/raw: https://github.com/theNetworkChuck/mac-studio-m5-ultra-local-ai/blob/main/data/language-models/summary.json ; https://github.com/theNetworkChuck/mac-studio-m5-ultra-local-ai/blob/main/data/language-models/runs.jsonl . Method: https://github.com/theNetworkChuck/mac-studio-m5-ultra-local-ai/blob/main/docs/method.md . Merged suite window 14:41:54.718–16:55:25.045 UTC; native cells have no individual timestamps. Native 32768-input/128-output headline: PP=391.157, TG=31.768 tokens/s, separate five-trial medians, not pooled across lengths. Input is exactly 32768 synthetic tokens. Configured KV capacity unknown and blank. Batch 1; one device. Hardware: Apple Mac Studio M3 Ultra (32-core CPU, 80-core GPU); 512GB installed unified memory; macOS26.5.1; applegpu_g15d; CPU composition 24 performance+8 efficiency; 8TB storage. Unified RAM and allocator peaks are not dedicated VRAM. M3 was Chuck's own; a stuck indexer consumed one CPU core during these tests. Effect unmeasured, no corrective rerun. OS/background effects were not isolated. Power, thermals, clocks, swap and offload count unknown. Model: mlx-community/Qwen3.8-27B-4bit; 4-bit; MLX Safetensors. Public identity/precision verified 2026-10-13: https://huggingface.co/mlx-community/Qwen3.8-27B-4bit . Source labels dense, nominal total/active 27/27B; rounded labels, not exact architecture counts. Tested HF revision/hash and group/mixed-precision details unknown; weights not hash-checked. Current HF does not identify the historical snapshot/runtime. Runtime: mlx-lm0.31.3/MLX0.32.2; fingerprint 0.31.3-0.32.2-macOS-26.5.1-arm64-arm-64bit-applegpu_g15d. Harness command: mlx_lm.benchmark --model <folder> -p <input> -g 128 -n 5. Command/parser: https://github.com/theNetworkChuck/mac-studio-m5-ultra-local-ai/blob/main/harness/src/bench/ssh.ts ; https://github.com/theNetworkChuck/mac-studio-m5-ultra-local-ai/blob/main/harness/src/bench/engine.ts . Model already loaded; one discarded warm-up, five trials. Rates printed to 3 decimals; harness uses their median/min/max, not tool means. Versioned runtime semantics: https://github.com/ml-explore/mlx-lm/blob/v0.31.3/mlx_lm/benchmark.py and https://github.com/ml-explore/mlx-lm/blob/v0.31.3/mlx_lm/generate.py . Seed 0, random IDs rather than HTTP essay, EOS disabled, batch 1, default prefill step 2048, no trial delay. Native prompt_tps=prompt.size/time through first generated token; includes first-token work, not pure kernels. generation_tps=count/time after that boundary through final detokenization, count 128. Fresh KV each call: no prompt_cache passed. No draft/KV-quant/activation-quant flags; flash selection and build/source commit unknown. Native rows: input/output; PP/TG median[min,max] tokens/s; peak decimal GB; n trials: 512/128: PP=426.402[426.113,426.797]; TG=37.613[37.61,37.625]; peak=16.391GB; n=5 4096/128: PP=440.166[439.832,440.531]; TG=36.662[36.613,36.681]; peak=18.549GB; n=5 16384/128: PP=419.404[419.193,419.763]; TG=34.253[34.226,34.29]; peak=20.562GB; n=5 32768/128: PP=391.157[391.007,391.261]; TG=31.768[31.767,31.787]; peak=23.246GB; n=5 Peak=max printed MLX peak_memory, mx.get_peak_memory()/1e9 allocator high-water, not RSS/disk/RAM/VRAM. No trial reset shown; warm-up/earlier allocations may contribute. Load-time/energy unknown. HTTP: serial LAN streaming from separate controller; mlx_lm.server prompt-cache-size 0, thinking disabled, temperature 0, cap 128. Repeated sample.txt essay+summary instruction, unique run-ID prefix. One discarded warm-up+five measured requests/cell, 30-second rests; 100K has one measured run. Raw: cachedTokens 0, output 128, firstWordField=content. Uncached input, warm-loaded model; not cold loading. Controller monotonic timing starts before request; first nonempty text ends TTFT. Client read=input/TTFT; write=(output−1)/(last−first text time); total ends at stream completion. Network/HTTP included; not native PP/TG. Code: https://github.com/theNetworkChuck/mac-studio-m5-ultra-local-ai/blob/main/harness/src/lane.ts . HTTP rows: nominal; actual in/out; read/write tokens/s; first/total seconds; median[min,max], n. Shown to 10 significant digits; raw retains full precision. 512: in=510[509,511]; out=128[128,128]; read=365.5672888[364.9190224,366.5407252]; write=32.8371052[32.78281437,32.87826705]; first=1.395092[1.391387,1.399296]; total=5.26326[5.257062,5.27352]; n=5 4K: in=4095[4095,4096]; out=128[128,128]; read=428.5118271[428.4438146,428.6736283]; write=31.91895501[31.87186206,31.96037115]; first=9.556329[9.552722,9.558784]; total=13.535332[13.530005,13.543677]; n=5 16K: in=16263[16262,16264]; out=128[128,128]; read=414.3083238[414.2026107,414.3782211]; write=29.52265585[29.45467795,29.76088651]; first=39.253375[39.248068,39.260979]; total=43.560296[43.52102,43.562865]; n=5 32K: in=32493[32492,32493]; out=128[128,128]; read=388.4928822[388.4450068,388.667172]; write=27.79181958[27.72220603,27.82305414]; first=83.638598[83.601092,83.646332]; total=88.207674[88.170197,88.222767]; n=5 Separate earlier M3 cohort: MLX 0.31.2 versus main 0.32.2; mlx-lm 0.31.3; timestamps 2026-09-20T11:59:09.611Z to 2026-09-20T12:15:59.604Z. Source: https://github.com/theNetworkChuck/mac-studio-m5-ultra-local-ai/blob/main/data/language-models/baseline-m3-ultra/runs.jsonl and https://github.com/theNetworkChuck/mac-studio-m5-ultra-local-ai/blob/main/data/language-models/baseline-m3-ultra/engine.json . Later Sept20 baseline manifest does not cover all earlier timestamps. Earlier results excluded from headline. HTTP cachedTokens 0/output 128; one warm-up+five repeats, except 100K one measured. Same formulas; medians/ranges recomputed from raw times/counts. 512: in=511[509,511]; out=128[128,128]; read=357.2807356[354.9542866,358.1398315]; write=32.43838304[32.42378241,32.50086754]; first=1.428243[1.42465,1.439622]; total=5.341095[5.339991,5.356814]; n=5 4K: in=4096[4095,4097]; out=128[128,128]; read=416.5061266[416.1040813,416.8738735]; write=31.55226197[31.47394985,31.59191406]; first=9.835378[9.823115,9.843691]; total=13.858506[13.855747,13.869052]; n=5 16K: in=16269[16268,16269]; out=128[128,128]; read=403.654217[403.4194416,403.7293685]; write=29.24961158[29.18044313,29.46557794]; first=40.301821[40.296796,40.327754]; total=44.643474[44.612797,44.683133]; n=5 32K: in=32501[32500,32502]; out=128[128,128]; read=378.8949741[378.8573912,379.0772012]; write=27.46601567[27.29535682,27.52788933]; first=85.7798[85.739791,85.786897]; total=90.39477[90.383003,90.415871]; n=5 Earlier native MLX 0.31.2, same method, exact input/output; PP/TG median[min,max]; allocator peak GB; n: 512/128: PP=414.475[414.35,414.64]; TG=37.509[37.487,37.521]; peak=16.391GB; n=5 4096/128: PP=427.738[427.299,427.923]; TG=36.652[36.642,36.671]; peak=18.549GB; n=5 16384/128: PP=408.222[408.101,408.317]; TG=34.263[34.239,34.267]; peak=20.562GB; n=5 32768/128: PP=381.406[381.333,381.436]; TG=31.781[31.779,31.801]; peak=23.246GB; n=5 No independent model-quality validation.

generation:

89.3 tokens/s

prompt processing (prefill):

2258.1 tokens/s

quant:

Q4 (MLX)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

NetworkChuck, measured 2026-09-20; recovered 2026-10-13, not locally rerun. Measurements/docs: CC BY 4.0. Native data: https://github.com/theNetworkChuck/mac-studio-m5-ultra-local-ai/blob/main/data/language-models/engine.json ; HTTP summary/raw: https://github.com/theNetworkChuck/mac-studio-m5-ultra-local-ai/blob/main/data/language-models/summary.json ; https://github.com/theNetworkChuck/mac-studio-m5-ultra-local-ai/blob/main/data/language-models/runs.jsonl . Method: https://github.com/theNetworkChuck/mac-studio-m5-ultra-local-ai/blob/main/docs/method.md . Merged suite window 14:41:54.718–16:55:25.045 UTC; native cells have no individual timestamps. Native 32768-input/128-output headline: PP=2258.113, TG=89.266 tokens/s, separate five-trial medians, not pooled across lengths. Input is exactly 32768 synthetic tokens. Configured KV capacity unknown and blank. Batch 1; one device. Hardware: Apple Mac Studio M3 Ultra (32-core CPU, 80-core GPU); 512GB installed unified memory; macOS26.5.1; applegpu_g15d; CPU composition 24 performance+8 efficiency; 8TB storage. Unified RAM and allocator peaks are not dedicated VRAM. M3 was Chuck's own; a stuck indexer consumed one CPU core during these tests. Effect unmeasured, no corrective rerun. OS/background effects were not isolated. Power, thermals, clocks, swap and offload count unknown. Model: mlx-community/Qwen3.6-35B-A3B-4bit; 4-bit; MLX Safetensors. Public identity/precision verified 2026-10-13: https://huggingface.co/mlx-community/Qwen3.6-35B-A3B-4bit . Source labels moe, nominal total/active 35/3B; rounded labels, not exact architecture counts. Tested HF revision/hash and group/mixed-precision details unknown; weights not hash-checked. Current HF does not identify the historical snapshot/runtime. Runtime: mlx-lm0.31.3/MLX0.32.2; fingerprint 0.31.3-0.32.2-macOS-26.5.1-arm64-arm-64bit-applegpu_g15d. Harness command: mlx_lm.benchmark --model <folder> -p <input> -g 128 -n 5. Command/parser: https://github.com/theNetworkChuck/mac-studio-m5-ultra-local-ai/blob/main/harness/src/bench/ssh.ts ; https://github.com/theNetworkChuck/mac-studio-m5-ultra-local-ai/blob/main/harness/src/bench/engine.ts . Model already loaded; one discarded warm-up, five trials. Rates printed to 3 decimals; harness uses their median/min/max, not tool means. Versioned runtime semantics: https://github.com/ml-explore/mlx-lm/blob/v0.31.3/mlx_lm/benchmark.py and https://github.com/ml-explore/mlx-lm/blob/v0.31.3/mlx_lm/generate.py . Seed 0, random IDs rather than HTTP essay, EOS disabled, batch 1, default prefill step 2048, no trial delay. Native prompt_tps=prompt.size/time through first generated token; includes first-token work, not pure kernels. generation_tps=count/time after that boundary through final detokenization, count 128. Fresh KV each call: no prompt_cache passed. No draft/KV-quant/activation-quant flags; flash selection and build/source commit unknown. Native rows: input/output; PP/TG median[min,max] tokens/s; peak decimal GB; n trials: 512/128: PP=2296.962[2290.489,2309.915]; TG=106.413[106.313,106.699]; peak=20.178GB; n=5 4096/128: PP=3060.709[3054.646,3062.875]; TG=104.767[103.309,104.876]; peak=21.524GB; n=5 16384/128: PP=2686.749[2685.706,2687.926]; TG=94.701[94.503,95.185]; peak=22.564GB; n=5 32768/128: PP=2258.113[2256.12,2259.217]; TG=89.266[88.876,89.409]; peak=23.999GB; n=5 Peak=max printed MLX peak_memory, mx.get_peak_memory()/1e9 allocator high-water, not RSS/disk/RAM/VRAM. No trial reset shown; warm-up/earlier allocations may contribute. Load-time/energy unknown. HTTP: serial LAN streaming from separate controller; mlx_lm.server prompt-cache-size 0, thinking disabled, temperature 0, cap 128. Repeated sample.txt essay+summary instruction, unique run-ID prefix. One discarded warm-up+five measured requests/cell, 30-second rests; 100K has one measured run. Raw: cachedTokens 0, output 128, firstWordField=content. Uncached input, warm-loaded model; not cold loading. Controller monotonic timing starts before request; first nonempty text ends TTFT. Client read=input/TTFT; write=(output−1)/(last−first text time); total ends at stream completion. Network/HTTP included; not native PP/TG. Code: https://github.com/theNetworkChuck/mac-studio-m5-ultra-local-ai/blob/main/harness/src/lane.ts . HTTP rows: nominal; actual in/out; read/write tokens/s; first/total seconds; median[min,max], n. Shown to 10 significant digits; raw retains full precision. 512: in=510[510,512]; out=128[128,128]; read=1392.91362[1380.224895,1407.223564]; write=86.305141[85.79266682,86.34874012]; first=0.366139[0.363837,0.369505]; total=1.837072[1.83542,1.847129]; n=5 4K: in=4097[4096,4097]; out=128[128,128]; read=2685.95678[2682.464201,2691.548655]; write=84.74101278[84.66141766,84.96767883]; first=1.525341[1.522172,1.527327]; total=3.022171[3.01864,3.025994]; n=5 16K: in=16273[16272,16274]; out=128[128,128]; read=2559.706696[2557.059316,2562.225937]; write=77.2879359[77.07016844,77.85588569]; first=6.357369[6.350728,6.364342]; total=8.005671[7.982217,8.007679]; n=5 32K: in=32511[32510,32511]; out=128[128,128]; read=2192.970326[2191.257683,2194.364095]; write=71.49874314[69.93815705,71.76896947]; first=14.825098[14.815226,14.836685]; total=16.598931[16.591711,16.652469]; n=5 Separate earlier M3 cohort: MLX 0.31.2 versus main 0.32.2; mlx-lm 0.31.3; timestamps 2026-09-20T12:32:00.205Z to 2026-09-20T12:36:29.626Z. Source: https://github.com/theNetworkChuck/mac-studio-m5-ultra-local-ai/blob/main/data/language-models/baseline-m3-ultra/runs.jsonl and https://github.com/theNetworkChuck/mac-studio-m5-ultra-local-ai/blob/main/data/language-models/baseline-m3-ultra/engine.json . Later Sept20 baseline manifest does not cover all earlier timestamps. Earlier results excluded from headline. HTTP cachedTokens 0/output 128; one warm-up+five repeats, except 100K one measured. Same formulas; medians/ranges recomputed from raw times/counts. 512: in=511[510,511]; out=128[128,128]; read=1371.142153[1360.127762,1377.558998]; write=84.38263472[84.21951718,84.49002619]; first=0.372682[0.370899,0.3757]; total=1.878181[1.876894,1.880795]; n=5 4K: in=4095[4095,4098]; out=128[128,128]; read=2626.44286[2585.746381,2638.281216]; write=82.85312229[81.04244838,83.01131045]; first=1.559143[1.553284,1.583682]; total=3.094791[3.086665,3.126361]; n=5 16K: in=16268[16267,16269]; out=128[128,128]; read=2506.748434[2501.353766,2509.861714]; write=74.05295311[73.99694806,74.74778508]; first=6.489283[6.481632,6.504078]; total=8.205734[8.185321,8.219329]; n=5 32K: in=32501[32500,32502]; out=128[128,128]; read=2152.18143[2149.123274,2154.740764]; write=68.43015302[68.36462136,68.68039013]; first=15.101516[15.083485,15.122911]; total=16.962611[16.932815,16.985637]; n=5 Earlier native MLX 0.31.2, same method, exact input/output; PP/TG median[min,max]; allocator peak GB; n: 512/128: PP=2233.48[2232.533,2237.758]; TG=101.953[101.735,102.087]; peak=20.178GB; n=5 4096/128: PP=2978.571[2977.652,2981.538]; TG=100.248[98.96,100.331]; peak=21.524GB; n=5 16384/128: PP=2622.995[2622.075,2624.993]; TG=91.219[90.886,91.288]; peak=22.564GB; n=5 32768/128: PP=2214.475[2212.581,2215.649]; TG=85.625[85.385,86.23]; peak=23.999GB; n=5 No independent model-quality validation.

generation:

49.5 tokens/s

prompt processing (prefill):

784.5 tokens/s

quant:

Q4 (MLX)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

NetworkChuck, measured 2026-09-20; recovered 2026-10-13, not locally rerun. Measurements/docs: CC BY 4.0. Native data: https://github.com/theNetworkChuck/mac-studio-m5-ultra-local-ai/blob/main/data/language-models/engine.json ; HTTP summary/raw: https://github.com/theNetworkChuck/mac-studio-m5-ultra-local-ai/blob/main/data/language-models/summary.json ; https://github.com/theNetworkChuck/mac-studio-m5-ultra-local-ai/blob/main/data/language-models/runs.jsonl . Method: https://github.com/theNetworkChuck/mac-studio-m5-ultra-local-ai/blob/main/docs/method.md . Merged suite window 14:41:54.718–16:55:25.045 UTC; native cells have no individual timestamps. Native 32768-input/128-output headline: PP=784.528, TG=49.498 tokens/s, separate five-trial medians, not pooled across lengths. Input is exactly 32768 synthetic tokens. Configured KV capacity unknown and blank. Batch 1; one device. Hardware: Apple Mac Studio M3 Ultra (32-core CPU, 80-core GPU); 512GB installed unified memory; macOS26.5.1; applegpu_g15d; CPU composition 24 performance+8 efficiency; 8TB storage. Unified RAM and allocator peaks are not dedicated VRAM. M3 was Chuck's own; a stuck indexer consumed one CPU core during these tests. Effect unmeasured, no corrective rerun. OS/background effects were not isolated. Power, thermals, clocks, swap and offload count unknown. Model: mlx-community/Qwen3.5-122B-A10B-4bit; 4-bit; MLX Safetensors. Public identity/precision verified 2026-10-13: https://huggingface.co/mlx-community/Qwen3.5-122B-A10B-4bit . Source labels moe, nominal total/active 122/10B; rounded labels, not exact architecture counts. Tested HF revision/hash and group/mixed-precision details unknown; weights not hash-checked. Current HF does not identify the historical snapshot/runtime. Runtime: mlx-lm0.31.3/MLX0.32.2; fingerprint 0.31.3-0.32.2-macOS-26.5.1-arm64-arm-64bit-applegpu_g15d. Harness command: mlx_lm.benchmark --model <folder> -p <input> -g 128 -n 5. Command/parser: https://github.com/theNetworkChuck/mac-studio-m5-ultra-local-ai/blob/main/harness/src/bench/ssh.ts ; https://github.com/theNetworkChuck/mac-studio-m5-ultra-local-ai/blob/main/harness/src/bench/engine.ts . Model already loaded; one discarded warm-up, five trials. Rates printed to 3 decimals; harness uses their median/min/max, not tool means. Versioned runtime semantics: https://github.com/ml-explore/mlx-lm/blob/v0.31.3/mlx_lm/benchmark.py and https://github.com/ml-explore/mlx-lm/blob/v0.31.3/mlx_lm/generate.py . Seed 0, random IDs rather than HTTP essay, EOS disabled, batch 1, default prefill step 2048, no trial delay. Native prompt_tps=prompt.size/time through first generated token; includes first-token work, not pure kernels. generation_tps=count/time after that boundary through final detokenization, count 128. Fresh KV each call: no prompt_cache passed. No draft/KV-quant/activation-quant flags; flash selection and build/source commit unknown. Native rows: input/output; PP/TG median[min,max] tokens/s; peak decimal GB; n trials: 512/128: PP=813.225[811.699,814.645]; TG=58.625[58.222,58.744]; peak=69.846GB; n=5 4096/128: PP=1015.859[1015.543,1016.89]; TG=56.721[56.387,56.861]; peak=71.834GB; n=5 16384/128: PP=911.011[910.96,911.133]; TG=53.218[52.378,53.438]; peak=73.727GB; n=5 32768/128: PP=784.528[784.324,786.333]; TG=49.498[49.337,49.572]; peak=76.32GB; n=5 Peak=max printed MLX peak_memory, mx.get_peak_memory()/1e9 allocator high-water, not RSS/disk/RAM/VRAM. No trial reset shown; warm-up/earlier allocations may contribute. Load-time/energy unknown. HTTP: serial LAN streaming from separate controller; mlx_lm.server prompt-cache-size 0, thinking disabled, temperature 0, cap 128. Repeated sample.txt essay+summary instruction, unique run-ID prefix. One discarded warm-up+five measured requests/cell, 30-second rests; 100K has one measured run. Raw: cachedTokens 0, output 128, firstWordField=content. Uncached input, warm-loaded model; not cold loading. Controller monotonic timing starts before request; first nonempty text ends TTFT. Client read=input/TTFT; write=(output−1)/(last−first text time); total ends at stream completion. Network/HTTP included; not native PP/TG. Code: https://github.com/theNetworkChuck/mac-studio-m5-ultra-local-ai/blob/main/harness/src/lane.ts . HTTP rows: nominal; actual in/out; read/write tokens/s; first/total seconds; median[min,max], n. Shown to 10 significant digits; raw retains full precision. 512: in=511[510,511]; out=128[128,128]; read=632.0612122[631.3638324,632.6185571]; write=51.32250415[51.16119806,51.43680826]; first=0.807997[0.806173,0.809359]; total=3.283269[3.275353,3.290922]; n=5 4K: in=4096[4095,4099]; out=128[128,128]; read=958.0874749[952.5661145,958.441652]; write=49.721423[49.59467095,49.86798727]; first=4.276734[4.273744,4.299964]; total=6.831157[6.820577,6.860899]; n=5 16K: in=16275[16274,16275]; out=128[128,128]; read=886.9589511[885.5433457,887.0338855]; write=45.11657483[45.10539034,45.16531929]; first=18.348087[18.346537,18.378547]; total=21.16384[21.162137,21.192029]; n=5 32K: in=32514[32514,32515]; out=128[128,128]; read=770.7526728[769.9743079,771.3100191]; write=42.28536763[42.09665192,42.43907737]; first=42.186036[42.154256,42.227383]; total=45.199644[45.150373,45.234488]; n=5 100K: in=99152[99152,99152]; out=128[128,128]; read=448.5236327[448.5236327,448.5236327]; write=31.64594024[31.64594024,31.64594024]; first=221.063045[221.063045,221.063045]; total=225.088704[225.088704,225.088704]; n=1 Separate earlier M3 cohort: MLX 0.31.2 versus main 0.32.2; mlx-lm 0.31.3; timestamps 2026-09-19T16:37:36.675Z to 2026-09-19T16:55:00.521Z. Source: https://github.com/theNetworkChuck/mac-studio-m5-ultra-local-ai/blob/main/data/language-models/baseline-m3-ultra/runs.jsonl and https://github.com/theNetworkChuck/mac-studio-m5-ultra-local-ai/blob/main/data/language-models/baseline-m3-ultra/engine.json . Later Sept20 baseline manifest does not cover all earlier timestamps. Earlier results excluded from headline. HTTP cachedTokens 0/output 128; one warm-up+five repeats, except 100K one measured. Same formulas; medians/ranges recomputed from raw times/counts. 512: in=510[509,512]; out=128[128,128]; read=612.0733866[606.3718986,613.1237274]; write=50.03944054[49.9811095,50.13744371]; first=0.833437[0.832389,0.841068]; total=3.37435[3.36884,3.374909]; n=5 4K: in=4096[4095,4096]; out=128[128,128]; read=930.7486629[929.2834784,932.1454547]; write=48.54174112[48.44205396,48.5922407]; first=4.399924[4.394164,4.407697]; total=7.021613[7.010378,7.021709]; n=5 16K: in=16269[16268,16270]; out=128[128,128]; read=864.0536016[863.8583094,864.3269631]; write=43.75977057[43.67120056,43.88585393]; first=18.828693[18.823895,18.831792]; total=21.734329[21.723297,21.740319]; n=5 32K: in=32502[32501,32502]; out=128[128,128]; read=753.5246788[751.2126047,753.8611028]; write=41.6975294[41.32669763,41.81270935]; first=43.133292[43.114043,43.264716]; total=46.187097[46.151546,46.33805]; n=5 100K: in=99122[99122,99122]; out=128[128,128]; read=443.3709763[443.3709763,443.3709763]; write=31.46559039[31.46559039,31.46559039]; first=223.564476[223.564476,223.564476]; total=227.61373[227.61373,227.61373]; n=1 No independent model-quality validation.

generation:

35.8 tokens/s

prompt processing (prefill):

570.1 tokens/s

quant:

Q4 (MLX)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

NetworkChuck campaign measured 2026-09-20, recovered 2026-10-13; historical source evidence, no new/local run. Merged window 2026-09-20T14:41:54.718Z–2026-09-20T16:55:25.045Z; native-cell timestamps unavailable. Credit NetworkChuck, CC BY 4.0 data/docs. Repository: https://github.com/theNetworkChuck/mac-studio-m5-ultra-local-ai/ . Native data: https://github.com/theNetworkChuck/mac-studio-m5-ultra-local-ai/blob/main/data/language-models/engine.json . Client summary: https://github.com/theNetworkChuck/mac-studio-m5-ultra-local-ai/blob/main/data/language-models/summary.json . Client requests: https://github.com/theNetworkChuck/mac-studio-m5-ultra-local-ai/blob/main/data/language-models/runs.jsonl . Method: https://github.com/theNetworkChuck/mac-studio-m5-ultra-local-ai/blob/main/docs/method.md . Headline: native 32768-input/128-generation cell, PP median 570.075, TG median 35.801 tokens/s across five trials; not pooled-duration or client rates. Context 32768 is actual native input length, not supported maximum or configured KV capacity. Batch 1. Hardware: one Apple Mac Studio M3 Ultra (32-core CPU, 80-core GPU), CPU core composition 24 performance+8 efficiency, 512 GB installed unified memory, macOS 26.5.1, GPU architecture applegpu_g15d. Both review units had 8 TB storage. Installed unified memory is not dedicated VRAM. A stuck background indexing process occupied 100% of one of this M3 Ultra's 32 CPU cores throughout this campaign, until 2026-09-21 10:10 CT. The effect was not measured by rerunning; do not silently correct the rates. This was Chuck's own machine. Power, energy, thermals, clock rates, exact storage part, explicit GPU-offload count and swap state are not reported for these rows. Model: mlx-community/Qwen3-14B-4bit, 4-bit MLX Safetensors; public identity verified 2026-10-13: https://huggingface.co/mlx-community/Qwen3-14B-4bit . Source calls it dense, nominal 14B total/active, local folder ~/bench-models/Qwen3-14B-4bit. Source disk label 7.8 GB is not runtime memory. Tested HF revision/hash unrecorded; author says weights were not hash-verified. Current HF main, conversion version and download size do not identify the historical snapshot/runtime. Quant group size/mixed-precision exceptions unverified. Runtime: mlx-lm 0.31.3 and MLX 0.32.2; server fingerprint 0.31.3-0.32.2-macOS-26.5.1-arm64-arm-64bit-applegpu_g15d. Native harness command is mlx_lm.benchmark --model <local-model-folder> -p <prompt-count> -g 128 -n 5, verified in https://github.com/theNetworkChuck/mac-studio-m5-ultra-local-ai/blob/main/harness/src/bench/ssh.ts . Model loaded before timing; one discarded warm-up then five trials. Harness median/min/max uses three-decimal printed trials; raw output retained. Warm-loaded, excludes cold loading. Native method: versioned https://github.com/ml-explore/mlx-lm/blob/v0.31.3/mlx_lm/benchmark.py seeds RNG 0, uses random token IDs, disables EOS stopping and calls stream_generate; no HTTP essay/chat workload. Published command leaves batch 1, prefill step 2048 and delay 0 defaults. https://github.com/ml-explore/mlx-lm/blob/v0.31.3/mlx_lm/generate.py defines prompt_tps=prompt.size/runtime time to first generated token, including that boundary rather than pure matrix-kernel time. Generation timing restarts there; generation_tps=generated count/runtime generation-window time, including final detokenizer handling, final count 128. This differs from client (128−1)/first-to-last text. No prompt_cache passed, so generate_step creates fresh KV. No draft model, KV-quantization or activation-quantization flag supplied. Flash/kernel selection and runtime build commit unrecorded; hardware alone does not establish a toggle. All native rows, input/output; PP median [min–max]; TG median [min–max], tokens/s; source peak-memory GB; trial count: 512/128; PP 813.779 [812.584–814.936]; TG 69.365 [69.32–69.391]; peak 8.826 GB; n=5 4096/128; PP 814.254 [813.998–815.156]; TG 62.896 [62.85–62.918]; peak 9.701 GB; n=5 16384/128; PP 690.643 [690.481–691.238]; TG 47.455 [47.42–47.465]; peak 11.602 GB; n=5 32768/128; PP 570.075 [569.782–570.199]; TG 35.801 [35.795–35.812]; peak 14.332 GB; n=5 Memory semantics: source peakMemoryGb is the maximum of printed native peak_memory values across five trials. Upstream peak_memory is mx.get_peak_memory()/1e9, an MLX allocator high-water mark in decimal GB; it is not RSS, installed RAM, dedicated VRAM, or a separately reset per-trial allocation. Warm-up and earlier allocation in that benchmark process can contribute to the high-water mark. No model-load timing or energy number is supplied by these rows. Separate HTTP arm, same model/Mac: serial streaming /v1/chat/completions from LAN controller to mlx_lm.server; --prompt-cache-size 0 and --chat-template-args '{"enable_thinking":false}' , temperature 0, max_tokens 128. Prompt repeats prompts/sample.txt with unique run-ID prefix and summary instruction. One discarded warm-up, five measured requests/cell, 30 s between cells. Single 100K requests, if listed, are not five-repeat cells. Actual counts from usage. All checked raw requests in this group: cachedTokens 0, outputTokens 128, firstWordField content. HTTP cache evidence does not mean cold model loading. Controller monotonic timing begins before request; first text is first nonempty delta. Reading=input tokens/client first-text seconds, including LAN/HTTP. Writing=(output−1)/(last-text−first-text seconds). Total ends at stream completion. These differ from native metrics. Implementation: https://github.com/theNetworkChuck/mac-studio-m5-ultra-local-ai/blob/main/harness/src/lane.ts and https://github.com/theNetworkChuck/mac-studio-m5-ultra-local-ai/blob/main/harness/src/timing.ts . All HTTP rows, each metric shown as median [min–max]; nominal length; actual token counts; client reading/writing tokens/s; first-text and total seconds; repetitions: 512 (512 nominal); actual input 507 [507–508], output 128 [128–128]; TTFT-derived reading 681.304886619 [680.15894523–683.748234999]; client writing 61.7620769176 [61.5733692754–61.8337956262]; first text 0.744678 [0.741501–0.745628] s; total 2.801364 [2.797872–2.808342] s; n=5 4K (4096 nominal); actual input 4097 [4095–4097], output 128 [128–128]; TTFT-derived reading 776.541288513 [774.603443497–777.677154077]; client writing 55.7032446921 [55.5113601595–55.8309246173]; first text 5.275959 [5.266477–5.289158] s; total 7.555642 [7.54761–7.577066] s; n=5 16K (16384 nominal); actual input 16280 [16279–16281], output 128 [128–128]; TTFT-derived reading 669.146411504 [668.529108244–669.423496892]; client writing 41.7188837078 [41.4444331315–41.7777862576]; first text 24.330998 [24.320927–24.350473] s; total 27.382021 [27.368085–27.394455] s; n=5 32K (32768 nominal); actual input 32521 [32520–32521], output 128 [128–128]; TTFT-derived reading 550.950577802 [550.6959282–551.18283725]; client writing 31.5317764581 [31.4994611608–31.5856519565]; first text 59.025261 [59.0022–59.054368] s; total 63.05821 [63.031217–63.090287] s; n=5 Independent M3/M5 invocations were merged with shared settings checked. Different OS, M3 indexing load and unverified weight snapshots confound silicon-only comparisons. No independent model-quality evaluation.

generation:

62.6 tokens/s

prompt processing (prefill):

1077.3 tokens/s

quant:

MXFP4/BF16 MLX (MLX)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports gpt-oss 117B-A5.1B at 62.6 t/s generation and 1,077.3 t/s prompt processing on an M3 Ultra 512GB. Setup is MLX with MXFP4/BF16 MLX weights, 32,006 actual input tokens and 128 generated tokens per run, three runs, peak memory 67.2 GB. The figures are medians; prompt processing p10/p90 are 1,074.7 and 1,078.3 t/s, and the configured context cap is unknown.

generation:

37.9 tokens/s

quant:

MLX affine 4-bit, group 64, 4.501 effective bpw (MLX)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8 at 37.9 t/s on an M3 Ultra 512GB, from a single run of one 68-token prompt and 120 generated tokens. Setup is MLX with affine 4-bit quantization, group size 64 and 4.501 effective bpw, peaking at 15.5 GB of memory. The author describes the figure as a rough guide rather than a repeated controlled benchmark; no prompt-processing throughput or TTFT is given.

generation:

3.0 tokens/s

quant:

Q4 (MLX)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User benchmarks Llama 3.1 405B at 2.99 t/s on a single M3 Ultra 512GB node. Setup is MLX with Q4 quantisation at 1K context, single node. This is part of a 200+ configuration campaign across 5 models and 9 topologies on a 5-node M3 Ultra cluster; the report also covers Qwen 2.5 32B, Mixtral 8x7B, DeepSeek V3.2 and Kimi K2.5, with TP and PP scaling results.

generation:

45-47 tokens/s

prompt processing (prefill):

1012.0 tokens/s

quant:

4-bit (MLX)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports DeepSeek-V4-Flash-0731 at 4-bit served across two Mac Studio M3 Ultra (512GB) boxes: prefill 1012 tok/s on a 13.9K-token prompt (TTFT 13.7 s), decode 45-47 tok/s single-stream and 112-117 tok/s aggregate at batch 8. Setup is MLX with an oMLX 0.6.3rc2 serving overlay, MTP/speculative machinery enabled, over a single Thunderbolt 5 cable using the jaccl RDMA mesh backend. The decode figure is given only as a range, so no single generationTps is recorded. The repo is a reproducibility playbook; the oMLX 0.6.3rc2 build artifact is no longer available and must be obtained upstream.

generation:

29.9 tokens/s

prompt processing (prefill):

426.5 tokens/s

quant:

4bit (MLX)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports DeepSeek V4 Flash 4bit at 29.9 tok/s generation and 426.5 tok/s prompt processing on a Mac Studio with M3 Ultra 512GB unified memory. Setup is MLX 0.32.0 (draft PR #1189) with a standard 8,000-byte prompt, 300 generated tokens, temperature 0, and no speculative decoding. The same machine also ran DeepSeek R1-0528 4bit MLX at 206.9 / 20.3, GLM-5.2 4bit MLX at 182.6 / 17.9, Kimi K2.7 Code Q2_K_XL GGUF on llama.cpp 571d0d5 at 144.8 / 24.6, Qwen3.5 397B-A17B 4bit MLX at 510.3 / 38.1, gpt-oss-120B MXFP4 MLX at 1,405.9 / 79.2, Qwen3.6 35B-A3B 4bit MLX at 2,892.2 / 94.6, Gemma 4 31B IT 4bit at 347.8 / 29.2, Gemma 4 26B-A4B IT 4bit at 2,508.5 / 104.9, Qwen3-Coder-Next 4bit at 2,099.8 / 77.2, and Devstral 2 123B 4bit at 90.4 / 8.9 (prompt / generation tok/s).

generation:

2.0 tokens/s

quant:

MXFP4 (MXFP4)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Kimi-K3 (2.78T, MXFP4) at ~2.0 tok/s single-stream decode on four M3 Ultra 512GB Mac Studios in tensor parallel over a Thunderbolt 5 RDMA mesh. Setup is MLX 0.32.0 / mlx-lm 0.31.3 with MXFP4 weights, full 1,048,576-token context (29 GB KV), 420.8 GB resident per rank, 229.6 s load. Aggregate throughput reaches 21.2 tok/s at 32 concurrent streams; chunked prefill is ~170 tok/s. User attributes decode at batch 1 to fixed per-call dispatch overhead (~170 ms/token) rather than bandwidth, with collectives at 1% and MoE gather at 7%; ~330 ms of the ~500 ms/token remains unattributed. An unresolved multi-chunk prefill hang under TP is described, following the host rather than the rank index.

generation:

31.2 tokens/s

quant:

Q4 (MLX)

kv:

FP16

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User benchmarks Qwen 32B at 31.2 t/s generation on 1K context using an M3 Ultra 512GB. Setup is MLX with Q4 quantization and FP16 KV cache, batch size 1, 256 generation tokens, median of 3 trials. The post reports a systematic sweep of 5 models, 6 quants, and 7 context lengths (276 runs total), showing generation drops to 8.5 t/s at 128K context and prompt processing falls from ~345 t/s at 1K to ~154 t/s at 128K.