Find which open-weight LLMs fit in your GPU or Mac and compare the speeds people report on setups like yours.
Pick your GPU or Mac, then read the speeds people reported on it, or estimate which models fit and how fast they run.
Free to use. No account needed. Memory estimates and community measurements are labelled separately.
Search for a model to see its reported speeds across GPUs and Macs.
These are individual setups, not a controlled benchmark. Compare GPU count, quantization, context and offloading before comparing speeds. How to read a report →
Exact recorded values. Context may be a configured limit; matching filters does not establish identical prompts, offloading or concurrency.
M3 Ultra 96GB · 100,000 ctx
User asks what throughput to expect from Qwen 3.8 27B on an M3 Ultra 96GB at 100k context, and whether it would be twice as fast as their current setup. Their current two-eGPU setup runs the Q4_K_M model at 23-30 tk/s, which they call usable but slow. No M3 Ultra figures were measured; the post is a question, not a benchmark.
User reports running Qwen3.8-Flash-Next oQ4 through oMLX on a Mac Studio M3 Ultra 96GB at around 60-70 t/s decode. Setup is oMLX with an MLX oQ4 quant of the model. The figure is given as a range, and the user asks whether the Splash engine running a GGUF model could beat MLX native decode speed, and about the quality trade-offs.
M3 Ultra 96GB · oMLX · 200,000 ctx
User reports Flash-Next running through oMLX plus Splash on a Mac Studio M3 Ultra 96GB, used mainly for agentic coding with Pi. Generation is given as 70–100 tokens/s and prefill as 900–1,000 tokens/s, both sustained up to 200K tokens of context with no noticeable degradation. No quant, KV cache format or single measured figure is stated.
M3 Ultra 96GB · llama.cpp
User reports Qwen3.5-35B-A3B at 73.43 tokens/s generation and 1808 tokens/s prefill on an Apple M3 Ultra 96GB. Setup is llama.cpp build 8860 with Q8_0 GGUF weights and f16 KV cache, Metal backend, single machine. The figures are separate synthetic tests (TG128 and PP512), not an end-to-end request. Historical TG128 85.48, turbo3 KV 63.23 and PP8192 1183 belong to other configurations and are kept for comparison only.
User reports 73 t/s generation and 347 t/s prompt processing on a Mac Studio M3 Ultra with 96GB, running mlx_lm.server with no speculative decoding and one request at a time. Source evidence (verbatim): Source: https://elitetestengineering.substack.com/p/gpt-oss-120b-on-two-generations-of # GPT-OSS-120B on Two Generations of Mac Studio: The Long-Context Gap Gets Bigger, Not Smaller Two Mac Studios running gpt-oss-120b (mlx-community/gpt-oss-120b-mxfp4-bf16, about 65GB) through mlx_lm.server. M5 Ultra: 30 CPU cores, 64-core GPU, 256GB unified memory. M3 Ultra: 28 CPU cores, 60-core GPU, 96GB unified memory. Short prompt of about 146 tokens: M5 Ultra prefill about 555 tok/s and decode 99 tok/s, M3 Ultra prefill about 347 tok/s and decode 73 tok/s. Long prompt of about 24,400 tokens: M5 Ultra prefill about 3,955 tok/s and decode 76 tok/s, M3 Ultra prefill about 1,101 tok/s and decode 55 tok/s. No speculative decoding was used. The M5 Ultra ran with decode concurrency set above 1 while the M3 Ultra ran with concurrency 1 and prompt caching off, and the article states the M5 Ultra setting inconsistently (2 and 4).
User reports 49.1 t/s generation. Source evidence (verbatim): Source: https://arxiv.org/html/2605.00519v1 # First-hand M3 Ultra 96GB Qwen3-Next 80B native MLX 4-bit generation measurement Original experimental paper Silicon Showdown, arXiv 2605.00519v1, May 2026. Selected Table 5 row: Apple M3 Ultra 96GB, 28 CPU cores, 60 GPU cores; installed unified memory 96GB. macOS 26 Tahoe, MLX v0.30.6. Qwen3-Next-80B-A3B native MLX 4-bit: generation 49.1 tokens/s, 0% CPU offload. Long prompt approximately 496 tokens; output 256 tokens, greedy decoding. Authors report arithmetic mean of three consecutive timed runs after two untimed warmups and fixed random seed. This is their published generation-phase measurement, not a speed derived from TTFT. Prompt-processing speed unreported; leave promptTps null. Context capacity, KV quantization, speculation, concurrency and resident allocation unreported. Verified base-model reference explicitly cited by the paper: https://huggingface.co/Qwen/Qwen3-Next-80B-A3B-Instruct . This is not the tested MLX quantization repository; exact quantized artifact and revision unknown. Do not infer disk size or resident memory. Table 2 names the 96GB machine, not the followed 256GB configuration. Other models, machines and Nvidia offload comparisons are separate runs. Source: https://arxiv.org/html/2605.00519v1 .
M3 Ultra 96GB · llama.cpp
User reports Qwen3.8-27B at 29 t/s generation and 274 t/s prompt processing on an M3 Ultra 96GB. Setup is llama.cpp 0.4.0 with the Q4_K_M GGUF, run natively through llama-bench on one Apple GPU via Metal. The native benchmark gives 28.97 +/- 0.03 tok/s generation and 273.98 +/- 0.03 tok/s prompt processing; context length, batch settings and offload flags are unreported.
User reports Qwen3.5-35B-A3B at 109.7 tokens/s generation and 1770.1 tokens/s prefill on an Apple Mac Studio M3 Ultra with 96 GB unified memory. Setup is MLX with 4-bit weights loaded via mlx_lm.load() and run through mlx_lm.generate.stream_generate() without an HTTP server, at a 512-token context setting, generating 50 tokens at temperature 0.0. Source reports 20.20 GB peak memory and 77.4 MB KV memory for this row. 512 is the tested context setting, not a configured maximum; runtime version, GPU core count, KV-cache precision and maximum context are not stated. This is a short-context standalone result, not an end-to-end server throughput measurement.
M3 Ultra 96GB · ds4 · 262,144 ctx
User reports Qwen3.8 Flash Next Q4 running through ds4 on an M3 Ultra 96GB with the full 262K context configured, delivering 55–60 tok/s decode and 667 tok/s prefill. About 80GB is used for model, KV cache and buffers, while the model's 95GB n-gram table streams from SSD. The user runs it daily through a codex harness as a headless box.
User compares Mac Studio configs (M5 Ultra 96GB, M5 Max 128GB, M5 Max 64GB) for local LLM use. They discuss Qwen3.8-27B at Q8 and Qwen3.8-Flash-Next (6B active) at various quants, including Flash-Next on 96GB Ultra with UD-IQ4_XS (93.7GB, 91.1% retention) and on 128GB Max with UD-Q4_K_XL (111GB, 93.5% retention), with the N-gram layer offloaded to SSD. User asks for advice on bandwidth versus quant tier, and whether to just get the 64GB box. No actual benchmark numbers are reported.
On Qwen3.8 27B at 4-bit, plain single-GPU runs. Full ranking