Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8 27B at around 20 t/s sustained on an M5 Pro 64GB, with about 30 t/s in the first 2-4k tokens.
User finds it unbearably slow, possibly because the model thinks so much.
User also tried antirez's DS4 with DeepSeek V4 Flash and gets around 8 t/s sustained, which they call slow and basically unusable.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
agenticcoding
User reports Nex-N2.5-mini at 133.6 t/s on an Apple M5 Max.
Setup is MLX with the MLX-4bit quant, using temperature 0.7, top_p 0.95, top_k 40, and reasoning_effort high.
User calls the generation speed and prompt processing really good, the memory footprint okay, and the quality good across the board, and plans to use it for agents and possibly coding.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User benchmarks GLM-5.3 321B across a MacBook M5 Max 128GB and an AMD Strix Halo 128GB connected over TB4.
The default tensor split was slower than the Mac alone. A Mac-heavy split improved prompt processing from 188 to 333 t/s and generation from 17 to 24 t/s.
User also reports 12-15 t/s for model sizes of 148-200 GB and plans to add two Asus GX10 next week.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
summarization
User asks whether about 9 t/s is normal for their setup, and is new to local LLMs.
User also asks for model recommendations for PDF summaries on 16 GB of RAM.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
codingsummarization
User asks about a Mac Studio M5 Max 128GB for local Qwen3.8-flash-next.
The user mentions the oQ4e quant at ~55 tok/s and also the oQ5e quant.
The user is debating whether to buy now or wait for the M7 Ultra, for private documents, notes, coding and general assistant work.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
long-contextcoding
User reports about 40 t/s on prose and 75 t/s on coding at roughly 760k context.
Setup uses 8-bit dense layers and 4-bit expert layers, with peak memory around 117 GB and a requirement of iogpu.wired_limit_mb=120000. Model weights are at ddalcu/Qwen3.8-Flash-Next-MLX-Serve-mixed-4-8bit.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
agentic
User compares an M5 Ultra 96GB against an M5 Max 128GB for running Qwen3.8-27B at Q8, and reports about 15 t/s on the Ultra.
The user also discusses Qwen3.8-Flash-Next, a multimodal MoE with 176B total parameters and about 6B active, and estimates it will not fit in 96GB.
MLX and llama.cpp are mentioned as possible engines, but the user does not confirm using either.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
coding
User benchmarks a 128 GB M5 Max MacBook Pro across multiple models, selecting Qwen 3.6 27B MLX Q8 as the primary one.
Other models tested include Gemma 4, MiniMax M2.7 and DeepSeek V4 Flash.
The user is enthusiastic about local AI.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User benchmarks several models on an M5 Max 128GB and selects Qwen3.6 35B A3B as the primary model for its MoE detail.
Other models tested include Gemma 4, Qwen 3.6 27B, MiniMax M2.7, Mistral Medium 3.5, Step 3.7 Flash and DeepSeek V4 Flash.
Results show strong performance for MoE models.
User compares DeepSeek-V4-Flash on an M5 Max MacBook with 128 GB against 2x DGX Spark. Terminal-Bench 2.1 scores 54% (47/87) on the Mac and 52% (45/86) on the Spark pair.
The Mac runs a heavily quantized GGUF at IQ2_XXS/Q2_K, about 2.45 bits per weight. The Spark pair runs the native FP8/FP4 checkpoint.
The Mac wins 11 tasks outright and the Spark pair 9, with no significant difference in task-solving. The Spark pair offers higher speed at 54-58 tok/s single-stream and 253 tok/s aggregate at concurrency 8, plus a larger 262K server context window.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
agenticcoding
User reports about 30 t/s on an M5 Max with a quantized DeepSeek V4 Flash running on the DS4 engine.
The user states this is double the speed of llama.cpp.
The user asks for feedback from CUDA and ROCm users.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports DeepSeek-V4-Flash 284B-A13B at 4.8 tok/s on a 24 GB M5 Pro with about 5.3 GB of memory in use.
Setup is Mference, an open-source engine. The user also runs Gemma 4 26B-A4B and Qwen 3.6 35B-A3B.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports DeepSeek V4 Flash 0731 at about 50 t/s prefill and about 1 t/s decode on an M5 Air with 32 GB.
Setup uses the streamed experts trick with a model of roughly 300B total parameters.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
codingtool-usemath
User compares DeepSeek V4 Flash 9B distill against its base model Qwen3.5 9B on a MacBook Pro M5 Pro with 48 GB RAM.
Both models gave the same answers on 6/8 tasks, but the distill used fewer tokens (5480 vs 8975). Throughput was near identical (44 vs 41 tok/s).
The distill was slower on open-ended tasks (1467 tokens vs 867 for a buffer overflow explanation). The base model failed arithmetic due to a token cap.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports 10-17 t/s in SSD streaming mode.
Setup keeps routed experts partly in RAM cache and pulls them from the GGUF on cache misses. Experts and output head are Q8_0, while router, embeddings and V4 auxiliary blocks are FP16.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
agentic
User benchmarks Qwen3.8-27B on an M5 Max 128GB across four engines.
Setup is llama.cpp with native MTP at 31.8 t/s sustained and 22.7 t/s at 56k context, TTFT 125ms, prefix reuse 0.80, RAM 36.8GB; oMLX 0.6 at 29.8 t/s and 24.6 t/s at depth, TTFT 1968ms, prefix reuse 0.82, ~17GB; mlx-vlm with MTP at 43.3 t/s and 28.9 t/s at depth, TTFT 9982ms, no prefix reuse, 15.6GB; and MTPLX 2.6 at 27.1 t/s and 23.1 t/s at depth, TTFT 13400ms, no prefix reuse.
The fastest engine, mlx-vlm, is unusable for agents due to 10s TTFT and no prefix reuse. Without --spec-type draft-mtp, llama.cpp drops the MTP head; a draft cap of 4 gives 31.8 t/s versus 21.9 t/s with no speculation, cap 2 from Qwen3.6 presets gives only 22.2 t/s, and cap 6 loses gains. At 262k context with q8_0 KV cache llama.cpp gives 29.6 t/s and 18.5 t/s at depth; at 131k with f16 it gives 31.8 t/s and 22.7 t/s at depth. The model does 5 searches then calls more tools instead of writing the brief and dies at max_turns. reasoning_effort 'none' throws an exception on 3.8, so enable_thinking: false is needed; oMLX does not activate MTP on 3.8; the model field of the response should be checked rather than /v1/models; and the M5 Max throttles to 50% after 80-110s. No generational verdict versus 3.6 is given.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
codingagentic
User asks which model to use for agentic coding.
User reports DeepSeek V4 Flash at 10-11 t/s with 1M context, and Qwen3.8 27B at 15-16 t/s with FP8 quants.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports 2.04 t/s with an 8 GB expert cache and 1.23 t/s with a 32 GB expert cache, streaming from SSD.
Speculative decoding and prefetch both decreased throughput.
Codebook 2-bit quantization failed the quality gate.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
long-context
User reports Qwen3.8-Flash-Next at 79 GB in 2-bit running at 350K context for 3.5 hours on a 128 GB M5 Max, over 100 turns.
Prefill runs 1,561 t/s at 5.6K context tapering to 318 t/s at 111K, with normal turns at 77-854 t/s from prefix reuse. Decode is about 30-35 t/s at small context down to 11.5 t/s at 169K, with a dip to 7.7 t/s at around 140K due to Low Power Mode. Decode figures are effective throughput with speculative decoding.
The user calls the experience strong for the first ~100K context, then reports role confusion on long-tail tasks, suspecting the 2-bit quant or preview-model long-context quality.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User benchmarks Qwen3.8-27B MTP quants on an Apple M5 Max.
The oQ4e-mtp quant reaches 36.6 t/s with a quality score of 86.8.
User calls oQ4e-mtp the best balance of speed and quality and recommends it as the sweet spot.