Local model performance on your hardware

generation:

40-46 tokens/s

prompt processing (prefill):

970-1000 tokens/s

quant:

Q4_K_XL (GGUF)

kv:

q4_0

User reports Qwen3.6-35B-A3B on an RTX 5050 Laptop 8GB at ~970-1000 tok/s prefill and ~40-46 tok/s decode on a 21.8k-token Python prompt generating 400 tokens. Setup is a llama.cpp fork with a hybrid MoE expert cache (800 MiB), Q4_K_XL GGUF, q4_0 KV cache, 90000 context, ubatch 4096, n-cpu-moe 40 and MTP draft speculation; experts ran from system RAM on a 32 GB DDR5 laptop. The baseline config (n-cpu-moe 34, ubatch 1024, no MoE cache) gave ~480 tok/s prefill and 41-43 tok/s decode, so the main gain is roughly 2x prefill. The user also tried the Moex engine at ~59-63 tok/s decode and ~826 tok/s prefill, and their branch reached 57-62 tok/s decode at 252K context.

generation:

118.0 tokens/s

quant:

Q2_K_XL (GGUF)

User reports Qwen3.6-35B-A3B at about 118 tok/s decode on a short prompt with their RustingStrata engine on an RTX 3060 12GB, versus 68 tok/s for llama.cpp. Setup is RustingStrata with Q2_K_XL GGUF weights (11.7 GiB) on a 3060 12GB, 7600X and 32 GB RAM, with experts that do not fit in VRAM running on the CPU. Decode is 113 tok/s after a 4K prompt and 92 tok/s at 32K; prefill is 2216 tok/s at 4K against about 2236 for llama.cpp, and 1800 tok/s at 32K. With MTP drafts decode reaches 160 tok/s on an easy prompt, with about +16% expected on normal text. Qwen3-Next-80B IQ1_S decodes at about 92 tok/s versus around 44 for llama.cpp.

generation:

105-125 tokens/s

quant:

NVFP4 (NVFP4)

User introduces SparkHub, a self-hosted LLM server and control panel for a single DGX Spark, with an OpenAI-compatible API and one-click switching between preset models. The fast preset runs Qwen3.6-35B-A3B in NVFP4, following NVIDIA's playbook. It reports about 105-125 tok/s single stream, and 60-70 tok/s per agent with 4 agents running.

generation:

82-89 tokens/s

User reports Qwen3.6-35B-A3B decoding at 82 to 89 t/s on one V100 16GB with their PXA 4.0 engine, a fork of llama.cpp. Setup is PXA 4.0 with greedy decoding, three or six repetitions, compared against mainline llama.cpp on the same box and quant class file. The figure is a range; the post claims 2.3x to 2.5x over mainline llama.cpp and 1.3x to 1.45x over PXA 3.1. Other rows cover Nemotron-3-Nano-30B-A3B at 106 to 115 t/s on one V100, Nemotron-3.5-Super-VL-120B-A12B at 24 to 26 t/s on two P100s, Nemotron-3.5-Bolt-74B-A9B at 44 to 45 t/s on two P100s, GLM-5.3-Flash at 9 to 10 t/s on two V100s, Qwen3.8-27B at 42 to 97 t/s on one V100, and Gemma-4-26B-A4B at 123 to 128 t/s decode and 2,400 to 4,400 t/s prompt processing on one V100. Qwen3.8 Flash-Next prompt processing with experts in system RAM reaches 550 to 660 t/s on one V100. The author notes the 97 t/s dense 27B figure is the code class with MTP drafting, prose lands in the 40s to 60s on one V100, and big MoE decoding on one V100 with experts in system RAM is not yet published.

generation:

27.0 tokens/s

prompt processing (prefill):

616.0 tokens/s

kv:

INT8

User reports Qwen3.6/3.8-27B decoding at 27.0 tok/s without MTP on a single modded 22 GB RTX 2080 Ti under NInfer v0.3.0 on Windows 11, with prefill of 616 tok/s on a 512-token prompt. Setup uses a 64k context with an INT8 KV cache; the weights take 15.4 GiB and the engine runs entirely in VRAM with no system RAM offloading. MTP speculative decoding is on by default. With MTP, decode reaches 47.5 tok/s on a real code prompt (26.3 without) and 28.8 tok/s on a real prose prompt (26.0 without); prefill on a 64k-token prompt is 458 tok/s. The engine supports only Qwen3.6/3.8-27B, and dropping RTX 5090, FP8, FP4 and 35B-A3B code paths cut the binaries from 1.56 GB to 0.28 GB.

generation:

10.0 tokens/s

quant:

Q4

User reports Qwen3.6 35B-A3B at Q4 (about 20GB) reaching 10 t/s on a laptop with an AMD Ryzen 7 7840U and 16GB DDR5, with no GPU. The run is CPU-only; the user built their own interface (Roaster) after Colibri gave about 6 t/s with the same settings. A parallel pipeline generating for two contexts at once gives around 4.6 t/s each. The user plans to use it for coding and asks for tuning suggestions.

generation:

122.5 tokens/s

prompt processing (prefill):

76.7 tokens/s

quant:

UD-Q4_K_XL (GGUF)

User compares a Mac Studio M5 Max (40-core GPU, 64GB) against their current dual RTX 4060 Ti 16GB setup and asks what prefill and decode speeds Mac owners get. On the 2x 4060 Ti they report Qwen3.6-35B-A3B at UD-Q4_K_XL with 262k context reaching 76.7 t/s prefill and 122.5 t/s decode. A second row gives Qwen3.8-27B dense at UD-Q4_K_XL with 200k context at 55.5 t/s prefill and 34.3 t/s decode. No engine is named.

generation:

32.6 tokens/s

prompt processing (prefill):

3581.0 tokens/s

quant:

bf16 (BF16)

kv:

bf16

User reports Qwen3.6-35B-A3B at 3,581 t/s prefill and 32.6 t/s decode at 2k context on a Strix Halo 128GB APU, comparing stock vLLM against the same vLLM with the libstrix HIP kernel plugin. Setup is vLLM with bf16 weights and bf16 KV cache, single stream on one APU; the plugin installs via pip and only activates on ROCm gfx1151. At 64k the same run goes from 260 to 2,617 t/s prefill and 12.8 to 26.9 t/s decode; at 128k with int8 KV it reaches 1,751 t/s prefill and 26 t/s decode, and a 256k prompt prefills in 241 s versus 7,568 s stock. KLD against stock is 0.001-0.006 with 97-99% top-1 agreement, and the user notes decode gains are absent where the replaced kernels are not the bottleneck.

prompt processing (prefill):

3556.7 tokens/s

quant:

Q4_K_M (GGUF)

User benchmarks a llama.cpp CUDA PR that fuses MoE gate/up matmuls and GLU in MMQ, reporting prompt-processing throughput on DGX Spark (GB10) and RTX 5090. On DGX Spark with Qwen3.6-35B-A3B Q4_K_M, prompt processing rises from 3235.8 to 3556.7 t/s (+9.9%); the Qwen3-30B-A3B Q4_K_M row goes from 2911.4 to 3410.8 t/s (+17.2%). On RTX 5090 the same Qwen3.6-35B-A3B Q4_K_M row goes from 15000.3 to 15852.0 t/s (+5.7%), with UD-Q4_K_M at 15030.2 to 15947.8 t/s (+6.1%). No generation speeds are given.

generation:

11.0 tokens/s

prompt processing (prefill):

66.0 tokens/s

User reports Qwen3.6-35B reading at 66 tok/s and writing at about 11 tok/s on a 2013 Mac Pro with two AMD FirePro D700 cards and 64 GB RAM. Setup is llama.cpp with Vulkan/MoltenVK on original macOS 12, measured with llama-bench. User warns that an earlier Metal path produced wrong results and looked about 9x faster than the correct run, so a tok/s figure without a correctness check is meaningless.

generation:

15.0 tokens/s

quant:

4-bit (MLX)

User reports Qwen3.6-35B-A3B at 15.0 tokens/s on an M5 MacBook Air with 16 GB RAM, keeping routed experts on SSD and loading only those selected by the MoE router. Setup uses MLX with a 4-bit checkpoint, around 1.7 GB of MLX memory, top-8 routing unchanged and no experts dropped; output is byte-identical to stock mlx_lm. The 15.0 t/s is a controlled benchmark, 72.6% faster than direct SSD streaming; with other apps open it is around 11.8 t/s, and time to first token for a short prompt is around 2.3 seconds.

generation:

40.0 tokens/s

kv:

8-bit

User reports Qwen 3.6 35B-A3B at about 40 tok/s on an M5 MacBook Pro with 32GB unified memory, against about 17 tok/s for a dense 27B at 4-bit. Setup uses oMLX with an SSD-backed cache server, 8-bit KV cache and context set to roughly 32K. The user says the 16GB dense model plus KV cache filled the 32GB machine and caused a shutdown; the MoE has been stable since.

generation:

155.0 tokens/s

User compares qwen3:4b at ~141 tokens/s with qwen3.6:35b at ~155 tokens/s in Ollama on a Mac Studio with 128 GB unified memory, using eval rate from --verbose so loading time is excluded. The 35B is a mixture-of-experts model with 8 of 256 experts active per token. With a very long context set in Ollama, even the 2.5 GB 4B model showed ~43 GB in ollama ps. User cautions that one test is not proof and the puzzle was well known, so both models may have seen it before.

generation:

40-41 tokens/s

kv:

INT8

User reports roughly 40-41 tok/s decode on a Jetson AGX Orin 64GB running Qwen3.6-35B-A3B with sampled MTP-2 enabled, in a fork of Strata v0.1.45. Setup uses unified INT8 KV caching, prefix checkpoints and optional KV preallocation. The user notes these were short-prompt tests and not matched llama.cpp-versus-Strata benchmarks. For comparison, the user got around 21 tok/s decode with MTP-3 in llama.cpp and around 30 tok/s in earlier Strata runs.

generation:

375-400 tokens/s

quant:

Splash

User reports running Qwen3.6-35B-A3B (Splash build) on an M5 Max 128GB MacBook Pro in LM Studio at roughly 375-400 tok/s with a 262k context. Setup is LM Studio with the Splash build at 262k context on the 128GB machine. The speed is given only as a range, so no single figure is recorded. The post's main subject is a cloud agent running a training-and-grading program against the local model, not a benchmark of it.

generation:

62.0 tokens/s

prompt processing (prefill):

464.0 tokens/s

quant:

4-bit group-64 (MLX)

kv:

Q8

User benchmarks Qwen3.6-35B-A3B on an M1 Max 64GB, reporting 464 tok/s prefill and 62 tok/s decode at a 128K prompt with the MTPLX M1 fork. Setup is the MTPLX M1 fork v2.12.2-m1.11 with mlx-community 4-bit group-64 weights plus an MTP head, 8-bit paged affine KV cache, and speculative decoding on. Splash-M1 1.3.2-m1.1 reached 302 tok/s prefill and 41 tok/s decode at 128K, and oMLX 385 tok/s prefill and 33 tok/s decode. The fork was faster from 8K up on agent logs, 1.46x prefill and 1.5x decode at 128K, while Splash-M1 used less memory (23.5 GB vs 28.1 GB at 128K). The same post also compares Qwen3.8-27B, where the fork reached 132 tok/s prefill and 25.0 tok/s decode at 128K. Only the 2K, 32K and 128K agent-log cells were run three times; the rest are single runs.

generation:

375-400 tokens/s

User reports a Qwen3.6-35B-A3B Splash build in LM Studio on an M5 Max 128GB MacBook Pro at roughly 375-400 tok/s, used to generate and cold-solve 800 math problems. A Qwopus 122B MLX build at about 38 tok/s acts as judge for disputed items; the cloud agent Muse orchestrates over Tailscale. Of 800 problems, 731 verified clean, 62 had corrected answer keys, 7 were excluded, and 100 were held out; the solve-and-adjudication passes logged 4.1M+ completion tokens. The user notes the 122B judge went 1-for-2 on the hardest disputes.

prompt processing (prefill):

70-100 tokens/s

quant:

Q8_0 (GGUF)

User reports a llama.cpp fork for legacy NVIDIA GPUs running MTP Qwen3.6-35B-A3B at Q8_0 on 4x Tesla K40c (Kepler, compute capability 3.5), reaching above 60 t/s with Windows NCCL, up from 35 t/s. Setup is a custom llama.cpp fork with CUDA graphs and Windows NCCL integration; CUDA graphs took the run from about 35 to 55 t/s. Prefill speeds varied between 70-100 t/s, and the user calls these preliminary results from their own system rather than universal benchmarks.

generation:

125.5 tokens/s

prompt processing (prefill):

1874.7 tokens/s

quant:

Q4_K_M (GGUF)

User reports Qwen3.6-35B-A3B at 125.47 t/s generation on an RX 7900 XTX 24GB. Setup is llama.cpp (Vulkan backend, v0.7.0) via LM Studio with Q4_K_M GGUF and no speculative decoding. Three consecutive runs gave 125.47, 125.07 and 126.47 t/s generation; prompt processing was 1874.71 t/s over 427 tokens, 1288.99 t/s over 311 tokens and 770.07 t/s over 157 tokens.

generation:

122.7 tokens/s

quant:

4bit (MLX)

User benchmarks Qwen3.6-35B-A3B at 122.7 tok/s on an M3 Max 36GB with Kapitsa 0.2.12, tying oMLX at 122.9 tok/s. Setup is Kapitsa with a 4-bit MLX build and MTP head plus DFlash2 drafter, sustained 256-token prose, median of 3 runs. Other engines on the same model: Rapid-MLX 110.3, Splash 97.0, MTPLX 79.9 tok/s. On dense Qwen3.8-27B-4bit Kapitsa leads at 41.3 tok/s, ahead of oMLX 34.1, Splash 33.0, MTPLX 31.6 and Rapid 26.2. Flint 40B-A2.5B reaches 134.2 tok/s on Kapitsa and 103.0 on Rapid. Draft acceptance was 75-78% dense and 81-85% MoE. No formal quality eval was run.