Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User benchmarks Qwen3.6-35B-A3B at 76.0 t/s single-stream decode and 1,225 t/s prefill on one Intel Arc Pro B70 32GB.
Setup is llama.cpp with the Vulkan backend, Q4_K_M quant and q4_0 KV cache, no speculative decoding. The host has 32 GB RAM.
With --ubatch-size 2048 prefill reaches 1,824 t/s at pp2048 and 8-stream aggregate decode is 624 t/s. The SYCL backend gives 77.3 t/s decode and 979 t/s prefill at pp2048. A dense Qwen3.6-27B Q4_K_XL comparison runs at 20 t/s single stream.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports a 35B-A3B model at 144 tok/s on a 2021 M1 Max using Inco Splash.
The post is a head-to-head comparison with oMLX and MTPLX covering speed, temperature, power, and memory.
The M1 Max memory size is not stated.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.6 27B at 32.17 t/s generation on a single NVIDIA Tesla V100 SXM2 32GB.
Setup is llama.cpp (build 6eab47181) with Q4_K_M GGUF weights, run via llama-bench with -p 0 and -n 128,256,512,1024,2048, steady across those generation lengths.
Other quants measured on the same card: Q5_K_M 29.14 t/s, Q6_K 25.11 t/s, Q8_0 22.80 t/s. BF16 required an added RTX 3090 24GB and ran 15.27 t/s.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.6 35B-A3B at IQ4 on a Tesla P40 24GB, getting 37-83 tok/s generation with MTP enabled.
Setup is llama.cpp with IQ4 quant. Prefill starts at 600 t/s for short prompts and degrades to 300-400 t/s by the end of a long prompt.
User asks whether IQ quants are actually slower on the P40, noting they did not feel a slowdown moving from Q4 to IQ4.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User benchmarks Qwen3.6 27B with MTP speculative decoding at 117.95 tok/s on an RTX Pro 6000 Blackwell, compared to 43.22 tok/s without MTP, a 2.73x speedup.
Setup is llama.cpp with Q8_0 GGUF weights, 3000 token context, and 4 speculative tokens. Hardware is an RTX Pro 6000 Blackwell with 96GB VRAM and 92GB system RAM.
A separate vLLM run of Gemma 4 31B FP8 with MTP averaged 124.26 tok/s over 10 runs, totaling 4740 tokens in 38.14s.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
agentictool-use
User reports Qwen3.6 35B-A3B at 112.4 tok/s on an RTX 3090 24GB with 64K context.
Setup is Ollama 0.32.14 with a Modelfile alias baking in num_ctx 65536; at 64K, 96.6% of layers were on GPU with about 0.71 GiB spilled to CPU.
The same runner at 32K reached 139.6 tok/s, so 64K was about 20% slower. The user also compares agent task success rates: warning the model to fix wrong values gave 1/9 successful runs with 4 false completions, deterministic tool-layer correction gave 8/15 with 2 false completions, and the same tool-layer approach at 64K gave 5/6 with 0 false completions.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
codingcreative-writing
User reports Qwen3.6 35B A3B at 54-60 t/s generation (66-72 t/s on code) on a single Tesla P100 16GB, up from 30-35 t/s prose on an RX 6600 XT.
Setup is llama.cpp with shinbunbun patches, UD_Q4_K_XL quant, 16-bit KV cache, MTP speculative decoding, and --n-cpu-moe 22, running 32k context.
Prefill is 600 t/s at 0 ctx dropping to 440-500 t/s by 10k. The user notes the P100 is underrated for the price and has a second card coming for full offload.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.6-35B-A3B-NVFP4 at 139.1 tok/s on a single NVIDIA Jetson AGX Thor with 117 GB unified memory.
Setup is vLLM built from source for sm_110a with DFlash speculative decoding (12 tokens), marlin MoE backend, flash_attn attention backend, 65536 context length, and 0.78 GPU memory utilization.
The post also benchmarks Qwen3.5-4B-NVFP4 at 155.8 tok/s, Qwen3.6-27B-NVFP4 at 50.1 tok/s, and Qwen3.5-122B-A10B-NVFP4 at 52.6 tok/s, all at concurrency 1. The 122B requires cutlass MoE and TRITON_ATTN due to a Marlin crash at 256 experts, and achieves 27-42 tok/s with DFlash versus 10.9 tok/s autoregressive.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
codingtool-useagenticmathlong-context
User reports Qwen3.6-35B-A3B at 120 tok/s on an RTX 3090.
Setup is llama.cpp with Unsloth's Q3_K_M GGUF quant and bf16 KV cache, using 23GB of the card's 24GB VRAM at 262144 tokens of context.
The user says it is the first local model to saturate their personal task benchmark, running it for three days through OpenCode, though they note some tasks needed reinforced prompts and adapted tooling.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.6-35B-A3B at 706.0 t/s prompt processing on an AMD Radeon RX 9070 XT 16GB under ROCm.
Setup is llama.cpp with Q4_K_M GGUF, flash attention enabled, -ncmoe 40 and -ub 512, on a Ryzen 9 5950X with 64 GB DDR4-3200.
The same configuration under Vulkan reached 333.4 t/s prompt and 29.3 t/s generation; the user retired Vulkan in favour of ROCm.
User reports Qwen3.6-27B-MTP UD-Q8_K_XL running on 2x RTX 3090 with llama.cpp build b9455, using tensor-split 50,50 and MTP speculative decoding. Decode speeds range from 54 to 81 t/s across runs, with prefill between 823 and 1421 t/s at contexts up to 68K. The user notes vLLM previously achieved 70+ t/s but lacked good quants, and praises the code output quality with this setup.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
codingagentic
User reports Qwen3.6 at 20 to 50 tokens per second on dual RTX 3090 GPUs.
Setup is llama.cpp with Q6 quantization and MTP speculative decoding; the cards are undervolted and limited to 65°C.
The user compares Q4 to Q6 quality for a coding agent, calls the Q6 improvement outstanding, and says the local server now works similarly to paid APIs.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.6-35B-A3B at 12.48 t/s generation and 222 t/s prefill on a single RTX 3070 Laptop 8GB with 32GB system RAM.
Setup is Krasis runtime with HQQ4 quantization and k4v4 KV cache, streaming the model from system RAM through VRAM.
Figures are best throughput obtained during the benchmark, not averages. The post also lists results for RTX 5080, RTX A4500, and RTX 5090.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
coding
User benchmarks Qwen3.6 27B on a single RTX 3090 at an enforced 370 W power cap, comparing ik_llama.cpp with IQ4_KS against mainline llama.cpp with Q4_K_M. ik_llama.cpp reaches ~72 t/s code decode and ~60 t/s narrative decode versus ~58 and ~50 t/s for mainline, an 18-20% gain, with quality tied at 100-103/150 on an 8-pack and ~0.5 GB lower VRAM.
Setup uses q4_0 KV cache, MTP n=2, -np 1, and a 200K context configuration that fills to ~183K tokens. A two-stage ngram plus MTP speculator on the same engine lifts code decode to ~98 t/s at the same power cap, with narrative flat at ~59 t/s.
The user notes the earlier reported tie was a measurement artifact from a stale mainline container answering on the shared port, and that both engines are strongly power-sensitive between 230 W and 370 W.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
agenticcodinglong-context
User reports Qwen3.6-35B-A3B at 144 tok/s on an M1 Max 64GB, up from 99 tok/s with the previous Splash build.
Setup is Inco Splash 1.0.2-m1.1 with 4-bit weights, an 8-bit KV cache and DFlash speculative decoding with 8 draft tokens; time to first token for a 38K prompt is 72 s, down from 106 s.
In a head-to-head on the same Mac with Qwen3.8-27B, Splash decodes at 41.5 tok/s versus 25.9 for MTPLX and 19.8 for oMLX, at 75.3 C and 1.06 J per token; prefill at 44K tokens is 102 tok/s against 125-127 tok/s for the MLX engines.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
long-context
User reports Qwen3.6-35B-A3B at 51.6 tok/s on a Tesla P40 24GB under WSL2.
Setup is llama.cpp built for sm_61 with UD-Q3_K_XL (16.85 GB, 17.23 GB with MTP), full GPU offload, MTP speculative decoding at --spec-draft-n-max 2, and Q8_0 KV cache at 131072 context.
Baseline without speculation is 37.9 tok/s; MTP with speculation disabled lands at 38.3 tok/s, confirming the speedup is real. Prefill is 982 tok/s (pp512). Context scaling from 4096 to 131072 costs about 1 GB VRAM and about 2% throughput. Needle-in-a-haystack passed 9 of 9 at 10/50/90 percent depth, with 6315-token prompts at about 790 tok/s prefill and 25111-token prompts at 422-609 tok/s prefill. All generation figures are single-stream.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
agenticcodingtool-use
User reports Qwen3.6 27B at 19.63 t/s on an AMD Strix Halo 128GB, within 2.3% of a DGX Spark's 20.1 t/s on the MLPerf v6.1 Edge Agentic benchmark.
Setup is the official MLPerf Edge Agentic harness with NVFP4 quant, same engine and quant recipe on both boxes, using 1,007 recorded agentic coding turns and BFCL function-calling accuracy gates.
A separate native Windows/Linux build for Qwen3.8 27B NVFP4 was verified on a Framework Desktop at 17.3 t/s on gfx1151 with an HA-20 BFCL score of 84.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.6-35B-A3B at 63.66 t/s sustained generation on an Intel Arc Pro B70.
Setup is llama.cpp commit 210a657 with SYCL backend, Q4_K_XL quant and q8_0 KV cache, 262144 context.
Short bursts peak at ~65 t/s; sustained generation over 1000+ tokens holds at 62-64 t/s. llama-bench on the same machine gives tg128 70.54 t/s and pp512 977.40 t/s.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.6 27B at 50.27 t/s narrative and 58.92 t/s code on a single RTX 3090 at 200K context.
Setup is llama.cpp with MTP speculative decoding and the Unsloth Q4_K_M GGUF; prefill measured 1025 t/s.
Soak testing gave a p50 of 54.5 t/s with 102 percent retention, and a verify-stress ladder showed 183K fill capacity with margin.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
agentic
User reports Qwen3.6-35B-A3B at ~85 t/s generation and ~300 t/s prompt processing on 2x Tesla P100.
Setup is llama.cpp with the Q4_K_XL GGUF, MTP speculative decoding, and community P100 patches that added about 50% decode; a single P100 on Q2_K_XL reached ~76 t/s generation and ~250 t/s prompt processing.
Card count barely affects single-stream speed, and PCIe lane width (x16/x16, x16/x8, x8/x8) made no difference. The user corrects an earlier 3-card result that was invalidated by stuck 405 MHz core clocks.