Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
agentic
User asks how to configure a multi-agent setup across several local machines, with one orchestrator and smaller workers for atomic tasks.
Main PC runs Qwen 3.6 27B at ~30 t/s, the wife's PC runs Qwen 3.6 35B at ~55 t/s, and two Steam Decks run Qwen 3.5 2B at ~30 t/s. No GPU, engine, quant or context length is stated.
User asks whether the Steam Decks have a practical role and whether the model mix is a good choice.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User benchmarks four MLX engines on a 14" M3 Max MacBook Pro (36 GB) across Qwen3.6-35B-A3B and Qwen3.8-27B at 4-bit.
Rapid-MLX leads the MoE at 110.2 tok/s decode, while Splash leads the dense 27B at 31.7 tok/s (MTPLX 31.5, effectively a tie). oMLX trails on dense at 17.9 tok/s but is level with the leaders on the MoE at 104.2 tok/s.
Each engine used its own weight build: Rapid and MTPLX ran MTP-augmented 4-bit builds, Splash paired its own DFlash2 draft, and oMLX ran the plain mlx-community 4-bit. Sustained test was 3 runs x 256 output tokens, median, temperature 0, thinking off. User kept Rapid-MLX because the MoE is their daily model. Qwen3.8-Flash-Next (125B) does not fit on 36 GB.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
codingagentictool-usesummarization
User reports Qwen3.6 35B A3B at 62 t/s on an AMD RX 9070 XT 16GB, with vLLM on ROCm reaching 48 t/s in the same comparison.
Setup is llama.cpp with the Vulkan backend, 32k context, on a Ryzen 7 9800X3D with 64GB DDR5; the MoE model spills part of its weights to system RAM.
User notes the setup is early and numbers are not settled, that 16GB VRAM rules out big dense models, and that the local model needs supervision.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
summarizationmultilingualcodingtool-use
User reports Qwen3.6-35B-A3B at 63.8 t/s median decode on a Mac mini M6 with 32 GB unified memory.
Setup is LM Studio with the MLX 4-bit build and a 32,768-token context; the MLX build ignored the context setting and loaded 34k to 165k. Reasoning off, max_tokens 600, temperature 0.7, median of 3 runs per prompt.
The same model's Splash build (speculative decoding with DFlash2 drafts) reached 51.5-234.1 t/s across the five prompts, slower on German prose and faster on code and JSON. The user recommends Splash where it exists and notes the MLX build was faster than Splash on German prose (63 vs 52-59 t/s).
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Ornith-Agents-A1-3.6-35B-A3B (a dare_ties merge on Qwen3.6-35B-A3B with a grafted MTP block) at up to ~260 tok/s on an RTX 6000 Ada with 96GB VRAM.
Setup is llama.cpp with Q4_K_M GGUF, q8_0 KV cache, 131072 context, --parallel 4, and native MTP speculative decoding (--spec-type draft-mtp, --spec-draft-n-max 4). Model used ~25 GB of the 95.6 GB VRAM.
MTP draft acceptance stayed in the 42-48% range across generation lengths; measured throughput converged to ~245-260 tok/s. Q8_0 with MTP gave 190 tok/s, MTP disabled gave 150-200 tok/s, and vLLM measured 130 tok/s. spec-draft-n-max 3 gave 270 tok/s and spec-draft-n-max 2 gave 277 tok/s.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
coding
User reports Qwen3.6-35B-A3B at 105 tok/s decode at 128 context and 97 tok/s at 8k on an M5 Max 128 GB.
Setup is llama.cpp with Q8_0 GGUF weights, MTP speculative decoding on, 65536 context, 45 GB memory.
Without MTP the same model gives 93 tok/s at 128 and 87 tok/s at 8k; MLX gives 85 and 79 tok/s. MTP adds 75% on the dense 27B (17 to 32 tok/s) but only 12% on the MoE 35B-A3B. llama.cpp beats MLX by 10-24% on this machine. DeepSeek-V4-Flash under ds4 at q2-q4 (91 GB) reaches 33 tok/s at 128 and 28 tok/s at 8k.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
codingagentictool-uselong-context
User reports Qwen3.6-35B-A3B at 25.4 tok/s decode and 38.6 tok/s prefill on a MacBook Pro M1 Pro 32GB.
Setup is llama.cpp build b9466 with Q4_K_S GGUF and q8_0/q4_0 KV cache at 32K context, single sequence, all layers on the Metal GPU.
User compares the MoE against dense Qwen3.6-27B on the same host: the dense model decodes at 5.3 tok/s and prefills at 12.9 tok/s, uses 17.9 GB wired versus 22.4 GB, and swaps ~1 GB versus ~7.6 GB. Prefill is the bottleneck; cold TTFT on a 7.5K prompt is 92.8 s versus 0.2 s warm. Tool-calling was clean in 10/10 episodes.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.6-35B-A3B at 73.4 t/s on an Apple M5 Pro.
Setup is MLX with an MLX-4bit quant.
The same run also measured 74.7 t/s for the same model, and 37.8 and 36.9 t/s for qwen3.8-flash-next under llama.cpp.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive at 28.74 t/s generation on an RTX 4070 with a Ryzen 7 7700X and 32GB DDR5, roughly half the speed they get from llama.cpp.
Setup is ik_llama.cpp with a Q3_K_P GGUF, q8_0 KV cache, 131072 context, and 12 threads; the same command under llama.cpp gave 63.82 t/s generation and 70.52 t/s prompt eval.
The user is asking what could be done about the shortfall.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User benchmarks dflash-mlx at 121.2 t/s decode on a 32,768-token prompt, the fastest of four MLX frameworks tested on an Apple M5 Max with 64 GB unified memory.
Setup is dflash-mlx v0.1.7 with the 4-bit MLX Qwen3.6-35B-A3B model and a companion draft model for speculative decoding, single sequence, prefix caching disabled, cooldown=60 between runs.
Decode medians across the ladder: 148.7 t/s at 64 tokens, 138.9 at 512, 154.5 at 2,048, 127.7 at 4,096, 124.6 at 8,192, 119.0 at 16,384, and 121.2 at 32,768. TTFT at 32K is 8.1 s. The other three frameworks were tested at cooldown=2 and would score higher at the longer cooldown.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
codingagentic
User reports Qwen3.6-35B-A3B at 78.3 tok/s generation on an ASUS Ascent GX10 with NVIDIA GB10 and 128 GB unified memory.
Setup is vLLM with NVFP4 weights at 131,072-token context, one model at a time on the machine.
The run scored 86/86 hidden tests in 21:01 wall clock, the fastest perfect run in the collection, at 57.9 end-to-end tokens per second. The same model at FP8 scored 68/86 at 50.0 tok/s, a difference the user attributes to one all-or-nothing task rather than precision.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User benchmarks Qwen3.6-35B-A3B at 49.19 t/s generation and 937.47 t/s prompt processing on two AMD MI50 16GB cards.
Setup is llama.cpp build b11325 with Vulkan, Q5_K_XL GGUF, flash attention on, 99 GPU layers, and each card's power limit lowered to 145 watts.
The table also covers eight other models on the same dual-MI50 rig, with generation rates from 12.05 to 60.57 t/s. A third MI50 is on hand but not yet benchmarked.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
coding
User reports Qwen3.6 35B-A3B at 3 t/s on a laptop with an i3 CPU and 8GB RAM, no GPU.
Setup is llama.cpp (llama-server.exe) with IQ2_XXS GGUF weights, Q8_0 KV cache, 8192 context, and -ngl 0 so the whole model ran on the CPU.
The user generated a single-file HTML RPG in 24 minutes and notes that being GPU poor in 2026 is not so bad.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User benchmarks Qwen3.6-35B-A3B at 50.8 t/s decode and 2384 t/s prefill on a Strix Halo 128GB using the gufo engine with Q8_K_XL quantization.
Setup is gufo with Q8_K_XL GGUF and no KV cache quantization, running on the Ryzen AI Max+ 395 with 128 GB unified memory. The user compares gufo against llama.cpp across multiple quantizations and speculative decoding modes.
Gufo achieves higher prefill speeds (up to 2702 t/s on Q6_K_XL) but llama.cpp decodes faster on Q6_K_XL by 3% to 9%. With DFlash2 speculative decoding at 7 draft tokens, gufo reaches 78.6 t/s on Q8_K_XL versus llama.cpp's 51.7 t/s. The user notes that speculative output is not bit-identical to plain greedy output on either engine.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
agenticsecurity-researchlearning
User reports about 25 tokens/s on short prompts running a 35B MoE model at 4-bit with 131K context on two base M4 Mac minis (16 GB each) over Thunderbolt 4 using llama.cpp RPC mode.
Setup splits the model across the two M4s with a 50/50 memory split; a 40/60 split OOMed one machine. An M1 Mac mini acts as gateway with an OpenAI-compatible API and HTTP proxy.
A 64K-token prompt takes almost five minutes to first token, and prompt cache hits arrive roughly 70x sooner. User notes reasoning was on by default and that the setup is good for background agents and learning but cloud models are still used for daily work.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen 3.6 27B at 18.3 tok/s decode on an Apple M4 Pro with 48 GB unified memory.
Setup is MLX with 4-bit weights, 4096 context, and native MTP speculative decoding at depth 3.
Without MTP the model decodes at about 7 tok/s; served over WiFi it reaches 14.3 tok/s, and llama.cpp with MTP gives 10.5 tok/s.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.6-35B-A3B at 33-36 t/s generation and 145 t/s prompt processing on an Intel Arc B390 integrated GPU after tuning llama.cpp.
Setup is llama.cpp 0.5.0-dev (build 158) with Vulkan backend, Q4_K_M quant, 30 GiB unified memory and 23.7 GiB device memory, MTP speculative decoding with spec-draft-n-max 2, batch and ubatch 2048, CPU_MOE disabled.
Before the changes the same setup gave 15-17 t/s generation and 130 t/s prompt processing; llama-bench reports 38.7 t/s generation.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.6-35B-A3B at 101.7 tok/s on short prompts on an RTX 3090 24GB.
Setup is llama.cpp with Q4_K_XL, q8_0 KV cache, 65536 context, flash attention on, and all layers offloaded to the GPU.
The same model on Ollama reached 98.9 tok/s at 32k context and 23.4 GB VRAM. Qwen3.5-35B-A3B on llama.cpp reached 142.2 tok/s at 65k context and 21.7 GB VRAM. The author corrected an earlier comparison: llama.cpp was 1.4x faster than Ollama, not 2.4x, because Ollama had overflowed to the CPU in the first test. No speculative decoding; only generation speed was measured.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.6-35B-A3B at Q3_K_XL generating 37.9 tok/s on a single Tesla P40 24GB under WSL2 with driver 582.70 in the MCDM driver model.
Setup is llama.cpp built with CUDA arch 61, GGML_CUDA_FORCE_MMQ=ON and GGML_CUDA_F16=OFF; llama-bench pp512 measures 982 tok/s prefill.
With MTP speculative decoding at --spec-draft-n-max 2 generation rises to 51.6 tok/s (n-max 3: 48.3, n-max 4: 41.9). Prefill falls to about 790 tok/s at 6K tokens, 422-609 at 25K and about 216 at 78K. With MTP n-max 2, 4096 context with f16 KV uses 21710 MiB at 51.45 tok/s, 32768 with q8_0 KV uses 22086 MiB at 50.79 tok/s, and 131072 with q8_0 KV uses 22782 MiB at 50.54 tok/s.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User benchmarks Qwen3.6-35B-A3B at 76.0 t/s single-stream decode and 1,225 t/s prefill on one Intel Arc Pro B70 32GB.
Setup is llama.cpp with the Vulkan backend, Q4_K_M quant and q4_0 KV cache, no speculative decoding. The host has 32 GB RAM.
With --ubatch-size 2048 prefill reaches 1,824 t/s at pp2048 and 8-stream aggregate decode is 624 t/s. The SYCL backend gives 77.3 t/s decode and 979 t/s prefill at pp2048. A dense Qwen3.6-27B Q4_K_XL comparison runs at 20 t/s single stream.