Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
codingagentic
User benchmarks Qwen3.8-27B on an Apple M2 Max 96GB, with MTPLX reaching 20-22 t/s decode and 109 t/s prefill.
Setup is 8-bit quants in GGUF and MLX with an unquantised KV cache at 262k context, across llama.cpp baseline, llama.cpp+MTP, MTPLX, oMLX, mlx-dspark (DFlash2/DSpark) and vllm-mlx, on an agentic coding benchmark with 4 phases.
MTPLX xhigh is worth the extra time, while oMLX xhigh is not. vllm-mlx leaks chain-of-thought into the output, and mlx-dspark DFlash2 xhigh thought 226K tokens and delivered nothing.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports a speed gap between LM Studio at 30-40 tok/s and raw llama-server at 50-55 tok/s on the same model and settings.
The model is a Q6_K quant.
User also asks about reasoning effort dropdown differences between lmstudio-community and Unsloth GGUFs.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports a 4070 Super run that was slow because only 38 of 66 layers were on GPU, with the rest running from system RAM.
The same model achieved 81.5 tok/s on a 5090.
The user introduces a tool called Picchio for diagnosing layer placement and speeds.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8:27b at 33.91 t/s on 2x RTX PRO 4000s and an RTX PRO 2000, up from 12.2 t/s.
Splitting the GPUs across VMs produced the gain.
User also reports muse-glimmer:30B at 22.61 t/s on the same setup, up from 14.3 t/s.
User asks for real-world sustained t/s on an R9700 with Qwen3.8-27B Q4 at 64K context and above.
User cites an AMD blog figure of 51.8 t/s and 5090 benchmarks showing degradation from 75 t/s at 4K to about 26 t/s at 64K.
User also asks about MTP speculative decoding stability.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8-27B at 130 t/s decode and 4000 t/s prefill on an RTX 5000 Pro with 48 GB VRAM, at 5 concurrencies.
Setup is SGLang with prefix offload to storage.
The user compares the result to a Claude subscription and notes the card's price has doubled since purchase.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8-27B-FP8 at roughly 50-56 t/s generation on two Radeon AI PRO R9700 GPUs in TP=2 on PCIe Gen5 x8, with an AMD Ryzen 5 7400 CPU and 64 GB of DDR5-6000.
Setup uses MTP3 speculative decoding, with accepted throughput of roughly 30-37 t/s.
User asks whether these numbers are reasonable and where to look for bottlenecks.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User benchmarks DFlash2 quants and MTP, with the best speed at 154.08 t/s for Q4 at n-max 7.
The best balance is DFlash2 Q2 at n-max 5, reaching 137.19 t/s with 121k context. MTP tops out at 128.32 t/s at n-max 6.
Context size is set to 113920 in the config.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports 12.55 t/s for a 256 token response and 11.65 t/s for a 512 token response on an RPC split across an RTX 4070 Ti and an M5 MacBook Air with 16 GB unified memory.
MTP off gives 9.72 t/s.
The use case is agentic coding.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User benchmarks dual R9700 GPUs with tensor parallelism, comparing PCIe gen4 x16+x4 against gen5 x8/x8.
Setup is vLLM with stilldeadcode/vllm-radiance:0.7.4 on a 9950X with 64GB DDR5 6000.
The gen5 x8/x8 configuration shows 18-32% improvements, with results given for concurrency 1, 2, and 4.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
tool-useagentic
User reports Qwen 3.8 27B at 86 tok/s on a single RTX 3090 with vision enabled at 150k context.
The model is locally hosted and run through the DeepSeek harness (dsh).
The user also mentions writing a Gmail plugin.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
agentic
User reports a hybrid attention model (Gated DeltaNet plus sparse QSA) with a 51B n-gram embedding table, which llama-bench reports as 176.94B all-in.
Setup is a 3-part GGUF, 87 GiB on disk, built from llama.cpp PR #27742 with a crash fix, on the Vulkan (RADV) backend.
The model uses about 91 GB resident at 131k context and loads in about 45s. The KV cache must stay F16, as quantized KV asserts.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
coding
User reports Qwen3.8 27B dense at 10 t/s on Strix Halo 128GB.
User considers eGPU options with 32GB and 48GB of VRAM for faster dense model inference and larger models.
User mentions 0731 (DeepSeek V4 Flash) and dflash speculation.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
coding
User compares Qwen 3.8 27B community quants on an RTX Pro 6000 96GB, with atomic ad-q6_k averaging 114.17 t/s as the best.
The quants tested are atomic ad-q6_k, unsloth ud-q6_k_l and bartowski q6_k.
The user also compares the results against a Claude Opus 4.6 subscription, with prompts generating self-playing 3D HTML games.
User reports Qwen3.8 27B at 972 t/s TG and 5,680 t/s PP on 4x MI100.
Setup is a custom INT8 vLLM fork with AITER and a GPTQ INT8 quant.
User claims near-reference quality via KLD diagnostics.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports 30 t/s decode and 120 t/s prefill on a dual GPU setup.
Setup is llama.cpp with a PR build.
User asks for optimization advice.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8-27B q8 GGUF at 20 t/s tg and 1000 pp on a Threadripper rig with 4x RTX 5060 Ti 16GB, VRAM only.
User also reports DeepSeek V4 Flash 0731 GGUF at 11 t/s tg and 200 pp with RAM offload.
User asks about vLLM tensor parallelism, NVFP4, a CPU upgrade, and other models.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports a 85 GB GGUF quant of Qwen 3.8 Flash Next running on a 64 GB MacBook with the ngram table offloaded to SSD, at 517.9 t/s prefill and 36 t/s decode.
The user notes the quant quality is far from perfect and better versions are planned.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
creative-writingcoding
User reports about 20 t/s on an RTX 3090 with 128 GB of DDR5, running Qwen3.8-Flash-Next.
Setup is a custom llama.cpp build from PR #27742, using the new qwen4exp architecture.
The figure is approximate and likely generation speed, and the run is a showcase of a generated 3D voxel demo rather than a formal benchmark.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen 3.8 Flash-Next at about 25 t/s on an RTX 5090 with 64 GB of RAM.
Setup is the Q4_K_XL quant, with the n-gram table unloaded to an SSD to fit in memory.
User advises that the n-gram table can be offloaded to an SSD with minimal speed cost.