Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8 27B at 98.03 t/s generation and 2829.27 t/s prompt eval on an RTX 5090.
Setup is llama.cpp with UD-Q6_K_XL GGUF, 48000 context, draft-mtp speculative decoding (acceptance 0.47494, mean len 2.42).
User notes only 48k context fits on the RTX 5090 with this quant.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8 Flash-Next at 57 t/s decode and 1239 tok/s encode on Strix Halo 128GB.
Setup uses MTP speculative decoding; the run processed 6204 chunks in 119.0 s.
User says the prefill speed is twice that of the fastest Strix Halo-specific llama.cpp fork they have used, and greater still versus mainline llama.cpp.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User benchmarks Qwen3.8 27B at 25.6 t/s generation and 512.14 t/s prompt processing on an Intel Arc Pro B70 32GB.
Setup is llama.cpp with Q4_K_M weights on Windows 11, comparing SYCL and Vulkan backends; the reported figures are the Vulkan llama-bench results (tg128 25.60 t/s, pp128 512.14 t/s).
Vulkan was roughly 3x faster at prompt processing and about 25% faster at generation than SYCL in llama-bench. In a real long business plan generation, Vulkan gave 517.15 t/s prompt processing and 16.17 t/s generation versus SYCL's 238.78 t/s and 17.39 t/s, but the user notes the Vulkan run processed 11,359 prompt tokens against 1,842 on SYCL, so those chat numbers are not apples to apples. The user also finds SYCL 10% to 35% faster on Linux.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
coding
User reports Qwen3.8-27B at 140.5 tok/s median on 2x RTX 5070 Ti 16GB, up from 69.7 tok/s without speculation.
Setup is llama.cpp b10990 with UD-Q4_K_XL GGUF, 32768 context, F16 KV cache, tensor split 1:1, and ungated MTP n-max 6.
Combined generation rate across three workloads is 126.74 tok/s (1.82x), aggregate acceptance 0.440, peak GPU memory 20.20 GiB, mean power 482 W. The four-request MTP result is 231.0 aggregate tok/s.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
agenticlong-contextcoding
User reports Qwen3.8 Flash at 429.68 t/s prefill and 23.74 t/s decode on an RX 7900 XTX 24GB plus Strix Halo 128GB over OCuLink.
Setup is a custom llama.cpp runtime with IU4 GGUF weights, F16 KV cache, MTP depth 3, batch/microbatch 1536, eight CPU threads and one inference slot, targeting 262,144 tokens of context.
The full-context request processed 261,888 prompt tokens in 609.49 seconds and generated 128 tokens in 5.35 seconds. A HermesAgent-20 agent workload with prompt caching on reports 590.24 t/s prefill and 50.00 t/s decode across 120 requests, with a decode range of 42.07-56.50 t/s for responses of at least 100 tokens. The user notes the approximately 50 t/s decode is an agent-workload result, not a filled-256K measurement.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8-Flash-Next at 20.44 t/s on an RTX 5070 Ti 16GB.
Setup is llama.cpp (unsloth build) with UD-IQ3_XXS GGUF and Q4_0 KV cache, 262144 context, 49 layers offloaded, CPU MoE, and ngram-mod speculative decoding.
Prompt processing measured at 129.29 t/s over 2409 tokens.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
visionlong-context
User reports Qwen3.8 27B running on an RTX 5060 Ti 16GB with MTP speculative decoding, reaching up to 50 t/s in the first half of a 100k context and dropping to about 25-30 t/s at the end.
Setup is a modified llama.cpp fork with adaptive KV cache streaming, IQ3_XXS GGUF weights, q8_0 K cache and q4_0 V cache, a 2100 MiB KV stream buffer, and a BF16 vision head.
The user notes the streaming fork only works on NVIDIA hardware and that the speed depends on context size versus KV stream buffer size.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
codinglong-contextagentic
User reports Qwen3.8 27B abliterated at 104.7 tok/s decode and 2,030 tok/s prefill on a 15,694-token prompt on an RTX 5070 Ti.
Setup is a patched llama.cpp build with IQ3_S GGUF, Q4_0 KV cache, 131,072-token context, and embedded MTP speculative decoding with up to 3 draft tokens.
At 128K a 128,794-token prompt prefilled at 1,576 tok/s and decoded at 91.3 tok/s; short-context decode reached 130.0-131.6 tok/s. The patch is not upstream and was written only for this model on this GPU.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
agentictool-uselong-contextcoding
User reports Qwen3.8-Flash-Next at ~34 tok/s single-stream decode on a single NVIDIA DGX Spark (GB10, 128 GB unified pool).
Setup is vLLM v0.30.0 with NVIDIA's NVFP4 checkpoint, hybrid fp8 side layers, MTP=2 speculative decoding, prefix caching, deterministic top-k, and YaRN 500k context.
Prefill is ~2,700-4,000 tok/s (8k-32k prompts) with a ~505-520k-token KV pool. The repo patches vLLM to serve the 48 GiB n-gram PLE table from NVMe via mmap, fixing prefix caching and non-deterministic top-k on GB10. A separate concurrency sweep on the same hardware reports 266.8 aggregate tok/s at 48 concurrent streams.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
codingmathlong-context
User reports Qwen3.8-Flash-Next at 24.5 tok/s decode on code without MTP on a single DGX Spark (GB10, 121 GB unified).
Setup is llama.cpp (danielhanchen qwen4exp/mtp branch) with UD-Q4_K_XL GGUF weights (103.7 GiB), 131,072 context, flash attention on and CUDA graphs active, with about 95 GB used serving.
With the Q8_0 shared MTP draft at --spec-draft-n-max 4, decode reaches 47.1 tok/s on code, 36.9 on thinking and 24.2 on prose. A 77,527-token voxel-pagoda run finished in 51 min at 25.3 tok/s end to end.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
agentictool-uselong-context
User reports Qwen3.8-27B at about 65-75 tok/s generation at 8K context on one Mac Studio M3 Ultra 512GB (one of four run as independent replicas).
Setup is omlx 0.6.1 with an oQ4e MLX quant, MTP speculative decoding with 3 draft tokens, and a 262144-token context cap. Prefill measured at 420-442 tok/s; batch-8 throughput reached 274 tok/s.
MTP draft depth 3 was the peak (65-75 tok/s vs 50.0 at depth 1 and 65.6 at depth 4). oQ8e was tied on quality but slower at 43 tok/s. A Mac mini M4 Pro 64GB reached 20.8 tok/s and OOM'd around 168k context. An RTX 5090 reference run under llama.cpp with NVFP4+MTP hit ~145 tok/s.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports approximately 15 tokens/s generation speed running Qwen3.8-27B on a DGX Spark (GB10) with llama.cpp.
Setup is llama.cpp with UD-Q4_K_XL GGUF quant and q8_0 KV cache, using MTP speculative decoding with draft-mtp and spec-draft-n-max 2, batch 4096, ubatch 2048, flash attention on, 99 GPU layers, 8 threads.
User notes that NVIDIA forum users reported similar performance with vLLM, but an official SGLang post reported 38 tokens/s decode throughput on DGX Spark, and asks whether a configuration or optimization is missing or whether the model/platform is not yet fully optimized in llama.cpp.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
agenticlong-context
User reports Qwen3.8 27B at 18 t/s generation and 150 t/s prompt processing on a 2007 Dell Precision T5400 with dual RX 6700 XT / RX 6700 (22GB total VRAM).
Setup is llama.cpp with UD-IQ3_XXS quant at 144k context, using MTP q4_0 speculative decoding, on 24GB DDR2 and dual Xeon X5460.
User compares five systems and argues older dual-Xeon dual-GPU setups beat a 2025 HP Omen with RTX 5070 in context length and speed, concluding DDR5 is not worth the cost for agentic tasks.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8-27B at 162.9 tok/s decode on an RTX 3090 with native DFlash2 speculative decoding in ExLlamaV3.
Setup is ExLlamaV3 (community fork, pin 355c6ee) with EXL3 4.00bpw target and DFlash2 draft, cq3 KV cache, at the full 262,144-token context.
Acceptance length 5.66; autoregressive decode 42.8 tok/s and MTP head 116.3 tok/s for comparison. At 150k-token depth: 25.3 tok/s decode, 594 tok/s prefill, peak 23.13 GB.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
codingtool-use
User reports Qwen3.8-27B at 47.5 tok/s on a DGX Spark (GB10).
Setup is the exllamav3 fork with EXL3 3.5bpw weights, nvfp4 KV cache, and DFlash2 speculative decoding (EXL3 5.0bpw draft, 1.4 GB).
The 47.5 tok/s figure is HumanEval-class decode at T=0.6 with acceptance 4.43 tokens/step; the kit also cites ~30 tok/s with MTP and ~34.5 tok/s with DFlash2 at 24 GB, and ~16.7 tok/s aggregate across 8 concurrent requests.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
codingagenticlong-context
User reports Qwen3.8 27B at 35.6 t/s decode on an Intel Arc Pro B70 (32 GB) with Vulkan, dropping to 4.5 t/s at 64k context.
Setup is llama.cpp b10883 with Q4_K_M and q8_0 KV cache, single GPU, full offload, MTP3 draft (Q8_0, n-max 3, p-min 0.10).
Decode collapses ~8x from 1k to 64k on Vulkan while SYCL degrades only ~2.4x (38.5 to 15.8 t/s); prefill falls 562 to 101 t/s on Vulkan. An f16-KV control collapses identically, and MTP gain goes negative at depth on Vulkan. A third-party RX 7900 XTX run holds ~58-60 t/s at 28-30k depth.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
coding
User compares Qwen3.8 27B and Qwen3.8 Flash-Next on a coding task using dual RTX 5090 GPUs. Qwen3.8 27B completed the task in 5-10 minutes with infill ~2000-3000 t/s and output 100-150 t/s. Flash-Next took 2.5 hours with infill ~1600-2300 t/s and output 60-100 t/s, and continued running tasks for ~2 hours after the last code modification. User asks why Flash-Next is slower and what settings would speed it up.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
visionagenticlong-context
User reports Qwen3.8 27B Swift-Genesis at 76 t/s in their benchmark on 2x RTX 5060 Ti 16GB, with a range of 40-113 t/s and 45-65 t/s under normal agentic work.
Setup is llama.cpp with GGUF weights at 262k context, MTP4 speculative decoding, and vision offloaded to system RAM.
The user notes the model fits under 17GB with vision, allowing full 262k context on 32GB VRAM, and asks what the tradeoff of this model is compared to other Swift NVFP4 models.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8-Flash-Next 177B NVFP4 at 9.06 tok/s decode on a single RTX 5060 Ti 16GB with 32GB DDR5, using a custom SSD-streaming engine that keeps 99 GiB of the 119 GiB model on NVMe.
Setup streams routed experts and a 50.7 GiB n-gram table from a Gen5 NVMe SSD on 8 read threads, with about 270 MiB read per token and roughly 75% of expert lookups hitting VRAM or RAM. Prefill is 49.2 tok/s on a 5.5k-token prompt.
Best decode turn reached 10.4 tok/s; llama.cpp on the same machine averaged 4.9 tok/s. The same engine runs Qwen3.6-35B-A3B NVFP4 at 47.3 tok/s decode and 591 tok/s prefill. v1 is limited to RTX 50-series on Windows 11 and WSL2 with greedy decoding only.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
agenticlong-contextvisioncoding
User reports Qwen3.8-Flash-Next (125B MoE, ~6B active) at 44.9 tok/s decode after a short question and 1217 t/s prefill on a 95.6K-token prompt, on a single Ryzen AI Max+ 395 (Strix Halo, 128 GB, 96 GB GPU memory in BIOS) running Windows.
Setup is a patched llama.cpp runtime (Strix Llama 0.3.0) with Unsloth's UD-IQ4_XS GGUF quant (94 GB), 256K context, MTP speculative decoding on (~2.9 tokens per pass), and an OpenAI-compatible API.
Decode drops to 35.8 tok/s at 86K context; with MTP off it is ~26 tok/s after a short question and ~23 tok/s at 86K context. Three and four concurrent conversations reach 58.8 and 62.6 tok/s combined. Decode is memory-bandwidth-bound (~5.8 GB of weights read per token at ~215 GB/s).