Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8-27B at 33.41 tok/s generation and 826.03 tok/s prefill on a single Tesla V100 32GB.
Setup is llama.cpp with UD-Q4_K_XL GGUF at 131,072 context; generation falls to 27.95 tok/s at 32k depth and 15.84 tok/s at 131k depth. Peak VRAM was 25,264 MiB of 32,768 MiB.
KV cache quantization (-ctk q8_0 -ctv q8_0) silently fell back to CPU with 0% GPU utilization; CUDA 13 dropped Volta support.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports running Qwen3.8-Flash-Next on an RTX 3080 Laptop GPU with 16GB VRAM at roughly 10-20 tok/s generation and an estimated 100-200 tok/s prompt processing, with about 65k context and only 1-2GB system RAM usage.
Setup uses a modified llama.cpp. The prompt processing figure is an estimate based on cloud GPU calculations, not a direct measurement on the laptop.
The user was mid-benchmark when their 180W power adapter cable failed and is asking for donations to replace it, promising to release the technique and source code regardless.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
codingagenticrp
User reports Swift-1.5-Qwen3.8-27b-oQ8e-mtp at 34.6 tok/s generation and around 400 tok/s prompt processing on an Apple M5 Max 64GB.
Setup is oMLX with oQ8e quant, 262k context window, thinking on at xhigh, MTP speculative decoding enabled.
User compares against base Qwen3.8 27B (32.8 tok/s, ~400 tok/s prompt processing) and notes Swift 1.5 generates fewer tokens per run (51k vs 77k average) with a similar quality score (85.8 vs 85.0).
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen 3.8 27B at ~30 t/s decode on an RX 7800 XT 16GB.
Setup is llama.cpp with Vulkan, UD-IQ4_XS GGUF quant, q8_0/q5_1 KV cache, 100096 context, no MTP, single request, full GPU offload.
User shares the setup because many believe Qwen 3.8 27B is infeasible on 16GB VRAM.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
agentictool-use
User reports Qwen3.8 Flash-Next running at approximately 40-50 tok/s on a Ryzen Strix Halo PC with 128GB unified RAM.
Setup is gufo with Q4 weights and 256k context, supporting 2 concurrent sessions.
The user notes the model impressed them with its context understanding and tooling, and that it provides capability comparable to good API models.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
coding
User reports Qwen 3.8 27B at 42 t/s on an AMD RX 9070 XT 16GB.
Setup is llama.cpp with MTP2 speculative decoding and a 64K context, using about 1 GB of working RAM during evaluation.
MTP2 delivered +41% to +53% throughput depending on the quantization, but the user notes the fastest configuration was not necessarily best for reasoning quality.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8-27B at ~18 tokens/s on an NVIDIA A100 80GB, compared with ~62 tokens/s for Qwen3.6-27B on the same hardware.
Setup is vLLM with unquantized (BF16/FP16) weights, same context length and prompt for both models.
User asks whether the 3.4x gap is expected and whether vLLM version, FlashAttention kernels, or Qwen3.8-specific configuration could explain it.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8 27B UD-Q2_K_XL at 0.65 tok/s decode on an Intel Arc B580 12GB.
Setup is the FreeToken-Intel fork with PyTorch XPU plus native SYCL packed matvecs, context 128 and KV cache 256 tokens; TTFT 37.58 s and model load 47.39 s, with 9.47 GB model state.
The run used a fixed 16-token forced decode length and is presented as a development baseline, not a general performance claim; the same table lists Llama 3.2 1B Q4_K_M at 18.29 tok/s, Gemma 4 12B Q4_K_XL at 2.85 tok/s and Ternary-Bonsai 27B PQ2_0 at 2.10 tok/s.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
agentictool-uselong-contextvisioncoding
User reports Qwen3.8-27B at 256K context on an RTX 5060 Ti 16GB, with decode staying around 30-40 tok/s.
Setup is a KVMem-enabled llama.cpp server (v0.14.0) with ISTA IQ3_S quant, MTP, Q8 KV cache, vision on GPU, and a 32K GPU window plus 16K reserved for new tokens; the rest of the KV cache stays in 32GB system RAM.
In a 33-request tool task ending at 262058/262144 tokens, prefill was 437 tok/s first pass and 243 tok/s overall, decode 30 tok/s over the tool rounds and 29 tok/s on the last 512 tokens, with 15.5 GB peak VRAM and 13.1 GB RAM. An IQ4_XS variant with Q5 KV and CPU vision reached 466/255 tok/s prefill and 33/35 tok/s decode. A shorter image and code task ran about 37-41 tok/s decode.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8-27B at 24.8 tok/s decode on an M4 Pro 48GB, with median TTFT 943 ms over three samples of 4096 prompt tokens and 128 generated tokens.
Setup is oMLX 0.6.4 with the oQ4e 4-bit MLX model and Lightning MTP draft depth 3 enabled via the mtp3 profile, at a 200k context window.
Enabling MTP through a profile rather than settings.json raised decode from 10 tok/s to 24.8 tok/s. A separate sweep documents the MTP-on prefill curve at 1.8k-30k prompts and a 5-turn agentic session reaching 157k context with 85% token reuse.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
codingagentic
User reports Qwen3.8 Flash-Next at 150+ t/s single-stream generation on 4x Radeon AI PRO R9700 32GB.
Setup uses the MXFP4-FP8 quant from tcclaviger, tested with agentic coding use-cases in a pi harness.
User also gets 3-5 concurrent streams at ~100 t/s generation each, and prefill of 10k+ t/s. Speed and quality both exceeded expectations.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
agenticcoding
User reports Qwen 3.8 27B at 31-34 t/s generation and around 1,080 t/s prompt processing on a single RTX 3090 24 GB.
Setup is llama.cpp with CUDA, UD-Q4_K_XL quant, 64K context, turbo3 KV cache, 2 parallel slots.
User compares four agent configurations: Qwen alone (1/5 scenes right, 26 min, $0), Sonnet 5.5 planning with Qwen coding (2/5, 85 min, $0.67), Qwen planning with Sonnet 5.5 coding (4/5, 14 min, ~$2.77), and Sonnet 5.5 alone (4/5, 9 min, $1.83). A 3060 12 GB was tried first and gave 4-6 t/s with ~7 GB of weights spilling into system RAM. User notes Qwen is a better planner than coder, that thinking budget had to be capped, and that crash checks reported zero errors while some scenes were visibly wrong. One run per setup, described as a field report rather than a benchmark.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8-Flash-Next at 25.7 t/s aggregate decode on an M3 Ultra 512GB, with throughput collapsing as concurrent long sequences are batched.
Setup is llama.cpp (Unsloth prebuilt b10830) with Q8_0 GGUF, 32k context, flash attention on, and 8 slots. At 32k context, N=1 gives 25.7 t/s aggregate, N=2 gives 8.6 t/s, and N=4 gives 5.5 t/s. Two separate -np 1 processes reach 39.4 t/s aggregate, 4.6x faster than one process batching the same two sequences.
User also sees the same cliff with GLM-5.3-Flash, going from 40 to 10.6 t/s aggregate at N=4 between 256 and 8k context, and uses two -np 1 processes behind a load balancer as a production workaround.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
long-contextvision
User reports Qwen3.8-27B-GSQ-RCO at 44.07 t/s decode on a hardware-modded RTX 2080 Ti 22GB, with 38~44 t/s quoted as the overall range.
Setup is a llama.cpp fork (TurboQuant 4-bit + Adaptive KV Streaming) with IQ3_S weights, 256K context (262,144 tokens), Q8_0 key cache and turbo4 value cache, a 2048 MiB GPU staging pool, and MTP speculative decoding with 2 draft tokens.
Needle tests at 8K/16K/32K gave 41.23, 39.00 and 34.84 t/s decode with 81-83% draft acceptance and 100% recall at 82% depth; a sustained 256K run gave 40.55 t/s at 89.17% acceptance. Peak VRAM was 18,619 MiB and GPU power peaked at 266.4 W.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
codingagenticlong-context
User reports Qwen3.8-27B Flash-Next at 433.19 t/s prompt processing and 27.72 t/s generation on a Tesla V100 32GB, comparing a custom llama.cpp fork against vanilla llama.cpp.
Setup is llama.cpp (v100-optimized fork) with UD-Q5_K_XL GGUF, q8_0 K/V cache, 262144 context, FlashAttention on, MTP n-max=3, 63 GPU layers, 100000 cached tokens plus a 1000-token appended prompt.
The fork gains +43.94% prompt processing at 100k cached + 1k appended (300.94 to 433.19 t/s) and +32.19% generation with the MTP3 decode stack (27.591 to 36.471 t/s), while standard llama-bench tg128 is effectively unchanged (27.89 to 27.72 t/s).
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
codingagenticlong-context
User reports Qwen3.8 Flash at 60-70 (peak 74.1) tok/s peak single-stream decode on one DGX Spark (GB10), with 60 to 70 tok/s on normal requests and 212.2 tok/s aggregate across 8 streams.
Setup is vLLM with W4A16 AutoRound for the MoE experts, FP8 side layers and INT8 lm_head, full 262K context, using the model's own MTP draft head with a draft vocab cut from 248K to 65K ids.
Cold prefill reaches 4,016 tok/s at 16K, 2,426 tok/s at 64K and 2,213 tok/s at 128K, versus 1,171 / 1,071 / 1,065 tok/s for the original recipe. Cached coding prompts start replying in about 0.57 s. Quality scored 93.1% and 93.3% on a fixed 492-question suite over two seeds. The model takes about 71 GiB with a 16 GB KV pool; CJK drafting is slower due to the English/code draft vocab.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports 50 t/s for Qwen 3.8 27B on an RX 9070 XT 16GB.
Setup is llama.cpp (Vulkan build) with a Q6 quant, 131072 context, Q8 K cache and Q4 V cache, and DFlash2 speculative decoding with 7 max drafts.
The rig also has an R9700 AI Pro 32GB and 64GB DDR5 6000; the user asks what speeds and flags others get, especially on Windows/AMD.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User benchmarks Swift 1.5 + HyperQwen at 104.0 tok/s decode on an RTX 3090 24GB.
Setup uses FP8 KV cache and 150k configured context, with speculative decoding via MTP and a Swift-specific 65,536-token draft vocabulary.
Swift 1.5 INT4 heads achieves roughly 37% lower average time per task than the HyperQwen fast quant, averaging 68.2 s and 5,669 output tokens per task versus 108.1 s and 8,985 tokens. The HyperQwen fast quant decodes at 112.1 tok/s, Swift 1.0 at 105.9 tok/s, and Swift 1.5 INT8 heads at 104.0 tok/s. Average task time covers about 630 tasks.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
codingagentic
User reports Qwen3.8-27B at 220.0 tok/s decode with MTP3 structured output on 2x RTX 5070 Ti, matching an RTX 5090's 219.8 tok/s on the same official NVFP4 weights.
Setup is the NInfer tensor-parallel fork v0.2.3 with NVFP4 weights, 7,680-token context, C=1, uncapped clocks, no P2P over PCIe 5.0 x8/x8. Without speculation decode is 70.9 tok/s and prefill 4,956 tok/s (59% of the 5090's 8,340).
Against llama.cpp on the same two cards: cold prefill 4,577 vs 2,197 tok/s, decode at 184k context 122.9 vs 64.0 tok/s, and a 16-turn agent session to ~94k context 46.5 s vs 82.8 s. QUASAR-QAT weights reach 254.5 tok/s decode but are lighter (17.0 vs 20.9 GiB).
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User benchmarks Qwen3.8-27B at 43.643 generation tok/s and 943.278 prompt tok/s on an RTX 4070 Ti 12GB.
Setup is llama.cpp b10448 with UD-IQ2_XXS GGUF, Q8 KV cache, 4096-token context, all 66 layers on GPU, one slot, 256-token outputs.
A second quant UD-Q2_K_XL decoded 38.030 tok/s and 995.095 prompt tok/s with 1,583 MiB more peak VRAM. An IQ2 context ladder gave 41.124 tok/s at 4K, 40.522 at 8K and 39.201 at 16K (12,831 prompt tokens, 11.119 s TTFT). An isolated IQ2 draft-mtp experiment raised throughput 47.284% on prose and 92.651% on Python code, but prose output diverged at token 16, so MTP stays off by default. In a 24-task quality pass, Q2 scored 10/24 and IQ2 9/24 (McNemar p = 1.0).