Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports running Qwen3.8 Flash-Next at 40 t/s on an M3 Max.
User says prefill feels stagnant and tried approximating the KV cache with QFN, getting about 2x prefill (400 t/s to ~850 t/s) but needle test accuracy suffered.
User asks whether anyone is working on improving prefill speed or if it is a hardware limitation.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
rplong-context
User benchmarks Qwen3.8-Flash-Next on a dual AMD Radeon AI PRO R9700 32GB setup (Lenovo P620, Threadripper Pro 3945WX, 128GB DDR4).
Setup is llama.cpp with Unsloth UD-Q3_K_XL GGUF, -ncmoe 5, -ngl 49, tensor-split 1.10,0.90, and 3 MTP draft tokens.
llama.cpp achieves 17.6 t/s generation and 616.5 t/s prompt processing, compared to 147.2 t/s and 6618.1 t/s on radiance 1.4.0, and 79.1 t/s and 1455.4 t/s on Strata 0.1.42. User notes radiance's prompt processing is unreasonably fast and adds a prefill slope metric. All three are considered equivalent in output quality for the user's purposes.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8-Flash-Next 125B-A6B at ~100 tok/s generation and ~3.4k tok/s prefill on a 15k prompt on 2x RTX 5070 Ti 16GB.
Setup is the capy-flash engine with NVFP4 weights, experts offloaded to RAM, tensor parallel across both GPUs, on a Ryzen 9 9900X with 96 GB DDR5.
User compares against Strata, which gets 78 tok/s generation and 2.7k tok/s prefill with the same NVFP4 weights; Strata's 3-bit IQ3_S build is still faster at generation. User also notes a tensor-parallel finding: writing partial sums in FP16 instead of BF16 restores single-GPU numerical parity at no real speed cost.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
tool-usecodinglong-context
User compares Ninfer6000, vLLM and llama.cpp running Qwen3.8-Flash-Next on an RTX Pro 6000 Blackwell 96GB. Ninfer6000 reaches 232 t/s decode at ~1k prompt versus 159 t/s for vLLM NVFP4 and 138 t/s for llama.cpp Q4_K_XL, with a mean decode of 238 t/s across prompts up to 114k.
Setup is Ninfer6000 with an NVFP4 conversion, vLLM with nvidia/Qwen3.8-Flash-Next-NVFP4 and llama.cpp with unsloth UD-Q4_K_XL, all on one RTX Pro 6000 Blackwell at 600W with a Ryzen 5 5600X and 96GB DDR4-3000.
Prefill trails vLLM (4,315 t/s vs 6,944 t/s at ~1k) but beats llama.cpp. Accuracy benchmarks are close across all three, with Ninfer6000 ahead on BFCL v4 tool calling (76.67 vs 74.56 and 73.00). User reports a crippling tool-calling formatting issue with Ninfer6000 in real-world use in Open-WebUI and Cline, and will keep using vLLM until it is resolved.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Swift-1.5-Qwen3.8-27B decoding at roughly 50 tok/s (44 to 52) on an RTX 3060.
Setup is the NInfer RTX 3060 port with an IQ3_XXS quant, 131K context and a 46,400-token KV cache.
The user says it passed a one-liner prompt on the first try.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports around 100 t/s running Qwen3.8 Flash-Next with QFN on Strata using 3x RTX 3090 at PCIe 4.0 x16 each.
Setup is Strata with QFN quant on three RTX 3090s.
User asks for advice on adding a fourth RTX 3090, considering PCIe bifurcation or a motherboard upgrade, and is concerned about x8 performance impact on Strata.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8-27B at 32 tok/s on a single RTX 3090 at 300k context.
Setup uses Strata on the RTX 3090; a Titan RTX 24GB sits idle in the same machine, with 48GB of free system RAM.
User asks whether Strata's multi-GPU support would hurt prefill speed when adding the second card without NVLink.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
codingagentic
User reports Qwen3.8 Flash-Next at 75 tok/s generation and 1200 tok/s prefill at 128k context on an RTX 5060 Ti 16GB.
Setup is Basalt (a Strata fork for Blackwell) with a Q2 quant and Q4 KV cache, on a system with 32GB RAM and about 5GB left for the OS.
User says the model is used as a dedicated coding agent and had to add a build flag to allocate more RAM.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports 176 t/s decode and 6,450 t/s prefill with Qwen3.8 Flash-Next Q4 on 2x R9700 with 64GB DDR5.
Setup uses Deadcode's Radiance inference engine, optimized for 2x R9700, with a Q4 quant. Benchmark tool is BetterBench.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8-Flash-Next at 90.2 t/s decode and 1375 t/s prefill on a Strix Halo 128GB at 87k tokens of context.
Setup is the strixite engine with cold, cache-defeated runs; the model is a hybrid recurrent/attention architecture whose KV cache is about 20 GiB at full depth.
Decode decays from 94.6 t/s at 21k to 90.0 t/s at 261k, a 10% cost across 50x the memory. The user retracted earlier 765k t/s prefill figures as prefix-cache hits and reports the box runs 176.9B parameters at full working depth.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8-Flash-Next at 4.4 t/s decode and ~2.9 t/s prefill on a RedMagic 10S Pro (Snapdragon 8 Elite, 24 GB RAM, 1 TB UFS).
Setup is llama.cpp with a 66 GB mixed-quant GGUF (2 shards), CPU only, no NPU or GPU path, pinned to one Oryon cluster with --cpu-mask 0x3f --cpu-strict 1 at -t 6. The weights are memory-mapped and demand-paged since they do not fit in RAM.
Speculative decoding with the MTP head was 4x slower (3.94 to 0.95 t/s) because the 2.8 GB sidecar pushed the expert working set past the page-cache cliff. Cluster pinning gave +26% and a NEON BF16 path gave +40% decode and +55% prefill. The user released the project as MonsterQwenDroid 0.1.0-alpha under MIT.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User compares Qwen3.8-27B at 47.9 tok/s on an RX 7900 XTX 24GB against Qwen3.8-Flash-Next at 36.8 tok/s on an MI50 32GB.
Setup is llama.cpp with UD-Q4_K_XL and F16 KV cache for the 27B; Strata with IQ3_S and fp16 KV cache for Flash-Next. Both ran through an OpenAI-compatible API with temp 1.0, top_p 0.95, top_k 20.
Speed figures cover AIME26 and LiveCodeBench generation (1.34M output tokens over 7.8 h for the 27B). The 27B scored 145/166 on the eval suite versus 151/166 for Flash-Next, a difference the user notes is not statistically significant (p = 0.18).
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports 136.7 t/s generation.
Source evidence (verbatim):
Source: https://kgptalkie.com/tutorials/llm-benchmarking/qwen-3-8-27b-llama-cpp-speed-settings
# Qwen 3.8 27B Speed Settings on llama.cpp
Hardware: NVIDIA RTX 5090 32GB. Engine: llama.cpp b10448. Model: Qwen 3.8 27B Q4_K_M. Baseline with MTP off (no speculative decoding, flash attention on, f16 KV cache, 17966 MB VRAM): decode 73.6 tok/s, prompt 3036 tok/s (the article does not explicitly tie the prompt figure to this run). Best MTP run (speculative decoding on: --spec-type draft-mtp, --spec-draft-n-max 2, q4_0 K and V cache, flash attention on, context 32768, -np 1, -ngl 999): decode 136.7 tok/s, 17412 MB VRAM. Best draft-depth-only run n=3 with f16 cache: 133.6 tok/s, reverse-order rerun 129.7 tok/s. MTP reduces prompt processing by roughly 5 to 8 percent.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8-Flash-Next at 121 prompt t/s and 12 generation t/s on an M1 Max 64GB.
Setup is MLX with oQ4e quant, 6-bit KV cache, 50% expert offload and MTP on.
A later fp16 build of the same model gave OOM errors and, at 20% expert offload, only 90 prompt t/s and 6 generation t/s; the user asks whether the model or the settings are at fault.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
agenticvisioncreative-writing
User reports Qwen3.8 Flash-Next at roughly 45-53 output tokens per second on an RTX 4070 Ti SUPER 16GB.
Setup is Strata with Q2 quantization and a 128K context configuration, running on a 16GB GPU with 64GB DDR4 DRAM.
The complete workflow used multiple computers; the LLM ran on the 16GB GPU while modelling, image generation and other services were handled elsewhere. The user notes Windows was using 15 to 20 GB of the system RAM.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8-Flash-Next GSQ-RCO IQ3_S at a median 161.3 tok/s generation on an RTX 5090 with 128GB DDR5-6000, measured on three 240,000-token inputs.
Setup is Strata 0.1.43.1 with INT8 KV cache, 32,768 resident KV cells, spec 4, draft floor 0.50, 12 pool workers, a 55 GiB resident-expert budget and a 700 MiB VRAM reserve, under a 262,144-token context ceiling.
Median prompt processing was 5537.8 tok/s, median first visible answer 43.475 s and median total request time 45.071 s; the generation range was 159.6-203.4 tok/s. The same nine requests at a 131K ceiling gave 173.5, 181.2 and 183.0 tok/s at 4,096, 32,768 and 128,000 input tokens, and the 262K ceiling gave 171.2, 175.0 and 179.2 tok/s. Unsloth UD-IQ4_XS reached a 121.2 tok/s short-task median after tuning, against 105.0 before, with 3/3 exact recall at 100,510 and 128,510 tokens. The user notes the RAM upgrade comparison is not a RAM-only A/B test and that no statistical significance was established.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8 27B running at 33-35 t/s on an M6 32GB Mac using the Inco Splash inference engine.
Other engines topped out at 9.6 t/s on the same machine.
The user is considering an M5 Pro for its 1.7x memory bandwidth and hopes Inco AI adds support for more models like Flash-Next.
2× 2x R9700 AI PRO 32GB · vLLM - Radiance · 262,144 ctx
generation:
150.0 tokens/s
prompt processing (prefill):
2000.0 tokens/s
quant:
4bit
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
coding
User compares Qwen3.8 Flash-Next 4-bit against Qwen3.8 27B FP8 on a coding task, both run on 2x R9700 AI PRO 32GB (64GB total VRAM) with vLLM - Radiance.
Flash-Next averaged ~2000 prefill and ~150 generation t/s at 262k context and finished in about 30 minutes, but produced a messy, unusable WebUI. The 27B FP8 averaged ~1500 prefill and ~100 generation t/s and took about 60 minutes, but produced a mostly working result.
User notes the speeds are real work averages, not benchmarks, and concludes a dense model outperforms the MoE on the same hardware.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8 Flash at about 25 t/s generation and 82 t/s prompt processing on a GTX 1080 plus GTX 1060 with 64GB DDR5.
Setup is Strata with an IQ3_XXS GGUF at 131k context.
The user says they did not expect such good results for hardware not made for this.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
agentic
User reports Qwen3.8 27B at just over 50 t/s decode and just under 900 t/s prompt processing on a single 16GB RTX 5060 Ti.
Setup is llama.cpp with an IQ3_XXS GGUF pruned to 9.3 GB via vocabulary pruning, Q8_0 KV cache, MTP speculative decoding, and everything in VRAM.
The user gets just over 100k context with Q8 KV cache and a little over 170k with Q4 KV cache. They previously ran Gemma 4 26B A4B QAT at 90 t/s generation and just under 3000 t/s prompt processing, and Qwen 3.6 35B A3B, but found their intelligence insufficient for agentic use cases.