Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
agenticvisionlong-context
User reports Qwen3.8-Flash-Next at 116 t/s decode and 2,908 t/s prompt read on two RTX 3090s with NVLink.
Setup is a modified Strata build (v0.1.38 plus 29 commits) with IQ3_S GGUF at 262K context, greedy, median of 3, on a Ryzen 9 3950X with 121 GB RAM. The second card acts as a peer expert tier read over NVLink; the pair holds about 19,500 of 24,576 IQ3_S experts with a 0.98 hit rate on real agent chats.
Decode gains from the second card are modest (+10%) because the verify window is latency-bound on the primary. Sampled with official settings it sits around 102-104 t/s, a 1.5K prompt reads at ~1,670 t/s, and switching chats takes 1 s. One stream at a time; answers are word-for-word identical to the one-card run.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
summarizationtool-uselong-context
User reports Qwen3.8 27B at about 15 tok/s on an RTX 5090.
Setup is Ollama with Row-Bot 5.0, running locally with no API keys or cloud.
The model read a year of bills, a tenancy agreement, an insurance policy and a rent increase letter, charted a CSV, quoted lease clauses, saved 8 linked memories to a local knowledge graph, and drafted an email. Some turns took over 2 minutes.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
codinglong-context
User reports Qwen3.8-27B GGUF quants fail to converge on long thinking tasks on 2x RTX 5060 Ti 16GB, while NVFP4 mixed-precision quants converge in ~17K thinking tokens.
Setup is llama.cpp and vLLM with UD-Q6_K_XL GGUF (23.6 GB) and f16 KV cache, tensor-parallel across 2 GPUs. Throughput was healthy at 23-41 t/s on llama.cpp and 22-23 t/s on vLLM.
The failure appears after 30K-38K thinking tokens with no closing token, tail-looping, or engine crash. The same model in NVFP4 (FP8 attention/GDN, FP4 MLP) converges in ~17K tokens on SGLang at 47-50 t/s. User hypothesizes recurrent state error accumulation under uniform quantization.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8-Flash-Next at 31.2 t/s generation and 713.7 t/s prompt processing on an Intel Arc Pro B70 32GB, after overclocking DDR5 memory from 3600MHz to 4800MHz.
Setup is llama.cpp with a customized SYCL backend, Q2_0 GGUF quant, q8_0 KV cache, 262144 context, MTP speculative decoding, and 34 MoE layers offloaded to CPU.
Overclocking gave about 10% higher prefill and 5% higher generation overall; the post compares base and overclocked llama-benchy numbers across context depths.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
coding
User reports Qwen 3.8 27B Q8_K_XL at 48.76 t/s generation and 142.46 t/s prompt processing on dual RTX 3090s.
Setup is llama.cpp via llama-swap with MTP speculative decoding (draft-mtp, n-max 3), 238000 token context, tensor split 50,50 across two cards, batch size 4096.
The figures are from the final turn of a multi-turn agentic coding run (95 prompt tokens, 90 generated, 43473 cached); per-turn generation speeds ranged from 39.58 to 77.38 t/s across the run.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports running Qwen3.8 Flash-Next quantized builds on an RTX 5090 with 64GB of system RAM, fitting Atomic Q4_K_M at 4.27bpw with 31 layers offloaded and Swift IQ4_XS with 32 layers offloaded.
Setup uses partial CPU offload to fit the models in memory; the user notes that offloading more layers would cause a significant slowdown.
User reports around 40 tok/s when staying in the 34-30 layer offload range, and asks whether a larger quant of a 27B model would be more capable than the cut-down Flash-Next builds.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8 27B at 40-70 tokens per second on an Intel Arc Pro B70.
Setup is vLLM with a Helm chart applying patches from the Intel Arc Pro B70 Inference Cookbook, at 128k context.
The user gives the speed as a range depending on workload.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
codingagentic
User reports Qwen3.8 Flash-Next at 150 prompt t/s and 13 decode t/s on a GMKTek Evo-X2 96GB (Strix Halo).
Setup is Q4_XS at 4K context; the user notes MTP and other optimizations are not yet enabled.
User asks whether adding a second RTX 3080 Ti to a gaming desktop (Ryzen 5700X3D, 3080 Ti 12GB) for a local swarm of Qwen 3.8 27B workers would speed up their Kimi Code CLI workflow.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
codinglong-context
User benchmarks Qwen3.8-27B at 46.72 t/s decode and 1021.1 t/s prefill at 1999 tokens of context on an unspecified NVIDIA GPU.
Setup is llama.cpp with IQ4_KS_KT GGUF weights and q4_0 KV cache, 99 GPU layers, 110000 context size, single request, no speculative decoding.
The post presents a context-length sweep from 2K to 107K tokens, with decode falling from 46.72 to 23.37 t/s and prefill from 1021.1 to 624.7 t/s. A second model with MTP enabled reaches 64.97 to 97.63 t/s decode, and a third configuration with KV cache 5_0/4_1 shows a VRAM cliff around 107K context.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8 Flash-Next 125B-A6B abliterated at around 28 t/s generation and 400-600 pp on an RTX 5070 Ti 16GB plus RTX A4000 16GB with 128GB system RAM.
Setup is stock llama.cpp with Q4 GGUF weights, q8_0 KV cache, 32768 context, 48 GPU layers, tensor-split 64,18 and ncmoe 35, so part of the model runs from system RAM.
User is happy with the speed but asks whether the Strata engine can use multiple GPUs and whether it would improve on the 60 t/s others report on 12GB laptops.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
codingtool-useagenticlong-contextvision
User reports Qwen3.8-27B at 92.9 tok/s decode on a single RTX 5090 with SparkInfer and no speculation.
Setup is SparkInfer v0.5.5 with an NVFP4 checkpoint (17.92 GB, two shards) and int8 KV cache at the full 262,144-token context, which costs 27.9 GB.
The same weights reach 85.8 tok/s on SGLang with FP8 KV, 161.7 tok/s on SGLang with the DSpark v2 drafter, and 264.8 tok/s overall (up to 420.2 on code) in SparkInfer's bench harness with DSpark. Under concurrency SparkInfer peaks at 344.3 tok/s aggregate at 8 requests, while vLLM reaches 546.6 tok/s at 16. A configured 32,022-token prefix cache cuts a repeated request from 5.75 s to 0.30 s.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8-27B at 96.7 t/s decode on two V100s with PXA v3.
Setup is PXA engine with PXQN2 quant, 4,096-token prompt, speculative decoding on.
Same model runs 85.8 t/s on one V100, 60.1 t/s on two P100s and 48.1 t/s on one P100. Gemma 4 26B-A4B reaches 168.6 t/s on one V100 and 71.1 t/s on one P100. A Flash-Next MoE runs 36.3 t/s on four P100s and 20.4 t/s on one P100 with system RAM.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
coding
User reports Qwen3.8 27B at 30-40 t/s decode on a 32GB VRAM machine.
Setup is Unsloth's Q6 GGUF with OpenCode and a 95k context window.
User finds small tasks fine but says the model gets stuck in thinking loops on nuanced tasks, consuming the whole context window. They question the gap between public sentiment and real-world effectiveness.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User benchmarks Swift-Qwen3.8-27B-Uncensored-MTP at 20.21 t/s generation and 176.86 t/s prompt processing on dual AMD Radeon Instinct MI50 (32GB combined VRAM) under ROCm 10.2, compared against 18.35 t/s and 149.30 t/s under Vulkan Mesa 26.2.
Setup is llama.cpp (llama-b11401 for ROCm, llama-b11382 for Vulkan) with Q6_K GGUF weights, both GPUs power limited to 145 watts.
ROCm shows +10.14% generation and +18.46% prefill over Vulkan for this model. Across the full nine-model sweep, ROCm gains up to +62.74% on generation but regresses on MoE prefill by up to -15.98%, with high variance (e.g. ±102.56 on Qwen3.5MoE Q4_K prefill).
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8 Swift-1.5 at 113.7 tok/s decode and 3191 tok/s prefill on an M5 Ultra 96GB Mac Studio.
Setup is mlx-serve 26.10.1 with a 4.7bpw MLX quantization, 107GB download, text-only, tested to 179,200 tokens of context.
Decode falls to 81.1 tok/s after a 95k prompt, where prefill is 2928 tok/s. User compares against a llama.cpp IQ3_XXS build of the same model, which reaches 62.7 tok/s decode after 4k and 1427 tok/s prefill at 25k. Top-1 agreement with Swift BF16 is 91.0% across 680 held-out positions, against 84.1% for the llama.cpp build.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8 Flash-Next at about 50 tok/s decode with MTP on a Strix Halo 128GB box, even at 116k context.
Setup is a forked Gufo engine with the Unsloth UD-IQ4_XS quant (about 89GB) and an MTP sidecar, running in the 120W performance profile.
Prefill is 1500+ tok/s from about 3k tokens up, peaking around 1570; on HumanEval prompts decode is about 76 tok/s with MTP versus 28 without. The balanced profile is around 10% less prefill, and very short prompts are slower (about 1350 at 2k) due to a fixed cost per request.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
codingtool-use
User reports Qwen3.8-Flash-Next at around 70 t/s for a single session on 4x Radeon AI PRO R9700 32GB.
Setup is vLLM with MXFP4-FP8-GPTQ weights, fp8 KV cache, tensor-parallel-size 4, 262144 max model length, and MTP speculative decoding with 3 tokens.
The user is troubleshooting a decode performance cap hit with newer vLLM image tags starting from 29.06.19, and notes the BetterBench report incorrectly labels the GPU as RTX 4000 Ada.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8-Flash-Next at 24.0 tok/s decode and 67.5 tok/s prompt on an RX 7600 8GB.
Setup is Strata 0.1.39 with IQ2_XS and a 32768-token context, splitting MoE experts across GPU, CPU and RAM with about 33 GB of pinned RAM.
User also reports Qwen3.6-35B-A3B Q6_K_XL on llama.cpp at 23.7 tok/s decode without MTP and 32.0 tok/s with MTP (98% draft acceptance), 161.7 tok/s prompt. Strata runs one model and one request at a time; prompt speed is limited by the 8 GB card reading prompts in 256-token chunks.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8-27B at 46.4 t/s generation and 43.1 t/s prompt processing on 2× RX 7900 XT (20 GB each).
Setup is llama.cpp with IQ4_XS, F16 KV cache, 8k context, MTP3 speculative decoding, Flash Attention, and ROCm 2.46.0. The 27B tests used roughly 700 input tokens and 1024 output tokens.
Two separate IQ4 copies reached 103.5 t/s combined with 16 concurrent requests. Flash Next UD-IQ4_XS (93.7 GB) ran at 8.6 t/s on one GPU and 5.5–6.5 t/s on two GPUs. User asks whether the x1 PCIe slot, CPU/RAM offloading, or backend settings are the bottleneck.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
coding
User reports Qwen3.8-Flash-Next running at 40-50 t/s for 10-15 minutes on an M5 Max via the Strata engine, with temperatures staying barely above idle.
Setup is Strata with an unspecified quant, running a JavaScript flocking simulation prompt through OpenCode.
The user compares this to running a 27B model via oMLX on the same M5 Max, where they had to manually set fans to 100% to manage heat, and asks why the Strata run stays cool.