Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
codinglong-context
User reports Qwen3.8-27B at 110+ t/s on a single RTX 5090 with a fork of llama.cpp called focus-llama.
Setup uses Q6_K_XL GGUF quantization with q8_0 KV cache, 256k logical context capped to an ~85k physical KV buffer, and adaptive MTP speculative decoding.
Speed ranges from about 120 t/s initially down to 60 t/s as context fills; the user cites 100-120 t/s depending on context fullness and says context usage stays at 18-30%.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
codingagentictool-use
User reports DeepSeek-V4-Flash at ~28 tok/s single-stream decode on one RTX 5090 with dual Xeon Gold 6138 and 256GB DDR4-2400.
Setup is stock ktransformers + sglang with int4 AMXINT4 CPU-offloaded experts, NUMA-sharded, top-4 adaptive-k, 8192 context, fp8 KV cache. The 284B-A13B model is 142 GB at 4-bit and does not fit the GPU; experts are computed on the CPU.
Re-verified at 27.9-28.0 tok/s steady-state decode. Decode is DDR4-bandwidth-saturated. EAGLE speculative decode (27.2), more GPU experts (27.5), and 80 hyperthreads (2.7) all lost. Coding canary passes 8/8.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
codingagenticlong-contexttool-usevision
User reports GLM-5.3-Flash at 21.96 t/s decode (narrative) and 26.10 t/s (code) on 2x RTX 5090.
Setup is llama.cpp with IQ4_XS GGUF weights, f16 KV cache, 204800-token context, DFlash2 Q4_K_M draft model with speculative decoding (n=3), and MoE expert tensors offloaded to CPU.
Prefill measured 421.07 t/s at ~10K prompt tokens and 448.20 t/s at ~87K. Soak test passed with p50 decode 20.70 t/s and 100% TPS retention over 25 turns. Expert cache was requested but not allocated (CACHE_DISABLED), so the run measured a half-cached path. Stress test failed the long-context recall ladder.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8-Flash-Next at 26.2 tok/s single-stream on an RTX 5090 32 GB with 254 GiB system RAM.
Setup is FreeToken 0.1.3 with NVFP4 routed experts streamed from system RAM (about 120 GB pinned for offloaded experts) and turbo4 KV cache; MTP is off because verification cost more than direct generation.
Aggregate throughput reaches 47.5 tok/s at concurrency 2, 66.9 at 4, 64.6 at 6 and 68.8 at 8, topping out around 65-77 tok/s, limited by PCIe bandwidth and per-step expert fetch volume.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
summarizationtool-uselong-context
User reports Qwen3.8 27B at about 15 tok/s on an RTX 5090.
Setup is Ollama with Row-Bot 5.0, running locally with no API keys or cloud.
The model read a year of bills, a tenancy agreement, an insurance policy and a rent increase letter, charted a CSV, quoted lease clauses, saved 8 linked memories to a local knowledge graph, and drafted an email. Some turns took over 2 minutes.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports running Qwen3.8 Flash-Next quantized builds on an RTX 5090 with 64GB of system RAM, fitting Atomic Q4_K_M at 4.27bpw with 31 layers offloaded and Swift IQ4_XS with 32 layers offloaded.
Setup uses partial CPU offload to fit the models in memory; the user notes that offloading more layers would cause a significant slowdown.
User reports around 40 tok/s when staying in the 34-30 layer offload range, and asks whether a larger quant of a 27B model would be more capable than the cut-down Flash-Next builds.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
codingtool-useagenticlong-contextvision
User reports Qwen3.8-27B at 92.9 tok/s decode on a single RTX 5090 with SparkInfer and no speculation.
Setup is SparkInfer v0.5.5 with an NVFP4 checkpoint (17.92 GB, two shards) and int8 KV cache at the full 262,144-token context, which costs 27.9 GB.
The same weights reach 85.8 tok/s on SGLang with FP8 KV, 161.7 tok/s on SGLang with the DSpark v2 drafter, and 264.8 tok/s overall (up to 420.2 on code) in SparkInfer's bench harness with DSpark. Under concurrency SparkInfer peaks at 344.3 tok/s aggregate at 8 requests, while vLLM reaches 546.6 tok/s at 16. A configured 32,022-token prefix cache cuts a repeated request from 5.75 s to 0.30 s.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
codinglong-context
User reports DeepSeek V4.1 Flash at 97.23 tok/s for a single interactive request on eight RTX 5090 32GB GPUs over PCIe with no NVLink.
Setup is vLLM with MXFP4 quantization, 32,768-token context, TP8 plus expert parallelism, and 9-11 GiB/rank of decoder expert offload to host RAM (503 GiB installed, about 264-279 GiB shared).
Aggregate output reaches 247.77 tok/s at eight concurrent requests and 368.55 tok/s at 128 concurrent text requests; 8K prefill at two concurrent requests is 3,145 tok/s. The checkpoint is about 476 GiB and its quantization is unchanged.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
codingsummarizationtool-useagenticlong-context
User reports Qwen3.8 27B at roughly 148 tok/s on an RTX 5090 with MTP enabled.
Setup is llama.cpp with NVFP4 weights and MTP speculative decoding on a single 32GB card.
Without MTP the same model ran at about 72 tok/s, and a Q4_K_M build at about 70 tok/s. Other models tested: GPT-OSS 20B MXFP4 at about 311 tok/s, Qwen3.6 hybrid at about 250 tok/s, Gemma 31B QAT at about 68 tok/s, and Flash Next at about 199 tok/s.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
codingagenticlong-contextvision
User reports Qwen3.8-27B at ~200 tok/s single stream on one RTX 5090.
Setup is Ignis with NVFP4 and DFlash2 speculative decoding (7 draft tokens), 512K context via YaRN x2, KV at 9 KB/token.
Aggregate throughput is >800 tok/s with 8 lanes; image questions cost 60-90 ms each; locate found the right log line 21/23 times at median 2.6 s.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
visiontool-uselong-contextagentic
User reports MiMo-V2.6-Flash-RL at 109.7 tok/s decode on one stream on one RTX 5090 plus four DGX Spark (GB10) systems.
Setup is the mimo26f-afd engine with MXFP4 weights and FP8 KV cache, attention-FFN disaggregation over RoCE v2 RDMA, DFlash speculative decoding, up to 1,048,576 tokens of context.
Decode reaches 279.1 tok/s across six streams and 416.7 tok/s across sixteen; cold prefill is 3,630 / 5,123 / 4,816 / 4,283 tok/s at 2K / 8K / 32K / 64K. The comparison is a 4-Spark vLLM TP4 reference without the 5090, so the uplift includes the fifth device.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
codingagenticvision
User reports Qwen3.8 27B at around 80 t/s on an RTX 5090, running Q6 with 160k context, medium thinking and vision enabled.
Setup uses the Hermes agent harness with Searxng in Docker for web browsing.
User finds it slow for real-world coding tasks, citing 15 minutes for a website one-shot versus 3-5 minutes with Claude, and questions whether the model's overthinking makes it weak for coding.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User compares Qwen3.8 27B across four setups on an RTX 5090 at 120k context, reporting decode speeds of ~147 t/s (ninfer non-NVFP4 with MTP), ~141 t/s (llama.cpp Q4_K_M with MTP), ~155 t/s (ninfer NVFP4 with MTP), and ~68 t/s (llama.cpp Q4_K_M without MTP).
Setup is llama.cpp with Q4_K_M GGUF, thinking on at xhigh, default sampling, one run per setup.
Stock llama.cpp without MTP is less than half as fast as ninfer, but MTP lifts it from 68 to 141 t/s. NVFP4 had the highest t/s but wrote the most tokens, finishing third on wall-clock time. All four runs produced working games; only the NVFP4 build could be won legally. User is sticking with the non-NVFP4 ninfer build.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
coding
User reports Qwen3.8 27B Swift at 143 tok/s decode and 2840 tok/s prefill on an RTX 5090.
Setup is llama.cpp with Q6_K quantization.
The figures come from a coding prompt asking for an HTML/JS calculator, which produced a working result in 1 minute 10 seconds; the user calls this the ceiling after two days of trials.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
agenticcoding
User compares NInfer's official Qwen3.8-27B quant (part NVFP4, part FP8) against QUASAR's QAT full-NVFP4 checkpoint with DFlash2 embedded, both on a single RTX 5090.
QUASAR reaches 484 tok/s on JSON output, 204 tok/s on prose, and 13.7k tok/s prefill, using 27.0 GB VRAM. The official quant gets 400 tok/s JSON, 176 tok/s prose, 11.4k tok/s prefill, and 30.8 GB VRAM.
On a real agentic task the QUASAR build averages 287 tok/s over 33k tokens. GPQA Diamond scores 89.9% versus 90.4% for the official quant, and perplexity is about 1.9% worse.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
codingagenticlong-context
User announces MegaCapybara, a purpose-built engine for RTX 5090 focused on Qwen3.8 27B. Claims roughly twice the decode speed of Ninfer, reaching up to 650 t/s in small bursts and 2600 t/s with 12 concurrent agent slots at 800k context. Setup includes custom kernels, Dflash2 with DSpark headers, draft trees, and smart VRAM-RAM-DISC cache management.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8-Flash-Next at 1,796 tok/s prompt and 128-134 tok/s decode on an RTX 5090 32 GB plus an RTX 4070 Ti SUPER 16 GB.
Setup is Strata with IQ3_XXS GGUF, int8 KV cache, 32K cells resident per layer, MTP spec 4, vision on, and 262,144 context; the 76 GB model runs with 31 GB of system RAM and experts streamed from an mmap'd GGUF.
Decode is a range of 128-134 tok/s median on real sampling and 95-108 greedy; stock 0.1.33 with a layer split at 36 gave 890 tok/s on an 80K prompt and about 110 tok/s decode. Three patches cut major faults from 24 million to 14 thousand and chat switching from 20-48 s to 0.6-1.2 s.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8-Flash-Next at IQ3_S running on a power limited RTX 5090 with 96GB of DDR5-6400, giving 150-200 tok/s decode and 5-6k prefill.
Setup is Strata with IQ3_S weights and 8-bit KV cache at 128k tokens of context.
Both figures are given as ranges, so no single throughput is recorded.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User benchmarks Qwen3.8-27B at 171 tok/s on an RTX 5090, solving 14 of 16 hard problems.
Setup is Ollama with Q4_K_M GGUF at 64K context, temperature 0.2 and a 32,768-token generation cap, using 17.5 GB of VRAM.
Nemotron 3.5 Lightning reached 216 tok/s and 14/16 at 25.4 GB, while Muse Glimmer reached 81 tok/s and 15/16 at 16.9 GB.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8-27B NVFP4 at 152 tok/s decode on one RTX 5090 under WSL2, with MTP speculative decoding and host embedding enabled.
Setup is SGLang v0.5.20 with NVFP4 weights, 147,456-token context, on a Ryzen 9 9950X3D with 96 GB RAM.
Plain 27B without speculation reached 93 tok/s at one stream (89.7-96.0) and 456 tok/s total at 5 streams; with MTP, 5 streams totalled 753 tok/s. Prefill of a 6K-token prompt was 13.4-15.0K tok/s plain and 9.2-12.4K tok/s with MTP. Energy was 2.91 J/token plain and 1.72 J/token with MTP.