Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
multilinguallong-contextvision
User reports Qwen3.8 27B at about 100 t/s on a single RTX 5090, but only about 82k tokens of usable context despite configuring 253,952.
Setup is SGLang serving the Huihui-Abliterated NVFP4 MTP VL build with an fp8_e4m3 KV cache, NEXTN speculative decoding at 3 steps and 4 draft tokens, flashinfer attention, chunked prefill of 1024, and a 16 GB hierarchical cache.
User asks how to tune the setup for more context without losing multimodal capability.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8 Flash-Next at 90-100 t/s decode and 3000 t/s prefill on two RTX 5090s across two machines.
Setup is llama.cpp with RPC over a ConnectX-5 100G link, using Unsloth Q2 quants.
User sees little practical reason to use this over the 27B model and asks what others get with 2x 5090 in one machine or a 6000 Pro.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8 27B at 173 to 179 t/s decode on an RTX 5090, with 0.79s first token at about 8K prompt and four concurrent workflows done in 8.15s.
Setup is vLLM with NVFP4 weights, fp8 KV cache, 262,144 context, MTP speculative decoding with 3 tokens, PIECEWISE cudagraphs, and prefix caching.
The same 27B family also runs on a Radeon AI PRO R9700 32GB via Radiance in MXFP4 at 71 to 73 t/s decode, and on the 5090 via ExLlamaV3 at 40 to 70 t/s decode and 600 to 1400 t/s prefill at full 262K depth.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
codingcreative-writinglong-contextvision
User reports Qwen3.8 27B Huihui abliterated NVFP4 at 205 t/s decode on a single RTX 5090 32GB, with prefill around 9,000 t/s at 8K context.
Setup is NInfer on Windows 11 + WSL2 with fp8 KV cache at 196,608 context, MTP speculative decoding with 3 draft tokens and --lm-head-draft, vision enabled, weights about 19.7 GB.
Decode varies with MTP acceptance: 253 t/s on predictable text, 205 t/s on coding, about 120 t/s on creative prose, and 60-80 t/s with MTP off. At 128K context decode is about 115 t/s and prefill about 4,500 t/s.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8 27B at ~162.1 t/s generation and ~4.26k t/s prefill on an RTX 5090.
Setup is the Ninfer Windows Edition with a Q8 KV cache at 200k context.
Switching from LM Studio Q6 to Ninfer roughly doubled both decode and prompt speeds.
User reports Qwen3.8-27B RVN Heretic (ARA abliterated) at 140-190 t/s on code and ~100 t/s on prose on a single RTX 5090 at 159,744 context.
Setup is llama.cpp master with the Q6_K GGUF, q8_0 K/V cache, DFlash2 speculative decoding at n=4, vision projector on GPU, and Sharp chat template. Without speculation the same model runs at 62 t/s.
Cold prefill measured without the drafter: 32K in 11.7 s, 64K in 28.6 s, ~113K in 60 s. VRAM is 29,996 MiB after load and 30,332 MiB after the first image. Tool calls parse and images read correctly with no Xid.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
coding
User reports Qwen3.8 Flash-Next at 73.6 t/s on an RTX 5090 with 32 GB VRAM and 192 GB system RAM.
Setup uses FreeToken with a 123 GB model load; the user notes running out of context after a few prompts.
The user compares the speed favorably to a 27B model and calls the experience impressive.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
codingagenticreasoning
User reports Qwen 3.8 Flash/Next 125B MoE at about 38.8 t/s decode and about 200 t/s prefill on an RTX 5090 with 64 GB of system RAM.
Setup is ik_llama.cpp with the AP quant at Q4_K_M, using about 29.7 GiB of VRAM.
User notes the run is slower and depends on system RAM bandwidth and CPU offload, and adds that Qwen 3.8 27B is extremely fast but weaker. User asks for advice on models and optimizations.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
long-contextagenticcoding
User reports a fork of NInfer with a custom NVFP4 KV cache, YaRN context extension to 555k, multi-level prefix reuse with a host KV safety net, tool calling improvements, and monitoring.
Benchmarks show decode at 117 tok/s at 400k+ context, cold prefill of 260s for 414k tokens at 1600 tok/s, H2D restore in 0.4s, and 30GB of host KV.
Quality results show LongBench matching int8, AIME at 96.7%, and needle-in-haystack at 100%.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User runs a 186 GiB MoE model on 96 GB of VRAM, using host RAM for pinned experts, and reports 24.5 t/s decode at 8k context and 27.0 t/s at 0k.
Prefill reaches 159 t/s at 27k. Host RAM is about 208 GB pinned, the CPU does no compute and only feeds experts, and transfers are NUMA-aware. The setup is NVIDIA SM120 only.
Prefix caching with mid-prompt checkpoints cuts TTFT from 67.5s to 18.4s at 8k and from about 923s to 79s at 97k.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports a benchmark on an RTX 5090 with 256k context reachable on 24 GB.
The model name is Glimmer, which is not in the canonical list and is extracted as raw.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
codingtool-usereasoning
User reports Muse Glimmer 30B at 253 t/s on an RTX 5090 at 262k context.
Setup is llama-server with PR #26842 and DFlash, using a dflash draft model.
The run used a code patch workload. The user notes Meta's published figure of 233 t/s.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
creative-writingcoding
User reports Muse-Glimmer with dflash on an RTX 5090 at roughly 90-160 tok/s depending on the task.
The user observes unusual reasoning traces that are disorganized, repetitive, and self-referential, using "we" and mentioning policy.
The user compares the reasoning to Gemini models and contrasts it with Qwen 3.6, questions whether reasoning helps or hinders, and also tests a coding prompt for a Flappy Bird HTML game.
User reports 421 tok/s at 25k context on an RTX 5090 with muse-glimmer 30B, using a v2.2 optimizer profile for vLLM.
Setup runs without DFlash speculative decoding, with TTFT at 224ms (p95 500ms) and TPOT at 23ms.
Cost drops from $3.41 to $0.65 per 1M output tokens and energy from 4.72 to 1.08 J/tok. The user mentions SWE-Bench agents.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
long-context
User reports DeepSeek V4 Flash at 1M context on an RTX 5090, with prefill at 159 t/s and decode at 13.7 t/s.
Setup is a patched llama.cpp, with peak VRAM around 31 GiB.
The user verified the run with a needle-in-haystack test.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports DeepSeek V4 Flash on an RTX 5090 with a llama.cpp fork, at 22.7 to 21.3 t/s TG and 1105 to 927 t/s PP across prompt lengths.
Setup is the Q2_K quant at 1M context, with CPU offloading for the MoE layers.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports prefill of ~650-700 t/s and decode of ~17 t/s.
Setup uses CPU offloading for MoE experts, with a loading time of 32 sec.
User notes the speed is not as impressive as Qwen models but sees room for optimization.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8 2.4T at 0.25 t/s on dual RTX 5090s and three RTX 3090s with 96 GB of system RAM.
The model is 397 GB, far exceeding available VRAM and RAM, at 64k context.
The user deems the speed unusable.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8-2.4T-A95B at ~0.80 tok/s generation on an RTX 5090 plus an RTX 5060 Ti.
Setup is llama.cpp (Unsloth build 10360) with the Unsloth Q1_0 GGUF (397 GiB), native MTP speculative decoding at n_max=3 and p_min=0.5, and 92 experts offloaded to CPU.
MTP acceptance is 90.48%, a +3.64% throughput gain over no MTP. VRAM use is 29.6 GB on the 5090 and 12.3 GB on the 5060 Ti.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports 700 pp/s and 18 tg/s with llama.cpp using --cpu-moe on a Threadripper 5965WX with an RTX 5090.
User notes llama.cpp does not support native FP8 cache for DFV4 Flash, so an FP16 cache is required, and speculative decoding is not working.
User suggests building a rig with an Epyc and 2x 5060 Ti.