Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8 27B averaging in the 170s t/s on an RTX 5090, peaking at 220 t/s.
Setup is ninfer-serve with the nvfp4 weights, MTP speculative decoding with 3 draft tokens, lm-head-draft, an fp8 KV cache and 240,000 context.
User calls it double or more the throughput of llama.cpp.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
long-contexttool-useagentic
User compares NInfer, llama.cpp and vLLM for Qwen3.8-27B on an RTX 5090. NInfer with NVFP4 and MTP3 reaches 158-213 t/s decode and 3904-7265 t/s prefill.
NInfer outperforms llama.cpp with Q5_K_M by 1.4-2.8x decode and 2.6-4.7x prefill.
Quality is statistically indistinguishable across engines. NInfer lacks json_mode support, and vLLM speed is not directly comparable due to wall-clock timing.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports prose decode at ~144 t/s and code at ~256 t/s, with parallel x2 at ~261/451 t/s.
Prefill TTFT runs from 0.33s at 5k to 23.46s at 150k, with a 175k KV pool and a host-RAM KV tier that resumes a ~100k conversation in ~1s.
External evals give GPQA Diamond 84.8%, GSM8K 96.8%, MATH-500 95.6%, AIME 2024 83.3%, HumanEval 56.7% and MBPP 75.0%. The run uses the froggeric template with xhigh reasoning and a 16k think cap.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User benchmarks DFlash2 speculative decoding against MTP on Qwen3.8-27B.
DFlash2 gives 20% faster generation but reduces context size by 38%.
The prompt and generation TPS figures are weighted averages.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
summarizationmultilingual
User benchmarks a DSpark PC Tree speculative decoding fork at 159.00 tok/s in its best config, PCTree k3/n16, against 94.27 tok/s for plain decoding.
User also tests Qwen3.8 27B Q4, with worse results.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
long-context
User reports a KV cache blending technique that splits the prompt into chunks, caches each part in isolation, and concatenates the caches for decode.
The method maintains needle-in-haystack retrieval with overlap.
Prefill speed is boosted 3x to ~1.3k tps at 256k tokens, comparable to Qwen3.8-27b on an RTX 5090.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
agenticcoding
User benchmarks 180K context with MTP5 enabled, reaching 2587 t/s prefill and 200.5 t/s decode.
The run also covers 64K and 120K contexts.
A synthetic corpus yields high MTP acceptance, while real agentic coding logs show about 51% accept and about 154 t/s.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
coding
User reports an average of 120 t/s across 3 parallel sessions.
Setup is a 451K global KV cache with vision enabled and power limited to 400W.
Detailed benchmarks are given in a table.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User benchmarks DFlash2 quants and MTP, with the best speed at 154.08 t/s for Q4 at n-max 7.
The best balance is DFlash2 Q2 at n-max 5, reaching 137.19 t/s with 121k context. MTP tops out at 128.32 t/s at n-max 6.
Context size is set to 113920 in the config.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen 3.8 Flash-Next at about 25 t/s on an RTX 5090 with 64 GB of RAM.
Setup is the Q4_K_XL quant, with the n-gram table unloaded to an SSD to fit in memory.
User advises that the n-gram table can be offloaded to an SSD with minimal speed cost.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports running unsloth/Qwen3.8-Flash-Next-GGUF on an RTX 5090.
The user asks how it runs on the 5090 and hopes for more benchmarks.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports speculative decoding with the draft model dspark-DeepSeek-V4-Flash-0731-Q8_0.gguf, peaking at 153.9 t/s on code, averaging 120 t/s on code and 80 t/s on non-code.
Setup is a 5090 on x8, an RTX PRO 6000 on x8 and an RTX PRO 5000 on x4 over PCIE Gen5, with a tensor split of 22,8,13.
Spec draft n-max is 5 and p-min is 0.6.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports a fork of NInfer with tensor parallelism and YaRN scaling decoding at 119 t/s with MTP and 57 t/s without at 653k context, and 48 t/s decode at 1M context, around 100 t/s with MTP.
Setup is a fork of NInfer with tensor parallelism and YaRN scaling.
Against vLLM at 42 t/s at 653k, MTP acceptance drops to zero past 262k. Prefill is 1.2-1.3x slower than vLLM. Two GPUs reach 75 t/s versus 54 t/s on one at 250k.
User reports building FlashMLA for consumer Blackwell (sm_120) GPUs, likely RTX 5090.
The build targets MLA inference and training workloads.
It delivers a 2-3x speedup over SDPA. No specific model family is mentioned; the focus is on kernel performance.
User compares NVFP4 against Q5_K_M on Qwen3.8-27B.
NVFP4 with adjusted sampling (temp=0.9, min_p=0.05) reaches 80% strict on IFBench, matching local BF16 and near official BF16 79.5%, while Q5_K_M stays at 76%.
NVFP4 runs about 3x faster and uses less VRAM. The Q5_K_M run uses llama.cpp.
User benchmarks Qwen3.5-9B, NVIDIA-Nemotron-3.5-Lightning-30B-A3B and Gemma-4-12B-it on a single RTX 5090.
Nemotron-3.5-Lightning-30B-A3B runs as a UD-Q4_K_XL GGUF and Gemma-4-12B-it as QAT w4a16.
Qwen3.5-9B scores 0.717 on GPQA Diamond and 0.713 on LiveCodeBench.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
coding
User reports 20 t/s average with MTP running a model in an FPS game built with it, on a 5900x CPU, 128 GB DDR4 RAM, an RTX 5090 and an RTX 4000 PRO with 32+24 GB VRAM.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
coding
User reports MTP support merged in ik_llama.cpp, with decode at 90 t/s on code, up from 45 without.
Draft acceptance is 93-99% on code and 60-65% on prose.
The same test on an RTX Pro 6000 gives 85 to 113 t/s on code and 83 to 59 t/s on story, and on a 12 GB 4070 it gives 9.5 to 12.5 t/s on code.