RTX 5090

reported speed:

73.6 tokens/s generation

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports Qwen3.8 Flash-Next at 73.6 t/s on an RTX 5090 with 32 GB VRAM and 192 GB system RAM. Setup uses FreeToken with a 123 GB model load; the user notes running out of context after a few prompts. The user compares the speed favorably to a 27B model and calls the experience impressive.

reported speed:

38.8 tokens/s generation · 200.0 tokens/s prompt processing

quant:

Q4_K_M (AP)

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports Qwen 3.8 Flash/Next 125B MoE at about 38.8 t/s decode and about 200 t/s prefill on an RTX 5090 with 64 GB of system RAM. Setup is ik_llama.cpp with the AP quant at Q4_K_M, using about 29.7 GiB of VRAM. User notes the run is slower and depends on system RAM bandwidth and CPU offload, and adds that Qwen 3.8 27B is extremely fast but weaker. User asks for advice on models and optimizations.

reported speed:

117.0 tokens/s generation · 1600.0 tokens/s prompt processing

kv:

NVFP4

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports a fork of NInfer with a custom NVFP4 KV cache, YaRN context extension to 555k, multi-level prefix reuse with a host KV safety net, tool calling improvements, and monitoring. Benchmarks show decode at 117 tok/s at 400k+ context, cold prefill of 260s for 414k tokens at 1600 tok/s, H2D restore in 0.4s, and 30GB of host KV. Quality results show LongBench matching int8, AIME at 96.7%, and needle-in-haystack at 100%.

reported speed:

24.5 tokens/s generation · 159.0 tokens/s prompt processing

quant:

UD-Q4_K_XL

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User runs a 186 GiB MoE model on 96 GB of VRAM, using host RAM for pinned experts, and reports 24.5 t/s decode at 8k context and 27.0 t/s at 0k. Prefill reaches 159 t/s at 27k. Host RAM is about 208 GB pinned, the CPU does no compute and only feeds experts, and transfers are NUMA-aware. The setup is NVIDIA SM120 only. Prefix caching with mid-prompt checkpoints cuts TTFT from 67.5s to 18.4s at 8k and from about 923s to 79s at 97k.

reported speed:

233.4 tokens/s generation

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports a benchmark on an RTX 5090 with 256k context reachable on 24 GB. The model name is Glimmer, which is not in the canonical list and is extracted as raw.

reported speed:

253.0 tokens/s generation

quant:

UD-Q5_K_M

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports Muse Glimmer 30B at 253 t/s on an RTX 5090 at 262k context. Setup is llama-server with PR #26842 and DFlash, using a dflash draft model. The run used a code patch workload. The user notes Meta's published figure of 233 t/s.

reported speed:

125.0 tokens/s generation

quant:

Q5_K_XL

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports Muse-Glimmer with dflash on an RTX 5090 at roughly 90-160 tok/s depending on the task. The user observes unusual reasoning traces that are disorganized, repetitive, and self-referential, using "we" and mentioning policy. The user compares the reasoning to Gemini models and contrasts it with Qwen 3.6, questions whether reasoning helps or hinders, and also tests a coding prompt for a Flappy Bird HTML game.

User reports 421 tok/s at 25k context on an RTX 5090 with muse-glimmer 30B, using a v2.2 optimizer profile for vLLM. Setup runs without DFlash speculative decoding, with TTFT at 224ms (p95 500ms) and TPOT at 23ms. Cost drops from $3.41 to $0.65 per 1M output tokens and energy from 4.72 to 1.08 J/tok. The user mentions SWE-Bench agents.

reported speed:

13.7 tokens/s generation · 159.0 tokens/s prompt processing

quant:

mixed Q8/Q4/Q2 (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports DeepSeek V4 Flash at 1M context on an RTX 5090, with prefill at 159 t/s and decode at 13.7 t/s. Setup is a patched llama.cpp, with peak VRAM around 31 GiB. The user verified the run with a needle-in-haystack test.

reported speed:

21.3 tokens/s generation · 1105.0 tokens/s prompt processing

quant:

Q2_K (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports DeepSeek V4 Flash on an RTX 5090 with a llama.cpp fork, at 22.7 to 21.3 t/s TG and 1105 to 927 t/s PP across prompt lengths. Setup is the Q2_K quant at 1M context, with CPU offloading for the MoE layers.

reported speed:

17.0 tokens/s generation · 675.0 tokens/s prompt processing

quant:

Q8_K_XL (GGUF)

kv:

q8_0

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports prefill of ~650-700 t/s and decode of ~17 t/s. Setup uses CPU offloading for MoE experts, with a loading time of 32 sec. User notes the speed is not as impressive as Qwen models but sees room for optimization.

reported speed:

0.3 tokens/s generation

quant:

UD-Q1_0

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports Qwen3.8 2.4T at 0.25 t/s on dual RTX 5090s and three RTX 3090s with 96 GB of system RAM. The model is 397 GB, far exceeding available VRAM and RAM, at 64k context. The user deems the speed unusable.

reported speed:

0.8 tokens/s generation · 0.8 tokens/s prompt processing

quant:

Q1_0 (GGUF)

kv:

q8_0

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports Qwen3.8-2.4T-A95B at ~0.80 tok/s generation on an RTX 5090 plus an RTX 5060 Ti. Setup is llama.cpp (Unsloth build 10360) with the Unsloth Q1_0 GGUF (397 GiB), native MTP speculative decoding at n_max=3 and p_min=0.5, and 92 experts offloaded to CPU. MTP acceptance is 90.48%, a +3.64% throughput gain over no MTP. VRAM use is 29.6 GB on the 5090 and 12.3 GB on the 5060 Ti.

reported speed:

18.0 tokens/s generation · 700.0 tokens/s prompt processing

quant:

Q8_K_XL (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports 700 pp/s and 18 tg/s with llama.cpp using --cpu-moe on a Threadripper 5965WX with an RTX 5090. User notes llama.cpp does not support native FP8 cache for DFV4 Flash, so an FP16 cache is required, and speculative decoding is not working. User suggests building a rig with an Epyc and 2x 5060 Ti.

User benchmarks Gemma4-12B on an RTX 5090, comparing vLLM stock, vLLM+Emmy, and llama.cpp. The runs report throughput and latency across various token lengths and concurrency levels. The comparison also covers speculative decoding (MTP) tests and kernel optimization details.

reported speed:

30.2 tokens/s generation · 64.0 tokens/s prompt processing

quant:

Q8_K_XL

kv:

bf16

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User benchmarks Q8_K_XL against IQ3_XXS on DeepSeek-V4-Flash-0731-UD. Q8_K_XL offloads about 40 GB to system RAM. IQ3_XXS is 2.2x faster at decode and 2.4x faster at prefill, with equal correctness. Q8_K_XL has more empty-answer failures, 8/125 against 5/125. A reasoning budget of 4096 eliminates empty answers, and disabling reasoning degrades trap handling. Q8_K_XL is slower but more reliable in some tasks.

reported speed:

15.0 tokens/s generation · 800.0 tokens/s prompt processing

kv:

fp8_ds_mla

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports full 1M context on a single RTX 5090 with CPU/RAM offloading, at about 800 t/s prefill and over 15 t/s decode. Setup uses DSpark speculative decoding with dynamic acceptance rates, with two routed MoE layers kept GPU-resident.

quant:

MXFP4

User reports 2000+ t/s aggregate and 30-40 t/s per session across 50 concurrent sessions on 8x RTX 5090. Setup is the REAP mxfp4 image with 250k KV per session.

reported speed:

13.8 tokens/s generation

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports DSpark K1/K2 at 13.78 t/s for reasoning and 17-18 t/s for content on a single RTX 5090 at full 1M context. Setup is a vLLM fork, guqiong96/Lvllmds4-x reporting vLLM 2.3.9, with phase-adaptive K1/K2 dual CUDA graphs and routed MoE experts in system RAM.

quant:

AD-IQ2_M

User compares quants of DeepSeek V4 Flash 0731 on 8x RTX 5090, benchmarking with wikitext-2 at 5632 context and reporting PPL and top-1 accuracy. The quant named AD-IQ2_M is 104 GB and reaches 83.6% top-1. The user mentions an H100 for comparison.

reported speed:

200.0 tokens/s generation

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports Qwen3.8-27B at about 200 t/s generation on a single RTX 5090 with speculative decoding, on day-0 support in NInfer. The setup uses a shared paged KV cache and ReplaySSM for GDN, with PDL in use. The user notes support for up to 8 concurrent requests.

reported speed:

200.0 tokens/s generation

quant:

NVFP4

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports Qwen3.8-27B at over 200 t/s decode on an RTX 5090 and an RTX Pro 6000, and 38 t/s on a DGX Spark. Setup is NVFP4 with DSpark on day-0 support. The user also mentions an H200 in the cookbook link.

reported speed:

58.1 tokens/s generation · 479.2 tokens/s prompt processing

quant:

Q6_K (GGUF)

kv:

Q8

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User compares Qwen3.8-27B against Qwen3.6-27B on an RTX 5090. The user reports prefill and decode speeds at concurrency 1, and finds the subjective quality improvement smaller than benchmarks suggest. Medium reasoning used fewer tokens than low on 3 of 5 prompts. The concurrency knee sits at 3 requests for Qwen3.8 versus about 7 for Qwen3.6.

quant:

FP8

User benchmarks 200 prompts at max concurrency 100, reaching 15,058.74 tok/s total token throughput and 1,673.19 tok/s output token throughput, with a peak output of 3,000 tok/s. Setup is vLLM with Model Runner V2, p2p enabled, and tensor parallel 2. Mean TTFT is 1,741.86 ms and mean TPOT is 45.74 ms. An AMD Radeon PRO W6600 is mentioned but not used for this benchmark.

User compares GGUFs of Qwen3.8-27B on 4x RTX 5090. Quants mentioned are Q8_0, IQ1_M, Q3_K_M, AD-IQ3_S and Q6_K, with KLD values reported. No tokens per second are given.

reported speed:

86.2 tokens/s generation · 588.2 tokens/s prompt processing

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports that DSpark GGUF for Qwen3.8 27B shows no performance improvement and is extremely memory hungry. Setup is llama.cpp with DSpark support merged in PR #25173, using a converted and quantized RadixArk/Qwen3.8-27B-DSpark GGUF, with context set to 64k to test the BF16 speculator. Draft acceptance is 0.28922 with a mean length of 3.02.

User reports KLD against BF16 for custom EXL3 quants of Qwen3.8-27B: 0.007406 for EXL3, 0.013126 for FP8, and 0.094978 for NVFP4. The quants require a custom vLLM fork, Gilded Gnosis.

reported speed:

100.0 tokens/s generation

quant:

Q8_0 (GGUF)

kv:

Q5_1

mtp (multi-token prediction):

on

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User benchmarks context length against KV cache quants on an RTX 5090 with Qwen3.8 27B MTP Q8_0 GGUF. Maximum stable context is 105,216 tokens with a Q8_0 KV cache, 137,984 with Q5_1, and 169,984 with Q4_0. Throughput dropped from 100 t/s to 25 t/s, which the user attributes to a llama.cpp issue.

reported speed:

200.0 tokens/s generation · 5950.0 tokens/s prompt processing

quant:

NVFP4

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports an aggregate 880 t/s at 6 parallel requests, peaking at 967 t/s, and over 200 t/s single stream with MTP speculative decoding. Prefill runs at about 5,950 t/s. Weights take 16.8 GiB, leaving about 13 GiB for KV cache. Benchmarks are HumanEval+ 152/164 and AIME25+AIME26 55/60, identical to int4 but 1.56x-1.98x faster. The run requires Blackwell FP4 cores and a 6-line patch not yet upstream.

reported speed:

59.0 tokens/s generation

quant:

5.5-bit (compressed-tensors)

kv:

fp8

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports Qwen3.8 27B at 59 t/s single stream on an RTX 5090, with aggregate throughput above 100 t/s for two requests and 279 t/s with four parallel jobs. Setup is vLLM 0.27.1 with the PrismaAqua 5.5-bit quant. The user also mentions the RTX 6000 Pro as target hardware. Quality is close to BF16 on a custom suite. The user prefers medium reasoning over xhigh because of fewer token-cap failures.

reported speed:

169.7 tokens/s generation

kv:

Q8

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports median decode speeds by context size on an unspecified setup: 169.7 t/s under 50k, 167.6 t/s at 50-100k, 156.9 t/s at 100-150k, 149.0 t/s at 150-200k, and 143.7 t/s at 200k and above. Context length is set to maximum. Peak decode for a single request is 222.0 t/s, with 211.2 t/s sustained over 5s.

reported speed:

120.0 tokens/s generation

quant:

Q5_K_L (gguf)

kv:

Q8

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports DFlash2 speculative decoding with Qwen3.8 27B on an RTX 5090, with bursts of ~200 tk/s on code, ~80-90 tk/s on thinking, and an average of ~120 tk/s for code generation. Setup uses a Q5_K_L base model and a Q4_K_M draft model. The user compares this to MTP at ~140 tk/s on code and ~100 tk/s otherwise, notes the setup is memory hungry and required reducing context from 220k to 160k, and reports better results previously with DFlash on Qwen3.6 27B.

reported speed:

75.0 tokens/s generation · 1100.0 tokens/s prompt processing

quant:

Q4_K_XL (gguf)

kv:

Q8

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports a DFlash2 draft model with a Q4_K_M quant, giving roughly 2x decode speedup with 5 draft tokens and roughly 2.7x with 7 draft tokens. Setup is a 4-bit Unsloth model at 256k context, with draft acceptance identical to Q8_0. Prefill slows by roughly 15% with the draft model.

reported speed:

82.6 tokens/s generation

quant:

W2

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports the Escha 2-bit quant of Qwen3.8-27B, 10.15 GB on disk, averages about 100% of FP8 performance across 8 benchmarks. On LiveCodeBench v6, Escha W2 scores 86.81 against 85.16 for FP8. The previous 2-bit Qwen3.6-35B MoE scored 62.6 against 67.0 for FP8 on the same benchmark.

reported speed:

220.0 tokens/s generation

quant:

NVFP4

kv:

FP8

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports up to 220 t/s, averaging 170 t/s, with ninfer. Setup uses speculative decoding with mtp and 3 draft tokens, plus lm-head-draft, with vision enabled. The user compares the result favorably to llama.cpp.

reported speed:

202.0 tokens/s generation · 3904.0 tokens/s prompt processing

quant:

NVFP4

kv:

FP8

mtp (multi-token prediction):

on

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User compares NInfer, llama.cpp and vLLM for Qwen3.8-27B on an RTX 5090. NInfer with NVFP4 and MTP3 reaches 158-213 t/s decode and 3904-7265 t/s prefill. NInfer outperforms llama.cpp with Q5_K_M by 1.4-2.8x decode and 2.6-4.7x prefill. Quality is statistically indistinguishable across engines. NInfer lacks json_mode support, and vLLM speed is not directly comparable due to wall-clock timing.

reported speed:

144.0 tokens/s generation

quant:

NVFP4 (safetensors)

kv:

FP8

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports prose decode at ~144 t/s and code at ~256 t/s, with parallel x2 at ~261/451 t/s. Prefill TTFT runs from 0.33s at 5k to 23.46s at 150k, with a 175k KV pool and a host-RAM KV tier that resumes a ~100k conversation in ~1s. External evals give GPQA Diamond 84.8%, GSM8K 96.8%, MATH-500 95.6%, AIME 2024 83.3%, HumanEval 56.7% and MBPP 75.0%. The run uses the froggeric template with xhigh reasoning and a 16k think cap.

reported speed:

97.7 tokens/s generation · 1995.8 tokens/s prompt processing

quant:

Q8_K_L (gguf)

kv:

Q8

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User benchmarks DFlash2 speculative decoding against MTP on Qwen3.8-27B. DFlash2 gives 20% faster generation but reduces context size by 38%. The prompt and generation TPS figures are weighted averages.

reported speed:

159.0 tokens/s generation

quant:

BF16 (gguf)

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User benchmarks a DSpark PC Tree speculative decoding fork at 159.00 tok/s in its best config, PCTree k3/n16, against 94.27 tok/s for plain decoding. User also tests Qwen3.8 27B Q4, with worse results.

reported speed:

1300.0 tokens/s prompt processing

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports a KV cache blending technique that splits the prompt into chunks, caches each part in isolation, and concatenates the caches for decode. The method maintains needle-in-haystack retrieval with overlap. Prefill speed is boosted 3x to ~1.3k tps at 256k tokens, comparable to Qwen3.8-27b on an RTX 5090.

reported speed:

200.5 tokens/s generation · 2587.0 tokens/s prompt processing

quant:

NVFP4

kv:

Q8

mtp (multi-token prediction):

on

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User benchmarks 180K context with MTP5 enabled, reaching 2587 t/s prefill and 200.5 t/s decode. The run also covers 64K and 120K contexts. A synthetic corpus yields high MTP acceptance, while real agentic coding logs show about 51% accept and about 154 t/s.

reported speed:

120.0 tokens/s generation

quant:

NVFP4

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports an average of 120 t/s across 3 parallel sessions. Setup is a 451K global KV cache with vision enabled and power limited to 400W. Detailed benchmarks are given in a table.

reported speed:

154.1 tokens/s generation

quant:

Q8_K_L (gguf)

kv:

Q8

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User benchmarks DFlash2 quants and MTP, with the best speed at 154.08 t/s for Q4 at n-max 7. The best balance is DFlash2 Q2 at n-max 5, reaching 137.19 t/s with 121k context. MTP tops out at 128.32 t/s at n-max 6. Context size is set to 113920 in the config.

reported speed:

25.0 tokens/s generation

quant:

Q4_K_XL (gguf)

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports Qwen 3.8 Flash-Next at about 25 t/s on an RTX 5090 with 64 GB of RAM. Setup is the Q4_K_XL quant, with the n-gram table unloaded to an SSD to fit in memory. User advises that the n-gram table can be offloaded to an SSD with minimal speed cost.

reported speed:

44.0 tokens/s generation

quant:

Q4

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

reported speed:

43.4 tokens/s generation

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports running unsloth/Qwen3.8-Flash-Next-GGUF on an RTX 5090. The user asks how it runs on the 5090 and hopes for more benchmarks.

reported speed:

120.0 tokens/s generation

quant:

Q4_K_XL (gguf)

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports speculative decoding with the draft model dspark-DeepSeek-V4-Flash-0731-Q8_0.gguf, peaking at 153.9 t/s on code, averaging 120 t/s on code and 80 t/s on non-code. Setup is a 5090 on x8, an RTX PRO 6000 on x8 and an RTX PRO 5000 on x4 over PCIE Gen5, with a tensor split of 22,8,13. Spec draft n-max is 5 and p-min is 0.6.

reported speed:

119.0 tokens/s generation

quant:

NVFP4

kv:

Q8

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports a fork of NInfer with tensor parallelism and YaRN scaling decoding at 119 t/s with MTP and 57 t/s without at 653k context, and 48 t/s decode at 1M context, around 100 t/s with MTP. Setup is a fork of NInfer with tensor parallelism and YaRN scaling. Against vLLM at 42 t/s at 653k, MTP acceptance drops to zero past 262k. Prefill is 1.2-1.3x slower than vLLM. Two GPUs reach 75 t/s versus 54 t/s on one at 250k.

User reports building FlashMLA for consumer Blackwell (sm_120) GPUs, likely RTX 5090. The build targets MLA inference and training workloads. It delivers a 2-3x speedup over SDPA. No specific model family is mentioned; the focus is on kernel performance.

quant:

NVFP4

User compares NVFP4 against Q5_K_M on Qwen3.8-27B. NVFP4 with adjusted sampling (temp=0.9, min_p=0.05) reaches 80% strict on IFBench, matching local BF16 and near official BF16 79.5%, while Q5_K_M stays at 76%. NVFP4 runs about 3x faster and uses less VRAM. The Q5_K_M run uses llama.cpp.