Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User benchmarks Qwen3.8-Flash-Next IQ3_XXS at 193.8 t/s generation and 4627.2 t/s prompt processing on an RTX 5090 with 128GB system RAM.
Setup is Strata 0.1.40 with INT8 KV cache, 4096-token context, MTP speculation with --spec 4, and CPU expert-pool offload. The IQ3_XXS GGUF is 75.84 GB with 22.34 GiB of GPU expert cache.
User compares three quants: IQ3_XXS (193.8 t/s at 4K), IQ3_S (161.0 t/s), and UD-IQ4_XS (116.3 t/s). Accuracy screening on 14 MMLU-Pro questions scored 12/14 for all three. IQ3_XXS had the highest median generation speed in the 4K test.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
codingcreative-writingrpagenticvision
User compares Strata and Infernix running Qwen3.8-Flash-Next Uncensored on a single RTX 5090 32GB with 192GB system RAM on Windows 11. Infernix NVFP4 reaches 137.0 tok/s decode and 3,497 tok/s prefill at 512k context (YaRN 2), while Strata Q4_K_S gets 126.3 tok/s decode and 3,045 tok/s prefill at the same context.
Setup uses int8 KV cache, MTP spec-4 with lm-head-draft, and a 52GB n-gram volume mmap'd from NVMe with 63.3 GiB of experts pinned in RAM. Decode was measured at roughly 39k effective context, with ±10-15 tok/s run noise.
User reports Infernix is 23% faster at 262k and 9% faster at 512k, calls it the daily driver, and notes vision works on both engines.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8 Flash-Next at 665 tok/s structured and 354 tok/s prose on an RTX 5090 + RTX 5060 Ti.
Setup is Basalt, a Strata fork for Blackwell, with IQ3_XXS weights, 64k context, MTP enabled, 400 W, on a 9950X with 32GB DDR5 RAM.
A single 5090 gives 585 struct and 316 prose, about 12% slower, with prefill unchanged. Concurrency of 8 streams totals 623 tok/s. IQ3_S prose is 300 tok/s at ~89% top-1 agreement.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
codingtool-use
User reports Qwen3.8-27B at about 304 t/s decode at 8K context on one RTX 5090, with 221 t/s at 60K and prefill about 12,500 t/s at 60K.
Setup is a vLLM main build newer than 0.27.1 with a custom NVFP4 checkpoint, DFlash2 drafter with 6 speculative tokens, fp8 KV cache, max model length 195,000 and GPU memory utilization 0.92.
Two concurrent slots get roughly 255-299 t/s end to end. An earlier vLLM 0.27.1 setup with qwen3_5_mtp and 3 speculative tokens reached about 123 t/s at 8K and 90 t/s at 60K, with aggregate throughput rising from 67 t/s at one stream to 579 t/s at 16 streams. NVFP4 has a small quality cost versus GGUF Q4_K_M (q_avg 92.5 vs 93.2; HumanEval 89.6 vs 94.5). The GPU draws roughly 380-520 W under vLLM versus about 120 W with llama.cpp.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports LFM2.5-8B-A1B at 563.12 t/s generation and 30,680.66 t/s prompt processing on an RTX 5090.
Setup is llama.cpp llama-bench (build b9653) with Q8_0 weights, all layers on GPU (ngl=99), single request, no speculative decoding.
The dataset card also lists llama-bench rows for gpt-oss-20b, Nemotron-3-Nano-30B-A3B, Qwen3.6-35B-A3B, Qwen3-Coder-Next, Gemma 4 12B, Qwen3.6-27B, Gemma 4 31B, and Ling-3.0-flash, plus rows for Qwen3.6-27B-MTP, Qwen3.6-27B Q6_K, gpt-oss-120B, Mistral Small 4 119B, and several quant ladders; some runs used vLLM and SGLang.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports GLM-5.2 at 5.8-6.8 tok/s single-request decode on 6x RTX 5090 with dual Xeon Silver 4510 and 251 GB RAM.
Setup is colibri with int4 container, full expert residency across VRAM+RAM, CUDA_EXPERT_GB=auto PIN_GB=all, COLI_CUDA_PIPE=2 TC_W4A16, DRAFT=0. TTFT about 13 s, expert hit 89-100%.
The model is a 744B frontier-class model on disk at ~370 GB; cold decode costs ~11 GB disk reads per token. On a 2-socket Xeon Silver 4510 host with 6x RTX 5090, selective COLI_NUMA=1 raised effective CPU-expert bandwidth from 42.42 to 58.26/65.89 GB/s and greedy decode from 7.66 to 9.02/9.17 tok/s (64 tokens, TEMP=0 DRAFT=0, byte-identical output).
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
agenticcodingtool-use
User reports Qwen3.8-Flash-Next at 125 t/s generation and 5,879 t/s prefill on an RTX 5090 with the Infernix engine.
Setup uses NVIDIA's NVFP4 quant with expert streaming from system RAM and SSD, n-gram drafting, and PCIe Gen5 x8. The engine is an offshoot of NInfer.
Compared to Strata running Unsloth's UD-Q4_K_XL quant, Infernix shows 82% lower average time to first token (5.9 s vs 32.9 s) and 65% lower workload wall time (7.8 min vs 22.3 min). At concurrency=2, output reaches 143 t/s. The user notes the model is broadly on-par quality-wise with the Unsloth quant and that PCIe Gen5 x16 may yield higher performance.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
codingagenticlong-context
User reports daily driving Qwen 3.8 Flash-Next MoE (uncensored NVFP4) on an RTX 5090 with 136 GB DDR5 RAM, seeing 135-153 tok/s decode over 8 hours of agentic coding.
Setup is Strata (NVFP4 fork) with NVFP4 MoE weights, BF16 embeddings, FP8 PLE and INT8 dynamic paged KV cache at up to 262k context; 22.3 GB of expert slots held in VRAM with the remainder paged from a 64 GB host RAM arena.
Decode stayed flat from 50k to 212k context (r = +0.005), with 134.5 tok/s at 52k, 146.7 tok/s at 205k and 141.2 tok/s at 212k; MTP speculative decoding had a 79.60% acceptance rate and the MoE VRAM cache hit rate averaged 89.9%. Over 2,411 requests the KV cache hit rate was 99.31%.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports a ~27B NVFP4 model on an RTX 5090 running 130-200 tok/s when the terminal window is focused, dropping to 50-60 tok/s when unfocused.
Setup is NInfer on Windows; the slowdown was traced to CPU-side scheduling (wait 16-17 ms focused vs 38-41 ms unfocused), not GPU throttling, since SM clocks were higher during slow runs (2550-2600 MHz vs 2100-2200 MHz) at the same 400 W power limit.
Running the server detached or headless fixed it, keeping throughput around 130-200 tok/s regardless of window focus; the same behavior reproduced with a native Windows build, so it is not WSL2-specific.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8-Flash-Next at 241.7 tok/s on code and 254.3 tok/s on prose on 2x RTX 5090.
Setup is ExLlamaV3 v1.5.0 with TabbyAPI, 2.50 bpw EXL3 quant, 8-bit KV cache, 262k context, MTP speculative decoding on, 2 cards.
Aggregate decode at 4 streams is 592 / 595 tok/s; prefill is 10,034 / 9,865 t/s at 86-172k-token prompts; peak decode power 302.8 W / 285.0 W per card.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
coding
User reports Qwen3.8 at 210.5 t/s decode and 6708.5 t/s prefill on a single RTX 5090.
Setup is NInfer with MTP speculative decoding, 1690 prompt tokens and 3029 generated tokens, 254 ms TTFT.
The user runs a demoscene generator where the model writes Three.js effects and revises them from captured video; LM Studio is also supported for the regular workflow.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8-27B at 139.7 tok/s on an RTX 5090 with SGLang and a D-Flash 2 drafter.
Setup is SGLang with NVFP4 weights and an FP8 KV cache, single-user at concurrency 1. Other profiles on the same card: DSpark v2 drafter 110.2 tok/s, native MTP under vLLM 100.8 tok/s, and a vLLM baseline with no speculation at 84.5 tok/s.
Prefill is about 9,081 tok/s on the 5090 against about 196 tok/s on an M4 Max 128GB running oMLX 4-bit MLX builds, where decode was 16.7 tok/s with speculation off, 35.5 tok/s with DFlash 2 greedy and 47.2 tok/s with DFlash 2 thinking on. A warm prefix cache cuts time to first token by about 90% on both machines.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Token Rush decoding Qwen3.8-27B at 70 tok/s without speculation at 200k tokens of context on one RTX 5090.
Setup is Token Rush with int4 GPTQ weights at 4.25 bits per weight and a 128-group scale, FP8 KV cache, speculation off for this figure, one stream, the whole model in VRAM.
With DFlash2 speculation the same engine decodes 229 tok/s on essay, 358 on code and 379 on math at short context, and 200-220 tok/s on prose at 200k. The full 256k window fits in 26 GB of VRAM.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports GLM-5.3-Flash NVFP4 on one RTX 5090 at ~15-18 tok/s decode and ~1000 tok/s prefill on a 64K-token prompt.
Setup is strata-glm, a Strata fork, with the NVFP4 checkpoint on one RTX 5090, experts streamed from 128 GB DDR5 RAM and two NVMe drives.
Decode on a ~1-1.4K chat prompt was 18.0 tok/s on Cyrillic chat and 14.8 on code by default, 16.5 and 12.9 in exact mode, and 15.3 in exact mode after the 64K prompt. llama.cpp on the same box managed 2.4-2.9 tok/s decode and 29 tok/s prefill. The project is archived.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
coding
User reports approximately 50 TPS on Qwen3.8 Flash Next on an RTX 5090 with 96GB DDR5.
Setup is a single RTX 5090 with 96GB DDR5 system RAM, with part of the 125B model in system RAM since it cannot fit in 32GB of VRAM.
The user also gets 200+ TPS on Qwen3.8 27B on the same card. No engine or quant is stated, and the post asks which model sits between the two for coding.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
agentictool-use
User benchmarks 18 tool-calling models on 33 recorded DevOps failure scenarios on a single RTX 5090 32GB, reporting gemma4:31b at 64 tokens/s with a 94% pass rate and 23.1 GiB measured VRAM.
Setup is Ollama 0.35.0 with each model's default build (Q4_K_M, except gpt-oss:20b at MXFP4), 32k context, temperature 0, one run each; VRAM is what nvidia-smi showed for the Ollama process.
The top six models tie between 88% and 94%, separated by speed and memory; the fastest model on the card (nemotron-3.5-lightning:30b at 427 tokens/s) passes 52%, and gemma4:12b passes 88% in 9.2 GiB. Ollama's own ollama ps size under-reported real GPU usage by 0.6 to 4.2 GiB for 17 of 18 models. north-mini-code-1.0 could not run due to a CUDA illegal memory access error. Nothing spilled to CPU.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User benchmarks Qwen3 MoE 30B-A3B at 352.06 t/s on an RTX 5090.
Setup is llama.cpp with Q4_K_M and flash attention, 99 layers offloaded to GPU.
This is the optimized run with kernel fusion and concurrent CUDA streams enabled; baseline was 246.96 t/s. The post also reports gpt-oss 20B MXFP4 at 419.14 t/s on the 5090 and both models on an RTX 4090.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User benchmarks Qwen3.8-Flash-Next with DFlash2 speculative decoding on an RTX 5090, reporting 118.63 tok/s single-stream decode.
Setup is TensorRT-LLM with NVFP4 weights, a 90-token prompt and 512 forced output tokens, five trials per batch size.
Aggregate decode throughput rises to 1,284.55 tok/s at batch 12 with the selector policy, while per-client throughput falls to 92.27 tok/s at batch 15. A V2 dual-cache deadlock was fixed before the batch-15 trials completed.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
codinglong-context
User reports Qwen3.8-27B NVFP4 on an RTX 5090 decoding at 267-282 tok/s on a mixed short-prompt set with the DFlash2 block drafter, against 192 tok/s for the recipe's MTP-10 and 77 tok/s with speculation off.
Setup is NInfer with NVFP4 weights and speculative decoding; the same campaign also measured llama.cpp, Ollama and vLLM, and a 5080 16 GB desktop for comparison.
The 450 tok/s NVFP4 recipe figure reproduced at 426-457 tok/s on a recall workload that copies from the prompt, while ordinary prose decodes at 113-141 tok/s; 8 concurrent requests totalled 673 tok/s with MTP-3.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
codingmathagentictool-usevision
User reports Qwen3.8-27B at 94 t/s on an RTX 5090.
Setup is llama.cpp with IQ4_XS GGUF quantization, single stream, flash attention on, thinking mode sampling.
With MTP speculative decoding it reaches 147 t/s (1.57x) and with DFlash2 167 t/s (1.78x). The model is 13.1 GB at 3.84 bits per weight.