Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User compares Qwen3.8 27B Q8 with q8_0 KV cache at 262k context on two RTX 5090s across two systems. On a 9950X with 64GB DDR5 6000, infill was 2500-3000 t/s and output 100-150 t/s. After moving the cards to a Proxmox VM on a 7402P with 256GB DDR4 3200 ECC, infill rose to 3800 t/s but output fell to around 60 t/s.
Setup is Unsloth Studio with Q8 weights and q8_0 KV cache, two 5090s passed through to the VM.
User asks what could cause the output drop and mentions memory bandwidth tests showing 90-120 GB/s on the 4-CCD CPU versus a theoretical 204 GB/s.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8-Flash-Next Uncensored NVFP4 at a median 50.5 t/s generation on one RTX 5090, measured at the maximum input of 259,601 tokens.
Setup is a modified FreeToken engine following the llama-split-bench protocol, with 96 GB system RAM and 1,000 tokens generated per test.
Across 3 rounds and 36 completed measurements, the median over 9 short practical prompts was 55.4 t/s.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8-Flash-Next at 48.02 t/s short decode and 1429 t/s prefill on an RTX 5090 32GB with 128GB DDR5 system RAM.
Setup is llama.cpp PR #27742 with UD-Q2_K_XL (78.9GB in 3 shards), 64k context, -ngl 999 and -ncmoe 24 keeping the first 24 layers' experts on the CPU, no speculative decoding, single stream.
Decode after about 16k tokens of input is 43.28 t/s, time to first token is 11.0 s at about 16k input, and VRAM use is about 30.9GB.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8 27B at 98.03 t/s generation and 2829.27 t/s prompt eval on an RTX 5090.
Setup is llama.cpp with UD-Q6_K_XL GGUF, 48000 context, draft-mtp speculative decoding (acceptance 0.47494, mean len 2.42).
User notes only 48k context fits on the RTX 5090 with this quant.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
coding
User compares Qwen3.8 27B and Qwen3.8 Flash-Next on a coding task using dual RTX 5090 GPUs. Qwen3.8 27B completed the task in 5-10 minutes with infill ~2000-3000 t/s and output 100-150 t/s. Flash-Next took 2.5 hours with infill ~1600-2300 t/s and output 60-100 t/s, and continued running tasks for ~2 hours after the last code modification. User asks why Flash-Next is slower and what settings would speed it up.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8-27B Uncensored at 81.3 tok/s generation on an RTX 5090 plus RTX 4070 Ti Super.
Setup is vLLM 0.30.0 with block-FP8 weights, fp8 KV cache, full 262,144 context, MTP K=3, and pipeline parallel across both cards.
Prefill is 3,160 tok/s at 32K, 68.5-68.8 s for 145K fresh prefill, and 169 s for 258K prefill; MTP acceptance is about 87% and the 5090 peaks at 31,472 MiB.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User ports DeepSeek-V4.1-Flash (552B) to a llama.cpp fork and runs it on a single RTX 5090 with 31.8 GiB VRAM and 125.7 GiB RAM, streaming routed experts from NVMe through a VRAM cache and pinned host tier. Reports 5.1 tokens/s on new content and 21 tokens/s on content already resident in the cache.
Setup uses MXFP4 experts as a lossless repack of the released blocks, with the engram tables (189 GiB of n-gram memory) kept on disk and read by the host. Logit correlation against the reference at 1401 tokens is 0.9967.
Measured ceilings are 6.2 tokens/s with no disk miss and 21.3 tokens/s with everything resident. Per remap call at a 72 GiB host tier: 20% compute, 26% PCIe, 54% NVMe. A prefetch oracle gives +30% only at 5 tokens of look-ahead; block verification with a perfect draft gives +17-20%; an n-gram draft loses 33% on realistic content; the DSpark draft head shows 51-97% acceptance but is neutral on mixed content.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
long-contextsummarizationtool-useagentic
User reports Qwen3.8-27B at 158 tok/s decode and 7,265 tok/s prefill on an RTX 5090 32GB (eGPU via OCuLink Gen4 x4).
Setup is NInfer with NVFP4 weights, FP8 KV cache, 240K context, MTP3 speculative decoding at 76% acceptance, and 2 concurrent lanes. Decode rises to 213 tok/s at 32K and 202 tok/s at 128K; prefill is 6,892 tok/s at 32K and 3,904 tok/s at 128K.
User compares against llama.cpp (Q5_K_M GGUF, q8_0 KV, 196K context, MTP on) at 114 tok/s decode and 1,545 tok/s prefill at 1K, and notes vLLM at about 70 tok/s decode at short context. Quality was statistically indistinguishable across engines on a 250+ item eval. NInfer does not support json_mode.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
long-contextvisiontool-useagentic
User reports Qwen3.8-27B Uncensored at 77.2 t/s decode after a 1K prompt and 64.7 t/s with 128K resident on a single RTX 5090.
Setup is vLLM 0.27.1 with an NVFP4 ModelOpt export, FP8 KV cache, prefix caching, vision and tool calling, at a 262,144-token context; the 19.18 GiB checkpoint leaves about 1.6 GiB free VRAM with KDE and Firefox running.
Prefill falls from 7,005 t/s at 8K to 1,578 t/s at 262K, and a 262,000-token prefill took 166 seconds. A shared 36,864-token prefix cut TTFT from 6.437 s to a 0.288 s median, a 22.3x speedup. The 128K and 262K rows are single runs.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User compares Qwen3.5-35B-A3B Q4_K_XL on an RTX 5090 against a Radeon AI PRO R9700, both 32 GB, using llama-bench with -ngl 99, flash attention and q8_0 KV cache.
On the RTX 5090 generation is 194.0 t/s and prompt processing is 7,026 t/s at 512 tokens, 6,953 at 1,024, 6,958 at 4,096 and 6,461 at 32,768.
The R9700 reaches 127.4 t/s generation and 2,713 / 2,643 / 2,546 / 1,877 t/s prompt processing at the same context lengths, a 1.52x generation and up to 3.44x prefill gap. The user notes these are single-stream llama-bench figures and calls the R9700 comfortable for interactive use at roughly 2.66x lower cost.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
codingagenticlong-contextvision
User reports Swift-1.5 Qwen3.8-27B at 160.8 t/s decode on a single RTX 5090 at 262,144 context.
Setup is the NInfer v3 engine with an all-NVFP4 (W4A4 gs16) artifact plus a z-lab DFlash2 drafter at K=7, k8v4 KV cache, 18.0 GiB of weights in VRAM, 450 W power cap, and concurrency 4.
Prefill at 200k context measured 3,269 t/s. IFBench prompt-strict 69.0, prompt-loose 72.7, instr-strict 70.4, instr-loose 73.6; GSM8K-200 95.0% (190/200); long-context needle at 250,031 tokens exact across 3 depths.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8 Flash-Next at approximately 20 tok/s single-stream decode over 256 output tokens on an RTX 5090 with 128 GB system RAM.
Setup is llama.cpp with GGUF UD-Q4_K_XL, with the model partly offloaded to system RAM.
With 4 parallel requests of roughly 4K prompt plus 256 output, aggregate end-to-end throughput is 13.8 tok/s; with roughly 8K prompt plus 256 output it is 8.81 tok/s. The user notes these aggregate figures are a different measurement from the single-request rate and should not be read as single-stream speed.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
codinglong-contexttool-use
User reports Qwen3.8-Flash-Next at 291.1 t/s single-stream decode on code and 273.5 t/s on prose, on two RTX 5090 32 GB GPUs with layer split [30, 30].
Setup is ExLlamaV3 v1.5.0 under TabbyAPI with a 2.50 bpw EXL3 pack, 8-bit KV cache, 262,144-token context and a 983,040-token page pool, using MTP draft depth 3 up to 4 jobs and 2 at 5 to 8 jobs.
Per-stream decode falls to 153.2 t/s on code and 156.8 t/s on prose at 4 streams, and 103.4 t/s on code and 103.6 t/s on prose at 8 streams. Aggregate decode across concurrent streams is 292 t/s code and 272 t/s prose at 1 stream, 615 t/s code and 627 t/s prose at 4 streams, and 822 t/s code and 831 t/s prose at 8 streams. Time to first token ranges from 0.13 to 0.14 s at 1 stream to 0.76 s at 8 streams. Cold prefill holds its rate to the top of the window: 199,844 tokens in 18.8 s and 240,047 tokens in 22.8 s. Quality runs on the same configuration give GSM8K 5-shot (n=500) 0.978, SWE-bench Verified 397/500 resolved (79.4%), and tool-eval-bench 84.0 +/- 2.4.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8 Flash-Next at 40 t/s decode on an RTX 5090 with 64GB system RAM.
Setup is llama.cpp with the Atomic Q4_K_M GGUF at 64,768 context, using mmap, lazy mode, flash attention, all GPU layers, and 32 MoE layers offloaded to CPU. The run used 27GB of VRAM and 8GB of system RAM.
User notes the prompt processing figure of 50 t/s is likely inaccurate due to the short prompt, and questions why the model is not buffering more into system RAM instead of reading from the NVMe drive.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
agenticcodinglong-context
User reports Qwen3.8 27B uncensored at 175 t/s on an RTX 5090 32GB.
Setup is NInfer with NVFP4 / groupwise-int quantization and MTP enabled, running at 262,144 context with a Q4 KV cache.
The user says the Q4 KV cache matched Q8 quality and that the longer context enabled long reasoning tasks in a Hermes harness.
User reports a native ninfer implementation of a Jev-like decision API running Qwen3.8 27B (Swift fine-tune) on an RTX 5090, scoring 84.4% (195/231) on the JevBench v1.2 public set with three in-flight decisions and a 19 s wall clock.
Setup drives the native POST /v1/decisions route directly rather than the chat-completions path; latency p50 is 127 ms and p95 is 699 ms, with hard-tier p95 prefill-dominated by ~3.7k-token states and tail latency inflated by a single serialized decision worker.
Calibration is ECE 0.045 and Brier 0.081; the user calls it a dirty POC with bugs and incorrect allocation still to fix, and notes latency is not great but accuracy is almost on par.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
summarizationlong-context
User reports Qwen3.8 27B at 97.9 t/s decode and 2,364 t/s prefill on an RTX 5090 at 61,136 tokens of context.
Setup is llama.cpp with the Unsloth UD-Q6_K GGUF, flash-attn on, q8_0 KV cache, batch 2048, ubatch 1024, and MTP speculative decoding with draft 3.
A second run at 97,374 tokens averaged 86.0 t/s decode and 1,929 t/s prefill. The user also tested Qwen3.8-Flash-Next Q4 across both GPUs and Qwen3.8 27B Q8 split across both cards at 262k context.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8 Flash-Next at 50 t/s generation and 2300 t/s prompt processing on a single RTX 5090 with 128 GB DDR5.
Setup is FreeToken with the nvidia/Qwen3.8-Flash-Next-NVFP4 model converted to FreeToken format, expert caching enabled, max sequence length 229376, and max running requests 2.
Generation speed ranges from 40 to 60 t/s depending on cache effectiveness. User notes llama.cpp lacks MoE expert caching, and FreeToken is early with no KV quantization or MTP yet.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
coding
User benchmarks Ternary Bonsai 2 27B at ~101 t/s on an RTX 5090 32GB, generating 106,396 tokens in ~17.5 minutes.
Setup is a PrismML llama.cpp fork (release prism-b10683-d8f26ee, CUDA 12.8) with the PQ2_0 GGUF at 7.2 GB, 262,144 context, flash attention on, and reasoning effort xhigh.
User compares it against Qwen 3.5 9B Q6_K at ~168 t/s and Gemma 4 12B Q8_0 at ~96 t/s on the same pagoda prompt, and rates Bonsai's output substantially better.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8 27B NVFP4 at 178 t/s on an RTX 5090 after undervolting the GPU and capping CPU power at 50%.
Setup uses the Ninfer inference engine. GPU memory was overclocked by 2400 MHz and the GPU was undervolted; CPU power was limited to 50% with no measurable TPS loss. A systemd service polls GPU usage and applies the CPU cap automatically during inference.
Stock settings gave 172 t/s in synthetic tests; undervolting and VRAM overclocking raised this to 178 t/s, and DFlash2 speculative decoding currently yields 200 t/s. GPU power dropped from 600 W to under 450 W and GPU temperature from 75°C to 62°C.