Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
codinglong-context
User reports 153.32 t/s at 32K context and 87.74 t/s at 200K context.
Setup uses a Q4 KV cache, speculative decoding with MTP and n-gram, and the vision encoder on the iGPU.
HumanEval scored 159/164 in 29.7 min.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8 27B averaging in the 170s t/s on an RTX 5090, peaking at 220 t/s.
Setup is ninfer-serve with the nvfp4 weights, MTP speculative decoding with 3 draft tokens, lm-head-draft, an fp8 KV cache and 240,000 context.
User calls it double or more the throughput of llama.cpp.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
long-contexttool-useagentic
User compares NInfer, llama.cpp and vLLM for Qwen3.8-27B on an RTX 5090. NInfer with NVFP4 and MTP3 reaches 158-213 t/s decode and 3904-7265 t/s prefill.
NInfer outperforms llama.cpp with Q5_K_M by 1.4-2.8x decode and 2.6-4.7x prefill.
Quality is statistically indistinguishable across engines. NInfer lacks json_mode support, and vLLM speed is not directly comparable due to wall-clock timing.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
codingagentictool-use
User reports sustained 80-90 t/s decode at full 262K context.
Setup uses MTP speculative decoding and prefix caching, with a 96.8% prefix cache hit rate and mean TTFT of 6.5s.
Multi-session throughput reaches about 3200 tokens/min across concurrent chats, and the benchmark ceiling is 106.9 t/s.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8 27B at ~25 tok/s with standard autoregressive decoding (MTP0) on a modded RTX 2080 Ti 22GB.
Setup is a Turing port of the NInfer engine with the official groupwise-int (W8A16) artifact and Q8 KV cache.
With MTP3 speculative decoding (draft window = 3) the title gives 45 tok/s and the body ~456 tok/s, at ~65% acceptance rate. VRAM usage is ~17.5 GiB with the MTP draft weights loaded, leaving ~4.5-5.0 GiB free for KV cache.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
codingagentic
User reports DFlash2 speculative decoding with LMCache KV offload, with decode speeds varying by task: code 268 t/s, agent 165 t/s, mixed 97 t/s, prose 76 t/s.
Prefill runs at 15-17k t/s, and the KV cache pool holds 18M tokens on NVMe L2.
The setup required patches to avoid corruption.
User reports running GSM8K, MATH-500, HumanEval and MBPP locally.
User found reasoning stalls causing significant accuracy drops on hard math.
User notes the HumanEval score is possibly inflated due to contamination, and that harness bugs were fixed.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports prose decode at ~144 t/s and code at ~256 t/s, with parallel x2 at ~261/451 t/s.
Prefill TTFT runs from 0.33s at 5k to 23.46s at 150k, with a 175k KV pool and a host-RAM KV tier that resumes a ~100k conversation in ~1s.
External evals give GPQA Diamond 84.8%, GSM8K 96.8%, MATH-500 95.6%, AIME 2024 83.3%, HumanEval 56.7% and MBPP 75.0%. The run uses the froggeric template with xhigh reasoning and a 16k think cap.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports 80-120 t/s generation and 12k t/s prefill for a single request on 4x R9700.
Setup is an optimized vLLM image with tcclaviger's MXFP4-FP8 quant, tensor-parallel-size 4, fp8 KV cache, and speculative decoding with MTP.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User benchmarks DFlash2 speculative decoding against MTP on Qwen3.8-27B.
DFlash2 gives 20% faster generation but reduces context size by 38%.
The prompt and generation TPS figures are weighted averages.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports decode speed rising from 18 tok/s to 58 tok/s with DFlash2 + XQA versus Flash2 + FA2 at 192K context.
Lossless BF16 weights and KV.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8-27B on an RTX 5060 Ti 16GB with llama.cpp, comparing the jpetrina IQ4_XS-pure and Unsloth UD-IQ4_XS quants.
Short-context generation runs at roughly 45-47 tok/s, dropping to about 31 tok/s after 55K prefill. The Unsloth quant has better fidelity (KLD 0.018 vs 0.0236), while jpetrina wins the multi-turn agent eval (40/80 vs 38/80).
Vision works but needs a separate profile. A Q8 baseline was used as a quality control.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User benchmarks Qwen3.8-27B and Gemma4-26B-A4B on an AMD V620 using llama.cpp with ROCm and Vulkan.
Qwen results are shown; Gemma results appear in a table but are not extracted as the primary model.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User benchmarks DFlash 2 speculative decoding on Qwen 3.8 27B with a llama.cpp PR build on an RTX PRO 6000 Blackwell 96GB and a Ryzen 9 9950X.
Setup is llama.cpp with 262144 context, an f16 KV cache and full offload.
The user reports 2.26x speedup on LiveCodeBench, from 67.97 to 153.91 tok/s, and 4.68x on multi-turn coding with n-gram lookup, and gives a detailed analysis of n-gram drafters and draft width.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User compares llama.cpp built with -DGGML_CUDA_NCCL=OFF against the same build with it ON on 2x RTX 3090.
Setup is the unsloth/Qwen3.8-27B-GGUF UD-Q6_K_XL model at 229k context with an F16 KV cache, tensor split 24,24, and speculative decoding with draft-mtp.
NCCL ON degrades performance: PP 1046.92 vs 958.58 t/s and TG 59.67 vs 54.80 t/s.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
agenticcoding
User benchmarks 180K context with MTP5 enabled, reaching 2587 t/s prefill and 200.5 t/s decode.
The run also covers 64K and 120K contexts.
A synthetic corpus yields high MTP acceptance, while real agentic coding logs show about 51% accept and about 154 t/s.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User compares two llama.cpp setups on the same GPU: Q5 with a Q8 KV cache, 32K context, 4 slots and MTP off at about 35 t/s, against Q4 with a Q4 KV cache, 131K context, 1 slot and MTP on at about 65 t/s.
User notes the trade-offs between the two configurations.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8-27B at 560/30 t/s on 2x RTX 3090.
Setup is Q8_0 quantization, run fully in VRAM.
The user calls it the local king and praises its quality. Other models mentioned are DeepSeek-V4-Flash, GLM-5.3-Flash, and Qwen3.8-Flash-Next.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
coding
User reports an average of 120 t/s across 3 parallel sessions.
Setup is a 451K global KV cache with vision enabled and power limited to 400W.
Detailed benchmarks are given in a table.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports 67 t/s with MTP3 speculative decoding enabled, against 24.4 tok/s without MTP.
Setup uses an INT8 KV cache with group-64.
The figure comes from a 128K NIAH benchmark with 130,048 prompt tokens, where MTP acceptance was 100% on a deterministic answer. Roughly 727 MiB of VRAM was left.