Local model performance on your hardware

generation:

304.0 tokens/s

prompt processing (prefill):

12500.0 tokens/s

quant:

NVFP4 (NVFP4)

kv:

FP8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B at about 304 t/s decode at 8K context on one RTX 5090, with 221 t/s at 60K and prefill about 12,500 t/s at 60K. Setup is a vLLM main build newer than 0.27.1 with a custom NVFP4 checkpoint, DFlash2 drafter with 6 speculative tokens, fp8 KV cache, max model length 195,000 and GPU memory utilization 0.92. Two concurrent slots get roughly 255-299 t/s end to end. An earlier vLLM 0.27.1 setup with qwen3_5_mtp and 3 speculative tokens reached about 123 t/s at 8K and 90 t/s at 60K, with aggregate throughput rising from 67 t/s at one stream to 579 t/s at 16 streams. NVFP4 has a small quality cost versus GGUF Q4_K_M (q_avg 92.5 vs 93.2; HumanEval 89.6 vs 94.5). The GPU draws roughly 380-520 W under vLLM versus about 120 W with llama.cpp.

generation:

97.2 tokens/s

quant:

MXFP4 (MXFP4)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports DeepSeek V4.1 Flash at 97.23 tok/s for a single interactive request on eight RTX 5090 32GB GPUs over PCIe with no NVLink. Setup is vLLM with MXFP4 quantization, 32,768-token context, TP8 plus expert parallelism, and 9-11 GiB/rank of decoder expert offload to host RAM (503 GiB installed, about 264-279 GiB shared). Aggregate output reaches 247.77 tok/s at eight concurrent requests and 368.55 tok/s at 128 concurrent text requests; 8K prefill at two concurrent requests is 3,145 tok/s. The checkpoint is about 476 GiB and its quantization is unchanged.

generation:

81.3 tokens/s

prompt processing (prefill):

3160.0 tokens/s

quant:

FP8 (FP8)

kv:

fp8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B Uncensored at 81.3 tok/s generation on an RTX 5090 plus RTX 4070 Ti Super. Setup is vLLM 0.30.0 with block-FP8 weights, fp8 KV cache, full 262,144 context, MTP K=3, and pipeline parallel across both cards. Prefill is 3,160 tok/s at 32K, 68.5-68.8 s for 145K fresh prefill, and 169 s for 258K prefill; MTP acceptance is about 87% and the 5090 peaks at 31,472 MiB.

generation:

77.2 tokens/s

quant:

NVFP4 (safetensors)

kv:

FP8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B Uncensored at 77.2 t/s decode after a 1K prompt and 64.7 t/s with 128K resident on a single RTX 5090. Setup is vLLM 0.27.1 with an NVFP4 ModelOpt export, FP8 KV cache, prefix caching, vision and tool calling, at a 262,144-token context; the 19.18 GiB checkpoint leaves about 1.6 GiB free VRAM with KDE and Firefox running. Prefill falls from 7,005 t/s at 8K to 1,578 t/s at 262K, and a 262,000-token prefill took 166 seconds. A shared 36,864-token prefix cut TTFT from 6.437 s to a 0.288 s median, a 22.3x speedup. The 128K and 262K rows are single runs.

generation:

173-179 tokens/s

quant:

NVFP4

kv:

fp8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8 27B at 173 to 179 t/s decode on an RTX 5090, with 0.79s first token at about 8K prompt and four concurrent workflows done in 8.15s. Setup is vLLM with NVFP4 weights, fp8 KV cache, 262,144 context, MTP speculative decoding with 3 tokens, PIECEWISE cudagraphs, and prefix caching. The same 27B family also runs on a Radeon AI PRO R9700 32GB via Radiance in MXFP4 at 71 to 73 t/s decode, and on the 5090 via ExLlamaV3 at 40 to 70 t/s decode and 600 to 1400 t/s prefill at full 262K depth.

User reports 421 tok/s at 25k context on an RTX 5090 with muse-glimmer 30B, using a v2.2 optimizer profile for vLLM. Setup runs without DFlash speculative decoding, with TTFT at 224ms (p95 500ms) and TPOT at 23ms. Cost drops from $3.41 to $0.65 per 1M output tokens and energy from 4.72 to 1.08 J/tok. The user mentions SWE-Bench agents.

User benchmarks Gemma4-12B on an RTX 5090, comparing vLLM stock, vLLM+Emmy, and llama.cpp. The runs report throughput and latency across various token lengths and concurrency levels. The comparison also covers speculative decoding (MTP) tests and kernel optimization details.

generation:

15.0 tokens/s

prompt processing (prefill):

800.0 tokens/s

kv:

fp8_ds_mla

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports full 1M context on a single RTX 5090 with CPU/RAM offloading, at about 800 t/s prefill and over 15 t/s decode. Setup uses DSpark speculative decoding with dynamic acceptance rates, with two routed MoE layers kept GPU-resident.

generation:

13.8 tokens/s

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports DSpark K1/K2 at 13.78 t/s for reasoning and 17-18 t/s for content on a single RTX 5090 at full 1M context. Setup is a vLLM fork, guqiong96/Lvllmds4-x reporting vLLM 2.3.9, with phase-adaptive K1/K2 dual CUDA graphs and routed MoE experts in system RAM.

quant:

FP8

User benchmarks 200 prompts at max concurrency 100, reaching 15,058.74 tok/s total token throughput and 1,673.19 tok/s output token throughput, with a peak output of 3,000 tok/s. Setup is vLLM with Model Runner V2, p2p enabled, and tensor parallel 2. Mean TTFT is 1,741.86 ms and mean TPOT is 45.74 ms. An AMD Radeon PRO W6600 is mentioned but not used for this benchmark.

User reports KLD against BF16 for custom EXL3 quants of Qwen3.8-27B: 0.007406 for EXL3, 0.013126 for FP8, and 0.094978 for NVFP4. The quants require a custom vLLM fork, Gilded Gnosis.

generation:

59.0 tokens/s

quant:

5.5-bit (compressed-tensors)

kv:

fp8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8 27B at 59 t/s single stream on an RTX 5090, with aggregate throughput above 100 t/s for two requests and 279 t/s with four parallel jobs. Setup is vLLM 0.27.1 with the PrismaAqua 5.5-bit quant. The user also mentions the RTX 6000 Pro as target hardware. Quality is close to BF16 on a custom suite. The user prefers medium reasoning over xhigh because of fewer token-cap failures.

generation:

120.0 tokens/s

quant:

NVFP4

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports an average of 120 t/s across 3 parallel sessions. Setup is a 451K global KV cache with vision enabled and power limited to 400W. Detailed benchmarks are given in a table.

User benchmarks abliteration tools Apostate, Huihui and Heretic on Qwen 2.5 7B on an RTX 5090 32GB. Setup is vLLM 0.19.0 in bf16, evaluated with lm-evaluation-harness. The run reports MMLU, GSM8K, HellaSwag, ARC Challenge, WinoGrande, TruthfulQA MC2, PiQA, LAMBADA ppl, HarmBench ASR and KL divergence. No tokens/sec is reported.

generation:

578.0 tokens/s

quant:

AWQ-4bit

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports 228 t/s baseline and 578 t/s with DFlash speculative decoding, a 2.56x speedup, on vLLM 0.19.2rc1. Setup uses the z-lab/gemma-4-26B-A4B-it-DFlash draft model, with 256 input tokens and 1024 output tokens.

generation:

73.6 tokens/s

prompt processing (prefill):

2883.0 tokens/s

quant:

NVFP4 (safetensors)

kv:

Q8

flash attention:

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports generation speed varying 59-111 tok/s with MTP enabled and 3 speculative tokens. Setup uses an fp8_e4m3 KV cache, with prefix caching tested. Mean MTP acceptance length is 2.28. A stability pass at 200k context completed 10/10 runs.

generation:

105-108 tokens/s

quant:

INT4 (safetensors)

kv:

Q8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.6-27B-INT4 at 105-108 t/s generation on an RTX 5090 at 256k context. Setup is vLLM 0.19 with the Lorbus AutoRound quant, an fp8_e4m3 KV cache, flashinfer attention, and MTP speculative decoding with 3 tokens.

generation:

80.0 tokens/s

quant:

NVFP4 (safetensors)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.6-27B at ~80 t/s on an RTX 5090 at 218k context. Setup is vLLM 0.19.1rc1 with NVFP4 quantization on a single card.