Local model performance on your hardware

generation:

304.0 tokens/s

prompt processing (prefill):

12500.0 tokens/s

quant:

NVFP4 (NVFP4)

kv:

FP8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B at about 304 t/s decode at 8K context on one RTX 5090, with 221 t/s at 60K and prefill about 12,500 t/s at 60K. Setup is a vLLM main build newer than 0.27.1 with a custom NVFP4 checkpoint, DFlash2 drafter with 6 speculative tokens, fp8 KV cache, max model length 195,000 and GPU memory utilization 0.92. Two concurrent slots get roughly 255-299 t/s end to end. An earlier vLLM 0.27.1 setup with qwen3_5_mtp and 3 speculative tokens reached about 123 t/s at 8K and 90 t/s at 60K, with aggregate throughput rising from 67 t/s at one stream to 579 t/s at 16 streams. NVFP4 has a small quality cost versus GGUF Q4_K_M (q_avg 92.5 vs 93.2; HumanEval 89.6 vs 94.5). The GPU draws roughly 380-520 W under vLLM versus about 120 W with llama.cpp.

generation:

97.2 tokens/s

quant:

MXFP4 (MXFP4)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports DeepSeek V4.1 Flash at 97.23 tok/s for a single interactive request on eight RTX 5090 32GB GPUs over PCIe with no NVLink. Setup is vLLM with MXFP4 quantization, 32,768-token context, TP8 plus expert parallelism, and 9-11 GiB/rank of decoder expert offload to host RAM (503 GiB installed, about 264-279 GiB shared). Aggregate output reaches 247.77 tok/s at eight concurrent requests and 368.55 tok/s at 128 concurrent text requests; 8K prefill at two concurrent requests is 3,145 tok/s. The checkpoint is about 476 GiB and its quantization is unchanged.

generation:

81.3 tokens/s

prompt processing (prefill):

3160.0 tokens/s

quant:

FP8 (FP8)

kv:

fp8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B Uncensored at 81.3 tok/s generation on an RTX 5090 plus RTX 4070 Ti Super. Setup is vLLM 0.30.0 with block-FP8 weights, fp8 KV cache, full 262,144 context, MTP K=3, and pipeline parallel across both cards. Prefill is 3,160 tok/s at 32K, 68.5-68.8 s for 145K fresh prefill, and 169 s for 258K prefill; MTP acceptance is about 87% and the 5090 peaks at 31,472 MiB.

generation:

77.2 tokens/s

quant:

NVFP4 (safetensors)

kv:

FP8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B Uncensored at 77.2 t/s decode after a 1K prompt and 64.7 t/s with 128K resident on a single RTX 5090. Setup is vLLM 0.27.1 with an NVFP4 ModelOpt export, FP8 KV cache, prefix caching, vision and tool calling, at a 262,144-token context; the 19.18 GiB checkpoint leaves about 1.6 GiB free VRAM with KDE and Firefox running. Prefill falls from 7,005 t/s at 8K to 1,578 t/s at 262K, and a 262,000-token prefill took 166 seconds. A shared 36,864-token prefix cut TTFT from 6.437 s to a 0.288 s median, a 22.3x speedup. The 128K and 262K rows are single runs.

generation:

15.0 tokens/s

prompt processing (prefill):

800.0 tokens/s

kv:

fp8_ds_mla

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports full 1M context on a single RTX 5090 with CPU/RAM offloading, at about 800 t/s prefill and over 15 t/s decode. Setup uses DSpark speculative decoding with dynamic acceptance rates, with two routed MoE layers kept GPU-resident.

generation:

13.8 tokens/s

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports DSpark K1/K2 at 13.78 t/s for reasoning and 17-18 t/s for content on a single RTX 5090 at full 1M context. Setup is a vLLM fork, guqiong96/Lvllmds4-x reporting vLLM 2.3.9, with phase-adaptive K1/K2 dual CUDA graphs and routed MoE experts in system RAM.

generation:

59.0 tokens/s

quant:

5.5-bit (compressed-tensors)

kv:

fp8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8 27B at 59 t/s single stream on an RTX 5090, with aggregate throughput above 100 t/s for two requests and 279 t/s with four parallel jobs. Setup is vLLM 0.27.1 with the PrismaAqua 5.5-bit quant. The user also mentions the RTX 6000 Pro as target hardware. Quality is close to BF16 on a custom suite. The user prefers medium reasoning over xhigh because of fewer token-cap failures.

generation:

120.0 tokens/s

quant:

NVFP4

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports an average of 120 t/s across 3 parallel sessions. Setup is a 451K global KV cache with vision enabled and power limited to 400W. Detailed benchmarks are given in a table.

generation:

578.0 tokens/s

quant:

AWQ-4bit

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports 228 t/s baseline and 578 t/s with DFlash speculative decoding, a 2.56x speedup, on vLLM 0.19.2rc1. Setup uses the z-lab/gemma-4-26B-A4B-it-DFlash draft model, with 256 input tokens and 1024 output tokens.

generation:

73.6 tokens/s

prompt processing (prefill):

2883.0 tokens/s

quant:

NVFP4 (safetensors)

kv:

Q8

flash attention:

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports generation speed varying 59-111 tok/s with MTP enabled and 3 speculative tokens. Setup uses an fp8_e4m3 KV cache, with prefix caching tested. Mean MTP acceptance length is 2.28. A stability pass at 200k context completed 10/10 runs.

generation:

80.0 tokens/s

quant:

NVFP4 (safetensors)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.6-27B at ~80 t/s on an RTX 5090 at 218k context. Setup is vLLM 0.19.1rc1 with NVFP4 quantization on a single card.