Local model performance on your hardware

generation:

139.7 tokens/s

prompt processing (prefill):

9081.0 tokens/s

quant:

NVFP4 (NVFP4)

kv:

FP8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B at 139.7 tok/s on an RTX 5090 with SGLang and a D-Flash 2 drafter. Setup is SGLang with NVFP4 weights and an FP8 KV cache, single-user at concurrency 1. Other profiles on the same card: DSpark v2 drafter 110.2 tok/s, native MTP under vLLM 100.8 tok/s, and a vLLM baseline with no speculation at 84.5 tok/s. Prefill is about 9,081 tok/s on the 5090 against about 196 tok/s on an M4 Max 128GB running oMLX 4-bit MLX builds, where decode was 16.7 tok/s with speculation off, 35.5 tok/s with DFlash 2 greedy and 47.2 tok/s with DFlash 2 thinking on. A warm prefix cache cuts time to first token by about 90% on both machines.

generation:

152.0 tokens/s

quant:

NVFP4 (NVFP4)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B NVFP4 at 152 tok/s decode on one RTX 5090 under WSL2, with MTP speculative decoding and host embedding enabled. Setup is SGLang v0.5.20 with NVFP4 weights, 147,456-token context, on a Ryzen 9 9950X3D with 96 GB RAM. Plain 27B without speculation reached 93 tok/s at one stream (89.7-96.0) and 456 tok/s total at 5 streams; with MTP, 5 streams totalled 753 tok/s. Prefill of a 6K-token prompt was 13.4-15.0K tok/s plain and 9.2-12.4K tok/s with MTP. Energy was 2.91 J/token plain and 1.72 J/token with MTP.

generation:

100.0 tokens/s

quant:

NVFP4 (NVFP4)

kv:

fp8_e4m3

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8 27B at about 100 t/s on a single RTX 5090, but only about 82k tokens of usable context despite configuring 253,952. Setup is SGLang serving the Huihui-Abliterated NVFP4 MTP VL build with an fp8_e4m3 KV cache, NEXTN speculative decoding at 3 steps and 4 draft tokens, flashinfer attention, chunked prefill of 1024, and a 16 GB hierarchical cache. User asks how to tune the setup for more context without losing multimodal capability.

generation:

200.0 tokens/s

quant:

NVFP4

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B at over 200 t/s decode on an RTX 5090 and an RTX Pro 6000, and 38 t/s on a DGX Spark. Setup is NVFP4 with DSpark on day-0 support. The user also mentions an H200 in the cookbook link.

generation:

82.6 tokens/s

quant:

W2

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports the Escha 2-bit quant of Qwen3.8-27B, 10.15 GB on disk, averages about 100% of FP8 performance across 8 benchmarks. On LiveCodeBench v6, Escha W2 scores 86.81 against 85.16 for FP8. The previous 2-bit Qwen3.6-35B MoE scored 62.6 against 67.0 for FP8 on the same benchmark.

generation:

144.0 tokens/s

quant:

NVFP4 (safetensors)

kv:

FP8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports prose decode at ~144 t/s and code at ~256 t/s, with parallel x2 at ~261/451 t/s. Prefill TTFT runs from 0.33s at 5k to 23.46s at 150k, with a 175k KV pool and a host-RAM KV tier that resumes a ~100k conversation in ~1s. External evals give GPQA Diamond 84.8%, GSM8K 96.8%, MATH-500 95.6%, AIME 2024 83.3%, HumanEval 56.7% and MBPP 75.0%. The run uses the froggeric template with xhigh reasoning and a 16k think cap.