Local model performance on your hardware

generation:

71.0 tokens/s

prompt processing (prefill):

861.5 tokens/s

kv:

INT8

User reports Qwen3.8-27B at 71.00 tok/s decode on one RTX 3090 24 GB. Setup is NInfer v0.6.1 with INT8 KV cache, ReplaySSM and MTP3 speculative decoding, CUDA Graphs, 8,192-token context, 19.6 GB VRAM used. Decode by cohort: C2 90.66, C4 100.28, C8 165.33 aggregate tok/s. Prefill with 4,362 fresh input tokens per request: C1 861.51 tok/s (TTFT 4,893 ms), C8 844.10 tok/s. MTP3 acceptance 61.13%, C1 TTFT 149 ms.

generation:

70.2 tokens/s

User reports Qwen3.8-27B at 70.19 tok/s end-to-end (71.00 tok/s decode) for a single request on an RTX 3090. Setup is the RTX 3090 port of NInfer, a C++/CUDA inference runtime, with MTP3 speculative decoding at 61.13% acceptance, CUDA Graphs enabled and 1,024 output tokens per request, peaking at 19,641 MiB VRAM. At eight concurrent requests decode reaches 165.33 tok/s. Qwen3-35B-A3B runs at around 260 tok/s single-stream on the same setup, and over 400 tok/s on highly repetitive workloads.