Local model performance on your hardware

generation:

96.5 tokens/s

kv:

q4

User reports Qwen 3.8 27B at a median 96.45 t/s decode on a 16GB RTX 5080, with a range of roughly 90-110 t/s during coding tasks. Setup is NInfer v1.5 with q4 KV cache, MTP speculative decoding with 3 draft tokens, 110,592 context allocated, and 11.86 GiB of weights leaving about 498 MiB of slack. Across 32 completed requests the lowest decode was 84.6 t/s and the highest 131.7 t/s, with median time to first token 1.4 seconds. About 68% of generated tokens were reasoning tokens, and one request that lost its prompt cache took almost 50 seconds to process a 79k prompt before generating at about 94 t/s.

generation:

71.6 tokens/s

prompt processing (prefill):

1380.6 tokens/s

quant:

Q3G64_F16S/Q4G64_F16S/Q5G64_F16S (ninfer)

kv:

Q4 group64

User reports Qwen3.8-27B at 71.57 t/s decode on a single RTX 5080 16GB at 131,072 context. Setup is NInfer v1.3 with mixed Q3/Q4/Q5 groupwise quantization (~3.95 BPW), Q4 group64 KV cache, MTP-3 speculative decoding, and Vision enabled. The 118,001-token prompt prefilled at 1380.61 t/s with 44.74% MTP acceptance. A short 512-token request averaged 84.7 t/s decode, and a video test reached 97.1 t/s decode with 100% MTP acceptance.