Local model performance on your hardware

generation:

27.0 tokens/s

User reports Qwen3.8 27B at about 27 t/s on dual RTX 5060 Ti 16GB with no speculation (no MTP, DFlash, n-gram or DSpark). Setup is vLLM nightly 0.32.1rc1 under WSL2 on Windows 11, with a Ryzen 5 3600 and 32GB DDR4; the second card sits in the first M.2 slot via an adapter. On a second machine (Ryzen 7 7800X3D, 64GB DDR5) the same run gave about 40 t/s, rising to 44.4 t/s after NCCL tuning flags; the user says NCCL was effectively disabled without them, since disabling it entirely reproduced the 27 t/s.

prompt processing (prefill):

2500-3000 tokens/s

quant:

FP8 (FP8)

User reports running Qwen3.8 27B with tensor parallelism across 4x RTX 5060 Ti 16GB in a Lenovo P620 (Threadripper PRO 3975WX, 128GB DDR4). Setup is vLLM with FP8 weights and MTP3 speculative decoding, headless Ubuntu, NCCL 2.30.7. Decode at C1 is 100-110 tokens/s and prefill is 2.5-3k tokens/s per stream. User compares NVFP4+MTP3 (137-141 t/s C1, 422 t/s C4 sum, ~3.5k prefill), SGLang NVFP4+DFlash (118-137 t/s C1, 357-386 t/s C4 sum, ~3.4k prefill), and llama.cpp Q6+MTP3 (95 t/s C1, 83 t/s C4 sum, ~1.2k prefill). NVFP4 was 1.4x faster than FP8 but degraded in MTP3 tests (F1 0.44 vs 0.73).

generation:

132.0 tokens/s

prompt processing (prefill):

4000.0 tokens/s

quant:

NVFP4

kv:

FP8

mtp (multi-token prediction):

on

User reports Qwen 3.8 27B NVFP4 on dual RTX 5060 Ti 16GB (32GB total VRAM) at 132 t/s decode for structured output and 72 t/s for prose, with ~4,000 tok/s prefill. Setup is stock vLLM 0.30.0 on headless Ubuntu with FP8 KV cache, full 262,144-token context, and MTP speculative decoding. GPUs are power limited to 150W with no throughput cost. Video input is disabled; the user notes this runs without an RTX 5090.

generation:

67.3 tokens/s

prompt processing (prefill):

952.5 tokens/s

quant:

NVFP4

kv:

FP8

mtp (multi-token prediction):

on

User compares NVFP4 on vLLM against Q6_K on llama.cpp, both on 2x RTX 5060 Ti. NVFP4 reaches 952.5 t/s prefill and 67.3 t/s decode with 0.696 MTP acceptance. Q6_K reaches 597.9 t/s prefill and 38.6 t/s decode. On quality, NVFP4 scores 65/68 correct, 64/68 exact format and 61/68 both, while Q6_K scores 66/68, 68/68 and 66/68. Both score 10/10 on long-context retrieval. NVFP4 is the default endpoint, with Q6_K kept for stricter output.