Local model performance on your hardware

generation:

234.4 tokens/s

prompt processing (prefill):

8000-9300 tokens/s

quant:

NVFP4 (GGUF)

User compares TensorSharp, llama.cpp, vLLM, SGLang, velox-inferer and NInfer running Qwen3.8 27B on one RTX 5090 32 GB. TensorSharp averages 234.4 t/s decode with DFlash2 speculation across code, prose and copying, against 80.0 t/s plain. Setup is TensorSharp's direct-CUDA backend with an NVFP4 GGUF target and Q4_K_M drafter, 40,960-token context, greedy decoding and thinking disabled. DFlash2 uses seven draft tokens per step. NInfer leads the speculative decode average at 280.5 t/s and four-request aggregate throughput at 488 t/s; llama.cpp has the lowest short-prompt TTFT at 27 ms plain. The author reports up to roughly 4% decode variation between runs and notes the engines used different quantized checkpoints.

generation:

80.0 tokens/s

quant:

NVFP4 (GGUF)

User compares six engines running Qwen3.8 27B on a single RTX 5090 32GB, reporting a plain-mode three-workload decode average of 80.0 t/s for TensorSharp rising to 234.4 t/s with DFlash2 speculation. Setup is TensorSharp's direct-CUDA backend with an NVFP4 GGUF target and Q4_K_M drafter, 40,960-token context, thinking disabled and vision enabled; vLLM and SGLang used NVFP4/FP8 safetensors, NInfer and velox a mixed NVFP4/FP8 .ninfer target. DFlash2 raises summarized single-request decode roughly 2.5-3.6x across engines; NInfer leads speculative decode at 280.5 t/s average and four-request throughput at 488 t/s, while llama.cpp has the lowest short-prompt TTFT at 27 ms. The author notes up to about 4% decode variation between runs and that quantized targets and cache formats differ between engines.