llamaperf

Qwen3.8 27B

on NVIDIA RTX 5090 · VoidInfer · 4,096 ctx

Tone: positiveNVIDIA hardware
Oct 24, 2026
Throughput
147.0 t/s gen
Quant
EXL3 6 bpw (EXL3)
KV cache
INT4
VRAM reported
32 GB

Use cases

agenticlong-context

Summary

User reports Qwen3.8-27B at 147 tok/s decode on one RTX 5090 under Windows 11, using their own VoidInfer engine. Setup is EXL3 6 bpw weights with OSCAR INT4 KV cache and VeriCache verification against FP16 KV history kept in host RAM, giving roughly 400k shared context on the single card. Against NInfer (NVFP4 + FP8 KV) on the same card, VoidInfer is about 25% slower decode (147 vs 190 tok/s) and much slower prefill (3.5K vs 9.3K tok/s at 3K prompt), but claims KL to BF16 of 0.014 vs 0.080 and 99.3% vs 96.0% agreement with BF16.