Qwen3.8 27B
on NVIDIA RTX 5090 · VoidInfer · 4,096 ctx
Use cases
agenticlong-context
Summary
User reports Qwen3.8-27B at 147 tok/s decode on one RTX 5090 under Windows 11, using their own VoidInfer engine.
Setup is EXL3 6 bpw weights with OSCAR INT4 KV cache and VeriCache verification against FP16 KV history kept in host RAM, giving roughly 400k shared context on the single card.
Against NInfer (NVFP4 + FP8 KV) on the same card, VoidInfer is about 25% slower decode (147 vs 190 tok/s) and much slower prefill (3.5K vs 9.3K tok/s at 3K prompt), but claims KL to BF16 of 0.014 vs 0.080 and 99.3% vs 96.0% agreement with BF16.