Unknown family 27B
on NVIDIA V100 32GB · ExLlamaV3
Summary
User reports 14.7 tok/s end-to-end on a 27B EXL3 SC pack on a V100-DGXS-32GB, matching the vLLM-plugin path within noise.
Setup is ExLlamaV3 with EXL3 6.00bpw mixed K 5-8 weights on a single device.
The PR ports EXL3 quantized-matrix kernels to Volta (sm_70), adding a GEMV decode kernel, arch gating, and reference tests. Single-device GEMV decode is 0.98-2.12x vs the hgemm floor across K2-4 shapes, with 664 GB/s aggregate weight-read at K=6 decode.