ExLlamaV2 benchmarks
An NVIDIA-focused runtime for EXL2-quantized models, designed for single-stream speed on one GPU.
0 community reports
ExLlamaV2 was written to get as much single-stream speed as possible out of NVIDIA cards. It uses its own EXL2 quantization format, which lets a model be quantized to any average bit width, and its own CUDA kernels.
It covers fewer model architectures than llama.cpp, and EXL2 files work only with it. ExLlamaV3 and its EXL3 format are the newer generation.
ExLlamaV2 against other engines
Pairs of plain runs on the same card, of the same model size at the same bit class: one device, one request, no speculative decoding, the whole model in memory. Context length and build still differ between the two sides, and each side shows its own.
No matched pair yet. No card has plain runs of one model size at one bit class on ExLlamaV2 and on another engine, so llamaperf can't say how it compares on speed. Add a run.
Frequently asked
Is ExLlamaV2 faster than other engines?
llamaperf has no matched comparison for ExLlamaV2 yet: no card has plain runs of the same model size at the same bit class on ExLlamaV2 and on another engine. Speed claims about engines need that pairing, so this page doesn't make one.