ExLlamaV3 benchmarks
An inference engine for running open-weight LLMs locally.
18 community reports
ExLlamaV3 results by GPU
Hardware with reported ExLlamaV3 results, with each report's model, quant and speed, newest first. Runs on several cards, with speculative decoding, with batched requests or with part of the model in system RAM say so, since each describes a different setup.
ExLlamaV3 on NVIDIA RTX 30906 reports
- GLM-5.3 320B · 18B active · exl3-3.05bpw28.6 t/s(part in system RAM)
- GLM-5.3 320B · 18B active · 3.05bpw17.3 t/s, 844 t/s prompt(part in system RAM)
- Qwen3.8 27B · exl3-5.00bpw60.0 t/s
- Qwen3.8 27B · EXL3-4.00bpw162.9 t/s(speculative)
- Qwen3.8 125B · 6B active · 3bpw80.0 t/s, 1,500 t/s prompt(3 cards, speculative)
ExLlamaV3 on NVIDIA RTX 40903 reports
- Qwen3.8 125B · 6B active · 3.05bpw_h5_ng520.0 t/s(part in system RAM)
- Qwen3.8 27B · EXL3 4.00 bpw35.9 t/s
- Qwen3.8 125B · 6B active · 4.05bpw33.6 t/s, 1,944 t/s prompt(part in system RAM)
ExLlamaV3 on NVIDIA RTX 50903 reports
- GLM-5.3 320B · 18B active · 2.05bpw EXL364.8 t/s, 1,973 t/s prompt(2 cards, speculative, part in system RAM)
- Qwen3.8 125B · 6B active · 2.50bpw241.7 t/s(2 cards, speculative)
- Qwen3.8 125B · 6B active · 2.50bpw EXL3291.1 t/s(2 cards, speculative)
ExLlamaV3 on AMD Radeon AI PRO R9700 32GB1 report
- Qwen3.8 125B · 6B active · exl3-5.05bpw34.7 t/s, 863 t/s prompt(speculative, part in system RAM)
ExLlamaV3 on AMD RX 7900 XTX1 report
- Qwen3.8 27B · EXL3 3.5bpw157.0 t/s(speculative)
ExLlamaV3 on NVIDIA CMP 170HX 64GB (unlocked)1 report
- GLM-5.3 320B · 18B active · 3.05bpw95.6 t/s, 1,529 t/s prompt(2 cards, speculative)
ExLlamaV3 on NVIDIA DGX Spark1 report
- Qwen3.8 27B · EXL3 3.5bpw47.5 t/s(speculative)
ExLlamaV3 on NVIDIA RTX 3080 20GB1 report
- Qwen3.8 125B · 6B active · 4.05 EXL325.0 t/s, 870 t/s prompt(2 cards, part in system RAM)
ExLlamaV3 on NVIDIA RTX 40801 report
- Qwen3.8 27B · SC_3.00bpw_H4_V456.5 t/s, 980 t/s prompt(speculative)
ExLlamaV3 on NVIDIA RTX 5060 Ti 16GB1 report
- Qwen3.8 27B · exl3-5.0bpw50.0 t/s(2 cards, speculative)
ExLlamaV3 on NVIDIA RTX 6000 Ada1 report
- Mimo 2.6 309B · 15B active · 2.20 bpw49.6 t/s, 2,370 t/s prompt(speculative, part in system RAM)
ExLlamaV3, GPU not identified2 reports
This engine doesn't yet have an editorial profile on llamaperf. The community reports below show how it's been used in practice across different hardware.
Compare GPUs running ExLlamaV3
Every card is compared on one model, Qwen3.8 27B at 4-bit, the size and bit class that most cards here have plain runs of. A card with no run of that model shows none rather than a figure from a different model.
| GPU | VRAM | Reports | Median speed, Qwen3.8 27B 4-bit |
|---|---|---|---|
| NVIDIA RTX 3090nvidia | 24GB | 7 | no Qwen3.8 27B 4-bit run reported |
| NVIDIA RTX 4090nvidia | 24GB | 3 | Gen 35.9 t/s |
| NVIDIA RTX 5090nvidia | 32GB | 3 | no Qwen3.8 27B 4-bit run reported |
| AMD RX 7900 XTXamd | 24GB | 1 | no Qwen3.8 27B 4-bit run reported |
| AMD RX 9070 XT 16GBamd | 16GB | 1 | no Qwen3.8 27B 4-bit run reported |
| AMD Radeon AI PRO R9700 32GBamd | 32GB | 1 | no Qwen3.8 27B 4-bit run reported |
| NVIDIA CMP 170HX 64GB (unlocked)nvidia | 64GB | 1 | no Qwen3.8 27B 4-bit run reported |
| NVIDIA DGX Sparknvidia | 128GB | 1 | no Qwen3.8 27B 4-bit run reported |
ExLlamaV3 against other engines
Pairs of plain runs on the same card, of the same model size at the same bit class: one device, one request, no speculative decoding, the whole model in memory. Context length and build still differ between the two sides, and each side shows its own.
No matched pair yet. No card has plain runs of one model size at one bit class on ExLlamaV3 and on another engine, so llamaperf can't say how it compares on speed. Add a run.
Top models on ExLlamaV3
Frequently asked
Is ExLlamaV3 faster than other engines?
llamaperf has no matched comparison for ExLlamaV3 yet: no card has plain runs of the same model size at the same bit class on ExLlamaV3 and on another engine. Speed claims about engines need that pairing, so this page doesn't make one.