Mistral 7B
on NVIDIA RTX 4090 · TensorRT-LLM
Summary
User benchmarks Mistral 7B at 170.63 t/s on an RTX 4090 with TensorRT-LLM, compared to 100.43 t/s with llama.cpp.
Setup is TensorRT-LLM 0.7.1 with AWQ INT4 quantization, batch size 1, input length 2048, output length 512, on a single RTX 4090 with 32GB system RAM.
TensorRT-LLM was 69.89% faster than llama.cpp on the same hardware. The post also reports results for RTX 3090 (144.19 t/s), RTX 4070 Laptop (51.57 t/s), and RTX 4090 eGPU (104.95 t/s).