llama.cpp on the NVIDIA V100 16GB
llamaperf has 3 llama.cpp reports on the NVIDIA V100 16GB, none of them a plain single-card run with a model size and quant, so there is no median to quote. The reports are listed below with how each one ran.
3 llama.cpp reports on the NVIDIA V100 16GB
Newest first. A run on several cards, with speculative decoding, with part of the model in system RAM or with batched requests says so, since each describes a different setup.
- MiniCPM-V 4.7 35B · 3B active · mix_q3gud_im93.0 t/s
- Qwen3.8 125B · 6B active · IQ3_XXS50.0 t/s(2 cards)
- Qwen3.8 27B · Q6_K_M39.9 t/s(2 cards)
Frequently asked
How fast is llama.cpp on the NVIDIA V100 16GB?
llamaperf has 3 llama.cpp reports on the NVIDIA V100 16GB, none of them a plain single-card run with a model size and quant, so there is no median to quote. The reports are listed below with how each one ran.