llama.cpp benchmarks

The C/C++ inference runtime behind the GGUF format for quantized open-weight LLMs.

398 community reports

llama.cpp results by GPU

Hardware with reported llama.cpp results, with each report's model, quant and speed, newest first. Runs on several cards, with speculative decoding, with batched requests or with part of the model in system RAM say so, since each describes a different setup.

llama.cpp on M5 Max 48GB38 reports

Model not named · Q4_K_M26.3 t/s, 685 t/s prompt
Qwen3.6 35B · 3B active · Q4_K_M124.3 t/s, 3,343 t/s prompt
Qwen3.5 9B · Q4_K_M86.5 t/s, 2,369 t/s prompt
Qwen3.5 4B · Q4_K_M135.6 t/s, 4,100 t/s prompt
Qwen3.5 35B · 3B active · Q4_K_M111.2 t/s, 3,378 t/s prompt

All llama.cpp results on the M5 Max 48GB Filter llama.cpp reports on the M5 Max 48GB

llama.cpp on M4 Max 128GB33 reports

Qwen3.8 27B · Q8_015.7 t/s, 256 t/s prompt
Qwen3.5 9B · Q8_048.4 t/s, 880 t/s prompt
Qwen3.5 4B · Q8_078.8 t/s, 1,568 t/s prompt
SmolLM3 3B · Q8_0119.7 t/s, 2,237 t/s prompt
SmolLM3 3B · Q4_K_M168.6 t/s, 2,143 t/s prompt

All llama.cpp results on the M4 Max 128GB Filter llama.cpp reports on the M4 Max 128GB

llama.cpp on NVIDIA RTX 3060 12GB29 reports

Gemma 4 31B · Q520.0 t/s(2 cards)
Qwen3.6 35B · 3B active · Q4_K_M79.0 t/s(speculative, part in system RAM)
Qwen3.8 · IQ4_XS12.9 t/s, 25 t/s prompt(2 cards)
Qwen3.8 27B · Q4_K_S30.8 t/s, 755 t/s prompt(2 cards)
Ornith1.5 35B · 3B active · IQ4_XS97.5 t/s, 1,870 t/s prompt(2 cards)

All llama.cpp results on the NVIDIA RTX 3060 12GB Filter llama.cpp reports on the NVIDIA RTX 3060 12GB

llama.cpp on NVIDIA RTX 5060 Ti 16GB29 reports

Qwen3.8 27B · IQ3_S40.0 t/s(speculative, part in system RAM)
Qwen3.8 125B · 6B active · IQ3_S15.0 t/s(part in system RAM)
Gemma 4 31B · QAT60.0 t/s, 1,100 t/s prompt(2 cards, speculative)
Bonsai 2300.0 t/s
Qwen3.8 27B · IQ3_XXS50.0 t/s, 900 t/s prompt(speculative)

All llama.cpp results on the NVIDIA RTX 5060 Ti 16GB Filter llama.cpp reports on the NVIDIA RTX 5060 Ti 16GB

llama.cpp on AMD RX 7900 XTX25 reports

Qwen3.6 35B · 3B active · Q4_K_M125.5 t/s, 1,875 t/s prompt
Qwen3.8 27B · Q4_K_M69.6 t/s, 383 t/s prompt(speculative)
Qwen3.8 27B · UD-Q4_K_XL37.1 t/s, 878 t/s prompt
Qwen3.8 27B · Q4_K_M78.5 t/s, 523 t/s prompt(speculative)
Qwen3.6 27B · IQ4_XS41.7 t/s, 1,076 t/s prompt(speculative)

All llama.cpp results on the AMD RX 7900 XTX Filter llama.cpp reports on the AMD RX 7900 XTX

llama.cpp on NVIDIA RTX 5070 Ti23 reports

Qwen3.8 27B · 4.2BPW54.7 t/s
Gemma 4 26B · 4B active · UD-IQ4_XS137.7 t/s, 5,951 t/s prompt
Qwen3.8 27B · IQ3_S107.0 t/s, 1,832 t/s prompt(speculative)
Qwen3.8 27B · UD-Q4_K_XL124.0 t/s(2 cards, speculative)
Gemma 4 31B · NVFP466.8 t/s(speculative)

All llama.cpp results on the NVIDIA RTX 5070 Ti Filter llama.cpp reports on the NVIDIA RTX 5070 Ti

llama.cpp on AMD Radeon AI PRO R9700 32GB17 reports

Qwen3.8 · Q6_K30.0 t/s(2 cards, part in system RAM)
Gemma 4 26B · 4B active · Q8_03,200 t/s prompt
Qwen3.8 27B · IQ4_NL29.8 t/s, 1,287 t/s prompt
Qwen3.8 125B · 6B active · UD-Q3_K_XL17.6 t/s, 617 t/s prompt(2 cards, speculative, part in system RAM)
DeepSeek V4 Flash 284B · 13B active40.0 t/s(8 cards)

All llama.cpp results on the AMD Radeon AI PRO R9700 32GB Filter llama.cpp reports on the AMD Radeon AI PRO R9700 32GB

llama.cpp on NVIDIA RTX 409016 reports

Qwen3.8 27B · UD-Q4_K_XL47.0 t/s
Qwen3.8 125B · 6B active · UD-IQ3_XXS30.0 t/s, 864 t/s prompt(part in system RAM)
Qwen3.5 122B · 10B active · UD-IQ4_XS18.8 t/s(2 cards, part in system RAM)
Qwen3.8 27B · Q4_K_M80.0 t/s
Qwen2.5 7B · Q4_K_M158.9 t/s

All llama.cpp results on the NVIDIA RTX 4090 Filter llama.cpp reports on the NVIDIA RTX 4090

llama.cpp on NVIDIA V100 32GB16 reports

Qwen3.8 27B · Q4_K_M65.1 t/s(speculative)
Qwen3.8 27B · UD-Q4_K_M57.3 t/s, 720 t/s prompt(speculative)
Qwen3.8 27B · UD-Q4-K-M50.0 t/s(speculative)
Qwen3.8 125B · 6B active · Q8_010.0 t/s, 52 t/s prompt(part in system RAM)
Qwen3.8 27B · IQ3_S47.2 t/s(speculative)

All llama.cpp results on the NVIDIA V100 32GB Filter llama.cpp reports on the NVIDIA V100 32GB

llama.cpp on AMD RX 9070 XT 16GB15 reports

Qwen3.5 27B · IQ3_S1,049 t/s prompt
Qwen3.8 27B · IQ3_M65.0 t/s
Qwen3.6 27B · IQ3_M46.1 t/s, 549 t/s prompt(speculative)
Qwen3.5 9B · Q6_K_XL62.0 t/s
Qwen3.8 27B · IQ4_XS36.5 t/s, 1,368 t/s prompt(speculative, part in system RAM)

All llama.cpp results on the AMD RX 9070 XT 16GB Filter llama.cpp reports on the AMD RX 9070 XT 16GB

llama.cpp on NVIDIA DGX Spark15 reports

Qwen3.6 35B · 3B active · Q4_K_M3,557 t/s prompt
Qwen3.8 27B · Q4_K_M900 t/s prompt
gpt-oss 117B · 5.1B active · MXFP452.9 t/s, 1,689 t/s prompt
Gemma 4 8B · Q4_K_M59.4 t/s, 3,634 t/s prompt
Nemotron 3 120B · 12B active · Q4_K14.4 t/s

All llama.cpp results on the NVIDIA DGX Spark Filter llama.cpp reports on the NVIDIA DGX Spark

llama.cpp on M4 Pro 64GB14 reports

Qwen2.5-Omni 3B · Q8_067.5 t/s, 1,159 t/s prompt
Qwen3-VL 4B · Q8_053.6 t/s, 879 t/s prompt
Qwen3.8 27B · UD-IQ4_XS12.8 t/s, 126 t/s prompt
Qwen3.8 27B · Q4_K_M11.5 t/s, 126 t/s prompt
Gemma 4 5.1B · Q8_078.2 t/s, 1,559 t/s prompt

All llama.cpp results on the M4 Pro 64GB Filter llama.cpp reports on the M4 Pro 64GB

llama.cpp on NVIDIA RTX Pro 6000 Blackwell14 reports

Model not named · Q8_0244.0 t/s, 5,230 t/s prompt
GLM-4.5-Air 106B · 12B active · Q4_K_M138.0 t/s
Qwen3.5 35B · 3B active · Q8_016,460 t/s prompt
GLM-5.3 320B · 18B active · IQ2_XXS69.9 t/s
DeepSeek V4 Flash 284B · 13B active · MXFP436.1 t/s, 497 t/s prompt(2 cards)

All llama.cpp results on the NVIDIA RTX Pro 6000 Blackwell Filter llama.cpp reports on the NVIDIA RTX Pro 6000 Blackwell

llama.cpp on Intel Arc Pro B7012 reports

Qwen3.8 27B · Q8_051.0 t/s, 838 t/s prompt(2 cards, speculative)
Qwen3.6 35B · 3B active · Q4_K_M20.7 t/s
Qwen3.8 27B · UD-Q8_K_L16.7 t/s, 1,047 t/s prompt
Qwen3.8 125B · 6B active · Q2_031.2 t/s, 714 t/s prompt(3 requests at once, speculative, part in system RAM)
Qwen3 8B · Q4_K_M77.7 t/s, 1,172 t/s prompt

All llama.cpp results on the Intel Arc Pro B70 Filter llama.cpp reports on the Intel Arc Pro B70

llama.cpp on M4 16GB11 reports

Qwen3.6 35B · 3B active · UD-Q4_K_M23.0 t/s, 30 t/s prompt(part in system RAM)
Qwen3.5 0.8B · Q8_090.1 t/s, 2,122 t/s prompt
Qwen3.5 4B · Q4_K_M28.3 t/s, 352 t/s prompt
Qwen2.5 3B · Q4_K_M47.4 t/s, 557 t/s prompt
Gemma 4 12B · Q4_K_M13.1 t/s

All llama.cpp results on the M4 16GB Filter llama.cpp reports on the M4 16GB

llama.cpp on M5 Max 128GB10 reports

DeepSeek V3.1 671B · INT43.6 t/s(speculative, part in system RAM)
Qwen3.8 122B · 10B active · Q8_020.4 t/s(speculative, part in system RAM)
Kimi K3 2800B · 104B active · 4-bit1.0 t/s(part in system RAM)
Qwen3.6 35B · 3B active · Q8_0105.0 t/s(speculative)
GLM-5.3 320B · 18B active · IQ1_S24.0 t/s, 333 t/s prompt(2 cards)

All llama.cpp results on the M5 Max 128GB Filter llama.cpp reports on the M5 Max 128GB

llama.cpp on Intel Arc Pro B60 24GB8 reports

Qwen3.8 27B · Q4_K_M15.6 t/s, 339 t/s prompt
Qwen3.8 27B53.0 t/s(speculative)
Qwen3.8 27B · Q4_K_M53.3 t/s, 420 t/s prompt(speculative)
Nemotron 3.5 Lightning 30B · 3B active · Q4_091.9 t/s, 1,760 t/s prompt(2 cards, speculative)
Qwen3.8 27B · Q4_K_M15.3 t/s, 513 t/s prompt

All llama.cpp results on the Intel Arc Pro B60 24GB Filter llama.cpp reports on the Intel Arc Pro B60 24GB

llama.cpp on M3 Ultra 512GB8 reports

Qwen3.5 35B · 3B active · MXFP4_MOE80.9 t/s, 2,779 t/s prompt
Gemma 4 26B · 4B active · MXFP4_MOE87.8 t/s, 2,595 t/s prompt
Gemma 4 26B · 4B active · UD-Q8_K_XL84.9 t/s, 2,635 t/s prompt
Gemma 4 26B · 4B active · UD-Q5_K_M88.1 t/s, 2,554 t/s prompt
Gemma 4 26B · 4B active · BF1662.4 t/s, 2,649 t/s prompt

All llama.cpp results on the M3 Ultra 512GB Filter llama.cpp reports on the M3 Ultra 512GB

llama.cpp on NVIDIA RTX 50808 reports

Qwen3 8B · Q4_K_M130.6 t/s
Qwen3.8 27B · Q3_K_XL80.0 t/s, 1,200 t/s prompt(speculative)
Qwen3.8 125B · 6B active · Q5_K_XL100 t/s prompt(2 cards, speculative, part in system RAM)
Qwen3.8 27B · NVFP412.0 t/s
Qwen3.8 125B · 6B active · IQ3_XXS6.0 t/s(part in system RAM)

All llama.cpp results on the NVIDIA RTX 5080 Filter llama.cpp reports on the NVIDIA RTX 5080

llama.cpp on NVIDIA Tesla P100 16GB8 reports

Qwen3.8 27B · Q4_033.3 t/s, 407 t/s prompt(2 cards)
Qwen3.8 125B · 6B active · IQ3_XXS5.8 t/s, 56 t/s prompt(part in system RAM)
Qwen3.8 27B · Q4_K_XL24.5 t/s(2 cards)
Qwen3.8 27B · Q6_K32.6 t/s, 493 t/s prompt(2 cards)
Nemotron 3.5 Lightning 30B · 3B active · Q4_0110.0 t/s(2 cards, speculative)

All llama.cpp results on the NVIDIA Tesla P100 16GB Filter llama.cpp reports on the NVIDIA Tesla P100 16GB

llama.cpp on AMD Gorgon Halo 192GB7 reports

Qwen3.8 27B · Q6_K_XL9.0 t/s, 463 t/s prompt
gpt-oss 117B · 5.1B active · MXFP455.7 t/s, 1,135 t/s prompt
DeepSeek V4 Flash 284B · 13B active · UD-IQ3_S17.8 t/s, 238 t/s prompt
MiniMax M2.7 230B · 10B active · UD-IQ4_XS30.2 t/s, 309 t/s prompt
Qwen3.8 27B · Q8_08.1 t/s, 466 t/s prompt

All llama.cpp results on the AMD Gorgon Halo 192GB Filter llama.cpp reports on the AMD Gorgon Halo 192GB

llama.cpp on Intel Arc B580 12GB6 reports

Bonsai 2 27B · ternary45.0 t/s
Llama 2 7B · Q4_04,454 t/s prompt
Bonsai 2 27B · ternary48.6 t/s, 1,480 t/s prompt
Bonsai 2 27B · PTQ1_042.3 t/s, 895 t/s prompt
GLM-4 9B · Q4_021.3 t/s

All llama.cpp results on the Intel Arc B580 12GB Filter llama.cpp reports on the Intel Arc B580 12GB

llama.cpp on M4 Max 64GB6 reports

Qwen3.8 27B · Q4_K_M23.9 t/s, 120 t/s prompt
Qwen3.6 35B · 3B active · UD-Q4_K_XL76.4 t/s, 320 t/s prompt
Qwen3.5 9B · Q4_K_M65.5 t/s, 306 t/s prompt
Gemma 4 31B · Q4_K_M22.4 t/s, 63 t/s prompt
Gemma 4 26B · 4B active · UD-Q4_K_M81.7 t/s, 380 t/s prompt

All llama.cpp results on the M4 Max 64GB Filter llama.cpp reports on the M4 Max 64GB

llama.cpp on NVIDIA RTX 40806 reports

Qwen3.8 27B · IQ3_XXS70.0 t/s(speculative)
Qwen3.8 27B · UD-IQ4_XS55.0 t/s(speculative)
Qwen3.8 125B · 6B active · Q4_K_M16.5 t/s, 350 t/s prompt(speculative, part in system RAM)
Qwen3.6 35B · 3B active · MXFP4_MOE45.0 t/s(speculative, part in system RAM)
Qwen3.8 27B · IQ4_XS19.0 t/s(speculative, part in system RAM)

All llama.cpp results on the NVIDIA RTX 4080 Filter llama.cpp reports on the NVIDIA RTX 4080

llama.cpp on NVIDIA Tesla P40 24GB6 reports

Ornith1.5 35B · 3B active · Q4_060.8 t/s, 1,104 t/s prompt
Kimi K2.5 1000B · 32B active · Q2_K_XL5.8 t/s, 11 t/s prompt(8 cards, part in system RAM)
Qwen3.6 27B · IQ4_XS23.7 t/s(speculative)
Qwen3.6 35B · 3B active · Q3_K_XL37.9 t/s, 982 t/s prompt
Qwen3.6 35B · 3B active · UD-Q3_K_XL51.6 t/s, 982 t/s prompt(speculative)

All llama.cpp results on the NVIDIA Tesla P40 24GB Filter llama.cpp reports on the NVIDIA Tesla P40 24GB

llama.cpp on AMD MI50 32GB5 reports

Qwen3.8 27B · Q6_K20.2 t/s, 177 t/s prompt(2 cards, speculative)
Qwen3-Coder 30B · 3B active · Q4_K_M66.1 t/s
Qwen3.8 125B · 6B active · UD-Q4_K_XL15.0 t/s(3 cards)
Model not named15.1 t/s, 410 t/s prompt
Qwen3.8 27B13.6 t/s, 379 t/s prompt(speculative)

All llama.cpp results on the AMD MI50 32GB Filter llama.cpp reports on the AMD MI50 32GB

llama.cpp on AMD Radeon 780M5 reports

Qwen3.8 27B · IQ3_S74 t/s prompt(speculative)
Gemma 4 26B · 4B active · Q4_K_M25.0 t/s, 209 t/s prompt
Qwen3.6 35B · 3B active · Q8_021.1 t/s, 287 t/s prompt
Qwen3.6 35B · 3B active · Q818.4 t/s, 311 t/s prompt
Qwen3.8100 t/s prompt

Filter llama.cpp reports on the AMD Radeon 780M

llama.cpp on NVIDIA RTX 40605 reports

Qwen3.8 125B · 6B active · IQ2_XS12.5 t/s(part in system RAM)
Qwen2.5 1.5B · 0.9B active · Q4_K_M253.0 t/s
Ornith1.5 9B15.0 t/s, 200 t/s prompt(speculative)
Qwen3.8 27B · UD-Q4_K_XL6.0 t/s(speculative, part in system RAM)
Qwen3.6 35B · 3B active · IQ4_XS24.2 t/s, 66 t/s prompt(part in system RAM)

All llama.cpp results on the NVIDIA RTX 4060 Filter llama.cpp reports on the NVIDIA RTX 4060

llama.cpp on NVIDIA RTX 4070 Ti Super5 reports

Qwen3-Next 80B · 3B active · Q4_K_M67.5 t/s(part in system RAM)
Qwen3.8 27B · Q510.0 t/s(part in system RAM)
Qwen3.8 27B · IQ2_XXS54.1 t/s, 1,522 t/s prompt
Qwen3.8 27B · Q4_K_M5.7 t/s
Qwen3.8 27B · Q4_K_M7.6 t/s(speculative)

All llama.cpp results on the NVIDIA RTX 4070 Ti Super Filter llama.cpp reports on the NVIDIA RTX 4070 Ti Super

llama.cpp on NVIDIA RTX 50705 reports

Qwen3.8 27B · Q4_K_M4.0 t/s(part in system RAM)
Qwen3.8 125B · 6B active · UD-IQ3_XXS11.5 t/s(part in system RAM)
Model not named1,800 t/s prompt
Qwen3.8 125B · 6B active · IQ3_XXS15.0 t/s, 105 t/s prompt(part in system RAM)
Qwen3.8 125B · 6B active · IQ1_S22.0 t/s, 25 t/s prompt(part in system RAM)

Filter llama.cpp reports on the NVIDIA RTX 5070

llama.cpp on AMD MI50 16GB4 reports

Qwen3.6 35B · 3B active · Q5_K_XL49.2 t/s, 937 t/s prompt(2 cards)
Qwen3.8 27B · Q4_048.0 t/s(2 cards, speculative)
Qwen3.8 27B · Q8_030.1 t/s(3 cards)
Qwen3.8 125B · 6B active · IQ416.9 t/s(part in system RAM)

Filter llama.cpp reports on the AMD MI50 16GB

llama.cpp on AMD Radeon Pro V620 32GB4 reports

Qwen3.8 27B · Q4_K_XL24.6 t/s, 492 t/s prompt
Muse 30B · Q6_K_XL35.4 t/s, 355 t/s prompt(2 cards, speculative)
DeepSeek V4 Flash 284B · 13B active · IQ3_XXS21.1 t/s, 276 t/s prompt(4 cards, speculative)
Qwen3.8 27B · Q5_K_P14.3 t/s, 364 t/s prompt(speculative)

Filter llama.cpp reports on the AMD Radeon Pro V620 32GB

llama.cpp on AMD RX 6700 XT4 reports

Qwen3.6 35B · 3B active · Q4_K_XL30.0 t/s(speculative, part in system RAM)
Qwen3 8B · Q4_K_M61.0 t/s, 851 t/s prompt
Gemma 4 12B · IQ4_NL34.6 t/s, 654 t/s prompt
Qwen3.8 27B · UD-IQ3_XXS18.0 t/s, 150 t/s prompt(2 cards, speculative)

Filter llama.cpp reports on the AMD RX 6700 XT

llama.cpp on AMD RX 9060 XT 16GB4 reports

Qwen3.8 27B · IQ4_XS42.7 t/s, 658 t/s prompt(speculative, part in system RAM)
Qwen3.6 35B · 3B active · UD-Q4_K_XL66.0 t/s(speculative, part in system RAM)
Qwen2.5 14B · Q4_K_M33.5 t/s, 1,210 t/s prompt
Ornith1.5 9B · Q6_K36.0 t/s, 950 t/s prompt

Filter llama.cpp reports on the AMD RX 9060 XT 16GB

llama.cpp on AMD RX 90704 reports

Qwen3.8 27B · UD-IQ4_XS81.0 t/s(speculative, part in system RAM)
Qwen3.8 27B · IQ3_S600 t/s prompt
Qwen3.8 27B · IQ3_XXS40.7 t/s, 779 t/s prompt(speculative)
Qwen3.6 27B · UD-Q5_K_XL46.9 t/s, 398 t/s prompt(2 cards, speculative)

Filter llama.cpp reports on the AMD RX 9070

llama.cpp on Intel Arc A770 16GB4 reports

Gemma 4 12B · QAT20.0 t/s, 1,100 t/s prompt(speculative)
Qwen3.5 9B · Q6_K49.0 t/s, 48 t/s prompt(speculative)
Qwen3.6 27B · 3B active · Q4_K_M14.4 t/s
Qwen3 1.7B · Q8_033.2 t/s

Filter llama.cpp reports on the Intel Arc A770 16GB

llama.cpp on M1 Max 64GB4 reports

Qwen3-Coder 30B · 3B active · Q4_K_M64.4 t/s, 645 t/s prompt
Model not named · IQ2_XS0.9 t/s
DeepSeek V4 Flash 284B · 13B active · IQ3-XXS8.0 t/s, 30 t/s prompt(part in system RAM)
Qwen3.8 125B · 6B active · Q4180 t/s prompt

Filter llama.cpp reports on the M1 Max 64GB

llama.cpp on M2 Ultra 192GB4 reports

MiniMax M2.7 230B · 10B active · UD-Q4_K_M32.7 t/s, 421 t/s prompt
Mimo 2.5 · UD-Q3_K_M37.9 t/s, 302 t/s prompt
Qwen3.8 27B · Q6_K_XL22.4 t/s, 360 t/s prompt
DeepSeek V4 Flash 284B · 13B active25.8 t/s, 250 t/s prompt

Filter llama.cpp reports on the M2 Ultra 192GB

llama.cpp on M4 24GB4 reports

Qwen 2.5 Coder 7B · Q4_K_M22.6 t/s, 240 t/s prompt
Llama 3.2 3B · Q4_K_M46.0 t/s, 530 t/s prompt
Qwen3.8 27B · UD-IQ4_XS6.5 t/s, 61 t/s prompt
Gemma 4 12B · UD-Q4_K_XL12.9 t/s, 139 t/s prompt

Filter llama.cpp reports on the M4 24GB

llama.cpp on NVIDIA RTX 4060 Laptop 8GB4 reports

LFM3 1.7B1.0 t/s(16 requests at once, part in system RAM)
Gemma 4 26B60.0 t/s(speculative, part in system RAM)
Qwen3.5 9B · Q4_K_M37.7 t/s, 1,829 t/s prompt
Model not named · UD-IQ3_XXS23.0 t/s, 35 t/s prompt(speculative, part in system RAM)

Filter llama.cpp reports on the NVIDIA RTX 4060 Laptop 8GB

llama.cpp on NVIDIA RTX 4060 Ti 16GB4 reports

Qwen3.6 35B · 3B active · Q5_K_XL19.5 t/s, 133 t/s prompt(part in system RAM)
Qwen3.8 125B · 6B active · Q5_K_M18.7 t/s, 210 t/s prompt(part in system RAM)
DeepSeek V4 Flash 284B · 13B active · UD-Q2_K_XL7.4 t/s, 15 t/s prompt(part in system RAM)
Qwen3.8 27B · IQ3_S11.0 t/s(speculative)

Filter llama.cpp reports on the NVIDIA RTX 4060 Ti 16GB

llama.cpp on NVIDIA RTX Pro 5000 Blackwell 72GB4 reports

Qwen3.6 35B · 3B active · Q8_0198.0 t/s, 5,772 t/s prompt
Gemma 4 31B · Q8_033.2 t/s, 2,146 t/s prompt
Muse 28B · Q8_038.5 t/s, 2,678 t/s prompt
Qwen3.8 27B · Q8_038.6 t/s, 2,276 t/s prompt

Filter llama.cpp reports on the NVIDIA RTX Pro 5000 Blackwell 72GB

llama.cpp on AMD RX 7800 XT 16GB3 reports

Qwen3.6 35B · 3B active · UD-Q5_K_M35.0 t/s(part in system RAM)
Qwen3.6 35B · 3B active · UD-IQ3_XXS110.0 t/s(speculative)
Qwen3.8 27B · UD-IQ4_XS30.0 t/s

Filter llama.cpp reports on the AMD RX 7800 XT 16GB

llama.cpp on AMD RX 7900 GRE 16GB3 reports

Qwen3 8B · Q8_0133.0 t/s(speculative)
Gemma 4 26B · 4B active · Q4_K_M51.9 t/s, 322 t/s prompt(2 cards)
Qwen3.8 27B · Q4_K_M11.8 t/s, 88 t/s prompt(part in system RAM)

Filter llama.cpp reports on the AMD RX 7900 GRE 16GB

llama.cpp on M1 Ultra 128GB3 reports

Gemma 4 8B · Q4_K_M69.7 t/s, 1,173 t/s prompt
Gemma 4 8B · Q4_K_M69.7 t/s, 1,173 t/s prompt
Qwen3.8 125B · 6B active · IQ4_NL31.2 t/s, 530 t/s prompt

Filter llama.cpp reports on the M1 Ultra 128GB

llama.cpp on M2 Pro 32GB3 reports

Model not named · Q4_K_M33.6 t/s, 201 t/s prompt
Model not named · Q4_K_M34.8 t/s, 228 t/s prompt
Qwen3.8 27B · IQ4_XS8.6 t/s, 22 t/s prompt

Filter llama.cpp reports on the M2 Pro 32GB

llama.cpp on M4 Pro 24GB3 reports

Qwen3 0.6B · Q4297.4 t/s, 4,993 t/s prompt
Qwen3-Next 80B · 3B active · Q4_K_M2.9 t/s(part in system RAM)
Qwen3.8 27B · Q4_K_M11.4 t/s, 97 t/s prompt

Filter llama.cpp reports on the M4 Pro 24GB

llama.cpp on M5 Ultra 256GB3 reports

Qwen3.8 125B · 6B active · IQ4_NL100.2 t/s, 2,150 t/s prompt(speculative)
GLM-5.3 320B · 18B active61.0 t/s(speculative)
Qwen3.8 125B · 6B active · Q8_017.0 t/s, 1,069 t/s prompt

Filter llama.cpp reports on the M5 Ultra 256GB

llama.cpp on NVIDIA CMP 170HX 64GB (unlocked)3 reports

Qwen3.6 27B · Q4_K_M38.0 t/s, 907 t/s prompt
Qwen3.6 35B · 3B active · Q6_K63.0 t/s, 1,700 t/s prompt(speculative)
DeepSeek V4 Flash 284B · 13B active · Q4_K_XL29.0 t/s, 450 t/s prompt(4 cards)

Filter llama.cpp reports on the NVIDIA CMP 170HX 64GB (unlocked)

llama.cpp on NVIDIA GTX 1080 Ti3 reports

Llama 2 7B · Q4_062.0 t/s, 1,084 t/s prompt
Gemma 4 26B · 4B active · Q6_K_XL52.0 t/s, 341 t/s prompt(2 cards)
Qwen3.6 35B · 3B active · MXFP419.7 t/s, 333 t/s prompt(2 cards)

Filter llama.cpp reports on the NVIDIA GTX 1080 Ti

llama.cpp on NVIDIA RTX 2080 Ti 22GB (modded)3 reports

Qwen3.6 35B · 3B active · UD-IQ4_XS69.0 t/s
Qwen3.6 35B · 3B active · Q4_K_P95.5 t/s, 466 t/s prompt(3 cards, speculative)
Qwen3.8 27B · IQ3_S44.1 t/s(speculative)

Filter llama.cpp reports on the NVIDIA RTX 2080 Ti 22GB (modded)

llama.cpp on NVIDIA RTX 30703 reports

Qwen3.6 35B · 3B active · UD-IQ4_XS27.0 t/s, 517 t/s prompt(part in system RAM)
Qwen3.5 9B · Q4_K_M69.8 t/s
Qwen3.6 35B · 3B active · Q5_K_S30.0 t/s(part in system RAM)

Filter llama.cpp reports on the NVIDIA RTX 3070

llama.cpp on NVIDIA RTX 3080 20GB3 reports

Qwen3.8 27B · Q821.0 t/s(2 cards, speculative)
Qwen3.8 27B · Q4_K_XL31.3 t/s, 997 t/s prompt
DeepSeek V4 Flash 284B · 13B active · IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ817.5 t/s, 273 t/s prompt(2 cards, part in system RAM)

Filter llama.cpp reports on the NVIDIA RTX 3080 20GB

llama.cpp on NVIDIA RTX 40703 reports

Qwen3.8 125B · 6B active · IQ3_XXS15.7 t/s(2 cards, speculative, part in system RAM)
Qwen3.8 125B · 6B active · 4.27 bpw20.6 t/s(speculative, part in system RAM)
Qwen3.8 27B · IQ4_XS11.7 t/s(speculative, part in system RAM)

Filter llama.cpp reports on the NVIDIA RTX 4070

llama.cpp on NVIDIA RTX 5060 Laptop 8GB3 reports

Qwen3.6 35B · 3B active43.0 t/s(part in system RAM)
Bonsai 2 27B · PTQ1_029.3 t/s
Gemma 4 12B · Q6_K40.0 t/s(speculative, part in system RAM)

Filter llama.cpp reports on the NVIDIA RTX 5060 Laptop 8GB

llama.cpp on NVIDIA RTX A2000 12GB3 reports

Gemma 4 8B · Q4_K_M51.9 t/s, 1,832 t/s prompt
Qwen3.8 125B · 6B active · UD-Q4_K_XL5.5 t/s, 118 t/s prompt(part in system RAM)
Qwen3.6 35B · 3B active · UD-Q5_K_M26.1 t/s, 233 t/s prompt(speculative, part in system RAM)

Filter llama.cpp reports on the NVIDIA RTX A2000 12GB

llama.cpp on NVIDIA RTX Pro 4000 Blackwell SFF3 reports

Qwen3.8 27B · NVFP4 hybrid 5.01 bpw50.4 t/s, 816 t/s prompt(speculative)
Muse 30B · Q5_K_M84.6 t/s(speculative)
Qwen3.8 27B · NVFP450.4 t/s, 227 t/s prompt(speculative)

Filter llama.cpp reports on the NVIDIA RTX Pro 4000 Blackwell SFF

llama.cpp on NVIDIA RTX Pro 4500 Blackwell 32GB3 reports

Bonsai 2 27B · PTQ1_036.5 t/s, 909 t/s prompt
Qwen3.8 125B · 6B active · UD-Q3_K_XL27.9 t/s, 150 t/s prompt(part in system RAM)
Qwen3.6 27B · IQ4_XS45.2 t/s, 2,023 t/s prompt

Filter llama.cpp reports on the NVIDIA RTX Pro 4500 Blackwell 32GB

llama.cpp on NVIDIA RTX PRO 6000 Max-Q3 reports

DeepSeek V4 Flash 284B · 13B active · FP4-FP835.7 t/s, 38 t/s prompt(2 cards)
Model not named16,384 t/s prompt
DeepSeek V4 Flash 284B · 13B active · UD-Q8_K_XL44.0 t/s, 564 t/s prompt(speculative, part in system RAM)

Filter llama.cpp reports on the NVIDIA RTX PRO 6000 Max-Q

llama.cpp on NVIDIA V100 16GB3 reports

MiniCPM-V 4.7 35B · 3B active · mix_q3gud_im93.0 t/s
Qwen3.8 125B · 6B active · IQ3_XXS50.0 t/s(2 cards)
Qwen3.8 27B · Q6_K_M39.9 t/s, 1,377 t/s prompt(2 cards)

Filter llama.cpp reports on the NVIDIA V100 16GB

llama.cpp on AMD FirePro D700 6GB2 reports

Qwen3.6 35B · 3B active11.0 t/s, 66 t/s prompt(2 cards)
Qwen3.5 9B · Q411.0 t/s(2 cards, speculative)

Filter llama.cpp reports on the AMD FirePro D700 6GB

llama.cpp on AMD Radeon 890M2 reports

Gemma 4 8B · Q4_K_M21.2 t/s, 569 t/s prompt
Qwen3.5 35B · 3B active · Q4_K_XL14.8 t/s, 212 t/s prompt

Filter llama.cpp reports on the AMD Radeon 890M

llama.cpp on AMD Radeon Pro W7800 48GB2 reports

Qwen3.6 35B · 3B active · UD-Q4_K_S154.1 t/s(2 cards, speculative)
Qwen3.8 125B · 6B active · IQ3_XXS90.0 t/s

Filter llama.cpp reports on the AMD Radeon Pro W7800 48GB

llama.cpp on AMD Radeon Pro W79002 reports

LFM2.5 24B · 2.5B active · Q895.0 t/s
RWKV7 2.9B · F1648.7 t/s, 4,316 t/s prompt

Filter llama.cpp reports on the AMD Radeon Pro W7900

llama.cpp on AMD RX 6600 XT2 reports

Model not named90 t/s prompt
Qwen3.6 35B · 3B active · UD_Q4_K_XL30.0 t/s(speculative, part in system RAM)

Filter llama.cpp reports on the AMD RX 6600 XT

llama.cpp on AMD RX 7600 XT 16GB2 reports

Qwen3.8 27B · IQ3_XXS18.0 t/s, 141 t/s prompt
Muse 30B · UD-Q2-K-XL20.0 t/s, 308 t/s prompt(speculative)

Filter llama.cpp reports on the AMD RX 7600 XT 16GB

llama.cpp on AMD RX 7900 XT2 reports

Qwen3.8 27B · IQ4_XS46.4 t/s(2 cards, speculative)
Qwen3.8 27B · IQ4_XS52.0 t/s, 708 t/s prompt(speculative)

Filter llama.cpp reports on the AMD RX 7900 XT

llama.cpp on M1 Pro 32GB2 reports

LFM2.5 24B · 2.5B active · Q4_K_M52.0 t/s
Qwen3.6 35B · 3B active · Q4_K_S25.4 t/s, 39 t/s prompt

Filter llama.cpp reports on the M1 Pro 32GB

llama.cpp on M2 Max 96GB2 reports

Qwen3.6 27B · F168.0 t/s(speculative)
Qwen3.6 27B · Q5_K_M28.0 t/s(speculative)

Filter llama.cpp reports on the M2 Max 96GB

llama.cpp on M2 Ultra 128GB2 reports

Qwen3-Coder 30B · 3B active · UD-Q4_K_XL71.2 t/s, 1,097 t/s prompt
Qwen3-Coder 30B · 3B active · UD-Q4_K_XL69.3 t/s, 1,081 t/s prompt

Filter llama.cpp reports on the M2 Ultra 128GB

llama.cpp on M3 Ultra 256GB2 reports

Qwen3.5 9B · Q4_K_M77.8 t/s
Qwen3.5 9B · Q4_K_M77.8 t/s, 1,010 t/s prompt

Filter llama.cpp reports on the M3 Ultra 256GB

llama.cpp on M3 Ultra 96GB2 reports

Qwen3.5 35B · 3B active · Q8_073.4 t/s, 1,808 t/s prompt
Qwen3.8 27B · Q4_K_M29.0 t/s, 274 t/s prompt

Filter llama.cpp reports on the M3 Ultra 96GB

llama.cpp on M4 32GB2 reports

Qwen3 8B · Q4_K_M19.7 t/s, 224 t/s prompt
Qwen3 4B · Q4_K_M36.2 t/s, 417 t/s prompt

Filter llama.cpp reports on the M4 32GB

llama.cpp on M5 Pro 64GB2 reports

Qwen3.8 27B · UD-IQ4_XS18.5 t/s, 382 t/s prompt
Qwen3.8 125B · 6B active · Q4_027.6 t/s, 367 t/s prompt(speculative, part in system RAM)

Filter llama.cpp reports on the M5 Pro 64GB

llama.cpp on NVIDIA L42 reports

Qwen3.8 27B · Q4_K_M14.6 t/s
Qwen3.8 27B · UD-Q4_K_XL33.0 t/s, 579 t/s prompt(speculative)

Filter llama.cpp reports on the NVIDIA L4

llama.cpp on NVIDIA P102-1002 reports

Qwen3.8 177B · IQ3_XXS10.9 t/s, 20 t/s prompt(4 cards, part in system RAM)
Qwen3.6 35B · 3B active · IQ4_XS23.5 t/s, 432 t/s prompt(2 cards, 3 requests at once)

Filter llama.cpp reports on the NVIDIA P102-100

llama.cpp on NVIDIA RTX 2080 Ti2 reports

GLM-5.3 320B · 18B active · Q47.0 t/s(part in system RAM)
Qwen3 14B · Q4_K_M68.1 t/s, 3,057 t/s prompt(2 cards, speculative)

Filter llama.cpp reports on the NVIDIA RTX 2080 Ti

llama.cpp on NVIDIA RTX 3080 10GB2 reports

Qwen3.5 4B · Q4_K_M662 t/s prompt(part in system RAM)
Bonsai 2 27B · PTQ1_052.2 t/s, 1,211 t/s prompt

Filter llama.cpp reports on the NVIDIA RTX 3080 10GB

llama.cpp on NVIDIA RTX 3090 Ti2 reports

DeepSeek V4 Flash 284B · 13B active · IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ815.0 t/s, 150 t/s prompt(part in system RAM)
Qwen3.6 27B · Q8_0100.0 t/s(2 cards, speculative)

Filter llama.cpp reports on the NVIDIA RTX 3090 Ti

llama.cpp on NVIDIA RTX 4070 Ti2 reports

Gemma 4 8B · Q8_069.7 t/s, 6,757 t/s prompt
Qwen3.8 27B · UD-IQ2_XXS43.6 t/s, 943 t/s prompt

Filter llama.cpp reports on the NVIDIA RTX 4070 Ti

llama.cpp on NVIDIA RTX 5060 8GB2 reports

Qwen3.8 27B · UD-IQ2_XXS30.1 t/s
Qwen3.6 35B · 3B active · Q4_K_XL40.0 t/s, 500 t/s prompt(part in system RAM)

Filter llama.cpp reports on the NVIDIA RTX 5060 8GB

llama.cpp on NVIDIA RTX 5070 Ti Laptop 12GB2 reports

Qwen3.6 35B · 3B active · Q4_K_M59.0 t/s, 156 t/s prompt(part in system RAM)
Qwen3.8 27B · Q4_K_XL5.0 t/s(speculative, part in system RAM)

Filter llama.cpp reports on the NVIDIA RTX 5070 Ti Laptop 12GB

llama.cpp on NVIDIA RTX 5090 Laptop 24GB2 reports

Gemma 4 8B · Q4_K_M165.6 t/s, 9,556 t/s prompt
Qwen3.8 27B · Q4_K_M30.0 t/s

Filter llama.cpp reports on the NVIDIA RTX 5090 Laptop 24GB

llama.cpp on NVIDIA RTX Pro 4000 Blackwell2 reports

Qwen3.8 27B · Q868.0 t/s(2 cards, 2 requests at once, speculative)
DeepSeek V4 Flash 284B · 13B active · MXFP413.6 t/s, 338 t/s prompt(2 cards, part in system RAM)

Filter llama.cpp reports on the NVIDIA RTX Pro 4000 Blackwell

llama.cpp on NVIDIA T1000 8GB2 reports

Ling-3.0 7.9B · 1.3B active · Q4_K_M54.0 t/s, 245 t/s prompt
Bonsai 2 27B · PTQ1_06.0 t/s, 18 t/s prompt

Filter llama.cpp reports on the NVIDIA T1000 8GB

llama.cpp on AMD RX 6800 16GB1 report

Qwen3.8 27B · IQ4_XS45.0 t/s(2 cards)

Filter llama.cpp reports on the AMD RX 6800 16GB

llama.cpp on AMD RX 6800 XT 16GB1 report

Qwen3.8 27B · Q6_K45.0 t/s, 450 t/s prompt(2 cards, speculative)

Filter llama.cpp reports on the AMD RX 6800 XT 16GB

llama.cpp on M1 16GB1 report

Qwen2.5 0.5B · Q495.6 t/s

Filter llama.cpp reports on the M1 16GB

llama.cpp on M1 8GB1 report

LFM2.5 2.6B · QAD-Q4_022.0 t/s, 200 t/s prompt

Filter llama.cpp reports on the M1 8GB

llama.cpp on M2 8GB1 report

Bonsai 2 27B · ternary7.6 t/s, 42 t/s prompt

Filter llama.cpp reports on the M2 8GB

llama.cpp on M2 Max 32GB1 report

Qwen3.6 35B · 3B active · Q4_K_M58.6 t/s, 671 t/s prompt

Filter llama.cpp reports on the M2 Max 32GB

llama.cpp on M2 Max 64GB1 report

Model not named · Q4_035.9 t/s, 428 t/s prompt

Filter llama.cpp reports on the M2 Max 64GB

llama.cpp on M2 Pro 16GB1 report

Qwen3 14B18.0 t/s

Filter llama.cpp reports on the M2 Pro 16GB

llama.cpp on M3 Pro 18GB1 report

Qwen3 8B · Q4_K_M25.2 t/s

Filter llama.cpp reports on the M3 Pro 18GB

llama.cpp on M4 Max 48GB1 report

Qwen3.5 4B · Q4_K_M85.0 t/s

Filter llama.cpp reports on the M4 Max 48GB

llama.cpp on M4 Pro 48GB1 report

Qwen3.5 4B · Q4_K_M150.0 t/s

Filter llama.cpp reports on the M4 Pro 48GB

llama.cpp on M5 16GB1 report

Qwen3.8 27B · Q3_xxs9.0 t/s

Filter llama.cpp reports on the M5 16GB

llama.cpp on M5 Max 64GB1 report

Gemma 4 26B · 4B active97.0 t/s

Filter llama.cpp reports on the M5 Max 64GB

llama.cpp on M5 Pro 24GB1 report

Qwen3 30B · 3B active · IQ3_XXS73.9 t/s

Filter llama.cpp reports on the M5 Pro 24GB

llama.cpp on M5 Pro 48GB1 report

Qwen3.8 27B · Q8_019.0 t/s(speculative)

Filter llama.cpp reports on the M5 Pro 48GB

llama.cpp on M6 32GB1 report

Qwen3.5 9B · Q4_K_M27.0 t/s

Filter llama.cpp reports on the M6 32GB

llama.cpp on NVIDIA A100 40GB1 report

DeepSeek V4 Flash 284B · 13B active · Q8_K_XL16.1 t/s(part in system RAM)

Filter llama.cpp reports on the NVIDIA A100 40GB

llama.cpp on NVIDIA CMP 170HX 40GB (unlocked)1 report

Qwen3.8 125B · 6B active · UD-Q4_K_XL96.9 t/s(4 cards, speculative)

Filter llama.cpp reports on the NVIDIA CMP 170HX 40GB (unlocked)

llama.cpp on NVIDIA Jetson Orin Nano Super 8GB1 report

SmolLM2 0.1B · Q4_K_M165.2 t/s

Filter llama.cpp reports on the NVIDIA Jetson Orin Nano Super 8GB

llama.cpp on NVIDIA RTX 3080 Laptop 8GB1 report

Ornith1.5 35B · 3B active · Q4_K_M28.6 t/s(part in system RAM)

Filter llama.cpp reports on the NVIDIA RTX 3080 Laptop 8GB

llama.cpp on NVIDIA RTX 4060 Ti 8GB1 report

Qwen3.5 35B · 3B active · Q28.0 t/s(part in system RAM)

Filter llama.cpp reports on the NVIDIA RTX 4060 Ti 8GB

llama.cpp on NVIDIA RTX 4070 Super1 report

Qwen3.8 27B · IQ3_S18.0 t/s(part in system RAM)

Filter llama.cpp reports on the NVIDIA RTX 4070 Super

llama.cpp on NVIDIA RTX 4080 Super1 report

Qwen3.8 27B · Q3_K_XL33.1 t/s

Filter llama.cpp reports on the NVIDIA RTX 4080 Super

llama.cpp on NVIDIA RTX 6000 Ada1 report

Qwen3.6 35B · 3B active · Q4_K_M260.0 t/s(4 requests at once, speculative)

Filter llama.cpp reports on the NVIDIA RTX 6000 Ada

llama.cpp on NVIDIA RTX A4000 16GB1 report

Qwen3.8 27B · Q8_023.0 t/s(3 cards, speculative)

Filter llama.cpp reports on the NVIDIA RTX A4000 16GB

llama.cpp on NVIDIA RTX A6000 48GB1 report

Gemma 4 5.1B · bf1616.9 t/s

Filter llama.cpp reports on the NVIDIA RTX A6000 48GB

llama.cpp, GPU not identified125 reports

Qwen3.8 125B · 6B active · Q4_K_XL87.9 t/s(speculative)
SenseNova-U1.5 8B · Q4_K_M125.0 t/s, 3,482 t/s prompt
Qwen3.8 35B · 3B active · IQ2_M14.5 t/s(part in system RAM)
Gemma 3 1B4.5 t/s
Mellum2.1 · Q412.0 t/s(speculative)

llama.cpp runs open-weight LLMs on your own hardware. It introduced GGUF, the file format most local LLM tools now read, and it has backends for CPU, CUDA, ROCm, Metal, Vulkan and SYCL.

Several easier tools are built on it, Ollama and LM Studio among them. llamaperf files a report under the tool the person named, so an Ollama report may well be llama.cpp underneath.

Its strengths are reach and choice. The same GGUF file runs on an NVIDIA card, an AMD card, a Mac or a CPU, and models come quantized at every size from 2-bit to 8-bit, plus the i-quants.

Compare GPUs running llama.cpp

Every card is compared on one model, Qwen3.8 27B at 4-bit, the size and bit class that most cards here have plain runs of. A card with no run of that model shows none rather than a figure from a different model.

GPU	VRAM	Reports	Median speed, Qwen3.8 27B 4-bit
AMD Strix Halo 128GBamd	128GB	100	Gen 14.0 t/sPP 365.8 t/s · n=2
NVIDIA RTX 3090nvidia	24GB	84	Gen 38.7 t/s
NVIDIA RTX 5090nvidia	32GB	45	Gen 84.2 t/s
M5 Max 48GBapple	48GB	38	no Qwen3.8 27B 4-bit run reported
NVIDIA RTX 5060 Ti 16GBnvidia	16GB	36	no Qwen3.8 27B 4-bit run reported
NVIDIA RTX 3060 12GBnvidia	12GB	35	no Qwen3.8 27B 4-bit run reported
M4 Max 128GBapple	128GB	33	Gen 21.8 t/sPP 236.6 t/s · n=1
AMD RX 7900 XTXamd	24GB	27	Gen 37.1 t/sPP 877.8 t/s · n=1

llama.cpp against other engines

Pairs of plain runs on the same card, of the same model size at the same bit class: one device, one request, no speculative decoding, the whole model in memory. Context length and build still differ between the two sides, and each side shows its own.

Model and GPU	llama.cpp	Other engine
Qwen3.6 35B · 3B active 8-bitAMD Strix Halo 128GB	24.1 t/sPrompt processing (prefill) 280.3 tokens/s (2)one generation report, 512K context, 1 not stated	Gufo50.8 t/sPrompt processing (prefill) 2384.0 tokens/s (1)one generation report, context not stated
Qwen3.8 125B · 6B active 4-bitAMD Strix Halo 128GB	23.0 t/sPrompt processing (prefill) 251.9 tokens/s (2)median of 3 generation reports, 128K to 160K context, 1 not stated	Gufo35.0 t/sPrompt processing (prefill) 1794.3 tokens/s (4)median of 5 generation reports, 16K to 134K context, 1 not stated
Qwen3.8 125B · 6B active 4-bitNVIDIA RTX Pro 6000 Blackwell	105.5 t/sPrompt processing (prefill) 1955.0 tokens/s (1)median of 2 generation reports, 2K to 240K context	ik_llama.cpp40.0 t/sPrompt processing (prefill) 2000.0 tokens/s (1)one generation report, 200K context
Gemma 3 12B 4-bitM5 Max 48GB	61.5 t/sPrompt processing (prefill) 1721.5 tokens/s (1)one generation report, context not stated	MLX66.9 t/sPrompt processing (prefill) 1834.9 tokens/s (1)one generation report, context not stated
Gemma 3 4B 4-bitM5 Max 48GB	150.5 t/sPrompt processing (prefill) 5760.7 tokens/s (1)one generation report, context not stated	MLX176.2 t/sPrompt processing (prefill) 5550.3 tokens/s (1)one generation report, context not stated
Qwen 2.5 Coder 14B 4-bitM5 Max 48GB	56.0 t/sPrompt processing (prefill) 1546.0 tokens/s (1)one generation report, context not stated	MLX60.8 t/sPrompt processing (prefill) 1719.3 tokens/s (1)one generation report, context not stated
Qwen 2.5 Coder 32B 4-bitM5 Max 48GB	26.3 t/sPrompt processing (prefill) 685.0 tokens/s (1)one generation report, context not stated	MLX28.4 t/sPrompt processing (prefill) 734.4 tokens/s (1)one generation report, context not stated
Qwen 2.5 Coder 7B 4-bitM5 Max 48GB	106.0 t/sPrompt processing (prefill) 2901.3 tokens/s (1)one generation report, context not stated	MLX121.8 t/sPrompt processing (prefill) 3362.9 tokens/s (1)one generation report, context not stated
Qwen3 0.6B 4-bitM5 Max 48GB	457.1 t/sPrompt processing (prefill) 23288.5 tokens/s (1)one generation report, context not stated	MLX570.6 t/sPrompt processing (prefill) 22693.8 tokens/s (1)one generation report, context not stated
Qwen3 1.7B 4-bitM5 Max 48GB	288.6 t/sPrompt processing (prefill) 11493.0 tokens/s (1)one generation report, context not stated	MLX366.0 t/sPrompt processing (prefill) 12362.2 tokens/s (1)one generation report, context not stated
Qwen3 14B 4-bitM5 Max 48GB	57.4 t/sPrompt processing (prefill) 1584.3 tokens/s (1)one generation report, context not stated	MLX62.6 t/sPrompt processing (prefill) 1738.6 tokens/s (1)one generation report, context not stated
Qwen3 32B 4-bitM5 Max 48GB	26.4 t/sPrompt processing (prefill) 667.0 tokens/s (1)one generation report, context not stated	MLX28.4 t/sPrompt processing (prefill) 730.9 tokens/s (1)one generation report, context not stated
Qwen3 4B 4-bitM5 Max 48GB	159.7 t/sPrompt processing (prefill) 4972.7 tokens/s (1)one generation report, context not stated	MLX186.0 t/sPrompt processing (prefill) 5505.7 tokens/s (1)one generation report, context not stated
Qwen3 8B 4-bitM5 Max 48GB	97.7 t/sPrompt processing (prefill) 2744.7 tokens/s (1)one generation report, context not stated	MLX110.6 t/sPrompt processing (prefill) 3150.6 tokens/s (1)one generation report, context not stated
Qwen3.6 35B · 3B active 4-bitM5 Max 48GB	124.3 t/sPrompt processing (prefill) 3342.7 tokens/s (1)one generation report, context not stated	MLX151.1 t/sPrompt processing (prefill) 3333.4 tokens/s (1)one generation report, context not stated
Qwen3.8 125B · 6B active 4-bitNVIDIA RTX Pro 6000 Blackwell	105.5 t/sPrompt processing (prefill) 1955.0 tokens/s (1)median of 2 generation reports, 2K to 240K context	NInfer172.0 t/sPrompt processing (prefill) 5905.0 tokens/s (1)one generation report, 512 context
Qwen3.8 27B 4-bitAMD RX 7900 XTX	37.1 t/sPrompt processing (prefill) 877.8 tokens/s (1)median of 3 generation reports, 64K context, 2 not stated	Ollama34.4 t/sPrompt processing (prefill) 215.8 tokens/s (1)one generation report, 64K context
Qwen3.8 125B · 6B active 4-bitM3 Ultra 512GB	31.1 t/sPrompt processing (prefill) 558.8 tokens/s (1)one generation report, 250K context	oMLX25.0 t/sone generation report, context not stated
Qwen3.8 125B · 6B active 4-bitNVIDIA DGX Spark	24.5 t/sone generation report, 128K context	SGLang35.0 t/sone generation report, 256K context
Qwen3.8 125B · 6B active 4-bitNVIDIA RTX Pro 6000 Blackwell	105.5 t/sPrompt processing (prefill) 1955.0 tokens/s (1)median of 2 generation reports, 2K to 240K context	SGLang-Prompt processing (prefill) 11352.0 tokens/s (1)no generation reports, 256K context
Qwen3.8 27B 4-bitNVIDIA RTX 5090	84.2 t/smedian of 3 generation reports, 40K to 120K context, 1 not stated	TensorSharp80.0 t/sone generation report, 40K context
Gemma 4 26B · 4B active 4-bitNVIDIA DGX Spark	69.9 t/sPrompt processing (prefill) 2657.0 tokens/s (1)one generation report, 256K context	vLLM48.2 t/sone generation report, context not stated
Qwen3.6 35B · 3B active 16-bitAMD Strix Halo 128GB	21.5 t/sPrompt processing (prefill) 484.0 tokens/s (1)one generation report, context not stated	vLLM32.6 t/sPrompt processing (prefill) 3581.0 tokens/s (1)one generation report, 2K context
Qwen3.6 35B · 3B active 4-bitNVIDIA DGX Spark	-Prompt processing (prefill) 3556.7 tokens/s (1)no generation reports, context not stated	vLLM78.3 t/sone generation report, 128K context
Qwen3.8 27B 4-bitNVIDIA RTX 3090	38.7 t/smedian of 4 generation reports, 128K to 256K context, 1 not stated	vLLM38.4 t/sPrompt processing (prefill) 561.0 tokens/s (2)one generation report, 144K to 200K context
Qwen3.8 27B 4-bitNVIDIA RTX 5090	84.2 t/smedian of 3 generation reports, 40K to 120K context, 1 not stated	vLLM73.2 t/sone generation report, context not stated

Top models on llama.cpp

Qwen3.8416 Qwen3.6136 Gemma 469 DeepSeek V4 Flash58 Qwen3.554 Qwen331 Qwen3-Coder18 Bonsai 215

Frequently asked

Is llama.cpp faster than other engines?

llamaperf has 24 matched pairs of llama.cpp and other-engine runs on the same card, model size and bit class, but none is comparable enough to say which engine is faster: the sides have too few reports or were run at different context lengths. A pair counts toward a direction only when each side has at least 3 generation reports with a stated context, and the two sides' contexts overlap. The measured figures and their conditions are in the table on this page.

What hardware does llama.cpp support?

CPUs with SIMD support, NVIDIA through CUDA, AMD through ROCm, Apple Silicon through Metal, most GPUs through Vulkan, and Intel through SYCL.

What is GGUF?

GGUF is the file format llama.cpp uses to package quantized model weights and their metadata in one file. It replaced the older GGML format.