Llama 3.1 8B
on Unknown GPU · ik_llama.cpp
Summary
User benchmarks LLaMA-3.1-8B on CPU with ik_llama.cpp and llama.cpp, reporting prompt processing of 370.11 t/s with Q8_K_R8 on a Ryzen 7950X.
Setup uses llama-bench for PP-512 and TG-128, with -rtr 1 for ik_llama.cpp; AVX2/Zen4 on Ryzen-7950X and ARM on M2-Max.
User reports ik_llama.cpp is up to 7x faster for prompt processing and notes the slowest ik_llama.cpp quant beats the fastest llama.cpp quant on Ryzen-7950X.