Llama 3.1 70B
on Unknown GPU · llama.cpp
Summary
User reports Llama 3.1 70B F16 at 2.27 t/s generation and 39.10 t/s prompt processing on a dual AMD Epyc Turin CPU system.
Setup is llama.cpp with GGUF F16 weights, 32 threads, --numa distribute, CPU only.
User found that token generation is bottlenecked by remote NUMA memory accesses caused by llamafile_sgemm() during prompt processing; running generation-only first raises it to 4.30 t/s. A similar effect is shown for phi-4 14B F16, where disabling GGML_USE_LLAMAFILE raises generation from 10.59 to 17.26 t/s but drops prompt processing from 187.03 to 85.66 t/s.