Unknown family
on M2 Max 64GB · llama.cpp
Summary
User benchmarks llama.cpp in a GPU-accelerated Podman container on an M2 Max Mac Studio with 64GB RAM, reporting 428.47 t/s prompt processing and 35.88 t/s token generation for llama 7B Q4_0.
Setup uses llama.cpp built with Vulkan and GPU-remoting via krunkit, with -ngl 99, in a Fedora 40 container.
The container is about 40% slower than native Metal (746.06 t/s pp512, 71.36 t/s tg128), but 3x faster than a CPU-only container in prompt processing and 25% faster in token generation. Enabling -fa in the Vulkan container drops performance to 312.92 t/s pp512 and 14.10 t/s tg128.