Unknown family
M2 Max 64GB · llama.cpp
- generation:
- 35.9 tokens/s
- prompt processing (prefill):
- 428.5 tokens/s
- quant:
- Q4_0 (GGUF)
User benchmarks llama.cpp in a GPU-accelerated Podman container on an M2 Max Mac Studio with 64GB RAM, reporting 428.47 t/s prompt processing and 35.88 t/s token generation for llama 7B Q4_0. Setup uses llama.cpp built with Vulkan and GPU-remoting via krunkit, with -ngl 99, in a Fedora 40 container. The container is about 40% slower than native Metal (746.06 t/s pp512, 71.36 t/s tg128), but 3x faster than a CPU-only container in prompt processing and 25% faster in token generation. Enabling -fa in the Vulkan container drops performance to 312.92 t/s pp512 and 14.10 t/s tg128.