Llama 2 70B Chat Q4_K_M / Metal b2164 / native 56-token prompt and 333 evaluation runs
on M1 Ultra 128GB · llama.cpp b2164 / 65085c71 / Metal · 512 ctx
Summary
User reports Llama 2 70B Chat at 11.1 t/s generation and 68.1 t/s prompt processing on a Mac Studio M1 Ultra with 128GB unified memory.
Setup is llama.cpp build 2164 with Q4_K_M quantization, 512-token context, F16 K and F16 V cache, all 81 layers offloaded to the Metal GPU.
The figures are the runtime's separate prompt-eval and eval counters, not wall-clock request throughput; prompt evaluation was 56 tokens and token evaluation was 333 runs.