CodeLlama 70B base / Q5_K_M / M2 Ultra Metal b2050 / native 9-token prompt and 399 eval runs
M2 Ultra 64GB · llama.cpp b2050 (19122117) / Metal · 512 ctx
- generation:
- 8.7 tokens/s
- prompt processing (prefill):
- 15.1 tokens/s
- quant:
- Q5_K_M (GGUF)
- kv:
- F16
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports CodeLlama-70B at 8.7 t/s generation and 15.1 t/s prompt processing on a Mac Studio M2 Ultra with 64 GB unified memory. Setup is llama.cpp build 2050 with Q5_K_M and F16 K/V cache, context and batch 512, 16 CPU threads, and 81/81 layers offloaded to the single Metal device. The source prints 8.72 t/s generation over 399 evaluation runs and 15.06 t/s prompt processing over 9 tokens; the stored figures are rounded to 0.1 t/s.