Mistral 7B OpenOrca / 0.2.29 / native 35-token prompt, 207 eval runs / 1/33-layer partial offload
on M2 Pro 16GB · llama-cpp-python 0.2.29 (llama.cpp Metal backend) · 10,240 ctx
Summary
User reports Mistral-7B-OpenOrca at 12 t/s generation and 8.7 t/s prompt eval on an M2 Pro with 16GB unified RAM.
Setup is llama-cpp-python 0.2.29 via LangChain with Q5_K_M, F16 K/V cache of 1280 MiB, 10240-token context and batch 2048, with only 1 of 33 layers offloaded to Metal.
The source's exact rates are 11.96 t/s over 207 eval runs and 8.72 t/s for 35 prompt tokens; the run is a single trial with partial offload, not a full-GPU benchmark.