Mistral 7B OpenOrca / 0.2.28 / native 35-token prompt, 207 eval runs / offloaded layers unknown
on M2 Pro 16GB · llama-cpp-python 0.2.28 (llama.cpp Metal backend) · 10,240 ctx
Summary
User reports Mistral-7B-OpenOrca at 25.3 t/s generation and 13 t/s prompt processing on an M2 Pro with 16GB.
Setup is llama-cpp-python 0.2.28 via LangChain with Q5_K_M, F16 K/V cache of 1280 MiB, 10240-token context and batch 2048.
The 25.31 t/s comes from 207 eval runs over 8177.10 ms, and the 12.98 t/s prompt rate from 35 tokens in 2696.68 ms; n_gpu_layers=1, so full-layer offload is not asserted.