Local model performance on your hardware

generation:

33.6 tokens/s

prompt processing (prefill):

200.7 tokens/s

quant:

Q4_K_M (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

First-party measurements by kun432 at https://zenn.dev/kun432/scraps/1fc03a48d5742c, dated 2023-09-16. The author explicitly identifies Mac mini 2023, Apple M2 Pro with 12 CPU cores, 19 GPU cores and 32 GB installed unified memory. GPU quantity is exactly one participating Metal device: this specific GPU-enabled run prints device discovery and explicit selection of Apple M2 Pro as the default device. This is run-bound selection, not an assumption based on the Mac model; 19 GPU cores are not 19 GPUs. The source also shows CPU-only measurements, which are not merged into this GPU profile. Model Code Llama 7B Instruct, quantization Q4_K_M, filename codellama-7b-instruct.Q4_K_M.gguf, GGUF. The source identifies the exact local filename but no verified tested HF artifact was established in this pass, so the artifact URL remains blank. Later 13B and 34B runs on the page lack their own device selection logs and are not promoted with this 7B result. Runtime llama.cpp ./main with Metal; source commit, build version, binary hash and compiler flags are unknown. Command uses the given GGUF and --temp 0.1 and the shown Japanese Fibonacci-code instruction prompt. The GPU-enabled command omits -ngl but logs establish active Metal default-device initialization. Exact layer count, context capacity, batch/microbatch, CPU threads, KV precision and flash attention are unreported and not inferred from a current version. Native prompt eval 323.85 ms / 65 tokens = 200.71 tokens/s. Native eval 9576.08 ms / 322 runs = 33.63 tokens/s. Sample count 323; exact output token count is not substituted for native eval-run count. These are actual request phase timers, not an end-to-end average. One displayed run, with no raw repetitions or warmup/cache protocol. No measured TTFT. Native prompt-eval duration and load duration must not be relabelled as TTFT. OS, clock speeds, power, thermals, swap state, disk size and measured peak process/GPU allocation are unknown. Installed 32 GB unified RAM is explicit and separate from file size or runtime buffer figures. No software, weights or benchmarks were executed to verify the author's result. Verified official base/reference model: https://huggingface.co/meta-llama/CodeLlama-7b-Instruct-hf . The public Meta Llama model card identifies the 7B instruction-tuned Transformers checkpoint. This is an upstream reference only, not identification of the measured Q4_K_M GGUF publisher, bytes, hash or historical revision. The tested quantized artifact remains unverified. Structured PP/TG are rounded to the form’s 0.1 tokens/s step; exact source rates remain above.

generation:

34.8 tokens/s

prompt processing (prefill):

227.6 tokens/s

quant:

Q4_K_M (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

First-party measurements by kun432 at https://zenn.dev/kun432/scraps/ea13185fd8a086, dated 2023-10-18. The author explicitly identifies Mac mini 2023, Apple M2 Pro with 12 CPU cores, 19 GPU cores and 32 GB installed unified memory. GPU quantity is exactly one participating Metal device: this specific GPU-enabled run prints device discovery and explicit selection of Apple M2 Pro as the default device. This is run-bound selection, not an assumption based on the Mac model; 19 GPU cores are not 19 GPUs. The source also shows CPU-only measurements, which are not merged into this GPU profile. Model ALMA-7B-Ja, quantization Q4_K_M, filename webbigdata-ALMA-7B-Ja-q4_K_M.gguf, GGUF. Source links mmnga/webbigdata-ALMA-7B-Ja-gguf and names the exact file. Current metadata independently verifies it at https://huggingface.co/mmnga/webbigdata-ALMA-7B-Ja-gguf/blob/40e7546c469f9e6d0dac307dbfa90bb3c9c88b07/webbigdata-ALMA-7B-Ja-q4_K_M.gguf; the historical tested revision/hash is unknown. This is the original ALMA-7B-Ja run, not the later v2 mentioned elsewhere in the page. Runtime llama.cpp ./main with Metal; source commit, build version, binary hash and compiler flags are unknown. Command uses the given GGUF and -n 128 with the shown Japanese-to-English translation template. The GPU-enabled command omits -ngl but logs establish active Metal default-device initialization. Exact layer count, context capacity, batch/microbatch, CPU threads, KV precision and flash attention are unreported and not inferred from a current version. Native prompt eval 347.14 ms / 79 tokens = 227.58 tokens/s. Native eval 1265.47 ms / 44 runs = 34.77 tokens/s. Sample count 45; exact output token count is not substituted for native eval-run count. These are actual request phase timers, not an end-to-end average. One displayed run, with no raw repetitions or warmup/cache protocol. No measured TTFT. Native prompt-eval duration and load duration must not be relabelled as TTFT. OS, clock speeds, power, thermals, swap state, disk size and measured peak process/GPU allocation are unknown. Installed 32 GB unified RAM is explicit and separate from file size or runtime buffer figures. No software, weights or benchmarks were executed to verify the author's result. Structured PP/TG are rounded to the form’s 0.1 tokens/s step; exact source rates remain above.

generation:

8.6 tokens/s

prompt processing (prefill):

21.9 tokens/s

quant:

IQ4_XS (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8 27B IQ4_XS with vision projector on an M2 MacBook Pro 32GB, at 21.9 t/s prompt and 8.6 t/s generation. Setup is llama.cpp built from source, with context set to 128K due to memory constraints. User notes the model is slow on Mac but that thinking quality is good, and mentions previous experience with Qwen3.5 35B-A3B.