Local model performance on your hardware

generation:

22.6 tokens/s

prompt processing (prefill):

239.7 tokens/s

quant:

Q4_K_M (GGUF)

kv:

F16

flash attention:

off

mtp (multi-token prediction):

off

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

Public source report by SpyderMYK, posted 2026-05-12T01:56:59Z: https://github.com/ggml-org/llama.cpp/discussions/4167#discussioncomment-16886673 . Historical result recovered and checked 2026-10-13; not an independent rerun. Exact measurement date is unknown. The source gives separate native llama-bench PP512 and TG128 measurements on an Apple M4 Mac mini with 24 GB RAM and macOS 26.4. This packet concerns qwen2.5-coder:7b only. Hardware: one participating physical Apple M4 GPU. The source's ambiguous phrase "10 cores" is preserved, rather than itself being treated as a GPU count. Apple's product specification for this exact M4 Mac mini class establishes 10 GPU cores, 10 CPU cores (4 performance + 6 efficiency) and 120 GB/s nominal memory bandwidth: https://support.apple.com/en-us/121555 . These are product-spec enrichments, not observed clocks or utilization. The benchmark explicitly reports four CPU execution threads. Installed 24 GB is unified system RAM, not dedicated VRAM. Peak/resident RAM, per-run GPU memory, swap, SSD capacity, clocks, power, thermals, host isolation and exact OS build are unreported. The native 4.36 GiB model-size column is a model tensor footprint, not measured process memory or disk capacity. Artifact: Qwen2.5-Coder-7B-Instruct, GGUF Q4_K_M; native description Q4_K - Medium, 7.62 B parameters (rounded). Source names Ollama tag qwen2.5-coder:7b and the tested GGUF blob sha256-60e05f2100071479f596b964f89f510f057ce397ea22f2833a0cfe029bfc2463. The official Ollama current manifest https://registry.ollama.ai/v2/library/qwen2.5-coder/manifests/7b independently lists the identical model-layer SHA256 and 4,683,074,048 bytes. Metadata page https://ollama.com/library/qwen2.5-coder:7b/blobs/60e05f210007 confirms Q4_K_M. Only manifests and rendered metadata were read; no weight bytes were downloaded. The exact source-reported tested content hash is therefore identified, even though the download route and original quantization author are unreported. Ollama is the verified public distributor/reference; do not attribute this run to a different GGUF publisher. Current manifest SHA256 dae161e27b0e90dd1856c8bb3209201fd6736d8eb66298e75ed87571486f4364 identifies the current wrapper, not the historical tested wrapper. This distinction matters because templates can change while the model blob remains identical. Hugging Face base/reference only: https://huggingface.co/Qwen/Qwen2.5-Coder-7B-Instruct . This HF link is not claimed to be a quantized GGUF repository, the benchmark's download source, or its tested HF revision. Tested HF revision is unknown. The exact quantized artifact reference is the Ollama model metadata page and matching digest above. The benchmark runtime is llama.cpp, not Ollama: the Ollama tag identifies model packaging only. Runtime: llama.cpp llama-bench build 9100, source short hash 2e97c5f, resolved source commit 2e97c5f96f9fe2bb26f794a348e05d7a1c74baa1. Source settings are -p 512 -n 128 -ngl 99 -r 3, explicitly described as Metal backend with full GPU offload. Native table backend BLAS,MTL, threads 4. Native mean and sample standard deviation in tokens/s: PP512 239.73 +/- 2.07; TG128 22.60 +/- 0.01. Three repetitions per phase. Raw per-repeat timings and a binary hash are not available. Method audited against the exact pinned file https://github.com/ggml-org/llama.cpp/blob/2e97c5f96f9fe2bb26f794a348e05d7a1c74baa1/tools/llama-bench/llama-bench.cpp . These are separate synthetic tests, not a paired 512-token natural-language prompt followed by a 128-token answer. PP performs 512 synthetic token evaluations in batches. TG performs 128 sequential one-token evaluations, sampling random token IDs, and synchronizes each one; no generated prose is counted. Prompt and generation operation counts are native phase labels, not application requested-output caps. The clock is std::chrono::high_resolution_clock, converted to nanoseconds. Arithmetic mean of the three per-run rates is printed with sample standard deviation. Model loading and warmup are outside timing; the memory/cache is cleared at each repetition. Under unchanged pinned defaults, depth zero, batch 2048, microbatch 512, F16 K/V, FA disabled, mmap enabled, warmup enabled (full prompt for PP; one token for TG), no delay. These omitted flags are implementation-bound defaults, not separately recorded per-run telemetry. Source provides a settings summary rather than a complete shell invocation. Pinned line1219 requests n_ctx = n_prompt + n_gen + n_depth: 512 for the separate PP case and 128 for the separate TG case. These are requested benchmark context sizes, not verified effective allocation. Effective allocated context and configured application context capacity remain unknown; model metadata maximum context is not substituted. The synthetic loop contains no draft model, MTP or speculative decoding. GPU quantity proof is source protocol plus pinned implementation, not merely the compiled backend column. The source explicitly claims full Metal offload with positive -ngl99. Device selection is in https://github.com/ggml-org/llama.cpp/blob/2e97c5f96f9fe2bb26f794a348e05d7a1c74baa1/src/llama.cpp (lines125-269); GPU-layer assignment is in https://github.com/ggml-org/llama.cpp/blob/2e97c5f96f9fe2bb26f794a348e05d7a1c74baa1/src/llama-model.cpp (lines1222-1245). In https://github.com/ggml-org/llama.cpp/blob/2e97c5f96f9fe2bb26f794a348e05d7a1c74baa1/ggml/src/ggml-metal/ggml-metal.cpp the backend defaults to one logical device (line18) and permits GGML_METAL_DEVICES logical instances (lines916-934). Crucially, every such instance binds MTLCreateSystemDefaultDevice in https://github.com/ggml-org/llama.cpp/blob/2e97c5f96f9fe2bb26f794a348e05d7a1c74baa1/ggml/src/ggml-metal/ggml-metal-device.m (lines631-647), so those logical contexts do not imply multiple physical GPUs. Combined with the stated single M4 host, the tested positive offload path uses one physical Apple GPU. No independently captured startup allocation trace, utilization log or exact per-layer residency is claimed. TTFT is not reported and is left blank. No reciprocal PP, prompt duration, model-loading time, full response latency or generation time is promoted to TTFT. The rates are native synthetic benchmark throughput, without quality validation. Reproduction limits include absent raw repetitions, unspecified run date/conditions and possible local binary modifications. No inference benchmark or downloaded source program was executed during this audit. LlamaPerf structured numeric fields are rounded to the supported 0.1 tokens/s step; the exact native rates and standard deviations remain above. Source and artifact references are public; no authentication or license acceptance was needed for reading the cited metadata.

generation:

46.0 tokens/s

prompt processing (prefill):

530.0 tokens/s

quant:

Q4_K_M (GGUF)

kv:

F16

flash attention:

off

mtp (multi-token prediction):

off

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

Public source report by SpyderMYK, posted 2026-05-12T01:56:59Z: https://github.com/ggml-org/llama.cpp/discussions/4167#discussioncomment-16886673 . Historical result recovered and checked 2026-10-13; not an independent rerun. Exact measurement date is unknown. The source gives separate native llama-bench PP512 and TG128 measurements on an Apple M4 Mac mini with 24 GB RAM and macOS 26.4. This packet concerns llama3.2:3b only. Hardware: one participating physical Apple M4 GPU. The source's ambiguous phrase "10 cores" is preserved, rather than itself being treated as a GPU count. Apple's product specification for this exact M4 Mac mini class establishes 10 GPU cores, 10 CPU cores (4 performance + 6 efficiency) and 120 GB/s nominal memory bandwidth: https://support.apple.com/en-us/121555 . These are product-spec enrichments, not observed clocks or utilization. The benchmark explicitly reports four CPU execution threads. Installed 24 GB is unified system RAM, not dedicated VRAM. Peak/resident RAM, per-run GPU memory, swap, SSD capacity, clocks, power, thermals, host isolation and exact OS build are unreported. The native 1.87 GiB model-size column is a model tensor footprint, not measured process memory or disk capacity. Artifact: Llama-3.2-3B-Instruct, GGUF Q4_K_M; native description Q4_K - Medium, 3.21 B parameters (rounded). Source names Ollama tag llama3.2:3b and the tested GGUF blob sha256-dde5aa3fc5ffc17176b5e8bdc82f587b24b2678c6c66101bf7da77af9f7ccdff. The official Ollama current manifest https://registry.ollama.ai/v2/library/llama3.2/manifests/3b independently lists the identical model-layer SHA256 and 2,019,377,376 bytes. Metadata page https://ollama.com/library/llama3.2:3b/blobs/dde5aa3fc5ff confirms Q4_K_M. Only manifests and rendered metadata were read; no weight bytes were downloaded. The exact source-reported tested content hash is therefore identified, even though the download route and original quantization author are unreported. Ollama is the verified public distributor/reference; do not attribute this run to a different GGUF publisher. Current manifest SHA256 a80c4f17acd55265feec403c7aef86be0c25983ab279d83f3bcd3abbcb5b8b72 identifies the current wrapper, not the historical tested wrapper. This distinction matters because templates can change while the model blob remains identical. Hugging Face base/reference only: https://huggingface.co/meta-llama/Llama-3.2-3B-Instruct . This HF link is not claimed to be a quantized GGUF repository, the benchmark's download source, or its tested HF revision. Tested HF revision is unknown. The exact quantized artifact reference is the Ollama model metadata page and matching digest above. The benchmark runtime is llama.cpp, not Ollama: the Ollama tag identifies model packaging only. Runtime: llama.cpp llama-bench build 9100, source short hash 2e97c5f, resolved source commit 2e97c5f96f9fe2bb26f794a348e05d7a1c74baa1. Source settings are -p 512 -n 128 -ngl 99 -r 3, explicitly described as Metal backend with full GPU offload. Native table backend BLAS,MTL, threads 4. Native mean and sample standard deviation in tokens/s: PP512 530.01 +/- 0.86; TG128 45.95 +/- 0.02. Three repetitions per phase. Raw per-repeat timings and a binary hash are not available. Method audited against the exact pinned file https://github.com/ggml-org/llama.cpp/blob/2e97c5f96f9fe2bb26f794a348e05d7a1c74baa1/tools/llama-bench/llama-bench.cpp . These are separate synthetic tests, not a paired 512-token natural-language prompt followed by a 128-token answer. PP performs 512 synthetic token evaluations in batches. TG performs 128 sequential one-token evaluations, sampling random token IDs, and synchronizes each one; no generated prose is counted. Prompt and generation operation counts are native phase labels, not application requested-output caps. The clock is std::chrono::high_resolution_clock, converted to nanoseconds. Arithmetic mean of the three per-run rates is printed with sample standard deviation. Model loading and warmup are outside timing; the memory/cache is cleared at each repetition. Under unchanged pinned defaults, depth zero, batch 2048, microbatch 512, F16 K/V, FA disabled, mmap enabled, warmup enabled (full prompt for PP; one token for TG), no delay. These omitted flags are implementation-bound defaults, not separately recorded per-run telemetry. Source provides a settings summary rather than a complete shell invocation. Pinned line1219 requests n_ctx = n_prompt + n_gen + n_depth: 512 for the separate PP case and 128 for the separate TG case. These are requested benchmark context sizes, not verified effective allocation. Effective allocated context and configured application context capacity remain unknown; model metadata maximum context is not substituted. The synthetic loop contains no draft model, MTP or speculative decoding. GPU quantity proof is source protocol plus pinned implementation, not merely the compiled backend column. The source explicitly claims full Metal offload with positive -ngl99. Device selection is in https://github.com/ggml-org/llama.cpp/blob/2e97c5f96f9fe2bb26f794a348e05d7a1c74baa1/src/llama.cpp (lines125-269); GPU-layer assignment is in https://github.com/ggml-org/llama.cpp/blob/2e97c5f96f9fe2bb26f794a348e05d7a1c74baa1/src/llama-model.cpp (lines1222-1245). In https://github.com/ggml-org/llama.cpp/blob/2e97c5f96f9fe2bb26f794a348e05d7a1c74baa1/ggml/src/ggml-metal/ggml-metal.cpp the backend defaults to one logical device (line18) and permits GGML_METAL_DEVICES logical instances (lines916-934). Crucially, every such instance binds MTLCreateSystemDefaultDevice in https://github.com/ggml-org/llama.cpp/blob/2e97c5f96f9fe2bb26f794a348e05d7a1c74baa1/ggml/src/ggml-metal/ggml-metal-device.m (lines631-647), so those logical contexts do not imply multiple physical GPUs. Combined with the stated single M4 host, the tested positive offload path uses one physical Apple GPU. No independently captured startup allocation trace, utilization log or exact per-layer residency is claimed. TTFT is not reported and is left blank. No reciprocal PP, prompt duration, model-loading time, full response latency or generation time is promoted to TTFT. The rates are native synthetic benchmark throughput, without quality validation. Reproduction limits include absent raw repetitions, unspecified run date/conditions and possible local binary modifications. No inference benchmark or downloaded source program was executed during this audit. LlamaPerf structured numeric fields are rounded to the supported 0.1 tokens/s step; the exact native rates and standard deviations remain above. Source and artifact references are public; no authentication or license acceptance was needed for reading the cited metadata.

generation:

6.5 tokens/s

prompt processing (prefill):

60.8 tokens/s

quant:

UD-IQ4_XS (GGUF)

kv:

F16

mtp (multi-token prediction):

off

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

Public source measurements by singula2045, published 2026-08-30. Primary source: https://singula2045.hatenablog.com/entry/2026/08/30/000000 . This is historical public-source recovery, not a newly executed benchmark or an independent reproduction. Hardware: Mac mini M4, 10 CPU cores, 10 GPU cores, 24 GB unified system RAM, all explicitly stated by the author. Unified capacity is not dedicated VRAM or measured model residency. The author dates the purchase to December 2025. macOS version, power mode, cooling, clocks, storage and thermal state for the August run are unreported. Model: Unsloth Qwen3.8-27B-GGUF, exact filename Qwen3.8-27B-UD-IQ4_XS.gguf, GGUF UD-IQ4_XS. Source native output labels it qwen35 27B IQ4_XS - 4.25 bpw, 13.26 GiB, 27.32 B parameters; the author explicitly clarifies that the tested model is Qwen3.8-27B despite the architecture label. Exact current file reference verified: https://huggingface.co/unsloth/Qwen3.8-27B-GGUF/blob/main/Qwen3.8-27B-UD-IQ4_XS.gguf . It currently lists 14.3 GB and SHA256 40fac4050e940397dbf13087afd50f4734a11805bf9d65ef8ddd7483470e6199. This is reference metadata only: the source does not publish its tested artifact hash/revision, so the current hash is not asserted as the measured bytes. Runtime: llama.cpp build 9544, source-reported commit 98d5e8ba8, resolved to 98d5e8ba8a2642710c9871d05ac1033a3328b884. The source prints the exact command with a model-path placeholder: llama-bench -m MODEL_PATH -p 512 -n 128. Its output reports MTL,BLAS and 4 CPU threads. Headline metrics preserve the native microbenchmarks: PP512 = 60.78 +/- 1.60 tokens/s; TG128 = 6.47 +/- 0.03 tokens/s. These are separate prompt and generation tests, not one 512-input/128-output request. No TTFT is reported for this run; none is calculated from PP. Participating GPU quantity is bound to the printed command/build and the exact upstream implementation: one physical Metal GPU. This is not inferred solely from the M4 name or GPU core count. At the pinned revision, llama-bench defaults n_gpu_layers=-1, no explicit devices, layer split, main GPU 0, and no RPC target. llama_model::n_gpu_layers maps the negative value to all layers plus output; default device selection picks available GPU devices. The Metal registry defaults to one logical device; even additional GGML_METAL_DEVICES indices are simulated virtual devices and all initialize via MTLCreateSystemDefaultDevice. The source's MTL,BLAS label records registered backends rather than a per-layer offload trace. No separate startup or layer-placement log is published; the quantity/offload binding is command-and-implementation-based, not an independently measured utilization trace. Sources: https://github.com/ggml-org/llama.cpp/blob/98d5e8ba8a2642710c9871d05ac1033a3328b884/tools/llama-bench/llama-bench.cpp ; https://github.com/ggml-org/llama.cpp/blob/98d5e8ba8a2642710c9871d05ac1033a3328b884/src/llama.cpp ; https://github.com/ggml-org/llama.cpp/blob/98d5e8ba8a2642710c9871d05ac1033a3328b884/src/llama-model.cpp ; https://github.com/ggml-org/llama.cpp/blob/98d5e8ba8a2642710c9871d05ac1033a3328b884/ggml/src/ggml-metal/ggml-metal.cpp ; https://github.com/ggml-org/llama.cpp/blob/98d5e8ba8a2642710c9871d05ac1033a3328b884/ggml/src/ggml-metal/ggml-metal-device.m . Version-bound defaults, not additional measured observations: 5 internal repetitions; warmup enabled; depth 0; batch 2048; microbatch 512; F16 K/V cache; flash attention AUTO; mmap enabled. Native displayed +/- is the standard deviation of per-repetition token rates; raw repetitions are absent. The code clears sequence memory before each repetition, after its warmup. The standalone microbench creates contexts for its separate tests; no fixed serving context capacity is asserted. TG128 is the depth-0 standalone generation microbenchmark, not decode after a 512-token prompt. No speculative-decoding or MTP option is used by this command. The article separately reports a vllm-mlx-bench run on mlx-community/Qwen3.8-27B-4bit: 50 requests, 3880 total input tokens and 5926 total output tokens; source-labelled TTFT mean1742.5ms, P95/Max4186.3ms, generation6.8tok/s and processing31.1tok/s. These different-workload MLX numbers are not blended into this report; its runtime revision and exact timing implementation are unpublished, so its claimed TTFT/processing rate are not promoted as independently verified native metrics here. The source also includes M5 Pro forecasts and AI-generated Reddit summaries, which are excluded. The same author's 2026-10-03 article repeats these M4 numbers as a baseline alongside a new M5 Pro64GB run: https://singula2045.hatenablog.com/entry/2026/10/03/000000 . Existing LlamaPerf report https://llamaperf.com/report/6296aa06-268b-444f-a940-1e5993e2e9fa covers that M5 PP381.62/TG18.45 campaign; this M4 PP60.78/TG6.47 campaign is distinct. The October macOS27.0.1 and newer-machine details are not transferred to the August M4 run. Structured rates are rounded to the form's 0.1 tok/s step; exact source rates are retained above.

generation:

12.9 tokens/s

prompt processing (prefill):

138.5 tokens/s

quant:

UD-Q4_K_XL (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

SOURCE: https://vangsebastian.ch/blog/the-memory-bus-does-not-negotiate/ HARDWARE/RUNTIME: Mac mini Mac16,10, base Apple M4, 10-core GPU, 24 GiB unified memory; macOS 26.5.1. Homebrew llama.cpp b10090, 7347430f4. GPU quantity: 1 physical GPU. Article filed 2026-07-28, verified 2026-08-07; exact measurement timestamp is not separately stated. Quoted command: llama-bench -m models/gemma-4-12b-it-UD-Q4_K_XL.gguf -p 512 -n 128 -r 5. A rendered terminal panel reports BLAS/MTL, 11.91 B parameters, PP512 138.51 ± 0.11 and TG128 12.91 ± 0.01 t/s and MTL0 (Apple M4). Five repetitions took 76 seconds total wall time, which is not either native timer. The source says this is verbatim rendered terminal text, not a screenshot. Its physicalMemory query reports 25,769,803,776 bytes = 24 GiB. RecommendedMaxWorkingSetSize 19,069,665,280 bytes is a device budget, not measured model residency. The article warns that later interleaved runs reached 12.71/12.79 t/s and PP drifted 129–134 under sustained load. Retain headline repeatability± SD and this between-run caveat. A separate two-repetition depth 0/2048/8192/32768 ladder reports PP 138.8/122.1/108.7/76.6 and TG 12.84/12.00/11.83/9.26; it is held outside the benchmark_cells because the already-read source section did not bind the extra timed-token counts/complete sweep command. KV memory diagrams are estimates. Source 7,366,423,360-byte disk file exactly matches current pinned HF Content-Length; tested local hash remains unknown. NATIVE METRIC: Native llama-bench tests are separate synthetic PP and TG passes, not one user request. PP counts all fixed synthetic prompt tokens; TG counts the fixed one-token decode loop, not a max-output allowance or natural language answer. The native timer is high_resolution_clock around test_prompt/test_gen, with final PP synchronization and synchronization in each TG iteration. Reported rates are means of per-repetition tokens/elapsed_time; native ± values are sample standard deviations, not total experimental uncertainty. Model load, warm-up and any prior-depth setup are outside the timed interval. No TTFT is measured or inferred. No benchmark was rerun. PARTICIPATING GPU QUANTITY: One physical Apple GPU, supported by this historical runtime Metal implementation selecting MTLCreateSystemDefaultDevice(), combined with the source-stated local Metal run. This is implementation-based physical-device proof, not a retained per-run enumeration. GPU core count is distinct from device quantity. RUNTIME PROOF: https://github.com/ggml-org/llama.cpp/blob/7347430f4466d4f55cfb841974ee64b80fc18d93/tools/llama-bench/llama-bench.cpp ; https://github.com/ggml-org/llama.cpp/blob/7347430f4466d4f55cfb841974ee64b80fc18d93/ggml/src/ggml-metal/ggml-metal-device.m ARTIFACT: https://huggingface.co/unsloth/gemma-4-12b-it-GGUF/blob/fc034cfff751157913579611efad8462ac1be606/gemma-4-12b-it-UD-Q4_K_XL.gguf . A pinned resolve HEAD returned 200 without downloading weight bytes. Current publisher/file identity and size are corroborated; the source does not provide the tested local file SHA256 or download revision. The source snapshot/hash is not the model hash. UNKNOWNS: CPU core count and benchmark thread count; full benchmark command/environment where not quoted; actual allocator-rounded context capacity; per-run duration samples; exact historical weight bytes; hardware/model resident-memory footprint for these native test windows. Unreported values remain unknown. RETRIEVAL LIMITATION: The benchmark claims above were captured from the original page through a public web reader before a later direct archival request returned HTTP 403. No local full-HTML archive was obtained. Exact historical runtime source and current pinned HF metadata were independently read from their public repositories. Post-restriction source excerpts are excluded from this report. LlamaPerf structured rates are rounded to 0.1 tokens/s to match form precision; exact native means and SDs are retained above.