Local model performance on your hardware

generation:

90.1 tokens/s

prompt processing (prefill):

2122.0 tokens/s

quant:

Q8_0 (GGUF)

flash attention:

on

mtp (multi-token prediction):

off

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

Public third-party measurements by Hack-Log, published 2026-10-06. Source: https://note.com/hacklog_stealth/n/nc6dfde6e4ee0?hl=en Mac mini Apple M4, 16 GB unified memory, macOS 26.4. System unified capacity is not dedicated VRAM or a measured working set. GPU core count 10 comes from the same author's 2026-10-09 Mac mini M4/16 GB companion: https://note.com/hacklog_stealth/n/n97927ee87cac?hl=en . CPU cores and threads are unpublished. One participating physical GPU is supported by the campaign's Metal startup excerpt, identifying MTL0 (Apple M4), and the exact new runtime's one-physical-device implementation. At llama.cpp b11429 revision d81235049384534c167caea52b85a694f6103d14, the Metal backend selects the system-default physical GPU; additional Metal devices would be virtual views. llama-bench's default GPU-layer count is -1, which this runtime maps to all model layers plus output. The article presents both models as the same local Metal old/new comparison. A per-model 0.8B startup trace and full 0.8B command are not published. Quantity evidence is therefore campaign hardware/backend plus pinned implementation, not a count assumed from the chip name alone. Sources: https://github.com/ggml-org/llama.cpp/blob/d81235049384534c167caea52b85a694f6103d14/ggml/src/ggml-metal/ggml-metal-device.m ; https://github.com/ggml-org/llama.cpp/blob/d81235049384534c167caea52b85a694f6103d14/tools/llama-bench/llama-bench.cpp ; https://github.com/ggml-org/llama.cpp/blob/d81235049384534c167caea52b85a694f6103d14/src/llama-model.cpp . Measured model label: Qwen3.5-0.8B, Q8_0 GGUF, 764 MiB according to the benchmark. Tested publisher, repository, revision and file hash are unknown. The benchmark does not show the exact 0.8B filename. The author's next-day inventory lists a file named Qwen3.5-0.8B-Q8_0.gguf but explicitly says its origin was not recorded: https://note.com/hacklog_stealth/n/n955380d3c86a . This is a provenance caveat, not evidence that a specific publisher produced the tested file. Reference identity only: https://huggingface.co/Qwen/Qwen3.5-0.8B/tree/2fc06364715b967f1860aea9cf38778875588b17 . That official post-trained-model repository is a base-model reference for the reported Qwen3.5-0.8B identity; it is not the tested Q8_0 artifact, and its revision must not be represented as a tested GGUF revision. New runtime: llama.cpp 0.6.0, official macOS arm64 build 11429, Metal, flash attention enabled. Native source values: prompt processing 2122 tokens/s; single-stream generation 90.1 tokens/s. These are the article's llama-bench rates, not end-to-end throughput, prompt-length/TTFT proxies or derived values. Same campaign baseline: Homebrew llama.cpp 0.4.1 build 10964, native prompt processing 2049 tokens/s and generation 87.3 tokens/s. The source says measurements were made old/new/old/new and the two invocations per runtime averaged. Its printed single-stream command is specifically for the 4B model and specifies prompt 512, generation 64, depths 0/4096 and 3 internal repetitions. The source does not print an equivalent 0.8B command or restate those lengths. Therefore 0.8B actual prompt/output counts, internal repetition count, context capacity and depth remain unknown. Do not copy the 4B configuration into those fields. Raw repetitions, spread, detailed timestamps, sampling, KV precision, CPU threads and thermal state are unpublished. Separate 0.8B batched generation totals, old/new tokens/s: 1 stream = 87.2/87.7; 4 streams = 183.5/224.2; 8 streams = 195.7/322.4. These are aggregate generation rates over all simultaneous streams, not per-stream TG. The article describes this batched experiment as reading a 512-token prompt and producing 32 tokens per stream, and prints a 4B command with context 8192 and unified KV. The exact 0.8B batched command and actual counts are absent, so those settings are contextual method descriptions only, not independently bound 0.8B run fields. The author omitted the 2-stream comparison because two results disagreed, 103.5 versus 146.3 tokens/s; no old/new mapping is assigned to those diagnostic values. No measured TTFT is published. Leave TTFT blank; do not calculate it from prompt throughput. This report contains the 0.8B model only; the 4B campaign has a separate report. No new benchmarks were run for this report.

generation:

28.3 tokens/s

prompt processing (prefill):

352.4 tokens/s

quant:

Q4_K_M (GGUF)

flash attention:

on

mtp (multi-token prediction):

off

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

Public third-party measurements by Hack-Log, published 2026-10-06. Source: https://note.com/hacklog_stealth/n/nc6dfde6e4ee0?hl=en Mac mini Apple M4, 16 GB unified memory, macOS 26.4. This is system unified capacity, not a measured working set or dedicated VRAM. GPU core count 10 is supplied by the same author's 2026-10-09 Mac mini M4/16 GB companion: https://note.com/hacklog_stealth/n/n97927ee87cac?hl=en . CPU cores and threads not reported. One participating physical GPU: the tested new-build startup excerpt identifies MTL0 (Apple M4). The stated benchmark command uses local llama-bench, without RPC/device/CPU-offload overrides. At the exact b11429 revision d81235049384534c167caea52b85a694f6103d14, llama-bench defaults n_gpu_layers=-1 and llama_model::n_gpu_layers maps negative values to all model layers plus output; the Metal implementation selects one system-default physical GPU. Extra Metal devices, if requested, are virtual views of that physical device. This is implementation-bound quantity evidence plus the actual source log, not a GPU count inferred solely from the chip name. Sources: https://github.com/ggml-org/llama.cpp/blob/d81235049384534c167caea52b85a694f6103d14/tools/llama-bench/llama-bench.cpp ; https://github.com/ggml-org/llama.cpp/blob/d81235049384534c167caea52b85a694f6103d14/src/llama-model.cpp ; https://github.com/ggml-org/llama.cpp/blob/d81235049384534c167caea52b85a694f6103d14/ggml/src/ggml-metal/ggml-metal-device.m . Model: Qwen3.5-4B, Q4_K_M GGUF, source size 2.54 GiB. Filename Qwen3.5-4B-Q4_K_M.gguf. Publisher binding comes from the author's next-day inventory, which names unsloth as the source of their Qwen3.5-4B GGUF: https://note.com/hacklog_stealth/n/n955380d3c86a . Their earlier download example gives the exact matching unsloth repository/filename: https://note.com/hacklog_stealth/n/n90637e37b463 . That earlier article labels its machine MacBook, so it is used only for artifact provenance, not this campaign's hardware. Current file reference: https://huggingface.co/unsloth/Qwen3.5-4B-GGUF/blob/e87f176479d0855a907a41277aca2f8ee7a09523/Qwen3.5-4B-Q4_K_M.gguf . This reference lists 2.74 GB and is compatible with the reported 2.54 GiB. Tested artifact hash/revision is unpublished; the HF pin is a current reference, not a claimed measured revision. New runtime: llama.cpp 0.6.0, official macOS arm64 build 11429, Metal. Report headline fields preserve the native llama-bench values: PP512 = 352.4 tokens/s; TG64 = 28.3 tokens/s. These are separate prompt-processing and generation microbenchmarks, not one 512-input/64-output request or an end-to-end rate. Flash attention enabled. Source command: llama-bench -m Qwen3.5-4B-Q4_K_M.gguf -fa 1 -p 512 -n 64 -d 0,4096 -r 3. The article's single-generation value is not explicitly mapped to one of the two requested depth settings; generation context depth remains unknown. The article separately identifies PP512 after 4096 existing tokens as 321.5 tokens/s. Context capacity is not a published fixed parameter of this standalone test. Same campaign baseline: Homebrew llama.cpp 0.4.1 build 10964, PP512 = 355.8, TG64 = 28.8; PP512 after 4096 tokens = 318.8. The author ran old/new/old/new and averaged the two invocations per runtime; the shown single-stream command specifies 3 internal repetitions. Raw repetitions, spread, exact timestamps and temperatures are not published. The new runtime's normal warmup is enabled because the shown command does not disable it; this is code-derived, not an additional source measurement. Separate batched generation sweep uses llama-batched-bench, FA on, prompt 512, generation 32 per stream, parallel streams 1/2/4/8, context 8192, unified KV enabled. Source-reported TOTAL generation tokens/s over all streams, old/new respectively: 1 = 28.2/28.3; 2 = 38.0/40.0; 4 = 36.6/64.3; 8 = 36.8/95.3. These batched aggregate rates must not be substituted for single-stream TG. No measured TTFT is published; TTFT stays blank and is not derived from PP. Sampling, KV precision, thermal state and memory footprint were not independently measured here. This report covers the 4B model. The same campaign also measured Qwen3.5-0.8B Q8_0; those rates belong in a separate model report with their own provenance caveats. No new benchmarks were run for this report.

generation:

47.4 tokens/s

prompt processing (prefill):

556.9 tokens/s

quant:

Q4_K_M (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

SOURCE: https://picklog.cc/blog/local-llm-mac-mini-16gb HARDWARE/RUNTIME: Mac mini Mac16,10, base Apple M4, 10-core GPU, 16 GB unified memory; macOS 26.4.1. Homebrew llama.cpp 0.4.0 build 10809; ggml 0.23.0. GPU quantity: 1 physical GPU. The first-party post explicitly dates measurements 2026-09-07. It reports three llama-bench repetitions on default Metal, all layers on GPU: PP512 556.85 ± 0.54 and TG128 47.37 ± 0.13 t/s. The site states its research/writing is done by an AI without a person at the keyboard; that authorship disclosure applies to these claims. The article claims raw JSONL/llama-bench JSON exist in its private research notes, but they are not linked or independently recovered here. Existing agent processes remained active (10.42 GiB baseline system RAM use), so this is not an idle-machine result. Separate CPU-only comparison: two repetitions, 4 threads, -ngl 0, PP512 52.19 ± 6.04 and TG128 34.17 t/s, not this GPU report. A separate real-generation answer reached 617 tokens at 46.60 t/s; 136 ms is model-load time, not TTFT. Its 3.28 GB peak process RSS, macmon power medians and system RAM growth describe a separate generation run, not native PP512/TG128 resident memory. The 1,929,903,264-byte published weight size matches the current pinned HF file Content-Length, but no historical measured hash is published. NATIVE METRIC: Native llama-bench tests are separate synthetic PP and TG passes, not one user request. PP counts all fixed synthetic prompt tokens; TG counts the fixed one-token decode loop, not a max-output allowance or natural language answer. The native timer is high_resolution_clock around test_prompt/test_gen, with final PP synchronization and synchronization in each TG iteration. Reported rates are means of per-repetition tokens/elapsed_time; native ± values are sample standard deviations, not total experimental uncertainty. Model load, warm-up and any prior-depth setup are outside the timed interval. No TTFT is measured or inferred. No benchmark was rerun. PARTICIPATING GPU QUANTITY: One physical Apple GPU, supported by this historical runtime Metal implementation selecting MTLCreateSystemDefaultDevice(), combined with the source-stated local Metal run. This is implementation-based physical-device proof, not a retained per-run enumeration. GPU core count is distinct from device quantity. RUNTIME PROOF: https://github.com/ggml-org/llama.cpp/blob/5266f24da75dc449bd56cbed7addb9c8e4a6a73e/tools/llama-bench/llama-bench.cpp ; https://github.com/ggml-org/llama.cpp/blob/5266f24da75dc449bd56cbed7addb9c8e4a6a73e/ggml/src/ggml-metal/ggml-metal-device.m ARTIFACT: https://huggingface.co/bartowski/Qwen2.5-3B-Instruct-GGUF/blob/f302c64a2269a69fb27b2f9473b362f5bb8e78d8/Qwen2.5-3B-Instruct-Q4_K_M.gguf . A pinned resolve HEAD returned 200 without downloading weight bytes. Current publisher/file identity and size are corroborated; the source does not provide the tested local file SHA256 or download revision. The source snapshot/hash is not the model hash. UNKNOWNS: CPU core count and benchmark thread count; full benchmark command/environment where not quoted; actual allocator-rounded context capacity; per-run duration samples; exact historical weight bytes; hardware/model resident-memory footprint for these native test windows. Unreported values remain unknown. RETRIEVAL LIMITATION: The benchmark claims above were captured from the original page through a public web reader before a later direct archival request returned HTTP 403. No local full-HTML archive was obtained. Exact historical runtime source and current pinned HF metadata were independently read from their public repositories. Post-restriction source excerpts are excluded from this report. LlamaPerf structured rates are rounded to 0.1 tokens/s to match form precision; exact native means and SDs are retained above.

generation:

13.1 tokens/s

quant:

Q4_K_M (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

Source-reported measurement by Mustafa.net, measured 2026-10-12 and published 2026-10-13. Primary source: https://mustafa.net/2026/10/13/local-llm-calculator-tested-four-machines/ . This report covers Gemma 4 12B-it Q4_K_M on ONE Mac mini M4 with 16 GB installed unified memory. Source says the official Mac llama.cpp build ran directly on Apple graphics; exact build, macOS version and CPU/GPU core counts are unstated. Installed unified memory is not dedicated VRAM. Model and quantization are source-backed. Verified upstream reference: https://huggingface.co/google/gemma-4-12B-it . This is the upstream model, NOT the tested GGUF download. Quantizer/publisher, GGUF filename, tested revision, checksum and conversion details are unknown; do not substitute a guessed Unsloth or other GGUF link. Generation headline is 13.1 tokens/s, the median of three requests after a short warm-up. Source reproduction extracts timings.predicted_per_second from llama.cpp /completion, so this is the server's generation-phase rate, not client wall throughput, TTFT, prefill or word-count estimation. Configured context is 8,192 tokens; actual prompt-token count is absent. The fixed reproduction prompt asks what a network switch does. Temperature is 0 and n_predict is 256. The prose calls these 256 generated tokens, but no raw per-request counters are published, so the command independently establishes the requested cap, not verified realized output count. Source reports roughly 1% agreement among repeats without raw samples or formal spread statistics. No prompt-processing rate, TTFT or end-to-end latency is supplied. KV precision, Flash Attention, speculative decoding/MTP, effective prefix-cache reuse, batch/microbatch, thread settings, power mode, thermals, background load and GPU placement count are unknown. The source explicitly omits Mac measured memory because its reading was unreliable; leave measured RAM/VRAM blank. This is a short warm-run source claim, not a long-context, sustained-load or quality test. Source reports the Gemma artifact as 7.1 GB; artifact size is not measured Mac memory. The article's 27.7/11.3 t/s Gemma comparison and graphics-card memory table concern the separate RTX 2060 Super machine and must not be inherited here.

generation:

17.7 tokens/s

quant:

Q4_K_M (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

Source-reported measurement by Mustafa.net, measured 2026-10-12 and published 2026-10-13. Primary source: https://mustafa.net/2026/10/13/local-llm-calculator-tested-four-machines/ . This report covers Qwen3.5 9B Q4_K_M on ONE Mac mini M4 with 16 GB installed unified memory. Source says the official Mac llama.cpp build ran directly on Apple graphics; exact build, macOS version and CPU/GPU core counts are unstated. Installed unified memory is not dedicated VRAM. Model and quantization are source-backed. Verified upstream reference: https://huggingface.co/Qwen/Qwen3.5-9B . This is the upstream model, NOT the tested GGUF download. Quantizer/publisher, GGUF filename, tested revision, checksum and conversion details are unknown; do not substitute a guessed Unsloth or other GGUF link. Generation headline is 17.7 tokens/s, the median of three requests after a short warm-up. Source reproduction extracts timings.predicted_per_second from llama.cpp /completion, so this is the server's generation-phase rate, not client wall throughput, TTFT, prefill or word-count estimation. Configured context is 8,192 tokens; actual prompt-token count is absent. The fixed reproduction prompt asks what a network switch does. Temperature is 0 and n_predict is 256. The prose calls these 256 generated tokens, but no raw per-request counters are published, so the command independently establishes the requested cap, not verified realized output count. Source reports roughly 1% agreement among repeats without raw samples or formal spread statistics. No prompt-processing rate, TTFT or end-to-end latency is supplied. KV precision, Flash Attention, speculative decoding/MTP, effective prefix-cache reuse, batch/microbatch, thread settings, power mode, thermals, background load and GPU placement count are unknown. The source explicitly omits Mac measured memory because its reading was unreliable; leave measured RAM/VRAM blank. This is a short warm-run source claim, not a long-context, sustained-load or quality test.

generation:

29.7 tokens/s

quant:

Q4_K_M (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

Source-reported measurement by Mustafa.net, measured 2026-10-12 and published 2026-10-13. Primary source: https://mustafa.net/2026/10/13/local-llm-calculator-tested-four-machines/ . This report covers Qwen3.5 4B Q4_K_M on ONE Mac mini M4 with 16 GB installed unified memory. Source says the official Mac llama.cpp build ran directly on Apple graphics; exact build, macOS version and CPU/GPU core counts are unstated. Installed unified memory is not dedicated VRAM. Model and quantization are source-backed. Verified upstream reference: https://huggingface.co/Qwen/Qwen3.5-4B . This is the upstream model, NOT the tested GGUF download. Quantizer/publisher, GGUF filename, tested revision, checksum and conversion details are unknown; do not substitute a guessed Unsloth or other GGUF link. Generation headline is 29.7 tokens/s, the median of three requests after a short warm-up. Source reproduction extracts timings.predicted_per_second from llama.cpp /completion, so this is the server's generation-phase rate, not client wall throughput, TTFT, prefill or word-count estimation. Configured context is 8,192 tokens; actual prompt-token count is absent. The fixed reproduction prompt asks what a network switch does. Temperature is 0 and n_predict is 256. The prose calls these 256 generated tokens, but no raw per-request counters are published, so the command independently establishes the requested cap, not verified realized output count. Source reports roughly 1% agreement among repeats without raw samples or formal spread statistics. No prompt-processing rate, TTFT or end-to-end latency is supplied. KV precision, Flash Attention, speculative decoding/MTP, effective prefix-cache reuse, batch/microbatch, thread settings, power mode, thermals, background load and GPU placement count are unknown. The source explicitly omits Mac measured memory because its reading was unreliable; leave measured RAM/VRAM blank. This is a short warm-run source claim, not a long-context, sustained-load or quality test.

generation:

5.8 tokens/s

prompt processing (prefill):

60.5 tokens/s

quant:

UD-Q4_K_M (GGUF)

flash attention:

on

mtp (multi-token prediction):

off

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

Source-reported historical llama-bench campaign by karlhendrik, added 2026-09-30. Pinned primary result: https://gist.github.com/karlhendrik/355bb6cbf98861b21177a6b7857e79f7/7ff331f5e10c68699eac04848ca3840708f41061#file-results-round2-md , section Added 30 Sep 2026: Qwen3.8-27B. Gist revision 7ff331f5e10c68699eac04848ca3840708f41061, last updated 2026-09-30T10:05:52Z; recovered 2026-10-13, not rerun. Exact common benchmark harness: https://gist.github.com/karlhendrik/355bb6cbf98861b21177a6b7857e79f7/7ff331f5e10c68699eac04848ca3840708f41061#file-run_llama_bench-sh . Hardware: TWO Apple Mac mini M4 machines, 16 GB installed unified RAM each, with model inference split 50/50 over a Thunderbolt 4 Bridge using llama.cpp RPC and Metal. Installed aggregate memory is 32 GB across separate hosts, not a contiguous 32 GB unified-memory device. Report hardware count must remain 2 and RAM 16 GB per device. M4a runs llama-bench; M4b runs ggml-rpc-server on MTL0. An M1 gateway mentioned elsewhere serves HTTP production clients and is not a third inference device in this native benchmark. Source GPU wired limit is 13,312 MB per mini, an allocation limit rather than measured memory. CPU/GPU core counts, macOS version, power mode, clocks, temperature, swap and precise background activity are unknown. Model: Qwen3.8-27B dense, source GGUF architecture qwen35, Unsloth UD-Q4_K_M, source size 16.5 GB. Source publisher/model/quant/size identify the matching artifact: https://huggingface.co/unsloth/Qwen3.8-27B-GGUF/blob/main/Qwen3.8-27B-UD-Q4_K_M.gguf . Verified current pinned file page: https://huggingface.co/unsloth/Qwen3.8-27B-GGUF/blob/287999af67436bfaa4f4ee09dc34af328a495ba6/Qwen3.8-27B-UD-Q4_K_M.gguf ; current observed SHA256 322e194ff79741c7baa497c240f677f54b201b0efab44ca8e50f122b39123482. Neither that current revision nor hash is stated in the measured source, so tested revision and measured file checksum remain unknown. Source does not publish this model's exact local path or raw run log. Retain UD prefix; do not relabel as ordinary Q4_K_M or infer a uniform 4-bit tensor layout. Exact parameter counts, offloaded memory and KV allocation are absent. Runtime: source-reported llama.cpp build 9731ad3 (18 Aug 2026), Metal and RPC. September 30 addition says same flags and build as above, with three repetitions and50/50 Thunderbolt split. Published common llama-bench command uses -fa 1 -ngl 99 -mmp 0 -b 1024 -ub 512 -p 512,4096 -n 128 -r 3 -o json, plus --rpc and -ts 50/50. Source notes comma-separated -ts 50,50 accidentally ran one host in an earlier discarded pass; the retained measurements use slash-separated split values. Big-model default 2048 batch failed compute-buffer allocation at the 13 GB limit; retained common flags reduce batch/microbatch to 1024/512. This is a serial single-stream benchmark across two devices, not two independent requests. Native results, source means over 3 repetitions: pp512=60.5 tokens/s; pp4096=99.4 tokens/s; tg128=5.8 tokens/s. Structured PP selects pp512 60.5; structured TG selects tg128 5.8. The pp512, pp4096 and tg128 are SEPARATE native llama-bench phase tests, not a combined 512-input/128-output or 4096-input/128-output completion. PP tests process synthetic prompt lengths 512 and 4096 respectively; TG measures the native 128-token generation microtest. No standard deviation or individual samples are published. Raw llama-bench JSON is available only on request according to the author and was not requested. Configured context capacity is unknown and left blank. The 131,072-context/q8_0-KV/temp 0.6 server config in README belongs to the production 35B RPC server, not these llama-bench settings, and must not be inherited. The benchmark harness does not set a KV dtype, CPU thread count or reusable request-prefix cache. Native bench rates are not HTTP end-to-end, TTFT or word-estimated rates. Exact timing internals of this author-reported build have not been independently reproduced. No speculative draft options are present in the native benchmark; the separate Ornith MTP experiment is not this model. Source says one mini cannot fit this artifact at its 13 GB GPU limit; there is no one-device speed result. The 16.5 GB source size refers to the model artifact, not peak RAM/VRAM. Keep measured memory fields blank. Synthetic throughput does not establish answer quality. All rates are historical source claims, not an expectation for a current build or current HF revision. Dedup: all 97 live approved M4-family catalog rows and all-status gist search read at recovery. Existing gist-linked report 1292 represents the 35B cluster campaign and is excluded; no Qwen3.8-27B two-M4 campaign was present. Existing report 1979 uses one 16 GB M4, TurboQuant 3-bit MLX, and is a different runtime/quant/device arrangement. No author outreach, model download, benchmark execution or external write was performed.

generation:

21.0 tokens/s

prompt processing (prefill):

229.2 tokens/s

quant:

Q4_K_M (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Llama 3.1 8B at 21 t/s generation and 229.2 t/s prompt processing on an M4 16GB Mac mini. Setup is llama.cpp 0.4.0 with Q4_K_M and Metal, all layers on GPU; context, KV precision and Flash Attention are unknown. The figures come from separate llama-bench tests, pp512 at 229.23 ±0.08 t/s and tg128 at 20.95 ±0.02 t/s over three runs, not a combined 512-input/128-output completion, with 10.29 GiB RAM already in use.

generation:

30.9 tokens/s

quant:

IQ4_XS (GGUF)

kv:

q4_0

flash attention:

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports a ~3.8x generation speed regression on Apple M4 Mac Mini (16 GB unified memory) with Gemma 4 26B A4B IQ4_XS. Master build (d12cc3d) achieves 8.14 tok/s versus 27.18 tok/s on tag b8680, both built from source with llama-server and identical flags. Setup uses Metal backend, --flash-attn on, q4_0 KV cache, 65536 context, and -ngl 99. The regression is reproducible and not explained by memory or swap pressure; prefill on b8680 was 55.7 tok/s.

generation:

25.0 tokens/s

quant:

4-bit (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports about 25 tokens/s on short prompts running a 35B MoE model at 4-bit with 131K context on two base M4 Mac minis (16 GB each) over Thunderbolt 4 using llama.cpp RPC mode. Setup splits the model across the two M4s with a 50/50 memory split; a 40/60 split OOMed one machine. An M1 Mac mini acts as gateway with an OpenAI-compatible API and HTTP proxy. A 64K-token prompt takes almost five minutes to first token, and prompt cache hits arrive roughly 70x sooner. User notes reasoning was on by default and that the setup is good for background agents and learning but cloud models are still used for daily work.