Local model performance on your hardware

generation:

297.4 tokens/s

prompt processing (prefill):

4993.0 tokens/s

quant:

Q4 (GGUF)

User reports 297.4 t/s generation and 4993 t/s prompt processing. Source evidence (verbatim): Source: https://arxiv.org/html/2607.00501v1 # Original M4 Pro 24GB Qwen3 0.6B four-bit llama.cpp baseline measurements Original Base Compute research paper, July 1, 2026, section 4 and Tables 1–2. Selected run is the authors' llama.cpp baseline, not BaseRT. Hardware: Apple M4 Pro 24GB unified memory, 16 GPU cores. Chassis, CPU core count and macOS version unreported; do not infer Mac mini. Model Qwen3-0.6B, GGUF weights, generic Q4 / 4-bit label only. Exact quantization recipe, tested artifact repository and revision unknown. Verified upstream model reference only: https://huggingface.co/Qwen/Qwen3-0.6B ; this safetensors repository is not the tested GGUF. Runtime llama.cpp Metal, source-labelled b9630 (June 2026), default settings. Full commit and command unreported. Generation 297.4 tokens/s over 128 autoregressive generated tokens (TG128). Prefill 4993 tokens/s at 512 prompt tokens (PP512). These are separate synthetic tests, not a combined 512-input/128-output request. Cache depth, KV precision, repetition count, warmup, concurrency and speculation unreported. Resident usage and weight-file size unreported. Standard deviations are omitted by the authors, who say they are generally below 3% for decode and 10% for prefill; these are not per-row error bars. Other runtime, prompt-length and quantization rows are distinct configurations, not averaged here.

generation:

2.9 tokens/s

quant:

Q4_K_M (GGUF)

User reports Qwen3-Next 80B-A3B at 2.9 tok/s on an M4 Pro 24GB MacBook Pro, running llama.cpp's WebGPU backend in a Chrome tab with MoE experts paged in from disk. Setup is llama.cpp (ggml-webgpu) with a Q4_K_M GGUF, 48.4 GB file, 4.89 GB resident on the GPU, 48 s initial load from disk. The 80B reads about 359 MB from disk per token; the author calls it a proof of concept. OLMoE 1B-7B Q4_K_M reached 22.4 tok/s and LFM2 8B-A1B Q4_K_M 17.1 tok/s in the same recording. Paged and unpaged replies were identical; against llama.cpp on Metal 4 to 8 of 9 replies matched. Chrome or Edge 137+ only, tested on Apple silicon only.

generation:

11.4 tokens/s

prompt processing (prefill):

96.8 tokens/s

quant:

Q4_K_M (GGUF)

flash attention:

on

User reports Qwen3.8 27B at 11.4 t/s generation and 96.8 t/s prompt on a Mac mini M4 Pro with 24GB unified memory. Setup is llama.cpp b10488 with Metal, Flash Attention on, all layers offloaded, and Q4_K_M quant. The figures come from separate llama-bench pp2048 and tg128 tests, mean of three repetitions, not one request; run-to-run variance is claimed under 3%.