Local model performance on your hardware

generation:

9.9 tokens/s

prompt processing (prefill):

160.8 tokens/s

quant:

IQ3_XXS (GGUF)

User benchmarks Qwen3.8-Flash-Next (125B) on an M4 Pro Mac mini with 24GB unified memory, comparing their own engine MoEShift against Strata. With a 6,009-token prompt and 256 generated tokens, MoEShift reached 160.8 tok/s prefill and 9.9 tok/s decode in 63.3s total, against Strata's 61.8 tok/s prefill and 7.8 tok/s decode in 129.9s. Both engines used the GSQ-RCO IQ3_XXS GGUF quant with an 8,192-token context window and no cached prompt tokens; times exclude process startup. On a short 66-token prompt MoEShift decoded at 10.44 tok/s versus Strata's 8.4 tok/s, with peak process footprint 6.4GB versus 14GB. The user notes these are first measurements and more benchmarking is planned; the code is not yet public.

generation:

11.0 tokens/s

prompt processing (prefill):

180.0 tokens/s

quant:

GSQ RCO IQ3 XXS

User reports Qwen3.8-Flash-Next 125B at 11 t/s generation and 180 t/s prefill on a Mac mini M4 Pro with 24GB RAM, using a custom inference implementation. Setup uses ISTA Lab's GSQ RCO IQ3 XXS quantization, roughly 76GB on disk, with support for up to 65K tokens of context. The model streams from storage, with peak runtime memory measured at 9.5GB. Speed is 11 t/s with short prompts and 8.5 t/s with longer contexts; the first implementation ran under 3 t/s generation and around 40 t/s prefill. The user is also testing it with a coding agent starting from over 10k tokens of context.

generation:

37-39 tokens/s

prompt processing (prefill):

856-895 tokens/s

quant:

3-bit

User reports Qwen3.8 Flash-Next at 37-39 tok/s writing and 856-895 tok/s reading the prompt at 60k context, split across five devices: an RTX 2060 Super PC, an RX 9060 XT PC, an M4 Pro MacBook with 24 GB, and two iPhone 17 Pro Maxes (A19 Pro). Setup chains the devices over Ethernet and 10 Gb/s USB-C, with every weight held in VRAM, pinned RAM or unified memory and no disk streaming. Pipelined speculation from the model's own MTP draft head lands about 2.2 tokens per step. At 16k context the user measured 42 tok/s writing and 728-744 tok/s prefill; at 32k, 39-41 tok/s and 844-852 tok/s. The agent session averaged 35 tok/s with bursts past 40. A second layout with 8-bit KV on the Mac and phones writes 31-33 tok/s at 60k and read one prompt past 164k tokens. PC 2's 16 GB caps prefill at about 980 tok/s.

generation:

297.4 tokens/s

prompt processing (prefill):

4993.0 tokens/s

quant:

Q4 (GGUF)

User reports 297.4 t/s generation and 4993 t/s prompt processing. Source evidence (verbatim): Source: https://arxiv.org/html/2607.00501v1 # Original M4 Pro 24GB Qwen3 0.6B four-bit llama.cpp baseline measurements Original Base Compute research paper, July 1, 2026, section 4 and Tables 1–2. Selected run is the authors' llama.cpp baseline, not BaseRT. Hardware: Apple M4 Pro 24GB unified memory, 16 GPU cores. Chassis, CPU core count and macOS version unreported; do not infer Mac mini. Model Qwen3-0.6B, GGUF weights, generic Q4 / 4-bit label only. Exact quantization recipe, tested artifact repository and revision unknown. Verified upstream model reference only: https://huggingface.co/Qwen/Qwen3-0.6B ; this safetensors repository is not the tested GGUF. Runtime llama.cpp Metal, source-labelled b9630 (June 2026), default settings. Full commit and command unreported. Generation 297.4 tokens/s over 128 autoregressive generated tokens (TG128). Prefill 4993 tokens/s at 512 prompt tokens (PP512). These are separate synthetic tests, not a combined 512-input/128-output request. Cache depth, KV precision, repetition count, warmup, concurrency and speculation unreported. Resident usage and weight-file size unreported. Standard deviations are omitted by the authors, who say they are generally below 3% for decode and 10% for prefill; these are not per-row error bars. Other runtime, prompt-length and quantization rows are distinct configurations, not averaged here.

generation:

2.9 tokens/s

quant:

Q4_K_M (GGUF)

User reports Qwen3-Next 80B-A3B at 2.9 tok/s on an M4 Pro 24GB MacBook Pro, running llama.cpp's WebGPU backend in a Chrome tab with MoE experts paged in from disk. Setup is llama.cpp (ggml-webgpu) with a Q4_K_M GGUF, 48.4 GB file, 4.89 GB resident on the GPU, 48 s initial load from disk. The 80B reads about 359 MB from disk per token; the author calls it a proof of concept. OLMoE 1B-7B Q4_K_M reached 22.4 tok/s and LFM2 8B-A1B Q4_K_M 17.1 tok/s in the same recording. Paged and unpaged replies were identical; against llama.cpp on Metal 4 to 8 of 9 replies matched. Chrome or Edge 137+ only, tested on Apple silicon only.

generation:

28.0 tokens/s

quant:

int4 affine group64 (MLX)

User reports Edge0 at 28 t/s on a single Apple Mac mini M4 Pro with 24 GB unified memory. Setup is the Edge0 MLX backend with int4 affine group-64 weights, prediction-as-routing staged decode, whole-layer prefill and SSD-streamed experts, using a shared LRU of 64 expert bundles. The figure is the paper's single-device warm decode result for the released Edge0 8B K=8 low-memory profile, a median over three paired rounds; a separate LRU1024 variant reaches 31.8 t/s with about 0.9 GiB more memory.

generation:

20.4 tokens/s

quant:

int4 affine group64 (MLX)

User reports Edge0 at 20.4 t/s on a single Apple Mac mini M4 Pro with 24 GB unified memory. Setup is the Edge0 MLX pipeline with int4 affine group-64 weights, a K=4 production profile, and SSD-streamed experts; allocated context is not reported. The figure is the paper's single-device warm decode result, a median over three paired rounds, and the timed output count is absent. The same paper reports prefill at 113 t/s cold and 140 t/s warm.

generation:

11.4 tokens/s

prompt processing (prefill):

96.8 tokens/s

quant:

Q4_K_M (GGUF)

flash attention:

on

User reports Qwen3.8 27B at 11.4 t/s generation and 96.8 t/s prompt on a Mac mini M4 Pro with 24GB unified memory. Setup is llama.cpp b10488 with Metal, Flash Attention on, all layers offloaded, and Q4_K_M quant. The figures come from separate llama-bench pp2048 and tg128 tests, mean of three repetitions, not one request; run-to-run variance is claimed under 3%.

generation:

36-38 tokens/s

prompt processing (prefill):

95-125 tokens/s

quant:

QAT (GGUF)

kv:

q8_0

User reports Gemma 4 26B-A4B QAT at 36-38 tok/s generation on an M4 Pro 24GB MacBook Pro. Setup is Ollama (llama.cpp) with QAT 4-bit weights, q8_0 KV cache, flash attention on, and a 64K context window; the model is 14 GB and 100% GPU-resident. Prefill is 95-125 tok/s on short prompts, cold load ~16 s, warm reload 0.4 s. The user notes the model squeezes the rest of the OS (9% memory free, swap nearly full) and that a dense 31B would spill to CPU.

generation:

23.6 tokens/s

prompt processing (prefill):

55.0 tokens/s

quant:

Q4_0 (GGUF)

User reports Gemma 4 26B-A4B at 23.6 tok/s decode and 55 tok/s prompt processing in a browser tab on an M4 Pro 24GB Mac. Setup is LocalMind (WebGPU) with Google's QAT Q4_0 GGUF (14.4 GB), a 4 GB expert cache, and routed experts streamed from disk; the Chrome GPU process sits at 6.9 GB with about 8.6 GB of experts on disk. llama.cpp Metal does 70.6 and 204 tok/s on the same Mac, so the tab is about 3x slower at decode. The user also ran Qwen3.6 35B-A3B (unsloth Q8_0, 36.9 GB) at 9.9 tok/s decode with 2.2 s to first token, a file larger than the machine's memory. Output matched llama.cpp on 9/9 Gemma conversations and 8/8 Qwen cuts.

generation:

82.8 tokens/s

quant:

int4

User benchmarks Gemma 4 E4B with MTP speculative decoding on a MacBook Pro M4 Pro 24GB, reporting 82.8 tok/s versus 44.6 tok/s baseline (1.86x mean, 2.03x best speedup). Setup is LiteRT-LM Metal GPU with int4 weights at batch=1, temperature=0, using real prompts; the E2B model reaches 144.3 tok/s with MTP versus 84.2 tok/s baseline. HF transformers + MPS with bf16 weights achieves only 12.1 to 15.1 tok/s (1.24x), and llama.cpp Metal Q8_0 cannot engage MTP for Gemma 4. The user notes that synthetic-token benchmarks show MTP as a regression (0.93x) because random sequences yield near-zero drafter acceptance, and warns that E4B bf16 on 24GB Macs can trigger kernel panics from Metal wired-memory pressure.