Local model performance on your hardware

generation:

18.0 tokens/s

User reports Qwen3-14B writing at 18 tok/s on a MacBook Pro M2 Pro with 16GB, where the model fits entirely in unified memory. Setup is llama.cpp with the Metal build, run through the MAFDET app, which reasons about the message while the user is still typing. The app cuts the wait from Enter to the first word from 32.7 s to 12.1 s on 6 word problems, with 6/6 accuracy both ways; on Qwen3-4B the mean/median wait drops from 18.1s/16.7s to 5.6s/4.1s over 24 problems. The same 14B on the user's RTX 2060 is split with CPU at 4.5-5 tok/s. Tested on one Mac only.

generation:

130.6 tokens/s

quant:

Q4_K_M (GGUF)

kv:

Q8

User compares an RTX 5080 desktop against a rented DGX Spark on Qwen3-8B Q4_K_M, where the PC decodes at 130.55 t/s versus 42.43 t/s on Spark at 512 input tokens, roughly three times faster. Both ran llama.cpp release b11889 with Q8 KV cache, Flash Attention and a 40,960-token context; the PC kept the whole model on its 16 GB card. Decode falls to 79.09 t/s on the PC and 25.64 t/s on Spark at 32,768 input tokens. With Qwen3-32B the PC needed CPU offload (40 GPU layers) and managed 4.92 t/s against Spark's 10.50 t/s. Aggregate batched throughput at 8 concurrent requests was 471.75 t/s on the PC and 209.02 t/s on Spark.

generation:

133.0 tokens/s

quant:

Q8_0 (GGUF)

User benchmarks Qwen3-8B Q8_0 with DSpark speculative decoding in llama.cpp on an RX 7900 GRE 16GB, reaching 133 t/s on ROCm versus 59 t/s without speculation, and 115 t/s versus 62 t/s on Vulkan. Setup is llama.cpp with Q8_0 weights, greedy decoding, one user and 32 prompts per setting. DFlash gave 1.5-1.9x, a 0.6B draft model about 1.3x, and n-gram lookup did nothing. The best number of guessed tokens is 3 on Vulkan and 7 on ROCm, and prompt processing is about 8% slower with DSpark loaded. Only 5-6 of 16 answers matched normal decoding word for word.

generation:

297.4 tokens/s

prompt processing (prefill):

4993.0 tokens/s

quant:

Q4 (GGUF)

User reports 297.4 t/s generation and 4993 t/s prompt processing. Source evidence (verbatim): Source: https://arxiv.org/html/2607.00501v1 # Original M4 Pro 24GB Qwen3 0.6B four-bit llama.cpp baseline measurements Original Base Compute research paper, July 1, 2026, section 4 and Tables 1–2. Selected run is the authors' llama.cpp baseline, not BaseRT. Hardware: Apple M4 Pro 24GB unified memory, 16 GPU cores. Chassis, CPU core count and macOS version unreported; do not infer Mac mini. Model Qwen3-0.6B, GGUF weights, generic Q4 / 4-bit label only. Exact quantization recipe, tested artifact repository and revision unknown. Verified upstream model reference only: https://huggingface.co/Qwen/Qwen3-0.6B ; this safetensors repository is not the tested GGUF. Runtime llama.cpp Metal, source-labelled b9630 (June 2026), default settings. Full commit and command unreported. Generation 297.4 tokens/s over 128 autoregressive generated tokens (TG128). Prefill 4993 tokens/s at 512 prompt tokens (PP512). These are separate synthetic tests, not a combined 512-input/128-output request. Cache depth, KV precision, repetition count, warmup, concurrency and speculation unreported. Resident usage and weight-file size unreported. Standard deviations are omitted by the authors, who say they are generally below 3% for decode and 10% for prefill; these are not per-row error bars. Other runtime, prompt-length and quantization rows are distinct configurations, not averaged here.

generation:

14.0 tokens/s

quant:

Q4_K_S (GGUF)

User compares DanilAI and LM Studio running Qwen3-30B-A3B-Instruct-2507 on an RTX 4050 Laptop (6 GB VRAM) with a Ryzen 7 7435HS and 24 GB RAM. LM Studio with Q4_K_S quantization reached 14.03 tokens/sec; DanilAI with hybrid FP16/8-bit/4-bit precision plus DanilDrive compression reached 25.0 tokens/sec. A separate test ran Qwen3-0.6B in FP16 with DanilDrive, reporting approximately 45% lossless compression. OBS Studio was recording throughout and may have affected performance.

generation:

89.0 tokens/s

prompt processing (prefill):

6700.0 tokens/s

User reports Qwen3 8B at 89 t/s decode and 6.7k tok/s prefill on an RTX 5090 using PulsarLM, a new inference engine that evicts low-attention token pages to keep the context window bounded. Setup is PulsarLM with custom attention kernels that accumulate attention mass per token page and remove low-mass pages at intervals; RoPE embeddings are re-rotated to keep positions contiguous. Context snapshots avoid re-prefills at the cost of extra RAM. Batched decode reaches 1.2k tok/s at 16 concurrent requests. The engine is early-stage, tested only on Linux x86_64, and supports only smaller/older models; the author removed a BM25+attention recall mechanism and recommends pairing with external RAG.

generation:

68.1 tokens/s

prompt processing (prefill):

3057.0 tokens/s

quant:

Q4_K_M (GGUF)

User reports Qwen3-14B at 68.1 t/s generation and 3057 t/s prompt processing on a 2x RTX 2080 Ti node. Setup is llama.cpp with Q4_K_M quant, 32768 context, and a Qwen3-0.6B-Q8_0 draft model for speculative decoding with depth 6. The user also reports Pixtral-12B at 77.0 t/s generation and 2862 t/s prompt processing, and a distributed 4-stream aggregate of 118.68 t/s. They ask if Linux would give better t/s and mention P2P is working on Windows.

generation:

6.4 tokens/s

prompt processing (prefill):

84.9 tokens/s

quant:

Q8_0 (GGUF)

flash attention:

on

User reports Qwen3-32B-Q8_0 at 6.4 t/s generation and 84.9 t/s prompt processing on an AMD Strix Halo 128GB system. Setup is llama.cpp with the Vulkan (RADV) backend, FA 1 and ngl 99, on a single AMD Radeon Graphics (RADV GFX1151) device. The figures come from separate llama-bench phases: PP512 at 84.87 +/- 0.35 t/s and TG128 at 6.43 +/- 0.01 t/s. Context depth, thread count and KV cache type are unreported.

generation:

6.4 tokens/s

prompt processing (prefill):

51.1 tokens/s

quant:

Q8_0 (GGUF)

flash attention:

on

User reports Qwen3-32B at 6.4 t/s generation and 51.1 t/s prompt processing on an AMD Strix Halo 128GB system. Setup is llama.cpp with Q8_0 and flash attention enabled, ngl 99, on one ROCm device (gfx1151). The figures are separate llama-bench phases: PP512 at 51.12 +/- 0.24 t/s and TG128 at 6.39 +/- 0.01 t/s, not a same-request pair. Context depth is not printed.

generation:

40.0 tokens/s

prompt processing (prefill):

586.0 tokens/s

quant:

MXFP4 (GGUF)

flash attention:

on

User reports Qwen3-Coder-Next-80B at 40 t/s generation and 586 t/s prompt processing on a single Radeon 8060S iGPU in an AMD Strix Halo system with 128 GB unified memory. Setup is llama.cpp with the MXFP4 quant, full offload, batch 2048 and flash attention, using the RADV Vulkan backend. The figures are separate 512-token prompt and 128-token generation microbenchmarks, not one end-to-end request; the source notes post-hoc MXFP4 can underperform conventional quants.

generation:

92.0 tokens/s

prompt processing (prefill):

1143.0 tokens/s

quant:

UD-Q4_K_XL (GGUF)

flash attention:

on

User reports Qwen3-30B-Instruct-2507 at 92 t/s generation and 1143 t/s prompt processing on an AMD Strix Halo 128GB system with a single Radeon 8060S iGPU. Setup is llama.cpp via RADV with UD-Q4_K_XL, full offload (ngl 999), batch 2048 and flash attention. The figures are separate llama-bench pp512 and tg128 microbenchmarks, not one end-to-end request; the AMDVLK run measured 936.0 t/s prompt and 93.0 t/s generation.

generation:

73.7 tokens/s

prompt processing (prefill):

843.3 tokens/s

quant:

Q4_K_M (GGUF)

flash attention:

on

mtp (multi-token prediction):

off

User reports Qwen3-30B-A3B at 73.7 t/s generation and 843.3 t/s prompt processing on an AMD Strix Halo 128GB box. Setup is llama.cpp with a Q4_K_M GGUF, full GPU offload (ngl 999) and flash attention on ROCm/HIP 7.2.4, one ROCm device. The headline is the d0 arm, PP843.25 ±5.39 and TG73.74 ±0.17 tok/s; at depth 4096 it drops to PP209.17 and TG47.48 tok/s, where the depth is pre-existing KV state rather than a timed prefill.

generation:

94.5 tokens/s

prompt processing (prefill):

806.0 tokens/s

quant:

Q4_K_M (GGUF)

kv:

F16

User reports Qwen3-30B-A3B-Instruct-2507 at 94.5 t/s generation and 806 t/s prompt processing on an M4 Pro with 64 GB unified memory. Setup is dasLLAMA version 16 with Q4_K_M weights, f16 KV cache, and one Metal GPU, with 640 positions allocated. The figures are native PP512 and TG128 means over five repetitions, not a single 512-input/128-output request.

generation:

77.2 tokens/s

prompt processing (prefill):

780.2 tokens/s

quant:

Q4_K_M (GGUF)

kv:

F16

User reports Qwen3-30B-A3B-Instruct-2507 at 77.2 t/s generation and 780.2 t/s prompt processing on an M4 Pro 64GB. Setup is llama.cpp commit 6fdd0ac89 with Q4_K_M, Metal backend, -ngl 99, and 10 CPU threads on one physical GPU. The figures are native llama-bench PP512 and TG128 means over five repetitions at depth 0, not a single chat request.

generation:

81.2 tokens/s

prompt processing (prefill):

518.0 tokens/s

quant:

4-bit (MLX)

mtp (multi-token prediction):

off

User reports Qwen3 8B 4-bit at 81.2 t/s generation and 518 t/s prompt processing on an M1 Ultra 128GB Mac Studio. Setup is mlxcel 0.7.0-beta.1 with FP16 KV cache, one Metal GPU, 512 input tokens and 128 output tokens, greedy temperature 0. The decode figure covers 128 output tokens over 127 forward steps; prefill excludes loading, tokenization and transport, so it is not end-to-end TTFT.

generation:

116.9 tokens/s

prompt processing (prefill):

942.3 tokens/s

quant:

4-bit (MLX)

mtp (multi-token prediction):

off

User reports Qwen3 4B 4-bit at 116.9 t/s generation and 942.3 t/s prompt processing on an M1 Ultra 128GB Mac Studio. Setup is mlxcel 0.7.0-beta.1 with FP16 KV cache, one request of 512 input tokens and 128 output tokens, greedy temperature 0, no speculative decoding. The decode figure covers 128 output tokens over 127 forward steps; prefill excludes loading, tokenization and transport, so it is not end-to-end TTFT.

generation:

19.7 tokens/s

prompt processing (prefill):

223.8 tokens/s

quant:

Q4_K_M (GGUF)

User reports Qwen3 8B at 19.7 t/s generation and 223.8 t/s prompt processing on a Mac mini M4 with 32 GB unified memory. Setup is llama.cpp b10929 with the Q4_K_M GGUF and Metal, all layers on the GPU, four CPU threads. The figures are separate llama-bench tests (pp512 and tg128), each the mean of three repetitions at zero initial depth, not one 512-input/128-output request; at 8192 prefill depth generation is 16.1 t/s.

generation:

36.2 tokens/s

prompt processing (prefill):

416.6 tokens/s

quant:

Q4_K_M (GGUF)

User reports Qwen3 4B at 36.2 t/s generation and 416.6 t/s prompt processing on a Mac mini M4 with 32 GB unified memory. Setup is llama.cpp b10929 with the Q4_K_M GGUF and Metal, all layers on the 10-core integrated GPU. The figures are separate llama-bench tests, pp512 and tg128, each the mean of three repetitions at zero initial depth, not one 512-input/128-output request.

generation:

80.3 tokens/s

quant:

4-bit (MLX)

User reports Qwen3-14B at 80.3 t/s on a Mac Studio M5 Ultra with 256 GB unified memory. Setup is LM Studio with 4-bit MLX weights on one machine, 36 CPU cores and 80 GPU cores. The 80.3 t/s is the mean of three runs; TTFT is 3.53 seconds, and the displayed 2,754 prompt tokens/s is prompt-token count divided by TTFT, not native prefill. Configured context capacity is unknown.

generation:

12.7 tokens/s

prompt processing (prefill):

116.0 tokens/s

quant:

4-bit (MLX)

mtp (multi-token prediction):

off

User reports Qwen3-14B-4bit at 12.7 t/s generation on one Apple Mac mini M4 with 16 GB unified memory. Setup is mlx-lm 0.29.1 native mlx_lm generate with 4-bit MLX safetensors, --temp 0 and --max-tokens 128, no speculative decoding or prompt cache. The separate long run reports 116 t/s prompt processing at 3,796 input tokens; the two short runs both measured 12.7 t/s, and peak memory was 8.4GB.