Local model performance on your hardware

generation:

18.0 tokens/s

User reports Qwen3-14B writing at 18 tok/s on a MacBook Pro M2 Pro with 16GB, where the model fits entirely in unified memory. Setup is llama.cpp with the Metal build, run through the MAFDET app, which reasons about the message while the user is still typing. The app cuts the wait from Enter to the first word from 32.7 s to 12.1 s on 6 word problems, with 6/6 accuracy both ways; on Qwen3-4B the mean/median wait drops from 18.1s/16.7s to 5.6s/4.1s over 24 problems. The same 14B on the user's RTX 2060 is split with CPU at 4.5-5 tok/s. Tested on one Mac only.

generation:

130.6 tokens/s

quant:

Q4_K_M (GGUF)

kv:

Q8

User compares an RTX 5080 desktop against a rented DGX Spark on Qwen3-8B Q4_K_M, where the PC decodes at 130.55 t/s versus 42.43 t/s on Spark at 512 input tokens, roughly three times faster. Both ran llama.cpp release b11889 with Q8 KV cache, Flash Attention and a 40,960-token context; the PC kept the whole model on its 16 GB card. Decode falls to 79.09 t/s on the PC and 25.64 t/s on Spark at 32,768 input tokens. With Qwen3-32B the PC needed CPU offload (40 GPU layers) and managed 4.92 t/s against Spark's 10.50 t/s. Aggregate batched throughput at 8 concurrent requests was 471.75 t/s on the PC and 209.02 t/s on Spark.

generation:

14.0 tokens/s

quant:

Q4_0

User reports Qwen3-4B-Thinking running at about 14 tok/s on the NPU of a Surface Laptop 7 (Snapdragon X Plus, 16 GB), and LFM2.5-8B-A1B at about 31 tok/s. Setup is the GenieX engine with Q4_0 weights, run fully on the NPU; time to first token was up to about 18x faster than CPU. In a GenieX vs Foundry Local express comparison, Foundry was faster on every comparable model (up to ~1.9x on 0.5B and 1.5B), and Phi-4-mini on Foundry Local scored 80/100 at 27.8 tok/s. Of 16 models fully tested, 10 ran on the NPU alone, 4 on NPU+CPU, 0 on CPU alone and 2 were undetermined.

generation:

133.0 tokens/s

quant:

Q8_0 (GGUF)

User benchmarks Qwen3-8B Q8_0 with DSpark speculative decoding in llama.cpp on an RX 7900 GRE 16GB, reaching 133 t/s on ROCm versus 59 t/s without speculation, and 115 t/s versus 62 t/s on Vulkan. Setup is llama.cpp with Q8_0 weights, greedy decoding, one user and 32 prompts per setting. DFlash gave 1.5-1.9x, a 0.6B draft model about 1.3x, and n-gram lookup did nothing. The best number of guessed tokens is 3 on Vulkan and 7 on ROCm, and prompt processing is about 8% slower with DSpark loaded. Only 5-6 of 16 answers matched normal decoding word for word.

generation:

297.4 tokens/s

prompt processing (prefill):

4993.0 tokens/s

quant:

Q4 (GGUF)

User reports 297.4 t/s generation and 4993 t/s prompt processing. Source evidence (verbatim): Source: https://arxiv.org/html/2607.00501v1 # Original M4 Pro 24GB Qwen3 0.6B four-bit llama.cpp baseline measurements Original Base Compute research paper, July 1, 2026, section 4 and Tables 1–2. Selected run is the authors' llama.cpp baseline, not BaseRT. Hardware: Apple M4 Pro 24GB unified memory, 16 GPU cores. Chassis, CPU core count and macOS version unreported; do not infer Mac mini. Model Qwen3-0.6B, GGUF weights, generic Q4 / 4-bit label only. Exact quantization recipe, tested artifact repository and revision unknown. Verified upstream model reference only: https://huggingface.co/Qwen/Qwen3-0.6B ; this safetensors repository is not the tested GGUF. Runtime llama.cpp Metal, source-labelled b9630 (June 2026), default settings. Full commit and command unreported. Generation 297.4 tokens/s over 128 autoregressive generated tokens (TG128). Prefill 4993 tokens/s at 512 prompt tokens (PP512). These are separate synthetic tests, not a combined 512-input/128-output request. Cache depth, KV precision, repetition count, warmup, concurrency and speculation unreported. Resident usage and weight-file size unreported. Standard deviations are omitted by the authors, who say they are generally below 3% for decode and 10% for prefill; these are not per-row error bars. Other runtime, prompt-length and quantization rows are distinct configurations, not averaged here.

generation:

14.0 tokens/s

quant:

Q4_K_S (GGUF)

User compares DanilAI and LM Studio running Qwen3-30B-A3B-Instruct-2507 on an RTX 4050 Laptop (6 GB VRAM) with a Ryzen 7 7435HS and 24 GB RAM. LM Studio with Q4_K_S quantization reached 14.03 tokens/sec; DanilAI with hybrid FP16/8-bit/4-bit precision plus DanilDrive compression reached 25.0 tokens/sec. A separate test ran Qwen3-0.6B in FP16 with DanilDrive, reporting approximately 45% lossless compression. OBS Studio was recording throughout and may have affected performance.

generation:

89.0 tokens/s

prompt processing (prefill):

6700.0 tokens/s

User reports Qwen3 8B at 89 t/s decode and 6.7k tok/s prefill on an RTX 5090 using PulsarLM, a new inference engine that evicts low-attention token pages to keep the context window bounded. Setup is PulsarLM with custom attention kernels that accumulate attention mass per token page and remove low-mass pages at intervals; RoPE embeddings are re-rotated to keep positions contiguous. Context snapshots avoid re-prefills at the cost of extra RAM. Batched decode reaches 1.2k tok/s at 16 concurrent requests. The engine is early-stage, tested only on Linux x86_64, and supports only smaller/older models; the author removed a BM25+attention recall mechanism and recommends pairing with external RAG.

generation:

68.1 tokens/s

prompt processing (prefill):

3057.0 tokens/s

quant:

Q4_K_M (GGUF)

User reports Qwen3-14B at 68.1 t/s generation and 3057 t/s prompt processing on a 2x RTX 2080 Ti node. Setup is llama.cpp with Q4_K_M quant, 32768 context, and a Qwen3-0.6B-Q8_0 draft model for speculative decoding with depth 6. The user also reports Pixtral-12B at 77.0 t/s generation and 2862 t/s prompt processing, and a distributed 4-stream aggregate of 118.68 t/s. They ask if Linux would give better t/s and mention P2P is working on Windows.

generation:

72.3 tokens/s

prompt processing (prefill):

178.0 tokens/s

quant:

UD-Q4_K_XL (GGUF)

User reports Qwen3 30B-A3B at 72.31 t/s generation and 178.03 t/s prompt processing on a pre-release Framework Desktop with an AMD Ryzen AI Max+ 395 / Radeon 8060S gfx1151. Setup is llama.cpp build 6adc3c3e (5684) with the Unsloth UD 2.0 Q4_K_XL GGUF over Vulkan, 99 GPU layers requested and batch 256. The 16.49 GiB model size is not resident RAM; context depth, KV quantization, speculative decoding and concurrency are not reported.

generation:

6.4 tokens/s

prompt processing (prefill):

84.9 tokens/s

quant:

Q8_0 (GGUF)

flash attention:

on

User reports Qwen3-32B-Q8_0 at 6.4 t/s generation and 84.9 t/s prompt processing on an AMD Strix Halo 128GB system. Setup is llama.cpp with the Vulkan (RADV) backend, FA 1 and ngl 99, on a single AMD Radeon Graphics (RADV GFX1151) device. The figures come from separate llama-bench phases: PP512 at 84.87 +/- 0.35 t/s and TG128 at 6.43 +/- 0.01 t/s. Context depth, thread count and KV cache type are unreported.

generation:

6.4 tokens/s

prompt processing (prefill):

51.1 tokens/s

quant:

Q8_0 (GGUF)

flash attention:

on

User reports Qwen3-32B at 6.4 t/s generation and 51.1 t/s prompt processing on an AMD Strix Halo 128GB system. Setup is llama.cpp with Q8_0 and flash attention enabled, ngl 99, on one ROCm device (gfx1151). The figures are separate llama-bench phases: PP512 at 51.12 +/- 0.24 t/s and TG128 at 6.39 +/- 0.01 t/s, not a same-request pair. Context depth is not printed.

generation:

6.4 tokens/s

prompt processing (prefill):

77.4 tokens/s

quant:

Q8_0 (GGUF)

flash attention:

on

User reports Qwen3-32B at 6.4 t/s generation and 77.4 t/s prompt processing on a Ryzen AI Max+ 395 / Radeon 8060S under Linux. Setup is llama.cpp with ROCm, Q8_0, flash attention enabled, ngl 99, and one local gfx1151 device. The stored figures are rounded from native llama-bench results of 6.43 t/s TG128 and 77.43 t/s PP512; a second run at 8192 prompt tokens gave 75.68 t/s PP8192 and 6.42 t/s TG128.

generation:

69.3 tokens/s

prompt processing (prefill):

163.8 tokens/s

quant:

UD-Q4_K_XL (GGUF)

flash attention:

on

User reports Qwen3-30B-A3B at 69.3 t/s generation and 163.8 t/s prompt processing on an AMD Ryzen AI Max+ 395 / Radeon 8060S under Linux. Setup is llama.cpp Vulkan with UD-Q4_K_XL, ngl 99, batch size 128 and flash attention enabled, on one local GPU. The two figures are separate llama-bench phases, PP512 and TG128, not a same-request pair; context depth is unreported.

generation:

40.0 tokens/s

prompt processing (prefill):

586.0 tokens/s

quant:

MXFP4 (GGUF)

flash attention:

on

User reports Qwen3-Coder-Next-80B at 40 t/s generation and 586 t/s prompt processing on a single Radeon 8060S iGPU in an AMD Strix Halo system with 128 GB unified memory. Setup is llama.cpp with the MXFP4 quant, full offload, batch 2048 and flash attention, using the RADV Vulkan backend. The figures are separate 512-token prompt and 128-token generation microbenchmarks, not one end-to-end request; the source notes post-hoc MXFP4 can underperform conventional quants.

generation:

92.0 tokens/s

prompt processing (prefill):

1143.0 tokens/s

quant:

UD-Q4_K_XL (GGUF)

flash attention:

on

User reports Qwen3-30B-Instruct-2507 at 92 t/s generation and 1143 t/s prompt processing on an AMD Strix Halo 128GB system with a single Radeon 8060S iGPU. Setup is llama.cpp via RADV with UD-Q4_K_XL, full offload (ngl 999), batch 2048 and flash attention. The figures are separate llama-bench pp512 and tg128 microbenchmarks, not one end-to-end request; the AMDVLK run measured 936.0 t/s prompt and 93.0 t/s generation.

generation:

73.7 tokens/s

prompt processing (prefill):

843.3 tokens/s

quant:

Q4_K_M (GGUF)

flash attention:

on

mtp (multi-token prediction):

off

User reports Qwen3-30B-A3B at 73.7 t/s generation and 843.3 t/s prompt processing on an AMD Strix Halo 128GB box. Setup is llama.cpp with a Q4_K_M GGUF, full GPU offload (ngl 999) and flash attention on ROCm/HIP 7.2.4, one ROCm device. The headline is the d0 arm, PP843.25 ±5.39 and TG73.74 ±0.17 tok/s; at depth 4096 it drops to PP209.17 and TG47.48 tok/s, where the depth is pre-existing KV state rather than a timed prefill.

generation:

44.6 tokens/s

prompt processing (prefill):

852.6 tokens/s

quant:

Q8_0 (GGUF)

kv:

F16

flash attention:

on

mtp (multi-token prediction):

off

User reports Qwen3-4B Q8_0 at 44.6 t/s generation and 852.6 t/s prompt processing on an AMD Ryzen AI Max+ 395 with Radeon 8060S. Setup is llama.cpp with Q8_0 and F16 K/V cache, one GPU, 16 threads, flash attention, ngl=99, 4096-token prompt and 512-token generation tests. The figures are separate synthetic llama-bench runs, not one chat: PP4096 852.58 ± 15.72 t/s and TG512 44.57 ± 0.38 t/s over five repetitions, with no speculative decoding or concurrency.

generation:

94.5 tokens/s

prompt processing (prefill):

806.0 tokens/s

quant:

Q4_K_M (GGUF)

kv:

F16

User reports Qwen3-30B-A3B-Instruct-2507 at 94.5 t/s generation and 806 t/s prompt processing on an M4 Pro with 64 GB unified memory. Setup is dasLLAMA version 16 with Q4_K_M weights, f16 KV cache, and one Metal GPU, with 640 positions allocated. The figures are native PP512 and TG128 means over five repetitions, not a single 512-input/128-output request.

generation:

77.2 tokens/s

prompt processing (prefill):

780.2 tokens/s

quant:

Q4_K_M (GGUF)

kv:

F16

User reports Qwen3-30B-A3B-Instruct-2507 at 77.2 t/s generation and 780.2 t/s prompt processing on an M4 Pro 64GB. Setup is llama.cpp commit 6fdd0ac89 with Q4_K_M, Metal backend, -ngl 99, and 10 CPU threads on one physical GPU. The figures are native llama-bench PP512 and TG128 means over five repetitions at depth 0, not a single chat request.

generation:

81.2 tokens/s

prompt processing (prefill):

518.0 tokens/s

quant:

4-bit (MLX)

mtp (multi-token prediction):

off

User reports Qwen3 8B 4-bit at 81.2 t/s generation and 518 t/s prompt processing on an M1 Ultra 128GB Mac Studio. Setup is mlxcel 0.7.0-beta.1 with FP16 KV cache, one Metal GPU, 512 input tokens and 128 output tokens, greedy temperature 0. The decode figure covers 128 output tokens over 127 forward steps; prefill excludes loading, tokenization and transport, so it is not end-to-end TTFT.