Local model performance on your hardware

generation:

74.3 tokens/s

prompt processing (prefill):

661.9 tokens/s

quant:

Q4_0 (GGUF)

User reports Llama 3.2 3B at 74.3 t/s generation and 661.9 t/s prompt processing on a Minisforum M1-S1 MAX with Ryzen AI Max+ 395 running Ubuntu 24.04.3. Setup is llama-bench on ROCm with the Q4_0 GGUF, one Radeon 8060S Graphics GPU (gfx1151) and -ngl 999. The headline pairs separate PP16 and TG64 phases; native values were 661.87 +/- 5.54 t/s and 74.31 +/- 0.05 t/s, with TG128 at 74.13 +/- 0.09 t/s.

generation:

28.0 tokens/s

prompt processing (prefill):

730.0 tokens/s

quant:

BF16

User reports Llama 3.2 3B at 28 t/s generation and 730 t/s prompt processing on an AMD Strix Halo 128GB machine. Setup is llama.cpp with Vulkan on the Radeon 8060S iGPU, BF16 quant, 6 threads and maximum GPU offload, run with llama-bench -p 512 -n 128. The 28 t/s and 730 t/s are separate synthetic phase tests, not an end-to-end rate; context depth, KV precision and batch settings are unreported.

generation:

93.0 tokens/s

prompt processing (prefill):

1912.0 tokens/s

quant:

Q4_K_XL

User reports Llama 3.2 3B at 93 t/s generation and 1912 t/s prompt processing on an AMD Strix Halo 128GB machine. Setup is llama.cpp with Vulkan on the Radeon 8060S iGPU, Q4_K_XL quant, 6 threads, and -ngl 999 for maximum GPU offload. The figures come from llama-bench with -p 512 -n 128, so prompt and generation are separate synthetic phase tests, not an end-to-end rate; rates are rounded and no raw timings, repetition count, or context depth are reported.

generation:

69.0 tokens/s

prompt processing (prefill):

1538.2 tokens/s

quant:

Q6_K_L (GGUF)

kv:

F16

flash attention:

on

mtp (multi-token prediction):

off

User reports Llama 3.2 3B at 69 t/s generation and 1538.2 t/s prompt processing on an AMD Strix Halo 128GB machine. Setup is llama.cpp with the Q6_K_L GGUF, f16 KV cache, flash attention on, and 999 GPU layers offloaded to the single Radeon 8060S iGPU. The figures are separate synthetic llama-bench PP512 and TG128 runs at n_depth=0, not generation after a 512-token prompt and not HTTP end-to-end throughput; no long-context rate is implied.

generation:

46.0 tokens/s

prompt processing (prefill):

530.0 tokens/s

quant:

Q4_K_M (GGUF)

kv:

F16

flash attention:

off

mtp (multi-token prediction):

off

User reports Llama 3.2 3B at 46 t/s generation and 530 t/s prompt processing on an M4 24GB Mac mini. Setup is llama.cpp llama-bench build 9100 with Q4_K_M, Metal backend, full GPU offload, F16 K/V cache, and four CPU threads. The figures are separate synthetic tests: PP512 at 530.01 +/- 0.86 t/s and TG128 at 45.95 +/- 0.02 t/s, three repetitions each, not a paired prompt-and-answer run.

generation:

85.1 tokens/s

prompt processing (prefill):

1917.9 tokens/s

quant:

Q4_K_M (GGUF)

flash attention:

on

mtp (multi-token prediction):

off

User reports Llama 3.2 3B at 85.1 t/s generation and 1917.9 t/s prompt processing on an AMD Strix Halo 128GB machine. Setup is llama.cpp with the Q4_K_M GGUF, ROCm/HIP 7.2.4, -ngl 999 and flash attention, on one enumerated gfx1151 device. The figures are llama-bench PP512 and TG128 means, not client-visible tokens; the source also lists Ollama prompt-eval 880.53 and eval 82.08, which the user excludes from the native metrics.

generation:

199.9 tokens/s

prompt processing (prefill):

6122.5 tokens/s

quant:

Q4_K_M (GGUF)

flash attention:

on

User reports Llama 3.2 3B at 199.9 t/s generation and 6122.5 t/s prompt processing on an M5 Max with 48GB unified memory. Setup is llama.cpp build 11146 with Metal, Q4_K_M weights, n_gpu_layers=99 and flash attention enabled. The figures are separate llama-bench microtests, tg128 and pp512, not one 512-input/128-output request or server concurrency throughput; context length is unspecified.

generation:

439.6 tokens/s

prompt processing (prefill):

16967.9 tokens/s

quant:

Q4_K_M (GGUF)

flash attention:

on

User reports Llama 3.2 1B at 439.6 t/s generation and 16967.9 t/s prompt processing on an M5 Max with 48GB unified memory. Setup is llama.cpp build 11146 with Metal, Q4_K_M weights, n_gpu_layers=99 and flash attention enabled. The figures are separate llama-bench microtests, tg128 and pp512, not one 512-input/128-output request or server concurrency throughput; context length is unspecified.

generation:

237.9 tokens/s

prompt processing (prefill):

7307.2 tokens/s

quant:

4-bit (MLX)

User reports Llama 3.2 3B Instruct 4-bit at 237.9 t/s generation and 7307.2 t/s prompt processing on an M5 Max with 48GB unified RAM. Setup is MLX via mlx-lm 0.31.3 with 4-bit Safetensors weights, batch size 1, unquantized KV cache and no speculative decoding. The figures are arithmetic means of five trials at prompt 512 and generation 128; the benchmark is synthetic token throughput, not end-to-end request throughput.

generation:

527.7 tokens/s

prompt processing (prefill):

17477.0 tokens/s

quant:

4-bit (MLX)

User reports Llama 3.2 1B Instruct 4-bit at 527.7 t/s generation and 17477 t/s prompt processing on an M5 Max 48GB. Setup is MLX with 4-bit weights and unquantized KV cache, batch size 1, one warmup plus five measured trials. The headline uses pp512 and tg128, not a best-of sweep; the source also lists pp128 7439.85, pp256 13408.64 and tg256 542.87 t/s.

generation:

298.6 tokens/s

prompt processing (prefill):

19397.4 tokens/s

quant:

Q4_K_M (GGUF)

User reports Llama 3.2 3B at 298.6 t/s generation and 19397.4 t/s prompt processing on an NVIDIA RTX 5080. Setup is Ollama with Q4_K_M at a configured 16384-token context, with an actual recorded prompt of 1657 tokens rather than a full 16384-token prefill. The figures are medians of 3 fresh-prefill runs, derived from Ollama eval_count/eval_duration rather than TTFT division, with temperature 0 and a unique nonce per repetition to avoid prompt-cache hits.

generation:

46.7 tokens/s

quant:

Q4_K_M (GGUF)

User reports Llama 3.2 3B at 46.7 t/s on an M4 16GB Mac mini running macOS 15.3.1. Setup is Ollama 0.31.2 with llama.cpp and Metal, Q4_K_M, default settings, fresh runtime launch. The figure is the median of three requests after one discarded warm-up, with a source-reported max-minus-min spread of 0.1 t/s; configured context, KV precision and speculative decoding are unknown.

generation:

14.1 tokens/s

prompt processing (prefill):

21.2 tokens/s

quant:

Q3_K_S (GGUF)

User reports Llama 3.2 3B at 14.1 t/s generation and 21.2 t/s prefill on an Xbox Series S, CPU-only with six inference threads. Setup is llama.cpp in xllama 1.6.0.1163 with Q3_K_S and a configured context of 2048, one sequential request per fresh app process. The figures are medians of three recorded runs, with generation ranging 14.10-14.12 t/s; peak working set is 1827 MB per run.

generation:

45.4 tokens/s

quant:

Q4_K_M (GGUF)

User reports Llama 3.2 3B at 45.4 t/s generation on a Mac mini M4 16GB. Setup is Ollama 0.30.8 at Q4_K_M on Metal, one request, speculative decoding off. Mean of two runs by ai-local-lab, measured 2026-06-16. A source-reported measurement, not independently reproduced.

generation:

6.0 tokens/s

quant:

Q4_K_M (GGUF)

kv:

q8_0

User reports a 3B roleplay finetune running fully on an iPhone at around 6 tok/s on an A18 chip. Setup is llama.cpp with Metal, Impish Llama 3B (a Llama 3.2 3B RP finetune) with a rank 16 LoRA fused and requantized to Q4_K_M, KV cache at q8_0, all layers offloaded, about 2.0 GB on disk. The user notes the speed is from an early measurement and has not been re-measured on the current build. Context is 4096 on 4 GB phones, 6144 on 6 GB phones, and 8192 on 8 GB phones. The user reports that scaling the history trim budget to about 0.68 x n_ctx raised long-conversation fact recall from 25% to 75%, that BM25 retrieval re-injected superseded facts (+20.8pp) until filtered by rare entities (-18.8pp, CI [-35.4, -6.2]), and that scene directions placed as their own block after the final user turn landed 16/16 versus 0/16 in the reminder slot.

generation:

14.6 tokens/s

prompt processing (prefill):

108.1 tokens/s

quant:

Q4_K_M (GGUF)

User benchmarks Llama 3.2 1B at 14.62 tok/s generation and 108.11 tok/s prompt processing on an Orange Pi 5 Plus (RK3588, 16GB LPDDR4x) using Ollama with 4 threads pinned to the Cortex-A76 big cores. Setup is Ollama with Q4_K_M GGUF weights, CPU-only inference; the same board also ran DeepSeek-Coder 1.3B at 16.90 tok/s, Qwen 2.5 1.5B at 14.48 tok/s, Llama 3.2 3B at 7.28 tok/s, Phi-3 Mini 3.8B at 6.56 tok/s and Llama 3.1 8B at 2.32 tok/s, all with 4 threads. Core pinning to 4 big cores gave up to +318% over the default 8 threads; the 8B model hit 85.0°C uncooled and the NPU path (RKLLM runtime) reached 21.55 tok/s on Qwen 1.5 0.5B with ~0% CPU load.

generation:

6.1 tokens/s

prompt processing (prefill):

8.8 tokens/s

quant:

q4f16_1 (MLC)

User benchmarks mlc-llm on an Orange Pi 5 Pro running Ubuntu off an m.2 NVMe SSD, with mlc-llm compiled for OpenCL. The table reports prefill and decode tok/s for seven models; the record captures Llama-3.2-3B-Instruct-q4f16_1-MLC at 8.8 prefill tok/s and 6.1 decode tok/s. Other rows: SmolLM-1.7B-Instruct 16.6/10.1, Qwen2.5-3B-Instruct 8.5/5.1, Phi-3.5-mini-instruct 5.5/4.8, SmolLM-360M-Instruct 44.4/25.7, SmolLM-1.7B-Instruct 18/9.8, TinyLlama-1.1B-Chat-v1.0 25.9/13.9.

generation:

15.4 tokens/s

prompt processing (prefill):

42.1 tokens/s

quant:

Q8_0 (GGUF)

User reports Llama 3.2 1B Instruct at 15.4 tokens/s on a Raspberry Pi 5 (8GB). Setup is llama.cpp compiled natively for Cortex-A76 with Q8_0 quantization, running CPU-only on 4 cores. A Q4_K_M build reached 17.8 tokens/s, prompt eval 42.1 tokens/s, and time-to-first-token 0.38 s, compared with 0.85 tokens/s and 18.97 s TTFT for a generic ARM64 build.

User reports FastDMS, a DMS KV-cache compression implementation, decoding 1.5-2x faster than vLLM BF16/FP8 with 5-8x less KV memory. Benchmarks cover Llama-3.2-1B and Qwen3-8B DMS checkpoints, with KLD and token match comparable or better than vLLM's FP8/TurboQuant. Training took about 20 min on an RTX Pro 6000 Blackwell.