Local model performance on your hardware

generation:

28.0 tokens/s

prompt processing (prefill):

730.0 tokens/s

quant:

BF16

User reports Llama 3.2 3B at 28 t/s generation and 730 t/s prompt processing on an AMD Strix Halo 128GB machine. Setup is llama.cpp with Vulkan on the Radeon 8060S iGPU, BF16 quant, 6 threads and maximum GPU offload, run with llama-bench -p 512 -n 128. The 28 t/s and 730 t/s are separate synthetic phase tests, not an end-to-end rate; context depth, KV precision and batch settings are unreported.

generation:

93.0 tokens/s

prompt processing (prefill):

1912.0 tokens/s

quant:

Q4_K_XL

User reports Llama 3.2 3B at 93 t/s generation and 1912 t/s prompt processing on an AMD Strix Halo 128GB machine. Setup is llama.cpp with Vulkan on the Radeon 8060S iGPU, Q4_K_XL quant, 6 threads, and -ngl 999 for maximum GPU offload. The figures come from llama-bench with -p 512 -n 128, so prompt and generation are separate synthetic phase tests, not an end-to-end rate; rates are rounded and no raw timings, repetition count, or context depth are reported.

generation:

69.0 tokens/s

prompt processing (prefill):

1538.2 tokens/s

quant:

Q6_K_L (GGUF)

kv:

F16

flash attention:

on

mtp (multi-token prediction):

off

User reports Llama 3.2 3B at 69 t/s generation and 1538.2 t/s prompt processing on an AMD Strix Halo 128GB machine. Setup is llama.cpp with the Q6_K_L GGUF, f16 KV cache, flash attention on, and 999 GPU layers offloaded to the single Radeon 8060S iGPU. The figures are separate synthetic llama-bench PP512 and TG128 runs at n_depth=0, not generation after a 512-token prompt and not HTTP end-to-end throughput; no long-context rate is implied.

generation:

46.0 tokens/s

prompt processing (prefill):

530.0 tokens/s

quant:

Q4_K_M (GGUF)

kv:

F16

flash attention:

off

mtp (multi-token prediction):

off

User reports Llama 3.2 3B at 46 t/s generation and 530 t/s prompt processing on an M4 24GB Mac mini. Setup is llama.cpp llama-bench build 9100 with Q4_K_M, Metal backend, full GPU offload, F16 K/V cache, and four CPU threads. The figures are separate synthetic tests: PP512 at 530.01 +/- 0.86 t/s and TG128 at 45.95 +/- 0.02 t/s, three repetitions each, not a paired prompt-and-answer run.

generation:

85.1 tokens/s

prompt processing (prefill):

1917.9 tokens/s

quant:

Q4_K_M (GGUF)

flash attention:

on

mtp (multi-token prediction):

off

User reports Llama 3.2 3B at 85.1 t/s generation and 1917.9 t/s prompt processing on an AMD Strix Halo 128GB machine. Setup is llama.cpp with the Q4_K_M GGUF, ROCm/HIP 7.2.4, -ngl 999 and flash attention, on one enumerated gfx1151 device. The figures are llama-bench PP512 and TG128 means, not client-visible tokens; the source also lists Ollama prompt-eval 880.53 and eval 82.08, which the user excludes from the native metrics.

generation:

199.9 tokens/s

prompt processing (prefill):

6122.5 tokens/s

quant:

Q4_K_M (GGUF)

flash attention:

on

User reports Llama 3.2 3B at 199.9 t/s generation and 6122.5 t/s prompt processing on an M5 Max with 48GB unified memory. Setup is llama.cpp build 11146 with Metal, Q4_K_M weights, n_gpu_layers=99 and flash attention enabled. The figures are separate llama-bench microtests, tg128 and pp512, not one 512-input/128-output request or server concurrency throughput; context length is unspecified.

generation:

439.6 tokens/s

prompt processing (prefill):

16967.9 tokens/s

quant:

Q4_K_M (GGUF)

flash attention:

on

User reports Llama 3.2 1B at 439.6 t/s generation and 16967.9 t/s prompt processing on an M5 Max with 48GB unified memory. Setup is llama.cpp build 11146 with Metal, Q4_K_M weights, n_gpu_layers=99 and flash attention enabled. The figures are separate llama-bench microtests, tg128 and pp512, not one 512-input/128-output request or server concurrency throughput; context length is unspecified.

generation:

237.9 tokens/s

prompt processing (prefill):

7307.2 tokens/s

quant:

4-bit (MLX)

User reports Llama 3.2 3B Instruct 4-bit at 237.9 t/s generation and 7307.2 t/s prompt processing on an M5 Max with 48GB unified RAM. Setup is MLX via mlx-lm 0.31.3 with 4-bit Safetensors weights, batch size 1, unquantized KV cache and no speculative decoding. The figures are arithmetic means of five trials at prompt 512 and generation 128; the benchmark is synthetic token throughput, not end-to-end request throughput.

generation:

527.7 tokens/s

prompt processing (prefill):

17477.0 tokens/s

quant:

4-bit (MLX)

User reports Llama 3.2 1B Instruct 4-bit at 527.7 t/s generation and 17477 t/s prompt processing on an M5 Max 48GB. Setup is MLX with 4-bit weights and unquantized KV cache, batch size 1, one warmup plus five measured trials. The headline uses pp512 and tg128, not a best-of sweep; the source also lists pp128 7439.85, pp256 13408.64 and tg256 542.87 t/s.

generation:

298.6 tokens/s

prompt processing (prefill):

19397.4 tokens/s

quant:

Q4_K_M (GGUF)

User reports Llama 3.2 3B at 298.6 t/s generation and 19397.4 t/s prompt processing on an NVIDIA RTX 5080. Setup is Ollama with Q4_K_M at a configured 16384-token context, with an actual recorded prompt of 1657 tokens rather than a full 16384-token prefill. The figures are medians of 3 fresh-prefill runs, derived from Ollama eval_count/eval_duration rather than TTFT division, with temperature 0 and a unique nonce per repetition to avoid prompt-cache hits.

generation:

46.7 tokens/s

quant:

Q4_K_M (GGUF)

User reports Llama 3.2 3B at 46.7 t/s on an M4 16GB Mac mini running macOS 15.3.1. Setup is Ollama 0.31.2 with llama.cpp and Metal, Q4_K_M, default settings, fresh runtime launch. The figure is the median of three requests after one discarded warm-up, with a source-reported max-minus-min spread of 0.1 t/s; configured context, KV precision and speculative decoding are unknown.

generation:

45.4 tokens/s

quant:

Q4_K_M (GGUF)

User reports Llama 3.2 3B at 45.4 t/s generation on a Mac mini M4 16GB. Setup is Ollama 0.30.8 at Q4_K_M on Metal, one request, speculative decoding off. Mean of two runs by ai-local-lab, measured 2026-06-16. A source-reported measurement, not independently reproduced.