Local model performance on your hardware

generation:

44.1 tokens/s

prompt processing (prefill):

892.5 tokens/s

quant:

Q4_K_M (GGUF)

flash attention:

off

User reports Llama 3.1 8B at 44.1 t/s generation and 892.5 t/s prompt processing on a Minisforum MS-S1 MAX with an AMD Ryzen AI Max+ 395 and integrated Radeon 8060S. Setup is llama.cpp build 7642 with the Q4_K_M GGUF, flash attention off, integer dot product disabled, 99 layers offloaded to one Vulkan0 device, and 64G of system RAM allocated to the GPU. The figures are separate llama-bench PP512 and TG128 phases, not one 512-input/128-output request, with 4.58 GiB model size and f16 K/V cache at prepared depth 0.

generation:

14.9 tokens/s

quant:

4-bit (MLX)

mtp (multi-token prediction):

off

User reports Llama 3.1 70B at 14.9 t/s on a Mac Studio with an M3 Ultra and 512 GB of unified memory. Setup is MLX with 4-bit quantization, group_size 64, no speculative decoding or batching, at a 4k prompt (4138 actual tokens) with 128 output tokens. The figure is one observation, not a repeated-run mean, from a single timed generation.

generation:

72.9 tokens/s

prompt processing (prefill):

566.4 tokens/s

quant:

4-bit (source config) (MLX)

User reports Llama 3.1 8B at 72.9 t/s generation and 566.4 t/s prompt processing on a single Apple M2 Max with 64 GB unified memory. Setup is MLX via a direct python -m mlx_lm benchmark, 4-bit quantized, batch size 1, with a requested 512-token prompt and 128-token generation cap. The figures are three-trial means after discarding Trial 1; peak memory was 5.116 in engine output units, and KV cache precision and prefill chunk size are unreported.

generation:

101.1 tokens/s

prompt processing (prefill):

2758.3 tokens/s

quant:

Q4_K_M (GGUF)

flash attention:

on

User reports Llama 3.1 8B at 101.1 t/s generation and 2758.3 t/s prompt processing on an M5 Max with 48GB unified memory. Setup is llama.cpp build 11146 with Metal, Q4_K_M weights, n_gpu_layers=99 and flash attention enabled. The figures are separate llama-bench microtests, tg128 and pp512, not one 512-input/128-output request; context length is unspecified and KV cache precision was not captured.

generation:

115.4 tokens/s

prompt processing (prefill):

3225.7 tokens/s

quant:

4-bit (MLX)

User reports Llama 3.1 8B at 115.4 t/s generation and 3225.7 t/s prompt processing on an M5 Max with 48GB unified RAM. Setup is MLX with 4-bit weights and unquantized KV cache, batch size 1, one model per process. The headline uses pp512 and tg128 from a five-trial mean, not a best-of sweep; the source also records pp128=1932.3, pp256=2487.89 and tg256=115.52 t/s.

generation:

21.0 tokens/s

prompt processing (prefill):

229.2 tokens/s

quant:

Q4_K_M (GGUF)

User reports Llama 3.1 8B at 21 t/s generation and 229.2 t/s prompt processing on an M4 16GB Mac mini. Setup is llama.cpp 0.4.0 with Q4_K_M and Metal, all layers on GPU; context, KV precision and Flash Attention are unknown. The figures come from separate llama-bench tests, pp512 at 229.23 ±0.08 t/s and tg128 at 20.95 ±0.02 t/s over three runs, not a combined 512-input/128-output completion, with 10.29 GiB RAM already in use.

generation:

8.2 tokens/s

prompt processing (prefill):

9.1 tokens/s

quant:

EVR-1 custom compression (~3-bit weights) (GGUF)

User reports Llama 3.1 8B at 8.2 t/s generation and 9.1 t/s prompt processing on an M4 16GB Mac mini. Setup is Evrmind's custom Metal binary with EVR-1 compression at roughly 3 bits per weight and full GPU offload. The figures come from separate pp512 and tg128 phase tests, not a paired 512-input/128-output run; the source table lists 9.11 and 8.16 t/s.

generation:

23.1 tokens/s

quant:

4bit

User benchmarks Llama 3.1 8B on a base M4 Mac, comparing MLX and llama.cpp. Best observed decode is MLX at 23.1 t/s on a quiet machine, with llama.cpp at 11.5 t/s; llama.cpp prefill is 118 t/s with flash-attention and 71 t/s without. Setup uses MLX 4-bit and llama.cpp GGUF Q4_K_M, with a 32K KV cache pre-allocated by llama.cpp. The user notes the memory gap between engines is an artifact of that pre-allocation, with a real weight delta of about 0.3 GB. The user reports decode is bandwidth-bound at roughly 18-22 t/s either way, and that the speed winner flips depending on background load. The MLX checkpoint is lighter at 4.50 bpw versus 4.90 bpw for llama.cpp.

generation:

3.0 tokens/s

quant:

Q4 (MLX)

User benchmarks Llama 3.1 405B at 2.99 t/s on a single M3 Ultra 512GB node. Setup is MLX with Q4 quantisation at 1K context, single node. This is part of a 200+ configuration campaign across 5 models and 9 topologies on a 5-node M3 Ultra cluster; the report also covers Qwen 2.5 32B, Mixtral 8x7B, DeepSeek V3.2 and Kimi K2.5, with TP and PP scaling results.

generation:

2.3 tokens/s

prompt processing (prefill):

39.1 tokens/s

quant:

F16 (GGUF)

User reports Llama 3.1 70B F16 at 2.27 t/s generation and 39.10 t/s prompt processing on a dual AMD Epyc Turin CPU system. Setup is llama.cpp with GGUF F16 weights, 32 threads, --numa distribute, CPU only. User found that token generation is bottlenecked by remote NUMA memory accesses caused by llamafile_sgemm() during prompt processing; running generation-only first raises it to 4.30 t/s. A similar effect is shown for phi-4 14B F16, where disabling GGML_USE_LLAMAFILE raises generation from 10.59 to 17.26 t/s but drops prompt processing from 187.03 to 85.66 t/s.

prompt processing (prefill):

370.1 tokens/s

quant:

Q8_K_R8 (GGUF)

User benchmarks LLaMA-3.1-8B on CPU with ik_llama.cpp and llama.cpp, reporting prompt processing of 370.11 t/s with Q8_K_R8 on a Ryzen 7950X. Setup uses llama-bench for PP-512 and TG-128, with -rtr 1 for ik_llama.cpp; AVX2/Zen4 on Ryzen-7950X and ARM on M2-Max. User reports ik_llama.cpp is up to 7x faster for prompt processing and notes the slowest ik_llama.cpp quant beats the fastest llama.cpp quant on Ryzen-7950X.

generation:

11.6 tokens/s

quant:

1.58-bit ternary

User reports LLaMA 3.1 8B at 11.6 tok/s sustained (11.9 peak) on a Motorola Moto G Stylus 5G with Snapdragon 6 Gen 1 and 8GB RAM, running CPU-only. Setup is a custom pure Rust runtime with 1.58-bit ternary quantization, speculative decoding, and mmap; model size is about 2.05 GB with 2.10 GB active RAM. Single-token decode without speculation is 4.2 tok/s. Qwen 2.5 1.5B reaches 20.3 tok/s at about 480 MB. WikiText-2 perplexity is 6.45 for the 8B and 9.15 for the 1.5B.

generation:

1.5 tokens/s

quant:

Q4_K_M (GGUF)

User reports Llama 3.1 8B at 1.5 t/s on an AMD RX 7800 XT under Ollama, compared with 65 t/s in LM Studio on the same machine. Setup is Ollama 0.4.7 on Windows with the Q4_K_M quant, Adrenalin Edition 24.10.1 drivers. The user asks whether the gap is their setup or an Ollama limitation.

generation:

1.2 tokens/s

quant:

Q4_K_M (GGUF)

User reports Llama 405B Q4_K_M at 1.2 t/s two years ago, and contrasts it with current speeds of 30-100 t/s for newer models including Kimi K2.6, DeepSeek V4 Flash, MiniMax 2.7, Step 3.5 Flash and Qwen3.5-397B. User also mentions running Qwen3.6-36B at 50 t/s for a few hundred dollars.