Llama 3.2 3B at 4-bit generates a median of 89.1 tokens per second on the AMD Strix Halo 128GB, the hardware it is most run on, from 2 plain runs (community reports, as of 25 Oct 2026).
As of 25 Oct 2026, Llama 3.2 3B at 4-bit on the hardware it is most run on, with the median of plain runs (one device, one request, no speculative decoding, the whole model in its memory):
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Llama 3.2 3B at 74.3 t/s generation and 661.9 t/s prompt processing on a Minisforum M1-S1 MAX with Ryzen AI Max+ 395 running Ubuntu 24.04.3.
Setup is llama-bench on ROCm with the Q4_0 GGUF, one Radeon 8060S Graphics GPU (gfx1151) and -ngl 999.
The headline pairs separate PP16 and TG64 phases; native values were 661.87 +/- 5.54 t/s and 74.31 +/- 0.05 t/s, with TG128 at 74.13 +/- 0.09 t/s.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Llama 3.2 3B at 28 t/s generation and 730 t/s prompt processing on an AMD Strix Halo 128GB machine.
Setup is llama.cpp with Vulkan on the Radeon 8060S iGPU, BF16 quant, 6 threads and maximum GPU offload, run with llama-bench -p 512 -n 128.
The 28 t/s and 730 t/s are separate synthetic phase tests, not an end-to-end rate; context depth, KV precision and batch settings are unreported.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Llama 3.2 3B at 93 t/s generation and 1912 t/s prompt processing on an AMD Strix Halo 128GB machine.
Setup is llama.cpp with Vulkan on the Radeon 8060S iGPU, Q4_K_XL quant, 6 threads, and -ngl 999 for maximum GPU offload.
The figures come from llama-bench with -p 512 -n 128, so prompt and generation are separate synthetic phase tests, not an end-to-end rate; rates are rounded and no raw timings, repetition count, or context depth are reported.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Llama 3.2 3B at 69 t/s generation and 1538.2 t/s prompt processing on an AMD Strix Halo 128GB machine.
Setup is llama.cpp with the Q6_K_L GGUF, f16 KV cache, flash attention on, and 999 GPU layers offloaded to the single Radeon 8060S iGPU.
The figures are separate synthetic llama-bench PP512 and TG128 runs at n_depth=0, not generation after a 512-token prompt and not HTTP end-to-end throughput; no long-context rate is implied.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Llama 3.2 3B at 46 t/s generation and 530 t/s prompt processing on an M4 24GB Mac mini.
Setup is llama.cpp llama-bench build 9100 with Q4_K_M, Metal backend, full GPU offload, F16 K/V cache, and four CPU threads.
The figures are separate synthetic tests: PP512 at 530.01 +/- 0.86 t/s and TG128 at 45.95 +/- 0.02 t/s, three repetitions each, not a paired prompt-and-answer run.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Llama 3.2 3B at 85.1 t/s generation and 1917.9 t/s prompt processing on an AMD Strix Halo 128GB machine.
Setup is llama.cpp with the Q4_K_M GGUF, ROCm/HIP 7.2.4, -ngl 999 and flash attention, on one enumerated gfx1151 device.
The figures are llama-bench PP512 and TG128 means, not client-visible tokens; the source also lists Ollama prompt-eval 880.53 and eval 82.08, which the user excludes from the native metrics.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Llama 3.2 3B at 199.9 t/s generation and 6122.5 t/s prompt processing on an M5 Max with 48GB unified memory.
Setup is llama.cpp build 11146 with Metal, Q4_K_M weights, n_gpu_layers=99 and flash attention enabled.
The figures are separate llama-bench microtests, tg128 and pp512, not one 512-input/128-output request or server concurrency throughput; context length is unspecified.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Llama 3.2 1B at 439.6 t/s generation and 16967.9 t/s prompt processing on an M5 Max with 48GB unified memory.
Setup is llama.cpp build 11146 with Metal, Q4_K_M weights, n_gpu_layers=99 and flash attention enabled.
The figures are separate llama-bench microtests, tg128 and pp512, not one 512-input/128-output request or server concurrency throughput; context length is unspecified.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Llama 3.2 3B Instruct 4-bit at 237.9 t/s generation and 7307.2 t/s prompt processing on an M5 Max with 48GB unified RAM.
Setup is MLX via mlx-lm 0.31.3 with 4-bit Safetensors weights, batch size 1, unquantized KV cache and no speculative decoding.
The figures are arithmetic means of five trials at prompt 512 and generation 128; the benchmark is synthetic token throughput, not end-to-end request throughput.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Llama 3.2 1B Instruct 4-bit at 527.7 t/s generation and 17477 t/s prompt processing on an M5 Max 48GB.
Setup is MLX with 4-bit weights and unquantized KV cache, batch size 1, one warmup plus five measured trials.
The headline uses pp512 and tg128, not a best-of sweep; the source also lists pp128 7439.85, pp256 13408.64 and tg256 542.87 t/s.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Llama 3.2 3B at 298.6 t/s generation and 19397.4 t/s prompt processing on an NVIDIA RTX 5080.
Setup is Ollama with Q4_K_M at a configured 16384-token context, with an actual recorded prompt of 1657 tokens rather than a full 16384-token prefill.
The figures are medians of 3 fresh-prefill runs, derived from Ollama eval_count/eval_duration rather than TTFT division, with temperature 0 and a unique nonce per repetition to avoid prompt-cache hits.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Llama 3.2 3B at 46.7 t/s on an M4 16GB Mac mini running macOS 15.3.1.
Setup is Ollama 0.31.2 with llama.cpp and Metal, Q4_K_M, default settings, fresh runtime launch.
The figure is the median of three requests after one discarded warm-up, with a source-reported max-minus-min spread of 0.1 t/s; configured context, KV precision and speculative decoding are unknown.
Xbox Series S, Zen 2 CPU-only (6 threads) · llama.cpp · 2,048 ctx
generation:
14.1 tokens/s
prompt processing (prefill):
21.2 tokens/s
quant:
Q3_K_S (GGUF)
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Llama 3.2 3B at 14.1 t/s generation and 21.2 t/s prefill on an Xbox Series S, CPU-only with six inference threads.
Setup is llama.cpp in xllama 1.6.0.1163 with Q3_K_S and a configured context of 2048, one sequential request per fresh app process.
The figures are medians of three recorded runs, with generation ranging 14.10-14.12 t/s; peak working set is 1827 MB per run.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Llama 3.2 3B at 45.4 t/s generation on a Mac mini M4 16GB.
Setup is Ollama 0.30.8 at Q4_K_M on Metal, one request, speculative decoding off.
Mean of two runs by ai-local-lab, measured 2026-06-16. A source-reported measurement, not independently reproduced.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
rpcreative-writing
User reports a 3B roleplay finetune running fully on an iPhone at around 6 tok/s on an A18 chip.
Setup is llama.cpp with Metal, Impish Llama 3B (a Llama 3.2 3B RP finetune) with a rank 16 LoRA fused and requantized to Q4_K_M, KV cache at q8_0, all layers offloaded, about 2.0 GB on disk.
The user notes the speed is from an early measurement and has not been re-measured on the current build. Context is 4096 on 4 GB phones, 6144 on 6 GB phones, and 8192 on 8 GB phones. The user reports that scaling the history trim budget to about 0.68 x n_ctx raised long-conversation fact recall from 25% to 75%, that BM25 retrieval re-injected superseded facts (+20.8pp) until filtered by rare entities (-18.8pp, CI [-35.4, -6.2]), and that scene directions placed as their own block after the final user turn landed 16/16 versus 0/16 in the reminder slot.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
agenticcodingsummarization
User benchmarks Llama 3.2 1B at 14.62 tok/s generation and 108.11 tok/s prompt processing on an Orange Pi 5 Plus (RK3588, 16GB LPDDR4x) using Ollama with 4 threads pinned to the Cortex-A76 big cores.
Setup is Ollama with Q4_K_M GGUF weights, CPU-only inference; the same board also ran DeepSeek-Coder 1.3B at 16.90 tok/s, Qwen 2.5 1.5B at 14.48 tok/s, Llama 3.2 3B at 7.28 tok/s, Phi-3 Mini 3.8B at 6.56 tok/s and Llama 3.1 8B at 2.32 tok/s, all with 4 threads.
Core pinning to 4 big cores gave up to +318% over the default 8 threads; the 8B model hit 85.0°C uncooled and the NPU path (RKLLM runtime) reached 21.55 tok/s on Qwen 1.5 0.5B with ~0% CPU load.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User benchmarks mlc-llm on an Orange Pi 5 Pro running Ubuntu off an m.2 NVMe SSD, with mlc-llm compiled for OpenCL.
The table reports prefill and decode tok/s for seven models; the record captures Llama-3.2-3B-Instruct-q4f16_1-MLC at 8.8 prefill tok/s and 6.1 decode tok/s.
Other rows: SmolLM-1.7B-Instruct 16.6/10.1, Qwen2.5-3B-Instruct 8.5/5.1, Phi-3.5-mini-instruct 5.5/4.8, SmolLM-360M-Instruct 44.4/25.7, SmolLM-1.7B-Instruct 18/9.8, TinyLlama-1.1B-Chat-v1.0 25.9/13.9.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Llama 3.2 1B Instruct at 15.4 tokens/s on a Raspberry Pi 5 (8GB).
Setup is llama.cpp compiled natively for Cortex-A76 with Q8_0 quantization, running CPU-only on 4 cores.
A Q4_K_M build reached 17.8 tokens/s, prompt eval 42.1 tokens/s, and time-to-first-token 0.38 s, compared with 0.85 tokens/s and 18.97 s TTFT for a generic ARM64 build.
User reports FastDMS, a DMS KV-cache compression implementation, decoding 1.5-2x faster than vLLM BF16/FP8 with 5-8x less KV memory.
Benchmarks cover Llama-3.2-1B and Qwen3-8B DMS checkpoints, with KLD and token match comparable or better than vLLM's FP8/TurboQuant.
Training took about 20 min on an RTX Pro 6000 Blackwell.
Individual observations from this page, not expected speeds or a ranking. Select a model to open its report, with the source and offloading conditions. Reported context may be a limit rather than actual input length.
The hardware Llama 3.2 3B at 4-bit is most run on: AMD Strix Halo 128GB (89.1 t/s median, 2 runs); M4 24GB (46.0 t/s median, one run); M5 Max 48GB (199.9 t/s median, one run). Medians of plain runs, as of 25 Oct 2026.
Which inference engines do people use for Llama 3.2?
Reports of Llama 3.2 name llama.cpp (8 reports), Ollama (4 reports), MLX (2 reports), vLLM (1 report), as of 25 Oct 2026.
Speed by hardware on Llama 3.2 3B at 4-bit
The size and quant level the most devices have plain runs of: one device, one request, no speculative decoding, and the whole model in the device's memory. Every speed in this table is the same model.