llamaperf

Local model performance on your hardware

Find which open-weight LLMs fit in your GPU or Mac and compare the speeds people report on setups like yours.

What runs on your hardware?

Pick your GPU or Mac, then read the speeds people reported on it, or estimate which models fit and how fast they run.

Free to use. No account needed. Memory estimates and community measurements are labelled separately.

Looking for a particular model?

Search for a model to see its reported speeds across GPUs and Macs.

Local model performance reports from the community

These are individual setups, not a controlled benchmark. Compare GPU count, quantization, context and offloading before comparing speeds. How to read a report →

Model: Qwen3-Coder
Compare setup details

Exact recorded values. Context may be a configured limit; matching filters does not establish identical prompts, offloading or concurrency.

Tone: positiveNVIDIA hardware
generation:
107.0 tokens/s
prompt processing (prefill):
3706.5 tokens/s
quant:
Q4_K_M (GGUF)
kv:
Q8_0
codingagentic

User reports Qwen3-Coder-30B-A3B-Instruct at 107.0 tok/s generation and 3,706.5 tok/s prefill on an RTX 5090 Laptop 24GB in an MSI Raider A18. Setup is llama.cpp with CUDA, Q4_K_M GGUF (17.28 GiB), Q8_0 KV cache, FlashAttention on, 65,536-token context, all layers on the GPU, one slot. The figures come from a repository-analysis workload with an 8,638-token prompt and 371 generated tokens; the user notes the screenshot captures one strong interval, not an average across the audit.

Nov 17, 2026
AMD hardware
generation:
97.7 tokens/s
prompt processing (prefill):
1115.3 tokens/s
quant:
Q4_K_S (GGUF)
kv:
f16
coding

User reports 97.73 t/s generation and 1115.3 t/s prompt processing. Source evidence (verbatim): Source: https://www.soothill.io/blog/2026/08/03/llamacpp-vulkan-vs-rocm-strix-halo/ # llama.cpp: Vulkan vs ROCm on Strix Halo Host: EVO-X3, 128GB integrated-memory Strix Halo desktop. llama.cpp Vulkan b10216 and ROCm b10085. 6 threads pinned, 32,768 context, flash attention on, batch/ubatch 2048/512, K/V cache f16, mmap off. Qwen3-Coder-30B-A3B-Instruct Q4_K_S (17.46 GB), prompt 512, generation 128, single stream: Vulkan 5 runs prompt 1,115.30 tok/s, generation 97.73 tok/s. ROCm 50 runs prompt 1,344.65, generation 73.65. Qwen3.5-122B-A10B Q4_K_XL (77.03 GB), ROCm: pp512 339.9, pp4096 523.0, generation 21.3 tok/s. Vulkan: pp512 285.0, pp4096 374.3, generation 22.9 tok/s. No speculative decoding stated.

Oct 16, 2026
Tone: positiveAMD hardware
generation:
173.0 tokens/s
quant:
BF16 (safetensors)
coding

User benchmarks Qwen3-Coder-30B-A3B in BF16 on a single MI300X 192GB, reporting 173 tok/s decode on a 32K prompt with SGLang defaults. Setup is SGLang 0.5.21 with a 131K context window, prefix caching off, and 256-token replies; vLLM 0.29.1 with AITER attention reached 164 tok/s and llama.cpp (HIP) 108 tok/s with a 20.5 s first token. SGLang was fastest from 1 to 64 users; vLLM only pulled ahead at 128 users. Prefix caching was the only tuning knob that helped, adding 8 to 11 percent. The user notes vLLM without AITER attention collapsed to 27 tok/s total on 16 users with 32K prompts versus 133 on SGLang.

Oct 16, 2026
AMD hardware
generation:
50.1 tokens/s
prompt processing (prefill):
442.1 tokens/s
quant:
Q4_K_M (GGUF)
kv:
Q8_0
coding

User reports 50.070235 t/s generation and 442.1505415 t/s prompt processing. Source evidence (verbatim): Source: https://www.soothill.io/blog/2026/08/19/vulkan-064-rocm-714-strix-halo/ # Strix Halo Qwen3-Coder Q4_K_M Vulkan measured 32K-depth benchmark First-hand Darren Soothill benchmark, measured/published 19 August 2026. Hardware GMKtec EVO-X3, Radeon 8060S gfx1151, 128GB unified memory (Strix Halo). CPU SKU and core counts not established by this article alone. Selected model Qwen3-Coder-30B-A3B-Q4_K_M, GGUF, llama.cpp Vulkan v0.6.4 build 10565 commit baf6360be (author short commit; full revision unknown). Selected measured context depth 32768 tokens already present before the operation; this is occupied depth, not allocated capacity. Microbatch 2048, batch2048, all available layers offloaded, Flash Attention enabled, Q8_0 K and V cache, one server slot, no-host loading. Native generation tg128 at that measured depth: 50.070235 tokens per second. Native prompt processing pp512 at the same depth and microbatch: 442.1505415 tokens per second. These are separate operation cells of the same matched synthetic configuration, not a single combined request or end-to-end rate. The author reports median of launch means. This 32K comparison used two launches per backend in ABBA order. Mesa26.3.0-devel RADV, libdrm2.4.134, shaderc2026.3-dev. IOMMU stayed enabled; no thermal, firmware or power-policy changes. Server slot count does not prove request concurrency; no concurrent request rate is inferred. Exact tested quant conversion repository, filename, checksum and revision are not published. Resident weight allocation and RAM consumption are unspecified. Original article https://www.soothill.io/blog/2026/08/19/vulkan-064-rocm-714-strix-halo/ . Original CSV https://www.soothill.io/assets/data/strix-halo-vulkan-064-rocm-714-2026-08-19.csv . Verified publisher model link https://huggingface.co/Qwen/Qwen3-Coder-30B-A3B-Instruct is a base-model reference, not the tested Q4_K_M artifact. Other backends, shallow cells, microbatches and served MTP requests are separate tests and must not be combined with the selected 32K synthetic cells.

Oct 14, 2026
AMD hardware
generation:
67.5 tokens/s
quant:
Q4_K_S (GGUF)
coding

User reports 67.5288 t/s generation. Source evidence (verbatim): Source: https://www.soothill.io/blog/2026/08/14/coding-model-benchmark-strix-halo/ # Qwen3-Coder 30B-A3B Q4_K_S: first-hand EVO-X3 coding decode measurement Original author Darren Soothill measured this deployment on 14 August 2026, published 14 August and updated 15 August. Hardware: GMKtec EVO-X3, Ryzen AI MAX+ 395, Radeon 8060S, 128GB unified memory (Strix Halo). Selected native decode result: Qwen3-Coder-30B-A3B-Instruct-Q4_K_S, llama.cpp, 67.5288 tokens per second. The author's CSV records this separately from wall_seconds_per_task 3.9879 and completion_tokens_per_task 250.8. This is an aggregate deployment result across ten first-response coding tasks, not a standalone llama-bench tg128 test. The publication does not specify whether the decode aggregate is arithmetic or token-weighted; do not reinterpret its statistic. Runtime build/version, CPU/GPU core counts, OS, concurrency, actual per-task prompt tokens and prefill rate are not supplied. Every coding request used a configured 32768-token context capacity and 2048-token output ceiling, temperature 0, top-p 1, presence penalty 0, thinking disabled and no repair turn. Configured context capacity is not the actual input length or occupied context depth. The model has 30.5B total and 3.3B active parameters. CSV installed_gib 16.26 describes installed storage, not resident RAM or GPU allocation. The 5/10 task pass score is quality, not throughput. Other model/quant rows and their speeds are separate deployments and must not be combined with this selected row. Original article: https://www.soothill.io/blog/2026/08/14/coding-model-benchmark-strix-halo/ . Author's machine-readable source: https://www.soothill.io/assets/data/evox3-coding-model-benchmark-2026-08-14.csv . Verified Hugging Face link supplied by the author: https://huggingface.co/Qwen/Qwen3-Coder-30B-A3B-Instruct . This is a base-model reference, not the exact tested Q4_K_S quantization artifact. Tested conversion repository, filename checksum and revision are not established; never infer them from the base repository.

Oct 14, 2026

Qwen3-Coder 30B (3B active)

AMD Strix Halo 128GB · llama.cpp / Vulkan KHR cooperative matrices / build unknown

AMD hardware
generation:
9.2 tokens/s
prompt processing (prefill):
154.0 tokens/s
quant:
BF16

User reports Qwen3-Coder 30B at 9.2 t/s generation and 154 t/s prompt processing on an AMD Strix Halo 128GB machine. Setup is llama.cpp with Vulkan on the Radeon 8060S iGPU, BF16 quantization, 6 threads and maximum GPU offload, run as llama-bench with -p 512 -n 128. The source gives 9.19 t/s TG and 154 t/s PP as separate synthetic phase tests, not an end-to-end rate; context depth, KV precision and memory use are unreported.

Oct 13, 2026
AMD hardware
generation:
69.6 tokens/s
prompt processing (prefill):
812.0 tokens/s
quant:
Q4_K_M (GGUF)
kv:
F16
flash attention:
on
mtp (multi-token prediction):
off

User reports Qwen3-Coder-30B-A3B-Instruct at 69.6 t/s generation and 812 t/s prompt processing on an AMD Strix Halo 128GB machine. Setup is llama.cpp with Q4_K_M weights, f16 K/V cache, flash attention, and one GPU (Radeon 8060S) with n_gpu_layers=999. The figures are separate synthetic llama-bench PP512 and TG128 tests at n_depth=0, not generation after a 512-token prompt and not HTTP end-to-end throughput; no long-context rate is implied.

Oct 13, 2026
AMD hardware
generation:
65.9 tokens/s
prompt processing (prefill):
401.5 tokens/s
quant:
UD-Q4_K_XL (GGUF)
flash attention:
on

User reports Qwen3-Coder-30B-A3B-Instruct at 65.9 t/s generation and 401.5 t/s prompt processing on an EVO-X2 with AMD Ryzen AI Max+ 395 and 128 GB unified memory. Setup is llama.cpp with the UD-Q4_K_XL GGUF, ROCm 6.4.0, Flash Attention on, and one gfx1151 iGPU at ngl=100. The source gives 65.93 +/- 0.18 t/s for tg128 and 401.48 +/- 2.72 t/s for pp512, with tg8192 at 63.27 t/s; these are separate synthetic microbenchmarks, not a same-request prompt and answer.

Oct 13, 2026
AMD hardware
generation:
59.8 tokens/s
prompt processing (prefill):
409.5 tokens/s
quant:
UD-Q4_K_XL (GGUF)
flash attention:
off

User reports Qwen3-Coder-30B-A3B-Instruct at 59.8 t/s generation and 409.5 t/s prompt processing on an EVO-X2 with AMD Ryzen AI Max+ 395 and 128 GB unified memory. Setup is llama.cpp with the UD-Q4_K_XL GGUF, ROCm 6.4.0, Flash Attention off, and one gfx1151 iGPU at ngl=100. The rates are separate llama-bench microbenchmarks, not a same-request prompt and answer; longer tests in the same arm give pp8192 254.28 t/s and tg8192 30.78 t/s.

Oct 13, 2026
AMD hardware
generation:
71.3 tokens/s
prompt processing (prefill):
399.8 tokens/s
quant:
UD-Q4_K_XL (GGUF)
kv:
F16
flash attention:
on
mtp (multi-token prediction):
off

User reports Qwen3-Coder-30B-A3B-Instruct at 71.3 t/s generation and 399.8 t/s prompt processing on an EVO-X2 with AMD Ryzen AI Max+ 395 and 128 GB unified memory. Setup is llama.cpp build 6060 with the UD-Q4_K_XL GGUF, Flash Attention on, Vulkan backend, and ngl=100 on the single integrated Radeon 8060S. The figures are separate llama-bench microbenchmarks: tg128 at 71.29 +/- 0.11 t/s and pp512 at 399.76 +/- 1.41 t/s, with longer runs of tg8192 at 59.91 +/- 1.95 t/s and pp8192 at 315.36 +/- 0.33 t/s.

Oct 13, 2026
AMD hardware
generation:
73.0 tokens/s
prompt processing (prefill):
391.5 tokens/s
quant:
UD-Q4_K_XL (GGUF)
kv:
F16
flash attention:
off
mtp (multi-token prediction):
off

User reports Qwen3-Coder-30B-A3B-Instruct at 73 t/s generation and 391.5 t/s prompt processing on an EVO-X2 with AMD Ryzen AI Max+ 395 and 128 GB unified memory. Setup is llama.cpp build 6060 with the UD-Q4_K_XL GGUF, Flash Attention off, Vulkan backend, and ngl=100 on the integrated Radeon 8060S. The source-native rates are separate llama-bench microbenchmarks: pp512 at 391.54 +/- 1.50 t/s and tg128 at 72.99 +/- 0.31 t/s, with longer tests at pp8192 284.71 t/s and tg8192 46.60 t/s.

Oct 13, 2026
Apple hardware
generation:
71.2 tokens/s
prompt processing (prefill):
1096.8 tokens/s
quant:
UD-Q4_K_XL (GGUF)
kv:
F16
flash attention:
on
mtp (multi-token prediction):
off

User reports Qwen3-Coder 30B-A3B at 71.2 t/s generation and 1096.8 t/s prompt processing on a Mac Studio M2 Ultra with 128GB unified memory. Setup is llama.cpp build 6060 with the UD-Q4_K_XL GGUF and flash attention enabled, run with -ngl 100 on one 60-core GPU. The figures are separate synthetic llama-bench tests: PP512 at 1096.84 +/- 5.48 t/s and TG128 at 71.23 +/- 0.08 t/s, with 16 CPU threads and a 16.45 GiB model footprint.

Oct 13, 2026
Apple hardware
generation:
69.3 tokens/s
prompt processing (prefill):
1080.5 tokens/s
quant:
UD-Q4_K_XL (GGUF)
kv:
F16
flash attention:
off
mtp (multi-token prediction):
off

User reports Qwen3-Coder 30B-A3B at 69.3 t/s generation and 1080.5 t/s prompt processing on a Mac Studio M2 Ultra with 128GB unified memory. Setup is llama.cpp build 6060 with the UD-Q4_K_XL GGUF and flash attention off, one GPU with -ngl 100 and 16 threads. The figures are separate synthetic llama-bench tests: PP512 and TG128, not a 512-input/128-output request. The same run measured 818.85 t/s PP8192 and 40.35 t/s TG8192.

Oct 13, 2026
AMD hardware
generation:
196.0 tokens/s
quant:
BF16 (safetensors)
coding

User benchmarks Qwen3-Coder-30B-A3B-Instruct in BF16 and FP8 on a single MI300X 192GB, reporting 196 tok/s at 1 concurrent request for BF16 and 136 tok/s for FP8. Setup is vLLM 0.29.1 in the rocm/vllm container on AMD Developer Cloud, 1024 prompt tokens and 256 output tokens, vllm bench serve. The 1-request figures are medians of 3 warm runs. FP8 is 31% slower at 1 request but 35% faster at 64 concurrent (3442 vs 2555 tok/s). Accuracy on GSM8K (250 problems) was 96.4% BF16 vs 94.0% FP8. llama.cpp with Q4_K_M reached 187 tok/s at 1 user. Also ran gpt-oss-120b (72 tok/s at 1, 1423 at 64) and Qwen2.5-72B BF16 (26 tok/s at 1, 421 at 32). Total cost about $5.73.

Oct 12, 2026
NVIDIA hardware
generation:
51.7 tokens/s
prompt processing (prefill):
1928.1 tokens/s
quant:
Q4_K_M (GGUF)

User reports Qwen3-Coder 30B-A3B at 51.7 t/s generation and 1928.1 t/s prompt processing on an NVIDIA RTX 5080. Setup is Ollama with Q4_K_M at a configured 16384-token context, with partial CPU/RAM offload since the model exceeds VRAM. The recorded prompt was 1347 tokens, not a full 16384-token prefill, and the figures are medians of 3 fresh-prefill runs; 74% GPU residency was measured separately at 8K context, not this cell.

Oct 12, 2026
Apple hardware
generation:
64.4 tokens/s
prompt processing (prefill):
645.0 tokens/s
quant:
Q4_K_M (GGUF)

User reports Qwen3-Coder 30B-A3B at 64.4 t/s generation and 645 t/s prompt processing on an M1 Max 64GB. Setup is llama.cpp with Q4_K_M, measured via llama-bench on Metal. The figure is a native prefill/decode rate, not a TTFT-derived estimate; exact test lengths, context, batch settings and KV precision are not given.

Oct 12, 2026
AMD hardware
generation:
62.8 tokens/s
prompt processing (prefill):
1692.0 tokens/s
quant:
Q8_0
mtp (multi-token prediction):
off

User reports Qwen3-Coder 30B-A3B at 62.8 t/s generation and 1692 t/s prompt processing on an AMD Gorgon Halo 192GB machine. Setup is llama.cpp with Q8_0 and the Vulkan backend, all layers on GPU, on a Minisforum MS-S1 MAX-P495 with Ryzen AI Max+ PRO 495 and 192 GB unified memory. The 1692 t/s and 62.8 t/s come from separate llama-bench pp512 and tg128 tests, not a paired 512-input/128-output run; the same source reports about 205 t/s prompt and 33.1 t/s generation at roughly 32k depth.

Oct 11, 2026
Tone: positiveAMD hardware
generation:
70.3 tokens/s
quant:
Q4_K_M (GGUF)
rating:
5/5
coding

User reports Qwen3-Coder-30B-A3B-Instruct at 70.25 t/s generation on an AMD Strix Halo (GMKTEC EVO-X2) with 128GB unified memory. Setup is llama.cpp (ROCm 7 RC, rocWMMA) with Q4_K_M GGUF, 8K context, all 99 layers offloaded to GPU, 2 parallel slots. Short prompt (9 tokens) prefill was 4.94 t/s; medium (51 tokens) 28.29 t/s and 71.00 t/s generation; large (88 tokens) 46.81 t/s and 68.09 t/s generation. User rates the setup 5/5 and notes generation is 62% faster than a previous Llama-2-7B baseline.

Oct 9, 2026
AMD hardware
generation:
66.1 tokens/s
quant:
Q4_K_M (GGUF)
coding

User benchmarks Qwen3 Coder 30B A3B at 66.11 t/s generation on one AMD Radeon Instinct Mi50 32GB. Setup is llama.cpp build 128d522c (6686) with the ROCm backend, Q4_K_M quant (17.28 GiB), -ngl 99, 112 threads and --numa distribute on a dual Xeon 8480+ host. Other quants on the same card: Q5_0 65.15 t/s, Q6_K 62.49 t/s, Q8_0 64.20 t/s. BF16 (56.89 GiB) required two GPUs and ran 41.41 t/s. With 16K tokens of output the rate falls to about 20 t/s.

Sep 30, 2026
Showing 1–19 of 19
Page 1 of 1

Community benchmarks snapshot

Records by GPU

AMD Strix Halo 128GB348NVIDIA RTX 3090271NVIDIA RTX 5090197NVIDIA DGX Spark99NVIDIA RTX 3060 12GB96NVIDIA RTX 5060 Ti 16GB88NVIDIA RTX Pro 6000 Blackwell81AMD Radeon AI PRO R9700 32GB81NVIDIA RTX 5070 Ti77NVIDIA RTX 409068

Records by model

3266 total
Qwen3.81567
Qwen3.6334
Gemma 4185
DeepSeek V4 Flash150
Qwen3.5145
DeepSeek V4.1 Flash103
other782

Records by engine

2308 total
llama.cpp1134
Strata234
vLLM227
MLX107
Ollama99
other507

Use cases

coding 937agentic 640long-context 442tool-use 281vision 174summarization 81math 77creative-writing 58multilingual 35rp 22text-generation 9reasoning 3
coding937agentic640long-context442tool-use281vision174summarization81math77creative-writing58

Median generation t/s by GPU

On Qwen3.8 27B at 4-bit, plain single-GPU runs. Full ranking

RTX 509080RTX Pro 600067RTX 5070 Ti55M5 Ultra 256GB50RTX 409040RX 7900 XTX40RTX 309038RX 7800 XT 16GB35RTX 5090 Laptop 24GB34RX 7900 XT32

Prompt processing, same model

M5 Ultra 256GB1800RTX 40901949RX 7900 XTX640RTX 3090561RX 7900 XT800M3 Ultra 512GB372V100 32GB941Radeon AI PRO R9700 32GB1287RTX 3080 20GB997Arc Pro B70512

Reports by model size

Qwen3.8 27B819Qwen3.8 125B · 6B active676Qwen3.6 35B · 3B active242DeepSeek V4 Flash 284B · 13B active133DeepSeek V4.1 Flash 552B · 16B active92Qwen3.6 27B81Gemma 4 26B · 4B active61GLM-5.3 320B · 18B active57

Quants

Q4_K_M395UD-Q4_K_XL189NVFP4174Q8_01194-bit106IQ4_XS86IQ3_XXS85IQ3_S84Q4_K_XL83Q480