llamaperf

Local model performance on your hardware

Find which open-weight LLMs fit in your GPU or Mac and compare the speeds people report on setups like yours.

What runs on your hardware?

Pick your GPU or Mac, then read the speeds people reported on it, or estimate which models fit and how fast they run.

Free to use. No account needed. Memory estimates and community measurements are labelled separately.

Looking for a particular model?

Search for a model to see its reported speeds across GPUs and Macs.

Local model performance reports from the community

These are individual setups, not a controlled benchmark. Compare GPU count, quantization, context and offloading before comparing speeds. How to read a report →

GPU: AMD RX 9070 XT 16GB
Compare setup details

Exact recorded values. Context may be a configured limit; matching filters does not establish identical prompts, offloading or concurrency.

Tone: positiveAMD hardware
generation:
11.0 tokens/s
prompt processing (prefill):
280.0 tokens/s
quant:
IQ4_XS (GGUF)
rp

User reports GLM-5.3-Flash at about 11 tok/s on an RX 9070 XT 16GB, with long prompts read at about 280 tok/s on Linux. Setup is the user's own engine with IQ4_XS GGUF, experts streamed from NVMe drives with hot experts kept on the GPU and in RAM, on a Core Ultra 7 265KF with 32 GB DDR5 and two NVMe drives. Windows gives about 10 tok/s and about 250 tok/s prompt reading. Testers measured about 11 tok/s on an Intel Arc Pro B70 32GB and 5.6 tok/s on an RTX 3090 plus RTX 3060; the 12-13 and 6.5-7 tok/s figures are expectations from logs, not measurements. The model needs about 157 GB on disk and 32 GB of RAM.

Nov 18, 2026
Tone: mixedAMD hardware
generation:
68.4 tokens/s
quant:
B12

User reports Qwen3-8B at 68.4 tok/s with speculative decoding on an RX 9070 XT 16GB, versus 42.7 tok/s plain, using a custom engine that packs bf16 weights into a lossless 12-bit format (B12) with a Qwen3-0.6B draft model. Setup is the user's own engine on Windows with ROCm PyTorch, batch-invariant GEMV kernels and seeded Gumbel-max sampling; speculative output matched plain decoding on 11 of 11 prompts. On 50 longer prompts of 512 tokens the speculative run reached 73.0 tok/s with 50/50 matches. The user notes llama.cpp Q8_0 with a draft is about 1.4x faster (98.8 tok/s ROCm, 95.0 Vulkan) but lossy, matching plain output on only 7 of 11 prompts.

Nov 16, 2026

Qwen3.6 35B (3B active)

AMD RX 9070 XT 16GB · ExLlamaV3 · 128,000 ctx

Tone: positiveAMD hardware
generation:
71.6 tokens/s
prompt processing (prefill):
1246.0 tokens/s
quant:
EXL3 4.0 bpw (EXL3)
kv:
Q8
agenticlong-context

User benchmarks MoE expert offloading on an RX 9070 XT 16GB with 32GB DDR4, reporting Qwen3.6-35B-A3B at 71.6 t/s decode and 1246 t/s prefill at 128K context. Setup is ExLlamaV3 with EXL3 4.0 bpw and Q8 KV cache, DFlash2 with 2 drafts, experts offloaded to system RAM. Prefill drops from 1900 t/s at 16K to 700 t/s at 128K on the 35B models. Ornith 1.5 35B-A3B reached 57.3 t/s decode and 1168 t/s prefill at 128K, and Qwen3.8-Flash-Next Coder on Polystrata with IQ1_M reached 44.0 t/s decode and 1572 t/s prefill at 128K.

Nov 12, 2026
Tone: mixedAMD hardware
generation:
70-75 tokens/s
quant:
IQ3_S (GGUF)

User asks which quant to settle on for Qwen3.8 Flash-Next run with Strata on an RX 9070 XT with 64GB DDR5 and a 9800X3D, all at 256k context. ISTA IQ3_S runs at ~70-75 tokens/s, Unsloth UD-IQ3_XSS (3.1) at ~60-65 tokens/s and Unsloth UD-IQ4_XS (3.1) at ~40-50 tokens/s. User notices no substantial quality difference between the three and questions whether the ~40% speed penalty of UD-IQ4_XS over IQ3_S is worth it.

Nov 9, 2026
Tone: positiveAMD hardware
generation:
88.0 tokens/s
prompt processing (prefill):
514.0 tokens/s
quant:
IQ3KS (GGUF)
agentictool-usecoding

User reports Qwen3.8-Flash-Next at 88 t/s generation and 514 t/s prompt processing on an RX 9070 XT 16GB. Setup is the Avalon llama.cpp fork with IQ3KS at xhigh effort, after upgrading the machine to 64 GB of 4800 MHz DDR5. The run used 5 symbols over 4 runs plus 1 failure, at 42-45 minutes per run. The user also compares against Anthropic Haiku 5.5 at 207 t/s and other local runs on Strata and llama.cpp.

Nov 5, 2026
Tone: positiveAMD hardware
generation:
7.0 tokens/s
quant:
IST-DAS 1.75 bpw
codingagenticlong-context

User reports DeepSeek V4.1 Flash at 1M context running on an RX 9070 XT 16GB with experts spread across VRAM, 64 GB DDR4 and an SSD, at about 7 tok/s generation. Setup is a patched Strata engine (HIP, gfx1201) with IST-DAS 1.75 bpw weights, plus a 5800X3D and PCIe 3.0 NVMe. The patch adds KV splicing and resident-only expert routing during prompt reads. Prompt reading for a ~37K-token agent system prompt took about 100 s before the patch; the user claims agent sessions now start 20x faster. A 1.2K-token prompt previously streamed 58 GB off the SSD. The splice is approximate and resident routing changes answers slightly; the user's coding suite scored 12/13 both ways.

Nov 2, 2026
Tone: negativeAMD hardware
prompt processing (prefill):
1049.1 tokens/s
quant:
IQ3_S (GGUF)
kv:
q8_0

User reports Qwen3.5 27B at IQ3_S prefill of 1049.13 t/s (pp512) on an RX 9070 XT with the Lemonade B1356 ROCm build, against 460.03 t/s on B1357. Setup is llama.cpp with IQ3_S weights (9.72 GiB), q8_0 KV cache, flash attention on, 99 layers offloaded and 512 batch. Prefill falls from 1049.13 to 460.03 t/s at pp512 between B1356 and B1357, and rebuilding llama.cpp over B1356 returns the slow result. No generation figures are given.

Oct 31, 2026
AMD hardware
generation:
65.0 tokens/s
quant:
IQ3_M (GGUF)

User reports Qwen3.8-27B at about 65 tok/s on a Radeon RX 9070 XT. Setup is llama.cpp with IQ3_M weights, run under both Vulkan and ROCm 10.1 on a Ryzen 7 7800X3D with 32 GB DDR5. User asks whether Strata can span two machines to run Qwen3.8-Flash-Next (125B-A6B), and whether llama.cpp RPC over a LAN is realistic; no figures are given for that setup.

Oct 31, 2026
Tone: positiveAMD hardware
prompt processing (prefill):
3700.0 tokens/s
quant:
fp8-iq4r
kv:
fp8_v4
vision

User reports ~3700 t/s prefill with Qwen3.8 Flash-Next fp8-iq4r on a single RX 9070 XT, with experts placed in RAM and a ~57GB host pool. KV cache uses fp8_v4 and --max-num-batched-tokens is set to 4096, which gave the best prefill results. The card is power limited during prefill at over 300W, and PCIe5 x16 is nearly saturated; gen 4 drops prefill to ~2000 t/s. Prefill stays flat across context while decode decays 18-19% over 100k tokens, with a worst case of ~45 t/s at 240k context. Vision is fully working. The user has 96GB of DDR5 and is still working on decode speed.

Oct 26, 2026
Tone: negativeAMD hardware
generation:
30-50 tokens/s
quant:
Q4_K_M (GGUF)
kv:
Q8

User reports Qwen3.8 27B at about 30-50 tok/s on an RX 9070 XT paired with an RTX 3080, with performance dropping noticeably at 64k versus 128k context. Setup is llama.cpp with Q4_K_M weights and Q8 KV cache, layer split rather than tensor split, MTP enabled, one parallel slot and flash attention on. MTP roughly doubles speed; the other tweaks mainly save VRAM. The user calls the pairing a bad choice for LLM work, citing the 9070 XT's lower memory bandwidth and weaker ROCm/Vulkan performance, and notes task manager shows only about 60% and 30% GPU usage on the two cards. Monitors must stay on the 9070 XT or performance collapses.

Oct 19, 2026
Tone: positiveAMD hardware
generation:
46.1 tokens/s
prompt processing (prefill):
549.3 tokens/s
quant:
IQ3_M (GGUF)
kv:
q8_0
rp

User reports 46.06 t/s generation and 549.27 t/s prompt processing. Source evidence (verbatim): Source: https://lemmy.world/post/48144487 # Qwen 3.6 27B running at 46 tok/s on an RX 9070 XT (llama.cpp + MTP Speculative Decoding is basically magic) Just got my hands on a new AMD Radeon RX 9070 XT (16GB) and wanted to share some inference numbers. I've been messing around with llama.cpp via their official ROCm Docker image, testing out Qwen 3.6 27B (Omnimerge-v4) in IQ3_M. Honestly, the performance you can squeeze out of a 27B model on a 16GB consumer card right now is blowing my mind. Here’s the breakdown: The Setup - GPU: AMD Radeon RX 9070 XT (RDNA4 / gfx1201) - 16 GB VRAM - CPU: AMD Ryzen 9 9950X3D - OS/Backend: Linux via Docker using ghcr.io/ggml-org/llama.cpp:server-rocm. (Props to the devs, it natively supports RDNA4 gfx1201 out of the box!) - Model: Qwen3.6-27B-Omnimerge-v4-IQ3_M.gguf (~13 GB) - Context: 16k Tweaks: - Set -np 1 since I'm just running it as a single-user chatbot in Open WebUI. - Slapped on 8-bit KV cache (--cache-type-k q8_0 --cache-type-v q8_0) to save about 50% VRAM. - Enabled MTP Speculative Decoding (--spec-type draft-mtp). The Numbers (512-token test) - Prompt Processing (TTFT): 549.27 tok/s (1220 tokens evaluated in ~2.2s). Latency: 1.82 ms/token. - Text Gen: 46.06 tok/s (512 tokens generated in ~11.1s). Latency: 21.71 ms/token. - MTP Stats: Draft acceptance rate was super high at 62.7% (333 drafts accepted / 531 generated). Aggregate speed (including prompt eval) hit roughly 48.97 tok/s. Memory Footprint - VRAM: Sitting at 14.46 GB out of 16 GB. This leaves about 1.5 GB of breathing room, which has been totally stable with zero OOM crashes so far. - System RAM: ~4.3 GB (mostly the host-side prompt cache helping speed up subsequent turns). RDNA4 is ready: The latest ROCm images and llama.cpp HIP libs support the 9070 XT natively. Didn't even need to mess with HSA_OVERRIDE_GFX_VERSION. KV Cache quantization is required: Pushing the KV cache to q8_0 is the only reason a 16k context window fits on a 16GB card alongside a 27B model. If anyone with a 16GB card is looking for the sweet spot, this has to be one of the best price-to-performance setups available right now. Let me know if you want my docker-compose.yml or run scripts... Edit: To see how much MTP actually helps, I ran the exact same 512-token prompt test with speculative decoding toggled off: Standard Inference (No Speculation): - Speed: 25.88 tokens/sec (Latency: 38.64 ms/token) - Prompt Eval: 239.46 tokens/sec MTP Speculative Decoding (MTP): - Speed: 46.06 tokens/sec (Latency: 21.71 ms/token) - Prompt Eval: 549.27 tokens/sec - Draft Acceptance Rate: 62.7% For more info, here are my logs ::: spoiler Logs ``` ROCm GPU and System Info: device_info: - ROCm0 : AMD Radeon RX 9070 XT (16304 MiB, 16200 MiB free) - ROCm1 : AMD Ryzen 9 9950X3D 16-Core Processor (15589 MiB, 25012 MiB free) - CPU : AMD Ryzen 9 9950X3D 16-Core Processor (31178 MiB, 31178 MiB free) I system_info: n_threads = 16 (n_threads_batch = 16) / 32 | ROCm : NO_VMM = 1 | PEER_MAX_BATCH_SIZE = 128 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | AVX512 = 1 | AVX512_VBMI = 1 | AVX512_VNNI = 1 | AVX512_BF16 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 Context Initialization (16k single-slot & 8-bit KV Cache): I srv load_model: [spec] estimated memory usage of MTP context is 178.02 MiB W llama_context: n_ctx_seq (16384) < n_ctx_train (262144) -- the full capacity of the model will not be utilized I srv load_model: initializing slots, n_slots = 1 I slot load_model: id 0 | task -1 | new slot, n_ctx = 16384 I srv load_model: prompt cache is enabled, size limit: 8192 MiB I srv llama_server: model loaded I srv llama_server: server is listening on http://0.0.0.0:8080/ Benchmark Request Timing Statistics (Generating 512 tokens): I slot launch_slot_: id 0 | task 3454 | processing task, is_child = 0 I slot print_timing: id 0 | task 3454 | n_decoded = 103, tg = 37.93 t/s I slot print_timing: id 0 | task 3454 | n_decoded = 247, tg = 43.16 t/s I slot print_timing: id 0 | task 3454 | n_decoded = 392, tg = 44.87 t/s I slot print_timing: id 0 | task 3454 | prompt eval time = 195.88 ms / 42 tokens ( 4.66 ms per token, 214.41 tokens per second) I slot print_timing: id 0 | task 3454 | eval time = 11116.24 ms / 512 tokens ( 21.71 ms per token, 46.06 tokens per second) I slot print_timing: id 0 | task 3454 | total time = 11312.12 ms / 554 tokens I slot print_timing: id 0 | task 3454 | graphs reused = 3560 I slot print_timing: id 0 | task 3454 | draft acceptance = 0.62712 ( 333 accepted / 531 generated) I statistics draft-mtp: #calls(b,g,a) = 6 3607 3607, #gen drafts = 3607, #acc drafts = 2824, #gen tokens = 10820, #acc tokens = 6605, dur(b,g,a) = 0.008, 31925.089, 2.740 ms I slot release: id 0 | task 3454 | stop processing: n_tokens = 553, truncated = 0 ``` ___ :::

Oct 19, 2026
Tone: positiveAMD hardware
generation:
40.0 tokens/s
quant:
UD-IQ4_XS (GGUF)
kv:
INT8
agenticlong-context

User reports Qwen3.8-Flash-Next at around 40 tokens/s on a single RX 9070 XT 16 GB, with experts partly held in 64 GB of system RAM and read from NVMe when not cached. Setup is Strata 0.1.44.1 with Unsloth UD-IQ4_XS weights, INT8 KV cache and a 292,000-token context using YaRN; the RAM expert budget is 48 GiB and the engine memory limit 57 GiB. Prompt processing is about 5,364 tokens/s but may reuse cached context, so it is not a cold-prefill figure. Thinking runs about 36.5 tokens/s and some runs reached about 51 tokens/s. A 49 GiB expert budget was OOM-killed on a roughly 161,000-token conversation.

Oct 18, 2026

Qwen3.5 9B

AMD RX 9070 XT 16GB · llama.cpp · 65,536 ctx

AMD hardware
generation:
62.0 tokens/s
quant:
Q6_K_XL (GGUF)

User reports 62 t/s generation. Source evidence (verbatim): Source: https://digtvbg.com/blog/llama-server-vulkan-rdna4-vllm-rocm-benchmark/ # Local LLM Inference on AMD RX 9070 XT: Vulkan vs ROCm Benchmarks on RDNA4 Hardware: AMD RX 9070 XT with 16GB VRAM on CachyOS. Model: Qwen3.5-9B-UD-Q6_K_XL (about 6.5 bits per weight). llama-server from llama.cpp built against Vulkan with --n-gpu-layers 999 -c 65536 --batch-size 2048 --ubatch-size 2048 --parallel 1: 62 tokens/second. vLLM on ROCm 7.2: 48 tokens/second. Same prompt, max tokens, temperature 0 and seed 42 for both; 10 measured runs per backend after 2 warmup runs. No speculative decoding is configured.

Oct 17, 2026

Qwen3.8 27B

AMD RX 9070 XT 16GB · llama.cpp · 262,144 ctx

Tone: positiveAMD hardware
generation:
36.5 tokens/s
prompt processing (prefill):
1368.4 tokens/s
quant:
IQ4_XS (GGUF)
kv:
Q8_0/Q4_0

User reports Qwen3.8-27B at 36.5 t/s decode on a single 16 GB RX 9070 XT, using a custom llama.cpp fork with adaptive KV streaming over PCIe. Setup is llama.cpp with IQ4_XS weights and Q8_0/Q4_0 KV cache, running the full 262,144-token context with the KV cache held in host RAM (~7.4 GB) and streamed in bounded working sets. Decode falls from 36.5 t/s at 1k to 9.5 t/s at 262k; with MTP speculative decoding it reaches 71.8-86.5 t/s at 1k-4k and 31.1 t/s at 262k. Prefill is 1368.4 t/s at 1k and 585.8 t/s at 262k. The card runs at a 231W limit with a 3.6% prefill penalty and no decode loss.

Oct 17, 2026

Qwen3.8 27B

AMD RX 9070 XT 16GB · ExLlamaV3 · 96,000 ctx

Tone: positiveAMD hardware
prompt processing (prefill):
1100-1250 tokens/s
quant:
EXL3 3.0 bpw (EXL3)
codingcreative-writing

User reports Qwen3.8-27B at ~150 tok/s on code at short context on a single RX 9070 XT. Setup is ExLlamaV3 with EXL3 3.0 bpw quant, DFlash2 speculative decoding, and TabbyAPI, running in a container with RDNA4 patches. Default context is 96k, with 128k available if nothing else uses the card. At 120k context code generation drops to ~105 tok/s, prose to ~60 tok/s. Prompt processing is ~1,100-1,250 tok/s on short prompts and ~900 tok/s at 128k. Up to 3 concurrent requests share the context. GPQA Diamond scores 88.4% at xhigh effort and ~81% at medium.

Oct 14, 2026
Tone: mixedAMD hardware
generation:
0.2 tokens/s
quant:
fmt=4 grouped int4, gs=64

User reports GLM-5.2 744B at 0.2414 tok/s (10-run median) on an RX 9070 XT 16GB with colibri's Vulkan backend at its default 320 resident experts. Setup is colibri's Vulkan backend with COLI_VK_EXPERTS=320, MTP drafting at 2.00 tokens per forward, fmt=4 grouped int4 weights with group size 64, experts streamed from a Gen3 x4 NVMe and 64GB DDR4 system RAM, Resizable BAR on, 40 generated tokens. The box is storage-bound. The HIP arms (0.3189 tok/s at 6 GB, 0.3331 at 12 GB) ran with MTP disabled by a guard bug, and the author has withdrawn that Vulkan-vs-HIP comparison. 640 resident experts gave 0.2475 tok/s, and with Resizable BAR off the Vulkan run fell to 0.11 tok/s.

Oct 8, 2026

Qwen3 8B

AMD RX 9070 XT 16GB · llama.cpp · 512 ctx

Tone: negativeAMD hardware
generation:
98.8 tokens/s
prompt processing (prefill):
4245.0 tokens/s
quant:
Q4_K_M (GGUF)

User reports Qwen3-8B Q4_K_M at 98.8 t/s decode and 4245 t/s prefill (pp512) on an RX 9070 XT with llama.cpp's HIP backend. Setup is llama.cpp b10290 official HIP build on one RX 9070 XT 16 GB, Q4_K_M weights, all layers on the GPU, Windows 11, no speculative decoding. The issue reports a Vulkan backend fault on the same card: Qwen3-8B Q4_K_M drops to 21.0 t/s decode and 791 t/s at pp128 there, and models with hidden size 4096 or more run 4.7-6.7x slower than on HIP, while Qwen3-4B is normal at 183.0 t/s. Qwen3-14B Q4_K_M runs 54.3 t/s on HIP against 8.1 on Vulkan.

Oct 8, 2026

Qwen3.5 9B

AMD RX 9070 XT 16GB · llama.cpp · 32,768 ctx

Tone: negativeAMD hardware
generation:
7.9 tokens/s
quant:
Q4_K_M (GGUF)
kv:
q8_0
flash attention:
on

User reports Qwen3.5 9B Q4_K_M decoding at 7.89 t/s on an AMD RX 9070 XT under a pinned llama.cpp Vulkan build (b10964), while LM Studio's Vulkan llama.cpp build reaches 29.89-30.46 t/s on the same GPU and byte-identical GGUF. Setup is llama.cpp with Q4_K_M and q8_0 KV cache at 32768 context, flash attention on, single RX 9070 XT; llama-bench -p 128 -n 64 -r 1 gives 451.45 pp128 t/s and 7.89 tg64 t/s at -ngl 99. -ngl 0 (CPU) decodes at 8.00 t/s and -ngl 99 at 7.83-7.89 t/s, so GPU offload buys nothing; the iGPU alone gives 3.04 t/s. Through the managed runtime decode is 7.41-7.49 t/s with prompt eval 98-222 t/s. LM Studio loaded the model at 255744 context and was still about 4x faster. Root cause not established.

Oct 8, 2026
Tone: mixedAMD hardware
generation:
62.0 tokens/s
codingagentictool-usesummarization

User reports Qwen3.6 35B A3B at 62 t/s on an AMD RX 9070 XT 16GB, with vLLM on ROCm reaching 48 t/s in the same comparison. Setup is llama.cpp with the Vulkan backend, 32k context, on a Ryzen 7 9800X3D with 64GB DDR5; the MoE model spills part of its weights to system RAM. User notes the setup is early and numbers are not settled, that 16GB VRAM rules out big dense models, and that the local model needs supervision.

Oct 5, 2026

Qwen3.8 27B

AMD RX 9070 XT 16GB · llama.cpp · 65,536 ctx

Tone: positiveAMD hardware
generation:
42.0 tokens/s
coding

User reports Qwen 3.8 27B at 42 t/s on an AMD RX 9070 XT 16GB. Setup is llama.cpp with MTP2 speculative decoding and a 64K context, using about 1 GB of working RAM during evaluation. MTP2 delivered +41% to +53% throughput depending on the quantization, but the user notes the fastest configuration was not necessarily best for reasoning quality.

Sep 29, 2026
Showing 1–20 of 27
Page 1 of 2

Community benchmarks snapshot

Records by GPU

AMD Strix Halo 128GB348NVIDIA RTX 3090271NVIDIA RTX 5090197NVIDIA DGX Spark99NVIDIA RTX 3060 12GB96NVIDIA RTX 5060 Ti 16GB88NVIDIA RTX Pro 6000 Blackwell81AMD Radeon AI PRO R9700 32GB81NVIDIA RTX 5070 Ti77NVIDIA RTX 409068

Records by model

3266 total
Qwen3.81567
Qwen3.6334
Gemma 4185
DeepSeek V4 Flash150
Qwen3.5145
DeepSeek V4.1 Flash103
other782

Records by engine

2308 total
llama.cpp1134
Strata234
vLLM227
MLX107
Ollama99
other507

Use cases

coding 937agentic 640long-context 442tool-use 281vision 174summarization 81math 77creative-writing 58multilingual 35rp 22text-generation 9reasoning 3
coding937agentic640long-context442tool-use281vision174summarization81math77creative-writing58

Median generation t/s by GPU

On Qwen3.8 27B at 4-bit, plain single-GPU runs. Full ranking

RTX 509080RTX Pro 600067RTX 5070 Ti55M5 Ultra 256GB50RTX 409040RX 7900 XTX40RTX 309038RX 7800 XT 16GB35RTX 5090 Laptop 24GB34RX 7900 XT32

Prompt processing, same model

M5 Ultra 256GB1800RTX 40901949RX 7900 XTX640RTX 3090561RX 7900 XT800M3 Ultra 512GB372V100 32GB941Radeon AI PRO R9700 32GB1287RTX 3080 20GB997Arc Pro B70512

Reports by model size

Qwen3.8 27B819Qwen3.8 125B · 6B active676Qwen3.6 35B · 3B active242DeepSeek V4 Flash 284B · 13B active133DeepSeek V4.1 Flash 552B · 16B active92Qwen3.6 27B81Gemma 4 26B · 4B active61GLM-5.3 320B · 18B active57

Quants

Q4_K_M395UD-Q4_K_XL189NVFP4174Q8_01194-bit106IQ4_XS86IQ3_XXS85IQ3_S84Q4_K_XL83Q480