llamaperf

Local model performance on your hardware

Find which open-weight LLMs fit in your GPU or Mac and compare the speeds people report on setups like yours.

What runs on your hardware?

Pick your GPU or Mac, then read the speeds people reported on it, or estimate which models fit and how fast they run.

Free to use. No account needed. Memory estimates and community measurements are labelled separately.

Looking for a particular model?

Search for a model to see its reported speeds across GPUs and Macs.

Local model performance reports from the community

These are individual setups, not a controlled benchmark. Compare GPU count, quantization, context and offloading before comparing speeds. How to read a report →

GPU: AMD Radeon Pro W7900
Compare setup details

Exact recorded values. Context may be a configured limit; matching filters does not establish identical prompts, offloading or concurrency.

Tone: mixedAMD hardware
generation:
13.7 tokens/s
quant:
ternary (AIM-W)

User reports BitNet b1.58 2B at 13.68 tok/s on a Radeon Pro W7900 with a self-built C inference stack, AimStack. The GPU backend loads the same 0.87GB ternary file, dequantizes weights on load and runs matmul in fp32; 256 of 256 greedy tokens matched the C engine bit-for-bit. The C engine runs the full model on a 2018 dual-core 8GB laptop at 3.20 tok/s with two workers, and its SIMD path measured 85.53 tok/s. BF16 is 2.4x faster at 32.96 tok/s but flips argmax on near-tied logits, so only FP32 passes validation. The GPU path gets the 78% memory saving but not yet a compute advantage; the UI server drives only the CPU C engine.

Nov 18, 2026
Tone: positiveAMD hardware
generation:
95.0 tokens/s
quant:
Q8 (GGUF)
agentictool-use

User reports LFM2.5-24B-A2B at ~95 t/s sustained on an AMD Radeon Pro W7900. Setup is llama.cpp with ROCm, Q8 quant, full 128k context, taking 30GB VRAM. User swapped it in for Ornith 1.5-35B in Pi Agent workflows and says it holds up well for mechanical and long-running recursive tasks; insight testing in council mode is pending.

Oct 16, 2026
AMD hardware
generation:
54.9 tokens/s
prompt processing (prefill):
982.2 tokens/s
quant:
Q8_0 (GGUF)

User asks whether Qwen3.8 27B Q8_0 numbers on a W7900 are reasonable. Table gives mean prefill 982.2 tok/s, mean generation 54.9 tok/s and mean server TTFT 2.27 s at depth 0, with 32K and 128K rows kept in features. No engine, context length or KV cache quant stated.

Oct 11, 2026
AMD hardware
generation:
48.7 tokens/s
prompt processing (prefill):
4315.7 tokens/s
quant:
F16 (GGUF)

User reports RWKV7 2.9B at 48.71 t/s generation and 4315.67 t/s prompt processing on an AMD Radeon Pro W7900. Setup is llama.cpp with F16 weights and ROCm backend, 99 GPU layers, 512-token prompt and 128-token generation. Q8_0 on ROCm reaches 58.59 t/s generation and 4033.24 t/s prompt; Vulkan backend gives 39.49 t/s (F16) and 45.21 t/s (Q8_0) generation.

Oct 8, 2026
Tone: mixedAMD hardware
generation:
63.5 tokens/s
prompt processing (prefill):
1119.0 tokens/s
quant:
UD-Q4_K_XL (GGUF)
codingcreative-writing

User reports Qwen3.8-Flash-Next at 63.5 t/s decode (code) and 1119 t/s prefill at 2.6K context on a Radeon Pro W7900 eGPU plus Strix Halo iGPU hybrid setup. Setup is llama-halo-hybrid fork with UD-Q4_K_XL quant at 128K context, MTP head enabled, dense layers on the W7900 and routed experts on the iGPU. Hybrid beats gufo iGPU-only (44.7 t/s decode code, 903 t/s prefill at 2.6K) and leaves 43-52 GiB RAM free versus 14-30. GLM-5.3-Flash UD-IQ3_XXS does 25 t/s (31.6 with MTP) and DS-V4-Flash UD-IQ3_XXS does 22.5 t/s. W7900 runs 100-104 C junction on long prefills with a locked 241 W power cap. One run each, no quality testing.

Oct 6, 2026
Showing 1–5 of 5
Page 1 of 1

Community benchmarks snapshot

Records by GPU

AMD Strix Halo 128GB348NVIDIA RTX 3090271NVIDIA RTX 5090197NVIDIA DGX Spark99NVIDIA RTX 3060 12GB96NVIDIA RTX 5060 Ti 16GB88NVIDIA RTX Pro 6000 Blackwell81AMD Radeon AI PRO R9700 32GB81NVIDIA RTX 5070 Ti77NVIDIA RTX 409068

Records by model

3266 total
Qwen3.81567
Qwen3.6334
Gemma 4185
DeepSeek V4 Flash150
Qwen3.5145
DeepSeek V4.1 Flash103
other782

Records by engine

2308 total
llama.cpp1134
Strata234
vLLM227
MLX107
Ollama99
other507

Use cases

coding 937agentic 640long-context 442tool-use 281vision 174summarization 81math 77creative-writing 58multilingual 35rp 22text-generation 9reasoning 3
coding937agentic640long-context442tool-use281vision174summarization81math77creative-writing58

Median generation t/s by GPU

On Qwen3.8 27B at 4-bit, plain single-GPU runs. Full ranking

RTX 509080RTX Pro 600067RTX 5070 Ti55M5 Ultra 256GB50RTX 409040RX 7900 XTX40RTX 309038RX 7800 XT 16GB35RTX 5090 Laptop 24GB34RX 7900 XT32

Prompt processing, same model

M5 Ultra 256GB1800RTX 40901949RX 7900 XTX640RTX 3090561RX 7900 XT800M3 Ultra 512GB372V100 32GB941Radeon AI PRO R9700 32GB1287RTX 3080 20GB997Arc Pro B70512

Reports by model size

Qwen3.8 27B819Qwen3.8 125B · 6B active676Qwen3.6 35B · 3B active242DeepSeek V4 Flash 284B · 13B active133DeepSeek V4.1 Flash 552B · 16B active92Qwen3.6 27B81Gemma 4 26B · 4B active61GLM-5.3 320B · 18B active57

Quants

Q4_K_M395UD-Q4_K_XL189NVFP4174Q8_01194-bit106IQ4_XS86IQ3_XXS85IQ3_S84Q4_K_XL83Q480