llamaperf

Local model performance on your hardware

Find which open-weight LLMs fit in your GPU or Mac and compare the speeds people report on setups like yours.

What runs on your hardware?

Pick your GPU or Mac, then read the speeds people reported on it, or estimate which models fit and how fast they run.

Free to use. No account needed. Memory estimates and community measurements are labelled separately.

Looking for a particular model?

Search for a model to see its reported speeds across GPUs and Macs.

Local model performance reports from the community

These are individual setups, not a controlled benchmark. Compare GPU count, quantization, context and offloading before comparing speeds. How to read a report →

GPU: NVIDIA RTX 5090 Laptop 24GB
Compare setup details

Exact recorded values. Context may be a configured limit; matching filters does not establish identical prompts, offloading or concurrency.

Tone: positiveNVIDIA hardware
generation:
107.0 tokens/s
prompt processing (prefill):
3706.5 tokens/s
quant:
Q4_K_M (GGUF)
kv:
Q8_0
codingagentic

User reports Qwen3-Coder-30B-A3B-Instruct at 107.0 tok/s generation and 3,706.5 tok/s prefill on an RTX 5090 Laptop 24GB in an MSI Raider A18. Setup is llama.cpp with CUDA, Q4_K_M GGUF (17.28 GiB), Q8_0 KV cache, FlashAttention on, 65,536-token context, all layers on the GPU, one slot. The figures come from a repository-analysis workload with an 8,638-token prompt and 371 generated tokens; the user notes the screenshot captures one strong interval, not an average across the audit.

Nov 17, 2026
Tone: positiveNVIDIA hardware
generation:
110.3 tokens/s
prompt processing (prefill):
3254.0 tokens/s
quant:
MLP-NVFP4 hybrid (GGUF)
agentic

User reports Muse Glimmer 30B at 110.3 tok/s greedy decode and 3,254 tok/s prefill on an RTX 5090 Laptop 24GB card. Setup is a patched llama.cpp with an MLP-NVFP4 hybrid quant, DFlash 2 speculative decoding at depth 15, and 128K context. A baseline llama.cpp build with unsloth UD-Q4_K_XL and Meta DFlash reached 80.6 tok/s decode and 1,430 tok/s prefill on the same 6,880-token prompt. A 100,554-token prompt prefilled at 2,367 tok/s, and an agent turn with a cached 46K prompt took 0.89 s. KLD vs Q8_0 rose to 0.053/0.030/0.034 (agent/English/Swedish) from 0.033/0.012/0.015.

Nov 16, 2026
NVIDIA hardware
generation:
30.0 tokens/s
quant:
NVFP4
codinglong-context

User reports about 30 t/s running Qwen3.8 Flash-Next in NVFP4 at 256k context on a laptop with a 16 GB Nvidia GPU and 96 GB DDR5 RAM. Setup details beyond the quant and context length are not stated. The figure is the user's current machine, cited while asking whether to buy a 24 GB VRAM / 128 GB RAM laptop or a 16 GB VRAM / 192 GB RAM laptop for local coding.

Nov 12, 2026
NVIDIA hardware
generation:
125.5 tokens/s
prompt processing (prefill):
2993.2 tokens/s
quant:
IQ2_XS (GGUF)

User reports Qwen3.8 Swift-1.5 Flash-Next at 125.5 tok/s generation and 2,993.2 tok/s prefill on an RTX 5090 Laptop 24 GB. Setup is PolyStrata with an IQ2_XS GGUF of about 63.5 GiB, configured for 131,072 tokens of context, with 128 GB of system RAM. The model is far larger than the card's 24 GB, so weights ran partly from system memory.

Oct 26, 2026
NVIDIA hardware
generation:
165.6 tokens/s
prompt processing (prefill):
9555.9 tokens/s
quant:
Q4_K_M (GGUF)

User reports 165.64 t/s generation and 9555.89 t/s prompt processing. Source evidence (verbatim): Source: https://blog.majid.info/llama-bench/golgotha.txt # NVIDIA GeForce RTX 5090 Laptop GPU 24GB: original Gemma4 E4B Q4_K_M llama-bench log First-hand author log linked from https://blog.majid.info/llama-bench/ (April 13, 2026 article). Measured hardware: NVIDIA GeForce RTX 5090 Laptop GPU 24GB. Gemma4 E4B Q4_K_M GGUF, llama.cpp e3f542d8 (4236), backend CUDA. pp512 9555.89 ±101.42 tokens/s; tg128 165.64 ±0.19 tokens/s. One device, ngl -1; 23983 MiB visible VRAM, not resident use. Laptop GPU, not desktop RTX5090. CPU and system RAM unknown. Command names --hf-repo unsloth/gemma-4-E4B-it-GGUF; exact tested quantization repository https://huggingface.co/unsloth/gemma-4-E4B-it-GGUF ; tested revision unknown. Log reports 4.62 GiB model size and 7.52B counted parameters; size is not resident memory. Context depth, KV quantization, measured power, driver version and explicit speculation/concurrency declarations unreported. Raw source: https://blog.majid.info/llama-bench/golgotha.txt .

Oct 14, 2026
Tone: negativeNVIDIA hardware
generation:
37.0 tokens/s
quant:
Q4_K_XL (GGUF)
codingagentic

User reports Qwen3.8-27B with vision at about 37 tok/s on an RTX 5090 Laptop 24GB with 64GB RAM. Setup is Q4_K_XL at 100K context, tested with the opencode and pi harnesses. User also ran a two-laptop cluster of RTX 5080 Laptop 16GB machines: Qwen3.8-27B Q6_K fine-tuned for coding agents at 67 to 77 tok/s, Qwen3.8-27B Q5_K_XL at 53 tok/s, Qwen3-Coder-Next Q3_K_XL at 48 tok/s, and Qwen3.8-Flash-Next IQ4_XS at 19 tok/s. On the single 5090 laptop, Qwen3-Coder-Next Q3_K_XL reached 55 tok/s and Qwen3.8-Flash-Next with vision IQ4_XS at 262K context reached 20 tok/s. User says none of the local models can complete even minor tasks, filling context without output or producing nonsense, and that hosted GPT-6 and Opus 5.5 handle the same tasks without issue.

Oct 8, 2026

Qwen3.8 27B

NVIDIA RTX 5090 Laptop 24GB · llama.cpp · 65,536 ctx

Tone: mixedNVIDIA hardware
generation:
30.0 tokens/s
quant:
Q4_K_M (GGUF)
kv:
8bit
coding

User reports Qwen3.8 27B at a steady 30 t/s on an RTX 5090 Laptop 24GB. Setup is llama.cpp with Q4_K_M and 8-bit KV cache at 65k context, using 21 GB of VRAM. The user also tested Unsloth UD_Q5_K_XL at 27 t/s, and NInfer models reaching 82.54 t/s (8-bit, 132k context), 92.4 t/s (NVFP4, 32k context), and 86.63 t/s (NVFP4, 4-bit, 64k context). The user says the laptop is too slow for coding and recommends a DGX Spark instead.

Oct 7, 2026
Tone: mixedNVIDIA hardware
generation:
3.6 tokens/s
quant:
INT4_SYM (OpenVINO IR)

User reports Qwen3-14B Heretic at 3.56 t/s on an Intel AI Boost NPU37XX in an Acer Predator Helios 16 AI laptop with 32 GB RAM and an RTX 5090 Laptop 24GB. Setup is OpenVINO GenAI with the machine-made-Fibre INT4_SYM OpenVINO IR export, group_size=-1, all_layers=true, loaded directly via openvino_genai.LLMPipeline on NPU with no conversion. Load took 33.52 s and 96 output tokens took 26.98 s; peak RAM was about 15.6 GB with a 9.7 GB working set. User also benchmarked Qwen3-8B INT4 SYM on the same NPU at 6.72 t/s with 6.15 s load, and found Qwen3-30B-A3B MoE impractical due to host memory pressure during OpenVINO preparation.

Sep 18, 2026
Showing 1–8 of 8
Page 1 of 1

Community benchmarks snapshot

Records by GPU

AMD Strix Halo 128GB348NVIDIA RTX 3090271NVIDIA RTX 5090197NVIDIA DGX Spark99NVIDIA RTX 3060 12GB96NVIDIA RTX 5060 Ti 16GB88NVIDIA RTX Pro 6000 Blackwell81AMD Radeon AI PRO R9700 32GB81NVIDIA RTX 5070 Ti77NVIDIA RTX 409068

Records by model

3266 total
Qwen3.81567
Qwen3.6334
Gemma 4185
DeepSeek V4 Flash150
Qwen3.5145
DeepSeek V4.1 Flash103
other782

Records by engine

2308 total
llama.cpp1134
Strata234
vLLM227
MLX107
Ollama99
other507

Use cases

coding 937agentic 640long-context 442tool-use 281vision 174summarization 81math 77creative-writing 58multilingual 35rp 22text-generation 9reasoning 3
coding937agentic640long-context442tool-use281vision174summarization81math77creative-writing58

Median generation t/s by GPU

On Qwen3.8 27B at 4-bit, plain single-GPU runs. Full ranking

RTX 509080RTX Pro 600067RTX 5070 Ti55M5 Ultra 256GB50RTX 409040RX 7900 XTX40RTX 309038RX 7800 XT 16GB35RTX 5090 Laptop 24GB34RX 7900 XT32

Prompt processing, same model

M5 Ultra 256GB1800RTX 40901949RX 7900 XTX640RTX 3090561RX 7900 XT800M3 Ultra 512GB372V100 32GB941Radeon AI PRO R9700 32GB1287RTX 3080 20GB997Arc Pro B70512

Reports by model size

Qwen3.8 27B819Qwen3.8 125B · 6B active676Qwen3.6 35B · 3B active242DeepSeek V4 Flash 284B · 13B active133DeepSeek V4.1 Flash 552B · 16B active92Qwen3.6 27B81Gemma 4 26B · 4B active61GLM-5.3 320B · 18B active57

Quants

Q4_K_M395UD-Q4_K_XL189NVFP4174Q8_01194-bit106IQ4_XS86IQ3_XXS85IQ3_S84Q4_K_XL83Q480