llamaperf

Local model performance on your hardware

Find which open-weight LLMs fit in your GPU or Mac and compare the speeds people report on setups like yours.

What runs on your hardware?

Pick your GPU or Mac, then read the speeds people reported on it, or estimate which models fit and how fast they run.

Free to use. No account needed. Memory estimates and community measurements are labelled separately.

Looking for a particular model?

Search for a model to see its reported speeds across GPUs and Macs.

Local model performance reports from the community

These are individual setups, not a controlled benchmark. Compare GPU count, quantization, context and offloading before comparing speeds. How to read a report →

GPU: NVIDIA Tesla P100 16GB
Compare setup details

Exact recorded values. Context may be a configured limit; matching filters does not establish identical prompts, offloading or concurrency.

Unknown family

2× NVIDIA Tesla P100 16GB

Tone: mixedNVIDIA hardware
generation:
38.0 tokens/s

User reports about 38 t/s average generation on 2x P100, with 60+ t/s on some tasks when MTP is enabled. Prefill is about 200 t/s, which the user calls painfully slow; the system draws 500W under load. The user says it works but is basically unusable for anything interactive and asks whether prefill can be sped up on this old hardware.

Nov 21, 2026

Qwen3.8 27B

2× NVIDIA Tesla P100 16GB · llama.cpp · 262,144 ctx

NVIDIA hardware
generation:
45.0 tokens/s
quant:
UD-Q5_K_XL (GGUF)
agentictool-uselong-contextmultilingual

User reports Qwen3.8-27B at about 45 tok/s in English and about 32 tok/s in Polish on 2x Tesla P100 16GB. Setup is llama.cpp with UD-Q5_K_XL at 262k context, row/tensor split with MTP, on an X79 platform (i7-3930K, ~44 GB DDR3) under Proxmox with GPU passthrough. The P100s serve as the main LLM backend for an agent with long prompts and many tool calls; a GTX 1080 handles vision, Whisper, embeddings and image generation. The user asks whether to replace the 1080 with a Titan V or 2080 Ti, and reports a Quadro GP100 was not detected at all on the board.

Nov 10, 2026
Tone: positiveNVIDIA hardware
generation:
30.0 tokens/s
agentictool-use

User reports about 30 tok/s with the Swift-1.5 version of Qwen3.8 Flash-Next on a Tesla P100 16GB in a PCIe3 x16 slot, after switching from llama.cpp to Strata. Setup is Strata with 64GB of ECC DDR4 at 2100 and a 131072-token context; the logs show expert cache hits of 63.5% to 77.8% with routed experts going over PCIe, so part of the model ran from system RAM. Individual runs in the pasted logs range from 28.7 to 33.3 tok/s on prompts of roughly 78k tokens, which were read in 2 to 3 s. The user says it serves their Hermes agent and calls it impeccable for a local LLM.

Oct 25, 2026
Tone: positiveNVIDIA hardware
generation:
33.3 tokens/s
prompt processing (prefill):
406.9 tokens/s
quant:
Q4_0 (GGUF)

User reports Qwen3.8-27B Q4_0 at 33.28 t/s decode and 406.9 t/s prefill on 2x Tesla P100 16GB with patched llama.cpp, up from 23.43 t/s decode and 233.6 t/s prefill on stock. Setup is llama.cpp v0.7.0 with 41 sm_60 patches, tensor split across two P100s, 5 interleaved rounds per config. The patch set also gives 77.6 t/s decode on Ornith-1.0-35B-A3B UD-Q4_K_M with tensor split, and layer split gives 20.68 t/s decode on the 27B. Tested only on 2x P100; some patches are generic for all cards.

Oct 16, 2026
Tone: mixedNVIDIA hardware
generation:
10.0 tokens/s
quant:
Q3_K_M (GGUF)
creative-writing

User reports Qwen3.8 27B at 10.01 t/s on a Tesla P100 16GB. Setup uses the Q3_K_M GGUF quant with a 2,180-token prompt that took 3 minutes 37 seconds. The user asks whether this speed is normal and whether a 14B model would be faster, noting the P100 is old hardware.

Oct 12, 2026
Tone: mixedNVIDIA hardware
generation:
5.8 tokens/s
prompt processing (prefill):
56.1 tokens/s
quant:
IQ3_XXS (GGUF)
codingagentictool-use

User reports Qwen3.8-Flash-Next at 5.84 t/s generation and 56.10 t/s prompt processing on an Nvidia P100 16GB. Setup is llama.cpp with IQ3_XXS quant and 80K context, running on an HP Z440 workstation with Intel Xeon E5-1650 v3 and 64GB RAM. The model is a 125B MoE with 6B active parameters, partially offloaded to CPU. User notes it is slow but works for non-time-critical tasks like email sorting and calendar management.

Oct 12, 2026
Tone: mixedNVIDIA hardware
generation:
25.0 tokens/s
quant:
Q6
coding

User reports Qwen3.8 35B at Q6 running at 25 t/s with about 128k context on 2x Tesla P100 16GB. User says the model cheats heavily on their personal benchmark (building a clock in Lua for Conky) and still has trouble getting it working. User asks whether their prompting is the problem or if they are jaded by frontier models, and whether to have Qwen push work to Opus/Sol.

Oct 12, 2026
Tone: positiveNVIDIA hardware
generation:
31.2 tokens/s
prompt processing (prefill):
326.0 tokens/s

User reports Qwen3.8 Flash-Next at 31.2 t/s decode on a single Tesla P100 16GB with PXA v3.1 Overdrive mode, using a 32 GB quant. Setup is PXA with a 32 GB quant, experts cached on card and rest streamed, single request. Overdrive off gives 20.0 t/s; two P100s reach 38.2 t/s, one V100 35.4 t/s, two V100s 55.7 t/s. Against Strata on the same P100, prose decode 31.1 vs 27.0 t/s, code 34.3 vs 34.1, prompt processing 4k 326 vs 226 t/s and 8k 338 vs 246 t/s.

Oct 11, 2026
NVIDIA hardware
generation:
24.5 tokens/s
quant:
Q4_K_XL (GGUF)
kv:
F16

User benchmarks llama.cpp CUDA patches on two Tesla P100-PCIE-16GB cards, reporting Qwen3.8-27B Q4_K_XL at 24.5 tok/s decode with -sm tensor, up from 22.3 tok/s stock. Setup is llama.cpp b10660 with patches 0001-0012, CUDA 12.4, F16 KV cache, tensor split across two cards. The README also reports Llama-3.1-8B Q4_K_M at 79.57 tok/s on one card (2.03x stock), Qwen3.8-27B pp8192 at 378.7 tok/s, and a real 74,919-token request at 20.92 tok/s with F16 KV versus 16.85 with q8_0. Four concurrent agents reach 20.3 tok/s each (72.9 aggregate).

Oct 9, 2026
Tone: positiveNVIDIA hardware
generation:
120-140 tokens/s
quant:
UD_Q4_K_XL (GGUF)
agentictool-use

User reports Qwen3.6 35B-A3B at 120-140 t/s generation on two Tesla P100s, with prefill around 1000 t/s at 0 context and 700 t/s at 13k context. Setup uses Unsloth's UD_Q4_K_XL quant; the engine is not named. The user praises the model for broad agentic tasks and notes it is fast enough to be conversational, but criticizes it for coding, persona holding, and hallucination. They also mention running Qwen3.8 Flash-Next at 30-40 t/s with Strata at Q3.

Oct 9, 2026
NVIDIA hardware
generation:
32.6 tokens/s
prompt processing (prefill):
493.0 tokens/s
quant:
Q6_K (GGUF)
kv:
q4_0

User reports Qwen3.8-27B Q6_K at 32.6 t/s decode (tg256, no MTP) on 2x Tesla P100 16GB with tensor split, versus 17.51 t/s upstream at the fork point. Setup is a llama.cpp fork with CUDA work for Pascal (sm_60), q4_0 KV cache, and fp16 math with fp32 accumulation. Prefill pp2048 at 0 context is 493 t/s versus ~250 t/s upstream. With MTP speculative decoding the fork reaches 54 t/s at 2k context and 29-35 t/s at 260k context. Prefill at 260k context is 123 t/s filling and 153 t/s for a question on a loaded context. Perplexity on the gate corpus at -c 4096 is 2.6101.

Oct 6, 2026
Tone: positiveNVIDIA hardware
generation:
110.0 tokens/s
quant:
Q4_0 (GGUF)
kv:
F16
codingagenticlong-context

User reports Nemotron-3.5-Lightning-30B-A3B at 110 tok/s writing code on two Tesla P100 16GB cards. Setup is llama.cpp b10970 with Q4_0 weights, F16 KV cache, tensor split across both cards, and the model's built-in MTP draft head at n-max 2. The same model reaches 95 tok/s on prose, 50 tok/s at 128k context, 39 tok/s at 256k, and 16 tok/s at 1M tokens. The study covers 545 speed measurements of 29 models from 2B to 122B parameters, all weights and KV cache in VRAM with no system RAM offload. A 119B MoE model runs 39 tok/s against 4.3 tok/s for a 70B dense model on the same cards. Tensor split makes dense models from 8B up 21-44% faster. A single P100 throttles to 906 MHz and loses 25% under sustained load, while two cards share the heat and lose 5.5%.

Oct 6, 2026

Qwen3.8 27B

2× NVIDIA Tesla P100 16GB · llama.cpp · 262,144 ctx

Tone: positiveNVIDIA hardware
generation:
50-60 tokens/s
prompt processing (prefill):
350.0 tokens/s
quant:
Q6_K (GGUF)
codingcreative-writing

User reports Qwen3.8 27B at 50-60 t/s generation and 350 t/s prefill on 2x Tesla P100 16GB with a custom llama.cpp fork. Setup is llama.cpp with Q6_K quant, 262144 context, batch 32768, ubatch 1024, and MTP speculative decoding. At 260k context, generation is 30-35 t/s and prefill is 110 t/s. GPUs are capped at 175W/250W each and run at 79C with minor thermal throttling; user estimates 5-10% higher numbers with better cooling.

Sep 27, 2026
Tone: positiveNVIDIA hardware
generation:
54-60 tokens/s
prompt processing (prefill):
440-500 tokens/s
quant:
UD_Q4_K_XL (GGUF)
kv:
16bit
codingcreative-writing

User reports Qwen3.6 35B A3B at 54-60 t/s generation (66-72 t/s on code) on a single Tesla P100 16GB, up from 30-35 t/s prose on an RX 6600 XT. Setup is llama.cpp with shinbunbun patches, UD_Q4_K_XL quant, 16-bit KV cache, MTP speculative decoding, and --n-cpu-moe 22, running 32k context. Prefill is 600 t/s at 0 ctx dropping to 440-500 t/s by 10k. The user notes the P100 is underrated for the price and has a second card coming for full offload.

Sep 27, 2026
Tone: mixedNVIDIA hardware
generation:
85.0 tokens/s
prompt processing (prefill):
300.0 tokens/s
quant:
Q4_K_XL (GGUF)
agentic

User reports Qwen3.6-35B-A3B at ~85 t/s generation and ~300 t/s prompt processing on 2x Tesla P100. Setup is llama.cpp with the Q4_K_XL GGUF, MTP speculative decoding, and community P100 patches that added about 50% decode; a single P100 on Q2_K_XL reached ~76 t/s generation and ~250 t/s prompt processing. Card count barely affects single-stream speed, and PCIe lane width (x16/x16, x16/x8, x8/x8) made no difference. The user corrects an earlier 3-card result that was invalidated by stuck 405 MHz core clocks.

Sep 24, 2026
Tone: positiveNVIDIA hardware
generation:
70.0 tokens/s
quant:
Q4_K_XL (GGUF)

User reports almost 70 t/s with Qwen3.6-35B-A3B Q4_K_XL on a budget build using three Nvidia Tesla P100 16GB GPUs. The GPUs cost $80 each and are split across three nodes to manage thermals; the motherboard required a patched BIOS to enable Above 4G Decoding. User is still testing and optimizing, and has published a GitHub repository for the build.

Sep 22, 2026
Tone: positiveNVIDIA hardware
generation:
40.0 tokens/s
quant:
Q6_K (GGUF)

User reports Qwen3.8 27B at 40+ t/s on two Tesla P100s, with a roughly 30 t/s average across long contexts and up to 55 t/s at 0 context. Setup is a custom llama.cpp fork with P100 kernel optimizations, Q6_K quant, both cards capped at 175W and communicating over PCIe gen 3. Speeds vary by about ±2 t/s; the user notes fp16 math saves 40% or more on prefill with negligible accuracy loss.

Sep 20, 2026
Showing 1–17 of 17
Page 1 of 1

Community benchmarks snapshot

Records by GPU

AMD Strix Halo 128GB348NVIDIA RTX 3090271NVIDIA RTX 5090197NVIDIA DGX Spark99NVIDIA RTX 3060 12GB96NVIDIA RTX 5060 Ti 16GB88NVIDIA RTX Pro 6000 Blackwell81AMD Radeon AI PRO R9700 32GB81NVIDIA RTX 5070 Ti77NVIDIA RTX 409068

Records by model

3266 total
Qwen3.81567
Qwen3.6334
Gemma 4185
DeepSeek V4 Flash150
Qwen3.5145
DeepSeek V4.1 Flash103
other782

Records by engine

2308 total
llama.cpp1134
Strata234
vLLM227
MLX107
Ollama99
other507

Use cases

coding 937agentic 640long-context 442tool-use 281vision 174summarization 81math 77creative-writing 58multilingual 35rp 22text-generation 9reasoning 3
coding937agentic640long-context442tool-use281vision174summarization81math77creative-writing58

Median generation t/s by GPU

On Qwen3.8 27B at 4-bit, plain single-GPU runs. Full ranking

RTX 509080RTX Pro 600067RTX 5070 Ti55M5 Ultra 256GB50RTX 409040RX 7900 XTX40RTX 309038RX 7800 XT 16GB35RTX 5090 Laptop 24GB34RX 7900 XT32

Prompt processing, same model

M5 Ultra 256GB1800RTX 40901949RX 7900 XTX640RTX 3090561RX 7900 XT800M3 Ultra 512GB372V100 32GB941Radeon AI PRO R9700 32GB1287RTX 3080 20GB997Arc Pro B70512

Reports by model size

Qwen3.8 27B819Qwen3.8 125B · 6B active676Qwen3.6 35B · 3B active242DeepSeek V4 Flash 284B · 13B active133DeepSeek V4.1 Flash 552B · 16B active92Qwen3.6 27B81Gemma 4 26B · 4B active61GLM-5.3 320B · 18B active57

Quants

Q4_K_M395UD-Q4_K_XL189NVFP4174Q8_01194-bit106IQ4_XS86IQ3_XXS85IQ3_S84Q4_K_XL83Q480