llamaperf

Local model performance on your hardware

Find which open-weight LLMs fit in your GPU or Mac and compare the speeds people report on setups like yours.

What runs on your hardware?

Pick your GPU or Mac, then read the speeds people reported on it, or estimate which models fit and how fast they run.

Free to use. No account needed. Memory estimates and community measurements are labelled separately.

Looking for a particular model?

Search for a model to see its reported speeds across GPUs and Macs.

Local model performance reports from the community

These are individual setups, not a controlled benchmark. Compare GPU count, quantization, context and offloading before comparing speeds. How to read a report →

GPU: NVIDIA RTX 4070 Super
Compare setup details

Exact recorded values. Context may be a configured limit; matching filters does not establish identical prompts, offloading or concurrency.

Tone: mixedNVIDIA hardware
generation:
60-70 tokens/s
quant:
IQ3_XXS (GGUF)
coding

User compares a local Qwen3.8 Flash-Next IQ3_XXS run in Strata against Claude on a coding task, generating a playable first-person motorcycle browser game in HTML. The local run used an RTX 4070 Super 12GB plus 64GB DDR5, with almost 50GB of RAM in use and only about 7GB left for other tasks. Thinking was capped at 16K tokens because the model otherwise consumed nearly 64K of context. Decode was around 60-70 tok/s with peaks over 100 tok/s, and the task took 396 s plus 274 s over two requests, about 11 minutes. The user found the result not really playable and concludes local setups are not worthwhile for coding on personal PCs.

Nov 4, 2026
Tone: positiveNVIDIA hardware
generation:
50-70 tokens/s
quant:
IQ3_XXS (GGUF)
tool-useagentic

User reports Qwen3.8-Flash-Next IQ3_XXS running at 50-70 tok/s at around 6k context on an RTX 4070 Super with 48GB DDR4-3200 system RAM, 60GB total memory. Setup is Strata with a 3-bit IQ3_XXS quant, with weights partly in system RAM. The speed is given only as a range. User says tool calls and web searches work well and the model does not seem degraded despite the 3-bit quant, but notes annoying refusals and a permissions quirk.

Oct 29, 2026
NVIDIA hardware
generation:
33.2 tokens/s
prompt processing (prefill):
947.0 tokens/s
quant:
GGUF (GGUF)
kv:
Q4

User reports running Qwen3.8-Flash-Next (125B MoE) on an RTX 4070 Super 12GB with 32GB system RAM, achieving 33.2 tok/s decode and 947 tok/s prefill at 8K context with the Gyro-M variant. Setup uses Strata with local source patches, GGUF weights, Q4 KV cache with 32K tokens GPU-resident, and MTP disabled. Experts are tiered across VRAM, 16 GiB of RAM, and SSD streaming. Gyro-S variant reaches 40.8 tok/s decode and 879 tok/s prefill at 8K, with vision support. Benchmarks cover 8K to 256K context; decode averages 3 cached runs.

Oct 14, 2026
Tone: positiveNVIDIA hardware
generation:
40.8 tokens/s
prompt processing (prefill):
879.0 tokens/s
kv:
Q4
agenticvisionlong-context

User reports running Qwen3.8 Flash-Next (125B MoE) on an RTX 4070 Super 12GB with 32GB system RAM, reaching 40.8 t/s decode and 879 t/s prefill at 8K context for the Gyro-S variant. Setup is upstream Strata with source patches, Q4 KV cache with 32K tokens GPU-resident, 256K context window, experts tiered across VRAM, 16 GiB in RAM and the rest streamed from SSD, MTP off. Gyro-M at 8K measured 33.2 t/s decode and 947 t/s prefill; Gyro-S is faster across the sweep, up to 41.9 t/s decode at 128K. Raising prefill chunk size from 7,168 to 8,192 tokens gave an 8-15% prefill gain. Vision and MTP both reduced speed and were left off Gyro-M.

Oct 14, 2026
Tone: mixedNVIDIA hardware
generation:
30-38 tokens/s
prompt processing (prefill):
950-1800 tokens/s

User reports Qwen3.8 Flash-Next running at 30-38 tok/s output and 950-1800 tok/s prefill on an RTX 4070 Super 12GB with 32GB system RAM. Setup is Strata with 256K context, MTP and vision off, model partially offloaded to system RAM. User notes speeds depend heavily on CPU, SSD, RAM speed and GPU bandwidth, and that 16GB VRAM may allow MTP.

Oct 14, 2026
Tone: positiveNVIDIA hardware
generation:
140.0 tokens/s
prompt processing (prefill):
2800-3300 tokens/s
quant:
Q4_K_M (GGUF)

User reports Qwen3.6-35B-A3B at 140 t/s decode around 8K context on an RTX 4070 Super 12GB. Setup is Polystrata with a HauhauCS uncensored Aggressive TTQ Q4_K_M GGUF and MTP enabled, on 32GB DDR5-6000 RAM under Windows 11. Prompt processing was 2,800-3,300 tok/s, and decode fell to 100 tok/s around 32K context. The user compares against llama.cpp at 2600 tok/s prefill and 77 tok/s decode, a roughly 1.3x prefill and 82% faster decode gain.

Oct 11, 2026
Tone: negativeNVIDIA hardware
generation:
12.0 tokens/s
codingtool-use

User reports Qwen3.8 Flash-Next at 12 t/s on an RTX 4070 Super with 80GB DDR4. Setup is Unsloth desktop, but the model errors out on anything beyond simple questions and refuses coding tasks with a message format error. The user also tried Qwen 27B in Q2 at 3 t/s, Qwen 3.6 35B at 20 t/s, and Ornith, which thinks for 30 minutes before erroring. They mention using LM Studio, Ollama, Open WebUI, and llama.cpp in the browser, and note that Qwen 27B with Q4 KV cache at 8K context gives 6 t/s.

Oct 8, 2026

Qwen3.8 27B

NVIDIA RTX 4070 Super · llama.cpp · 64,000 ctx

Tone: positiveNVIDIA hardware
generation:
18.0 tokens/s
prompt processing (prefill):
500-600 tokens/s
quant:
IQ3_S (GGUF)
kv:
q4_0

User reports Qwen3.8-27B at ~18 tok/s decode and ~500-600 tok/s prefill at 64k context on an RTX 4070 Super 12GB with 32GB system RAM. Setup is stock llama.cpp with an ISTA-DASLab GSQ-RCO IQ3_S quant (~11GB), q4_0 KV cache, -ngl 58 and token_embd offloaded to CPU. Only 16 of 64 layers need KV cache, so 64k context takes about 1.1GB instead of 4GB at f16. The same recipe works with 0bserverx' Qwen3.8-27B-Heretic-GSQ-RCO IQ3_S quant at about a 7% speed loss. User calls GSQ-RCO the best Qwen3.8-27B quant tested and says it performs close to full precision.

Oct 7, 2026

Qwen3.6 35B (3B active)

NVIDIA RTX 4070 Super · ik_llama.cpp · 131,072 ctx

Tone: positiveNVIDIA hardware
generation:
110.2 tokens/s
quant:
IQ4_XS-4.19bpw (GGUF)
kv:
Q8
mtp (multi-token prediction):
on
codingsummarizationmath

User benchmarks llama.cpp at 89.76 t/s against ik_llama.cpp at 110.24 t/s on Qwen3.6-35B-A3B with MTP. Setup is the IQ4_XS quant on a Ryzen 7 9700X running CachyOS, with the GPU used as a secondary card and the iGPU for display. The ik_llama.cpp run is a 23% speed increase.

May 21, 2026
Showing 1–9 of 9
Page 1 of 1

Community benchmarks snapshot

Records by GPU

AMD Strix Halo 128GB348NVIDIA RTX 3090271NVIDIA RTX 5090197NVIDIA DGX Spark99NVIDIA RTX 3060 12GB96NVIDIA RTX 5060 Ti 16GB88NVIDIA RTX Pro 6000 Blackwell81AMD Radeon AI PRO R9700 32GB81NVIDIA RTX 5070 Ti77NVIDIA RTX 409068

Records by model

3266 total
Qwen3.81567
Qwen3.6334
Gemma 4185
DeepSeek V4 Flash150
Qwen3.5145
DeepSeek V4.1 Flash103
other782

Records by engine

2308 total
llama.cpp1134
Strata234
vLLM227
MLX107
Ollama99
other507

Use cases

coding 937agentic 640long-context 442tool-use 281vision 174summarization 81math 77creative-writing 58multilingual 35rp 22text-generation 9reasoning 3
coding937agentic640long-context442tool-use281vision174summarization81math77creative-writing58

Median generation t/s by GPU

On Qwen3.8 27B at 4-bit, plain single-GPU runs. Full ranking

RTX 509080RTX Pro 600067RTX 5070 Ti55M5 Ultra 256GB50RTX 409040RX 7900 XTX40RTX 309038RX 7800 XT 16GB35RTX 5090 Laptop 24GB34RX 7900 XT32

Prompt processing, same model

M5 Ultra 256GB1800RTX 40901949RX 7900 XTX640RTX 3090561RX 7900 XT800M3 Ultra 512GB372V100 32GB941Radeon AI PRO R9700 32GB1287RTX 3080 20GB997Arc Pro B70512

Reports by model size

Qwen3.8 27B819Qwen3.8 125B · 6B active676Qwen3.6 35B · 3B active242DeepSeek V4 Flash 284B · 13B active133DeepSeek V4.1 Flash 552B · 16B active92Qwen3.6 27B81Gemma 4 26B · 4B active61GLM-5.3 320B · 18B active57

Quants

Q4_K_M395UD-Q4_K_XL189NVFP4174Q8_01194-bit106IQ4_XS86IQ3_XXS85IQ3_S84Q4_K_XL83Q480