llamaperf

Local model performance on your hardware

Find which open-weight LLMs fit in your GPU or Mac and compare the speeds people report on setups like yours.

What runs on your hardware?

Pick your GPU or Mac, then read the speeds people reported on it, or estimate which models fit and how fast they run.

Free to use. No account needed. Memory estimates and community measurements are labelled separately.

Looking for a particular model?

Search for a model to see its reported speeds across GPUs and Macs.

Local model performance reports from the community

These are individual setups, not a controlled benchmark. Compare GPU count, quantization, context and offloading before comparing speeds. How to read a report →

Model: Qwen3-Next
Compare setup details

Exact recorded values. Context may be a configured limit; matching filters does not establish identical prompts, offloading or concurrency.

Qwen3-Next 80B (3B active)

Arc iGPU · Janas-LLM

generation:
26.4 tokens/s
prompt processing (prefill):
68.4 tokens/s

User reports Qwen3-Next-80B-A3B at 26.4 tok/s generation and 68.4 tok/s prompt processing on Janas-LLM, against 3.0 and 17.7 tok/s on llama.cpp. Setup is Janas-LLM streaming MoE experts from NVMe on a Core Ultra 9 185H laptop with 32 GB DDR5-5600 and an Arc iGPU; the 48.4 GB model is larger than the RAM. The same table gives Qwen3.6-35B-A3B at 28.0 tok/s reply and 136 tok/s prompt on Janas-LLM versus 12.1 and 81.8 on llama.cpp, and Qwen3-30B-A3B at 33.3 and 133 versus 15.0 and 122. The post announces Brebus, a Linux desktop app built on the engine.

Nov 16, 2026

Qwen3-Next 80B (3B active)

Intel Arc iGPU · Janas-LLM · 1,024 ctx

Tone: positive
generation:
26.4 tokens/s
prompt processing (prefill):
68.4 tokens/s

User reports Qwen3-Next-80B-A3B at 26.4 tok/s generation and 68.4 tok/s prompt processing on a laptop with an Intel Core Ultra 9 185H, 32 GB DDR5, integrated Arc GPU and NPU, and NVMe Gen4. Setup is the author's Janas-LLM engine in plain C, streaming MoE experts from NVMe with an expert cache in RAM; replies run on the CPU, prompts on the iGPU or NPU. The 48.4 GB model is larger than system RAM. llama.cpp on the same files reached 3.0 tok/s generation and 17.7 tok/s prompt on this model. The post also tables nine other models, MTP drafts giving up to 63% faster chat replies on Qwen3.8-Flash-Next, and 18.2 tok/s generation at 8,192 tokens on Qwen3-4B with an 8-bit KV cache.

Oct 29, 2026
NVIDIA hardware
generation:
67.5 tokens/s
quant:
Q4_K_M (GGUF)

User benchmarks Qwen3-Next-80B-A3B-Instruct Q4_K_M on an RTX 4070 Ti Super 16GB, reporting 67.54 t/s median decode on real text and 80.673 t/s on a repeated prompt. Setup is llama.cpp b11617 with -ncmoe 48, --moe-cache-mib 10752, 4096 context and a single-CCD CPU affinity mask; the 46.64GB model streams experts from system memory over PCIe. All figures are warm-cache, 435 prompt tokens and 256 generated tokens. A 512 MiB expert cache was 23.8% slower than no cache, and decode fell 70.5% at -ncmoe 44. Random token IDs decoded at 93.61 t/s, about 38% faster than real text.

Oct 19, 2026
Tone: mixedApple hardware
generation:
2.9 tokens/s
quant:
Q4_K_M (GGUF)

User reports Qwen3-Next 80B-A3B at 2.9 tok/s on an M4 Pro 24GB MacBook Pro, running llama.cpp's WebGPU backend in a Chrome tab with MoE experts paged in from disk. Setup is llama.cpp (ggml-webgpu) with a Q4_K_M GGUF, 48.4 GB file, 4.89 GB resident on the GPU, 48 s initial load from disk. The 80B reads about 359 MB from disk per token; the author calls it a proof of concept. OLMoE 1B-7B Q4_K_M reached 22.4 tok/s and LFM2 8B-A1B Q4_K_M 17.1 tok/s in the same recording. Paged and unpaged replies were identical; against llama.cpp on Metal 4 to 8 of 9 replies matched. Chrome or Edge 137+ only, tested on Apple silicon only.

Oct 14, 2026
Apple hardware
generation:
49.1 tokens/s
quant:
4-bit (MLX)

User reports 49.1 t/s generation. Source evidence (verbatim): Source: https://arxiv.org/html/2605.00519v1 # First-hand M3 Ultra 96GB Qwen3-Next 80B native MLX 4-bit generation measurement Original experimental paper Silicon Showdown, arXiv 2605.00519v1, May 2026. Selected Table 5 row: Apple M3 Ultra 96GB, 28 CPU cores, 60 GPU cores; installed unified memory 96GB. macOS 26 Tahoe, MLX v0.30.6. Qwen3-Next-80B-A3B native MLX 4-bit: generation 49.1 tokens/s, 0% CPU offload. Long prompt approximately 496 tokens; output 256 tokens, greedy decoding. Authors report arithmetic mean of three consecutive timed runs after two untimed warmups and fixed random seed. This is their published generation-phase measurement, not a speed derived from TTFT. Prompt-processing speed unreported; leave promptTps null. Context capacity, KV quantization, speculation, concurrency and resident allocation unreported. Verified base-model reference explicitly cited by the paper: https://huggingface.co/Qwen/Qwen3-Next-80B-A3B-Instruct . This is not the tested MLX quantization repository; exact quantized artifact and revision unknown. Do not infer disk size or resident memory. Table 2 names the 96GB machine, not the followed 256GB configuration. Other models, machines and Nvidia offload comparisons are separate runs. Source: https://arxiv.org/html/2605.00519v1 .

Oct 14, 2026
Tone: positiveNVIDIA hardware
generation:
94.5 tokens/s
quant:
Q4_K_M (GGUF)

User reports Qwen3-Next-80B-A3B at 94.5 tok/s decode on an RTX 3090 with 16 GB RAM, using a patched llama.cpp that substitutes missing experts instead of waiting for SSD reads. Setup is llama.cpp with Q4_K_M, 1/4 of experts in VRAM, rest read from NVMe at ~5.7 GB/s, 16 threads, about 15.7 GB VRAM used. Stock llama.cpp gave 31.8 tok/s in the same 16 GB case; the patch also reached 108.4 tok/s with plenty of RAM and 89 tok/s reading every miss from SSD. Perplexity was 1.6% higher than stock, GSM8K lost 1.8 points, and greedy generation ran 64-74 tok/s. The 16 GB case was simulated by locking RAM on a bigger machine.

Oct 6, 2026

Qwen3-Next 80B (3B active)

Unknown GPU · llama.cpp · 32,768 ctx

Tone: positive
generation:
19.1 tokens/s
prompt processing (prefill):
130.0 tokens/s
quant:
Q4_K_M (GGUF)
coding

User reports Qwen3-Next-80B-A3B at 130 t/s prefill and 19.1 t/s generation on a CPU-only Lenovo ThinkStation P5 with Intel Xeon w5-2555X and 256 GB DDR5 ECC. Setup is llama.cpp built with GGML_NATIVE=ON, Q4_K_M quant, 32k context, mlocked into RAM, no GPU offload. The user also benchmarks Qwen3-Coder-30B-A3B at 166 t/s prefill and 33.7 t/s generation, gpt-oss-120b at 98 t/s prefill and 19.5 t/s generation, and Qwen3-235B-A22B at 24.8 t/s prefill and 5.6 t/s generation. An NVIDIA T1000 8GB was tested and found to slow prefill versus CPU-only, so it was removed from inference. The user notes that -ub tuning varies per model and that Q8 quant of Qwen3-Next-80B scored 59/61 on a coding suite versus 55-57 for Q4_K_M.

Oct 3, 2026

Qwen3-Next Flash

6× BC-250 · llama.cpp · 100,000 ctx

Tone: positive
generation:
28.0 tokens/s
quant:
IQ2_XS (GGUF)

User reports Qwen3-Next Flash IQ2_XS at around 28 tok/s for short generation on a cluster of 4 BC-250 boards, dropping to 24 tok/s at 50k context with around 115 t/s prefill. Setup is llama.cpp with Vulkan and RPC over 1Gb Ethernet, 100k context, across 6 BC-250 ex-mining boards in an ASRock 4U12G case. The other two boards run Qwen3.6 35B Q4 at 60 tok/s with 100k context and 450 t/s prefill.

Oct 2, 2026
NVIDIA hardware
generation:
116.0 tokens/s
prompt processing (prefill):
6800-7950 tokens/s
quant:
W4A16

User reports Qwen3-Next-80B at 116 t/s single-stream decode on a single NVIDIA CMP 170HX with 64 GB HBM2e, at a 150 W cap. Setup is vLLM 0.27.1 with W4A16 weights (40.9 GB), torch 2.13.0+cu130, CUDA 13.0, Ubuntu 26.04 LTS, on an AMD Ryzen Threadripper PRO 3945WX with 128 GB DDR4 ECC. Prefill over about 8.9k tokens measured 6800 to 7950 t/s; power draw 137 to 145 W. Aggregate throughput at 8 concurrent requests was 352 t/s, saturating at 4 slots. Also measured on the same card for context: Ornith-1.5-35B FP8 at 122.5 t/s and Qwen3.8-27B W4A16 with DFlash2 at 127 t/s single-stream.

Sep 23, 2026
Showing 1–9 of 9
Page 1 of 1

Community benchmarks snapshot

Records by GPU

AMD Strix Halo 128GB348NVIDIA RTX 3090271NVIDIA RTX 5090197NVIDIA DGX Spark99NVIDIA RTX 3060 12GB96NVIDIA RTX 5060 Ti 16GB88NVIDIA RTX Pro 6000 Blackwell81AMD Radeon AI PRO R9700 32GB81NVIDIA RTX 5070 Ti77NVIDIA RTX 409068

Records by model

3266 total
Qwen3.81567
Qwen3.6334
Gemma 4185
DeepSeek V4 Flash150
Qwen3.5145
DeepSeek V4.1 Flash103
other782

Records by engine

2308 total
llama.cpp1134
Strata234
vLLM227
MLX107
Ollama99
other507

Use cases

coding 937agentic 640long-context 442tool-use 281vision 174summarization 81math 77creative-writing 58multilingual 35rp 22text-generation 9reasoning 3
coding937agentic640long-context442tool-use281vision174summarization81math77creative-writing58

Median generation t/s by GPU

On Qwen3.8 27B at 4-bit, plain single-GPU runs. Full ranking

RTX 509080RTX Pro 600067RTX 5070 Ti55M5 Ultra 256GB50RTX 409040RX 7900 XTX40RTX 309038RX 7800 XT 16GB35RTX 5090 Laptop 24GB34RX 7900 XT32

Prompt processing, same model

M5 Ultra 256GB1800RTX 40901949RX 7900 XTX640RTX 3090561RX 7900 XT800M3 Ultra 512GB372V100 32GB941Radeon AI PRO R9700 32GB1287RTX 3080 20GB997Arc Pro B70512

Reports by model size

Qwen3.8 27B819Qwen3.8 125B · 6B active676Qwen3.6 35B · 3B active242DeepSeek V4 Flash 284B · 13B active133DeepSeek V4.1 Flash 552B · 16B active92Qwen3.6 27B81Gemma 4 26B · 4B active61GLM-5.3 320B · 18B active57

Quants

Q4_K_M395UD-Q4_K_XL189NVFP4174Q8_01194-bit106IQ4_XS86IQ3_XXS85IQ3_S84Q4_K_XL83Q480