llamaperf

Local model performance on your hardware

Find which open-weight LLMs fit in your GPU or Mac and compare the speeds people report on setups like yours.

What runs on your hardware?

Pick your GPU or Mac, then read the speeds people reported on it, or estimate which models fit and how fast they run.

Free to use. No account needed. Memory estimates and community measurements are labelled separately.

Looking for a particular model?

Search for a model to see its reported speeds across GPUs and Macs.

Local model performance reports from the community

These are individual setups, not a controlled benchmark. Compare GPU count, quantization, context and offloading before comparing speeds. How to read a report →

GPU: NVIDIA CMP 170HX 64GB (unlocked)
Compare setup details

Exact recorded values. Context may be a configured limit; matching filters does not establish identical prompts, offloading or concurrency.

Tone: positiveNVIDIA hardware
generation:
55-65 tokens/s
prompt processing (prefill):
900-1100 tokens/s
quant:
IQ2_XXS GSQ-RCO-GGUF (GGUF)

User reports DeepSeek V4.1 Flash running at 55-65 t/s decode and 900-1100 t/s prefill on a single unlocked CMP 170HX 64GB at 150W. Setup is Project Maya with an IQ2_XXS GSQ-RCO-GGUF (2.0 bpw) quant. Both figures are given as ranges. The user plans to retry at 250-300W once a server shroud arrives.

Nov 2, 2026
Tone: mixedNVIDIA hardware
generation:
25-33 tokens/s
prompt processing (prefill):
900-1100 tokens/s
quant:
IQ2_XXS (GGUF)

User reports DeepSeek V4.1 Flash at around 25-33 t/s decode and 900-1100 t/s prefill at 128-256K context on an unlocked CMP 170HX 64GB, with prefill dropping to 520 t/s at 1M context. Setup is Strata (Project Maya) with an IQ2_XXS GSQ-RCO GGUF at 2.0 bpw, on a Dell R740 server with 256GB DDR4-2133 ECC RAM, GPU capped at 150W and running at 85C without a fan. User asks how others run this card with this model, and says the low-bit DeepSeek V4.1 Flash beats the Qwen3.8-Flash-Next Q6 they ran before on quality, though they avoid expert-preference optimizations that trade quality for speed.

Nov 1, 2026
NVIDIA hardware
generation:
38.0 tokens/s
prompt processing (prefill):
907.0 tokens/s
quant:
Q4_K_M (GGUF)

User reports Qwen3.6 27B at 38 t/s generation on an unlocked NVIDIA CMP 170HX 64GB. Setup is llama.cpp with Q4_K_M and flash attention, full GPU layer offload, zero context depth. Prompt processing measured 907 t/s at pp512 and 888 t/s at pp8192, five repetitions each; the model file is about 16GB.

Oct 14, 2026
Tone: positiveNVIDIA hardware
generation:
100.0 tokens/s
quant:
Q4_K_XL (GGUF)
agentictool-use

User reports Qwen3.8 Flash-Next at 100 tok/s on a CMP 170HX 64GB and one RTX 3090, with system RAM offload. Setup is Strata with Q4_K_XL weights; the run used two GPUs plus system RAM, and the user notes about a quarter of the results came from Strata .38, an eighth from .39, and the rest from .40. Before Strata, the same model ran on three GPUs (CMP 170HX and two RTX 3090s) under Unsloth Studio at 39 tok/s with 84 s average TTFT; the Strata run cut average TTFT to 15.7 s and total LLM time from 5,795m 9s to 1,228m 53s.

Oct 9, 2026
Tone: positiveNVIDIA hardware
generation:
95.6 tokens/s
prompt processing (prefill):
1529.0 tokens/s
quant:
3.05bpw (EXL3)
kv:
Q8
codingagenticlong-context

User reports GLM-5.3-Flash at 95.6 t/s decode and 1,529 t/s prefill on an 8K input, running on two CMP 170HX 64GB cards. Setup is ExLlamaV3 1.5.4 with an EXL3 3.05bpw target, DFlash2 EXL3 6bpw K7 drafter and Q8 KV cache, target weights fully resident in HBM at about 116.6GiB across the two cards. Decode stays near 90 t/s out to a 385K input (90.1 t/s, 1,535 t/s prefill); DFlash acceptance was around 87%, against roughly 60 t/s on the older MTP d2 path. The user also compared a Qwen3.8-Flash-Next AWQ INT4 + FP8 PLE vLLM setup on the same machine, which reached 129.5 t/s decode and 5,513 t/s prefill at 8K, but GLM finished three one-shot coding tasks sooner (4m40s vs 5m30s, 10m05s vs 14m40s, 4m40s vs 24m10s).

Oct 8, 2026
NVIDIA hardware
generation:
117.0 tokens/s
prompt processing (prefill):
6066.0 tokens/s
kv:
fp8

User reports DeepSeek-V4.1-Flash at 117 tok/s decode on one stream at 128k context and 6,066 tok/s prefill at 105k tokens on eight 64 GB CMP 170HX mining cards. Setup is vLLM with fp8 KV cache, PP=8, DSpark speculative decoding with 5 draft tokens, Engram tables in pinned host RAM, and 1M context. Decode drops to 96 tok/s at 512k. Eight concurrent streams reach 532 tok/s aggregate (66 per stream) at 105k. KV pool holds 6.17M tokens. One card was capped to 180 W after PCIe bus drops.

Oct 5, 2026
Tone: positiveNVIDIA hardware
generation:
101.3 tokens/s
prompt processing (prefill):
3191.0 tokens/s
quant:
W4A16 (W4A16)
kv:
BF16
codinglong-contexttool-usemultilingual

User reports Qwen3.8-Flash-Next at 101.3 t/s decode on Japanese prose and 170.0 t/s on code, single stream, on two NVIDIA CMP 170HX cards. Setup is vLLM with W4A16 weights, BF16 KV cache, MTP k=4 speculative decoding, 262,144-token context, and expert parallel across the two cards. The PLE n-gram table was converted to FP8 locally to fit in 92 GiB of host RAM. Aggregate throughput reaches 392 t/s at 4 concurrent requests. Prefill measures 3,191 t/s at 6,954 tokens and 3,261 t/s at 27,853 tokens. A 200,087-token prompt returned the planted code with 70.5 s time to first token and 69.0 t/s decode at depth. MTP is worth about 1.6x on prose and 2.6x on code. The user notes xhigh reasoning effort spent the whole budget without answering in 5 of 6 runs.

Sep 27, 2026
NVIDIA hardware
generation:
95.0 tokens/s
quant:
FP8
kv:
FP8
codinglong-context

User reports DeepSeek V4 Flash Vision at up to ~95 tok/s on 4× CMP 170HX 64GB (256GB aggregate HBM2e). Setup is vLLM with pipeline parallel 4, max_model_len 262144, FP8 KV cache, DSpark speculative decoding with 6 tokens, and max_num_seqs 2. Normal coding workload runs ~50–70 tok/s, strong speculative-decoding periods ~80–90 tok/s, peak ~95 tok/s. Cards sit around 56–62GB VRAM each at 50–60°C. Next tests planned for Qwen3.8 Flash Next and GLM 5.3.

Sep 24, 2026
NVIDIA hardware
generation:
116.0 tokens/s
prompt processing (prefill):
6800-7950 tokens/s
quant:
W4A16

User reports Qwen3-Next-80B at 116 t/s single-stream decode on a single NVIDIA CMP 170HX with 64 GB HBM2e, at a 150 W cap. Setup is vLLM 0.27.1 with W4A16 weights (40.9 GB), torch 2.13.0+cu130, CUDA 13.0, Ubuntu 26.04 LTS, on an AMD Ryzen Threadripper PRO 3945WX with 128 GB DDR4 ECC. Prefill over about 8.9k tokens measured 6800 to 7950 t/s; power draw 137 to 145 W. Aggregate throughput at 8 concurrent requests was 352 t/s, saturating at 4 slots. Also measured on the same card for context: Ornith-1.5-35B FP8 at 122.5 t/s and Qwen3.8-27B W4A16 with DFlash2 at 127 t/s single-stream.

Sep 23, 2026
NVIDIA hardware
generation:
98.1 tokens/s
prompt processing (prefill):
5300.0 tokens/s
quant:
MXFP4+FP8

User reports DeepSeek-V4-Flash-0731 at 98.1 t/s single-stream decode on 4x CMP 170HX (64 GB HBM2e each), up from a 50.8 t/s baseline with DSpark speculative decoding. Setup is vLLM with pipeline parallelism, native MXFP4+FP8 weights (~155.4 GiB), context verified to 1,047,736 tokens. At 100k context single-stream decode is 38.8 t/s with 14.6 s time to first token. Aggregate throughput across 64 concurrent requests is 712.8 t/s with DSpark (472.0 t/s baseline), and 90.0 t/s at 100k context. Prefill across the 25k-77k context range is about 5,300 t/s.

Sep 23, 2026
Tone: positiveNVIDIA hardware
generation:
147.0 tokens/s
quant:
W4A16 (W4A16)
kv:
FP8
long-context

User reports Qwen3.8-27B at 147.0 t/s single-stream decode on one NVIDIA CMP 170HX 64GB, with 134.7 t/s at 4K, 100.1 t/s at 65K, ~90 t/s at 126K, and 64.9 t/s at 250K context. Setup is vLLM 0.27.1 with a W4A16 target, a DFlash2 W4A16 drafter, FP8 target KV, BF16 draft KV, a custom SM80 split-KV verifier, full CUDA Graph, 35 verifier segments / 140 CTAs, k=3 draft tokens, MAX_SEQS=1, and 1350 MHz / 180W. These are decode-only numbers, not end-to-end throughput including prefill. FP8 beat INT8 at long context (48.3 vs 115.4 ms/iter at 250K). Allowing 2 concurrent long requests made makespan and slowest-request throughput worse at both 126K and 250K.

Sep 20, 2026
Tone: mixedNVIDIA hardware
generation:
50.0 tokens/s
quant:
Q8
coding

User reports Qwen3.8 27B at around 50 t/s on a CMP 170HX, running at Q8 or BF16. The user finds the model hallucinates classes and APIs in 90%+ of answers to domain-specific enterprise Java questions, while DeepSeek and Microsoft Copilot produce working code 99% of the time. The user asks whether Qwen's agentic optimization is responsible and requests tips for a harness, system prompt, or skills to reduce hallucination.

Sep 18, 2026
Tone: positiveNVIDIA hardware
generation:
63.0 tokens/s
prompt processing (prefill):
1700.0 tokens/s
quant:
Q6_K
kv:
F16

User reports Qwen3.6-35B-A3B at 63 t/s on 4x CMP 170HX 8GB cards flashed to 64GB each, 256GB total. Setup uses MTP with little-MoE default. Generation reaches 110 t/s with MTP optimistic.

Sep 7, 2026
NVIDIA hardware
generation:
29.0 tokens/s
prompt processing (prefill):
450.0 tokens/s
quant:
Q4_K_XL

User reports testing multiple models on 4x CMP 170HX 64GB cards, including DeepSeek V4-Flash 0731 with a Q4_K_XL quant, 13B active MoE, and 1M context, plain no-spec. The cards are cut-down A100 mining cards with 64GB each, PCIe Gen2 x4, and no NVLink. Other models tested include gpt-oss-120B, Qwen3.6-35B-A3B, GLM-4.5-Air, and MiniMax-M2.7.

Sep 7, 2026
Tone: positiveNVIDIA hardware
prompt processing (prefill):
3468.0 tokens/s
quant:
FP8

User reports an unlocked CMP 170HX, A100 silicon with a firmware unlock, reaching 193 TFLOPS tensor throughput, up from 6.3 TFLOPS. In llama.cpp, pp512 rises from 599.6 to 3468 tok/s. Serving Qwen3.8-27B-FP8 under vLLM, one 170HX beats a 2x3090 tensor-parallel pair on prefill at 197W versus 454W.

Aug 28, 2026
Showing 1–15 of 15
Page 1 of 1

Community benchmarks snapshot

Records by GPU

AMD Strix Halo 128GB348NVIDIA RTX 3090271NVIDIA RTX 5090197NVIDIA DGX Spark99NVIDIA RTX 3060 12GB96NVIDIA RTX 5060 Ti 16GB88NVIDIA RTX Pro 6000 Blackwell81AMD Radeon AI PRO R9700 32GB81NVIDIA RTX 5070 Ti77NVIDIA RTX 409068

Records by model

3266 total
Qwen3.81567
Qwen3.6334
Gemma 4185
DeepSeek V4 Flash150
Qwen3.5145
DeepSeek V4.1 Flash103
other782

Records by engine

2308 total
llama.cpp1134
Strata234
vLLM227
MLX107
Ollama99
other507

Use cases

coding 937agentic 640long-context 442tool-use 281vision 174summarization 81math 77creative-writing 58multilingual 35rp 22text-generation 9reasoning 3
coding937agentic640long-context442tool-use281vision174summarization81math77creative-writing58

Median generation t/s by GPU

On Qwen3.8 27B at 4-bit, plain single-GPU runs. Full ranking

RTX 509080RTX Pro 600067RTX 5070 Ti55M5 Ultra 256GB50RTX 409040RX 7900 XTX40RTX 309038RX 7800 XT 16GB35RTX 5090 Laptop 24GB34RX 7900 XT32

Prompt processing, same model

M5 Ultra 256GB1800RTX 40901949RX 7900 XTX640RTX 3090561RX 7900 XT800M3 Ultra 512GB372V100 32GB941Radeon AI PRO R9700 32GB1287RTX 3080 20GB997Arc Pro B70512

Reports by model size

Qwen3.8 27B819Qwen3.8 125B · 6B active676Qwen3.6 35B · 3B active242DeepSeek V4 Flash 284B · 13B active133DeepSeek V4.1 Flash 552B · 16B active92Qwen3.6 27B81Gemma 4 26B · 4B active61GLM-5.3 320B · 18B active57

Quants

Q4_K_M395UD-Q4_K_XL189NVFP4174Q8_01194-bit106IQ4_XS86IQ3_XXS85IQ3_S84Q4_K_XL83Q480