llamaperf

Local model performance on your hardware

Find which open-weight LLMs fit in your GPU or Mac and compare the speeds people report on setups like yours.

What runs on your hardware?

Pick your GPU or Mac, then read the speeds people reported on it, or estimate which models fit and how fast they run.

Free to use. No account needed. Memory estimates and community measurements are labelled separately.

Looking for a particular model?

Search for a model to see its reported speeds across GPUs and Macs.

Local model performance reports from the community

These are individual setups, not a controlled benchmark. Compare GPU count, quantization, context and offloading before comparing speeds. How to read a report →

GPU: NVIDIA RTX 3090
Compare setup details (1 active)

Exact recorded values. Context may be a configured limit; matching filters does not establish identical prompts, offloading or concurrency.

Tone: positiveNVIDIA hardware
generation:
72-86 tokens/s
quant:
Q6_K_XL (GGUF)
codingagenticlong-contexttool-use

User compares Qwen3.8 Flash-Next Q6_K_XL on Strata against several DeepSeek V4.1 Flash quants on 2x RTX 3090s capped at 300W, in a 3975WX/256GB DDR4 box. Qwen decodes at 72-86 tok/s and prefills at about 3,950 tok/s at 120K context; the DeepSeek runs decode at 23-62 tok/s depending on quant. On a 242-question accuracy benchmark Qwen Q6 scored 93.0%, DeepSeek EXL3 3.5bpw 90.1%, Maya-XS 89.7% and Maya-S 88.4%. Qwen Q6 finished six agent tasks in 14 minutes against 32 for Maya-XS and 81 for EXL3, and answered 5/4 of the 100K/180K long-document questions where Maya-XS got 1/0 and EXL3 0/0. DeepSeek models often exhausted the reply budget on thinking tokens; a thinking budget below the reply limit fixed it. The user notes one run per agent task and only 16 long-document questions, and that the DeepSeek setups drew 700-800W to finish 2-6x slower.

Nov 20, 2026
Tone: mixedNVIDIA hardware
generation:
38.0 tokens/s
quant:
Q4 (GGUF)
long-contextagentic

User reports Qwen3.8 Flash-Next at around 38 t/s in Q4 and 50 t/s in Q3 on an RTX 3090 with 64GB of system RAM. Setup is Strata at 256k context with max thinking settings, run for legal research. User asks why a 20-page document analysis takes at least 3 hours with 4 sub agents, and says Strata keeps using sub agents even when they are turned off by prompt. Lowering thinking to high/medium still takes at least 2 hours.

Nov 18, 2026
Tone: mixedNVIDIA hardware
generation:
130.0 tokens/s
quant:
IQ3_S (GGUF)

User reports Qwen3.8-Flash-Next at about 130 t/s on an RTX 3090 plus RTX 5080 (40 GB VRAM) with 32 GB system RAM. Setup is Strata with the standard IQ3_S GGUF. User says the model runs great but that the Unsloth UD_IQ3_XXS build of the same size stalls during quick tune, runs at only about 20 t/s without tuning, and reads more from the SSD.

Nov 17, 2026

Edge0 145B

NVIDIA RTX 3090 · Strata · 16,384 ctx

Tone: mixedNVIDIA hardware
generation:
48.6 tokens/s
prompt processing (prefill):
1481.2 tokens/s
quant:
q3k
kv:
int8
agentictool-use

User reports Edge0-145B at 48.6 t/s chat decode on a single RTX 3090, with a 16K prompt read at 1,481.2 t/s and agent decode at 52.8 t/s. Setup is Strata with the q3k quant, int8 KV cache, speculative decoding 4, 131K max context, and direct PLE I/O from disk; the FP8 PLE table alone is 51 GB. GPU peak was 23,929 of 24,576 MiB and the engine process held 43.4 GiB of system RAM, so the user does not know the memory floor yet. The agent figure is speed only: three temperature-0 runs gave three different tool calls, one invalid.

Nov 17, 2026
Tone: positiveNVIDIA hardware
generation:
80.4 tokens/s
prompt processing (prefill):
1036.0 tokens/s
quant:
UD IQ4_XS (GGUF)
long-contextagentic

User reports Qwen3.8-Flash-Next at a median 80.4 tok/s chat decode on an RTX 3090 with an RTX 2080 SUPER as helper card, over 3 runs (range 67.6-87.8, n=9). Setup is Strata with Unsloth UD IQ4_XS weights, 96GB DDR4 3600 system RAM. 4K prompt processing ran at 1,036 tok/s and 16K at 1,915 tok/s; agent session decode was 101.7 tok/s with a 69.4% cache hit. A 66,371-token needle test prefilled in about 34 s (~2,050 tok/s) and decoded at 76.1 tok/s with the full KV resident, retrieving the mid-context needle. The user pins the 3090 memory clock to avoid a driver freeze bug, idling at ~110 W instead of ~25 W, and saw 79°C peak under a 3-minute load.

Nov 16, 2026
NVIDIA hardware
generation:
139-144 tokens/s
quant:
UD-Q4_K_XL (GGUF)
kv:
int8
long-context

User benchmarks Qwen3.8-Flash-Next (UD-Q4_K_XL, Q8_0 n-gram table) on 4x RTX 3090 24 GB via Strata, reaching 139-144 tok/s on short prompts and 85-96 tok/s after 50K context with all four cards. Setup is Strata 0.2.13 with layer split, DFlash2 drafter, 256K context, int8 KV, one request at a time, on a Threadripper 2920X with 128 GB RAM; experts that miss VRAM are computed on the CPU. Three cards give 112-137 tok/s and two cards 90-95 tok/s; a fourth card mainly speeds prompt reading, which is twice as fast as on two or three cards. Q5_K_XL decodes 28% slower than Q4 at 0.55% lower perplexity. A cold 190K prompt takes 91 s, after which a short turn at 200K decodes at 78 tok/s. Ranges are two runs each.

Nov 12, 2026
Tone: positiveNVIDIA hardware
generation:
70-80 tokens/s
quant:
IQ3_XXS (GGUF)

User reports running Strata QFN (IQ3_XXS) on an RTX 3090 with 64GB of system RAM at around 70-80 t/s, and says it is their daily driver. The run uses both the GPU and system RAM, so the machine is fully occupied while serving. The figure is given only as a range. The user asks whether 70-80 t/s is realistic on a Strix Halo, which they are considering buying as a dedicated inference box for 3-3.5k EUR.

Nov 12, 2026
Tone: mixedNVIDIA hardware
generation:
41.0 tokens/s
quant:
IQ4

User reports Strata IQ4 Qwen3.8 Flash-Next decoding at 41 t/s on an RTX 3090 with a Xeon E5 and 96GB DDR4 ECC. Setup is Strata with IQ4 weights, used to edit a Qwen 0.6B model running fully on CPU in FP32, over a 39.8M-token run lasting 3h 12m. The edit steered the 0.6B model to answer a favourite-band question with Polyphia; the user calls the setup weird and questions whether models above 20B would benefit compared with traditional gradient training.

Nov 4, 2026
Tone: positiveNVIDIA hardware
generation:
50.0 tokens/s
quant:
UD-Q5_K_XL (GGUF)
kv:
8-bit
codingvisionlong-contextagentic

User reports Qwen 3.8 Flash Next at 50 t/s with UD-Q5_K_XL and 120 t/s with Gyro-M on an RTX 3090 24GB. Setup is Strata on Ubuntu 24.04 with 128K context and 8-bit KV cache, on a Dell T5810 with a Xeon 2699 v4 and 128 GB of ECC DDR4. The user asks which smarter model reaches at least 20 t/s, how to improve long-term context and memory, and what cost-efficient upgrade path to take up to mid-2027.

Nov 3, 2026
NVIDIA hardware
generation:
90-110 tokens/s
quant:
IQ3_S (GGUF)
codingagenticlong-context

User reports Qwen3.8 Flash-Next at 90-110 t/s generation with over 2000 t/s prefill on a single RTX 3090 24GB. Setup is Strata with an IQ3_S quant, on a Ryzen 9 5950X with 96 GB DDR4. The user has bought a second 3090 and asks whether to buy a motherboard with two full x16 slots or use an NVLink bridge, since the B550 board's second slot is x4 from the chipset.

Nov 3, 2026
Tone: mixedNVIDIA hardware
generation:
120-151 tokens/s
quant:
IQ2_XS (GGUF)
tool-useagentic

User reports Qwen3.8 Flash-Next served by Strata at IQ2_XS with 64k context on a single RTX 3090, running at 120-151 tok/s with the GPU at 53°C and no throttling. Setup is an Unraid box with an Intel i7-9700K and 64 GB DDR4, with Open WebUI calling Strata directly and via Hermes-Agent, plus Home Assistant's MCP server for tools; the fixed prompt was trimmed from about 9,500 to 4,560 tokens. The user asks why new chats are slow while follow-ups are fast: fresh-chat tasks take 8-26 s against 1-2 s for repeats, and Strata's log shows 4,552 of 4,557 prompt tokens reused from cache even in a new chat. Suspected causes are background jobs after the first reply, a redundant first tool call, and per-chat MCP tool setup, none confirmed.

Nov 3, 2026
Tone: mixedNVIDIA hardware
generation:
70.0 tokens/s
prompt processing (prefill):
2400.0 tokens/s
quant:
Q8_0 (GGUF)

User reports Qwen3.8 Flash-Next at Q8_0 running at about 70 t/s generation and about 2400 t/s prompt processing on four RTX 3090s (96GB VRAM) with 192GB of octa-channel 3200MHz system RAM. Setup is Strata (0.2.9-cuda13) with a 262144-token context, PARALLEL 1, vision on, the qwen-flash preset and a 65536 MiB conversation cache across 16 slots. User asks whether others still run Qwen on Strata with larger quants, whether they retune the config on every Strata update, and what their configs and performance look like, citing concerns that Strata updates sometimes trade quality for speed without an opt-in.

Nov 2, 2026
Tone: positiveNVIDIA hardware
generation:
94.2 tokens/s
quant:
Q8_0 (GGUF)
kv:
int8

User reports Qwen3.8-Flash-Next Q8_0 at 94.2 t/s single-stream on 2x RTX 3090, with 456 t/s summed across 8 concurrent streams. Setup is Strata with an int8 KV cache, 131072 context, speculative decoding on, and the 188.2 GB GGUF streamed from 504 GB of system RAM on an AMD EPYC 7642 box. The summed 8-stream figure is the arithmetic total of per-stream rates; the measured wall-clock aggregate stays at 56.89 t/s because streams overlap only partially. The same box ran QFN IQ3_S at 121.94 t/s single-stream and 714 t/s summed at C=6, GLM-5.3-Flash at 35.51 t/s and DeepSeek-V4.1-Flash at 44.86 t/s (XS) and 29.87 t/s (S).

Nov 1, 2026
Tone: positiveNVIDIA hardware
generation:
170-210 tokens/s
prompt processing (prefill):
1500-2000 tokens/s
quant:
UD-Q5_K_XL (GGUF)
kv:
F16
agenticcoding

User reports Qwen3.8-Flash-Next Gyro-M at 170-210 t/s decode and up to 5k t/s prefill on a dual RTX 3090 machine with 128GB DDR4, using Strata with a UD-Q5_K_XL GGUF and F16 KV cache. All experts and KV cache fit in VRAM, with the PLE in system RAM. On a second machine (dual RTX 5070 Ti, 32GB DDR4) the same model ran at 105-115 t/s decode and up to 3k t/s prefill with int8 KV cache, slower to decode than the IQ3_XXS build it replaced but with higher prefill. The user says Gyro-M quality is on par with Q4 and nearly as capable as Q5, with no reasoning loops or file rewrites, and has used it as a subagent for a couple of weeks. The previous IQ3_XXS build ran at 120 t/s decode and 2-2.5k t/s prefill on the 5070 Ti pair, and the Q5 build at 85-95 t/s decode and 1.5-2k t/s prefill on the 3090 pair.

Nov 1, 2026
NVIDIA hardware
generation:
30.0 tokens/s
quant:
IQ4_XS (GGUF)
kv:
q4_0
coding

User reports about 30 t/s with IQ4_XS on two RTX 3090s running Strata, and around 100 t/s with IQ3_S and 20 t/s or less with Q4_K_XL. Setup is Strata in Docker with 256K context, q4_0 KV cache, both GPUs, and 128GB of system RAM on a Ryzen 9 3900X. The user is tuning the configuration for coding and asks whether the larger IQ4_XS quant is worth the speed drop over IQ3_S.

Oct 28, 2026
Tone: mixedNVIDIA hardware
prompt processing (prefill):
3536.0 tokens/s

User reports a GPU unlock module raising prefill on Qwen3.8 Flash-Next under Strata on 2 RTX 3090s, from 3205 to 3536 tok/s at 90K context, a 10% gain. The module, gpu-unlock by Zagitta, sets FMLA32 to full before the NVIDIA driver loads; the user measured 4K and 10K prefill gains of 12% each. BF16 tensor math rose from 82 to 155 TFLOPS and TF32 from 41 to 79 TFLOPS, but outputs were unchanged byte for byte over 1996 teacher-forced positions. The user asks whether it works on 40-series cards.

Oct 24, 2026
Tone: positiveNVIDIA hardware
generation:
110-120 tokens/s
prompt processing (prefill):
2000-2500 tokens/s
quant:
IQ3_S (GGUF)

User reports Qwen3.8 Flash-Next Swift-1.5 at 110-120 tok/s decode and 2000-2500 tok/s prefill on an RTX 3090 with Strata. Setup is Strata with IQ3_S at 262144 context on the 24GB card in a 128GB DDR4 system. The same model on an RTX 4060 office PC with IQ2_XS and Q4 KV cache gives around 37-38 tok/s decode and 250 tok/s prefill. On an RTX 5060 with 32GB RAM, Strata prefill was around 100 tok/s. The user says hand-tuned llama.cpp only reached 30-40 tok/s decode, while Strata with the tiel variant gives at least 60 tok/s.

Oct 21, 2026
Tone: positiveNVIDIA hardware
generation:
100.0 tokens/s
quant:
IQ3_S (GGUF)
codingmathlong-context

User reports Qwen3.8 Flash-Next at around 100 tokens/s on 2x RTX 3090. Setup is Strata with IQ3_S quant. User is happy with the speed and says quality is a bit better than 27B at Q8. User also owns five more RTX 3090s (one ASUS ROG Strix, four Zotacs) and is deciding whether to upgrade the X299 machine, build a second 4x3090 rig, or sell the extra cards.

Oct 16, 2026
NVIDIA hardware
generation:
100.0 tokens/s
quant:
QFN

User reports around 100 t/s running Qwen3.8 Flash-Next with QFN on Strata using 3x RTX 3090 at PCIe 4.0 x16 each. Setup is Strata with QFN quant on three RTX 3090s. User asks for advice on adding a fourth RTX 3090, considering PCIe bifurcation or a motherboard upgrade, and is concerned about x8 performance impact on Strata.

Oct 15, 2026

Qwen3.8 27B

NVIDIA RTX 3090 · Strata · 300,000 ctx

NVIDIA hardware
generation:
32.0 tokens/s

User reports Qwen3.8-27B at 32 tok/s on a single RTX 3090 at 300k context. Setup uses Strata on the RTX 3090; a Titan RTX 24GB sits idle in the same machine, with 48GB of free system RAM. User asks whether Strata's multi-GPU support would hurt prefill speed when adding the second card without NVLink.

Oct 15, 2026
Showing 1–20 of 33
Page 1 of 2

Community benchmarks snapshot

Records by GPU

AMD Strix Halo 128GB348NVIDIA RTX 3090271NVIDIA RTX 5090197NVIDIA DGX Spark99NVIDIA RTX 3060 12GB96NVIDIA RTX 5060 Ti 16GB88NVIDIA RTX Pro 6000 Blackwell81AMD Radeon AI PRO R9700 32GB81NVIDIA RTX 5070 Ti77NVIDIA RTX 409068

Records by model

3266 total
Qwen3.81567
Qwen3.6334
Gemma 4185
DeepSeek V4 Flash150
Qwen3.5145
DeepSeek V4.1 Flash103
other782

Records by engine

2308 total
llama.cpp1134
Strata234
vLLM227
MLX107
Ollama99
other507

Use cases

coding 937agentic 640long-context 442tool-use 281vision 174summarization 81math 77creative-writing 58multilingual 35rp 22text-generation 9reasoning 3
coding937agentic640long-context442tool-use281vision174summarization81math77creative-writing58

Median generation t/s by GPU

On Qwen3.8 27B at 4-bit, plain single-GPU runs. Full ranking

RTX 509080RTX Pro 600067RTX 5070 Ti55M5 Ultra 256GB50RTX 409040RX 7900 XTX40RTX 309038RX 7800 XT 16GB35RTX 5090 Laptop 24GB34RX 7900 XT32

Prompt processing, same model

M5 Ultra 256GB1800RTX 40901949RX 7900 XTX640RTX 3090561RX 7900 XT800M3 Ultra 512GB372V100 32GB941Radeon AI PRO R9700 32GB1287RTX 3080 20GB997Arc Pro B70512

Reports by model size

Qwen3.8 27B819Qwen3.8 125B · 6B active676Qwen3.6 35B · 3B active242DeepSeek V4 Flash 284B · 13B active133DeepSeek V4.1 Flash 552B · 16B active92Qwen3.6 27B81Gemma 4 26B · 4B active61GLM-5.3 320B · 18B active57

Quants

Q4_K_M395UD-Q4_K_XL189NVFP4174Q8_01194-bit106IQ4_XS86IQ3_XXS85IQ3_S84Q4_K_XL83Q480