llamaperf

Local model performance on your hardware

Find which open-weight LLMs fit in your GPU or Mac and compare the speeds people report on setups like yours.

What runs on your hardware?

Pick your GPU or Mac, then read the speeds people reported on it, or estimate which models fit and how fast they run.

Free to use. No account needed. Memory estimates and community measurements are labelled separately.

Looking for a particular model?

Search for a model to see its reported speeds across GPUs and Macs.

Local model performance reports from the community

These are individual setups, not a controlled benchmark. Compare GPU count, quantization, context and offloading before comparing speeds. How to read a report →

GPU: NVIDIA RTX 3090
Compare setup details (1 active)

Exact recorded values. Context may be a configured limit; matching filters does not establish identical prompts, offloading or concurrency.

Tone: mixedNVIDIA hardware
generation:
38.0 tokens/s
quant:
Q4 (GGUF)
long-contextagentic

User reports Qwen3.8 Flash-Next at around 38 t/s in Q4 and 50 t/s in Q3 on an RTX 3090 with 64GB of system RAM. Setup is Strata at 256k context with max thinking settings, run for legal research. User asks why a 20-page document analysis takes at least 3 hours with 4 sub agents, and says Strata keeps using sub agents even when they are turned off by prompt. Lowering thinking to high/medium still takes at least 2 hours.

Nov 18, 2026
Tone: mixedNVIDIA hardware
generation:
130.0 tokens/s
quant:
IQ3_S (GGUF)

User reports Qwen3.8-Flash-Next at about 130 t/s on an RTX 3090 plus RTX 5080 (40 GB VRAM) with 32 GB system RAM. Setup is Strata with the standard IQ3_S GGUF. User says the model runs great but that the Unsloth UD_IQ3_XXS build of the same size stalls during quick tune, runs at only about 20 t/s without tuning, and reads more from the SSD.

Nov 17, 2026

Edge0 145B

NVIDIA RTX 3090 · Strata · 16,384 ctx

Tone: mixedNVIDIA hardware
generation:
48.6 tokens/s
prompt processing (prefill):
1481.2 tokens/s
quant:
q3k
kv:
int8
agentictool-use

User reports Edge0-145B at 48.6 t/s chat decode on a single RTX 3090, with a 16K prompt read at 1,481.2 t/s and agent decode at 52.8 t/s. Setup is Strata with the q3k quant, int8 KV cache, speculative decoding 4, 131K max context, and direct PLE I/O from disk; the FP8 PLE table alone is 51 GB. GPU peak was 23,929 of 24,576 MiB and the engine process held 43.4 GiB of system RAM, so the user does not know the memory floor yet. The agent figure is speed only: three temperature-0 runs gave three different tool calls, one invalid.

Nov 17, 2026
Tone: positiveNVIDIA hardware
generation:
80.4 tokens/s
prompt processing (prefill):
1036.0 tokens/s
quant:
UD IQ4_XS (GGUF)
long-contextagentic

User reports Qwen3.8-Flash-Next at a median 80.4 tok/s chat decode on an RTX 3090 with an RTX 2080 SUPER as helper card, over 3 runs (range 67.6-87.8, n=9). Setup is Strata with Unsloth UD IQ4_XS weights, 96GB DDR4 3600 system RAM. 4K prompt processing ran at 1,036 tok/s and 16K at 1,915 tok/s; agent session decode was 101.7 tok/s with a 69.4% cache hit. A 66,371-token needle test prefilled in about 34 s (~2,050 tok/s) and decoded at 76.1 tok/s with the full KV resident, retrieving the mid-context needle. The user pins the 3090 memory clock to avoid a driver freeze bug, idling at ~110 W instead of ~25 W, and saw 79°C peak under a 3-minute load.

Nov 16, 2026
Tone: mixedNVIDIA hardware
generation:
41.0 tokens/s
quant:
IQ4

User reports Strata IQ4 Qwen3.8 Flash-Next decoding at 41 t/s on an RTX 3090 with a Xeon E5 and 96GB DDR4 ECC. Setup is Strata with IQ4 weights, used to edit a Qwen 0.6B model running fully on CPU in FP32, over a 39.8M-token run lasting 3h 12m. The edit steered the 0.6B model to answer a favourite-band question with Polyphia; the user calls the setup weird and questions whether models above 20B would benefit compared with traditional gradient training.

Nov 4, 2026
Tone: positiveNVIDIA hardware
generation:
50.0 tokens/s
quant:
UD-Q5_K_XL (GGUF)
kv:
8-bit
codingvisionlong-contextagentic

User reports Qwen 3.8 Flash Next at 50 t/s with UD-Q5_K_XL and 120 t/s with Gyro-M on an RTX 3090 24GB. Setup is Strata on Ubuntu 24.04 with 128K context and 8-bit KV cache, on a Dell T5810 with a Xeon 2699 v4 and 128 GB of ECC DDR4. The user asks which smarter model reaches at least 20 t/s, how to improve long-term context and memory, and what cost-efficient upgrade path to take up to mid-2027.

Nov 3, 2026
Tone: mixedNVIDIA hardware
generation:
70.0 tokens/s
prompt processing (prefill):
2400.0 tokens/s
quant:
Q8_0 (GGUF)

User reports Qwen3.8 Flash-Next at Q8_0 running at about 70 t/s generation and about 2400 t/s prompt processing on four RTX 3090s (96GB VRAM) with 192GB of octa-channel 3200MHz system RAM. Setup is Strata (0.2.9-cuda13) with a 262144-token context, PARALLEL 1, vision on, the qwen-flash preset and a 65536 MiB conversation cache across 16 slots. User asks whether others still run Qwen on Strata with larger quants, whether they retune the config on every Strata update, and what their configs and performance look like, citing concerns that Strata updates sometimes trade quality for speed without an opt-in.

Nov 2, 2026
Tone: positiveNVIDIA hardware
generation:
94.2 tokens/s
quant:
Q8_0 (GGUF)
kv:
int8

User reports Qwen3.8-Flash-Next Q8_0 at 94.2 t/s single-stream on 2x RTX 3090, with 456 t/s summed across 8 concurrent streams. Setup is Strata with an int8 KV cache, 131072 context, speculative decoding on, and the 188.2 GB GGUF streamed from 504 GB of system RAM on an AMD EPYC 7642 box. The summed 8-stream figure is the arithmetic total of per-stream rates; the measured wall-clock aggregate stays at 56.89 t/s because streams overlap only partially. The same box ran QFN IQ3_S at 121.94 t/s single-stream and 714 t/s summed at C=6, GLM-5.3-Flash at 35.51 t/s and DeepSeek-V4.1-Flash at 44.86 t/s (XS) and 29.87 t/s (S).

Nov 1, 2026
NVIDIA hardware
generation:
30.0 tokens/s
quant:
IQ4_XS (GGUF)
kv:
q4_0
coding

User reports about 30 t/s with IQ4_XS on two RTX 3090s running Strata, and around 100 t/s with IQ3_S and 20 t/s or less with Q4_K_XL. Setup is Strata in Docker with 256K context, q4_0 KV cache, both GPUs, and 128GB of system RAM on a Ryzen 9 3900X. The user is tuning the configuration for coding and asks whether the larger IQ4_XS quant is worth the speed drop over IQ3_S.

Oct 28, 2026
Tone: mixedNVIDIA hardware
prompt processing (prefill):
3536.0 tokens/s

User reports a GPU unlock module raising prefill on Qwen3.8 Flash-Next under Strata on 2 RTX 3090s, from 3205 to 3536 tok/s at 90K context, a 10% gain. The module, gpu-unlock by Zagitta, sets FMLA32 to full before the NVIDIA driver loads; the user measured 4K and 10K prefill gains of 12% each. BF16 tensor math rose from 82 to 155 TFLOPS and TF32 from 41 to 79 TFLOPS, but outputs were unchanged byte for byte over 1996 teacher-forced positions. The user asks whether it works on 40-series cards.

Oct 24, 2026
Tone: positiveNVIDIA hardware
generation:
100.0 tokens/s
quant:
IQ3_S (GGUF)
codingmathlong-context

User reports Qwen3.8 Flash-Next at around 100 tokens/s on 2x RTX 3090. Setup is Strata with IQ3_S quant. User is happy with the speed and says quality is a bit better than 27B at Q8. User also owns five more RTX 3090s (one ASUS ROG Strix, four Zotacs) and is deciding whether to upgrade the X299 machine, build a second 4x3090 rig, or sell the extra cards.

Oct 16, 2026
NVIDIA hardware
generation:
100.0 tokens/s
quant:
QFN

User reports around 100 t/s running Qwen3.8 Flash-Next with QFN on Strata using 3x RTX 3090 at PCIe 4.0 x16 each. Setup is Strata with QFN quant on three RTX 3090s. User asks for advice on adding a fourth RTX 3090, considering PCIe bifurcation or a motherboard upgrade, and is concerned about x8 performance impact on Strata.

Oct 15, 2026

Qwen3.8 27B

NVIDIA RTX 3090 · Strata · 300,000 ctx

NVIDIA hardware
generation:
32.0 tokens/s

User reports Qwen3.8-27B at 32 tok/s on a single RTX 3090 at 300k context. Setup uses Strata on the RTX 3090; a Titan RTX 24GB sits idle in the same machine, with 48GB of free system RAM. User asks whether Strata's multi-GPU support would hurt prefill speed when adding the second card without NVLink.

Oct 15, 2026
NVIDIA hardware
generation:
62.0 tokens/s
prompt processing (prefill):
1400.0 tokens/s
quant:
IQ3_S GSQ-RCO (GGUF)
kv:
INT8

User reports Qwen3.8 Flash-Next at ~62 t/s decode and ~1400 t/s prefill on a single RTX 3090 with 8-channel DDR4 server RAM. Setup is Strata with IQ3_S GSQ-RCO quant and 256K INT8 KV context at a 220W power cap. User also mentions GLM 5.3 Flash "Maya L" 156.3GB running on a single Tesla V100 32GB via Project Maya, but gives no throughput figure for it. User asks what benchmarks to use to compare the two models on quality.

Oct 14, 2026
Tone: mixedNVIDIA hardware
generation:
59-62 tokens/s
prompt processing (prefill):
1325.0 tokens/s
quant:
IQ3_XXS (GGUF)
kv:
int8
long-context

User reports Qwen3.8-Flash-Next 125B-A6B at IQ3_XXS running at the full 262,144-token context on a single RTX 3090 with 32 GB of DDR5, using the Strata engine with int8 KV cache and MTP speculation. On a 121,566-token prompt the run reached 1325 tok/s prefill and 59-62 tok/s decode; a 249K-token prompt gave 1008 tok/s prefill and 41.8 tok/s decode with 90.5% expert-cache hits, and short-context decode climbs to 63-100 tok/s. Setup uses --resident-cpu-experts with a 21 GiB resident budget, 12.2 GiB of experts in VRAM, 21 GiB locked in RAM and 6.6 GiB left on disk, covering 83% of routings without disk reads. The process is pinned to P-cores with taskset -c 0-15 and --pool-affinity p-cores; --prefill auto:16384 raised prefill from 1014 to 1325 tok/s on the 121K prompt. --aux-cpus gave no gain on this chip (60.1 vs 59.6 tok/s decode). The user asks whether these numbers are good and what others get on comparable hardware, and notes the KV cache is about 3.1 GiB at 262K because only 12 of 48 layers keep growing KV.

Oct 12, 2026
NVIDIA hardware
generation:
19.1 tokens/s
prompt processing (prefill):
41.0 tokens/s
quant:
GSQ-RCO 3.5-bit (GGUF)

User reports GLM-5.3-Flash at 19.1 t/s decode and 41 t/s prefill on an RTX 3090 with a 47-token prompt. Setup is Strata with GSQ-RCO 3.5-bit GGUF at 32,768 tokens of context, most experts in RAM (121 GiB of 177 GiB system RAM used). Prefill rises with prompt length to 974 t/s at 26,641 tokens; decode stays 18.7-19.0 t/s across contexts. On 9 GPUs (8x RTX 5060 Ti 16 GB plus the 3090) decode is about 28 t/s and 8K prompts run 1150-1210 t/s. Calibration kept the engine's own settings at 21.6 t/s.

Oct 9, 2026
Tone: positiveNVIDIA hardware
generation:
100.0 tokens/s
quant:
IQ4_XS (GGUF)

User reports Qwen3.8-Flash-Next-OrcaRouter IQ4_XS at a bit over 100 t/s decode on short context on 2x RTX 3090 24GB with Strata. Setup is Strata on 2x RTX 3090 with IQ4_XS weights, which keeps the useful experts on the GPUs and the rest in 64GB DDR5 system RAM, vision enabled, experimental context extension configured for 512K. A prior vLLM setup did about 60-70 t/s decode. Context extension runs: 200K prompt at ~2,004 t/s prompt processing and ~73 t/s decode; 400K at ~1,633 t/s and ~46 t/s; 900K at ~1,110 t/s and ~44 t/s.

Oct 8, 2026
Tone: positiveNVIDIA hardware
generation:
43.9 tokens/s
prompt processing (prefill):
1817.0 tokens/s
quant:
Q6_K_XL (GGUF)
kv:
int8

User reports Qwen3.8-Flash-Next at 43.9 tok/s generation and 1817 tok/s prompt processing on an RTX 3090 + RTX A4000 at 32k context. Setup is Strata with UD-Q6_K_XL GGUF and int8 KV cache, 262144 native context, speculative decoding with MTP, experts offloaded to CPU. At 200k context Strata Q6 does 48.8 tok/s versus llama.cpp's 10.6 tok/s, and TTFT drops from 595 s to 118 s. Q6 is 12-31% faster than Q8 and 19 GB smaller.

Oct 7, 2026
Tone: mixedNVIDIA hardware
generation:
97.0 tokens/s
quant:
UD-Q4_K_XL (GGUF)
codingagentic

User reports Qwen3.8-Flash-Next at 97 t/s decode on 2x RTX 3090 with Strata, with about half the experts in system RAM. Setup is Strata v0.1.40.1 with unsloth UD-Q4_K_XL GGUF, a layer split, speculative decoding, and 121 GB DDR4 on a Ryzen 9 3950X. The 97 t/s is at a 95% expert hit rate; shrinking the cache to 90% and 81% hit rates gave 84 t/s and 68 t/s. The user estimates +34% decode from eliminating misses on their box and about +75% headroom at a single-24GB-card share, and reports +16% decode from --pcie-frac 0.2 --spec-min-p 0.7 over about 280 runs. The prefetch feature is not implemented yet.

Oct 7, 2026
NVIDIA hardware
generation:
19.8 tokens/s
prompt processing (prefill):
437.8 tokens/s
quant:
UD-Q4_K_XL (GGUF)

User reports GLM-5.3-Flash at 19.8 tok/s decode and 437.8 tok/s prefill on one RTX 3090 24 GB with 251 GB RAM. Setup is the Strata engine with a UD-Q4_K_XL GGUF pack, --chunk 4096, 16K-token prompt, tiered expert cache streaming from RAM and NVMe. Decode splits into 12.4 ms expert copies, 0.2 ms expert kernels and 37.2 ms dense/sync per 49.8 ms token. A 64K prompt prefills at 420.7 tok/s; a 1K prompt at 132 tok/s prefill and 19.8 tok/s decode; disk-only tier decodes at 1.15 tok/s. --chunk 8192 OOMs on 24 GB.

Oct 6, 2026
Showing 1–20 of 25
Page 1 of 2

Community benchmarks snapshot

Records by GPU

AMD Strix Halo 128GB348NVIDIA RTX 3090271NVIDIA RTX 5090197NVIDIA DGX Spark99NVIDIA RTX 3060 12GB96NVIDIA RTX 5060 Ti 16GB88NVIDIA RTX Pro 6000 Blackwell81AMD Radeon AI PRO R9700 32GB81NVIDIA RTX 5070 Ti77NVIDIA RTX 409068

Records by model

3265 total
Qwen3.81567
Qwen3.6334
Gemma 4185
DeepSeek V4 Flash150
Qwen3.5145
DeepSeek V4.1 Flash103
other781

Records by engine

2307 total
llama.cpp1133
Strata234
vLLM227
MLX107
Ollama99
other507

Use cases

coding 937agentic 640long-context 442tool-use 280vision 173summarization 81math 77creative-writing 58multilingual 35rp 22text-generation 9reasoning 3
coding937agentic640long-context442tool-use280vision173summarization81math77creative-writing58

Median generation t/s by GPU

On Qwen3.8 27B at 4-bit, plain single-GPU runs. Full ranking

RTX 509080RTX Pro 600067RTX 5070 Ti55M5 Ultra 256GB50RTX 409040RX 7900 XTX40RTX 309038RX 7800 XT 16GB35RTX 5090 Laptop 24GB34RX 7900 XT32

Prompt processing, same model

M5 Ultra 256GB1800RTX 40901949RX 7900 XTX640RTX 3090561RX 7900 XT800M3 Ultra 512GB372V100 32GB941Radeon AI PRO R9700 32GB1287RTX 3080 20GB997Arc Pro B70512

Reports by model size

Qwen3.8 27B819Qwen3.8 125B · 6B active676Qwen3.6 35B · 3B active242DeepSeek V4 Flash 284B · 13B active133DeepSeek V4.1 Flash 552B · 16B active92Qwen3.6 27B81Gemma 4 26B · 4B active61GLM-5.3 320B · 18B active57

Quants

Q4_K_M395UD-Q4_K_XL189NVFP4174Q8_01194-bit106IQ4_XS86IQ3_XXS85IQ3_S84Q4_K_XL83Q480