Find which open-weight LLMs fit in your GPU or Mac and compare the speeds people report on setups like yours.
Pick your GPU or Mac, then read the speeds people reported on it, or estimate which models fit and how fast they run.
Free to use. No account needed. Memory estimates and community measurements are labelled separately.
Search for a model to see its reported speeds across GPUs and Macs.
These are individual setups, not a controlled benchmark. Compare GPU count, quantization, context and offloading before comparing speeds. How to read a report →
Exact recorded values. Context may be a configured limit; matching filters does not establish identical prompts, offloading or concurrency.
User reports Qwen3.8 Flash-Next at up to 275 tok/s peak generation and 5500 tok/s prompt processing on a single RTX 5080 16GB with 64 GB DDR5-6400 and a Core Ultra 9 285K. Setup is the Strata fork Basalt with IQ3_S weights, run during a live opencode coding session. Code and tool calls peak at 275 tok/s while reasoning traces reach up to 190 tok/s; the user says the IQ3_S quant is very usable.
2× NVIDIA RTX 5080 · Ollama · 65,536 ctx
User reports Qwen3.8 27B at about 40 tokens/second on an RTX 5080 plus RTX 5070, with MTP disabled. Setup is Ollama with Q8_0 weights and 128k of context; the vision encoder and MTP were turned off to fit the split across the two cards. With MTP enabled at 64k context the user sees about 54 tokens/second. They ask whether Ollama's per-GPU buffer can be lowered, having had no luck with LLAMA_ARG_FIT_TARGET, or whether to switch to llama.cpp.
NVIDIA RTX 5080 · llama.cpp · 40,960 ctx
User compares an RTX 5080 desktop against a rented DGX Spark on Qwen3-8B Q4_K_M, where the PC decodes at 130.55 t/s versus 42.43 t/s on Spark at 512 input tokens, roughly three times faster. Both ran llama.cpp release b11889 with Q8 KV cache, Flash Attention and a 40,960-token context; the PC kept the whole model on its 16 GB card. Decode falls to 79.09 t/s on the PC and 25.64 t/s on Spark at 32,768 input tokens. With Qwen3-32B the PC needed CPU offload (40 GPU layers) and managed 4.92 t/s against Spark's 10.50 t/s. Aggregate batched throughput at 8 concurrent requests was 471.75 t/s on the PC and 209.02 t/s on Spark.
NVIDIA RTX 5080 · Basalt · 262,144 ctx
User reports Qwen3.8-Flash-Next IQ3_S on an RTX 5080 16 GB with a Core Ultra 9 285K and 62 GB DDR5-6800, averaging 154.9 t/s generation and 5,475 t/s prompt processing in a context sweep of fresh 15K-207K prompts. Setup is Basalt with IQ3_S weights, 262K context and int8 KV cache, using 15.7 GB VRAM and 54.1 GB system RAM. An agent coding session over 103 requests reached 153.0 t/s average generation with a 262 t/s peak, and 5,011 t/s average prefill; 99.5% of prompt tokens came from cache with a 0.2 s median time to first token.
NVIDIA RTX 5080 · Strata · 65,536 ctx
User reports Qwen3.8-Flash-Next at 150 tok/s decode and about 4,400 tok/s prompt on a single RTX 5080 16 GB. Setup is Strata v0.2.1 with IQ3_XXS GGUF, int8 KV cache and 64K context, MTP speculation on, 15 CPU expert workers and an expert cache of 5,310 experts (8.61 GiB VRAM). Decode median 150.0 tok/s over 9 runs (126.7-170.2); fresh prompt runs gave 3,755 tok/s at 4K, 4,396 at 16K and 4,412 at 32K. The card was tuned with LACT (memory +6000, core capped 2850 MHz, 360 W), so stock cards may be slower.
NVIDIA RTX 5080 · NInfer
User compares Qwen3.8 Flash-Next on Strata against a Qwen3.8 27B build for the RTX 5080 on NInfer, reporting 89-112 t/s decode on the 27B. Setup is NInfer on an RTX 5080 with 32GB DDR5, vision flags removed and thinking budget raised to 4096; the user plugged the display into the iGPU and reports 94.6% VRAM usage while using the desktop. The user says the 27B solved in one prompt a coding task that the Flash-Next coder and an IQ2_XS build needed multiple prompts for, and calls it insanely fast at prefill and decode.
NVIDIA RTX 5080 · Ollama · 4,096 ctx
User reports Devstral 24B at 34.9 t/s generation and 5334.83 t/s prompt processing on an NVIDIA RTX 5080. Setup is Ollama with Q4_K_M at a configured 4096-token context; the recorded prompt was 2597 tokens, not a full 4096-token prefill. The figure is a median of 3 fresh-prefill runs at temperature 0 and seed 42, with a unique nonce per repetition to avoid prompt-cache hits; the source captured only 4096-context cells and warns against extrapolating to 8K or 16K.
NVIDIA RTX 5080 · Ollama · 16,384 ctx
User reports Qwen3-Coder 30B-A3B at 51.7 t/s generation and 1928.1 t/s prompt processing on an NVIDIA RTX 5080. Setup is Ollama with Q4_K_M at a configured 16384-token context, with partial CPU/RAM offload since the model exceeds VRAM. The recorded prompt was 1347 tokens, not a full 16384-token prefill, and the figures are medians of 3 fresh-prefill runs; 74% GPU residency was measured separately at 8K context, not this cell.
NVIDIA RTX 5080 · Ollama · 16,384 ctx
User reports Qwen2.5 Coder at 93.4 t/s generation and 4842.2 t/s prompt processing on an NVIDIA RTX 5080. Setup is Ollama with Q4_K_M at 16384 tokens of configured context, on one 16GB card. The recorded prompt was 1368 tokens, not a 16384-token prefill, and the figures are medians of 3 fresh-prefill runs at temperature 0 with a 256-token target.
NVIDIA RTX 5080 · Ollama · 16,384 ctx
User reports Qwen2.5 Coder-7B at 171.4 t/s generation on one NVIDIA RTX 5080 16GB. Setup is Ollama with Q4_K_M at 16384 tokens of configured context, on a Ryzen 7 7800X3D under Windows 11. The figure is the median of 3 fresh-prefill runs with a 1368-token prompt, not a 16384-token prefill; prompt processing was 9825.5 t/s.
NVIDIA RTX 5080 · Ollama · 16,384 ctx
User reports Qwen2.5 7B at 176.6 t/s generation and 9984.4 t/s prompt processing on an NVIDIA RTX 5080. Setup is Ollama with Q4_K_M at a configured 16384-token context, on one 16GB card. The recorded prompt was 1674 tokens, not a full 16384-token prefill, and the figures are medians of 3 fresh-prefill runs.
NVIDIA RTX 5080 · Ollama · 16,384 ctx
User reports Llama 3.2 3B at 298.6 t/s generation and 19397.4 t/s prompt processing on an NVIDIA RTX 5080. Setup is Ollama with Q4_K_M at a configured 16384-token context, with an actual recorded prompt of 1657 tokens rather than a full 16384-token prefill. The figures are medians of 3 fresh-prefill runs, derived from Ollama eval_count/eval_duration rather than TTFT division, with temperature 0 and a unique nonce per repetition to avoid prompt-cache hits.
NVIDIA RTX 5080 · Ollama · 16,384 ctx
User reports Qwen2.5 14B at 94.6 t/s generation and 4864.7 t/s prompt processing on one NVIDIA RTX 5080 16GB. Setup is Ollama with Q4_K_M at a configured 16384-token context; the recorded prompt was 1674 tokens, not a full 16384-token prefill. The figures are medians of 3 fresh-prefill runs, with 256 target output tokens and a unique nonce per repetition to avoid prompt-cache hits.
NVIDIA RTX 5080 · Ollama · 16,384 ctx
User reports gpt-oss 21B-A3.6B at 178.3 t/s on an NVIDIA RTX 5080. Setup is Ollama with MXFP4 at 16384 configured context, on one 16GB card; the recorded prompt was 1360 tokens, not a full 16384-token prefill. The figure is a median of 3 fresh-prefill runs at temperature 0 and seed 42, with a unique nonce per repetition to avoid prompt-cache hits; it is the Aug 26 re-capture, not the faster Aug 12 capture.
3× NVIDIA RTX 5080 · LM Studio
User reports Qwen3.8 27B at about 30 tok/s on an RTX 5080 with two RTX 5060 Ti 16GB cards in a multi-GPU setup. Setup is LM Studio with the default Qwen parameters; the two secondary cards run at PCIe 4.0 x4. The user asks whether swapping in an RX 7900 XT or switching to Vulkan would improve speed, and whether a Thunderbolt 3 eGPU would only affect model loading time.
NVIDIA RTX 5080 · LM Studio
User reports Qwen3.8 27B IQ2_M at 80-100 tok/s on an RTX 5080 with 64GB system RAM. Setup is LM Studio with IQ2_M quant; the model is partially offloaded to system RAM because the 16GB card has about 1.5GB used at idle. The user finds the model too slow for web-search automation and says it loops on web search until hitting the context limit without retrieving the price.
NVIDIA RTX 5080 · NInfer · 110,592 ctx
User reports Qwen 3.8 27B at a median 96.45 t/s decode on a 16GB RTX 5080, with a range of roughly 90-110 t/s during coding tasks. Setup is NInfer v1.5 with q4 KV cache, MTP speculative decoding with 3 draft tokens, 110,592 context allocated, and 11.86 GiB of weights leaving about 498 MiB of slack. Across 32 completed requests the lowest decode was 84.6 t/s and the highest 131.7 t/s, with median time to first token 1.4 seconds. About 68% of generated tokens were reasoning tokens, and one request that lost its prompt cache took almost 50 seconds to process a 79k prompt before generating at about 94 t/s.
2× NVIDIA RTX 5080 · Strata · 65,536 ctx
User reports Qwen3.8-Flash-Next at 110.6 t/s generation and 2773 t/s prefill on two RTX 5080s with a Ryzen 9 9900X, using a custom Strata fork. Setup is Strata with UD-Q4_K_XL, 64K context, INT8 K/V, 8K prompt chunk, MTP on and expert adaptation on. Upstream v0.1.38 reached 78.1 t/s generation and 2016 t/s prefill with layer split, and 70.3 t/s generation and 1522 t/s prefill with the host peer tier. With static caches custom was about 26% ahead of upstream layer split (55 vs 44 t/s). Letting the second GPU compute experts during prompt processing doubled custom's 32K prefill (1366 to 2731 t/s), and a 16K chunk brought it to 3724 t/s. Putting the expert tier on the x8 PCIe card improved 4K prefill by 58% and 32K by 34%. The saved answer came from the custom fork, so it may favor its drafts.
2× NVIDIA RTX 5080 · Strata · 262,144 ctx
User reports Qwen3.8-Flash-Next (Swift 1.5 IQ2_XS) at 143 tok/s code generation on an RTX 5080 + RTX 4060 Ti with 32 GB system RAM. Setup is a dual-GPU fork of the Strata engine at 256K context, with experts that don't fit in VRAM read from the SSD and a fine-tuned draft layer for speculative decoding. The same machine with the 5080 alone reached 29 tok/s code and 333 tok/s on a 32K prompt; the fork with both cards reached 1,940 tok/s on the 32K prompt. In real use with the Pi coding agent it peaks above 200 tok/s (209 so far) and a 150K-token context reads at about 1,850 tok/s. Perplexity was 7.24 vs upstream's 7.28.
NVIDIA RTX 5080 · llama.cpp · 160,000 ctx
User reports Qwen3.8 27B at 80+ t/s generation and 1200+ t/s prompt processing on an RTX 5080 with 16GB VRAM, using CUDA unified memory to fit the model. Setup is llama.cpp with the Unsloth UD Q3_K_XL GGUF, 160000 context, q8_0 KV cache, flash attention, and MTP speculative decoding with ngram-mod draft. User asks whether Q3 quants are good enough for harness coding with opencode, pi, and qwen-cli, and notes this is the only quant that fits the whole model in VRAM.
On Qwen3.8 27B at 4-bit, plain single-GPU runs. Full ranking