Find which open-weight LLMs fit in your GPU or Mac and compare the speeds people report on setups like yours.
Pick your GPU or Mac, then read the speeds people reported on it, or estimate which models fit and how fast they run.
Free to use. No account needed. Memory estimates and community measurements are labelled separately.
Search for a model to see its reported speeds across GPUs and Macs.
These are individual setups, not a controlled benchmark. Compare GPU count, quantization, context and offloading before comparing speeds. How to read a report →
Exact recorded values. Context may be a configured limit; matching filters does not establish identical prompts, offloading or concurrency.
User reports Qwen3.8 Flash-Next at up to 275 tok/s peak generation and 5500 tok/s prompt processing on a single RTX 5080 16GB with 64 GB DDR5-6400 and a Core Ultra 9 285K. Setup is the Strata fork Basalt with IQ3_S weights, run during a live opencode coding session. Code and tool calls peak at 275 tok/s while reasoning traces reach up to 190 tok/s; the user says the IQ3_S quant is very usable.
NVIDIA RTX 5080 · Strata · 65,536 ctx
User reports Qwen3.8-Flash-Next at 150 tok/s decode and about 4,400 tok/s prompt on a single RTX 5080 16 GB. Setup is Strata v0.2.1 with IQ3_XXS GGUF, int8 KV cache and 64K context, MTP speculation on, 15 CPU expert workers and an expert cache of 5,310 experts (8.61 GiB VRAM). Decode median 150.0 tok/s over 9 runs (126.7-170.2); fresh prompt runs gave 3,755 tok/s at 4K, 4,396 at 16K and 4,412 at 32K. The card was tuned with LACT (memory +6000, core capped 2850 MHz, 360 W), so stock cards may be slower.
2× NVIDIA RTX 5080 · Strata · 65,536 ctx
User reports Qwen3.8-Flash-Next at 110.6 t/s generation and 2773 t/s prefill on two RTX 5080s with a Ryzen 9 9900X, using a custom Strata fork. Setup is Strata with UD-Q4_K_XL, 64K context, INT8 K/V, 8K prompt chunk, MTP on and expert adaptation on. Upstream v0.1.38 reached 78.1 t/s generation and 2016 t/s prefill with layer split, and 70.3 t/s generation and 1522 t/s prefill with the host peer tier. With static caches custom was about 26% ahead of upstream layer split (55 vs 44 t/s). Letting the second GPU compute experts during prompt processing doubled custom's 32K prefill (1366 to 2731 t/s), and a 16K chunk brought it to 3724 t/s. Putting the expert tier on the x8 PCIe card improved 4K prefill by 58% and 32K by 34%. The saved answer came from the custom fork, so it may favor its drafts.
2× NVIDIA RTX 5080 · Strata · 262,144 ctx
User reports Qwen3.8-Flash-Next (Swift 1.5 IQ2_XS) at 143 tok/s code generation on an RTX 5080 + RTX 4060 Ti with 32 GB system RAM. Setup is a dual-GPU fork of the Strata engine at 256K context, with experts that don't fit in VRAM read from the SSD and a fine-tuned draft layer for speculative decoding. The same machine with the 5080 alone reached 29 tok/s code and 333 tok/s on a 32K prompt; the fork with both cards reached 1,940 tok/s on the 32K prompt. In real use with the Pi coding agent it peaks above 200 tok/s (209 so far) and a 150K-token context reads at about 1,850 tok/s. Perplexity was 7.24 vs upstream's 7.28.
On Qwen3.8 27B at 4-bit, plain single-GPU runs. Full ranking