Find which open-weight LLMs fit in your GPU or Mac and compare the speeds people report on setups like yours.
Pick your GPU or Mac, then read the speeds people reported on it, or estimate which models fit and how fast they run.
Free to use. No account needed. Memory estimates and community measurements are labelled separately.
Search for a model to see its reported speeds across GPUs and Macs.
These are individual setups, not a controlled benchmark. Compare GPU count, quantization, context and offloading before comparing speeds. How to read a report →
Exact recorded values. Context may be a configured limit; matching filters does not establish identical prompts, offloading or concurrency.
AMD Strix Halo 128GB · Gufo · 16,384 ctx
User reports Qwen3.8-Flash-Next at 34.1 tok/s decode on a Ryzen AI Max+ 395 128GB mini PC. Setup is Gufo at 16K context, with the model weights taking around 103.7 GiB. With MTP decode rises to 53.7 tok/s, and adding the NPU gives 34.2 tok/s decode with about 10% faster prefill. The user questions whether the largest model is the best use of the memory.
User reports Qwen3.8-27B at about 1180 t/s prompt processing and 95 t/s decode on an M5 Ultra Studio with 96GB of RAM. The model runs with MTP speculation and an oQ4e quant; SSD cache is disabled and the hot cache is set to 4GB with a 10% hit rate. System RAM sits just over 50% during heavy work, and the user asks whether a larger hot cache would be reasonable.
AMD RX 7900 XTX · llama.cpp · 16,384 ctx
User reports Qwen3.8-27B at 87.4 t/s decode and 872 t/s prefill at 16K context on one RX 7900 XTX, using a fork of llama.cpp tuned for RDNA3 Vulkan. Setup is the fork's llama.cpp with UD-Q4_K_M GGUF, q8_0 KV cache, flash attention and a DFlash2 draft model for speculative decoding; the card is 24 GB at stock 291 W on Mesa RADV. At 128K context the fork gives 69.1 t/s decode and 657 t/s prefill, against 26.3 t/s and 442 t/s upstream. The fork reaches up to 220K context where upstream fits about 192K, and DFlash2 no longer hits vk::ErrorDeviceLost past ~60K tokens. Only the 7900 XTX was tested.
User reports DeepSeek V4.1 Flash at 49.9 tok/s decode on one RTX PRO 4500 32 GB, median of three runs. Setup is the author's FLE engine fork with a 2-bit IQ2_XXS GGUF (232 GiB on disk), experts in 120 GB of system RAM with the busiest on the GPU, and DeepSeek's DSpark draft model proposing five tokens. Without the draft model decode is 37.2 tok/s. Against the original weights the 2-bit quant has KL divergence 0.248 and top-1 agreement 83.5 %. Speed is bound by DDR5-4800 memory across four sticks. With thinking on at 16k context, a budget of 8000 tokens lets both test tasks finish, while at 64k without a budget both thought 60,000 tokens and never answered. The author also notes Qwen3.8-Flash-Next 125B runs from 16 GB at about 15 tok/s and 91 tok/s on the 32 GB card.
User benchmarks XeMaya, a new Intel Arc inference engine forked from XeStrata, on DeepSeek V4.1 Flash, GLM 5.3 Flash and Qwen3.8 Flash-Next. The regular DeepSeek V4.1 Flash model runs at 11.4 t/s prefill and 14.56 t/s generation on a B70 32GB plus B65 32GB. Setup is XeMaya over SYCL and Level Zero on an Ubuntu server with a B70 32GB, B65 32GB, B60 24GB, Ryzen 9950X and 128GB RAM. GLM 5.3 Flash reaches 152-161 t/s prefill and 17.4 t/s generation, around 20 t/s with MTP; DeepSeek V4.1 Flash Coder reaches 224.8 / 21.7 t/s after optimization. Using all three GPUs is slower than two for the regular DeepSeek model (11.2 / 11.58 t/s) because the B60 sits on a slower chipset link. The user notes the comparisons are not perfectly controlled, with different model files, prompts and GPU configurations.
2× NVIDIA RTX 3080 10GB · Speedy · 200,000 ctx
User reports Qwen3.8-Flash-Next on 2x RTX 3080 10GB with experts in system RAM, reaching 3,957 t/s prefill at a 200K cold prompt and decode of ~145-185 t/s with UD-IQ4_XS. Setup is the Speedy fork of Strata with UD-IQ4_XS, a 200K-token cold prompt, two pipeline windows and speculative decoding; the box is a Threadripper PRO 5975WX with 128 GB DDR4-3200. Decode varies with reply length and context, sitting at the top of the range for short replies and the bottom after a 200K prompt; a 200K cold prompt took 51 s. The 1x RTX 3080 table gives ~113-128 t/s decode and 1,792 t/s prefill at 200K for the same quant.
User reports Qwen3.8 27B at around 32 t/s on 2x RTX 3060 12GB with tensor split 1,1 after moving to an x470 SLI motherboard running both cards at x8. Previously, with one card at x16 and the other at x1 on a B550 board, the user got 11-16 t/s in layer split mode. The tensor-split run eventually gets stuck in loops, including repeated tool calls and endless question marks in thinking, which did not happen in layer split mode.
User benchmarks Qwen3.8-Flash-Next IQ2_XS on an RTX 3060 12GB, reporting 33.0 tok/s generation and 109 tok/s prompt processing in Strata versus 25.6 tok/s and 31 tok/s in llama.cpp. Setup is Strata against llama.cpp with the same prompt and a 128-token reply, six warm runs each on a Ryzen 7600X machine. Generation ranged 29.3–35.8 tok/s for Strata and 23.2–26.9 for llama.cpp, about 29% faster. Over 9,180 WikiText-2 tokens perplexity was 2.9212 versus 2.9156, with 96.78% same most-likely next token and mean KLD 0.0098.
2× NVIDIA RTX 4000 Ada · llama.cpp · 32,768 ctx
User reports Qwen3.8-27B at 29.39 t/s generation and 350.96 t/s prompt processing on two RTX 4000 Ada GPUs (20GB each). Setup is llama.cpp (llama-server) with Q8_0 GGUF weights, q8_0 KV cache, 32768 context, flash attention on, unified KV cache, and an MTP draft model plus vision projector, all layers offloaded to the GPUs. The figures come from a first prompt of 376 tokens (1.1s) and 109 generated tokens (3.7s). The user asks for tuning advice and for help wiring Claude Code and Cline in VS Code to the local server.
2× NVIDIA Tesla P100 16GB · llama.cpp · 262,144 ctx
User reports Qwen3.8-27B at about 45 tok/s in English and about 32 tok/s in Polish on 2x Tesla P100 16GB. Setup is llama.cpp with UD-Q5_K_XL at 262k context, row/tensor split with MTP, on an X79 platform (i7-3930K, ~44 GB DDR3) under Proxmox with GPU passthrough. The P100s serve as the main LLM backend for an agent with long prompts and many tool calls; a GTX 1080 handles vision, Whisper, embeddings and image generation. The user asks whether to replace the 1080 with a Titan V or 2080 Ti, and reports a Quadro GP100 was not detected at all on the board.
2× NVIDIA RTX 3090 · llama.cpp · 170,000 ctx
User reports running Unsloth's Qwen3.8 27B Q5 at around 60 tokens/sec with a 170K context window on an RTX 3090 plus RTX 3080 Ti. Setup is llama.cpp with a 32GB DDR4 system RAM configuration. User asks for advice on better hardware use and says agentic tasks such as generating and working with .docx files are painfully slow.
User reports Qwen3.8 Flash-Next at Q3 running at 90 tok/s on an RTX 5070 Ti with 64GB of system RAM. Setup is Strata. User also reports DeepSeek V4.1 Flash at Q2 on Strata averaging 5-7 tok/s on the same card, and asks whether adding an Intel Arc B580 12GB alongside the 5070 Ti could combine to 28GB of VRAM without tanking speed.
NVIDIA RTX 3080 Ti · llama.cpp
User reports Qwen 3.5 9B at about 100 t/s on an RTX 3080 Ti, matching figures others have posted. Setup is llama.cpp with Q4_K_M weights, run at a 260W power limit instead of the stock 400W. Reducing the limit from 450W to 260W showed no meaningful drop in tok/s; at 220W the rate fell to about 80 t/s. The user notes VRAM bandwidth never maxes out and wonders whether another bottleneck is at play.
NVIDIA RTX 5070 Ti · Strata · 256,000 ctx
User compares Strata and Basalt running Qwen3.8-Flash-Next iq3s at 256k context on an RTX 5070 Ti with 128GB DDR4 and a 12900k, using opencode as the harness. Strata averaged 99.2 t/s generation and 839 t/s prefill over 11.1M tokens, with DFlash 2 and vision on, using 15.3GB VRAM and 69.2GB RAM. Basalt averaged 100 to 118 t/s and peaked at 145 t/s, with prefill peaking at 4200 t/s, using 15.8GB VRAM and 52.7GB RAM. User says Strata stalled between phases and failed to compact or finish the work, while Basalt ran smoothly and completed the project with fewer errors and less RAM. They plan to test NVFP4 and Swift next.
User reports Qwen3.8 Flash-Next Uncensored at 239.3 t/s generation and 4316.6 t/s prompt processing on an M5 Ultra 256GB, on a 4096-token prompt with 3000 output tokens. Setup uses oQ5e weights with MTP speculative decoding, single request, peaking at 122.9 GB of memory. Longer prompts slow decode to 155.0 t/s at 16384 tokens and 127.1 t/s at 65536 tokens. A 2x batch run gives 172.8 t/s aggregate generation and 1828.2 t/s prompt processing, a 0.72x speedup over baseline.
M4 Max 48GB · llama.cpp · 32,768 ctx
User reports the Qwen3.5 4B notes model in their LokalBot app generating at about 85 tok/s on an M4 Max with 48 GB. Setup is llama.cpp (bundled llama-server, Metal) with Q4_K_M at 32K context; the app loads five local models totalling 9.6 GB on disk and evicts them under a budget of half the machine's RAM. Notes for a 26-minute meeting took 33 s with the model warm and 60 s including the load. Autocomplete measured 56 ms median and 102 ms p95 per keystroke, query embedding 9 ms, and the Nemotron 3 speaker model 14.6% DER on 8 AMI recordings at about 500x real time. All numbers come from that one machine; no 16 GB, M1, M2 or M3 figures were measured.
NVIDIA RTX 3060 12GB · RustingStrata · 5,290 ctx
User benchmarks Qwen3.6-35B-A3B at 90.9 tok/s decode without MTP on an RTX 3060 12 GB, with 2330 tok/s prefill on a 5290-token prompt. Setup is RustingStrata with the Unsloth Q3_K_M GGUF (15.9 GiB, does not fit in VRAM) on a Ryzen 5 7600X with 32 GB RAM, 128 tokens generated, mean of 3 reps. With MTP drafts decode rises to 128.4 tok/s. The post compares RustingStrata against PolyStrata and llama.cpp.
M5 Max 128GB · ds4
User reports Kimi 3.1 running locally on a MacBook Pro M5 Max 128GB at 2 tok/s decode and 8.4 tok/s prefill. The run uses a source build of the ds4 engine with weights downloaded separately. No quant, context length or KV cache setting is stated.
2× NVIDIA RTX Pro 4000 Blackwell · NInfer · 39,843 ctx
User benchmarks Qwen3.8-27B at 155.9 t/s decode on a ~40k-token prompt using two RTX PRO 4000 Blackwell 24GB cards in tensor parallel 2. Setup is NInfer v0.9.0 (ninfer-serve, CUDA) with a QUASAR mixed NVFP4/FP8 artifact, int8 KV cache, 262144-token context and DFlash2 speculative decoding with 7 draft tokens; GPUs are power-capped to 145 W each. A second long-context run gave 163.8 t/s decode and 5590 t/s prefill; prose decode was 111.1 t/s and short reasoning tasks averaged 204.7 t/s with 10/10 correct. A ~9k-token prompt had a median TTFT of 1.31 s. Timings are client-side, one request at a time.
NVIDIA RTX 5090 · vLLM
User reports Qwen3.8-27B at 86 tokens/sec median single-request generation on an RTX 5090, up from 51 tokens/sec with speculative decoding off. Setup is vLLM on the 5090 (32 GB) in a tower with a Ryzen 9 9950X3D and 96 GB RAM; quant and context length are not stated. The user notes speculative decoding left less memory for requests in flight, and that this was a generation benchmark rather than a 70% speedup on code reviews.
Local model performance covers how fast a model generates a reply and how much memory its setup needs. These community measurements show generation and prompt processing speed for specific setups. To estimate memory fit on your hardware, use the VRAM calculator.
| Hardware | Model and setup | Generation | Prompt processing | Evidence |
|---|---|---|---|---|
| AMD RX 7900 XTX | Qwen3.8 27B 27B UD-Q4_K_M · llama.cpp 16,384 reported context | 87.4 tokens/s | 872 tokens/s | Full setupOriginal source ↗ |
| NVIDIA RTX 5070 Ti | Qwen3.8 125B Flash-Next iq3s · Strata 256,000 reported context | 99.2 tokens/s | 839 tokens/s | Full setupOriginal source ↗ |
| M4 Max 48GB | Qwen3.5 4B 4B Q4_K_M · llama.cpp 32,768 reported context | 85 tokens/s | - | Full setupOriginal source ↗ |
| M5 Pro 64GB | Qwen3.8 27B 27B oQ4e · Inco Splash 88,535 reported context | 78.6 tokens/s | - | Full setupOriginal source ↗ |
| AMD Strix Halo 128GB | Qwen3.8 125B Flash-Next UD-Q4_K_XL · Gufo 16,384 reported context | 34.1 tokens/s | 1,888 tokens/s | Full setupOriginal source ↗ |
| Intel Arc Pro B70 | Qwen3.8 27B 27B Q6 · llama.cpp 16,384 reported context | 56.5 tokens/s | 1,032 tokens/s | Full setupOriginal source ↗ |
Reported context may be a configured limit rather than the prompt length used for timing. Check the full setup for KV cache, offloading, batching and speculative decoding; missing details are unknown. What affects local model performance? →
On Qwen3.8 27B at 4-bit, plain single-GPU runs. Full ranking