Find which open-weight LLMs fit in your GPU or Mac and compare the speeds people report on setups like yours.
Pick your GPU or Mac, then read the speeds people reported on it, or estimate which models fit and how fast they run.
Free to use. No account needed. Memory estimates and community measurements are labelled separately.
Search for a model to see its reported speeds across GPUs and Macs.
These are individual setups, not a controlled benchmark. Compare GPU count, quantization, context and offloading before comparing speeds. How to read a report →
Exact recorded values. Context may be a configured limit; matching filters does not establish identical prompts, offloading or concurrency.
NVIDIA RTX 5060 Ti 16GB · llama.cpp · 81,920 ctx
User reports about 40 tokens per second running a Swift-1.5 Qwen3.8 27B IQ3_S GGUF on an RTX 5060 Ti 16GB. Setup is llama.cpp with MTP speculative decoding at 81920 context size, on a Ryzen 7 9800X3D with 32GB DDR5-6000. User asks whether this is the best they can do for general use, coding and agentic tasks.
NVIDIA RTX Pro 4000 Blackwell SFF · 8,000 ctx
User reports Qwen3.8 27B at 58.53 tok/s on an RTX PRO 4000 Blackwell SFF, up from 35.04 tok/s with DFlash2 speculation after power and clock tuning. Setup is NVFP4 weights at 8,000 input tokens with ECC off, best at a 120 W limit, VRAM +20% and core +180 MHz; the baseline was 70 W with zero offsets. A second model, Qwen3.6 35B A3B in Q4_K_S with no drafter, went from 84.97 to 115.56 tok/s at 120 W, VRAM +33.3% and core offset 0. The user notes 120 W exceeds the 75 W PCIe slot budget on this connectorless card and calls it an experiment, not a recommended daily profile.
M5 Ultra 96GB · Inco Splash · 262,144 ctx
User reports Qwen3.8 Flash-Next (Swift 1.5) at 144.6 tok/s decode and 3,444 tok/s prefill on a Mac Studio M5 Ultra 96GB, at a median context of 147.5K tokens. Setup is Splash with IQ3_S GGUF weights and a 262,144-token context window, driven by the DeepSeek Harness agent framework for agentic coding. With two concurrent requests combined decode is 200.4 tok/s (100.2 per request) and prefill 3,006 tok/s; with three it is 218.0 tok/s combined (72.7 per request) and 3,253 tok/s prefill, a result the user calls preliminary. The user is happy with the numbers and finds the model less lazy than Sol at troubleshooting.
NVIDIA RTX Pro 4000 Blackwell SFF · 8,000 ctx
User reports Qwen3.8 27B generation rising from 35.04 to 58.53 tok/s (+67%) on one RTX PRO 4000 Blackwell SFF after raising the power limit from 70 W to 120 W with VRAM +20% and core +180 MHz offsets. Setup used a modified VBIOS loaded into memory via a patched loader and NVIDIA open kernel modules, with no EEPROM flash; headline comparisons used 8,000 input tokens with ECC off. Only 1-3 requests per configuration across 158 total, and the 70 W baseline used the same modified VBIOS rather than factory firmware. Pushing VRAM from +20% to +33.3% slightly reduced 27B generation, 57.70 to 56.91 tok/s. Qwen3.6 35B A3B was almost flat between 106 and 120 W.
NVIDIA RTX 4090 · Strata · 145,000 ctx
User reports Qwen3.8-Flash-Next at 2076 tok/s prompt processing on a 145k-token prompt and 95-105 tok/s decode depending on task, with peaks of 145 tok/s, on a single RTX 4090 24 GB. Setup is Strata 0.1.43 with an IQ3_S pack, int8 KV cache, 204800 context, resident experts and MTP speculation, on a Ryzen 9 3900X with 48 GiB DDR4-3200 under Windows 10. A grid search over flags raised decode from 97 to 142 tok/s; the best arm was --kv-resident 20480 plus --adapt-async 1 and STRATA_ADAPT_LAG=2, giving 1526 and 2309 tok/s prompt processing at 10k and 20k. The user notes Windows does not pin memory with CUDA, so every host-to-GPU expert copy is CPU-driven.
2× NVIDIA RTX 3080 20GB · vLLM · 8,192 ctx
User benchmarks Qwen3.8 27B (Swift-1.5, AWQ-INT4) on dual RTX 3080 20GB cards in tensor parallel, reporting 2080 t/s prefill at 8192 prompt tokens with gpu-unlock plus P2P versus 967.1 t/s stock, a 115% gain. Setup is vLLM 0.32.0 with FP8 KV cache, MTP speculative decoding with 3 tokens, and both cards limited to 240W in PCIe 3.0 x16 slots. Prefill gains range from 70% to 115% across prompts of 8192 to 261900 tokens. Decode is reported as inter-token latency step rates of about 15 to 16.6 steps/s, which the user says corresponds to roughly 45 tok/s at an MTP acceptance length of 2; decode changes are within noise except at 261k tokens.
NVIDIA RTX 3090 · llama.cpp · 262,144 ctx
User reports Qwen3.8-27B at 66 t/s decode at 262,144-token context depth on one RTX 3090, with the full context fitting in 23.5 GB. Setup is a custom engine with unsloth UD-Q4_K_XL weights, K cache at 6 bits, V at 5 bits, and the newest 2,048 positions kept at q8_0, saving 1.1 GB versus a full q8_0 cache at 120k context. KV quality was scored by KL divergence against llama.cpp q8_0 output: 0.00160 at 64k versus 0.00163 for q8_0 and 0.00161 for fp16, +8% at 32k, and the same 230/282 on the author's question set. The author notes Qwen3.8 is a hybrid with only 16 of 64 layers attention, so full-attention models may differ.
AMD RX 9070 · llama.cpp · 262,144 ctx
User reports Qwen3.8-27B ASCII-Condensed UD-IQ4_XS at about 81 t/s code generation on an RX 9070 16 GB card with 32 GB system RAM, running on Windows. Setup is achyudh's llama.cpp ROCm KV-streaming fork with q8_0 K / q4_0 V cache, 262144 context, --kv-stream with 300/1800 MiB resident KV, MTP n=4 plus n-gram speculation, and 6 threads. Decode was 75 t/s at 9k and 95 t/s at 21k tokens, prefill about 1000+ t/s; these are single runs on short prompts, not filled 64k-262k contexts, and output quality was not checked. The user notes their numbers are below the fork README's ~100+ t/s at 16k-64k.
NVIDIA Tesla P100 16GB · Strata · 131,072 ctx
User reports about 30 tok/s with the Swift-1.5 version of Qwen3.8 Flash-Next on a Tesla P100 16GB in a PCIe3 x16 slot, after switching from llama.cpp to Strata. Setup is Strata with 64GB of ECC DDR4 at 2100 and a 131072-token context; the logs show expert cache hits of 63.5% to 77.8% with routed experts going over PCIe, so part of the model ran from system RAM. Individual runs in the pasted logs range from 28.7 to 33.3 tok/s on prompts of roughly 78k tokens, which were read in 2 to 3 s. The user says it serves their Hermes agent and calls it impeccable for a local LLM.
2× NVIDIA T4 16GB · ExLlamaV3
User benchmarks Qwen3.8-27B on two free Kaggle T4s (no P2P, 70W cap per card) with EXL3 4.0bpw under exllamav3, tensor parallel over PCIe, and a DFlash2 block drafter of 7 tokens. Overall throughput is 55-57 tok/s with drafting versus ~18-19 without; HumanEval reaches 154 tok/s and prose 35-37 tok/s. Prose acceptance is the limit at about 2.5 accepted tokens per step versus ~7 on code. On the same cards llama.cpp best was 16.5 tok/s and vLLM int4 with its own MTP drafter did 40.8 tok/s.
NVIDIA DGX Spark · llama.cpp · 16,384 ctx
User benchmarks a llama.cpp PR that fuses FFN gate/up matmuls and GLU in MMQ, on a DGX Spark (GB10) and an RTX 5090. Setup is llama.cpp with llama-bench, -ngl 999, flash attention on, 16384-token prompts, F16 KV cache, batch 2048, 5 repetitions. On the DGX Spark, Qwen3.8-27B Q4_K_M prompt processing rises from 840.1 to 899.9 tok/s (+7.1%). On the RTX 5090 the same model and quant goes from 3688.4 to 3790.9 tok/s. The table also covers Qwen3.6-27B and Llama-3.1-8B across Q4_K_M, Q5_K_M, Q6_K, Q8_0 and IQ4_XS; only prompt processing was measured (-n 0).
M5 24GB · Inco Splash · 256,000 ctx
User reports Qwen3.8-Flash-Next at 49 t/s decode on an M5 24GB MacBook Air. Setup is Splash 1.4.0 with a small fork, ISTA GSQ-RCO Q2_0 weights and a Flash-Next-DFlash2 draft model at 256K context. About 28% of expert slots (6,800 of 24,576, ~8.8 GB) stay resident and the rest are streamed from the SSD. Cold prefill without prompt cache is 522 t/s on a 32,620-token prompt and 485 t/s on 63,201 tokens. Real agent use in pi averages ~45 t/s with no prompt-cache losses, and draft acceptance is ~.70 for code and ~.80 for tool calls.
User reports Qwen3.8 Flash-Next at 164.6 tok/s decode on an RTX 5090 32GB, with prefill of 6,000~6,500 tok/s. Setup is Strata with an IQ3_S GGUF quant, on a Ryzen 9 9950X3D system with 64GB DDR5. The user previously ran IQ3_XSS and switched to IQ3_S for more speed and quality. The prompt asked for a Minecraft-style Eiffel Tower, which the model built in detail.
M5 Max 48GB · oMLX · 4,096 ctx
User benchmarks oMLX 0.7.2 serving Inco AI's Splash package for Qwen3.8-27B (4-bit target plus DFlash 2 draft) on a MacBook Pro M5 Max with 48 GB, using a Python code context and 128 generated tokens per test. At a 4096-token prompt the single request reaches 93.8 t/s decode and 1207.0 t/s prefill; longer prompts give 189.4 t/s decode at 16382 tokens and 110.2 t/s at 65536 tokens. Batched continuous batching reaches 413.7 t/s aggregate at 2x and 407.0 t/s at 4x, with peak memory between 19.1 GB and 22.2 GB.
NVIDIA RTX 4090 · ExLlamaV3 · 262,000 ctx
User reports Qwen3.8-Flash-Next 125B at 20.0 tok/s decode on an RTX 4090 at the full 262k context window, with experts streamed from NVMe SSD. Setup is ExLlamaV3 with a 3.05bpw_h5_ng5 quant and Q4 KV cache, 16 GB VRAM and only 8 GB system RAM allocated. At 100k context the same card gives 20.8 tok/s decode and ~1,536 tok/s prefill; short 8k context gives 28-33 tok/s prose and 19-22 tok/s code with ~1,080 tok/s prefill. An RTX 3080 Laptop with 16 GB VRAM and 32 GB RAM reaches ~18-21 tok/s decode and ~650-700 tok/s prefill at 100k context, peaking at 15.5 GB VRAM.
User reports Qwen3.8-Flash-Next 180B-A6B at 46-53 tok/s generation and ~770 tok/s prompt processing on a Strix Halo mini PC with 128 GB unified memory. Setup is llama.cpp and later the Strata engine, with UD-IQ4_XS GGUF weights in 3 shards at 94 GB, plus MTP speculative decoding and a vision projector. The same box ran Qwen3.6-35B-A3B at 76.5 tok/s, and that model did 11.7 tok/s CPU-only on the user's Ryzen 7 6800U laptop. The 46-53 tok/s figure is a range, so no single generation rate is recorded.
NVIDIA RTX 5080 · Basalt · 262,144 ctx
User reports Qwen3.8-Flash-Next IQ3_S on an RTX 5080 16 GB with a Core Ultra 9 285K and 62 GB DDR5-6800, averaging 154.9 t/s generation and 5,475 t/s prompt processing in a context sweep of fresh 15K-207K prompts. Setup is Basalt with IQ3_S weights, 262K context and int8 KV cache, using 15.7 GB VRAM and 54.1 GB system RAM. An agent coding session over 103 requests reached 153.0 t/s average generation with a 262 t/s peak, and 5,011 t/s average prefill; 99.5% of prompt tokens came from cache with a 0.2 s median time to first token.
NVIDIA RTX 5090 · SGLang
User reports Qwen3.8 27B at an average 208.2 tok/s on a single RTX 5090 32GB. Setup is SGLang in Docker with the gittensor NVFP4 build and the DSpark speculative drafter, served over an OpenAI-compatible API. Five runs of 256 tokens with thinking on ranged from 148.3 tok/s (cold start) to 232.4 tok/s, averaging 208.2; the user says it sits in the 220s once warm and peaks above 230.
AMD RX 7900 XTX · hipfire
User reports Qwen3.8 27B averaging 95 t/s decode and 1500 t/s prefill on a single RX 7900 XTX using a hipfire fork. No quant, context length or KV cache settings are stated. The user calls the result a removal of AMD's governor and is enthusiastic about it.
4× NVIDIA V100 16GB · NInfer · 256,000 ctx
User reports Qwen3.8 27B at 353 t/s on four V100 16GB cards. Setup is NInfer with NVFP4 weights, Q8 KV cache, 256k max context and MTP 4/0 speculation, tensor parallel across 4 GPUs. The user previously ran 2 V100s at 181 t/s and notes the repo now supports tp4.
On Qwen3.8 27B at 4-bit, plain single-GPU runs. Full ranking