Use the calculator to check which models fit in 24 GB of VRAM. The reports below include different GPU counts and memory setups; a model listed here may need extra cards or CPU offloading.
Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.
codinglong-contexttool-use
User reports Qwen3.8 Flash-Next at ~120-126 t/s on 3x RTX 3090 (72GB VRAM) with MTP speculative decoding.
Setup is TabbyAPI with ExLlamaV3 1.5.0, 3.05bpw exl3 quant, 262,144 context, and MTP draft mode. Without MTP the model runs at ~90 t/s; MTP adds ~35% with ~40% acceptance. First request after boot reads ~58 t/s due to warmup.
Needle-in-haystack at ~97k tokens found with ~31s prefill; 4 concurrent requests complete in ~4s each. Tool calling works and reasoning splits into reasoning_content.
Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.
codingtool-use
User reports Qwen3.8 27B at an average 100 t/s on an RTX 3090, compared to 20 t/s on a MacBook M5 Pro 48GB.
Setup is vLLM with a Q4 model on Linux, served over an OpenAI endpoint. The Mac run used llama.cpp with MTP and an Unsloth Q6 quant; Q4 on Mac made little speed difference.
User notes tool calls fail more often but the model self-corrects, and considers the setup good enough to replace a Claude subscription.
Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.
tool-usemathlong-context
User reports Qwen3.8 27B at 38.39 t/s generation and 871.93 t/s prefill on an RTX 3090 24GB, averaged over 3 hours of constant running.
Setup is vLLM 0.27.1 with the Club 3090 patch, INT4 AutoRound weights, FP8 E4M3 KV cache, 147,456 token context, single sequence, and AOT compilation on WSL2.
At 10K prompt the run reached 42.3 t/s generation and 1000.26 t/s prefill; at 90K prompt it reached 34 t/s generation and 743.59 t/s prefill. The user notes llama.cpp with Q5 UD GGUF at 81,920 context plus MTP gave 25-30 t/s, and that 154K context was the practical ceiling.
Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.
codingagenticlong-context
User reports GLM-5.3 Flash at around 28 t/s across four RTX 3090s at a quant of 3.
Setup is four RTX 3090 24 GB cards on a Threadripper PRO 3975WX with 256 GB RAM, PCIe only, no NVLink.
The user is planning a Qwen 3.8 Flash Next or Qwen 3.8 27B setup and asks for advice on model and hardware configuration.
Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.
agentic
User reports Qwen3.8 Flash-Next 177B-A6B at 38 t/s single stream on 2x RTX 3090 with an EPYC 7551 and 128 GB DDR4-2133.
Setup is llama.cpp with an IQ4_XS GGUF; experts live in system RAM with hot ones cached in VRAM.
Two parallel requests drop to about 4 t/s each. User asks whether a third RTX 3090 or a CPU upgrade would help more, and whether faster RAM matters.
Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.
User reports Qwen3.8 27B at around 70 t/s sustained on a 2x RTX 3090 build.
Setup uses the Q4 quant from the syv ais repo and config, with 64GB DDR4 4000MHz system RAM.
The figure is described as sustained throughput; the user offers more details on request.
Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.
User reports about 2 t/s running DeepSeek-R1-0528 Q3 GGUF on 2x RTX 3090 with 48 GB VRAM and 512 GB DDR5 system RAM.
No engine is named.
The user asks for upgrade advice to reach 15+ t/s with a very large model, and asks about Threadripper, EPYC and Xeon platforms.
Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.
codingsummarizationagentic
User compares ninfer-3090 against llama.cpp on an RTX 3090 for Qwen3.6-35B-A3B and Qwen3.8-27B.
The primary record is Qwen3.6-35B-A3B with ninfer-3090. llama.cpp with the UD-IQ4_XS GGUF reached 2439.7 t/s prefill, 148.6 t/s generation and 3410 ms TTFT. For Qwen3.8-27B, ninfer reached 187911.6 t/s prefill, 34.9 t/s generation and 27 ms TTFT, while llama.cpp reached 1012.7 t/s prefill, 38.5 t/s generation and 8160 ms TTFT.
The benchmark used 7 prompts (4 long at roughly 7.7k-12.9k tokens, 1 medium at roughly 2.3k, 2 short at roughly 400-600), 3 repeats each, max_tokens=1024, seed=42 and cache busting via nonce. The ninfer-3090 system is a Ryzen 5 2600 with 16 GB RAM; the llama.cpp system is a Threadripper PRO 3945WX with 128 GB RAM. The user notes the hardware is not apples-to-apples but that inference is GPU-contained.
Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.
User reports token generation improving from 20 t/s to 49 t/s with the flashnext-e06 fork on 2x RTX 3090, an Intel Ultra 7 270k Plus, and 192 GB DDR5@5600 MHz.
Setup uses an MTP draft model in Q8_0 and ngram-mod speculative decoding.
Prompt processing is faster on the main branch.
Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.
User reports prefill improvements of 2.2-2.5x on Qwen3.8-Flash-Next across 8k, 37k, and 119k contexts on 2x RTX 3090 with dual Broadwell Xeon and DDR4, with decode performance unchanged.
Setup is llama.cpp on a custom branch flashnext-e06, a 150-slot expert cache moved off GPU during prompt processing, 261k context, and an f16 KV cache. Hardware changed from 4 DIMMs to 6x32GB DDR4-2133 ECC.
Prefill is 223.7 t/s at 8k (was 99.9), 212.6 t/s at 37k (was 88.1), and 206.5 t/s at 119k (was 81.3). Decode is 34.3 t/s at 8k, 41.2 t/s at 37k, and 33.9 t/s at 119k. Time to first token improved from 82s to 37s at 8k, 424s to 176s at 37k, and 1461s to 575s at 119k. A quality screen showed no regression, and MTP acceptance is 0.79-0.83.
Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.
User reports a 285B MoE at 60+ t/s on 10 RTX 3090s and 120+ t/s on 12 RTX 3090s.
Setup is FP4 experts with FP8 attention and 157GB of weights, using TP2xPP5 on 10 GPUs at a 240W cap and TP4xPP3 on 12 GPUs.
The 60+ t/s figure is on 10 GPUs. Context runs to 1M without offload and 4M with RAM offload, with long-context prefill around 3500 t/s when batched. Vision, speculative decoding and tool calls work. A Docker image and repo are provided.
Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.
long-context
User reports median decode of 33.3 t/s with radix-select versus 30.2 t/s with the old top-k at ~119k context.
Quality screen scores 238/240 versus 235/240 correct.
No regression is detected.
Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.
long-context
User reports median decode improving from 30.2 to 33.3 t/s with radix-selection top-k fallback.
A quality screen scored 238/240 correct against 235/240 for the control, with no regression detected.
Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.
visionlong-contextagentic
User reports Muse Glimmer fits on a single RTX 3090 at Q4_K_XL with full context, DFlash and mmproj, using about 22-23 GB of VRAM.
Generation speed ranges from 64 to 124 t/s depending on prose versus code, with an average of 94 t/s. Prompt processing runs at about 1400 t/s.
The user notes Qwen3.6-27B and Gemma-4-31B are borderline unusable on the same GPU.
Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.
tool-use
User reports a custom harness called burrito fixes gpt-oss tool calling, running 320,192 evals, 8 seeds, 3.49B tokens and 1,062 GPU hours on a single RTX 3090.
Setup reaches about 200 tps at 128k context.
Effort levels affect accuracy: Low 38.3%, Medium 97.1%, High 100.0%. The user mentions llama.cpp and vLLM had issues with the model.
Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.
User reports GLM-5.2 UD-IQ2_M at ~7.3 tok/s decode and 135 tok/s prefill on 4x RTX 3090 (1 Ti + 3 FE), 96GB total VRAM with only ~83GB used.
Setup is llama.cpp master with the glm-dsa arch, a 223GB on-disk model at 744B total / 40B active, expert offload to CPU, explicit expert placement via -ot to avoid OOM, and a ~6GB KV cache at 128K that scales linearly.
Decode is bound by CPU compute, not bandwidth. IQ1_M and IQ2_M are identical in A/B tests (7.30 vs 7.29 tok/s), and 6 vs 12 CPU threads gave +22% (5.83 vs 7.14 tok/s). Thinking off gives ~5-6x wall-clock speedup.
Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.
coding
User reports Qwen3.8 27B at 25.6 t/s on an RTX 3090 at 131,072 context.
Setup is llama.cpp with the Huihui abliterated Q4_K_S GGUF and a Q4 KV cache, the whole 24 GB card in use.
A second run at 196,608 context averaged 25.8 t/s. The user also mentions a 5090 with Qwen3.6 27B at ~140 tok/s and DeepSeek V4 Flash on dual RTX 6000 at ~80-100 tok/s, but the primary benchmarked setup is the 3090.
Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.
User reports DeepSeek V4 Flash at 37.47 t/s pp and 6.62 t/s tg on a dual Xeon E5-2660 v2 with an RTX 3090.
Setup is the Q4_K_XL quant at 8k context.
The user also tested Cydonia v4.3, a Mistral Small 3.2 finetune, at 308.92 t/s pp and 7.8 t/s tg.
Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.
User benchmarks DeepSeek V4 Flash GGUF in IQ3_XXS-AS and IQ2_S quants on an RTX 3090 with 128 GB DDR4.
Setup is mainline llama.cpp b10064, which matches the fairydreaming fork's performance.
User prefers IQ3_XXS-AS as the sweet spot.
Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.
coding
User reports DeepSeek V4 Flash at 37.91 t/s prompt processing and 9.07 t/s generation on two RTX 3090 24GB cards and one RTX 5090 32GB card, 80GB VRAM total, with 128GB DDR4 RAM.
Setup is llama.cpp server with the unsloth Q8_K_XL GGUF, tensor splitting, and MoE layers offloaded to CPU.
User says performance is better than Qwen 3.5 v2 on coding tasks and asks for speed improvement suggestions.
Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.
User reports llama-bench results on 3x RTX 3090 with Q3_K_XL quantization.
The model is 119.40 GiB with 284.33 B params.
Prompt processing runs at 116.04 t/s and generation at 7.71 t/s.
Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.
User reports DeepSeek-V4-Flash-0731 UD-IQ3_S running under the llama.cpp loader in text-generation-webui.
Setup uses --n-cpu-moe 39 to offload MoE experts to system RAM, with the loader estimating roughly 136 GB total and relying on 128 GB of DDR5 at 5600 MHz.
Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.
User requantized only the 129 routed expert tensors of DeepSeek-V4-Flash-0731 to IQ3_XXS, with down projections at IQ3_S, leaving attention, shared experts and router at Q8_0, BF16 and F32. The result is 111.37 GiB in four shards.
Against the MXFP4 source, KLD is a mean of 0.2386 versus 0.2936 for UD-IQ3_S, top-1 is 84.65% versus 82.78%, and delta PPL is +0.536 versus +0.685, but the requant is 2.12 GiB larger and has a worse max KLD of 12.53 versus 11.13.
On 5 mixed GPUs (2x 3090, 5060 Ti, 2x 4060 Ti, 96 GiB VRAM) with expert spill to CPU, speed is 13.91/13.57/13.26 t/s at depths 0/4096/16384 versus 9.88/9.69/9.51 for the MXFP4 source, roughly 1.4x. antirez's flat Q2 is 80.76 GiB, about 98% resident, at 30.27 t/s. The user notes known DSV4 SWA/rollback stall issues in llama.cpp and used a patched build.
Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.
User reports 1M lossless context on 3x RTX 3090 with DDR5, at 15 t/s on cold start and 10 t/s steady.
Setup is a llama.cpp fork with deepseek4 architecture support. A fourth RTX 3090 runs gemma 12b and flux2klein.
Prefill starts at 300 t/s and degrades to 100 t/s after 64,000 tokens.
Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.
User compares four quants of a model on a single RTX 3090 at 128K context, with the best speed around 14.9 t/s on UD-IQ2_M.
Setup uses --n-cpu-moe to offload MoE experts to CPU, with Text Generation WebUI as the frontend.
User compares a W4A16 quant of Nemotron 3.5 Lightning 30B-A3B against an IQ4_XS GGUF on an RTX 3090.
Setup is vLLM for W4A16 and llama.cpp for IQ4_XS.
The two are near-parity in instruction following benchmarks, with about 4.5x throughput by B16. The user describes the model as fast and reliable, and suitable for batch labelling and agentic responses.
Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.
User reports 34-36 t/s on an RTX 3090 with Qwen3.8 27B.
Setup is llama.cpp with the IQ4_NL quant and a Q8 KV cache.
The user compares this to 50-60 t/s on Qwen3.6 27B.
Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.
coding
User reports Qwen3.8-27B at 73 t/s on code and 58.5 t/s on prose on dual RTX 3090s (24 GB each, 48 GB total).
Setup is a Q8_K_XL quant (31.5 GB) with mmproj-F16 (0.93 GB), 200K context and an f16 KV cache, tensor split 50/50. MTP draft depth 2 gives the best acceptance at 92% on code and 68% on prose.
The speeds are medians of 3 draws at temp 0. The hybrid architecture has 48 of 64 layers as Gated DeltaNet (linear attention), with only 16 full attention layers plus the MTP head, for 17 KV-caching layers. Cold start is 32s. The split is ~1.1 GB lopsided, causing OOM risk at 220K context. User asks for optimization suggestions for dual 3090 setups.
Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.
User reports Qwen3-35B-A3B at about 260 t/s single-stream on an RTX 3090, and over 400 t/s on repetitive workloads.
Setup is NInfer, a custom C++/CUDA inference runtime, with CUDA Graphs enabled and 1024 output tokens per run.
The 3090 port targets Ampere GPUs, and the figures are from sustained runs.
Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.
User reports Qwen3.8 27B at 82 t/s on an RTX 3090.
Setup is llama.cpp with a Q4_K_S GGUF and a Q4 KV cache.
The 82 t/s is single request generation speed. A 672 t/s peak is not counted because it is unclear whether it is generation or prompt processing.
User benchmarks Qwen3.8-27B on 4x RTX 3090 (24 GB each) with vLLM 0.20.1, reaching 587.4 tok/s at c32 with TP=2 on one NVLink pair versus 417.9 tok/s with TP=4.
Setup uses tensor parallelism across the cards, with a KV pool of 1.45M tokens on TP=4 and 503k tokens on TP=2, and a max context of 262k.
MTP speculative decoding is slower at all concurrency levels, and pipeline-parallel throughput stays flat with concurrency.
Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.
User reports Qwen3.8-27B at about 124 t/s greedy on a single RTX 3090.
Setup uses fp8 KV cache, int8 lm_head and embed_tokens, MTP-4 drafts, GPTQ-int4 lm_head and MTP module, a split-KV attention kernel, and KVarN 4/2-bit KV cache for 262k context.
Peak concurrent throughput is about 1000 t/s at 64 concurrent.
Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.
User reports Qwen3.8 27B at 82 t/s single request on an RTX 3090, with up to 672 t/s peak and 417 t/s sustained at 64 concurrent.
Setup is vLLM with W4A16 quantization, an FP8 KV cache, and int8 lm_head and embed_tokens, at up to 195k context.
The title says Qwen3.8-27b while the body mentions Qwen3.6-28B.
Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.
creative-writing
User reports Qwen3.8 27B at about 40 t/s for SVG generation on 2x RTX 3090.
Setup is the cyankiwi/Qwen3.8-27B-AWQ-INT4 model with TP2 and an FP8 KV cache.
Speed started near 60 t/s and dropped to the 20s after the user removed a power limit mid-generation. The run took 20 minutes, 12 of them thinking, and the user was impressed with the result.
Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.
User reports Qwen 3.8 27B at 100+ t/s generation at high context on 4x RTX 3090, each throttled to 200W.
Setup is vLLM with the flashinfer backend, INT8/W8A16, and MTP.
User adds that Gemma models also benefit, but vision models need --language-model-only.
Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.
User reports Qwen3.8-27B at 99 t/s single request and 1150 t/s batched on an RTX 3090.
Setup uses a hyper-optimized inference engine with an fp8 KV cache, int8 lm_head and embeddings, fp16 recurrent state, and int8 activations, at 64k context in fast mode.
Prefill reaches 1812 t/s at 1k context and 1000 t/s at 100k context.
Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.
User reports 120.1 t/s decode on narrative text and 218.3 t/s on code with vLLM v0.26.1rc1 and AutoRound INT4 (group 128) on 2x RTX 3090 over PCIe Gen4 x16/x16 with no NVLink and patched P2P.
Prefill reaches 1342 tok/s at 10k and 628 tok/s at 90k. Speculative decoding uses 7 draft tokens with an acceptance length of 3.35 and 47.8% acceptance.
Peak VRAM is 22.3 GB per card with zero leak, and the context ceiling is 131k because the DFlash2 drafter takes about 13.5 GB. Power is capped at 220/250 W, and the user used Kimi K3 for vLLM fixes.
Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.
User reports Qwen3.8-27B at 55 tok/s on an RTX 3090.
Setup is the IQ4_XS quant.
The user also mentions GLM-5.3 pricing and intelligence, DeepSeek V4 Flash, and Strix Halo hardware, but does not run them.
Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.
User reports Qwen3-235B-A22B at about 7.5 t/s generation and about 75k pp on a single RTX 3090 with 128 GB of system RAM.
Setup is textgen with batch size 1024, 16384 context, autofit layers, and 38 of 48 threads on a dual Xeon DDR4 system.
Generation dips to about 7 t/s at 10k context. The user calls it a daily driver for months for uncensored ERP roleplay, and mentions Step 3.7 Flash as an alternative that is censored and verbose, along with Gemma 4 and MiniMax-M2.5, which were not run.
User reports Qwen3.8 27B Instruct scoring 36 and 34 composite on two ACT practice tests, with perfect reading scores.
Setup is 2x RTX 3090 with full offload and 32k context.
The two tests took 177 minutes in total.
Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.
agenticcoding
User reports 50-65 t/s generation on dual RTX 3090, sometimes dropping to 40s.
Setup is llama.cpp with the Qwen3.8-27B-UD-Q6_K_XL GGUF, --mmproj for vision, --kv-unified, --flash-attn, and q8_0 KV cache types, plus speculative decoding with draft-mtp.
The use case is agentic coding.
Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.
User reports Qwen3.8-27B at 382 t/s on an RTX 3090 when reproducing context, and about 133 t/s on real chat prompts.
Setup uses an fp8 KV cache, int8 activations, DFlash2 block drafting, lookup-augmented drafting, prefix caching, split-KV verify attention, and KVarN for 262k context.
Quality is unchanged, with GSM8K at 96.5%.
Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.
codingagentictool-use
User reports sustained 80-90 t/s decode at full 262K context.
Setup uses MTP speculative decoding and prefix caching, with a 96.8% prefix cache hit rate and mean TTFT of 6.5s.
Multi-session throughput reaches about 3200 tokens/min across concurrent chats, and the benchmark ceiling is 106.9 t/s.
Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.
codingagentic
User reports DFlash2 speculative decoding with LMCache KV offload, with decode speeds varying by task: code 268 t/s, agent 165 t/s, mixed 97 t/s, prose 76 t/s.
Prefill runs at 15-17k t/s, and the KV cache pool holds 18M tokens on NVMe L2.
The setup required patches to avoid corruption.
User reports running GSM8K, MATH-500, HumanEval and MBPP locally.
User found reasoning stalls causing significant accuracy drops on hard math.
User notes the HumanEval score is possibly inflated due to contamination, and that harness bugs were fixed.
Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.
User compares llama.cpp built with -DGGML_CUDA_NCCL=OFF against the same build with it ON on 2x RTX 3090.
Setup is the unsloth/Qwen3.8-27B-GGUF UD-Q6_K_XL model at 229k context with an F16 KV cache, tensor split 24,24, and speculative decoding with draft-mtp.
NCCL ON degrades performance: PP 1046.92 vs 958.58 t/s and TG 59.67 vs 54.80 t/s.
Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.
User compares two llama.cpp setups on the same GPU: Q5 with a Q8 KV cache, 32K context, 4 slots and MTP off at about 35 t/s, against Q4 with a Q4 KV cache, 131K context, 1 slot and MTP on at about 65 t/s.
User notes the trade-offs between the two configurations.
Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.
User reports Qwen3.8-27B at 560/30 t/s on 2x RTX 3090.
Setup is Q8_0 quantization, run fully in VRAM.
The user calls it the local king and praises its quality. Other models mentioned are DeepSeek-V4-Flash, GLM-5.3-Flash, and Qwen3.8-Flash-Next.