Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports 294.1 t/s generation.
Source evidence (verbatim):
Source: https://dev.to/sikamikanikobg/vllm-vs-llamacpp-vs-ollama-what-happens-when-your-model-doesnt-fit-in-24gb-vram-56eb
# vLLM vs llama.cpp vs Ollama: What Happens When Your Model Doesn't Fit in 24GB VRAM
Hardware: NVIDIA RTX 3090 24GB, 128GB system RAM. Concurrency 1 decode tok/s (llama.cpp / vLLM): Gemma 3 1B 294.1 / 235.5. Qwen3-Coder 30B-A3B 157.2 / 172.0 (vLLM AWQ). Gemma 4 26B-A4B 118.8 / 133.8. Ollama at concurrency 1: 125.6, 129.3, 84.5. Single-stream steady-state decode on the longest prompt tier with models that exceed VRAM (llama.cpp -ngl 12 for gpt-oss-120b): GPT-OSS-120B 7.65 tok/s llama.cpp, 7.6 Ollama. GLM-4.5-Air 4.58 llama.cpp, 4.59 Ollama. vLLM ran out of memory on both. Engine versions and context length not stated.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
codingtool-useagentic
User benchmarks Nemotron 3.5 Bolt 74B-A9B at 85.2 tok/s decode and 833 tok/s prompt on an RTX 3090, against Qwen3.8 27B Q4_K_M at 41.5 tok/s decode and 963 tok/s prompt.
Setup is llama.cpp with IQ2_XXS GGUF weights, q8_0 KV cache, 16K context, all 89 layers on GPU with no CPU offload, 21.3GB peak VRAM. No speculative decoding.
On a 5090 the same models ran at 148.3 vs 72.5 tok/s decode. Qwen passed more tasks on both cards (89/120 vs 75/120 on 3090, 84/120 vs 71/120 on 5090); the gap was almost entirely tool calling, where Nemotron had 20 truncations at a 512-token per-turn cap. Median time-to-finished-task was a tie because Nemotron writes about 1.5x more tokens per task.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports 123.49 t/s decode on a repasted Dell RTX 3090, up from 86.06 t/s before the repaste, with a Founders Edition RTX 3090 also in the system.
Both cards ran at stock clocks and 280 W. The Dell card's hotspot dropped from ~104 °C to ~83 °C and thermal throttling stopped.
The model, engine, quant and context length are not stated.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
coding
User reports Qwen3.8-27B at 151.3 tok/s greedy decode on short prompts on one RTX 3090, using a fork of the open-jet engine with the model's own MTP head (7 drafts) and a custom D' KV cache (K 6-bit, V 5-bit, 8-bit ring for the newest ~2.5k tokens).
Setup is the open-jet fork with UD-Q4_K_XL weights at 120k context, single stream, one request at a time. The same engine with q8 KV gives 150.2 tok/s; llama.cpp tuned with MTP 4 drafts and q8_0 KV gives 82.7 tok/s on the same card.
Sampled thinking runs at 134.9 tok/s, decode at 38k context at 114.3 tok/s, and at 101.6k context at 105.4 tok/s. The full 262,144-token context fits on the card at 23.4 GB, reading 260,896 tokens in 334 s (781 tok/s) then decoding at 63 tok/s. Greedy token ids are identical at draft depths 0, 4 and 7, so output is unchanged with speculation off. The user notes the card is an ASUS Strix 3090 with a raised power limit (~470 W) and memory at 9,851 MHz, and that numbers will differ on other hardware.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
codingmath
User reports Qwen 3.8 27B MTP Q4 at 60 tokens/s on 2x RTX 3090 with 256GB RAM and an EPYC 7443.
Setup is Ollama with shared scheduling across both cards; a single card gave 40 tokens/s.
User asks whether this is the sweet spot for the hardware and whether an NVLink bridge would help further.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
codingmathlong-context
User reports Qwen3.8 Flash-Next at around 100 tokens/s on 2x RTX 3090.
Setup is Strata with IQ3_S quant. User is happy with the speed and says quality is a bit better than 27B at Q8.
User also owns five more RTX 3090s (one ASUS ROG Strix, four Zotacs) and is deciding whether to upgrade the X299 machine, build a second 4x3090 rig, or sell the extra cards.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports around 100 t/s running Qwen3.8 Flash-Next with QFN on Strata using 3x RTX 3090 at PCIe 4.0 x16 each.
Setup is Strata with QFN quant on three RTX 3090s.
User asks for advice on adding a fourth RTX 3090, considering PCIe bifurcation or a motherboard upgrade, and is concerned about x8 performance impact on Strata.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8-27B at 32 tok/s on a single RTX 3090 at 300k context.
Setup uses Strata on the RTX 3090; a Titan RTX 24GB sits idle in the same machine, with 48GB of free system RAM.
User asks whether Strata's multi-GPU support would hurt prefill speed when adding the second card without NVLink.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8 Flash-Next at ~62 t/s decode and ~1400 t/s prefill on a single RTX 3090 with 8-channel DDR4 server RAM.
Setup is Strata with IQ3_S GSQ-RCO quant and 256K INT8 KV context at a 220W power cap.
User also mentions GLM 5.3 Flash "Maya L" 156.3GB running on a single Tesla V100 32GB via Project Maya, but gives no throughput figure for it. User asks what benchmarks to use to compare the two models on quality.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8-27B at 66.39 tok/s on one RTX 3090, versus a 41.55 tok/s non-speculative baseline, a +59.8% within-host effect.
Setup is llama.cpp with Q4_K_XL GGUF weights, q8_0 KV cache, 8192-token context, flash attention on, single stream, batch size 1, using the built-in MTP path at --spec-draft-n-max 2.
The faster path was not bit-exact with serial greedy decoding: 92-100% of requests diverged by a 1,600-token cap, and semantic quality was not measured. Deeper drafting was slower (MTP n-max 3 at 63.29 tok/s, DFlash2 n-max 4 at 63.13, MTP n-max 5 at 54.89, DFlash2 n-max 7 at 50.95). Board telemetry estimated a 37% decode energy reduction for a fixed 400-token workload.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
math
User reports running a quant of Qwen3.8 27B on an RTX 3090, getting around 50 t/s. The model was benchmarked on a noisy data science prediction problem against cloud models Luna 5.6 and Astra, where it performed better than Luna and competitively with Astra. The user notes the local model took 5-15 times longer than the cloud models.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
long-context
User reports Qwen3.8-Flash-Next 125B-A6B at IQ3_XXS running at the full 262,144-token context on a single RTX 3090 with 32 GB of DDR5, using the Strata engine with int8 KV cache and MTP speculation. On a 121,566-token prompt the run reached 1325 tok/s prefill and 59-62 tok/s decode; a 249K-token prompt gave 1008 tok/s prefill and 41.8 tok/s decode with 90.5% expert-cache hits, and short-context decode climbs to 63-100 tok/s.
Setup uses --resident-cpu-experts with a 21 GiB resident budget, 12.2 GiB of experts in VRAM, 21 GiB locked in RAM and 6.6 GiB left on disk, covering 83% of routings without disk reads. The process is pinned to P-cores with taskset -c 0-15 and --pool-affinity p-cores; --prefill auto:16384 raised prefill from 1014 to 1325 tok/s on the 121K prompt. --aux-cpus gave no gain on this chip (60.1 vs 59.6 tok/s decode).
The user asks whether these numbers are good and what others get on comparable hardware, and notes the KV cache is about 3.1 GiB at 262K because only 12 of 48 layers keep growing KV.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
coding
User benchmarks Qwen3.8 27B at 70 tok/s average on an RTX 3090, running in llama.cpp with 119K context and MTP enabled.
Setup is llama.cpp with MTP speculative decoding and 119K context.
The user also tested Qwen3.8 Flash-Next in Strata at 128K context, averaging 120 tok/s, and GPT 6.1 Sol (High reasoning) which one-shot the prompt.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8-Flash-Next at 57.6 t/s decode and 6574 t/s prefill on six RTX 3090s.
Setup is vLLM with a GPTQ-INT4 checkpoint and int8 KV cache, 250k context, PLE table on SSD, pipeline parallelism across six cards.
Decode falls to 29.4 t/s per user with seven concurrent sessions (206 t/s aggregate); llama.cpp-multigpu holds 30 t/s per user at five sessions. NVFP4 checkpoints are 10-12% slower than GPTQ on these cards. Load to healthy is 4-5 minutes.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Hauhau's Qwen3.8 27B Uncensored at around 20 tok/s on a single RTX 3090 with MTP enabled.
The base Qwen model outputs around 30-35 tok/s on the same card.
User asks why the uncensored fine-tune is substantially slower than the base model.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
coding
User reports Qwen3.8-27B at 140 tok/s on code generation on a single RTX 3090, using a custom CUDA megakernel that runs each speculative-decoding cycle in one launch.
Setup is the megakernel with Q4_K_M weights and fp16 KV cache up to about 80k context, then q8. The same hardware with llama.cpp and MTP gives 73 tok/s on code, 56 tok/s on coding with a 4K-token file in context, and about 1,100 tok/s prefill.
The megakernel is 1.4-1.9x faster than llama.cpp depending on task and context length, and is built for this model on 3090s only.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8 27B Swift 1.5 at almost 100 tok/s decode on 2x RTX 3090.
Setup is vLLM with an 8-bit AWQ + GPTQ W8A16 quant, 128k context and FP16 KV cache, MTP=5, around 21.6 GB used per card.
The quant was made on 4 RTX 3090; the user says it is less of an overthinker than the original Qwen3.8 27B.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports GLM-5.3-Flash at 19.1 t/s decode and 41 t/s prefill on an RTX 3090 with a 47-token prompt.
Setup is Strata with GSQ-RCO 3.5-bit GGUF at 32,768 tokens of context, most experts in RAM (121 GiB of 177 GiB system RAM used).
Prefill rises with prompt length to 974 t/s at 26,641 tokens; decode stays 18.7-19.0 t/s across contexts. On 9 GPUs (8x RTX 5060 Ti 16 GB plus the 3090) decode is about 28 t/s and 8K prompts run 1150-1210 t/s. Calibration kept the engine's own settings at 21.6 t/s.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports GLM-5.3-Flash at 28.55 tok/s decode on an RTX 3090.
Setup is ExLlamaV3 with exl3-3.05bpw and 131k context, all-RAM fast mode with CPU offload requiring ~238 GB free RAM.
Lab suite passed 6/6 with 28.6 tok/s decode and 989 tok/s prefill; at concurrency 4 the aggregate was 28.62 tok/s (28.45 per stream) because fast mode decodes one stream at a time.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports GLM-5.3-Flash at 17.8 tok/s decode and 996 tok/s prefill on an RTX 3090 with the 55 GB NVMe offload launch.
Setup is the glm53-flash-offload v4.4-nvme image, with the model's weights streamed from NVMe (55 GB) plus 16 GB resident; the 16 GB launch measured 13.3 tok/s decode and 986 tok/s prefill and failed the 15 tok/s speed gate.
Previous runs were 17.3 tok/s for 55 GB and 13.1 tok/s for 16 GB. The 55 GB launch passed all gates and was accepted; the 16 GB launch was reported. Host settings outside the launch hash were --gpus device=0 and --cpuset-cpus 2-39 with no read cap.