Local model performance on your hardware

generation:

701.2 tokens/s

quant:

UD-Q4_K_XL (GGUF)

kv:

Q8

flash attention:

on

mtp (multi-token prediction):

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8 27B at 701.2 t/s on one NVIDIA RTX 5090, the median of three repeats (695.3, 709.0, 701.2) for a copy/edit task from a 27,135-token prompt with 400 completion tokens. Setup is llama.cpp b11060 CUDA with UD-Q4_K_XL, q8_0 key and value cache, Flash Attention, greedy sampling, and draft-mtp plus ngram-mod speculation with draft maximum 16 and n-gram range 96-192. The 131,072-token context is the server limit, not the measured prompt length, and the figure is not general chat or novel-code speed. At 3,038 prompt tokens the same profile returned a 784.1 t/s median (range 768.0-814.3).

generation:

563.1 tokens/s

prompt processing (prefill):

30680.7 tokens/s

quant:

Q8_0 (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports LFM2.5-8B-A1B at 563.12 t/s generation and 30,680.66 t/s prompt processing on an RTX 5090. Setup is llama.cpp llama-bench (build b9653) with Q8_0 weights, all layers on GPU (ngl=99), single request, no speculative decoding. The dataset card also lists llama-bench rows for gpt-oss-20b, Nemotron-3-Nano-30B-A3B, Qwen3.6-35B-A3B, Qwen3-Coder-Next, Gemma 4 12B, Qwen3.6-27B, Gemma 4 31B, and Ling-3.0-flash, plus rows for Qwen3.6-27B-MTP, Qwen3.6-27B Q6_K, gpt-oss-120B, Mistral Small 4 119B, and several quant ladders; some runs used vLLM and SGLang.

generation:

352.1 tokens/s

quant:

Q4_K_M (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User benchmarks Qwen3 MoE 30B-A3B at 352.06 t/s on an RTX 5090. Setup is llama.cpp with Q4_K_M and flash attention, 99 layers offloaded to GPU. This is the optimized run with kernel fusion and concurrent CUDA streams enabled; baseline was 246.96 t/s. The post also reports gpt-oss 20B MXFP4 at 419.14 t/s on the 5090 and both models on an RTX 4090.

generation:

94.0 tokens/s

quant:

IQ4_XS (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B at 94 t/s on an RTX 5090. Setup is llama.cpp with IQ4_XS GGUF quantization, single stream, flash attention on, thinking mode sampling. With MTP speculative decoding it reaches 147 t/s (1.57x) and with DFlash2 167 t/s (1.78x). The model is 13.1 GB at 3.84 bits per weight.

generation:

22.0 tokens/s

prompt processing (prefill):

421.1 tokens/s

quant:

IQ4_XS (GGUF)

kv:

f16

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports GLM-5.3-Flash at 21.96 t/s decode (narrative) and 26.10 t/s (code) on 2x RTX 5090. Setup is llama.cpp with IQ4_XS GGUF weights, f16 KV cache, 204800-token context, DFlash2 Q4_K_M draft model with speculative decoding (n=3), and MoE expert tensors offloaded to CPU. Prefill measured 421.07 t/s at ~10K prompt tokens and 448.20 t/s at ~87K. Soak test passed with p50 decode 20.70 t/s and 100% TPS retention over 25 turns. Expert cache was requested but not allocated (CACHE_DISABLED), so the run measured a half-cached path. Stress test failed the long-context recall ladder.

generation:

148.0 tokens/s

quant:

NVFP4 (NVFP4)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8 27B at roughly 148 tok/s on an RTX 5090 with MTP enabled. Setup is llama.cpp with NVFP4 weights and MTP speculative decoding on a single 32GB card. Without MTP the same model ran at about 72 tok/s, and a Q4_K_M build at about 70 tok/s. Other models tested: GPT-OSS 20B MXFP4 at about 311 tok/s, Qwen3.6 hybrid at about 250 tok/s, Gemma 31B QAT at about 68 tok/s, and Flash Next at about 199 tok/s.

generation:

68.0 tokens/s

quant:

Q4_K_M (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User compares Qwen3.8 27B across four setups on an RTX 5090 at 120k context, reporting decode speeds of ~147 t/s (ninfer non-NVFP4 with MTP), ~141 t/s (llama.cpp Q4_K_M with MTP), ~155 t/s (ninfer NVFP4 with MTP), and ~68 t/s (llama.cpp Q4_K_M without MTP). Setup is llama.cpp with Q4_K_M GGUF, thinking on at xhigh, default sampling, one run per setup. Stock llama.cpp without MTP is less than half as fast as ninfer, but MTP lifts it from 68 to 141 t/s. NVFP4 had the highest t/s but wrote the most tokens, finishing third on wall-clock time. All four runs produced working games; only the NVFP4 build could be won legally. User is sticking with the non-NVFP4 ninfer build.

generation:

143.0 tokens/s

prompt processing (prefill):

2840.0 tokens/s

quant:

Q6_K (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8 27B Swift at 143 tok/s decode and 2840 tok/s prefill on an RTX 5090. Setup is llama.cpp with Q6_K quantization. The figures come from a coding prompt asking for an HTML/JS calculator, which produced a working result in 1 minute 10 seconds; the user calls this the ceiling after two days of trials.

generation:

48.0 tokens/s

prompt processing (prefill):

1429.0 tokens/s

quant:

UD-Q2_K_XL (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-Flash-Next at 48.02 t/s short decode and 1429 t/s prefill on an RTX 5090 32GB with 128GB DDR5 system RAM. Setup is llama.cpp PR #27742 with UD-Q2_K_XL (78.9GB in 3 shards), 64k context, -ngl 999 and -ncmoe 24 keeping the first 24 layers' experts on the CPU, no speculative decoding, single stream. Decode after about 16k tokens of input is 43.28 t/s, time to first token is 11.0 s at about 16k input, and VRAM use is about 30.9GB.

generation:

98.0 tokens/s

prompt processing (prefill):

2829.3 tokens/s

quant:

UD-Q6_K_XL (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8 27B at 98.03 t/s generation and 2829.27 t/s prompt eval on an RTX 5090. Setup is llama.cpp with UD-Q6_K_XL GGUF, 48000 context, draft-mtp speculative decoding (acceptance 0.47494, mean len 2.42). User notes only 48k context fits on the RTX 5090 with this quant.

generation:

5.1 tokens/s

quant:

MXFP4 (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User ports DeepSeek-V4.1-Flash (552B) to a llama.cpp fork and runs it on a single RTX 5090 with 31.8 GiB VRAM and 125.7 GiB RAM, streaming routed experts from NVMe through a VRAM cache and pinned host tier. Reports 5.1 tokens/s on new content and 21 tokens/s on content already resident in the cache. Setup uses MXFP4 experts as a lossless repack of the released blocks, with the engram tables (189 GiB of n-gram memory) kept on disk and read by the host. Logit correlation against the reference at 1401 tokens is 0.9967. Measured ceilings are 6.2 tokens/s with no disk miss and 21.3 tokens/s with everything resident. Per remap call at a 72 GiB host tier: 20% compute, 26% PCIe, 54% NVMe. A prefetch oracle gives +30% only at 5 tokens of look-ahead; block verification with a perfect draft gives +17-20%; an n-gram draft loses 33% on realistic content; the DSpark draft head shows 51-97% acceptance but is neutral on mixed content.

generation:

194.0 tokens/s

quant:

Q4_K_XL (GGUF)

kv:

q8_0

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User compares Qwen3.5-35B-A3B Q4_K_XL on an RTX 5090 against a Radeon AI PRO R9700, both 32 GB, using llama-bench with -ngl 99, flash attention and q8_0 KV cache. On the RTX 5090 generation is 194.0 t/s and prompt processing is 7,026 t/s at 512 tokens, 6,953 at 1,024, 6,958 at 4,096 and 6,461 at 32,768. The R9700 reaches 127.4 t/s generation and 2,713 / 2,643 / 2,546 / 1,877 t/s prompt processing at the same context lengths, a 1.52x generation and up to 3.44x prefill gap. The user notes these are single-stream llama-bench figures and calls the R9700 comfortable for interactive use at roughly 2.66x lower cost.

generation:

20.0 tokens/s

quant:

UD-Q4_K_XL (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8 Flash-Next at approximately 20 tok/s single-stream decode over 256 output tokens on an RTX 5090 with 128 GB system RAM. Setup is llama.cpp with GGUF UD-Q4_K_XL, with the model partly offloaded to system RAM. With 4 parallel requests of roughly 4K prompt plus 256 output, aggregate end-to-end throughput is 13.8 tok/s; with roughly 8K prompt plus 256 output it is 8.81 tok/s. The user notes these aggregate figures are a different measurement from the single-request rate and should not be read as single-stream speed.

generation:

40.0 tokens/s

quant:

Q4_K_M (GGUF)

flash attention:

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8 Flash-Next at 40 t/s decode on an RTX 5090 with 64GB system RAM. Setup is llama.cpp with the Atomic Q4_K_M GGUF at 64,768 context, using mmap, lazy mode, flash attention, all GPU layers, and 32 MoE layers offloaded to CPU. The run used 27GB of VRAM and 8GB of system RAM. User notes the prompt processing figure of 50 t/s is likely inaccurate due to the short prompt, and questions why the model is not buffering more into system RAM instead of reading from the NVMe drive.

generation:

97.9 tokens/s

prompt processing (prefill):

2364.0 tokens/s

quant:

Q6_K (GGUF)

kv:

q8_0

flash attention:

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8 27B at 97.9 t/s decode and 2,364 t/s prefill on an RTX 5090 at 61,136 tokens of context. Setup is llama.cpp with the Unsloth UD-Q6_K GGUF, flash-attn on, q8_0 KV cache, batch 2048, ubatch 1024, and MTP speculative decoding with draft 3. A second run at 97,374 tokens averaged 86.0 t/s decode and 1,929 t/s prefill. The user also tested Qwen3.8-Flash-Next Q4 across both GPUs and Qwen3.8 27B Q8 split across both cards at 262k context.

generation:

90-100 tokens/s

prompt processing (prefill):

3000.0 tokens/s

quant:

Q2 (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8 Flash-Next at 90-100 t/s decode and 3000 t/s prefill on two RTX 5090s across two machines. Setup is llama.cpp with RPC over a ConnectX-5 100G link, using Unsloth Q2 quants. User sees little practical reason to use this over the 27B model and asks what others get with 2x 5090 in one machine or a 6000 Pro.

quant:

Q6_K (GGUF)

kv:

q8_0

User reports Qwen3.8-27B RVN Heretic (ARA abliterated) at 140-190 t/s on code and ~100 t/s on prose on a single RTX 5090 at 159,744 context. Setup is llama.cpp master with the Q6_K GGUF, q8_0 K/V cache, DFlash2 speculative decoding at n=4, vision projector on GPU, and Sharp chat template. Without speculation the same model runs at 62 t/s. Cold prefill measured without the drafter: 32K in 11.7 s, 64K in 28.6 s, ~113K in 60 s. VRAM is 29,996 MiB after load and 30,332 MiB after the first image. Tool calls parse and images read correctly with no Xid.

generation:

253.0 tokens/s

quant:

UD-Q5_K_M

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Muse Glimmer 30B at 253 t/s on an RTX 5090 at 262k context. Setup is llama-server with PR #26842 and DFlash, using a dflash draft model. The run used a code patch workload. The user notes Meta's published figure of 233 t/s.

generation:

13.7 tokens/s

prompt processing (prefill):

159.0 tokens/s

quant:

mixed Q8/Q4/Q2 (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports DeepSeek V4 Flash at 1M context on an RTX 5090, with prefill at 159 t/s and decode at 13.7 t/s. Setup is a patched llama.cpp, with peak VRAM around 31 GiB. The user verified the run with a needle-in-haystack test.

generation:

21.3 tokens/s

prompt processing (prefill):

1105.0 tokens/s

quant:

Q2_K (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports DeepSeek V4 Flash on an RTX 5090 with a llama.cpp fork, at 22.7 to 21.3 t/s TG and 1105 to 927 t/s PP across prompt lengths. Setup is the Q2_K quant at 1M context, with CPU offloading for the MoE layers.