Local model performance on your hardware

generation:

66.4 tokens/s

quant:

Q4_K_XL (GGUF)

kv:

q8_0

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B at 66.39 tok/s on one RTX 3090, versus a 41.55 tok/s non-speculative baseline, a +59.8% within-host effect. Setup is llama.cpp with Q4_K_XL GGUF weights, q8_0 KV cache, 8192-token context, flash attention on, single stream, batch size 1, using the built-in MTP path at --spec-draft-n-max 2. The faster path was not bit-exact with serial greedy decoding: 92-100% of requests diverged by a 1,600-token cap, and semantic quality was not measured. Deeper drafting was slower (MTP n-max 3 at 63.29 tok/s, DFlash2 n-max 4 at 63.13, MTP n-max 5 at 54.89, DFlash2 n-max 7 at 50.95). Board telemetry estimated a 37% decode energy reduction for a fixed 400-token workload.

generation:

70.0 tokens/s

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User benchmarks Qwen3.8 27B at 70 tok/s average on an RTX 3090, running in llama.cpp with 119K context and MTP enabled. Setup is llama.cpp with MTP speculative decoding and 119K context. The user also tested Qwen3.8 Flash-Next in Strata at 128K context, averaging 120 tok/s, and GPT 6.1 Sol (High reasoning) which one-shot the prompt.

generation:

21.5 tokens/s

quant:

GSQ-RCO 3.0bit Q4_K attention (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports GLM-5.3-Flash at 21.5 t/s decode on 2x RTX 3090 with a VRAM-filling MoE expert cache, up from 13.8 t/s without it. Setup is a neurall/llama.cpp fork with a GSQ-RCO 3.0-bit GGUF (Q4_K attention), 32k context, experts in RAM with a GPU expert cache, on a Ryzen 7 3700X with 125 GB DDR4. Long 12k-token code prompt: 181 t/s prompt processing and 15.7 t/s decode with the cache, versus 217 t/s and 12.3 t/s without. Perplexity 3.5871 on wikitext-2, 0.95% above the original quant's 3.5534.

generation:

19.2 tokens/s

quant:

Q2_K Medium (GGUF)

kv:

F16

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports GLM-5.3-Flash at 19.17 tok/s decode on 2x RTX 3090. Setup is llama.cpp with Q2_K Medium GGUF and F16 KV cache, 65536 context, layer split across two cards with static expert offload to CPU and an external DFlash2 Q4_K_M drafter (SPEC_N=2). Engine-side deep-prefill mean 117.78 tok/s; 10K prefill mean 118.99 tok/s. DFlash fired on 16/16 requests with mean acceptance 0.681. Both cards power-capped at 210 W. Soak-continuous was skipped.

generation:

111.7 tokens/s

prompt processing (prefill):

3865.4 tokens/s

quant:

Q4_K_M (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User benchmarks LLaMA 3 8B at 111.74 t/s generation and 3865.39 t/s prompt processing on an RTX 3090 24GB. Setup is llama.cpp with Q4_K_M GGUF weights, 1024-token generation and 1024-token prompt eval, single card. The same card reaches 115.42 t/s at 512 tokens and drops to 87.49 t/s at 8192 tokens; the F16 build of the same model runs at 46.51 t/s generation and 4239.64 t/s prompt processing.

generation:

17.5 tokens/s

quant:

IQ4_XS (GGUF)

kv:

Q8_0

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Step-3.5-Flash at about 17/18 t/s on llama.cpp with an RTX 3090 and two RTX 5060 Ti 16GB cards, 64GB RAM and a Ryzen 9 5950X. Setup is llama.cpp with an IQ4_XS GGUF around 96GB, 16384 context and Q8_0 KV cache, with the model split across GPU and system RAM. The same model on ik_llama.cpp reaches only 10 t/s and is a bit unstable, so the user asks whether the multi-GPU mix is the problem or whether the setup can be refined.

generation:

129.1 tokens/s

prompt processing (prefill):

5124.4 tokens/s

quant:

Q4_0 (GGUF)

kv:

f16

flash attention:

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User benchmarks the new tensor parallel (split mode tensor) implementation in llama.cpp against graph parallel in ik_llama.cpp on a 2x RTX 3090 system with 48 GiB of VRAM. Setup is llama.cpp with Q4_0 quantized Llama 3 8B, f16 KV cache, 65536 context, 100 GPU layers, and flash attention enabled. The run crashed with CUDA out of memory at 34816 tokens of context. User reports 129.08 t/s generation and 5124.38 t/s prompt processing at zero context, degrading to 26.89 t/s generation and 3092.68 t/s prompt processing at 34816 tokens. User calls the PR a gimmick not ready for prime time, noting memory is not released and performance lags well behind ik_llama.cpp graph parallel.

generation:

44.2 tokens/s

prompt processing (prefill):

1710.2 tokens/s

quant:

Q8_K_XL (GGUF)

kv:

q8_0

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User benchmarks Qwen3.8-27B on dual RTX 3090s, comparing llama.cpp against vLLM int8. llama.cpp with Q8_K_XL GGUF and q8_0 KV cache reaches 44.22 tok/s decode on narrative prompts, roughly half the 84.56 tok/s of the vLLM int8 w8a16 container with fp8 KV cache and MTP. Setup uses two RTX 3090s with a 0.55/0.45 split and ngl=99 for llama.cpp; vLLM uses custom all-reduce with P2P granted. Prefill at 10K tokens is 1710.16 tok/s for llama.cpp versus 1891.57 tok/s for vLLM. TTFT is about 5x slower on llama.cpp (337 ms vs 69 ms narrative). The 90K prefill run for llama.cpp was stopped early. User considers adding more system RAM for LMCache instead.

generation:

94.5 tokens/s

quant:

Q4_K_M (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3-Next-80B-A3B at 94.5 tok/s decode on an RTX 3090 with 16 GB RAM, using a patched llama.cpp that substitutes missing experts instead of waiting for SSD reads. Setup is llama.cpp with Q4_K_M, 1/4 of experts in VRAM, rest read from NVMe at ~5.7 GB/s, 16 threads, about 15.7 GB VRAM used. Stock llama.cpp gave 31.8 tok/s in the same 16 GB case; the patch also reached 108.4 tok/s with plenty of RAM and 89 tok/s reading every miss from SSD. Perplexity was 1.6% higher than stock, GSM8K lost 1.8 points, and greedy generation ran 64-74 tok/s. The 16 GB case was simulated by locking RAM on a bigger machine.

generation:

48.8 tokens/s

prompt processing (prefill):

142.5 tokens/s

quant:

Q8_K_XL (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen 3.8 27B Q8_K_XL at 48.76 t/s generation and 142.46 t/s prompt processing on dual RTX 3090s. Setup is llama.cpp via llama-swap with MTP speculative decoding (draft-mtp, n-max 3), 238000 token context, tensor split 50,50 across two cards, batch size 4096. The figures are from the final turn of a multi-turn agentic coding run (95 prompt tokens, 90 generated, 43473 cached); per-turn generation speeds ranged from 39.58 to 77.38 t/s across the run.

generation:

36.5 tokens/s

quant:

Q4_K_S (GGUF)

kv:

q5_0/q4_1

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Gemma-4-31B at 36.5 tok/s on a single RTX 3090 with full big context, using mainline llama.cpp built from source with -DGGML_CUDA_FA_ALL_QUANTS=ON. Setup is llama.cpp with Q4_K_S weights and q5_0/q4_1 KV cache on one 24 GB card; the stock prebuilt image only reaches ~12 tok/s because the flag is off by default. The post compares engines: beellama.cpp with DFlash speculative decoding reaches 54-193 tok/s depending on workload (131-200K context), vLLM gets 86-145 tok/s on dual cards at 262K, and ik_llama is fast but context-capped near 24K. Quality on an 8-pack was 109/150 with reasoning off and 114/150 with reasoning on.

generation:

38.1 tokens/s

prompt processing (prefill):

1263.0 tokens/s

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8 27B at 38.1 t/s generation and 1263 t/s prefill on 2x RTX 3090, asking whether NVLink would improve performance over a better motherboard. Setup is llama.cpp b10586 (CUDA) behind llama-swap on Windows 11, with a Ryzen 7 5800X3D and 64GB DDR4. Cold load took 24s. At 28k context the run gave 1250 t/s prefill and 36.2 t/s generation; at 85k context 901 t/s prefill and 28.9 t/s generation. User also runs nightly data classification jobs and asks whether prefill speed would benefit.

generation:

169.4 tokens/s

quant:

Q4_K_M (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Ornith-1.5-35B-A3B-BigBang-MTP at 169.4 tok/s with MTP speculative decoding on a single RTX 3090, up 29% from 131.0 tok/s without the draft. Setup is llama.cpp with Q4_K_M main weights and a Q8_0 draft, single RTX 3090, temp 0.1. The grafted Qwen3.6 MTP head accepts 0.55-0.75 of tokens (code high, chat lower) with mean accepted runs of 3.2-4.0 tokens, versus ~0.13 acceptance for stock Ornith-1.5's random-init head. Stock Ornith-1.5 measured 133.9 tok/s without a draft. Perplexity on mixed code and prose was 3.34 versus 3.41 for stock, and both scored 15/15 on a 15-task Python pass@1. The author calls this a small local eval, not a benchmark suite.

generation:

24.5 tokens/s

prompt processing (prefill):

300.2 tokens/s

quant:

IQ2_XXS (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports DeepSeek V4 Flash 0731 at 24.48 t/s generation and 300.24 t/s prompt processing on 4x RTX 3090 (WRX80 platform, Threadripper PRO 3975WX, 251GB RAM). Setup is llama.cpp (club-3090 build) with UD-IQ2_XXS GGUF quant, 204800 token context, MoE cache enabled, speculative decoding on (draft acceptance 0.60833, mean len 4.04), 4 GPUs with PCIe-only interconnect (no NVLink, P2P unreachable). Model is 284.33B params, 84.62 GiB file at 2.56 BPW; GPUs are power-capped to 250W. Figures are from a single request (n_tokens 2797 prompt, 96 tokens generated).

generation:

1.8 tokens/s

prompt processing (prefill):

29.8 tokens/s

quant:

UD-Q8_K_XL (GGUF)

kv:

q8_0

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports DeepSeek V4 Flash at ~1.78 t/s generation and ~29.8 t/s prompt processing on dual RTX 3090s with 64GB DDR4 and 78GB swap. Setup is llama.cpp with UD-Q8_K_XL GGUF weights, q8_0 KV cache, 65536-token context, 11 GPU layers and tensor split 1,1. The user notes the generation figure used only GPU and RAM with no swap, and that prefill is fast because the GPUs handle it.

generation:

6.0 tokens/s

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User asks which inference engine is fastest for a full quant on 96GB of VRAM, with 4x RTX 3090 and 256GB of 8-channel DDR4. They report getting at most 6 t/s on short prompts with llama.cpp CUDA. No model, quant, or context length is named.

generation:

30.0 tokens/s

quant:

Q4_K_M (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Muse Glimmer 30B at 29.96 t/s median completion throughput on an RTX 3090. Setup is llama.cpp (build 10437, CUDA) with GGUF Q4_K_M at 8,192 tokens of context, one concurrent request and no speculative decoding. The model used about 18.0 GB of VRAM on the 24 GB card, which ran at a fixed 280 W power limit. Median latency was 6.94 seconds per question; prompt processing speed was not reported separately.

generation:

36.4 tokens/s

quant:

Q4_K_M (GGUF)

kv:

q8_0

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B at 36.4 tok/s on an RTX 3090 24GB, with throughput dropping to 20.9 tok/s at 120K context. Setup is llama.cpp with Q4_K_M and q8_0 KV cache, all layers on GPU, 131,072-token context, flash attention, one generation slot. The 36.4 tok/s figure is for a 2,073-token input with thinking disabled; cached continuations reuse almost the entire prefix and are not representative of fresh prompts. Peak GPU memory was 22,162 MiB. Three of four bounded coding tasks passed their checks, with one failing after spending its full 8,192-token output allowance on reasoning.

generation:

101.7 tokens/s

quant:

Q4_K_XL (GGUF)

kv:

q8_0

flash attention:

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.6-35B-A3B at 101.7 tok/s on short prompts on an RTX 3090 24GB. Setup is llama.cpp with Q4_K_XL, q8_0 KV cache, 65536 context, flash attention on, and all layers offloaded to the GPU. The same model on Ollama reached 98.9 tok/s at 32k context and 23.4 GB VRAM. Qwen3.5-35B-A3B on llama.cpp reached 142.2 tok/s at 65k context and 21.7 GB VRAM. The author corrected an earlier comparison: llama.cpp was 1.4x faster than Ollama, not 2.4x, because Ollama had overflowed to the CPU in the first test. No speculative decoding; only generation speed was measured.

generation:

31-34 tokens/s

prompt processing (prefill):

1080.0 tokens/s

quant:

UD-Q4_K_XL (GGUF)

kv:

turbo3

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen 3.8 27B at 31-34 t/s generation and around 1,080 t/s prompt processing on a single RTX 3090 24 GB. Setup is llama.cpp with CUDA, UD-Q4_K_XL quant, 64K context, turbo3 KV cache, 2 parallel slots. User compares four agent configurations: Qwen alone (1/5 scenes right, 26 min, $0), Sonnet 5.5 planning with Qwen coding (2/5, 85 min, $0.67), Qwen planning with Sonnet 5.5 coding (4/5, 14 min, ~$2.77), and Sonnet 5.5 alone (4/5, 9 min, $1.83). A 3060 12 GB was tried first and gave 4-6 t/s with ~7 GB of weights spilling into system RAM. User notes Qwen is a better planner than coder, that thinking budget had to be capped, and that crash checks reported zero errors while some scenes were visibly wrong. One run per setup, described as a field report rather than a benchmark.