Local model performance on your hardware

generation:

30.0 tokens/s

quant:

IQ4_XS (GGUF)

kv:

q4_0

User reports about 30 t/s with IQ4_XS on two RTX 3090s running Strata, and around 100 t/s with IQ3_S and 20 t/s or less with Q4_K_XL. Setup is Strata in Docker with 256K context, q4_0 KV cache, both GPUs, and 128GB of system RAM on a Ryzen 9 3900X. The user is tuning the configuration for coding and asks whether the larger IQ4_XS quant is worth the speed drop over IQ3_S.

generation:

41.0 tokens/s

quant:

Q2 (GGUF)

User reports DeepSeek-V4.1-Flash-1017 at 41 tok/s on a single RTX 3090 with 192 GB of system RAM, averaged across multiple prompts. Setup is the user's delta-inference-engine (a Windows port of Xolotl, built on Strata) running nazeshinjite's Q2 GGUF natively on Windows, with the model streamed from a 366 GB download and a 1M context. The user says output matches DeepSeek's API next token 90.2% of the time on general text and 92.4% on code/math. The same table gives 33 tok/s with 128 GB RAM, and the needs list gives 31 tok/s at 16 GB VRAM and 17 tok/s at 12 GB VRAM. Other quants measured 38 to 42 tok/s.

generation:

66.0 tokens/s

quant:

UD-Q4_K_XL (GGUF)

kv:

q8_0

User reports Qwen3.8-27B at 66 t/s decode at 262,144-token context depth on one RTX 3090, with the full context fitting in 23.5 GB. Setup is a custom engine with unsloth UD-Q4_K_XL weights, K cache at 6 bits, V at 5 bits, and the newest 2,048 positions kept at q8_0, saving 1.1 GB versus a full q8_0 cache at 120k context. KV quality was scored by KL divergence against llama.cpp q8_0 output: 0.00160 at 64k versus 0.00163 for q8_0 and 0.00161 for fp16, +8% at 32k, and the same 230/282 on the author's question set. The author notes Qwen3.8 is a hybrid with only 16 of 64 layers attention, so full-attention models may differ.

generation:

96.0 tokens/s

User reports Raon-1 mini preview at 96 tok/s on an RTX 3090 while running a Korean RAG benchmark over 300 questions and 64 PDFs. Setup details are not stated; the run used the same top 5 retrieved pages and prompt for every model, with thinking on and temp 0. Raon-1 scored around 74-75% and looped on 16 of 320 questions at temp 0 with thinking on. Qwen3.8 27B scored 77.3% versus Opus 5.5 at 76.6% (p=0.82 on McNemar), and thinking off dropped Qwen about 6 points. gpt-oss 20B went from 57.5% to 69.6% after raising the token limit from 1024 to 8192. BM25 hit@5 was 56% with whitespace splitting and 80% with a Korean morphological tokenizer. The user notes 11 verdicts flipped across hardware, so a 1 point gap means nothing.

generation:

100.0 tokens/s

quant:

Q5_K_XL (GGUF)

kv:

F16

User compares two quant builds of the same unnamed model on dual RTX 3090s with NVLink and 128GB RAM. The UD Q5_K_XL build with F16 KV cache runs at around 100 t/s with the Q8 PLE on SSD, while the iQ3_S int8 setup runs at 150 t/s. All quality-reducing speed flags are off in the Q5 run. The user says the Q5 build thinks less and is much more logical, takes fewer steps in agentic tasks and needs fewer corrections. No benchmarks were run; the comparison is subjective.

prompt processing (prefill):

3536.0 tokens/s

User reports a GPU unlock module raising prefill on Qwen3.8 Flash-Next under Strata on 2 RTX 3090s, from 3205 to 3536 tok/s at 90K context, a 10% gain. The module, gpu-unlock by Zagitta, sets FMLA32 to full before the NVIDIA driver loads; the user measured 4K and 10K prefill gains of 12% each. BF16 tensor math rose from 82 to 155 TFLOPS and TF32 from 41 to 79 TFLOPS, but outputs were unchanged byte for byte over 1996 teacher-forced positions. The user asks whether it works on 40-series cards.

generation:

190.0 tokens/s

quant:

Q4_K_M (GGUF)

kv:

8-bit

User reports Qwen3.8-27B at 190 tok/s on two RTX 3090s power-limited to 280 W over plain PCIe, at 0 context. Setup is the Geezuz engine, a dual-GPU fork of L-Forster's open-jet megakernel, with Q4_K_M GGUF weights and 8-bit KV cache; the model drafts with its own MTP head, verifying 5 drafted tokens per pass. At 8 concurrent requests the total is 622 tok/s; Q8_0 gives 143 tok/s single and 610 tok/s at 8. The user says llama.cpp's fastest two-GPU setup did about 100 tok/s on the same prompts, and that Geezuz averaged 164 tok/s and 214 on code at the same KL divergence. At 10K context it reports 159 tok/s single-request and 550 at 8 requests; vLLM still leads at 8 parallel requests by about 15%.

generation:

15.5 tokens/s

prompt processing (prefill):

548.0 tokens/s

quant:

Q4_K_XL (GGUF)

User reports Ornith-1.5-122B at 15.5 t/s decode and 548 t/s prefill on a single RTX 3090 at 32K context. Setup is llama.cpp with Q4_K_XL GGUF and routed experts kept in system RAM via --n-cpu-moe 38, on a cloud pod with 32 vCPU EPYC 7763 and 125 GB RAM. Decode ranged 12-22 t/s across runs; Q3 reached 26 t/s and 128K context fits with --n-cpu-moe 40. Q4 peaked at 55 GiB RAM and the pod cost $1.53.

generation:

40.9 tokens/s

prompt processing (prefill):

1150-1210 tokens/s

quant:

Q4_K_XL (GGUF)

kv:

q8_0

User benchmarks Qwen3.8-27B (Q4_K_XL GGUF, 17.6 GB) at 40.9 tok/s decode on an RTX 3090 against a Mac mini M5 Pro 64GB at 15.6 tok/s. Setup is llama.cpp with q8_0 KV cache, 131072 context, 256-token completions, prompts of about 8,300 tokens. Prefill was 1,150 to 1,210 tok/s on the 3090 and 322 to 339 tok/s on the Mac. On a real agent task the 3090 finished in 59 min 51 s against the Mac's 3 h 7 min, writing 27.3 against 10.4 output tokens per second. MLX with nvfp4 weights decoded at 29.6 tok/s on the Mac, and MTP lifted the Mac's llama.cpp decode from 15.6 to 25.6 tok/s. The user calls the ratios rough, with one task and one run per machine.

generation:

38.0 tokens/s

prompt processing (prefill):

900-1200 tokens/s

quant:

Q4_K_M (GGUF)

kv:

q8_0

User reports Qwen3.8-27B Q4_K_M at a median of about 38 tokens/s on an RTX 3090 in a USB4 eGPU enclosure attached to a Beelink GTR7 mini PC. Setup is llama.cpp with Q4_K_M weights, q8_0 KV cache, 64K context, flash attention on, and n-gram speculative decoding enabled. Prompt evaluation runs about 900-1200 tokens/s on large prompts, with spikes up to about 280 t/s when speculation hits on repetitive code. VRAM use is 20.7 of 24.6 GB with the full 64K context.

generation:

39.3 tokens/s

quant:

UD-Q4_K_XL (GGUF)

User reports DeepSeek-V4.1-Flash at 39.3 tok/s on code and 24.6 tok/s on prose with 4x RTX 3090 (96 GB VRAM) plus an EPYC 9575F and 768 GB DDR5. Setup is llama.cpp built from PR #30817 with the UD-Q4_K_XL GGUF quant, 65536 context, flash attention on, and the DSpark Q8_0 drafter (8 GB) for speculative decoding with up to 3 draft tokens. All figures are one request at a time with thinking off, single runs. Prefill and generation on an 8K prompt were 326 / 20.6 tok/s and on a 32K prompt 363 / 25.4 tok/s. DSpark roughly doubles code speed with about 80% of guesses accepted, but only about 30% are accepted on prose, where it is sometimes a bit slower; the user keeps it on because they mostly use the model for code.

generation:

26.0 tokens/s

prompt processing (prefill):

170.0 tokens/s

quant:

2.6bit

User reports DeepSeek V4.1 Flash at 26 t/s generation and 170 t/s prompt processing on 2x RTX 3090s. Setup is the dsx engine with a 2.6-bit quant (2.6127 bpw experts) and Engram tables, at 262,144 context; the 187.6 GB weights plus 203 GB Engram tables ran from a SATA SSD, and loading from an HDD took 90-110 minutes. The build scores KL 0.1972, 82.59% top-1 and perplexity 9.3626 against the FP8/FP4 teacher; the engine does not support images yet.

prompt processing (prefill):

1600.0 tokens/s

quant:

Q4_K_M (GGUF)

kv:

8-bit

User reports 140 t/s writing code with a custom inference engine built from scratch for Qwen3.5-27B on an RTX 3090 24GB, with CPU offload. Setup uses the unsloth Q4_K_M GGUF weights with additional 8-bit quantization of the KV cache and hidden states. Coding with a 4K-token file in context drops to 95 t/s, and prefill is about 1,600 tok/s. The user compares these figures against llama.cpp with MTP on the same hardware.

generation:

40.6 tokens/s

prompt processing (prefill):

1616.0 tokens/s

quant:

Q4_K_M (GGUF)

kv:

F16

User benchmarks Qwen3.8-27B Q4_K_M across two RTX 3090s in llama.cpp, with a 4096-token prompt and TG128 generation. The layer split reached 1,616 t/s prompt and 40.56 t/s generation, against 1,235 / 39.16 t/s on a single GPU. Setup is llama.cpp build 10775 with F16 KV cache, -ngl 999, flash attention on, no speculative decoding and a single request; the 16.46 GB file fits on one 24 GB card. Both cards ran at PCIe Gen2 x1 with no P2P. Tensor split was slower: built-in AllReduce gave 140 t/s prompt and 41.10 t/s generation, NCCL 169 t/s and 31.25 t/s. The user concludes layer split was clearly more practical in both speed and latency on this machine.

generation:

75-80 tokens/s

prompt processing (prefill):

980.0 tokens/s

quant:

IQ4_NL (GGUF)

User reports Qwen3.5-35B-A3B at ~980 t/s prompt processing and 75-80 t/s generation on a single RTX 3090 24GB. Setup is llama.cpp with the RPC backend, IQ4_NL GGUF, flash attention on, 64K context per agent, and all layers offloaded to the GPU. The host is an AMD EPYC 7663 with 128GB DDR4 RAM. Generation is given only as a 75-80 t/s range. Two instances run simultaneously (planner and coder).

generation:

15.0 tokens/s

quant:

Q2

User reports about 15 t/s running GLM-5.3 Flash at Q2 on an RTX 3090 with DDR4 RAM and SSD offload, under an engine called Project Maya. The setup streams part of the model from system memory and disk, since a Q2 GLM-5.3 Flash does not fit in the card's 24GB alone. User's main complaint is behavioral rather than speed: directed to start a new project, the model continued on an old project with a similar name and made little to no improvement, a mistake no other model or engine made in the same environment.

generation:

83.0 tokens/s

quant:

IQ3_S (GGUF)

User reports about 83 tok/s generation on an RTX 3090 24 GB running a Swift-1.5 fine-tune of Qwen3.8 27B at IQ3_S GGUF with MTP (draft 4) and vision on. Setup is Ollama 0.35.1 on Windows 11, one request at a time, 131072-token window fully on the card. The figure is for a local agent driving ten Windows apps; the user notes writing files directly is still about 10x faster and made the agent stop doing that by default.

generation:

86.6 tokens/s

prompt processing (prefill):

822.0 tokens/s

quant:

2-bit (GGUF)

kv:

Q8_0

flash attention:

on

User benchmarks Nemotron 3.5 Bolt (74B, 2-bit) and Qwen3.8 27B (Q4_K_M) on an RTX 3090, comparing context limits and speed at various fill depths. Setup is llama.cpp with Q8_0 KV cache, flash attention on, and no CPU offload. Bolt fits its full 256K context at 21.8GB VRAM, while Qwen3.8 27B maxes out at about 216K (221,184 tokens) at 23.2GB. At empty context, Bolt decodes at 86.6 t/s and Qwen at 42.3 t/s. At 128K fill, Bolt decodes at 52.2 t/s and Qwen at 23.3 t/s. Prompt processing is faster for Qwen at empty context (1237 vs 822 t/s), but Bolt is faster from 64K onward. The user notes that ubatch 128 was carried over from a previous test, so prompt numbers are lower than a tuned setup would achieve.

generation:

82.0 tokens/s

quant:

Q6

User reports Qwen3.6 35B-A3B at 82 t/s decode on a 4x RTX 3090 rig with MTP disabled. Setup is a Q6 quant with MTP speculative decoding off; the same rig reached about 70 t/s with MTP enabled. The user attributes the gain to MTP's draft-and-verify step needing extra sync passes across the four layer-split GPUs, where PCIe hops cost more than the accepted draft tokens save.

generation:

294.1 tokens/s

User reports 294.1 t/s generation. Source evidence (verbatim): Source: https://dev.to/sikamikanikobg/vllm-vs-llamacpp-vs-ollama-what-happens-when-your-model-doesnt-fit-in-24gb-vram-56eb # vLLM vs llama.cpp vs Ollama: What Happens When Your Model Doesn't Fit in 24GB VRAM Hardware: NVIDIA RTX 3090 24GB, 128GB system RAM. Concurrency 1 decode tok/s (llama.cpp / vLLM): Gemma 3 1B 294.1 / 235.5. Qwen3-Coder 30B-A3B 157.2 / 172.0 (vLLM AWQ). Gemma 4 26B-A4B 118.8 / 133.8. Ollama at concurrency 1: 125.6, 129.3, 84.5. Single-stream steady-state decode on the longest prompt tier with models that exceed VRAM (llama.cpp -ngl 12 for gpt-oss-120b): GPT-OSS-120B 7.65 tok/s llama.cpp, 7.6 Ollama. GLM-4.5-Air 4.58 llama.cpp, 4.59 Ollama. vLLM ran out of memory on both. Engine versions and context length not stated.