Local model performance on your hardware

generation:

70.0 tokens/s

quant:

IQ3_XXS (GGUF)

kv:

q8_0

User reports Qwen3.8 27B (Swift-1.5 fine-tune, IQ3_XXS GGUF) at 70 t/s on an RTX 4080 16GB with 81920 context, using MTP plus n-gram speculation. Setup is llama.cpp (llama-server CUDA build in WSL2/Debian) with q8_0 KV cache, flash attention on, all layers on GPU, and 14.9 GB of the 16 GB VRAM used. With n-gram only the user got 45 t/s at 204800 context. They note MTP was worth the tradeoff and could grow context further by dropping MTP or allowing spillover into system memory.

generation:

55.0 tokens/s

quant:

UD-IQ4_XS (GGUF)

User reports Qwen3.8-27B Heretic (UD-IQ4_XS) at 55 tok/s on code and 50 tok/s on prose on an RTX 4080 16GB with MTP 2 draft tokens. Setup is llama.cpp b11457 with a 15K-token prompt, 32K context, MTP 2 drafts, mean of 3 fixed seeds. Without MTP the same config gives 29 tok/s on both code and prose. User notes 2 draft tokens beat 3 (68 vs 56 tok/s on code, 55 vs 36 on prose at 40K), 48K vs 40K produced identical tokens but was 22% slower, and VRAM spill from the desktop dropped code decode to 68, 51 or 42 tok/s depending on 1.2, 1.4 or 1.9 GB of shared VRAM. MTP head precision barely matters (q6_K vs IQ3_S: 83% vs 83% acceptance on code, 58% vs 54% on prose). The 12GB and 24GB context numbers are computed from llama.cpp buffers, not measured.

generation:

166.7 tokens/s

prompt processing (prefill):

2719.9 tokens/s

quant:

GSQ (safetensors)

kv:

rk4v4-e8

User reports Qwen3.8 27B at 166.7 t/s generation and 2719.9 t/s prefill on an RTX 4080 16GB. Setup is NInfer with GSQ safetensors and rk4v4-e8 KV cache, single card, DFlash2 speculative decoding at 8K context. User also reports MTP3 decode at 151.2 t/s and DFlash2 K=7 decode at 262.3 t/s at 32K, with prefill up to 2720 t/s and 100k context support.

generation:

55.0 tokens/s

prompt processing (prefill):

590.0 tokens/s

quant:

Q3

kv:

Q8

User reports Qwen3.8 Flash-Next at 55 t/s on an RTX 4080 with 128 GB DDR5 RAM, using Strata with Q3 weights and Q8 KV cache at roughly 120-150k context. Setup is Strata with the 3060 left unused; adding the second GPU drops generation to 45 t/s and prompt processing to 250 t/s from 590 t/s. With Unsloth Studio the same model reached 27 t/s on the 4080 alone, 23 t/s with both GPUs, and 16 t/s with both GPUs on an older PCIe 4.0 x1 link. The user asks whether the 3060 is counter-productive alongside the 4080.

generation:

16.5 tokens/s

prompt processing (prefill):

350.0 tokens/s

quant:

Q4_K_M (GGUF)

kv:

Q8

User reports Qwen3.8 Flash-Next at 16.5 t/s generation and 350 t/s prompt processing on an RTX 4080 16GB with 64GB system RAM, at 131k context. Setup is llama.cpp built from a pr-mtp-fix branch with the Unsloth MTP head as the draft model, AtomicChat AD-4.27bpw Q4_K_M target weights, and a Q8 KV cache. The --spec-draft-cpu-moe flag keeps draft experts in RAM so the target's hot experts fit in the 16GB of VRAM. An IQ4_XS quant was also tested and reached about 10 t/s.

generation:

45.0 tokens/s

quant:

MXFP4_MOE (GGUF)

kv:

q8_0

flash attention:

on

User reports Qwen3.6 35B A3B at 45 t/s on an RTX 4080 16GB, and Gemma 4 12B at 34 t/s. Setup is llama.cpp with CUDA 13.3, MXFP4_MOE GGUF, 65535 context, q8_0 KV cache, and draft-MTP speculative decoding with n-max 4. User finds MTP speculative decoding consistently reduces throughput rather than improving it, contrary to reports of 20-30% gains.

generation:

56.5 tokens/s

prompt processing (prefill):

980.2 tokens/s

quant:

SC_3.00bpw_H4_V4 (EXL3)

kv:

6,5

mtp (multi-token prediction):

on

User reports Qwen3.8 27B at 56.48 t/s decode and 980.2 t/s prefill on an RTX 4080 16GB at 131,072 context with 102,400 active input tokens. Setup is ExLlamaV3 1.5.0 with TabbyAPI, the turboderp SC_3.00bpw_H4_V4 EXL3 quant, a 6,5 KV cache, and MTP k=2 with a Q6 draft cache. VRAM peaked around 15.2GB of 16.4GB. Without MTP decode was 33.68 t/s, so MTP gave about a 68% decode increase at roughly a 5% prefill cost. Q4 draft cache was about 9% slower than Q6, and dynamic drafting was slower than fixed k=2.

generation:

19.0 tokens/s

quant:

IQ4_XS (GGUF)

kv:

kvarn4

mtp (multi-token prediction):

on

User reports 8 t/s without MTP, 14 t/s with MTP, and about 19 t/s with full optimizations. Setup is beellama 0.4.3, a llama.cpp fork.

generation:

8.0 tokens/s

quant:

Q4_K_M (GGUF)

User reports running a 182B model on an RTX 4080 with 64GB RAM by offloading ngrams to SSD. User claims it is faster and more intelligent than Qwen3.8 27B and provides the launch command.