Local model performance on your hardware

generation:

70.0 tokens/s

quant:

IQ3_XXS (GGUF)

kv:

q8_0

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8 27B (Swift-1.5 fine-tune, IQ3_XXS GGUF) at 70 t/s on an RTX 4080 16GB with 81920 context, using MTP plus n-gram speculation. Setup is llama.cpp (llama-server CUDA build in WSL2/Debian) with q8_0 KV cache, flash attention on, all layers on GPU, and 14.9 GB of the 16 GB VRAM used. With n-gram only the user got 45 t/s at 204800 context. They note MTP was worth the tradeoff and could grow context further by dropping MTP or allowing spillover into system memory.

generation:

55.0 tokens/s

quant:

UD-IQ4_XS (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B Heretic (UD-IQ4_XS) at 55 tok/s on code and 50 tok/s on prose on an RTX 4080 16GB with MTP 2 draft tokens. Setup is llama.cpp b11457 with a 15K-token prompt, 32K context, MTP 2 drafts, mean of 3 fixed seeds. Without MTP the same config gives 29 tok/s on both code and prose. User notes 2 draft tokens beat 3 (68 vs 56 tok/s on code, 55 vs 36 on prose at 40K), 48K vs 40K produced identical tokens but was 22% slower, and VRAM spill from the desktop dropped code decode to 68, 51 or 42 tok/s depending on 1.2, 1.4 or 1.9 GB of shared VRAM. MTP head precision barely matters (q6_K vs IQ3_S: 83% vs 83% acceptance on code, 58% vs 54% on prose). The 12GB and 24GB context numbers are computed from llama.cpp buffers, not measured.

generation:

16.5 tokens/s

prompt processing (prefill):

350.0 tokens/s

quant:

Q4_K_M (GGUF)

kv:

Q8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8 Flash-Next at 16.5 t/s generation and 350 t/s prompt processing on an RTX 4080 16GB with 64GB system RAM, at 131k context. Setup is llama.cpp built from a pr-mtp-fix branch with the Unsloth MTP head as the draft model, AtomicChat AD-4.27bpw Q4_K_M target weights, and a Q8 KV cache. The --spec-draft-cpu-moe flag keeps draft experts in RAM so the target's hot experts fit in the 16GB of VRAM. An IQ4_XS quant was also tested and reached about 10 t/s.

generation:

45.0 tokens/s

quant:

MXFP4_MOE (GGUF)

kv:

q8_0

flash attention:

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.6 35B A3B at 45 t/s on an RTX 4080 16GB, and Gemma 4 12B at 34 t/s. Setup is llama.cpp with CUDA 13.3, MXFP4_MOE GGUF, 65535 context, q8_0 KV cache, and draft-MTP speculative decoding with n-max 4. User finds MTP speculative decoding consistently reduces throughput rather than improving it, contrary to reports of 20-30% gains.

generation:

19.0 tokens/s

quant:

IQ4_XS (GGUF)

kv:

kvarn4

mtp (multi-token prediction):

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports 8 t/s without MTP, 14 t/s with MTP, and about 19 t/s with full optimizations. Setup is beellama 0.4.3, a llama.cpp fork.

generation:

8.0 tokens/s

quant:

Q4_K_M (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports running a 182B model on an RTX 4080 with 64GB RAM by offloading ngrams to SSD. User claims it is faster and more intelligent than Qwen3.8 27B and provides the launch command.