Local model performance on your hardware

generation:

21.0 tokens/s

quant:

Q8 (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B at about 21 t/s on two modded RTX 3080 20GB cards, with MTP raising it to 33 t/s. Setup is llama.cpp with Q8 weights and 131.1k context plus vision, run on the two 3080s; the 3080s are power limited to 220W. The user also ran Gemma 4-26B-A4B on a single RTX 4070 Ti 12GB and found coding output poor, and has just reinstalled the 4070 Ti so has limited data with all three cards.

generation:

31.3 tokens/s

prompt processing (prefill):

997.0 tokens/s

quant:

Q4_K_XL (GGUF)

kv:

Q8_0

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User benchmarks Qwen3.8 27B at ~31.3 t/s generation and ~997 t/s prompt processing on a single modded RTX 3080 20GB. Setup is llama.cpp with Q4_K_XL GGUF and Q8_0 KV cache at ~98k context. Native MTP raises generation to ~56.9-60.7 t/s but cuts context to ~68k. Adding an RTX 3070 8GB via layer split reaches ~223k context but drops generation to ~27.8 t/s; tensor parallelism over the chipset x4 slot hits ~59.7 t/s generation with a ~20% prompt-processing penalty, while moving the second card to a CPU-attached Oculink x4 link recovers ~976-1,005 t/s prompt processing at ~60.4 t/s generation and ~204k context.

generation:

17.5 tokens/s

prompt processing (prefill):

273.2 tokens/s

quant:

IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8 (GGUF)

kv:

q8_0

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports 17.53 t/s generation and 273.21 t/s prefill at 128k context on 2x RTX 3080 20GB with 64GB DDR5. Setup is the llama.cpp fork by fairydreaming with an 86.7 GB model, --n-cpu-moe 28 and split-mode layer.