Local model performance on your hardware

generation:

80.0 tokens/s

prompt processing (prefill):

1200.0 tokens/s

quant:

Q3_K_XL (GGUF)

kv:

q8_0

flash attention:

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8 27B at 80+ t/s generation and 1200+ t/s prompt processing on an RTX 5080 with 16GB VRAM, using CUDA unified memory to fit the model. Setup is llama.cpp with the Unsloth UD Q3_K_XL GGUF, 160000 context, q8_0 KV cache, flash attention, and MTP speculative decoding with ngram-mod draft. User asks whether Q3 quants are good enough for harness coding with opencode, pi, and qwen-cli, and notes this is the only quant that fits the whole model in VRAM.

generation:

18-20 tokens/s

prompt processing (prefill):

100.0 tokens/s

quant:

Q5_K_XL (GGUF)

kv:

q8_0

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-Flash-Next at around 100 t/s prefill and 18-20 t/s decode on 2x RTX 5080 with 128 GB DDR5. Setup is llama.cpp with the unsloth UD-Q5_K_XL GGUF, a Q8_0 KV cache, 262,144 context, MTP speculative decoding with a Q4_K_M draft model, and ncmoe42 offloading 42 MoE layers to CPU. User asks whether switching to vLLM or SGLang would give better performance, noting the scarcity of NVFP4/FP8 GGUF files.

generation:

12.0 tokens/s

quant:

NVFP4 (GGUF)

kv:

q4_0

flash attention:

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8 27B at around 12 t/s on an RTX 5080 16GB at 49,152 context. Setup is llama.cpp with an NVFP4 GGUF, q4_0 K cache and iq4_nl V cache, flash attention on, 512 batch and ubatch, 16 threads. User is asking for pi.dev plugin suggestions for context, compaction and memory management, and is disappointed that compaction settings are kept separate from model settings.

generation:

6.0 tokens/s

quant:

IQ3_XXS (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8-Flash-Next at 6 t/s on an RTX 5080 16GB with 64GB RAM, and finds it unusable. Setup is llama.cpp with offloading to CPU/RAM. The first attempt used the IQ3_XXS quant, the second the Q4_K_M quant at 4.27bpw AD.

generation:

75.0 tokens/s

quant:

IQ4_XS (GGUF)

kv:

Q4

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports hybrid quantization to fit MTP and long context in 16 GB VRAM. Setup uses MTP draft with spec-type draft-mtp and spec-draft-n-max 3, at 85,000 context, with q4_0 KV cache quantization for both cache types.

generation:

13.2 tokens/s

quant:

Q4_K_M (GGUF)

kv:

Q4

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports a first tuning attempt with selective FFN offload to CPU improving speed over whole-layer offload. True Q4_K_M runs at 13.2 tok/s near 50-61K context. MTP is worse at deep context. Coding quality improved over IQ3.

generation:

56.0 tokens/s

prompt processing (prefill):

1584.0 tokens/s

quant:

Q4_K_XL (GGUF)

kv:

Q8

flash attention:

on

mtp (multi-token prediction):

off

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports a 35B Q4_K_XL model at 128k context runs best with no MTP and --fit-target 1536. MTP does not help at 128k context. A 27B IQ3 model fits fully on GPU and benefits from MTP at 73 tok/s.