Local model performance on your hardware

generation:

5.8 tokens/s

prompt processing (prefill):

10.7 tokens/s

quant:

Q2_K_XL (GGUF)

kv:

f16

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Kimi K2.5 at 5.85 t/s generation and 10.67 t/s prompt processing on eight Tesla P40 24GB cards. Setup is llama.cpp with a Q2_K_XL GGUF (349.01 GiB, 2.92 BPW) at 262144 context, f16 KV cache, flash attention on, experts offloaded to CPU. User says smaller MoE models reach over 20 t/s and Kimi K2.5 gives 5+ t/s; a second request measured 5.47 t/s generation and 7.48 t/s prompt processing.

generation:

23.7 tokens/s

quant:

IQ4_XS (GGUF)

kv:

turbo3

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen 3.6 27B at 23.66 tok/s decode on a Tesla P40 24GB. Setup is llama.cpp with IQ4_XS GGUF, turbo3 KV cache, MTP self-speculative decoding (N=2, p_min=0.5), 65536 context, single card. MTP accept rate 83.7%, a +28.0% gain over the non-MTP Q4_K_M baseline at 18.48 tok/s. A cache sweep found turbo3 best at 22.43 tok/s and 18.1 GB VRAM, while q8_0 broke MTP with 0% draft acceptance.

generation:

37.9 tokens/s

prompt processing (prefill):

982.0 tokens/s

quant:

Q3_K_XL (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.6-35B-A3B at Q3_K_XL generating 37.9 tok/s on a single Tesla P40 24GB under WSL2 with driver 582.70 in the MCDM driver model. Setup is llama.cpp built with CUDA arch 61, GGML_CUDA_FORCE_MMQ=ON and GGML_CUDA_F16=OFF; llama-bench pp512 measures 982 tok/s prefill. With MTP speculative decoding at --spec-draft-n-max 2 generation rises to 51.6 tok/s (n-max 3: 48.3, n-max 4: 41.9). Prefill falls to about 790 tok/s at 6K tokens, 422-609 at 25K and about 216 at 78K. With MTP n-max 2, 4096 context with f16 KV uses 21710 MiB at 51.45 tok/s, 32768 with q8_0 KV uses 22086 MiB at 50.79 tok/s, and 131072 with q8_0 KV uses 22782 MiB at 50.54 tok/s.

generation:

51.6 tokens/s

prompt processing (prefill):

982.0 tokens/s

quant:

UD-Q3_K_XL (GGUF)

kv:

Q8_0

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.6-35B-A3B at 51.6 tok/s on a Tesla P40 24GB under WSL2. Setup is llama.cpp built for sm_61 with UD-Q3_K_XL (16.85 GB, 17.23 GB with MTP), full GPU offload, MTP speculative decoding at --spec-draft-n-max 2, and Q8_0 KV cache at 131072 context. Baseline without speculation is 37.9 tok/s; MTP with speculation disabled lands at 38.3 tok/s, confirming the speedup is real. Prefill is 982 tok/s (pp512). Context scaling from 4096 to 131072 costs about 1 GB VRAM and about 2% throughput. Needle-in-a-haystack passed 9 of 9 at 10/50/90 percent depth, with 6315-token prompts at about 790 tok/s prefill and 25111-token prompts at 422-609 tok/s prefill. All generation figures are single-stream.

generation:

46.0 tokens/s

prompt processing (prefill):

444.4 tokens/s

quant:

Q8_0 (GGUF)

kv:

F16

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports about 46 t/s on typical short interactive decode in use on dual P40s, up from roughly 15 t/s without speculative decoding. Setup is MTP speculative decoding with draft-mtp and ngram-simple, draft maximum 6, Qwen reasoning medium and the F16 vision projector loaded. A synthetic 128-token decode reached 48 t/s, long-context interactive decode runs at about 20 t/s, and prefill is 444 t/s at pp512. The model is a fine-tune of Qwen3.8 27B.