Local model performance on your hardware

generation:

1-2 tokens/s

quant:

UD-Q4_K_XL (GGUF)

kv:

turbo3_tcq

User profiles Kimi K2.6 on a Tesla T4 16GB with an Intel Xeon Gold 5120 and 740 GB RAM, reporting it works but is unusable at 1-2 tokens/s with a 50k context. Setup is llama.cpp with the Unsloth UD-Q4_K_XL GGUF (~544 GiB, 14 shards merged), turbo3_tcq KV cache, 49920 context, --n-cpu-moe 61 so all 61 layers and 384 experts run on CPU. The Q8_K_XL quant is not viable on the T4, and the notes also profile hypothetical RTX 4090 and RTX 5090 hosts where all layers still run on CPU.

generation:

112.5 tokens/s

kv:

fp8

User reports Kimi-K2.6 at 112.5 tok/s single-stream decode on 8x RTX PRO 6000 Blackwell. Setup is vLLM with fp8 KV cache, TRITON_MLA attention, TP=8, DCP=8, and Eagle3 MTP speculative decoding with 3 tokens using the lightseekorg/kimi-k2.5-eagle3-mla draft model. The 112.5 tok/s is the DCP=1 + MTP=3, 0-context, concurrency-1 cell from an aggregate decode matrix; the page also gives DCP=1 no-MTP (91.5), DCP=4 MTP (101.5), DCP=8 MTP (89.4) and DCP=8 no-MTP (75.5) at the same cell, plus prefill of 7892 tok/s at 8k and 5604 tok/s at ~83k. An erratum notes the old 128k rows were actually ~83k prompt tokens.

generation:

9.7 tokens/s

prompt processing (prefill):

264.0 tokens/s

User reports Kimi K2.6 at 264 t/s prompt processing and 9.7 t/s generation on 32x AMD MI50 32GB GPUs.