Kimi K2.6
on NVIDIA T4 16GB · llama.cpp · 49,920 ctx
Summary
User profiles Kimi K2.6 on a Tesla T4 16GB with an Intel Xeon Gold 5120 and 740 GB RAM, reporting it works but is unusable at 1-2 tokens/s with a 50k context.
Setup is llama.cpp with the Unsloth UD-Q4_K_XL GGUF (~544 GiB, 14 shards merged), turbo3_tcq KV cache, 49920 context, --n-cpu-moe 61 so all 61 layers and 384 experts run on CPU.
The Q8_K_XL quant is not viable on the T4, and the notes also profile hypothetical RTX 4090 and RTX 5090 hosts where all layers still run on CPU.