Kimi K3.1

generation:

2.0 tokens/s

prompt processing (prefill):

8.4 tokens/s

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Kimi 3.1 running locally on a MacBook Pro M5 Max 128GB at 2 tok/s decode and 8.4 tok/s prefill. The run uses a source build of the ds4 engine with weights downloaded separately. No quant, context length or KV cache setting is stated.

generation:

10.8 tokens/s

quant:

2.81/2.56/1.44 bpw 3-tier MoE (~1.92 bpw avg)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Kimi-K3.1-Pro-Coder at 10.8 tok/s single-stream decode and 36.8 tok/s batch/prefill eval on a single RTX PRO 6000 Blackwell (96 GB) with 176 GB RAM in Colab. Setup is the Strata 3-tier MoE offloading engine with dr-triton fused dequant kernels, a 775 GB quant (down from 1.77 TB) with routed experts at 2.81/2.56/1.44 bpw (~1.92 bpw average) and shared experts, attention and embeddings in BF16/F32; experts split across GPU VRAM, pinned RAM and NVMe mmap with router lookahead prefetching at a 97.9% hit rate. The 2.8T model keeps 96.67% on both AIME 2025 and LiveCodeBench (29/30 each), with 95.20% top-1 and 99.92% top-5 token agreement against the base and mean KLD 0.0191. The post also lists projected throughput for other hardware (e.g. ~4-5.5 tok/s decode on a single 4090/5090, ~18-22 tok/s on 2x PRO 6000/H100), which are estimates rather than measurements.