Local model performance on your hardware

generation:

0.5-0.6 tokens/s

quant:

VQ3R

User reports Kimi K3 2.78T running at 0.45-0.62 tok/s on a 64 GB MacBook Pro with M5 Pro, streaming experts from internal NVMe. Setup is the WARP engine with VQ3R 3-bit expert quantization, a 46.39 GB memory budget including a 17.56 GB expert cache, and a 982 GB converted container. The 64-token run is a range; the 200-token column is absent because one run takes ten minutes and reads 4.6 TB. The engine also runs DeepSeek-V4.1-Flash 552B at 3.77/3.71 tok/s and GLM-5.3-Flash 313B at 3.32/3.86 tok/s on the same machine.

generation:

0.3 tokens/s

User reports running the full 2.8-trillion-parameter Kimi K3 at 0.2901 token/s (3.447 s/token) on an M1 Max laptop using the Deltafin native binary. Setup uses the Deltafin engine with all 16 experts routed and no pruning, plus the Kimi-K3-DSpark draft model (6.635 GiB on disk, ~4.49 GiB at runtime) for speculative decoding; an optional Qwen 0.6B/1.7B add-on (4.337 GiB) speeds raw completion by 2.7x on a 17-token test. The figure is 1.9% higher than the previous update; historical M1 Max results were 0.2847, 0.2660, 0.1311 and 0.0141 token/s. The full model is 1.7 TB on disk, or 215 GB with streaming setup.

generation:

1.0 tokens/s

quant:

4-bit

User reports Kimi K3 (2.78T) at 0.96 tok/s on a MacBook Pro M5 Max 128GB, with the trunk in memory and 1.44 TB of 4-bit routed experts streamed from NVMe drives. Setup is a llama.cpp-derived fork with experts split across the internal SSD plus up to three Thunderbolt 5 enclosures, each holding a byte-identical replica. Decode scales with drive count: 0.55 tok/s on one drive, 0.75 with one enclosure, 0.89 with two, 0.96 with three (1.75x), and 1.00 steady over 512 generated tokens, 1.13 over 128.

generation:

0.3-0.3 tokens/s

User reports running the full 2.78T-parameter Kimi K3 on a 64 GB MacBook Pro at around 0.32–0.34 tokens per second. Setup is the WASTE (Weight-Aware Streaming Tensor Engine) with a 982 GiB NVMe-optimized container, streaming routed experts from disk and caching them in RAM, at a 4K context with a measured minimum memory requirement of approximately 29 GB. The engine needs roughly 17 GB of expert data per token and more than half of decode time is spent reading experts from disk. The same engine runs Kimi-Linear 48B from a 19 GB container at 8.92 tokens per second with an 8 GB memory budget.

generation:

20.0 tokens/s

User reports Kimi K3 at 20 t/s on 16 DGX Sparks at 300k context, up from an unusable 7 t/s at 100k context. Setup is vLLM across two joined 8-node DGX Spark clusters. The user also ran Qwen 397B at 30 t/s on 4x GB10 and 50-60 t/s on 16x3090, Qwen 235B at 10-12 t/s and DeepSeek 671B at 8 t/s on a Threadripper with 512GB DDR4 and experts offloaded to RAM.

generation:

5.5 tokens/s

quant:

mxfp4 (MLX)

kv:

Q8

User reports Kimi K3 REAP73 at 5.51 t/s on a 512 GiB M3 Ultra. Setup is mlx-lm with mxfp4 experts and Q8 non-experts, 242 of 896 experts retained, single machine. Figures are the mean of three prompts x 96 tokens; the 350 GB REAP80 build decodes at 5.54 t/s and the graded two-bank build at 2.68 t/s. Prefill runs at ~73 tok/s. Unpruned tiers are 883 GB and above and do not fit any Apple Silicon machine.

generation:

2.0 tokens/s

quant:

MXFP4 (MXFP4)

User reports Kimi-K3 (2.78T, MXFP4) at ~2.0 tok/s single-stream decode on four M3 Ultra 512GB Mac Studios in tensor parallel over a Thunderbolt 5 RDMA mesh. Setup is MLX 0.32.0 / mlx-lm 0.31.3 with MXFP4 weights, full 1,048,576-token context (29 GB KV), 420.8 GB resident per rank, 229.6 s load. Aggregate throughput reaches 21.2 tok/s at 32 concurrent streams; chunked prefill is ~170 tok/s. User attributes decode at batch 1 to fixed per-call dispatch overhead (~170 ms/token) rather than bandwidth, with collectives at 1% and MoE gather at 7%; ~330 ms of the ~500 ms/token remains unattributed. An unresolved multi-chunk prefill hang under TP is described, following the host rather than the rank index.

generation:

92.0 tokens/s

quant:

MXFP4 (MXFP4)

User reports Kimi K3 (2.8T parameters) at 92 tok/s decode on 8x B300 via Modal. Setup is vLLM with MXFP4 weights, tensor parallel 8, cold boot about 27 min for a 1.56 TB load. TTFT is 0.92 to 1.02 s and average decode over 4 prompts is 83 tok/s. Cost is $56.79 per hour, $190 per million output tokens, about $36 per run, or $1,363 a day left warm. User also ran Unsloth's Dynamic GGUF 1-bit UD-IQ1_S (594 GB) on 8x A100-80GB via llama.cpp at about 9 tok/s with TTFT 7 to 60 s, $19.99 per hour and about $620 per million tokens, 3.3x more expensive per token. Quality at 1-bit was fine.

generation:

30.0 tokens/s

prompt processing (prefill):

750-910 tokens/s

User reports running Kimi K3 (2.8T) on a 16x GB10 cluster at ~30 t/s generation (peaking ~38 t/s) for coding and agentic tasks. Setup uses a customized gb10-vllm stack with custom MLA/KV kernels, dual MikroTik switches, and 4x 400G-to-4x100G breakout cables. Prefill throughput is ~750–910 t/s. Stable multi-hundred-thousand token context runs with multiple 500k compaction; handles concurrent requests without dropping token generation rates.

generation:

1.1 tokens/s

prompt processing (prefill):

5.7 tokens/s

quant:

UD-Q4_K_XL (GGUF)

User reports Kimi K3 at 1.05 t/s generation and 5.66 t/s prompt processing on an RTX 3090 with 2 TB of Intel Optane PMem 200 in Memory Mode. Setup uses a Supermicro X12SPA-TF with Intel Xeon 8370C, 256 GB DDR4-3200, and the UD-Q4_K_XL quant at 16k context. App Direct mode gave lower throughput at 0.62 t/s generation and 4.9 t/s prompt processing. The user also benchmarked Kimi K2.5, GLM-5.3-753B, Hermes-4-405B, DeepSeek-V4-Flash, Nemotron-3-Super-120B-A12B, gpt-oss-120b, and Ornith-1.5-35B-A3B on the same system.

generation:

20.0 tokens/s

prompt processing (prefill):

750.0 tokens/s

User reports the full Kimi K3 model running on a 16x DGX Spark cluster at an average of 20+ t/s, peaking at 38 t/s, with prefill at 750 t/s. This is a first run, and the user plans to optimize and publish a vLLM image.