Local model performance on your hardware

generation:

1.0 tokens/s

quant:

4-bit

User reports Kimi K3 (2.78T) at 0.96 tok/s on a MacBook Pro M5 Max 128GB, with the trunk in memory and 1.44 TB of 4-bit routed experts streamed from NVMe drives. Setup is a llama.cpp-derived fork with experts split across the internal SSD plus up to three Thunderbolt 5 enclosures, each holding a byte-identical replica. Decode scales with drive count: 0.55 tok/s on one drive, 0.75 with one enclosure, 0.89 with two, 0.96 with three (1.75x), and 1.00 steady over 512 generated tokens, 1.13 over 128.

generation:

20.0 tokens/s

User reports Kimi K3 at 20 t/s on 16 DGX Sparks at 300k context, up from an unusable 7 t/s at 100k context. Setup is vLLM across two joined 8-node DGX Spark clusters. The user also ran Qwen 397B at 30 t/s on 4x GB10 and 50-60 t/s on 16x3090, Qwen 235B at 10-12 t/s and DeepSeek 671B at 8 t/s on a Threadripper with 512GB DDR4 and experts offloaded to RAM.

generation:

5.5 tokens/s

quant:

mxfp4 (MLX)

kv:

Q8

User reports Kimi K3 REAP73 at 5.51 t/s on a 512 GiB M3 Ultra. Setup is mlx-lm with mxfp4 experts and Q8 non-experts, 242 of 896 experts retained, single machine. Figures are the mean of three prompts x 96 tokens; the 350 GB REAP80 build decodes at 5.54 t/s and the graded two-bank build at 2.68 t/s. Prefill runs at ~73 tok/s. Unpruned tiers are 883 GB and above and do not fit any Apple Silicon machine.

generation:

2.0 tokens/s

quant:

MXFP4 (MXFP4)

User reports Kimi-K3 (2.78T, MXFP4) at ~2.0 tok/s single-stream decode on four M3 Ultra 512GB Mac Studios in tensor parallel over a Thunderbolt 5 RDMA mesh. Setup is MLX 0.32.0 / mlx-lm 0.31.3 with MXFP4 weights, full 1,048,576-token context (29 GB KV), 420.8 GB resident per rank, 229.6 s load. Aggregate throughput reaches 21.2 tok/s at 32 concurrent streams; chunked prefill is ~170 tok/s. User attributes decode at batch 1 to fixed per-call dispatch overhead (~170 ms/token) rather than bandwidth, with collectives at 1% and MoE gather at 7%; ~330 ms of the ~500 ms/token remains unattributed. An unresolved multi-chunk prefill hang under TP is described, following the host rather than the rank index.

generation:

92.0 tokens/s

quant:

MXFP4 (MXFP4)

User reports Kimi K3 (2.8T parameters) at 92 tok/s decode on 8x B300 via Modal. Setup is vLLM with MXFP4 weights, tensor parallel 8, cold boot about 27 min for a 1.56 TB load. TTFT is 0.92 to 1.02 s and average decode over 4 prompts is 83 tok/s. Cost is $56.79 per hour, $190 per million output tokens, about $36 per run, or $1,363 a day left warm. User also ran Unsloth's Dynamic GGUF 1-bit UD-IQ1_S (594 GB) on 8x A100-80GB via llama.cpp at about 9 tok/s with TTFT 7 to 60 s, $19.99 per hour and about $620 per million tokens, 3.3x more expensive per token. Quality at 1-bit was fine.

generation:

30.0 tokens/s

prompt processing (prefill):

750-910 tokens/s

User reports running Kimi K3 (2.8T) on a 16x GB10 cluster at ~30 t/s generation (peaking ~38 t/s) for coding and agentic tasks. Setup uses a customized gb10-vllm stack with custom MLA/KV kernels, dual MikroTik switches, and 4x 400G-to-4x100G breakout cables. Prefill throughput is ~750–910 t/s. Stable multi-hundred-thousand token context runs with multiple 500k compaction; handles concurrent requests without dropping token generation rates.

generation:

1.1 tokens/s

prompt processing (prefill):

5.7 tokens/s

quant:

UD-Q4_K_XL (GGUF)

User reports Kimi K3 at 1.05 t/s generation and 5.66 t/s prompt processing on an RTX 3090 with 2 TB of Intel Optane PMem 200 in Memory Mode. Setup uses a Supermicro X12SPA-TF with Intel Xeon 8370C, 256 GB DDR4-3200, and the UD-Q4_K_XL quant at 16k context. App Direct mode gave lower throughput at 0.62 t/s generation and 4.9 t/s prompt processing. The user also benchmarked Kimi K2.5, GLM-5.3-753B, Hermes-4-405B, DeepSeek-V4-Flash, Nemotron-3-Super-120B-A12B, gpt-oss-120b, and Ornith-1.5-35B-A3B on the same system.

generation:

20.0 tokens/s

prompt processing (prefill):

750.0 tokens/s

User reports the full Kimi K3 model running on a 16x DGX Spark cluster at an average of 20+ t/s, peaking at 38 t/s, with prefill at 750 t/s. This is a first run, and the user plans to optimize and publish a vLLM image.