Local model performance on your hardware

generation:

23.4-25.2 tokens/s

prompt processing (prefill):

398-500 tokens/s

quant:

W4A16 GPTQ (GPTQ)

kv:

FP8

User benchmarks KAT-Coder-V2.5-Dev, a 35B MoE, on an Intel Arc 140V iGPU in an MSI Claw 8 AI+ handheld with 32 GB shared LPDDR5x. Setup is vLLM 0.31.0+xpu with W4A16 GPTQ weights, FP8 KV cache, 131072-token context and MTP speculative decoding with 2 draft tokens. GPTQ decodes at 23.4-25.2 tok/s, peaking at 29.1 tok/s, and finishes 164 HumanEval problems in 42m 12s versus 47m 33s for MXFP4. Prefill runs 51-140 tok/s on short prompts and 398-500 tok/s on 2k-4k prompts, with both quants hitting a 1,247.1 tok/s chunked-prefill ceiling. MTP accepts 2.55-2.71 tokens per step at a 77.3-82.5 acceptance rate, roughly doubling throughput from about 11 tok/s.

generation:

14.0 tokens/s

quant:

Q4_K_M (GGUF)

User reports KAT-Coder V2.5-Dev at 14 t/s on an 8-core CPU with no GPU. Setup is llama.cpp with Q4_K_M GGUF weights. The user finds the result impressive for a small model on a potato iGPU and pure CPU, completing a solar system demo in under 10 minutes.

generation:

14.2 tokens/s

User benchmarks a 25-task coding-agent benchmark on an RTX 3060 12GB. KAT-Coder completed 21/25 tasks and won. Qwen3.8-27B scored 13/25 at 14.73 tps, Nemotron 3.5 Lightning 30B-A3B 8/25 at 29.96 tps, Tiel-Coder 35B-A3B 3/25 strict at 20.57 tps, and Ternary-Bonsai-27B 0/25 at 59.21 tps. User plans to tweak settings to reach 18 tps.