Local model performance on your hardware

generation:

57.4 tokens/s

quant:

Q6_K (GGUF)

kv:

F16

User benchmarks Tencent's Hy3, a 295B MoE with 21B active, at 57.4 tok/s generation with the Q6_K GGUF on 8x RTX PRO 6000 Blackwell. Setup is llama.cpp llama-bench with layer split and F16 KV cache, with the MTP/NextN block excluded. On the same cards Q4_K_M runs at 67.3 tok/s, Q3_K_L at 63.3 tok/s and IQ2_M at 78.7 tok/s. The user's Nemotron-Labs-Audex-30B-A3B quants run at 287 (Q8_0), 334 (Q5_K_M), 345 (Q4_K_M) and 329 (MXFP4_MOE) tok/s on 2x RTX PRO 6000 Max-Q.

generation:

32.4 tokens/s

quant:

UD128 (GGUF)

kv:

Q8

User reports Tencent HY3 295B-A21B MoE at 32.4 t/s generation at empty context and 16.3 t/s at 16K context on Metal. Setup is llama.cpp with the UD128 quant (107 GB), a q8_0 KV cache, and 24K context. With MTP n=2, generation peaks at 38 t/s. Prefill runs 528 t/s at pp512 empty and 124 t/s at pp512 at 16K. User compares the result favorably to DeepSeek V4 Flash.