llamaperf
Oct 9, 2026
Throughput
69.9 t/s gen
Quant
IQ2_XXS (GGUF)
KV cache
Q8_0
VRAM reported
96 GB

Summary

User reports GLM-5.3-Flash at 69.89 t/s on an RTX PRO 6000 Blackwell 96 GB. Setup is llama.cpp with an IQ2_XXS GGUF and Q8_0 KV cache, fully resident on the card. Decode falls to 52.74 t/s at 8192 context, 31.70 at 32768 and 12.24 at 131072. The same file on a single RTX 3090 24 GB with 64 GB DDR5 and -ncmoe 40 gives 11.55 t/s decode and 36.69 t/s prefill. The IQ3_XXS build is not benchmarked.