- generation:
- 90.6 tokens/s
- quant:
- NVFP4 (NVFP4)
- kv:
- BF16
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports GLM-5.1 at 90.6 t/s single-stream decode on 8x RTX PRO 6000 Blackwell (96 GB each, 768 GB total). Setup is vLLM with an NVFP4 checkpoint (lukealonso/GLM-5.1-NVFP4-MTP), BF16 KV cache, DCP1, MTP speculative decoding on, 371456 KV cache tokens, max model length 202752. Aggregate decode reaches 1165.1 t/s at 64 concurrent requests; cold prefill is 4333 t/s at 8k context and 3170 t/s at 128k. DCP2 and DCP4 trade decode speed for larger KV cache, and DCP8 MTP-off was unstable at long context.