Local model performance on your hardware

generation:

8.3 tokens/s

prompt processing (prefill):

444.2 tokens/s

quant:

UD-Q3_K_XL (GGUF)

kv:

q8_0

User benchmarks GLM-4.5-Air with Unsloth UD-Q3_K_XL quant on an RTX PRO 6000 96GB with CPU offload for some experts, using ik_llama.cpp at 16k context with q8_0 KV cache. Setup is ik_llama.cpp with UD-Q3_K_XL and q8_0 KV cache, offloading some expert layers to a Ryzen 9 7950X with 128GB DDR5-3600. At 12288 tokens of context, prompt processing reaches 444.23 t/s and generation 8.25 t/s. The user also compares mainline llama.cpp and an Ubergarm IQ3_KT quant, noting a batch warm-up quirk in the forked llama-sweep-bench.

generation:

40.0 tokens/s

prompt processing (prefill):

2000.0 tokens/s

quant:

IQ4 (GGUF)

kv:

Q8

User reports prefill around 2000 t/s and decode around 40 t/s for a single request. With 2-4 parallel requests, prefill drops to 500-1000 t/s and decode to 10-20 t/s. User asks for a better setup for 2-4 slots with q4 quants or better and q8 KV cache or better, and mentions trying vLLM recipes before.